Please note that this post is a linear and unedited brain dump of what I did. Many things might have changed meanwhile, and I may have learned how to do things better. This is an experiment in progress.

This is a continuation of [[lass week’s post /posts/2016-03-28-cooperating-with-travisci.mdwn]], refreshing my exiscan module using retrospec-puppet. In the last installment I managed to get the full-system tests running in docker. Yay! It turns out that the SUT from within the docker “container” changes my sound settings and causes weird interactions with my desktop environment. Boo! Isolating myself from the contents of the container will be the goal for today. Since it’s the 15th anniversary of finding my significant other, I’ll spend most of the day at the Festival of Colours.

Isolating the “container”

Of course, I am aware that I’m using docker not in the way it was designed for: I’m booting a “complete” system(d), instead of just “the one process” of the application. On the other hand, the security story of “do not run things that will do unspeakable things to your kernel”, is not very convincing.

Beaker as hypervisor manager doesn’t allow passing arbitrary options to docker, so any of the possibilities of enabling seccomp profiles or disabling capabilities are impossible to use without patching beaker. Additionally it would mean to handcraft a policy that disallows futzing with my desktop (plausible for the audio, a unknown for the colord issue, and who knows what else?). Alternatively, I could just run docker within a VM (yay vagrant) and get proper containment of the system running in docker.

Here’s the relevant vagrant provision command to setup a puppetlabs/debian-8.2-64-nocm box to be able to run beaker in docker on that VM. This has also the advantage, that anyone following along at home gets a pristine, knwon-good environment to run all of this.

config.vm.provision "shell", inline: <<-SHELL
  # do not use dash's built-in echo, which doesn't understand -e
  /bin/echo -e "en_US.UTF-8 UTF-8\nde_AT.UTF-8 UTF-8" > /etc/locale.gen
  locale-gen

  apt-get update
  apt-get install -y apt-transport-https ca-certificates

  echo deb https://apt.dockerproject.org/repo debian-jessie main > /etc/apt/sources.list.d/docker.list
  apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

  apt-get update
  apt-get install -y docker-engine ruby-dev ruby bundler build-essential libxml2-dev zlib1g-dev

  systemctl enable docker
  systemctl start docker

  adduser vagrant docker

  su - vagrant -c 'echo "cd /vagrant" >> /home/vagrant/.bashrc;
    echo "exec sudo -i" >> /home/vagrant/.bash_history;
    echo "BEAKER_provision=no BEAKER_destroy=no PUPPET_INSTALL_TYPE=agent BEAKER_set=docker/debian-8 bundle exec rake beaker" >> /home/vagrant/.bash_history;
    cd /vagrant;
    bundle install --path=/tmp/bundle;
    PUPPET_INSTALL_TYPE=agent BEAKER_set=docker/debian-8 bundle exec rake beaker'
SHELL

I used vagrant init puppetlabs/debian-8.2-64-nocm in a temporary directory and added the above commands to the generated Vagrantfile to get a box that had all the necessary software installed. In the end I added a few commands to the bash history and installed the required gems, so I have them ready when the machine is finalized.

Waiting for gems to compile, I figured adding a little bit more oomph to the box would help too. Researching the exact incantation I also found the linked_clone option, which should help when recycling the box.

config.vm.provider "virtualbox" do |vb|
  vb.cpus = 4
  vb.memory = "2048"
  vb.linked_clone = true
end

While the gems were compiling (the next time after adding more -dev packages) I considered storing the compiled gems in a persistent folder outside of the box. Turns out this is as easy as:

config.vm.synced_folder "/tmp/vbox_gems", "/tmp/bundle"

Depending on your system setup, /tmp might be a ram disk. In that case you might want to use a different place.

All of this together means that I can run the beaker tests with vagrant ssh, enter, up, enter. Building the initial docker file also takes a bit of time, so I add a initial beaker run to the vagrant provision step. (See above, there are limits to my no-editing/braindump rule)

Preparing the docker image also takes its time. I guess I’ll mount /var/lib/docker for persisting the docker images.

The first full run failed on

==> default:   Error: Evaluation Error: Error while evaluating a Resource Statement, Could not find declared class exiscan at /tmp/apply_manifest.pp.ndZ2Lh:1:7 on node debian-8-x64

after 20 minutes.

To debug this, look at docker ps output and connect to that port with ssh. The default password setup by beaker is root.

vagrant@localhost:/vagrant$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                   NAMES
069b9b964ede        3e3ff809749a        "/sbin/init"        2 minutes ago       Up 2 minutes        0.0.0.0:32769->22/tcp   nauseous_hopper
vagrant@localhost:/vagrant$ ssh root@localhost -p 32769
The authenticity of host '[localhost]:32769 ([::1]:32769)' can't be established.
ECDSA key fingerprint is 81:ff:c9:01:3f:4d:7b:93:a7:a1:e2:af:3c:4f:ef:a9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:32769' (ECDSA) to the list of known hosts.
root@localhost's password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@debian-8-x64:~#

One thing that is always annoying with the stripped down systems, like docker images, is missing locales. I thought I had already fixed that by installing the locales package and genning the data for the en_US.UTF-8 locale, but there is something more going on, keeping the locale data missing, even after reinstalling the locales package. A divert perhaps? Nope:

root@debian-8-x64:~# dpkg-divert --list
local diversion of /sbin/initctl to /sbin/initctl.distrib
diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash
diversion of /bin/sh to /bin/sh.distrib by dash
root@debian-8-x64:~#

It turns out that I was completely confused about the actual syntax of the locale.gen (already fixed above). Leave me alone in my shame!

Back to the original problem of the exiscan class not being found. This does not look healthy.

root@debian-8-x64:/etc/puppetlabs/code/modules# ls
exiscan			       vagrantfilesdefaultconf.dretry	   vagrantfilesgreylistexim4conf.dmain	vagrantfilesspamassassinpostgres
tp			       vagrantfilesdefaultconf.drewrite    vagrantfilesgreylistpostgres		vagrantjunit
vagrantfiles		       vagrantfilesdefaultconf.drouter	   vagrantfilesscanner			vagrantjunitdebian-8
vagrantfilesdefault	       vagrantfilesdefaultconf.dtransport  vagrantfilesscanneracls		vagrantjunitdefault
vagrantfilesdefaultconf.d      vagrantfilesgreylist		   vagrantfilesscannerconf.d		vagrantmanifests
vagrantfilesdefaultconf.dacl   vagrantfilesgreylistexim4	   vagrantfilesscannerconf.dacl		vagranttemplates
vagrantfilesdefaultconf.dauth  vagrantfilesgreylistexim4conf.d	   vagrantfilesscannerconf.dmain
vagrantfilesdefaultconf.dmain  vagrantfilesgreylistexim4conf.dacl  vagrantfilesspamassassin
root@debian-8-x64:/etc/puppetlabs/code/modules#

It does appear to be a weird issue with copying the module into the docker container. The weird thing is how apparently all the slashes seem to be missing from the filenames, e.g. vagrantfilesgreylistexim4conf.dmain should be /vagrant/files/greylist/exim4/conf.d/main. Looking at that, even more suspicious is that these are only the directories, and none of the contained files. Maybe beaker or scp gets confused by copying from /vagrant? I’ll run an experiment from /tmp/foo/bar.

On the performance side, the next thing I’d need to look into is having a local mirrors of debian, puppetlabs, and rubygems. But that may be a problem for another weekend, meanwhile I’ll bask in the speed of my home cable downlink. Thinking more about this, replacing the puppetlabs’ repo will be a pain, since the list file is installed by beaker internals. ngngngng.

Meanwhile the testrun where I copied the module to /tmp/foo/bar finished successfully. Let’s see if there is anything “obvious” in beaker’s source. The scp_to function looks straight forward enough. @ssh is a Net::SSH object, but the documentation doesn’t seem to contain any references to scp. Looking into the gemspec I realize that there is a separate net-scp gem. (“this project is in maintenance mode”) Helpfully, there is a progress block attached to beaker’s call of the scp. Its output is stored in the usual beaker result object, so let’s check that output in our setup code.

result = copy_module_to(host, :source => proj_root, :module_name => 'exiscan')
puts result.stdout

Again I run afoul of my unfounded optimism. Looking at the copy_module_to source, the scp_to result is not passed up the call chain. In these cases I usually start modifying the installed gem source that is used by bundler. Having a separate install dir for the vbox makes that not as terrible as it sounds at first. Having that directory persistent across boxes is both a boon and an annoyance: I don’t lose my local changes, but when I forget to clean up, I’ll confuse the heck out of future-me. Except that the retrospec templates install beaker from git, which puts it somewhere else. I remove the git references, since the recent releases are just fine anyways. Trying again.

The debug output is not very helpful:

copying /vagrant/Gemfile.lock:          0/8562
copying /vagrant/Gemfile.lock:       8562/8562
SCP'ed file /vagrant/Gemfile.lock to debian-8-x64:/etc/puppetlabs/code/modules/vagrant

Yes. that’s all. Maybe the chunk size of the ssh connection is interfering? I spy a rsync transfer protocol implementation in the same method, so I try that out before continuing down the scp rabbit hole.

copy_module_to(host, source: proj_root, module_name: 'exiscan', protocol: 'rsync')

411 kB/s is not enough download speed for the debian packages. :-(

rsync requires manual password entry, and then fails, because the target container doesn’t have rsync installed.

Another dead end. Instead of further fighting these demons, I simply switch out the mount on /vagrant to /tmp/exiscan. I bet you it’ll turn out that some weirdness of the vboxsf, and not the path, was the issue when copying. -.-

I’ve also cleaned out the bundler cache to revert my changes to beaker. I’ll not wait for another 20 minutes until this box has built. There is literally not enough time in this day left: it’s after 23:00.

Good night.

Update

Rebuilding the box suddenly failed with docker complaining about:

Apr 03 15:18:56 localhost docker[29239]: time="2016-04-03T15:18:56.491483351-07:00" level=fatal msg="Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to get bridge network configurations from store: error while populating kmap: invalid argument"

Why do I even bother??