Ephemeral Containers for Gitlab CI

An ephemeral container is a temporary container created when started and destroyed when stopped. There is no persistence between individual starts. In this tutorial, we will create such ephemeral containers using Debian 10 (Buster), SSH, Systemd's socket activation, and systemd-nspawn.

SSH Keys for Gitlab CI Runner

We will use the usual Gitlab CI Multi Runner with SSH Executor. Before we create the container, we will prepare SSH keys:

# ssh-keygen -f /etc/gitlab-runner/id_rsa

Creating the container

First, create a minimal system for the container:

# debootstrap --variant=minbase \
    --include=dbus,systemd-container,openssh-server,build-essential,git,composer,rsync,wget,curl \
    stable /var/lib/container/ephemeral-builder/

We will also need the public key for the Gitlab Runner; therefore, we should copy it into the container:

# cp /etc/gitlab-runner/id_rsa.pub /var/lib/container/ephemeral-builder/home/

Then, start the container in a non-ephemeral way, and create a builder user:

# systemd-nspawn -D /var/lib/container/ephemeral-builder
# adduser builder --uid 9000 --gid 9000 --disabled-password

… Then move the SSH key to its place (note the rename):

# mkdir /home/builder/.ssh
# mv /home/id_rsa.pub /home/builder/.ssh/authorized_keys
# chown builder:builder -R /home/builder/.ssh

We also should remove some potential problems from the builder's profile files. Make sure there is no clear_console in the /home/builder/.bash_logout file, and no Bash Completion or similar redundant stuff in the ~/.bashrc file. CI builds may fail without explanation because of this.

Finally, we create a copy of the builder's home directory, so that we can easily copy it back into tmpfs:

# cp -ar /home/builder /home/builder.template

Now, the container is ready, and we can go back to the host system:

# exit

Socket Activation

Socket activation is a lovely feature of Systemd, which basically replaces inetd. Systemd listens on defined sockets, and when it detects an incoming connection, it starts a defined service and passes the connection to the service.

We will create a TCP socket on localhost, port 24. Create the following file in /etc/systemd/ephemeral-builder.socket:

[Unit]
Description=Ephemeral builder container

[Socket]
ListenStream=127.0.0.1:24
Accept=true

[Install]
WantedBy=sockets.target

Then we need to define the service to start when the socket is activated. Create the following file in /etc/systemd/system/ephemeral-builder@.service:

[Unit]
Description=Ephemeral builder container

[Service]
StandardInput=socket
ExecStart=-/bin/dash -c "exec /usr/bin/systemd-nspawn --quiet --read-only
    -M ephemeral-builder-$$$$ -D /var/lib/container/ephemeral-builder
    --link-journal=try-host --tmpfs=/home/builder
    /bin/dash -c 'mkdir /var/run/sshd ;
        cp -ar /home/builder.template/. /home/builder ;
        exec /usr/sbin/sshd -i'"

CPUQuota=25%
MemoryHigh=128M
MemoryMax=256M
TasksMax=64

Warning: The ExecStart= must be a single line. It is wrapped here for better readability only.

The trick in this service file is that we start sshd -i inside a systemd-nspawn container. The container has a unique name, thanks to the variable in the -M argument expanded by the first dash instance. The container's filesystem is read-only except a tmpfs mounted into the builder's home directory. Also, we copy the template of the home directory into the tmpfs before starting the sshd.

Once the services are configured, we can enable them:

# systemctl enable ephemeral-builder.socket ephemeral-builder@.service
# systemctl start ephemeral-builder.socket

We start only the socket because the socket will start the service when a connection arrives.

At this point, we should be able to log in into the container:

# ssh localhost -p 24 -i /etc/gitlab-runner/id_rsa -l builder

Also, we should see one container for each SSH connection we create:

# ssh localhost -p 24 -i /etc/gitlab-runner/id_rsa -l builder  &
# ssh localhost -p 24 -i /etc/gitlab-runner/id_rsa -l builder  &
# ssh localhost -p 24 -i /etc/gitlab-runner/id_rsa -l builder  &
# machinectl list
MACHINE                CLASS     SERVICE        OS     VERSION ADDRESSES
ephemeral-builder-2925 container systemd-nspawn debian 10      -
ephemeral-builder-2945 container systemd-nspawn debian 10      -
ephemeral-builder-2958 container systemd-nspawn debian 10      -

Registering the Runner

Our container behaves just like any other SSH server. Therefore, we register Gitlab Runner as usual:

# gitlab-runner register
Running in system-mode.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://git.example.com/
Please enter the gitlab-ci token for this runner:
xybarbarfoofoobar123
Please enter the gitlab-ci description for this runner:
[runner]: ephemeral-builder
Please enter the gitlab-ci tags for this runner (comma separated):
shell,container,composer
Whether to run untagged builds [true/false]:
[false]: true
Whether to lock Runner to current project [true/false]:
[false]: true
Registering runner... succeeded                     runner=xfoobarfoo
Please enter the executor: docker, parallels, ssh, virtualbox, docker+machine, docker-ssh+machine, docker-ssh, shell, kubernetes:
ssh
Please enter the SSH server address (e.g. my.server.com):
localhost
Please enter the SSH server port (e.g. 22):
24
Please enter the SSH user (e.g. root):
builder
Please enter the SSH password (e.g. docker.io):

Please enter path to SSH identity file (e.g. /home/user/.ssh/id_rsa):
/etc/gitlab-runner/id_rsa
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Because each started container has a unique name and lives only in memory, we can run jobs in parallel. Add the concurrent option and the limit option to /etc/gitlab-runner/config.toml:

concurrent = 4

[[runners]]
  name = "ephemeral-builder"
  url = "https://git.example.com/"
  token = "xybarbarfoofoobar123"
  executor = "ssh"
  limit = 4
  [runners.ssh]
    user = "builder"
    host = "localhost"
    port = "24"
    identity_file = "/etc/gitlab-runner/id_rsa"
  [runners.cache]

… And restart the Runner to apply the changes in the configuration:

# systemctl restart gitlab-ci-multi-runner.service

Fun with the Container

Because the container gets created when SSH connects and destroyed on disconnect, we must maintain the SSH connection as long as we need the container. However, we cannot connect multiple times to a single container because each connection gets its own container. To overcome this difficulty, we can utilize the multiplexing feature of OpenSSH:

#!/bin/sh
ssh_ctl="/tmp/ssh.master.$$"
ssh="ssh localhost -p 24"
set -e

echo "> Connecting ..."

$ssh -M -S "$ssh_ctl" -f -N
$ssh -S "$ssh_ctl" -O check

echo "> Connected."

$ssh -S "$ssh_ctl" echo "Hello world"
$ssh -S "$ssh_ctl" date

echo "> Disconnecting ..."

$ssh -S "$ssh_ctl" -O exit
$ssh -S "$ssh_ctl" -O check

This simple script creates a single connection to our container and executes two commands, both in the same ephemeral container.

The Gitlab Runner works on the same principle. Its SSH Executor creates a single connection to a server, the ephemeral container in our case, and executes the commands via that one connection. Another instance of the runner creates its own SSH connection, and thus it creates its own container.

Conclusion

Systemd-nspawn with a socket–activated SSH provides a lightweight way to create ephemeral containers where we can run our Gitlab CI pipelines. It may be a convenient alternative to Docker containers. While it is not as universal, it is much better integrated with the host system, which may be useful when dealing with unusual scenarios.