Ephemeral Containers for Gitlab CI
An ephemeral container is a temporary container created when started and destroyed when stopped. There is no persistence between individual starts. In this tutorial, we will create such ephemeral containers using Debian 10 (Buster), SSH, Systemd's socket activation, and systemd-nspawn.
SSH Keys for Gitlab CI Runner
We will use the usual Gitlab CI Multi Runner with SSH Executor. Before we create the container, we will prepare SSH keys:
# ssh-keygen -f /etc/gitlab-runner/id_rsa
Creating the container
First, create a minimal system for the container:
# debootstrap --variant=minbase \ --include=dbus,systemd-container,openssh-server,build-essential,git,composer,rsync,wget,curl \ stable /var/lib/container/ephemeral-builder/
We will also need the public key for the Gitlab Runner; therefore, we should copy it into the container:
# cp /etc/gitlab-runner/id_rsa.pub /var/lib/container/ephemeral-builder/home/
Then, start the container in a non-ephemeral way, and create a builder user:
# systemd-nspawn -D /var/lib/container/ephemeral-builder # adduser builder --uid 9000 --gid 9000 --disabled-password
… Then move the SSH key to its place (note the rename):
# mkdir /home/builder/.ssh # mv /home/id_rsa.pub /home/builder/.ssh/authorized_keys # chown builder:builder -R /home/builder/.ssh
We also should remove some potential problems from the builder's profile files.
Make sure there is no
clear_console in the
and no Bash Completion or similar redundant stuff in the
CI builds may fail without explanation because of this.
Finally, we create a copy of the builder's home directory, so that we can easily copy it back into tmpfs:
# cp -ar /home/builder /home/builder.template
Now, the container is ready, and we can go back to the host system:
Socket activation is a lovely feature of Systemd, which basically replaces inetd. Systemd listens on defined sockets, and when it detects an incoming connection, it starts a defined service and passes the connection to the service.
We will create a TCP socket on localhost, port 24.
Create the following file in
[Unit] Description=Ephemeral builder container [Socket] ListenStream=127.0.0.1:24 Accept=true [Install] WantedBy=sockets.target
Then we need to define the service to start when the socket is activated.
Create the following file in
[Unit] Description=Ephemeral builder container [Service] StandardInput=socket ExecStart=-/bin/dash -c "exec /usr/bin/systemd-nspawn --quiet --read-only -M ephemeral-builder-$$$$ -D /var/lib/container/ephemeral-builder --link-journal=try-host --tmpfs=/home/builder /bin/dash -c 'mkdir /var/run/sshd ; cp -ar /home/builder.template/. /home/builder ; exec /usr/sbin/sshd -i'" CPUQuota=25% MemoryHigh=128M MemoryMax=256M TasksMax=64
ExecStart= must be a single line.
It is wrapped here for better readability only.
The trick in this service file is that we start
sshd -i inside a systemd-nspawn container.
The container has a unique name, thanks to the variable in the
expanded by the first dash instance.
The container's filesystem is read-only except a tmpfs mounted into the builder's home directory.
Also, we copy the template of the home directory into the tmpfs before starting the sshd.
Once the services are configured, we can enable them:
# systemctl enable ephemeral-builder.socket ephemeral-builder@.service # systemctl start ephemeral-builder.socket
We start only the socket because the socket will start the service when a connection arrives.
At this point, we should be able to log in into the container:
# ssh localhost -p 24 -i /etc/gitlab-runner/id_rsa -l builder
Also, we should see one container for each SSH connection we create:
# ssh localhost -p 24 -i /etc/gitlab-runner/id_rsa -l builder & # ssh localhost -p 24 -i /etc/gitlab-runner/id_rsa -l builder & # ssh localhost -p 24 -i /etc/gitlab-runner/id_rsa -l builder & # machinectl list MACHINE CLASS SERVICE OS VERSION ADDRESSES ephemeral-builder-2925 container systemd-nspawn debian 10 - ephemeral-builder-2945 container systemd-nspawn debian 10 - ephemeral-builder-2958 container systemd-nspawn debian 10 -
Registering the Runner
Our container behaves just like any other SSH server. Therefore, we register Gitlab Runner as usual:
# gitlab-runner register Running in system-mode. Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): https://git.example.com/ Please enter the gitlab-ci token for this runner: xybarbarfoofoobar123 Please enter the gitlab-ci description for this runner: [runner]: ephemeral-builder Please enter the gitlab-ci tags for this runner (comma separated): shell,container,composer Whether to run untagged builds [true/false]: [false]: true Whether to lock Runner to current project [true/false]: [false]: true Registering runner... succeeded runner=xfoobarfoo Please enter the executor: docker, parallels, ssh, virtualbox, docker+machine, docker-ssh+machine, docker-ssh, shell, kubernetes: ssh Please enter the SSH server address (e.g. my.server.com): localhost Please enter the SSH server port (e.g. 22): 24 Please enter the SSH user (e.g. root): builder Please enter the SSH password (e.g. docker.io): Please enter path to SSH identity file (e.g. /home/user/.ssh/id_rsa): /etc/gitlab-runner/id_rsa Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Because each started container has a unique name and lives only in memory,
we can run jobs in parallel.
concurrent option and the
limit option to
concurrent = 4 [[runners]] name = "ephemeral-builder" url = "https://git.example.com/" token = "xybarbarfoofoobar123" executor = "ssh" limit = 4 [runners.ssh] user = "builder" host = "localhost" port = "24" identity_file = "/etc/gitlab-runner/id_rsa" [runners.cache]
… And restart the Runner to apply the changes in the configuration:
# systemctl restart gitlab-ci-multi-runner.service
Fun with the Container
Because the container gets created when SSH connects and destroyed on disconnect, we must maintain the SSH connection as long as we need the container. However, we cannot connect multiple times to a single container because each connection gets its own container. To overcome this difficulty, we can utilize the multiplexing feature of OpenSSH:
#!/bin/sh ssh_ctl="/tmp/ssh.master.$$" ssh="ssh localhost -p 24" set -e echo "> Connecting ..." $ssh -M -S "$ssh_ctl" -f -N $ssh -S "$ssh_ctl" -O check echo "> Connected." $ssh -S "$ssh_ctl" echo "Hello world" $ssh -S "$ssh_ctl" date echo "> Disconnecting ..." $ssh -S "$ssh_ctl" -O exit $ssh -S "$ssh_ctl" -O check
This simple script creates a single connection to our container and executes two commands, both in the same ephemeral container.
The Gitlab Runner works on the same principle. Its SSH Executor creates a single connection to a server, the ephemeral container in our case, and executes the commands via that one connection. Another instance of the runner creates its own SSH connection, and thus it creates its own container.
Systemd-nspawn with a socket–activated SSH provides a lightweight way to create ephemeral containers where we can run our Gitlab CI pipelines. It may be a convenient alternative to Docker containers. While it is not as universal, it is much better integrated with the host system, which may be useful when dealing with unusual scenarios.