Deploying CI-tron

The CI-tron project is shipped as a container image which can be run on existing infrastructure, or booted directly using Boot2container either in a virtual machine or netbooted over the internet using the Boot2ipxe server.

Security considerations

Danger

You are about to deploy a system that potentially lets people on the internet run their own code on your machines. If done improperly, this presents a serious risk for the confidentiality of your data / identity, and may be used to commit crimes.

Please read this section very carefully before moving any further.

CI-tron makes it really easy to control your machines, which means it can be very easy for bad actors to use your infrastructure to spam, scam, DDoS other web services, or use your resources to mine cryptocurrencies.

To reduce the chances of this happening, CI-tron encourages the right behavior:

  • The CI-tron gateway’s firewall drops all traffic towards the internet that is not HTTP(s), SSH, Wireguard, or SNMP;

  • Test machines should be plugged into a separate network (the private network) do not have unmediated Internet access;

  • Control equiment such as PDUs should be plugged into a separate network (the mgmt network), and not have access to the internet nor the DUTs.

    graph LR
        internet[fa:fa-globe Internet] <-->|fa:fa-sitemap eth0| gateway[fa:fa-server Testing<br>Gateway]
        gateway <--> |fa:fa-sitemap eth1 / Port 1 | switch[fa:fa-sitemap Private<br>Network Switch]
        gateway <--> |fa:fa-sitemap eth2 | pdu[fa:fa-sitemap Industrial PDU]
        gateway <--> |fa:fa-wifi WiFi AP| smartplug[fa:fa-wifi Smart plug]
        gateway <--> |fa:fa-usb USB cable | usbhub[fa:fa-usb USB Hub with PPPS]
    
        pdu <--> |fa:fa-plug Power cable| dut1[fa:fa-desktop DUT 1]
        switch <--> |fa:fa-sitemap Port 2| dut1
        smartplug <--> |fa:fa-plug Power cable| dut2[fa:fa-laptop DUT 2]
        switch <--> |fa:fa-sitemap Port 3| dut2
        usbhub <--> |fa:fa-usb Port 1| dut3[fa:fa-mobile-phone DUT 3]

    Expected network configuration

Requirements

The CI-tron gateway container is meant to be run on the gateway machine of a CI farm, and comes with the following requirements:

Hardware Requirements

For optimal performance when serving 20-30 DUTs, we recommend the following hardware specifications for the gateway machine:

  • CPU: 16-core processor

  • Memory: 32 GB RAM

  • Storage: 1 TB NVMe SSD (fast storage is crucial for /config and /cache volumes, especially when serving network boot images and build artifacts)

  • Network: One Ethernet network interface, connected to the internet, with the following optional interfaces:

    • WiFi adapters: Will be used to create a WiFi network with the wanted SSID and passphrase, and then added to the mgmt network. This is meant for PDUs (see Adding the PDU to your CI-tron instance), or allowing local SSH access.

    • Ethernet adapters: Any extra Ethernet network interface will be added to the private network, which is used to connect test machines to the CI-tron gateway. We recommend guaranteeing 100 Mbps of bandwidth per DUT by using multi-gig NICs and spreading DUTs across multiple NICs/switches.

Container Requirements

  • Volumes: The container requires a volume to store persistent data (mounted at /config), and optionally a temporary volume which acts as a cache across reboots (mounted at /cache).

  • Container: The container provides network services for multiple network protocols (DHCP, DNS, TFTP, HTTP, …), and thus requires to be run as a privileged container using --network host, or the following:

    • Capabilities:

      • CAP_NET_ADMIN: For binding ports < 1024

      • CAP_NET_RAW: For our DHCP server

      • CAP_AUDIT_CONTROL: For logind, to be able to log-in using SSH

    • Network:

      • --network bridge: The network the container uses to access the internet , AKA the public network;

      • podman network create -d macvlan -o parent=$PRIVATE_NIC -o mode=passthru -o no_default_route=1 --ipam-driver=none ci-tron-private-nic1 and --network ci-tron-private-nic1: The private interface you want to share to use for testing (repeat as needed).

The latest CI-tron container can be accessed using registry.freedesktop.org/gfx-ci/ci-tron/gateway:latest, but you may want to pin it to a certain version. Until we start creating proper releases, you will find all the different versions of CI-tron in our container registry.

Deploying CI-tron

CI-tron is meant to provide a plug-and-play experience, on a dedicated machine. As such, it is best deployed as the sole application on a dedicated machine.

It can however operate with a reduced feature-set on already-existing infrastructure if needs be.

Let’s review your options!

Netbooting CI-tron without a host operating system (recommended)

The best way to deploy a CI-tron instance is to netboot it using an Boot2ipxe server. This self-hostable service provides:

  • per-machine boot images: for various architectures and single-board computers, flashable on USB pendrives (recommended) or disk;

  • automatic backup/restore: move the pendrive to another machine, and it will restore the configuration automatically;

  • offline config editing: never get locked out after a failed update;

  • access control: allow others to deploy new kernels when you are on holiday.

Here is a list of public servers you may use:

Booting a boot image in QEMU
$ fallocate -l 10G disk.img
$ qemu-system-x86_64 -enable-kvm -no-reboot \
  -M q35 -smp 4 -m size=2048 \
  -nic user,model=e1000,hostfwd=tcp::2222-:22 \
  -device usb-tablet \
  -vga none -device virtio-vga-gl,hostmem=4G,blob=true,venus=true \
  -display gtk,gl=on,grab-on-hover=on \
  -drive format=raw,file=${OUTPUT_IMAGE} \
  -drive format=raw,file=disk.img
Netbooting without using the dedicated boot servers

Warning

This method does not provide any automatic backup/restore, nor any editing of the ci-tron configuration. You may thus accidentally lock yourself out and be unable to fix it without physical access.

If for some reason you do not want to use any public Boot2ipxe server instance nor re-deploy it yourself, you may want to reproduce parts of what the service is doing by using Boot2container, with the following configuration expressed as an iPXE script:

#!ipxe

# Set the defaults for the gateway
set farm_name my_farm
set gateway_cache_device auto
set gateway_extra_cmdline
set gateway_img_path registry.freedesktop.org/gfx-ci/ci-tron/gateway:latest

set default_cmdline initrd=initrd.magic net.ifnames=0 b2c.ntp_peer=auto b2c.pipefail b2c.swap=32G b2c.volume="ci-tron-cache" b2c.volume="ci-tron-config" b2c.hostname=${farm_name}-gateway b2c.run="-ti --pull=always -v ci-tron-cache:/cache -v ci-tron-config:/config -v /dev:/dev -v /lib/modules:/lib/modules -v /lib/firmware:/lib/firmware docker://${gateway_img_path}" b2c.cache_device="${gateway_cache_device}" b2c.shutdown_cmd="reboot -f" ${gateway_extra_cmdline}

kernel https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/linux-x86_64 ${default_cmdline}
initrd https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/initramfs.linux_amd64.cpio.xz
initrd https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/linux-x86_64.depmod.cpio.xz
initrd https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/linux-x86_64.gpu.cpio
initrd https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/linux-x86_64.wifi.cpio
initrd https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/linux-x86_64.ucode.cpio  # For Intel-based machines
boot

Rather than embedding the above script in iPXE directly (see iPXE Crypto for more details), you could store it in a public git repository and then source it from another embedded script which would never need to change:

#!ipxe

set boot_script_url https://gitlab.freedesktop.org/mupuf/ci-tron-boot-cfg/-/raw/main/my_farm.ipxe

echo Welcome to the CI-tron iPXE boot script

:retry
echo Acquiring an IP
dhcp || goto retry # Keep retrying indefinitely
echo Got the IP: ${netX/ip} / ${netX/netmask}

echo

echo Updating the current time using NTP
ntp pool.ntp.org || goto retry
echo Current unixtime: ${unixtime} (use `date --date=@$$((${unixtime}))` to decode it)

echo

echo Chainloading from the iPXE server ${IPXE_SERVER_FQDN} ...
chain ${boot_script_url} || goto retry

# The above command may "succeed" but we actually fail to
# boot. This could happen if the iPXE boot file is
# successfully returned, but the URLs to the kernel and
# ramdisk are invalid, for example. In cases like these,
# continuously retry the netboot, rather than exiting iPXE and
# potentially getting stuck indefinitely in the firmware's next
# boot method. The sleep acts as a simple rate limiter.
sleep 60
goto retry

Once your container has booted and shows the dashboard, you will want to add your SSH keys to /config/ssh/authorized_keys by killing the dashboard (CTRL+Q) then typing nano /config/ssh/authorized_keys.

On an existing gateway, but letting CI-tron take over everything

Warning

This mode is not very well supported, as the services of your existing system may conflict with the ones provided by CI-tron.

Work to improve support for this mode is welcome, and slowly happening

Assuming you already have internet connectivity in your dedicated machine, the container should be executed in this fashion, all as root:

# FARM_NAME=myfarm

# # Create the volumes that will hold the config and caches
# podman volume create ci-tron-config
# podman volume create ci-tron-cache

# podman run --rm -ti --pull=always --privileged \
      --name ci-tron --network host --hostname $FARM_NAME-gateway \
      -v ci-tron-config:/config -v ci-tron-cache:/cache \
      registry.freedesktop.org/gfx-ci/ci-tron/gateway:latest
On an existing gateway, as a rootless unprivileged container

Unsupported features:

  • ❌ Physical (non-virtual) DUTs

  • ❌ Chronyd (NTP)

  • ❌ NFS / imagestore

  • ❌ Gitlab support (nested podman)

$ # Create the volumes that will hold the config and caches
$ podman volume create ci-tron-config
$ podman volume create ci-tron-cache

$ podman run --rm -ti --pull=always                \
    --cap-add CAP_NET_ADMIN                        \
    --cap-add CAP_NET_RAW                          \
    --cap-add CAP_AUDIT_CONTROL                    \
    --device /dev/fuse                             \
    --device /dev/net/tun                          \
    --device /dev/kvm                              \
    --name ci-tron --hostname vivian-mupuf-gateway \
    -v config:/config -v cache:/cache              \
    --network bridge                               \
    -p 60022:22                                    \
    -p 8000:80                                     \
    -p 8006:8006                                   \
    registry.freedesktop.org/gfx-ci/ci-tron/gateway:latest

Once your container has booted and shows the dashboard, you will want to add your SSH keys to the container for remote access. You may do so by mounting the config volume using podman unshare podman volume mount ci-tron-config and then editing $MOUNT_POINT/ssh/authorized_keys.

On an existing gateway, as a rootfull unprivileged container

Unsupported features:

  • ❌ Dynamic network interfaces (needed for ethernet over USB, for Fastboot devices)

  • ❌ Hardware-based serial console devices (needs an external service)

  • ❌ NFS / imagestore

  • ❌ Gitlab support (nested podman)

# # Allow ip forwarding, since we can't do that from inside the container
# # without requiring insane privileges. May not even be needed.
# echo 1 > /proc/sys/net/ipv4/ip_forward

# Create the networks
# podman network create -d macvlan -o parent=$PRIVATE_NIC -o mode=passthru -o no_default_route=1 --ipam-driver=none ci-tron-private-nic
# podman network create -d macvlan -o parent=$MGMT_NIC -o mode=passthru -o no_default_route=1 --ipam-driver=none ci-tron-mgmt-nic

# Create the volumes that will hold the config and caches
# podman volume create ci-tron-config
# podman volume create ci-tron-cache

# # TODO: add support for the USB-based PDUs
# # TODO: write an external service that forwards consoles
# # to SALAD using socat or similar
# podman run --rm -ti --pull=always                   \
    --cap-add CAP_NET_ADMIN                           \
    --cap-add CAP_NET_RAW                             \
    --cap-add CAP_AUDIT_CONTROL                       \
    --cap-add CAP_SYS_TIME                            \
    --device /dev/fuse                                \
    --device /dev/net/tun                             \
    --device /dev/kvm                                 \
    --name ci-tron --hostname vivian-mupuf-gateway    \
    -v ci-tron-config:/config -v ci-tron-cache:/cache \
    --network bridge                                  \
    --network ci-tron-private-nic                     \
    -p 60022:22                                       \
    -p 8000:80                                        \
    -p 8006:8006                                      \
    registry.freedesktop.org/gfx-ci/ci-tron/gateway:latest

Once your container has booted and shows the dashboard, you will want to add your SSH keys to the container for remote access. You may do so by mounting the config volume using podman unshare podman volume mount ci-tron-config and then editing $MOUNT_POINT/ssh/authorized_keys.

In vivian, for local development of CI-tron

Unsupported features:

  • ❌ Physical (non-virtual) DUTs

If you want to develop CI-tron, you could use built-in test environment: vivian.

$ cd ~/src/ci-tron
$ make vivian

You may then use $ make vivian-connect to get a shell inside the CI-tron container, that you can then use to edit the config files, or shutting down the virtual machine using poweroff.

You are now ready to go on to the quick guide :)