================= Deploying CI-tron ================= The ``CI-tron`` project is shipped as a container image which can be run on existing infrastructure, or booted directly using `Boot2container`_ either in a virtual machine or netbooted over the internet using the `Boot2ipxe server`_. .. _Boot2container: https://gitlab.freedesktop.org/gfx-ci/boot2container .. _Boot2ipxe server: https://gitlab.freedesktop.org/mupuf/boot2ipxe Security considerations ======================= .. danger:: You are about to deploy a system that potentially lets people on the internet run their own code on your machines. If done improperly, this presents a serious risk for the confidentiality of your data / identity, and may be used to commit crimes. Please read this section **very carefully** before moving any further. CI-tron makes it really easy to control your machines, which means it can be very easy for bad actors to use your infrastructure to spam, scam, DDoS other web services, or use your resources to mine cryptocurrencies. To reduce the chances of this happening, CI-tron encourages the right behavior: * The CI-tron gateway's firewall drops all traffic towards the internet that is not HTTP(s), SSH, Wireguard, or SNMP; * Test machines should be plugged into a separate network (the ``private`` network) do not have unmediated Internet access; * Control equiment such as :ref:`PDUs ` should be plugged into a separate network (the ``mgmt`` network), and not have access to the internet nor the DUTs. .. mermaid:: :fit: contain :align: center :alt: Network configuration: Internet <-- Public NIC --> Testing Gateway <-- Private NIC --> DUT Switch :caption: Expected network configuration graph LR internet[fa:fa-globe Internet] <-->|fa:fa-sitemap eth0| gateway[fa:fa-server Testing
Gateway] gateway <--> |fa:fa-sitemap eth1 / Port 1 | switch[fa:fa-sitemap Private
Network Switch] gateway <--> |fa:fa-sitemap eth2 | pdu[fa:fa-sitemap Industrial PDU] gateway <--> |fa:fa-wifi WiFi AP| smartplug[fa:fa-wifi Smart plug] gateway <--> |fa:fa-usb USB cable | usbhub[fa:fa-usb USB Hub with PPPS] pdu <--> |fa:fa-plug Power cable| dut1[fa:fa-desktop DUT 1] switch <--> |fa:fa-sitemap Port 2| dut1 smartplug <--> |fa:fa-plug Power cable| dut2[fa:fa-laptop DUT 2] switch <--> |fa:fa-sitemap Port 3| dut2 usbhub <--> |fa:fa-usb Port 1| dut3[fa:fa-mobile-phone DUT 3] Requirements ============ The CI-tron gateway container is meant to be run on the gateway machine of a CI farm, and comes with the following requirements: Hardware Requirements --------------------- For optimal performance when serving 20-30 DUTs, we recommend the following hardware specifications for the gateway machine: - **CPU**: 16-core processor - **Memory**: 32 GB RAM - **Storage**: 1 TB NVMe SSD (fast storage is crucial for ``/config`` and ``/cache`` volumes, especially when serving network boot images and build artifacts) - **Network**: One Ethernet network interface, connected to the internet, with the following optional interfaces: - **WiFi adapters**: Will be used to create a WiFi network with the wanted SSID and passphrase, and then added to the ``mgmt`` network. This is meant for PDUs (see :ref:`PDU`), or allowing local SSH access. - **Ethernet adapters**: Any extra Ethernet network interface will be added to the ``private`` network, which is used to connect test machines to the CI-tron gateway. We recommend guaranteeing 100 Mbps of bandwidth per DUT by using multi-gig NICs and spreading DUTs across multiple NICs/switches. Container Requirements ---------------------- - **Volumes**: The container requires a volume to store persistent data (mounted at ``/config``), and optionally a temporary volume which acts as a cache across reboots (mounted at ``/cache``). - **Container**: The container provides network services for multiple network protocols (DHCP, DNS, TFTP, HTTP, ...), and thus requires to be run as a privileged container using ``--network host``, or the following: - `Capabilities `_: - **CAP_NET_ADMIN**: For binding ports < 1024 - **CAP_NET_RAW**: For our DHCP server - **CAP_AUDIT_CONTROL**: For logind, to be able to log-in using SSH - Network: - ``--network bridge``: The network the container uses to access the internet , AKA the ``public`` network; - ``podman network create -d macvlan -o parent=$PRIVATE_NIC -o mode=passthru -o no_default_route=1 --ipam-driver=none ci-tron-private-nic1`` and ``--network ci-tron-private-nic1``: The private interface you want to share to use for testing (repeat as needed). The latest CI-tron container can be accessed using ``registry.freedesktop.org/gfx-ci/ci-tron/gateway:latest``, but you may want to pin it to a certain version. Until we start creating proper releases, you will find all the different versions of CI-tron in our `container registry `_. Deploying CI-tron ================= CI-tron is meant to provide a plug-and-play experience, on a dedicated machine. As such, it is best deployed as the sole application on a dedicated machine. It can however operate with a reduced feature-set on already-existing infrastructure if needs be. Let's review your options! .. accordion:: .. dropdown:: Netbooting CI-tron without a host operating system (recommended) :name: deploy_boot2ipxe :open: The best way to deploy a CI-tron instance is to netboot it using an `Boot2ipxe server`_. This self-hostable service provides: * **per-machine boot images**: for various architectures and single-board computers, flashable on USB pendrives (recommended) or disk; * **automatic backup/restore**: move the pendrive to another machine, and it will restore the configuration automatically; * **offline config editing**: never get locked out after a failed update; * **access control**: allow others to deploy new kernels when you are on holiday. Here is a list of public servers you may use: * `boot.ci-tron.dev `__: The recommended CI-tron boot server **for anyone getting into hardware CI**, requires a `Freedesktop Gitlab`_ account and being a member of `ci-tron-users `_; * `nura.ci-tron.dev `__: The recommended `**NURA** <_Nura.eco>`_ boot server, requires a `Nura.eco Gitlab`_ account; * `demo.ci-tron.dev `__: A demo instance for **quick testing** in a public sandbox that is reset periodically. .. dropdown:: Booting a boot image in QEMU :color: light :animate: fade-in-slide-down .. code-block:: $ fallocate -l 10G disk.img $ qemu-system-x86_64 -enable-kvm -no-reboot \ -M q35 -smp 4 -m size=2048 \ -nic user,model=e1000,hostfwd=tcp::2222-:22 \ -device usb-tablet \ -vga none -device virtio-vga-gl,hostmem=4G,blob=true,venus=true \ -display gtk,gl=on,grab-on-hover=on \ -drive format=raw,file=${OUTPUT_IMAGE} \ -drive format=raw,file=disk.img .. _Freedesktop Gitlab: https://gitlab.freedesktop.org .. _gfx_ci_ci_tron: https://gitlab.freedesktop.org/gfx-ci/ci-tron .. _ci_tron_users: https://gitlab.freedesktop.org/groups/ci-tron-users/-/group_members/request_access .. _Nura.eco: https://nura.eco/ .. _Nura.eco Gitlab: https://gitlab.nura.eco .. dropdown:: Netbooting without using the dedicated boot servers :name: deploy_netboot_ipxe .. warning:: This method does not provide any automatic backup/restore, nor any editing of the ci-tron configuration. You may thus accidentally lock yourself out and be unable to fix it without physical access. If for some reason you do not want to use any public `Boot2ipxe server`_ instance nor re-deploy it yourself, you may want to reproduce parts of what the service is doing by using `Boot2container`_, with the following configuration expressed as an `iPXE`_ script: .. code-block:: #!ipxe # Set the defaults for the gateway set farm_name my_farm set gateway_cache_device auto set gateway_extra_cmdline set gateway_img_path registry.freedesktop.org/gfx-ci/ci-tron/gateway:latest set default_cmdline initrd=initrd.magic net.ifnames=0 b2c.ntp_peer=auto b2c.pipefail b2c.swap=32G b2c.volume="ci-tron-cache" b2c.volume="ci-tron-config" b2c.hostname=${farm_name}-gateway b2c.run="-ti --pull=always -v ci-tron-cache:/cache -v ci-tron-config:/config -v /dev:/dev -v /lib/modules:/lib/modules -v /lib/firmware:/lib/firmware docker://${gateway_img_path}" b2c.cache_device="${gateway_cache_device}" b2c.shutdown_cmd="reboot -f" ${gateway_extra_cmdline} kernel https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/linux-x86_64 ${default_cmdline} initrd https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/initramfs.linux_amd64.cpio.xz initrd https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/linux-x86_64.depmod.cpio.xz initrd https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/linux-x86_64.gpu.cpio initrd https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/linux-x86_64.wifi.cpio initrd https://gitlab.freedesktop.org/gfx-ci/boot2container/-/releases/v0.11.0/downloads/linux-x86_64.ucode.cpio # For Intel-based machines boot Rather than embedding the above script in iPXE directly (see `iPXE Crypto`_ for more details), you could store it in a public git repository and then source it from another embedded script which would never need to change: .. code-block:: #!ipxe set boot_script_url https://gitlab.freedesktop.org/mupuf/ci-tron-boot-cfg/-/raw/main/my_farm.ipxe echo Welcome to the CI-tron iPXE boot script :retry echo Acquiring an IP dhcp || goto retry # Keep retrying indefinitely echo Got the IP: ${netX/ip} / ${netX/netmask} echo echo Updating the current time using NTP ntp pool.ntp.org || goto retry echo Current unixtime: ${unixtime} (use `date --date=@$$((${unixtime}))` to decode it) echo echo Chainloading from the iPXE server ${IPXE_SERVER_FQDN} ... chain ${boot_script_url} || goto retry # The above command may "succeed" but we actually fail to # boot. This could happen if the iPXE boot file is # successfully returned, but the URLs to the kernel and # ramdisk are invalid, for example. In cases like these, # continuously retry the netboot, rather than exiting iPXE and # potentially getting stuck indefinitely in the firmware's next # boot method. The sleep acts as a simple rate limiter. sleep 60 goto retry Once your container has booted and shows the dashboard, you will want to add your SSH keys to ``/config/ssh/authorized_keys`` by killing the dashboard (``CTRL+Q``) then typing ``nano /config/ssh/authorized_keys``. .. _iPXE: https://ipxe.org/start .. _iPXE Crypto: https://ipxe.org/crypto .. dropdown:: On an existing gateway, but letting CI-tron take over everything :name: deploy_host_takeover .. warning:: This mode is not very well supported, as the services of your existing system may conflict with the ones provided by CI-tron. Work to improve support for this mode is welcome, and slowly happening Assuming you already have internet connectivity in your dedicated machine, the container should be executed in this fashion, all as root: .. code-block:: console # FARM_NAME=myfarm # # Create the volumes that will hold the config and caches # podman volume create ci-tron-config # podman volume create ci-tron-cache # podman run --rm -ti --pull=always --privileged \ --name ci-tron --network host --hostname $FARM_NAME-gateway \ -v ci-tron-config:/config -v ci-tron-cache:/cache \ registry.freedesktop.org/gfx-ci/ci-tron/gateway:latest .. dropdown:: On an existing gateway, as a rootless unprivileged container :name: deploy_rootless_unprivileged Unsupported features: * ❌ Physical (non-virtual) DUTs * ❌ Chronyd (NTP) * ❌ NFS / imagestore * ❌ Gitlab support (nested podman) .. code-block:: console $ # Create the volumes that will hold the config and caches $ podman volume create ci-tron-config $ podman volume create ci-tron-cache $ podman run --rm -ti --pull=always \ --cap-add CAP_NET_ADMIN \ --cap-add CAP_NET_RAW \ --cap-add CAP_AUDIT_CONTROL \ --device /dev/fuse \ --device /dev/net/tun \ --device /dev/kvm \ --name ci-tron --hostname vivian-mupuf-gateway \ -v config:/config -v cache:/cache \ --network bridge \ -p 60022:22 \ -p 8000:80 \ -p 8006:8006 \ registry.freedesktop.org/gfx-ci/ci-tron/gateway:latest Once your container has booted and shows the dashboard, you will want to add your SSH keys to the container for remote access. You may do so by mounting the config volume using ``podman unshare podman volume mount ci-tron-config`` and then editing ``$MOUNT_POINT/ssh/authorized_keys``. .. dropdown:: On an existing gateway, as a rootfull unprivileged container :name: deploy_rootfull_unprivileged Unsupported features: * ❌ Dynamic network interfaces (needed for ethernet over USB, for Fastboot devices) * ❌ Hardware-based serial console devices (needs an external service) * ❌ NFS / imagestore * ❌ Gitlab support (nested podman) .. code-block:: console # # Allow ip forwarding, since we can't do that from inside the container # # without requiring insane privileges. May not even be needed. # echo 1 > /proc/sys/net/ipv4/ip_forward # Create the networks # podman network create -d macvlan -o parent=$PRIVATE_NIC -o mode=passthru -o no_default_route=1 --ipam-driver=none ci-tron-private-nic # podman network create -d macvlan -o parent=$MGMT_NIC -o mode=passthru -o no_default_route=1 --ipam-driver=none ci-tron-mgmt-nic # Create the volumes that will hold the config and caches # podman volume create ci-tron-config # podman volume create ci-tron-cache # # TODO: add support for the USB-based PDUs # # TODO: write an external service that forwards consoles # # to SALAD using socat or similar # podman run --rm -ti --pull=always \ --cap-add CAP_NET_ADMIN \ --cap-add CAP_NET_RAW \ --cap-add CAP_AUDIT_CONTROL \ --cap-add CAP_SYS_TIME \ --device /dev/fuse \ --device /dev/net/tun \ --device /dev/kvm \ --name ci-tron --hostname vivian-mupuf-gateway \ -v ci-tron-config:/config -v ci-tron-cache:/cache \ --network bridge \ --network ci-tron-private-nic \ -p 60022:22 \ -p 8000:80 \ -p 8006:8006 \ registry.freedesktop.org/gfx-ci/ci-tron/gateway:latest Once your container has booted and shows the dashboard, you will want to add your SSH keys to the container for remote access. You may do so by mounting the config volume using ``podman unshare podman volume mount ci-tron-config`` and then editing ``$MOUNT_POINT/ssh/authorized_keys``. .. dropdown:: In vivian, for local development of CI-tron Unsupported features: * ❌ Physical (non-virtual) DUTs If you want to develop CI-tron, you could use built-in test environment: ``vivian``. .. code-block:: console $ cd ~/src/ci-tron $ make vivian You may then use ``$ make vivian-connect`` to get a shell inside the CI-tron container, that you can then use to edit the config files, or shutting down the virtual machine using ``poweroff``. You are now ready to go on to the quick guide :)