Release notes

The project is currently released as containers, versioned by release date. The latest stable version is tagged :latest, while release candidates are tagged :unstable.

Upcoming (unstable)

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:XXXX-XX-XX.1 (amd64, arm64)

Summary here

New contributors

  • Get your name here :)

Highlights

The main changes found in this release are:

  • Nothing yet

2026-06-17.1: Gitlab 19 support for the experimental runner, more transparent caching proxy, native DynDNS (unstable)

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2026-06-17.1 (amd64, arm64)

This release mostly brings fixes to our experimental runner, including GitLab 19 support. It also contains improvements related to the job HTTP server, and adds native support for hosting CI-tron on dynamic IPs.

Highlights

The main changes found in this release are:

  • Gitlab Runners:

    • Allowed using the new experimental runner for gateway runners

    • Added support for Gitlab v19 to our experimental runner

    • Obey the X-GitLab-Trace-Update-Interval header

    • Hardened the code to better handle future server upgrades

    • Better handling of unexpected server errors in the logging module

    • Improved cloning reliability, and --depth handling

    • Stop spamming stderr with exit statuses

    • Misc fixes

  • Executor:

    • Made the job HTTP server act like a forwarding proxy by forwarding the HTTP headers

  • Dynamic DNS support (Ddclient: dynamic DNS)

2026-06-03.1: Experimental GitLab runner, automatic proxy registry cleanups, HTTP request headers (unstable)

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2026-06-03.1 (amd64, arm64)

This release brings the long-awaited re-implementation of the GitLab runner which allows us to increase reliability, provide better artifact and git caching, support private repos, and significantly cut our CPU usage. It is also the first step towards allowing us to make executing jobs in CI-tron fully transparent, and enabling preemption support. Read up more on our Gitlab runner module page.

This new runner is however not used by default. You will have to opt-in by editing /config/mars_db.yaml by changing runner: auto into runner: ci-tron for every priority queue of every runner you want to test it on. The plan will be to switch to it by default in the next release, then drop support for the original runner in the following one.

Highlights

The main changes found in this release are:

  • Gitlab Runners:

    • Updated the official runner to v18.11.3 (runner: auto, or runner: gitlab-runner)

    • Previewing a re-implemented runner (runner: ci-tron)

  • Proxy registries:

    • Images pulled in fdo_proxy will get marked for deletion 2 months after creation

    • Automatic pruning of expired blobs at service start up

  • Executor:

    • Fully drop the size limit on control messages

    • Flush all control messages before stopping the job (cleaner shutdown)

    • Ability to set request headers for HTTP-backed artifacts

  • Dashboard:

    • Do not consider an empty file as a valid version

    • Improved reliability by disabling markup support for Labels

  • DUT hardware support:

    • Google Pixel 3a (google-sargo)

Migrations

Enabling automatic pruning of images in user-defined proxy registries

To enable automatic pruning of container images in registries defined in /config/registries/, you will need to set storage: delete: true and proxy: ttl: 1440h.

See /etc/registryd/8100_quay.yml.j2.example.

Manually pruning the existing proxy registry caches

Job-breaking operation

This migration requires shutting down the proxy registry, which means any job trying to access the registry will fail. Do not attempt it unless the gateway is idle and, ideally, not accepting jobs!

Container images downloaded before automatic pruning was enabled will remain on disk indefinitely. If you would like to remove the images, you will have to run the following commands:

# systemctl stop fdo_proxy-registry
# rm -rf /cache/registries/fdo_proxy
# systemctl restart fdo_proxy-registry

Please replace fdo_proxy with whatever proxy registry name you would like to apply this solution to.

2026-03-31.1: Fewer Valve-isms, better quality of life

Breaking changes

This release switches the dashboard’s display server from Cage (Wayland) to KMSCON to improve reliability, at the (temporary) expense of losing touchscreen support.

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2026-03-31.1 (amd64, arm64)

This release continues on its work to make the CI-tron codebase less of a Valve infra and more of a generic and user-controllable infrastructure in two key ways: public boot servers, and opt-in telemetry.

Quality of life has also been massively improved through improved reliability, error reporting, lower resource usage (-200MB image size, lower CPU usage), our new boot infrastructure with public boot servers, and massive improvements to the documentation (in part from our new contributors).

Finally, in this release, we introduce the concept of unstable releases. These releases are stable release candidates, which will be tagged as the latest stable release after sufficient testing has happened. If you wish to always use the latest unstable release, you may use the :unstable tag.

New contributors

  • Pablo Correa Gómez (@pabloyoyoista)

  • Alexis Deruelle (@alxdrl)

  • Janne Grunau (@jannau)

Highlights

The main changes found in this release are:

  • Monitoring:

    • Unhardcoded the monitoring configuration, config folder at /config/telegraf.d/

    • Dropped the automatic push of all telemetry to Valve’s servers

  • iPXE Boot Server:

  • Executor:

    • Control messages and DUT logs are no longer interleaved

    • Improved handling of Gitlab runner tokens to prevent any mismatch between local and remote runners

    • Better error reporting related to invalid jobs, artifacts, archives, …

    • Shaved 5 seconds from every job setup

    • Cleaner error log when the job is invalid or on error conditions

    • Added support for Intellinet’s PDU 163682

  • Dashboard:

    • Better visualisation of the validity of registration, access, and runner tokens

    • Switch from Cage (Wayland) to KMSCON to improve reliability. See 5dc6c4104491 for more details.

    • User-configurable keyboard layout, font, screen rotation, … at /config/kmscon/kmscon.conf

    • Invalid characters in the DUT console are now ignored rather than crashing

  • SALAD:

    • Polling has been limited to 1kHz to save CPU cycles to increase the number of characters sent per network frame

  • Misc:

    • Live-provisioning improvements and documentation

    • Better support for building our images with SELinux enabled

    • Restructuring of the documentation

2026-02-04.2: Fix for the access control script on GitLab runners on the gateway

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2026-02-04.2 (amd64, arm64)

This release only brings in critical bug fixes, reliability improvements, and debug options:

  • Fix the use of gateway runners on Gitlab;

  • Improve the reliability of the Gitlab ACL check in the presence of Gitlab errors;

  • Improve the resilience to unexpected owner/permissions in the config directory;

  • Add debug parameters related to Gitlab/Github job submission.

2026-02-04.1: Gitlab job prioritisation tag enforcement, distributed farms, OnePlus 6/6T

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2026-02-04.1 (amd64, arm64)

Bug in this release

This release contains a bug, breaking the use of the gateway runners on GitLab. Please use the next release, 2026-02-04.2, if this is a feature you need.

Breaking changes

This release starts enforcing that the jobs are correctly setting their priority tag (introduced in release 2025-11-07.1). If a job starts failing with “Priority tag is missing from the job” and you don’t care about priorities, use ci-tron:priority:default, as it’s always available, unless the project has a different policy (e.g. Mesa uses high/default/low priorities for various jobs and situations).

New contributors

  • Neil Armstrong (@narmstrong)

Highlights

The main changes found in this release are:

  • Gitlab:

    • Start enforcing that job priorities are set in every job

    • Allow setting timeouts:boot_cycle:retries value without also providing a timeout

  • Executor:

    • Allow farms to join a distributed virtual farm that can be targeted by forges jobs (FARM_ALIASES=name1,name2,... in /config/config.env)

    • Sergeant Hartman: Ability to disable training retrials

  • DUT hardware support:

    • Modernized support for the HDK8*50

    • OnePlus 6/6T (fajita/enchilada)

  • PDUs:

    • Add auto-detection for APC PDU hardware generations

  • Misc:

    • About 2x faster live deployment by making use of Ansible mitogen

    • Lots of documentation improvements

2025-12-08.1: Attached artifacts, user-configurable Wireguard tunnels, new PDU drivers

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2025-12-08.1 (amd64, arm64)

This release, on top of the usual fixes, brings support for attached artifacts: Artifacts that are uploaded to the gateway by the client and referenced in the job description with attached_artifact instead of url or data.

It also continues the trend of turning CI-tron into a generic project by un-hardcoding the Wireguard configuration (see /config/wireguard/wg0.conf).

Finally, this release also shines by how many new PDU drivers also landed:

2025-11-07.1: Multiplying job sources

Breaking changes

Upon rebooting on the new image, all your Gitlab runners will get removed and re-registered with an extra new tag: ci-tron:priority:$PRIORITY.

Before rebooting on the new image, please:

  • Back up /config/mars_db.yaml

  • Check that your Gitlab registration/access tokens are still valid by either asking the Gitlab instance admin to confirm your tokens are valid, or by using the following command and making sure they do not return error 401:

    curl  --header "PRIVATE-TOKEN: $token" "https://gitlab.freedesktop.org/api/v4/runners"

If any of your tokens have expired, please refer to MarsDB to see which scopes to request.

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2025-11-07.1 (amd64, arm64)

The main changes found in this release are:

  • Job prioritisation: Get more control over which jobs will run next

  • Initial Github support [1]

  • The dashboard v2 now replaces the original dashboard

  • It is now possible to disable exposing all runners by default on a job source, by setting expose_all_runners_by_default=false

  • Switch to Fedora 43

As you can see, this release was focused on increasing compatibility with forges while keeping the worst-case job execution latency of important jobs in check by allowing farm admins to specify the priorities of different forges/projects/users.

Finally, the new dashboard allows interacting with the new features more easily than having to edit /config/mars_db.yaml by hand.