Release notes

The project is currently released as containers, versioned by release date. The latest stable version is tagged :latest, while release candidates are tagged :unstable.

XXXX-XX-XX.1

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:XXXX-XX-XX.1 (amd64, arm64)

New contributors

  • Get your name here :)

Highlights

The main changes found in this release are:

  • Proxy registries:

    • Images pulled in fdo_proxy will get marked for deletion 2 months after creation

    • Automatic pruning of expired blobs at service start up

  • Executor:

    • Fully drop the size limit on control messages

    • Flush all control messages before stopping the job (cleaner shutdown)

    • Ability to set request headers for HTTP-backed artifacts

  • Dashboard:

    • Do not consider an empty file as a valid version

    • Improved reliability by disabling markup support for Labels

Migrations

Enabling automatic pruning of images in user-defined proxy registries

To enable automatic pruning of container images in registries defined in /config/registries/, you will need to set storage: delete: true and proxy: ttl: 1440h.

See /etc/registryd/8100_quay.yml.j2.example.

Manually pruning the existing proxy registry caches

Job-breaking operation

This migration requires shutting down the proxy registry, which means any job trying to access the registry will fail. Do not attempt it unless the gateway is idle and, ideally, not accepting jobs!

Container images downloaded before automatic pruning was enabled will remain on disk indefinitely. If you would like to remove the images, you will have to run the following commands:

# systemctl stop fdo_proxy-registry
# rm -rf /cache/registries/fdo_proxy
# systemctl restart fdo_proxy-registry

Please replace fdo_proxy with whatever proxy registry name you would like to apply this solution to.

2026-03-31.1: Fewer Valve-isms, better quality of life (unstable)

Breaking changes

This release switches the dashboard’s display server from Cage (Wayland) to KMSCON to improve reliability, at the (temporary) expense of losing touchscreen support.

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2026-03-31.1 (amd64, arm64)

This release continues on its work to make the CI-tron codebase less of a Valve infra and more of a generic and user-controllable infrastructure in two key ways: public boot servers, and opt-in telemetry.

Quality of life has also been massively improved through improved reliability, error reporting, lower resource usage (-200MB image size, lower CPU usage), our new boot infrastructure with public boot servers, and massive improvements to the documentation (in part from our new contributors).

Finally, in this release, we introduce the concept of unstable releases. These releases are stable release candidates, which will be tagged as the latest stable release after sufficient testing has happened. If you wish to always use the latest unstable release, you may use the :unstable tag.

New contributors

  • Pablo Correa Gómez (@pabloyoyoista)

  • Alexis Deruelle (@alxdrl)

  • Janne Grunau (@jannau)

Highlights

The main changes found in this release are:

  • Monitoring:

    • Unhardcoded the monitoring configuration, config folder at /config/telegraf.d/

    • Dropped the automatic push of all telemetry to Valve’s servers

  • iPXE Boot Server:

  • Executor:

    • Control messages and DUT logs are no longer interleaved

    • Improved handling of Gitlab runner tokens to prevent any mismatch between local and remote runners

    • Better error reporting related to invalid jobs, artifacts, archives, …

    • Shaved 5 seconds from every job setup

    • Cleaner error log when the job is invalid or on error conditions

    • Added support for Intellinet’s PDU 163682

  • Dashboard:

    • Better visualisation of the validity of registration, access, and runner tokens

    • Switch from Cage (Wayland) to KMSCON to improve reliability. See 5dc6c4104491 for more details.

    • User-configurable keyboard layout, font, screen rotation, … at /config/kmscon/kmscon.conf

    • Invalid characters in the DUT console are now ignored rather than crashing

  • SALAD:

    • Polling has been limited to 1kHz to save CPU cycles to increase the number of characters sent per network frame

  • Misc:

    • Live-provisioning improvements and documentation

    • Better support for building our images with SELinux enabled

    • Restructuring of the documentation

2026-02-04.2: Fix for the access control script on GitLab runners on the gateway

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2026-02-04.2 (amd64, arm64)

This release only brings in critical bug fixes, reliability improvements, and debug options:

  • Fix the use of gateway runners on Gitlab;

  • Improve the reliability of the Gitlab ACL check in the presence of Gitlab errors;

  • Improve the resilience to unexpected owner/permissions in the config directory;

  • Add debug parameters related to Gitlab/Github job submission.

2026-02-04.1: Gitlab job prioritisation tag enforcement, distributed farms, OnePlus 6/6T

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2026-02-04.1 (amd64, arm64)

Bug in this release

This release contains a bug, breaking the use of the gateway runners on GitLab. Please use the next release, 2026-02-04.2, if this is a feature you need.

Breaking changes

This release starts enforcing that the jobs are correctly setting their priority tag (introduced in release 2025-11-07.1). If a job starts failing with “Priority tag is missing from the job” and you don’t care about priorities, use ci-tron:priority:default, as it’s always available, unless the project has a different policy (e.g. Mesa uses high/default/low priorities for various jobs and situations).

New contributors

  • Neil Armstrong (@narmstrong)

Highlights

The main changes found in this release are:

  • Gitlab:

    • Start enforcing that job priorities are set in every job

    • Allow setting timeouts:boot_cycle:retries value without also providing a timeout

  • Executor:

    • Allow farms to join a distributed virtual farm that can be targeted by forges jobs (FARM_ALIASES=name1,name2,... in /config/config.env)

    • Sergeant Hartman: Ability to disable training retrials

  • DUT hardware support:

    • Modernized support for the HDK8*50

    • OnePlus 6/6T (fajita/enchilada)

  • PDUs:

    • Add auto-detection for APC PDU hardware generations

  • Misc:

    • About 2x faster live deployment by making use of Ansible mitogen

    • Lots of documentation improvements

2025-12-08.1: Attached artifacts, user-configurable Wireguard tunnels, new PDU drivers

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2025-12-08.1 (amd64, arm64)

This release, on top of the usual fixes, brings support for attached artifacts: Artifacts that are uploaded to the gateway by the client and referenced in the job description with attached_artifact instead of url or data.

It also continues the trend of turning CI-tron into a generic project by un-hardcoding the Wireguard configuration (see /config/wireguard/wg0.conf).

Finally, this release also shines by how many new PDU drivers also landed:

2025-11-07.1: Multiplying job sources

Breaking changes

Upon rebooting on the new image, all your Gitlab runners will get removed and re-registered with an extra new tag: ci-tron:priority:$PRIORITY.

Before rebooting on the new image, please:

  • Back up /config/mars_db.yaml

  • Check that your Gitlab registration/access tokens are still valid by either asking the Gitlab instance admin to confirm your tokens are valid, or by using the following command and making sure they do not return error 401:

    curl  --header "PRIVATE-TOKEN: $token" "https://gitlab.freedesktop.org/api/v4/runners"

If any of your tokens have expired, please refer to MarsDB to see which scopes to request.

Image: registry.freedesktop.org/gfx-ci/ci-tron/gateway:2025-11-07.1 (amd64, arm64)

The main changes found in this release are:

  • Job prioritisation: Get more control over which jobs will run next

  • Initial Github support [1]

  • The dashboard v2 now replaces the original dashboard

  • It is now possible to disable exposing all runners by default on a job source, by setting expose_all_runners_by_default=false

  • Switch to Fedora 43

As you can see, this release was focused on increasing compatibility with forges while keeping the worst-case job execution latency of important jobs in check by allowing farm admins to specify the priorities of different forges/projects/users.

Finally, the new dashboard allows interacting with the new features more easily than having to edit /config/mars_db.yaml by hand.