Skip to content

tempus.build

Why tempus.build builds its own runner image instead of using ubuntu-latest

A managed GitHub Actions runner makes one promise: change runs-on, and your workflow keeps working. That promise rests mostly on the runner image — the languages, tools and environment variables your workflow steps depend on. The easy route would be to take GitHub’s image, ubuntu-latest (currently Ubuntu 24.04), as is. We don’t; we build our own (opens in new tab). Here’s why, and what it costs us.

Why GitHub’s image doesn’t fit as is

ubuntu-latest is an Azure VM image built with Packer from actions/runner-images (opens in new tab). It isn’t a container image. Its build scripts assume a full VM: systemd as PID 1, background services (the Docker daemon, PostgreSQL, MySQL), swap.

It’s also large. On a standard private ubuntu-latest runner (2 vCPU) we measured a root filesystem of about 77 GB with about 14 GB free — the 14 GB GitHub lists in its documentation (opens in new tab). The image itself takes 59–63 GB.

tempus.build runners work differently. Each runner is an ephemeral container managed by Actions Runner Controller (ARC), each pod runs in its own Kata Containers A lightweight VM: a strong VM boundary (its own kernel, hardware virtualization), but close to a container in weight and start-up time., and Docker comes from a separate dind sidecar. That setup needs a container image with the GitHub Actions runner agent inside — ARC’s gha-runner-scale-set requires it. A VM image doesn’t fit, so we build the image ourselves.

Reproducibility and a verifiable supply chain

This is the main technical difference. GitHub’s image installs whatever is current at build time: the latest apt packages, curl | bash installers, toolset entries like “8.3” with no patch level. Two builds a day apart can differ. GitHub publishes a software report of what’s installed, but no signature, Software Bill of Materials — a machine-readable inventory of every component and dependency inside an artifact (image): what is in it and at which versions. tempus.build images publish it in the CycloneDX format. or A cryptographically signed statement of an artifact's origin: what built it, from which source and in what environment. In SLSA terms, a build provenance attestation. that a consumer could verify for the image itself.

Ours is built differently:

  • Every downloaded artifact is pinned: an exact version plus SHA256/512, or a key-verified apt repository. The base image is pinned by digest. Renovate bumps the pins in separate PRs that go through the same checks as any other change.
  • Every published image is signed with A Sigstore tool for signing and verifying container images and artifacts. In keyless mode the signature is bound to an OIDC identity and needs no long-lived keys. keyless (GitHub OIDC, bound to build.yml on refs/heads/main) and carries an SBOM and Supply-chain Levels for Software Artifacts — an industry standard of supply-chain security levels; the levels describe how strongly an artifact's origin is proven. provenance.
  • A A vulnerability scanner: checks an image for known CVEs in OS packages and dependencies. In CI it acts as a gate that blocks publishing on fixable HIGH/CRITICAL findings. gate blocks publication on HIGH/CRITICAL findings. Exceptions are documented in .trivyignore.yaml (opens in new tab), each with a rationale and an expiry date.

Anyone can verify which image ran their code, using the commands from SECURITY.md (opens in new tab):

IMAGE=ghcr.io/tempusbuild/runner-ubuntu-24.04
DIGEST=$(docker buildx imagetools inspect "${IMAGE}:vYYYYMMDD" --format '{{.Manifest.Digest}}')
cosign verify \
  --certificate-identity-regexp '^https://github\.com/tempusbuild/runner-images/\.github/workflows/build\.yml@refs/heads/main$' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  "${IMAGE}@${DIGEST}"
gh attestation verify "oci://${IMAGE}@${DIGEST}" --owner tempusbuild

The “Set up job” step of every job shows which image ran it: the runner reads /imagegeneration/imagedata.json and prints the image name, its vYYYYMMDD version, the runner version, and links to the README at the build commit and to the ghcr package. Signing, the SBOM and the CVE gate are covered in depth in our runner-image supply chain post.

Parity where it matters for speed

The compatibility contract is GitHub’s official Ubuntu 24.04 image README (opens in new tab). The image ships cloud CLIs, compilers (GCC 12–14, Clang 16–18), and browsers with their drivers; the full inventory is in the image README (opens in new tab). For speed, though, the list matters less than whether setup-* actions find the requested version in the image or go to the network for it.

How setup-* actions find a preinstalled tool

Many workflows start with actions/setup-python, setup-node, setup-go, setup-java or similar. Each one makes a single decision: use the version in the image, or download it.

  • Toolcache lookup. The first-party actions look the version up via @actions/tool-cache (opens in new tab): a $RUNNER_TOOL_CACHE/<tool>/<version>/<arch> directory with an <arch>.complete marker file next to it. No marker, no version — even if Python 3.12 is installed somewhere else. setup-java uses its own naming, e.g. Java_Temurin-Hotspot_jdk/<version>/x64.
  • A miss means a download. When the version isn’t found, the action downloads it (from a versions manifest, go.dev, Adoptium or another vendor) and puts it in the toolcache — for the rest of that job only. Our runners are ephemeral, so without a baked toolcache every job would pay for that download again.
  • The runner mode changes behavior. The runner sets RUNNER_ENVIRONMENT (opens in new tab) to github-hosted or self-hosted, and some actions branch on it. ARC runners are always self-hosted, ours included. ruby/setup-ruby, for instance, expects the toolcache at /opt/hostedtoolcache, writable by the runner user, on self-hosted runners (opens in new tab).

So our layout matches GitHub’s, and setup-* actions resolve these versions locally:

  • Python 3.10–3.14, Node 22 and 24, Go 1.25 and 1.26, Ruby 3.2–4.0, PyPy 3.9–3.11;
  • Java Temurin 8, 11, 17, 21 and 25;
  • .NET SDK 8, 9 and 10 in /usr/share/dotnet, the default install dir of actions/setup-dotnet, so it reuses them;
  • PHP 8.3 for shivammathur/setup-php (see the example below).

As on ubuntu-latest, the image also carries a cache of common action archives, so the runner doesn’t download them on every run. The environment contract that setup-* actions and builds rely on is kept too: ImageOS, RUNNER_TOOL_CACHE, JAVA_HOME_<v>_X64 and the rest.

Worked example: setup-php

Here’s what a missed precondition looks like, from a customer’s PHP project. The shivammathur/setup-php (opens in new tab) step took 62–72 seconds on our runners versus about 6 seconds on GitHub, even though PHP 8.3 was already in the image.

setup-php sees RUNNER_ENVIRONMENT=self-hosted and switches to its self-hosted mode. In that mode it reuses an installed PHP only if php-config reports the requested series, and php-config ships in php8.3-dev. It also installs helper packages via apt, apt-transport-https among them. Neither php-config nor apt-transport-https was in the image, so every run re-added a PPA, ran apt-get update across a dozen repositories and reinstalled PHP.

The fix ships PHP 8.3 from the Ubuntu archive (opens in new tab), as GitHub does, with no third-party PPA, the full extension set, php8.3-dev and the helper packages.

How we keep the fast path

Parity isn’t “the same tools are installed”. It’s the same preconditions each action checks: paths, marker files, helper binaries, environment variables. And they have to hold in the mode the action actually runs in — self-hosted, in our case.

So the image tests (opens in new tab) check the preconditions themselves, not just versions: a .complete marker for every baked version, ImageOS and RUNNER_TOOL_CACHE, a writable toolcache. The new PHP test asserts setup-php’s fast-path preconditions: php-config for the right series and the helper packages. If a workflow still falls off the fast path, we treat it as a compatibility bug and fix it in the image.

Where we deviate from ubuntu-latest — on purpose

Some differences come from security and correctness. They’re spelled out in the inclusion policy in the image README:

  • End-of-life Go minors (1.22–1.24) aren’t baked in. Go patches only the two latest minors, so a baked toolcache for an EOL minor would carry stdlib CVEs the image can’t fix. The default go is 1.26; on ubuntu-latest it’s 1.24, which is already EOL. setup-go installs older minors on demand.
  • Older toolcache Pythons get their bundled setuptools upgraded. The bundled version ships vulnerable vendored copies of libraries; we upgrade it rather than ship it as is.
  • No systemd-coredump. systemd isn’t PID 1 in a container, so the package would do nothing except add attack surface.
  • More native-build headers than ubuntu-latest: imaging, crypto, DB and ODBC, compression, Kafka, BLAS/HDF5, plus protobuf, meson and ccache. Native Python packages such as Pillow, mysqlclient, python-ldap, pycurl, pyodbc, PyNaCl and h5py build without an apt-get install step. The caveat: workflows that rely on this won’t run on GitHub-hosted runners without changes.
  • Jobs run as the non-root runner user, with sudo available, as on GitHub-hosted. There are no secrets in image layers, ENV or ARG.
  • Nothing in the image depends on our infrastructure. Everything environment-specific is applied at runtime, so our runners run exactly the image published on ghcr.

What it costs us

Parity gaps cost speed

Every precondition a setup-* action checks is ours to meet. A missed one doesn’t break the workflow — it slows it down, as in the setup-php example, so such gaps have to be hunted down in step timings, not in errors.

Maintenance is on us

Bumping pins and recomputing SHAs is our job. So is the runner update window: GitHub stops queueing jobs (opens in new tab) to a runner that hasn’t been updated within 30 days of a new release. We refresh the runner version on that cadence; at the time of writing it’s 2.338.0.

Size

The image is about 17 GB compressed (amd64) — a heavy pull for a node that doesn’t have it yet. So every new image version is pulled onto our runner nodes before it’s rolled out, and its size doesn’t affect job start.

The bottom line

Building our own image isn’t walking away from ubuntu-latest compatibility — it’s a different way to deliver it: a container image instead of a VM image, pinned versions instead of floating ones, a signature and an SBOM instead of taking our word for it. The differences from GitHub are listed in the image README, and the repository is public: you can see what’s inside, build the image yourself and verify the signature.

If a workflow runs slower or behaves differently on tempus.build than on GitHub-hosted runners, contact us or open an issue in runner-images (opens in new tab). How the isolation around the image works is on our Security & isolation page.

Related articles

← back to the blog