Kata Containers: how every CI job gets its own kernel

A container is a process on the host, isolated by kernel features: namespaces, cgroups, capabilities, seccomp. But the kernel is shared by everyone — the host kernel. For CI that is an awkward boundary: a job runs someone else’s, effectively untrusted code (build steps, dependencies from registries, and often Docker-in-Docker), and the whole thing stands on the same kernel as the neighbouring jobs and the host itself. One kernel exploit and the container boundary stops meaning anything.

Kata Containers changes the boundary itself: each container (a pod, in Kubernetes terms) runs inside a separate lightweight virtual machine with its own guest kernel. The isolation boundary is not a set of kernel primitives but hardware virtualization. And from the outside Kata still looks like an ordinary OCI runtime: the same images, the same containerd, the same workflow.

Let’s walk through how this works in Kata 3.x, and why it’s the right level of isolation for CI.

What Kata Containers is

Kata Containers↗ (opens in new tab) is an open-source project (now under OpenInfra / the Linux Foundation) that “runs a lightweight virtual machine and uses a guest Linux kernel to create container workloads.” The idea is to close the historical gap between two approaches:

• Containers — fast and light, but they share the host kernel (a weak isolation boundary).
• Virtual machines — a strong boundary (own guest kernel, hardware virtualization), but heavy and slow to start.

Kata takes the VM property (a separate kernel + a hardware boundary) and packages it to approach a container in weight and start-up time — hence the term micro-VMA lightweight VM: a strong VM boundary (its own kernel, hardware virtualization), but close to a container in weight and start-up time..

Kata is OCI-compatible. It’s the same standard as Docker containers, and the same one under CRI (Container Runtime Interface) in Kubernetes. You don’t rebuild images — Kata works as just another runtime under containerd or CRI-O. To the orchestrator it’s simply a different class of runtime environment, not a different container format.

How it works inside

What makes it look like a container on the outside and run as a VM on the inside is a handful of components.

Hypervisor and the guest VM

Kata brings up the VM through a hypervisor on top of hardware virtualization KVMKernel-based Virtual Machine — the Linux kernel's hardware-virtualization layer that the VMMs launched by Kata run on.. Kata 3.x supports several VMMVirtual Machine Monitor (hypervisor) — the process that creates and runs the VM. In Kata: QEMU, Cloud Hypervisor, Firecracker or Dragonball., and they all work through KVM:

Each VMM has its own trade-offs between maturity, start-up speed, overhead and feature set. The choice of a specific VMM is an operational question; the technology allows any of them.

Inside the VM runs its own guest kernel and a lightweight guest image. The guest rootfs is attached efficiently, via a DAXDirect Access — the hypervisor maps the guest image straight into guest memory (zero-copy), saving both memory and boot time. mapping (zero-copy), which cuts both memory and start-up time — a micro-VM doesn’t boot like a full VM.

kata-agent inside the VM

Inside the guest VM lives the kata-agent — a long-lived process written in Rust. It manages the containers inside the VM: it creates them with the right cgroups and namespaces (so the familiar container isolation doesn’t go away — it’s simply applied inside the VM, on top of the already-hardware boundary) and tracks the workload’s lifecycle.

The host-side runtime talks to the agent over ttRPCA compact gRPC-style protocol for low-level IPC, without HTTP/2 — built for minimal overhead. on VSOCK — a virtual socket between host and VM. No ordinary network is needed for the control channel.

Runtime and shim: containerd-shim-kata-v2

On the host side Kata is implemented as a shimv2Containerd Runtime V2 (Shim API): one long-lived shim process per pod instead of invoking the runtime for every operation. runtime — the containerd-shim-kata-v2 binary (it implements the Containerd Runtime V2 / Shim API). Instead of invoking the runtime as a separate process for every operation, containerd creates a single socket and talks to a single shim instance per pod. It’s the shim that starts the hypervisor and its VM and maintains the link to the agent.

virtio: how the VM “reaches” what it needs

The guest VM is isolated, but the container needs files, network and I/O. Those are passed through by paravirtual virtio devices:

• virtio-fs — passes the OCI bundle (the container rootfs) into a directory inside the guest rootfs.
• virtio-net / vhost-net — networking.
• virtio-block / virtio-scsi — block devices.
• virtio-vsock — the agent’s control channel.

This is a narrow, explicitly defined set of interfaces between the VM and the host — not “the whole host kernel,” as with an ordinary container.

The sandbox model: a pod is a VM

In Kata terms one pod = one VM = one sandbox. Kubernetes can run several containers in a pod — they all share one VM; a service pause container (the sandbox container) accompanies them, setting up the pod’s shared namespaces. shimv2 serves the whole pod with a single shim instance — instead of separate shim and proxy processes per container, as in the older architecture.

The isolation is two-level:

• Between pods — hardware. Different pods are different VMs with different guest kernels.
• Within a pod — namespaces. Containers of one pod are isolated from each other via cgroups and namespaces created by the guest kernel inside the VM.

For CI this is exactly what you want: one job is one pod, so one VM. A neighbouring job is a different VM with a different kernel; all they share is the physical machine under the hypervisor.

Networking: how the host’s network namespace connects to the VM

Through CNI the orchestrator creates a separate network namespace for the pod and usually a veth pair: one end in the pod’s namespace, the other on the host. You can’t attach a veth to the VM directly — the VM needs a TAP interface. Kata wires one to the other transparently; the method is set by the interworking model (NetInterworkingModelA Kata setting: how the pod's CNI interface is wired to the VM's TAP device — tcfilter, macvtap or none.):

• tcfilter (default) — Traffic Control (TC) rules redirect traffic between the veth in the pod’s namespace and the VM’s TAP device. Works with ipvlan/macvlan too.
• macvtap — the pod’s interface is attached to the VM through a MACVTAP device.
• none — when the VM sits in the host’s network namespace.

The key point: the hypervisor gets a separate network namespace, and the guest sees only the TAP device it’s been given. The guest has no direct access to the host NIC — traffic goes through a host-controlled TC/TAP boundary.

Storage: virtio-fs, DAX and a block rootfs

The VM’s own guest rootfs is attached via DAX: the hypervisor maps the guest image into guest memory (onto a /dev/pmem* device) zero-copy — which saves both memory and boot time.

Container files are passed through differently. By default it’s virtio-fsA paravirtual shared filesystem: the virtiofsd daemon on the host exposes the container rootfs to the guest; host and guest see one file tree.: a virtiofsd daemon runs on the host and exposes the OCI bundle (the container rootfs) into a directory inside the guest rootfs (kataShared). virtio-fs is a shared-fs model: host and guest see the same file tree.

The alternative is a block device: the rootfs or a volume is passed into the VM as virtio-block, for example through a devicemapper snapshotter that exposes the container image as a block device.

The trade-off between the two models:

• virtio-fs is more flexible — live directory passthrough, bind-mounting volumes, overlays — but it’s a separate daemon and a cache-coherence layer between host and guest.
• a block rootfs is simpler and usually faster on “raw” I/O, but less flexible and requires a block snapshotter.

The guest image and kata-agent as init

Guest components come in two kinds:

• a rootfs image — a full root filesystem; init is either systemd (flexible, easy to customize) or the kata-agent itself.
• initrd — an image unpacked into memory; init is the kata-agent.

This is built with the AGENT_INIT=yes flag: then the kata-agent runs as the init process (PID 1) of the guest — with no systemd and no extra userspace. The minimal variant, initrd + agent-as-init, is the fastest and most compact. The point is that the guest userspace is reduced to the essentials: the kernel, the agent and whatever the container needs. Less code — less attack surface and a faster boot.

Memory and a fast start

A micro-VM’s fast start comes from several things:

• a minimal guest kernel and a trimmed guest image (see above);
• VM templating / factory — a pre-booted, initialized and paused VM template from which new VMs are cloned faster than they’d boot from scratch (enable_template, kata-runtime factory init; the mode requires initrd and is incompatible with virtio-fs as shared_fs).

Memory is handled just as pragmatically. A VM has a starting size (default_memory), and beyond that hotplug kicks in: when a container has a memory limit set, Kata adds memory to the guest on the fly — via virtio-mem (enable_virtio_mem) or memory hotplug / balloon, depending on the VMM and architecture. That removes the need to give every VM memory “with a margin.”

The overhead is honestly still there: each VM carries its own kernel and its own starting memory — that’s the price of a hardware boundary (see the trade-offs below).

The flow: from containerd to a process in the VM

How Kata slots into the standard container launch pipeline:

1. The orchestrator selects Kata — in Kubernetes via a RuntimeClass, in CRI-O via annotations. No changes to the images themselves.
2. From runtime_type (e.g. io.containerd.kata.v2) containerd derives the shim binary name by substituting the prefix: io.containerd.kata.v2 → containerd-shim-kata-v2. It creates a socket and hands it to a single shim instance.
3. The shim starts the hypervisor, which loads the micro-VM from the guest components (image + guest kernel).
4. The shim calls the agent’s API inside the VM (CreateSandbox / container creation) over ttRPC/VSOCK.
5. The kata-agent brings up the container process inside the VM — with cgroups and namespaces, from the rootfs passed through via virtio-fs.
6. The job finishes — the VM is destroyed along with everything in it.

Registering Kata as a runtime handler in containerd is essentially one config section (an illustration from the Kata docs, not our production config):

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
  runtime_type = "io.containerd.kata.v2"

And in Kubernetes workloads are directed to this runtime via a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata
---
apiVersion: v1
kind: Pod
metadata:
  name: untrusted-job
spec:
  runtimeClassName: kata
  containers:
    - name: build
      image: ghcr.io/example/runner:latest

For whoever writes the workflow, all of this is invisible: they don’t know — and don’t need to know — that their job runs in a VM rather than a container.

Why it’s the right boundary for untrusted CI code

Kata’s threat model states the goal plainly: prevent an untrusted container workload from gaining control of the host, extracting information from it, or changing anything on it.

The difference from an ordinary container is fundamental:

• A shared kernel is ruled out. The guest has its own kernel. A kernel exploit from untrusted code hits the guest kernel inside the VM, not the host kernel directly.
• A container escape ≠ host access. Having broken out of the container, the code is still inside the VM. To reach the host or a neighbouring workload it has to break the virtualization boundary as well — and that’s a fundamentally narrower and harder surface.
• No access to the host’s filesystem or network by default. The guest doesn’t see the host filesystem without an explicitly passed-through device; the network is isolated via TAP, with no direct access to the host NIC.
• Resource exhaustion is bounded by the VM. A fork bomb or a memory leak runs into the quota of its own micro-VM.

That’s exactly why Kata is a natural layer for CI: jobs by definition run someone else’s code, and the container boundary is not enough for that.

Trade-offs — honestly

Hardware isolation isn’t free:

• Virtualization overhead. Each micro-VM is a separate guest kernel and memory for it; start-up and resource use are higher than an ordinary container’s. Kata and its lightweight VMMs (Firecracker, Dragonball, Cloud Hypervisor) minimize this, but don’t reduce it to zero.
• The hypervisor is an attack surface too. The boundary has moved from the host kernel to the hypervisor; a vulnerability in QEMU/KVM or in a virtio backend could in theory allow a VM escape. That’s why it matters which VMM is chosen and how the backends are configured (userspace vhost-user is safer than in-kernel vhost).
• The guest kernel needs updating too. Having its own kernel doesn’t make the guest invulnerable — its CVEs don’t go away, they just stop being shared with the host.

The sober conclusion: Kata doesn’t “cancel” container security — it adds a second, independent boundary underneath it. It’s defense-in-depth, not a silver bullet.

runc, gVisor and Kata: where the boundary runs

Kata is easy to position through two neighbouring runtimes — the ordinary container one and the “sandbox” one.

In short: runc is maximally fast and maximally trusting; gVisor wedges SentrygVisor's userspace “kernel”: it intercepts the application's syscalls and services them itself, passing only a narrow set through to the host kernel. (a userspace kernel) between the application and the host; Kata raises a full hardware boundary. For untrusted CI code the last one is what matters.

Confidential Containers: when the host is untrusted too

A separate capability of the technology — not a mandatory part of the base isolation — is running the guest inside a hardware TEETrusted Execution Environment — a hardware-isolated, encrypted execution region (Intel TDX, AMD SEV-SNP, IBM Secure Execution).: Intel TDX, AMD SEV-SNP, IBM Secure Execution. In that mode the guest’s memory is encrypted and protected even from the host and the hypervisor, and remote attestationCryptographic proof that a workload runs inside a genuine, expected TEE — before any secrets or code are handed to it. lets you cryptographically verify the environment before a workload is handed to it.

This inverts Kata’s base threat model: there the host is protected from an untrusted guest — here, conversely, the guest is protected from an untrusted host and operator. This is the Confidential Containers direction. (QEMU is the most supported VMM for TDX and SEV-SNP.)

An important caveat: everything above is a property of the Kata technology itself. In this article Confidential Containers are described as a capability, not as a specific tempus.build configuration.

How it works at tempus.build

At tempus.build every runner job runs in its own hardware micro-VM on Kata Containers. Untrusted CI code — and its Docker-in-Docker — doesn’t reach the host kernel or another job’s VM: the boundary between jobs is hardware, not “trust the container.” A VM is created for a specific job and destroyed when it finishes, along with everything that ended up inside it.

Our VM has a fixed shape for a specific runner SKU: the vCPU count and memory are set by the hypervisor default, not by hotplug — a job gets exactly the machine it pays for, with no dynamic memory growth. The backend is QEMU/KVM (compatible with privileged Docker-in-Docker inside the VM); we’re considering Cloud Hypervisor as a candidate for faster start-up down the line. We keep the guest kernel and the hypervisor at a current security floor and update them when guest→host-class fixes ship: a second boundary is worth exactly as much as it is patched.

Kata is only one of the layers of our isolation model; how the rest are built and how to verify the runner image yourself is on the “Security & isolation” page.