The layer Docker hides, written out and measured.

What it is

mars implements the OCI runtime-spec. It is the layer Docker and Kubernetes sit on top of — the one that actually creates the namespaces, writes the cgroups, assembles the rootfs and drops the capabilities.

It is not meant for production, and it is not trying to displace runc. It exists because container failures in production happen in the layer Docker hides, and the only way to build a model of that layer is to write it.

Install

Check the host first. Most of what can go wrong is the host, not the build.

./scripts/preflight.sh

On a Debian or Ubuntu host that passes preflight:

sudo apt-get install -y build-essential pkg-config libseccomp-dev jq attr uidmap
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
cargo build --release
sudo install -m 0755 target/release/mars /usr/local/bin/mars

Linux only: it needs namespaces, cgroups and libseccomp, and does not build on macOS. A Lima VM definition is included so the environment is reproducible.

limactl start --name=mars-dev ./lima/mars-dev.yaml
limactl shell mars-dev

First container

Generate a bundle, put a rootfs in it, run it.

mars spec --bundle /tmp/demo
./scripts/make-rootfs.sh /tmp/demo/rootfs
cd /tmp/demo && sudo mars run demo

That is a memory limit, a PID namespace, a pivoted root and a seccomp filter, all built by writing to the kernel rather than by asking a library to do it.

$ docker run --rm --runtime=mars --memory=64m alpine:3.20 cat /sys/fs/cgroup/memory.max
67108864

Docker, using mars instead of runc, enforcing a limit through a cgroup that mars created and wrote itself.

Lifecycle

The full spec surface is implemented: create, start, state, kill, delete, exec, list, ps, pause, resume, events, update, spec, features, all five lifecycle hooks, and a console socket over SCM_RIGHTS.

The three-level fork chain is forced by the kernel, not a style choice.

mars create
  │  socketpair(AF_UNIX)
  ├─ fork() ─────────► [intermediate]
  │                      unshare(NEWUSER|NEWNS|NEWPID|NEWUTS|NEWIPC|NEWNET|NEWTIME)
  │  ◄── "map me" ────
  │  write /proc/<pid>/setgroups, uid_map, gid_map
  │  ─── "mapped" ──►   setresuid(0) — a new user namespace leaves you unmapped
  │                      fork() ──────────► [container init, PID 1]
  │                        exit                mounts, pivot_root, devices
  │  write pid to cgroup.procs                 caps, seccomp, no_new_privs
  │  ◄── "ready" ─────────────────────────────  block on exec.fifo
mars create returns; the container stays alive, waiting
mars start ──── opens the fifo ─────────────────► execve(user process)

unshare(CLONE_NEWPID) does not move the caller into the new PID namespace — the next fork() is what becomes PID 1, and the same holds for CLONE_NEWTIME. uid_map has to be written from outside the user namespace by a privileged process, because a process cannot map itself, so the two ends need two-way synchronisation. And create must return while the container stays alive, so the wait has to be on something a later unrelated process can reach: a fifo, opened O_PATH by the parent so it does not count as an opener, reopened by the init through /proc/self/fd/N because the path is gone after pivot_root.

Rust instead of Go, deliberately. setns(2) refuses to move a multi-threaded process into a new mount or user namespace. Go is already multi-threaded when main starts, which is why runc ships a C constructor that runs before the Go runtime initialises. Rust stays single-threaded until the fork, so mars exec calls setns directly — and asserts the property by counting /proc/self/task rather than assuming it.

cgroup v2

The driver is hand-written against cgroupfs rather than delegated to a crate — memory, cpu, pids, cpuset and io, written directly. Delegating it would delegate away the main thing this project is for.

Writing one pid to cgroup.procs is the most expensive step in starting a container — 55% to 79% of cold start, measured. Creating the cgroup and writing every limit takes 279µs; moving one process into it takes 7–15ms, because the first migration into a fresh cgroup pays for per-cgroup controller setup and an RCU grace period across every CPU. Every container gets a fresh cgroup, so nothing is ever amortised.

trace 10a2ae966c82f09af3c6d2282991b79c  25 spans  11892us total
     458us    279us  cgroup                      create it, write every limit
    1012us    895us  intermediate.unshare.net    a whole network stack
    2358us   7065us  cgroup.attach               write one pid to one file
    9438us    492us  init.rootfs.mount
    9931us    232us  init.pivot_root

Layered rootfs

The overlay rootfs is a documented extension, not a spec feature. The runtime-spec has no field for image layers — that is the image-spec's job, done by containerd or Docker before the runtime is called. mars reads three dev.mars.overlay.* annotations instead, so the config.json stays valid for any other runtime, which will ignore them.

sudo -E ./scripts/oci-bundle.sh -i alpine:3.20 /tmp/layered
cd /tmp/layered && sudo mars run demo
find /tmp/layered/diff -mindepth 1        # everything the container wrote

The .wh. markers from the image tarballs are converted into real OverlayFS whiteouts, and process, env and cwd are taken from the image config.

Hardening

Capabilities, seccomp, no_new_privs, read-only rootfs, maskedPaths and readonlyPaths, sysctls, rlimits, oomScoreAdj, and user namespaces with a newuidmap fallback.

readonlyPaths cannot simply remount a path read-only, because a path is not a mount. It has to be made one first, by bind-mounting it onto itself, and only then remounted with MS_RDONLY. maskedPaths binds /dev/null over a file and an empty read-only tmpfs over a directory — which is how /proc/kcore, a mapping of all physical memory, stops being readable inside a container.

Run the OCI validation suite against either runtime and compare:

sudo -E ./scripts/run-validation.sh
sudo -E RUNTIME=runc ./scripts/run-validation.sh

As a Docker runtime

sudo ./scripts/install-docker-runtime.sh          # TRACE=1 also logs how Docker calls it
docker run --rm -it --runtime=mars alpine:3.20 sh

TRACE=1 is there because the interesting part is not that it works, but the exact sequence of calls Docker makes to a runtime it has never seen before.

Startup traces

Every startup phase emits an OTLP span, accepted by Tempo. No collector needed to look at them:

./scripts/otlp-echo.py 4318 &
MARS_OTLP_ENDPOINT=127.0.0.1:4318 sudo -E mars run demo

The exporter is hand-written for a hard reason. The OpenTelemetry SDK runs its exporter on a background thread. A process that forks must not have one — only async-signal-safe work is legal in the child — and a process that calls setns must not either. So the exporter is around 120 lines that build OTLP/HTTP JSON and write one POST: no threads, nothing running at fork() time.

Attack surface

A runtime spends its whole life as root. It holds CAP_SYS_ADMIN, it creates the namespaces and writes the cgroups, and then it exits. Everything dangerous it will ever do, it does in that window — and published runtime comparisons measure startup latency and per-container memory, not that.

scripts/hap-bench.sh counts the distinct host kernel functions a runtime traverses while privileged, traced with ftrace, following Bottomley's horizontal attack profile. Five runs per cell, median, idle subtracted, all three runtimes sharing one unmodified seccomp profile:

24% less kernel than crun and 35% less than runc on a plain start, and 44% / 56% less on exec — the operation a Kubernetes exec probe repeats for the lifetime of a pod.

It is not that mars skips work. Namespace creation, pivot_root, cgroup setup and capability handling all appear at parity. Of the 1000 functions runc reaches and mars does not, only 10 are thread, futex or scheduler functions — the Go runtime is not the explanation. 301 are file, path and /proc traversal.

Widening that benchmark to cover seccomp is what found the one real defect so far: a rule that restates the filter's default action, which libseccomp refuses as redundant and mars was treating as fatal. Every standard profile carries dozens of them.

Failure modes

Nine failures that happen in production, reproduced here with the evidence read out of the kernel:

• a pod is OOMKilled with exit 137 — and memory.events says how close it had been, for how long
• a container ignores SIGTERM for the full grace period — because PID 1 gets no default handlers
• zombies pile up until fork() fails — because PID 1 was never written to be an init
• CPU throttles at 10% utilisation — because cpu.max is a quota, not a share
• a fix applied with exec survives a restart but not a recreate — because it lived in the OverlayFS upper layer, which is the container
• a rootless bind mount is unwritable at mode 0777 — because the uid has no mapping and reads as nobody
• EPERM mounting /sys/fs/cgroup — because the bundle asked for cgroup v1 on a v2 host

Before production

Not finished: rootless without any privilege. The user namespace machinery works and is tested, but mars still expects to be started with privilege. A fully rootless run also needs a delegated cgroup under user.slice, fuse-overlayfs or userxattr for whiteouts, and slirp4netns for networking.

Verified environment: Ubuntu 24.04, kernel 6.8, aarch64, pure cgroup v2 with cpu cpuset io memory pids delegated, unprivileged user namespaces enabled, and runc 1.5.1 plus Docker 29.7.2 alongside for comparison.

Out of scope

Left out on purpose, each because something else already owns it: