Firecracker on Linux

grain can launch sandboxes with Firecracker instead of QEMU on Linux + KVM.

Support policy

Default remains hypervisor: qemu. Firecracker on grain is supported (not experimental) for the rows above. Jailer-less launch (single-tenant only). Nested KVM: works when the outer hypervisor exposes vmx/svm so /dev/kvm exists in this guest.

macOS, missing Firecracker binary, or unusable /dev/kvm fail with clear errors (grain doctor and create). Networking failures without CAP_NET_ADMIN mention privilege in the error.

When to use this path

Quick config

# ~/.grain/config.yaml
hypervisor: firecracker
firecracker_binary: firecracker   # PATH lookup (default)
kernel_path: ""                   # optional; default ~/.grain/kernels/vmlinux

See Configuration.

Requirements

Operator checklist (vFC-1 production path)

1. Linux host with /dev/kvm RDWR for the daemon user (add user to kvm group if needed).
2. Install Firecracker and put it on PATH (or set firecracker_binary).
3. Pull catalog artifacts: grain image pull fc-kernel and grain image pull grain-ubuntu-fc (or BYO: place vmlinux / grain image import …).
4. Set hypervisor: firecracker and preferably image: grain-ubuntu-fc in ~/.grain/config.yaml, then grain up (restart daemon if it was already running).
5. Run grain doctor and fix every ✗ before grain new -i grain-ubuntu-fc --wait agent.
6. Optional BYO: raw rootfs via grain image import ./rootfs.ext4 --id my-fc-rootfs (not catalog qcow2 cloud images).

grain doctor (Firecracker)

With hypervisor: firecracker in config:

grain doctor

Hard failures print ✗ and exit non-zero. Soft items print · and do not fail doctor.

Doctor distinguishes “no kernel at the default path” (missing Grain/BYO artifact) from “you set kernel_path and that file is gone” (BYO misconfigured).

If grain new fails, prefer the create error and ~/.grain/logs/<name>.log over later agent/vsock messages — Firecracker often exits immediately when KVM is unavailable (firecracker exited immediately + KVM hint).

Image / rootfs notes

grain’s QEMU catalog images (ubuntu-cloud, grain-ubuntu, alpine-cloud) are qcow2 cloud images aimed at QEMU + cloud-init. They are not drop-in Firecracker guests.

Firecracker catalog IDs (vFC-1 production)

These IDs are explicit (not dual-use of grain-ubuntu qcow2) so operators and tooling never confuse QEMU cloud images with FC raw + kernel. Pull is the happy path; BYO import remains first-class.

Pull (published fc-latest)

# config: hypervisor: firecracker
grain image pull fc-kernel          # → data_dir/kernels/vmlinux
grain image pull grain-ubuntu-fc    # → images/grain-ubuntu-fc/disk.raw
grain new -i grain-ubuntu-fc --wait agent
# or: ./scripts/smoke-fc.sh

Catalog digests use companion .sha256 sidecars on the fc-latest release (fail-closed; same pattern as grain-ubuntu).

Bake (Linux rebuild)

# curl, qemu-img, unsquashfs, mkfs.ext4, go
./scripts/bake-fc.sh --all
# → dist/fc/vmlinux-<arch> + grain-ubuntu-fc-<arch>.raw (+ .sha256)

grain image import dist/fc/vmlinux-amd64 --id fc-kernel
grain image import dist/fc/grain-ubuntu-fc-amd64.raw --id grain-ubuntu-fc

Defaults: Firecracker CI v1.12 vmlinux-6.1.128 + ubuntu-24.04.squashfs → ext4 with agent systemd unit (vsock :7475). Override with FC_CI_VERSION / FC_KERNEL_VER / FC_UBUNTU_SQFS. Workflow Bake Firecracker artifacts rewrites the fc-latest release.

Operator path today (BYO)

1. Prefer a raw golden rootfs (ext4/squashfs layout that boots with the FC kernel’s root=/dev/vda).
2. If the VM disk is still qcow2, Start runs qemu-img convert -O raw into disk.raw under the VM dir (when qemu-img is available). Otherwise Start refuses with a conversion hint.
3. Standard Ubuntu cloud images need a matching Firecracker-capable kernel; they are not drop-in FC guests without extra work (kernel + init + virtio drivers).

# BYO kernel → catalog id (installs under data_dir/kernels/vmlinux)
grain image import ./vmlinux --id fc-kernel

# BYO raw rootfs → catalog id (images/grain-ubuntu-fc/disk.raw; keeps format raw)
grain image import ./rootfs.ext4 --id grain-ubuntu-fc

# Create with the FC rootfs id (hypervisor: firecracker in config)
grain new -i grain-ubuntu-fc --wait agent

See also Images for the QEMU/golden workflow. FC agent production (vFC-1) is a separate catalog/kernel path from QEMU cloud images.

Suggested layout

~/.grain/
  kernels/
    vmlinux              # Firecracker guest kernel
  config.yaml            # hypervisor: firecracker
  vms/<name>/
    disk.raw             # rootfs (converted or imported)
    firecracker.json     # generated config
    firecracker.sock     # FC API unix socket
    fc-vsock.sock        # host end of virtio-vsock
    firecracker.pid

Networking and agent

This backend is CNI-less / jailer-less. Agent uses vsock; optional TAP + host TCP proxy provides publish/fwd (not SLIRP).

Publish example (Linux + CAP_NET_ADMIN)

# config: hypervisor: firecracker
grain image pull fc-kernel
grain image pull grain-ubuntu-fc
grain up
grain new -i grain-ubuntu-fc -n fcweb -P 18080:80 --wait agent
# Guest eth0 is configured via agent after boot; then:
curl -sS http://127.0.0.1:18080/   # if guest serves :80
grain fwd add fcweb 19000:7475     # live TCP proxy to guest
grain fwd ls fcweb

Smoke: ./scripts/smoke-fc-net.sh (guest HTTP listener + host curl for create-time -P and live fwd).

On Start, grain:

• Allocates a guest CID (AgentCID) and configures Firecracker vsock with uds_path = …/fc-vsock.sock
• When net is enabled: creates TAP, allocates SSH/agent/publish host ports; after agent is up, configures guest eth0 via agent exec and starts host TCP proxies for create-time -P, SSH, and agent TCP (same path as live grain fwd add)

Firecracker’s host-side vsock is not AF_VSOCK//dev/vhost-vsock. Host clients connect to the UDS and send CONNECT <port>\n (see Firecracker vsock docs). Guest agent listens on AF_VSOCK port 7475. Host agent.Dial prefers UDS + CONNECT (vFC-1).

For the QEMU networking model (SLIRP, publish, live forwards), see Networking.

Start / stop / pause

Logs: ~/.grain/logs/<name>.log (Firecracker stdout/stderr). grain logs --qemu <name> shows that hypervisor log (name is historical).

Cloud-init seed

If seed.iso exists in the VM dir, it is attached as a second read-only drive (cidata). NoCloud typically expects a labeled ISO/FAT volume; success depends on the guest image. Prefer baking keys/agent into a FC-oriented rootfs for reliable boots.

Example

# ~/.grain/config.yaml
hypervisor: firecracker
firecracker_binary: firecracker
image: grain-ubuntu-fc
cpus: 2
memory_mb: 1024

# vFC-1 production path (published fc-latest)
grain image pull fc-kernel
grain image pull grain-ubuntu-fc
grain up
grain doctor
grain new -i grain-ubuntu-fc --wait agent

# BYO alternative
# grain image import ./rootfs.ext4 --id my-fc-rootfs
# grain new -i my-fc-rootfs --wait agent

Known limitations vs QEMU

Not on FC today (use QEMU): multi-host CNI, overlay L2, 9p/virtiofs mounts, SLIRP proxy, virtio GPU. Jailer multi-tenant claims are out of scope.

vFC-1 (agent) + vFC-2 (partial net) shipped: pullable fc-latest, vsock agent, TAP publish/fwd. Full table: Hypervisor matrix.

Boot metric (reference SKU)

Primary project metric: wall time for grain new -i grain-ubuntu-fc --wait agent (create through agent ready).

Nested virt is slower than bare-metal KVM; re-run bench-fc.sh on your class before publishing numbers in a release.

Related

• Hypervisor matrix — QEMU vs FC today + vFC-1 / vFC-2 targets
• Images — base images, golden bake (QEMU-oriented); FC rootfs notes
• Networking — QEMU SLIRP / hostfwd (FC uses TAP + TCP proxy)
• Guest agent — guest agent HTTP API
• Troubleshooting — doctor and logs (includes Firecracker doctor rows)
• Configuration — hypervisor, firecracker_binary, kernel_path
• Concepts — hypervisor glossary
• Product surface — FC agent production vs QEMU-only net