Guest agent (exec, shell, files, health)

The guest agent is a small HTTP server that runs inside each Linux VM. The host CLI and daemon talk to it over either virtio-vsock (when the host supports it) or a QEMU SLIRP hostfwd to guest port 7475, so common operations work without opening an interactive SSH session.

On Firecracker there is no SLIRP hostfwd: the guest still listens on AF_VSOCK port 7475, and the host side is Firecracker’s vsock UDS (fc-vsock.sock + CONNECT). Optional host TCP proxy can expose guest ports (including agent :7475) after TAP is up. See Firecracker on Linux.

What it provides

Buffered exec returns JSON (stdout/stderr/exit_code). Streaming exec (buffered=false) emits NDJSON frames: started → stdout/stderr → exit.

Lifecycle

grain new / start
    → SSH ready (default --wait ssh)
    → if image has no baked agent: SCP grain-agent-linux-$arch + systemd unit
    → wait GET /health (soft-fail unless --wait agent|userdata)

Golden images (grain-ubuntu via grain image import / bake) set has_agent so create prefers probing the agent before SSH deploy. See Images.

Deploy (host → guest)

Deploy runs only when a Linux agent binary is found. Search order (internal/agent.LinuxBinaryPath):

1. Directory of the running grain executable
2. bin/ next to that executable
3. ./bin/ / . relative to cwd
4. ~/.grain/agent/grain-agent-linux-$GOARCH (data dir)

Build both arches:

just agent-linux
# → bin/grain-agent-linux-arm64
# → bin/grain-agent-linux-amd64

Guest install path: /usr/local/bin/grain-agent
Unit: /etc/systemd/system/grain-agent.service (listen :7475)

grain doctor warns (does not fail) when the binary is missing — VMs still work SSH-only.

CLI

Health

grain agent health
grain agent health sbox-1
# {
#   "hostname": "sbox-1",
#   "agent_version": "0.2.0",
#   "agent_uptime_sec": 42,
#   "userdata_ran": true
# }

Stats

Guest resource basics (GET /stats on the agent): uptime, memory total/available, load average, optional disk totals. Collected from /proc on Linux. Useful for agents and CI to decide whether the sandbox is healthy under load. Host-level counters live on the daemon Prometheus endpoint (GET /metrics, grain_vms_*).

Secrets

The guest agent can materialize a secret file (POST /secrets/materialize) with base64 payload, optional mode/uid/gid, default path /run/grain/secrets/<name>. The host keeps a file-backed store under ~/.grain/secrets/ (mode 0700) for daemon-mediated inject workflows. Prefer short-lived lab credentials; do not treat this as a full KMS.

Exec

grain x sbox-1 -- uname -a
grain x -- id                    # single VM: name optional
grain x --agent -- true          # require agent
grain x --ssh -- true            # force SSH

Interactive shell

grain sh prefers an interactive PTY over the guest agent WebSocket (GET /shell) so a shell works even when sshd is down, as long as the agent is healthy. The CLI dials the host-forwarded agent port directly (same pattern as cp / fs). Falls back to SSH when the agent is missing or unhealthy.

grain sh                         # auto-create if no VMs; agent PTY if up, else SSH
grain sh sbox-1
grain sh --agent sbox-1          # require agent PTY (error if unavailable)
grain sh --ssh sbox-1            # force classic SSH

Clipboard (OSC 52)

Copy from tools inside the guest (for example Grok Build) reaches your laptop in two hops:

Shims live on PATH at /var/lib/grain/bin/ (installed when the agent starts). They advertise TARGETS / MIME types for tools that probe before reading image data.

If Grok (or similar) says clipboard unavailable or paste fails:

1. Redeploy a current agent and open a new grain sh: grain agent deploy NAME (restarts the unit). Image paste needs both an updated CLI (host clipboard read) and guest agent.
2. Inside the guest: which pbcopy pbpaste → /var/lib/grain/bin/…. For Grok/arboard image paste: install xvfb in the guest (sudo apt-get install -y xvfb) so the agent can own X11 CLIPBOARD; confirm echo $DISPLAY is :7 in a fresh grain sh.
3. On the machine running grain sh, ensure system clipboard tools exist (pbcopy / AppKit via Swift on macOS, or wl-copy / wl-paste / xclip on Linux).
4. Paste needs an active interactive shell session; grain x has no clipboard bridge. The clipboard client is whichever host runs grain sh (local or remote GRAIN_API).
5. Terminal selection copy (mouse drag + Cmd/Ctrl+C) is the host terminal UI, not grain.

Terminal identity (Shift+Enter / keyboard protocol)

grain sh (local and remote/GRAIN_API) forwards host terminal identity into the guest PTY so TUIs and multiplexers can match local keyboard behavior. Without this, the guest only sees a generic TERM=xterm-256color and modified keys (especially Shift+Enter) often collapse to plain Enter.

Forwarded when set on the client:

Troubleshooting Shift+Enter / odd keys inside grain sh:

1. On the client (where you run grain sh), check echo $TERM_PROGRAM $LC_TERMINAL $TERM_FEATURES.
2. Inside the guest session: echo $TERM_PROGRAM $LC_TERMINAL $TERM_FEATURES — should match the client after a new grain sh (not an old session).
3. Use a CLI and guest agent that include terminal-env forwarding (0.5.0+ for the first cut; later builds add iTerm/Kitty/WezTerm keys). Redeploy the agent after upgrading the host: grain agent deploy NAME.
4. Host terminal must actually emit a modified-Enter sequence (iTerm2, Ghostty, Kitty, WezTerm with Kitty keyboard enabled: e.g. WezTerm enable_kitty_keyboard = true).
5. Nested tmux/screen: enable extended keys in the multiplexer (tmux 3.2+: set -s extended-keys on) and ensure the outer terminal identity still reaches the app.

If Shift+Enter still fails, try Alt+Enter as a fallback.

Protocol: WebSocket binary frames carry PTY bytes both ways; optional text JSON control frames resize the PTY ({"type":"resize","cols":N,"rows":M}). The agent spawns a login shell as uid 1000 when present, otherwise root. Local stdin is put in raw mode when attached to a TTY; SIGWINCH is forwarded as resize frames.

Disconnect / host daemon restart: Restarting the grain daemon drops all remote and local grain sh WebSocket sessions. The client restores cooked mode and clears common private modes (alternate screen, mouse tracking, Kitty keyboard, …) so the terminal is usable again. If the screen is still garbled, run reset. Reconnect with grain sh NAME once the daemon is back.

Copy

# host → guest
grain cp ./local.txt sbox-1:/tmp/local.txt
# guest → host (reverse)
grain cp sbox-1:/var/log/cloud-init.log ./cloud-init.log
grain cp --agent ./a sbox-1:/tmp/a
grain cp --ssh   ./a sbox-1:/tmp/a

Incremental sync

For iterative edit loops (seed → work in guest → bring results home), prefer grain sync over full-tree cp:

grain sync push ~/proj sbox-1:/work/proj
grain sh sbox-1
grain sync pull sbox-1:/work/proj ~/proj

Requires a healthy agent. See CLI reference.

Refresh / redeploy the guest agent

After upgrading the host CLI, an older in-guest agent may lack new features (terminal env, readiness fields, …). Redeploy so the guest picks up the new binary:

just agent-linux                 # if bin/grain-agent-linux-$arch is missing on the daemon host
grain agent deploy sbox-1        # SCP + systemd enable (local or remote CLI)
grain agent health sbox-1        # confirm agent_version

With GRAIN_API, the CLI calls POST /vms/{name}/agent/deploy so SSH hostfwd runs on the sandbox host. The agent binary must exist on the daemon host (not the laptop). Or recreate the VM from a golden image with a baked agent.

grain doctor reports whether the host-side agent binary is present (soft warning if missing).

Filesystem (agent only)

grain fs ls   sbox-1 /tmp
grain fs stat sbox-1 /etc/os-release
grain fs mkdir -p sbox-1 /tmp/a/b
grain fs rm -r sbox-1 /tmp/a

Networking

Every VM gets a host-forwarded agent port (metadata agent_port → guest 7475), same loopback-only pattern as SSH. List with:

grain fwd ls

Trust model

The guest agent has no auth token. Security depends on reachability:

Do not publish guest 7475 with -P, and do not expose hostfwd agent ports off loopback. On a shared host, prefer remote API access over tunneling raw agent ports. Overlay details: Overlay network. Broader model: Security model.

Agent transport (vsock vs TCP)

Config (~/.grain/config.yaml):

agent_transport: auto  # auto | tcp | vsock

• auto — prefer vsock when /dev/vhost-vsock exists; otherwise TCP.
• tcp — always SLIRP hostfwd (typical on macOS).
• vsock — force QEMU -device vhost-vsock-pci,guest-cid=<CID> (requires the host device).

TCP hostfwd is always configured as a fallback even when vsock is selected. The guest agent always listens on TCP :7475 and additionally tries AF_VSOCK port 7475 on Linux (listen failure is ignored — TCP-only is fine).

Host dial path (agent.Dial):

1. If agent_cid > 0, try vsock CID:7475
2. Else (or on vsock failure) use http://127.0.0.1:agent_port

Instance metadata:

Soft-fail and fallbacks

• Missing agent binary → no deploy; SSH still works.
• Unhealthy agent → grain sh / grain x / grain cp fall back to SSH/scp unless --agent.
• grain fs and grain sync require a healthy agent (no SSH fallback).

Go SDK and OpenAPI

Programmatic access uses the public client package and the daemon OpenAPI surface:

import "github.com/cxdy/grain/client"

c, err := client.DialUnix(filepath.Join(home, ".grain", "grain.sock"))
// c.Exec, c.ExecStream, c.AgentHealth, c.PutFile, …

• Spec: api/openapi.yaml (embedded in the binary for discovery)
• Auth: set api_token in config or GRAIN_TOKEN — see README API / Config

Related

• Images — golden agent-baked images
• Coding agent recipe — mount repo, run agent, cp results
• CI ephemeral recipe — create → x → rm
• Troubleshooting — doctor, logs, timeouts