use case / agent evals

every run starts from the same clean machine.

evals lie when state leaks between runs. boot identical microvms in milliseconds, run the task, keep the disk when you need evidence, and delete the rest.

measured release: microsandbox 0.4.5, inferred from benchmark chronology because the raw artifact does not record the binary version; owner confirmation is pending. median of 10 measured runs after 2 warmups; end-to-end wall time from cli invocation to process exit; pre-pulled alpine userspace; bare-metal linux x86_64 on a gcp c3-standard-192-metal host (intel sapphire rapids, ubuntu 24.04, /dev/kvm). microsandbox: 320 ms; docker: 463 ms; firecracker: 808 ms. firecracker used the same alpine userspace and harness; these numbers do not use firecracker's narrower kernel-to-userspace boundary. full harness and raw results.

dithered archive shelves representing reproducible saved disk state

reproducibility is a boundary problem

same image, same limits, same network policy, every run. the only variable left is the agent you're measuring.

start each task in a fresh microvm so one run cannot leave packages, files, or processes behind for the next.

keep the evidence, not the machine

snapshots capture the disk state of a stopped sandbox.

save the runs that matter and rerun them later from the same bytes.

scale the fleet, not the risk

run hundreds of tasks in parallel, each in its own kernel. one flaky run can't poison the batch.

each result belongs to one machine with its own limits and lifecycle, not a shared worker process.

keep local reproduction possible

one sdk. one api. local or cloud is a config change, not a rewrite.

pin the same image, snapshot name, resource request, and command when reproducing a failed task.

publish the harness with the number

the cold-start claim is labelled as microsandbox 0.4.5 because the public artifact does not record its binary version and that release is inferred from chronology. the benchmark page keeps that caveat adjacent.

how it works

repeat the machine, not the residue

snapshot a stopped base

prepare the filesystem once, stop the sandbox, and capture its disk state.

start each run from it

boot a fresh microvm from the same saved disk, image, limits, and network policy.

run the eval

execute the task and collect its output inside that run's own kernel.

keep or discard the disk

snapshot a stopped run when you need evidence and remove the rest.

see it work

create, archive, and run from a stopped-sandbox snapshot

bash

msb stop eval-base
msb snapshot create swebench-base --from eval-base
msb snapshot save swebench-base ./swebench-base.tar.zst --with-image
msb run --from-snapshot swebench-base --name task-0441 -- pytest -q
msb rm --force task-0441

know the limits

snapshots do not promise captured guest memory or live process state.
gpu support is not offered today.
the public cold-start artifact measures 0.4.5 by chronology, not the pinned 0.6.8 release.

evaluate the alternatives

vs modal vs firecracker

run your eval suite on disposable computers.