SPACE: Architecture

A sandbox, a lifecycle operation, and a command stream have separate lifetimes. A sandbox can outlive both the request that created it and the connection carrying its output.

Core concepts

A template reference such as seed:latest selects a published release; a digest reference selects specific content. A sandbox response reports the resolved template digest so callers can record what they ran.

Operators can restrict newly registered template artifacts with SPACE_TRUSTED_TEMPLATE_STORAGE_BUCKETS on the internal API: a comma-separated list of exact S3 bucket names. A configured policy rejects other buckets and all endpoint overrides before writing the release or its tags. An unset or empty policy preserves local S3-compatible storage. This policy does not audit existing releases or replace publisher authorization; provision the allowlist before promoting an enforcement image.

Template lookup rollout

Template lookup clients attach scope metadata and compatible servers return a receipt. A missing receipt preserves the existing lookup behavior during a rolling upgrade. A mismatched receipt fails the request. Authentication, organization checks, and cache decisions continue to use the existing rules.

The auth.template_lookup.compatibility counter records missing_scope, missing_receipt, and received, under the emitting service's identity. A receipt confirms only that one peer received the metadata. It does not certify that a downstream peer received it or that the scope was authorized. Check all participating services and exercise cache misses as well as hits; an idle service or a cached lookup cannot prove the catalog supports the protocol.

Deploy this compatibility phase before introducing stricter lookup rules. Strict enforcement requires a separate authorization acknowledgement; it must not accept the compatibility receipt as proof. Existing clients that omit scope must be migrated before enforcing it. Rollback to the previous version remains compatible with these requests and replies.

Request and execution flow

The API records desired state in PostgreSQL. Controllers and worker services reconcile the running system with that state, so an accepted request is not a readiness signal.

SPACE can manage workers directly or through Kubernetes. The direct adapter calls worker services without creating Kubernetes sandbox pods. The Kubernetes adapter is used by the local Kind installation in these guides.

Local Kind topology

The diagram below shows the Kind deployment used in these guides, not a requirement for every SPACE deployment.

Scroll horizontally to view the full diagram.

flowchart TB
    accTitle: SPACE local Kind request and execution flow
    accDescr: Management requests and command streams follow separate paths. The API stores desired state in PostgreSQL. The cluster controller, Kubernetes sandbox pod, runtime shim and worker services arrange the Firecracker microVM and guest process. Worker services exchange templates and snapshots with object storage. The data plane connects the application to the guest process.
    application["Application"]
    api["API"]
    postgres[("PostgreSQL")]
    controller["Cluster controller"]
    pod["Kubernetes sandbox pod"]
    shim["SPACE runtime shim"]
    workers["Worker services"]
    storage[("Object storage<br/>templates / snapshots")]
    vm["Firecracker microVM"]
    data["Data plane"]
    guest["Guest process"]

    application -->|Management| api
    api --> postgres
    postgres -->|desired state| controller
    controller --> pod
    pod --> shim
    shim --> workers
    workers <--> storage
    workers --> vm
    vm --> guest
    application -->|Command stream| data
    data <--> guest

• Create: The API resolves the template and accepts the requested sandbox. The client retains the sandbox ID and operation ID.
• Become ready: Controllers arrange execution through the selected adapter and worker services. In Kind, this includes Kubernetes sandbox pods and the runtime shim. The client waits for operation success, a running sandbox, and a reachable process endpoint.
• Execute: The API creates a command session. The client attaches to the returned events endpoint, starting the guest process and receiving output.
• Finish: The client checks the process exit result, then keeps the sandbox for another command or deletes it. A completed command does not itself delete the sandbox.

An operation's state describes the lifecycle request; the sandbox's state describes the environment. Neither is the exit status of a command.

Native create, pause, suspend, wake, and delete operations have a 30-minute deadline from when they were requested. One that has started and is still retrying when it passes fails with operation_failed and the reason operation_deadline_exceeded. A create, pause, suspend, or wake that has not started by then, because it waited behind earlier work on its sandbox, for a free execution slot, or for the cluster controller to come back, ends expired with the same code and reason instead of starting late. Inspect the sandbox before requesting it again. A delete still runs however late it starts: expiring it would leave running a sandbox its owner asked to remove.

Requesting a delete cancels the sandbox's create if that create is still queued or waiting to retry: the create ends canceled, with operation_failed and the reason superseded_by_delete, and the delete proceeds. A create the delete cannot cancel, because its attempt was running or its claim had gone stale, still holds the delete back until that attempt ends; if the attempt would retry, or the dispatcher hands the create back without running it, the create ends canceled instead, so it never runs after the delete, and its error keeps the attempt's last progress message and error fields beside superseded_by_delete. The delete's request and a create's requeue each lock the create first, so a delete arriving as an attempt ends is always seen by one of them. One case still requeues the create: a dispatcher that reclaims a create whose earlier execution lost its claim but still runs in the same process hands it back unchanged, since canceling it would let the delete start under that execution; the delete then waits for one more attempt.

A Native create can answer 202 before its execution records the sandbox. A DELETE /v1/sandboxes/{sandbox_id} in that window is accepted while the create is queued or running, for a caller who could see that create's operation: it answers with a delete operation, as for a sandbox already creating, and cancels or waits for the create as described above. A create that succeeds records the sandbox before it is marked succeeded, so a delete that races it still finds the sandbox. A create that fails, expires, or is canceled by an earlier delete can end without recording the sandbox, and a delete with a new Idempotency-Key then answers 404. Every other Native request naming the sandbox, such as GET /v1/sandboxes/{sandbox_id}, answers 404 until the sandbox is recorded; poll the create operation instead.

Inline-durable execution

Native lifecycle operations are durable records in PostgreSQL. By default the cluster controller's dispatcher claims and runs each one: it runs on the elected leader only, with a bounded number of execution slots and a per-organization slot limit. With inline execution on, the API process that admits a cold create, a :wake, :resume or :start, or the wake a command, process session or attach queues, can claim and run that operation itself, in the same transaction that records it. It does so only when:

• the env master SPACE_NATIVE_INLINE_OPERATIONS_ENABLED is on, the kind is in SPACE_NATIVE_INLINE_KINDS, and the space-native-inline-operations flag allows the organization;
• nothing else for the sandbox is queued or running;
• the process has one of its SPACE_NATIVE_INLINE_MAX_CONCURRENT permits free and is not draining. Admission takes the permit only here, after it has created the operation and found nothing ahead of it, so a request that joins other work, or waits on the sandbox's locks, holds none;
• the organization is under SPACE_NATIVE_INLINE_ORG_LIMIT.

Anything else records the operation queued exactly as before, for the dispatcher. configuration.md describes each setting. A create that wins an admission-time warm-pool claim runs inline under its own switch, SPACE_NATIVE_CREATE_INLINE_ENABLED, not these.

The master goes on only once every API process, on every surface, takes the per-sandbox queue lock below, and off before the dispatcher's keyed claim: the admission claim is sound only while both hold.

The record, the claim and the execution are the dispatcher's. The API runs the same executors through the same driver, so claim renewal, retry deferral and the operation deadline behave the same, and every retry is requeued for the dispatcher. The claim commits in the task that runs the operation, not in the request, so a client that disconnects mid-commit cannot leave a claim that nothing runs. If the API process dies mid-execution, its claim stops being renewed. After the claim staleness window (60 s) the dispatcher claims the same operation again, so nothing is lost, though that operation finishes later than a queued one would have. A graceful API shutdown waits for running inline work within its drain budget and queues new work meanwhile.

Inline claims keep one execution per sandbox, in request order:

• Every insert for a sandbox takes a per-sandbox queue lock until it commits. So the admission's check that nothing else is pending sees everything inserted before it. A later insert commits only after the claim, and the dispatcher's claim then sees that claim live and holds the later operation back.
• A caller's :wake joins a system wake only while that wake is still queued. Behind one the API is already running, the :wake is queued and completes as already done.

Two limits are counted separately. The organization limit counts the organization's live inline claims and is soft: admissions that commit at the same moment can pass it by up to their number less one. The dispatcher's per-organization slot limit does not count inline claims, and the inline limit does not count dispatcher claims. Suspend, pause, stop, delete and kill always run on the dispatcher.

In Kind, sandbox pods run in their own namespace, sandboxes, apart from the control plane in space. NetworkPolicies there admit only the API and cluster controller, on the port they dial, and limit sandbox egress to the public IPv4 internet and cluster DNS; the Kind harness README lists each rule's source.

Lifecycle and persistence

Snapshot upload streaming describes the worker transfer pipeline, memory bounds, failure ownership, and rollout behavior.

Pause and suspend both stop execution, but retain state differently. Use wake to return a paused or suspended sandbox to running, and wait for its operation and readiness before issuing another command.

The stored vCPU shape is the sandbox's activation allocation. A live CPU update changes the current execution's scheduling quota within its admitted ceiling; it does not resize the guest or replace that stored shape. Pause/resume keeps the live quota. A new start or restore returns to the activation allocation, so a temporary reduction does not permanently prevent raising the quota again.

By default, sandboxes created through the SPACE API suspend after six hours without access or command activity. This is an idle limit, not a maximum lifetime.

A created or overwritten file that can replace its destination is published atomically when its writes complete, and persisted at lifecycle boundaries, not per write: an acknowledged one is not guaranteed to survive a sandbox crash, forced stop or host failure. Guest file and disk I/O describes the guest's disk, its per-sandbox limits, the upload and read paths, and what each lifecycle operation persists.

The E2B Compatibility Wrapper (ECW) is SPACE's E2B-compatible API surface. It translates E2B-compatible requests and responses at the edge and is distinct from the SPACE API. The ECW guide defines its tested compatibility contract and configuration. ECW uses different lifecycle names: its pause maps to SPACE suspend, and its resume maps to SPACE restore.

Service-port ingress has a separate connection lifetime. With ingress protection enabled and completion framing negotiated, automatic lifecycle transitions defer until the HTTP response body or WebSocket connection completes; explicit lifecycle operations retain their existing authority. Ingress wake authorizes access and respects the sandbox's access-wake policy before preparing a dormant backend. Route resolution first prepares the current worker path. If activation or recovery is needed, the cluster API delegates through the authenticated IngressActivationApi to the API process's existing lifecycle controller. It re-resolves and reauthorizes after activation, within the original request deadline. The cluster API does not construct a Kubernetes substrate or retain the sandbox-pod reflector cache. Accepted activation remains owned by the API's tracked lifecycle tasks and existing execution leases; a caller timeout only ends its wait. Transport idle limits still apply, and SPACE does not generate WebSocket keepalives. During coordinated deployment, a temporary metadata opt-in falls back to ordinary stream termination for older callers or gateways. Drain existing attachments before rollback; old decoding forwarders are not a supported steady state. The internal API retains explicit loopback preparation until all workers provide attachment readiness. Fallback connections do not provide full downstream lifetime protection, so retain caller renewals. Remove this metadata opt-in and the fallback branches after deployment is complete. Reconnect deadline updates require a specific worker response-metadata acknowledgement; ordinary touch success cannot stand in for a successful extension. Reconnect extends a deadline floor, not the configured idle window; changing that window requires an explicit timeout update. When API identity publication lags, deadline operations obtain the current execution generation from the worker before applying a fenced mutation. This lookup remains necessary after rollout; missing API identity is not proof of a dormant execution. Ingress protection does not cover unrelated background operations. Accepted ingress activation survives request cancellation. The API's HTTP and activation listeners share its shutdown signal and drain budget. Draining is best-effort within the deployment termination grace; existing activation leases provide recovery if the process exits before completion.

Lifecycle policy at activation

Timeout updates use durable execution ownership rather than a lifecycle label alone. A Kubernetes sandbox can retain lifecycle started after its execution has retired and placement has cleared. With no execution, activation lease, route, or node assignment, the policy update succeeds without waking it. Direct-mode missing placement does not provide the same proof of absence.

The next activation reads policy after claiming execution ownership. Start (including snapshot restore), Wake, and resident Resume can carry that policy to the worker. The worker installs it atomically with activation admission, after snapshot materialization and before arming inactivity timers. Omission preserves the worker policy; local overrides remain authoritative. Retrying an already-running activation does not replace its policy or reset its deadline.

Resident policy updates compare the observed execution generation atomically with resume admission. An absent generation matches only a legacy resident that still has none. Wake carries this resume fence separately from fresh activation authority. If the resident retires after observation, Wake requests a fresh identity through the normal activation handshake. A fence without policy is invalid; policy omission preserves legacy behavior without a resume fence. Capability checks use the resolved runtime connection rather than asynchronous placement observations.

An identified execution still starting may be observed within the original operation deadline before a timeout reset is sent. Observation does not start another execution or retarget a replacement generation. The reset is sent once: an ambiguous transport failure does not establish that repeating a clock reset is safe.

Service responsibilities

Sandbox-manager waits for node-manager before advertising takeover readiness and again before acquiring the startup-recovery work gate. Temporary dependency failures keep the process waiting with bounded individual attempts and periodic warnings; there is no overall dependency-wait deadline. Its optional local health socket serves liveness while withholding readiness during these waits. Deployment probes must distinguish liveness from readiness; external startup deadlines still apply. Local execution-authority validation remains under recovery admission. These availability checks do not guarantee that node-manager will stay available through recovery, and do not extend any caller's request deadline.

Existing namespace containment runs with at most four workers, preserving the order of checks within each namespace. All checks finish before admission; any failure prevents activation. Shutdown stops scheduling new namespaces and waits for in-flight host commands before releasing service ownership. The pool opens on tracked blocking work so its host commands do not stall local liveness.

An opt-in Linux benchmark exercises production containment against synthetic network namespaces and nftables rules, comparing sequential and bounded parallel passes. Run only inside a disposable container with network namespace capabilities:

SPACE_RUN_CONTAINMENT_BENCHMARK=1 cargo test -p pplx-space-node-mgr --lib \
  linux_containment_benchmark -- --ignored --nocapture

Crate layout

The Rust workspace is layered, bottom-up: generic building blocks (l10_foundation), SPACE models and wire protocols without I/O (l20_domain), stores (l30_storage), use cases and service entry points with their clients (l40_api), and thin binaries (l50_app), with test_support and tools beside them. Each is a directory under crates/ whose AGENTS.md says what belongs there; "Layers" in archcheck's README gives the dependency rules CI enforces.

Security boundaries

Firecracker isolates guest execution. Host-side services still require privileges to manage VMs, storage, and networking.

The API authorizes management requests with a user-session or service bearer token. The data plane authorizes command connections with session-scoped attachment tokens. Management and attachment tokens are not interchangeable.

A data-plane replica that receives SIGTERM drains rather than dropping its streams. The listener closes, so new connections, readiness probes included, are refused, and idle keep-alive connections close; /readyz reports 503 only to a probe already in flight on an open connection. Behind a load balancer, use a preStop delay so it stops routing to the replica before the listener closes. A stream request whose headers had already arrived when the drain started gets 503 with Retry-After (plus Connection: close on HTTP/1); that includes a streamed ECW Process.StreamInput. A native WebSocket that was accepted but hadn't attached yet is closed with code 1012 (service restart). Streams that were already attached keep running, and the replica exits once none remain and in-flight requests finish. With a drain deadline set (--shutdown-grace-secs), streams still open when it passes are cancelled; without one, they run until they end or the pod's termination grace period kills the process. A second stop signal ends the drain the way the deadline does, but at once: a Process.Start still opening is cut the way SIGKILL would cut it. At the deadline: native event streams end, and a client that reconnects with Last-Event-ID resumes a process's output after the last event it received while the guest still retains it, and a file watch's live stream; native WebSockets close with 1012, including ones still opening their backend stream, and the client re-attaches the same session (a process or terminal session doesn't start its process again); ECW connect streams end with unavailable and need a new client request, a process stream after first delivering the output the backend has already produced; a Process.Start that hasn't reported itself gets up to 2 seconds to, and otherwise ends with unknown, since the command may be running and a retried start would run it twice; a streamed Process.StreamInput gets a 409, since some of its input may already have been delivered; other stream requests still opening their backend stream, and proxied sandbox-service requests still preparing their route, get the draining 503; proxied sandbox-service responses and other upgraded connections are cut. Other responses still in flight, such as a large file download, end when the process exits, at most about 5 seconds after the deadline, unless they finish first. Keep the deadline plus those 5 seconds within the pod's termination grace period minus its preStop delay; the telemetry flush on exit gets whatever time remains. The service rejects a deadline under 8 seconds, the stream-open budget.

Continue with Your first sandbox for API examples and recovery steps.

Internal TCP transport configuration and coordinated rollout requirements are documented in Internal RPC TLS.

Sandbox artifact transfer administration

Node-api exposes no sandbox or snapshot transfer RPCs to remote callers. Template transfer remains available through its existing API.

Sandbox and snapshot export/import remain available through the sandbox-manager Unix socket to root or an account with the local operator role. No transfer command is shipped in the sbx CLI. Performing a transfer requires custom code using the sandbox-manager client library: ExportSandbox / ExportSnapshot on the source and ImportSandbox / ImportSnapshot on the destination. That code must supply the destination RPC's existing import header and file chunks. Source lifecycle, encryption restrictions, and destination validation still apply. The node-api service account cannot invoke these local artifact methods.

TLS interception exemptions

NetworkPolicyV1.tls_interception_exemptions uses the same destination selectors as egress rules. After normal destination admission, the gateway relays matching SNI hostnames or destination IPs without terminating TLS, even when tls_interception is on. A new connection using a hostname exemption requires a live observed DNS binding from that hostname to the destination IP, including under default-allow or independent IP allowances. Without that binding, normal interception applies. IP exemptions also support TLS without SNI and apply regardless of the hostname served at that IP. Domain-only egress allowances still require the guest's DNS attribution and matching SNI. Explicit destination denies and platform guardrails take precedence. Exemptions grant no egress permission and bypass HTTP inspection and credential injection. Other destinations retain the policy's interception mode.

The API composes operator egress profiles into the tenant's policy on authenticated federated creates. A profile independently supplies allowances and TLS exemptions. DNS resolver allowances are not exempted. Selected names are persisted separately from the effective policy. Fresh creates, policy replacement, and explicit template inheritance resolve current definitions. Forks retain the selected names and saved effective policy; resume and restart do not refresh profile destinations. An established exempt flow retains its verified hostname binding for its lifetime, so DNS expiry or cache eviction does not disconnect it on an unrelated policy update. Each update still rechecks current egress permissions and exemptions; removing an exemption closes the flow when the updated policy requires interception. New connections require fresh DNS evidence.

Before configuring nonempty profile exemptions, upgrade every policy reader and running gateway to support TLS exemptions. Native sandbox and template policy writers must observe the same reader compatibility requirement. Upgrade queued operation executors before APIs begin admitting profile selections.

Before downgrading a reader, remove exemptions from stored policies and operator profile selections using compatible binaries, or retire the affected resources. Disabling configuration does not strip an existing sandbox's profile selection. Removing an exemption restores interception, so pinned clients need an appropriate operational fallback before rollback.