Configuration reference

Every SPACE_* environment variable listed here is a supported operator setting. Variables that are not listed are not read by release binaries. "Unset" means the built-in default applies. Variables marked required make the service fail at startup when missing.

Settings that exist only so tests can shorten a timer or flip a scenario are test hooks. They are read only by binaries built with the test-hooks Cargo feature (pplx-space-common-core/test-hooks), which the local kind image enables and release images never do. They are intentionally undocumented, and scripts/source_guards/env_test_hook_boundary.py fails CI if code names a variable listed here in a test-hook read, directly or through a string constant from the same crate.

Secrets are passed as files: set the *_PATH variable to a mounted file. The inline forms (SPACE_ECW_CREDENTIAL_KEY, SPACE_ECW_CREDENTIAL_PREVIOUS_KEY, SPACE_API_ROUTE_ED25519_SIGNING_KEY, SPACE_API_ROUTE_ED25519_VERIFICATION_BUNDLE) and SPACE_CLUSTER_NODE_API_ENDPOINT are deprecated and still read for one release so directly provisioned hosts can upgrade in place.

Reconcilers and kill switches

Each mutating loop has one *_ENABLED switch. Loops that are on by default treat any value other than a truthy one as off, so a typo stops the loop instead of leaving it running. The one exception is SPACE_PUBLIC_OPERATION_DISPATCHER_ENABLED, which keeps the lenient parser because disabling it by mistake would leave every create queued forever.

Sandbox placement and nodes

API, admission and limits

Sizing dispatcher execution concurrency

SPACE_PUBLIC_OPERATION_DISPATCHER_EXECUTION_CONCURRENCY sets how many operations the leading cluster-controller replica runs at once. Raising it is a configuration change, but five limits bound what it buys, and the first two have to change with it:

1. The controller's Postgres pool. Executions share the controller's space_user pool with most reconcilers. POSTGRES_POOL_MAX_CONNECTIONS (default 25) sizes it, and the controller's space_creds pool too; a query waits up to POSTGRES_POOL_ACQUIRE_TIMEOUT_SECS (default 5) for a free connection. An execution holds a connection only while one of its statements or transactions runs, but every execution renews its claim every 20 s and writes its result when it ends, so the pool has to hold the executions' statements on top of what the reconcilers already use. Raise the pool with the concurrency, and confirm the headroom with the pool metrics below rather than by a fixed ratio.
2. Postgres max_connections. Every controller process opens both pools and one more connection for the node-drain listener, and the standby replica and a rolling-update surge pod can open as many. With a pool size of P, the controller can hold (2 × P + 1) × (replicas + surge pods) server connections: 153 at the default of 25 with two replicas and one surge pod, 387 at 64. The sum over every service has to stay below max_connections, less superuser_reserved_connections and the sessions operators open, so that a service runs out of its own pool before the server refuses it a connection. The Kind budget in infra/space/kind/charts/space-postgres/values.yaml is a worked example: it holds 1,328 of 1,400 connections, so a controller pool of 64 there (234 more) needs max_connections raised as well.
3. Claim cost. Each claim reads the whole due backlog under one cluster-wide advisory lock, and a tick that finds the lock held claims nothing, so claim cost grows with the backlog, not with the slot count. Freed slots refill on the next 25 ms tick, at most 64 per tick.
4. Node activation admission. Each sandbox-manager runs at most SPACE_SBXMGR_ACTIVATION_ADMISSION_MAX_CONCURRENT (5) activations at once, starting at most 5 a second after a burst of 10, and holds the rest up to 30 s before refusing them. Slots beyond that many per worker node wait in the sandbox-manager rather than doing work.
5. Cluster activation admission. Every activation in the cluster draws on one Redis token bucket, 60 a second by default (SPACE_CLUSTER_ACTIVATION_ADMISSION_REFILL_RPS). An activation it refuses is requeued without spending a retry attempt.

The slot reservations do not grow with the concurrency. Teardowns keep their reservation (SPACE_PUBLIC_OPERATION_DISPATCHER_TEARDOWN_RESERVED_SLOTS, 2), and an organization alone leaves its headroom free (SPACE_PUBLIC_OPERATION_DISPATCHER_ORG_SLOT_HEADROOM, 2), so one organization on an idle dispatcher gets the concurrency less four: 12 of the default 16. While another organization is waiting, each one is held to SPACE_PUBLIC_OPERATION_DISPATCHER_ORG_SLOT_LIMIT (8) whatever the concurrency, so raise that limit too if several organizations should share a larger dispatcher.

What pool starvation looks like. A claim renewal or result write that cannot get a connection within the acquire timeout fails, and the dispatcher abandons the execution: it logs public-operation-dispatcher execution failed with the pool timeout, records nothing, and drops the execution at its next await point, though work a node already started keeps going. The operation reads running until its claim goes stale, 60 s after its last renewal. The dispatcher then reclaims it before anything else on its sandbox starts and runs it again, repeating any work it had already done; the operation's events show a second claimed event. Callers see no error, only an operation that takes about a minute longer. operations.stale_claim counts a related case, an execution that finished after its claim went stale and found it already taken; it should stay at 0.

Metrics to watch while changing the concurrency or the pool:

Templates and builds

Prefetch requires the catalog to confirm generic-only release resolution from the same database snapshot as its storage locations. Cached visibility alone does not authorize a download. Upgrade the catalog before template-manager; an older catalog cannot acknowledge this restriction, so prefetch pauses until it is upgraded. On-demand pulls retain their existing authorization.

Direct/systemd deployments can publish API-submitted template builds without Kubernetes projected credentials. Generate a separate random publisher secret (for example, 32 random bytes encoded as hex), install it in a protected file readable only by the appropriate service identity, and set SPACE_TEMPLATE_PUBLICATION_TOKEN_PATH on ClusterApi and build-manager only. The file must contain 32–4096 visible ASCII bytes; surrounding whitespace is trimmed. Both processes still require their normal shared node bearer and internal RPC TLS configuration. Other node services must not receive the publisher secret.

Build-manager attaches the dedicated proof only to template and release registration. The shared node bearer alone cannot publish. Both tenant and system publication require the corresponding per-build grant issued by the API. A tenant grant must match the template, organization, and registered builder at the connection’s source address; no Kubernetes identity is required. The publisher credential cannot issue system grants itself, and does not enable ungranted system builds from the operator CLI. Kubernetes publication continues using projected tokens; combining Kubernetes auth with the direct receiver credential fails startup.

Upgrade and configure ClusterApi first, then build-manager, before submitting builds. Unconfigured receivers retain their existing rejection of static publication; a sender upgrade alone cannot enable publication. Missing or malformed configured files fail closed. The receiver loads its credential at startup; the sender reads it on each registration attempt. To rotate it, pause build submissions and drain publishing jobs, replace both files atomically, restart ClusterApi, and resume. Removing the setting from both services disables direct publication without changing previously published templates. Failed builds must be resubmitted after configuration is corrected.

For direct space-buildmgr build calls without --org-id, configure SPACE_INTERNAL_API_ENDPOINT and SPACE_CLUSTER_API_TOKEN_PATH in the CLI's process. The projected token must identify an account in the system publication list only. The CLI obtains a grant before submitting the build and sends it with the job; build-manager's dual-authority credential cannot issue grants. Configure SPACE_BUILD_MANAGER_TOKEN_PATH separately for permission to submit builds. Direct operator builds may still use custom system template names. API system builds remain limited to seed and code-runtime at the node gateway.

The grant follows the persisted build job through template and release registration, expires after seven days, and is consumed only when the release transaction commits. A failed API dispatch revokes only its own grant. Grants left by failed direct submissions expire and are swept when a new grant is issued. With no ClusterApi endpoint configured, the CLI retains local-only build behavior; cluster publication requires the operator configuration above.

Pause system build submissions and drain older system jobs before rolling out ClusterApi's grant enforcement. Apply the grant-table migration and role bootstrap first, upgrade build-manager and the node-api gateway, then upgrade the API and ClusterApi before resuming submissions. The gateway must understand the new field to forward each build's grant. Resubmit any older queued system jobs that could not be drained: jobs without a grant cannot publish through a dual-authority account after the rollout. System-only deployment publishers retain their existing publication authority.

Tenant builds follow the same pattern. Before dispatching an organization's build, the API records a tenant publication grant naming the target template and that organization, and sends it with StartBuild; it withdraws the grant if dispatch fails and pins it to the accepting builder node from the response. Build-manager forwards the grant to template and release registration. ClusterApi accepts a tenant-list account's publication only with a grant that names the requested template and organization and that covers the builder node identified by freshly reviewed pod-bound node name and UID, matched against live Kubernetes builder inventory. A builder whose registration has no inventory UID yet cannot publish; build-manager retries until inventory synchronization completes, within its bounded registration backoff. Updated build-managers also retry publication failures explicitly marked as uncommitted database contention; transport failures with an unknown commit outcome remain terminal. Both build-manager and ClusterApi must be upgraded for these retries. Standalone trusted deployments retain transport-address identity. A grant covers any registered builder for one hour while unpinned, then only the pinned builder for seven days. Release registration consumes the grant in its transaction. Direct space-buildmgr build --org-id submissions carry no grant, so their builds cannot publish to a cluster through a tenant-list account; submit organization builds through the API.

Deploy tenant grant enforcement only after the preparation release has fully rolled out to every API, ClusterApi, node-api gateway, and build-manager. That release produces and forwards grants while still accepting grantless tenant publication; the grant-table migration and role bootstrap must already be applied. Do not skip directly from an older release to enforcement.

Drain or resubmit every tenant job accepted before preparation finished; an older gateway or builder may have dropped its grant. Verify a newly submitted tenant build retains the grant through dispatch and publication, and that TokenReview returns the pod-bound node identity matched by builder inventory. Direct tenant CLI submissions must move to the API before enforcement.

Once these prerequisites hold, enforcement may roll out through the normal service order: existing preparation builders already carry grants and understand safe retry responses. Roll application components back to the preparation release if needed, retaining the additive schema. Remove enforcement from all ClusterApi replicas before rolling grant producers or forwarders back further.

Template balloon device

A template can opt its sandboxes into a Firecracker balloon device through balloon in its recipe's [metadata], which the build records in the template's metadata.json:

[metadata]
memory_mib = 8192
# One of:
balloon = { mode = "inflate", amount_mib = 1024, deflate_on_oom = true }
balloon = { mode = "report" }  # free-page reporting
balloon = { mode = "hint" }    # free-page hinting

stats_polling_interval_s (default 0, off) enables balloon statistics in any mode. amount_mib and deflate_on_oom are required for inflate and rejected otherwise; amount_mib must be below the sandbox's memory. Unknown modes and fields are rejected. Without balloon, the VM configuration is unchanged.

The device is rendered only when a sandbox cold-boots from the template build, which includes creating the template's base snapshot. Sandboxes restored from a snapshot keep the device that snapshot captured, so changing the setting takes effect only after rebuilding the template and its base snapshot. The REST build API does not accept the setting yet.

Hint mode also needs balloon_hint_timeout_ms (at most 30000) in the node's /space/etc/sandbox.toml. Before each suspend snapshot, a VM whose balloon has free-page hinting then runs one hinting pass bounded by this time; timeouts and errors are logged and the snapshot proceeds. Unset, suspend makes no balloon calls.

Object storage and snapshots

Authentication, TLS and credentials

External console sign-in

By default the console signs in with a SPACE username and password. A deployment that puts the console behind an identity-aware proxy can instead install an ExternalConsoleAuthenticator through ApiExtensions::with_external_console_authenticator and start the API with pplx_space_api::run_with. The stock binaries install none, so these variables have no effect on them. The authenticator verifies the proxy's signed assertion and applies the deployment's admission policy; SPACE holds no provider-specific policy and never reads an unsigned identity header. Install the same authenticator on every API surface the console calls, including the admin API.

With an authenticator installed:

• GET /auth/v1/config returns {"mode":"external","provider_name":…,"logout_url":…} (otherwise {"mode":"password"}), unauthenticated and no-store, so the console can choose its sign-in flow. A console that gets 404 here assumes password mode.
• A console request is one that carries X-Space-Console-Request: 1 exactly once; the console sends it on every request in this mode. Console requests authenticate only with the assertion: none is a 401, and a rejected one is a final 401, 403 or 503. A bearer token sent alongside is never used: console routes ignore it and the native API refuses the combination. State-changing methods also need a trusted Origin.
• The verified email maps to an existing local user by exact match. Nothing is provisioned and no role is granted: the user's own role and organization authorize the request, so a member still cannot reach admin-only routes. Federated owners and legacy disabled service accounts are not eligible.
• Password login (POST /auth/v1/sessions) is refused with 403 and existing password session tokens no longer authenticate, so an old session cannot bypass the mode. DELETE /auth/v1/session revokes nothing; the console ends the provider session through the logout URL instead.
• Requests without the console header are machine requests. The native API authenticates them with API keys exactly as before and never consults an assertion the proxy attached; console-gated routes refuse them. Session tokens are not accepted as native API credentials in this mode.
• Each console decision increments auth.external_console.decision with result accepted, missing_assertion, unauthenticated, forbidden, unavailable, csrf_rejected, no_user or lookup_failed.
• The authenticator runs inline on every console request with no timeout applied by SPACE; it must bound its own latency.
• Cross-origin requests never carry credentials: the API's CORS policy sets no Access-Control-Allow-Credentials. The origin and header checks above rely on that; do not relax it for the API host.

Removing the authenticator and restarting restores password sign-in; password sessions that have not expired become valid again.

Template mutation authority

On worker and builder node-api, DeleteTemplate and DeleteTemplateBuild require the trusted policy service-account capability. Template-manager's local roles and deletion dependency guards still apply.

With Kubernetes authentication disabled, the shared internal token remains the operator trust boundary and retains mutation authority. When Kubernetes authentication is enabled, static fallback has no mutation authority. Missing authentication context is denied.

Template transfer authority

SPACE_NODE_API_AUTH_TEMPLATE_TRANSFER_SERVICE_ACCOUNTS grants only artifact transfer, independently of mutation. Configure the controller and node accounts that perform transfers before upgrading. This is a cluster infrastructure role: it permits exporting cached private releases from any organization as well as system templates. Do not grant it to tenant-facing or general admitted services. Node-api replaces forwarded authority metadata, and template-manager requires that grant from its authenticated node-api Unix-socket peer. Upgrade node-api before template-manager; an old forwarder is denied by the new manager.

With Kubernetes authentication disabled, the shared internal token remains the operator trust boundary and retains transfer authority. When Kubernetes authentication is enabled, static fallback has no transfer authority. Missing authentication context is denied. Local root/operator exports remain available through template-manager's Unix socket.

A peer transfer has two callers, and each needs the grant and admission by SPACE_NODE_API_AUTH_ALLOWED_SERVICE_ACCOUNTS on the node it calls. The caller of PullTemplate on the target node starts it: the admin API's node template-pull route, which only the direct substrate serves, or an operator calling node-api directly. The target's node-api then calls NodeExportTemplate on the source node with its own SPACE_NODE_API_TOKEN_PATH token. List the account that starts pulls and the node-api account of every role that can be a target. An empty SPACE_NODE_API_AUTH_TEMPLATE_TRANSFER_SERVICE_ACCOUNTS disables peer transfers and node-api logs a startup warning. A deployment without peer transfers may leave it empty. Also list each node-api account in SPACE_NODE_API_AUTH_PEER_NODE_SERVICE_ACCOUNTS, which limits that account to the export call and GetNodeInfo.

Exact-digest export can use the local cache offline. An omitted digest resolves the requested catalog tag (default latest) and exports only that digest; a missing tag, unavailable catalog, or uncached resolved digest fails without falling back to a newer local build.

Policy authorization

Before enabling Kubernetes authentication or upgrading an authenticated worker or builder, set SPACE_NODE_API_AUTH_TRUSTED_POLICY_SERVICE_ACCOUNTS to the service accounts that own policy intent. It must be nonempty on both roles or node-api refuses to start. These accounts must also be admitted by the normal and internal surface allowlists for the RPCs they call. Policy trust is independent of the SpacedWaitReady trust set. Policy-bearing pool adoption uses the internal surface, so its caller needs an internal-audience token.

Shared static-token fallback cannot mutate policy when Kubernetes authentication is enabled; static-only deployments retain that authority. Policy-bearing cold creation and internal pool adoption require a trusted caller. Cold creation and pool adoption also require policy authority for a nonzero sandbox-policy generation, even without an override. Requests with no overrides and generation zero remain available to otherwise admitted callers and inherit the template's policy. Guest-bridge accounts already have separate method restrictions.

Local policy is node-administrator-owned, including normal standalone use and operator overrides on cluster-managed nodes. It requires a root or Operator Unix socket peer. Forwarders such as node-api and node-ctl must run under their dedicated non-root identities: UID 0 is classified as Root before the service-account lookup.

Forwarded imports carrying Local network or auto-lifecycle policy are rejected, and retrying does not help. To retain Local policy, an operator must review its ownership and perform the import through an authorized local peer; forwarding never relabels it.

Database

Service database pools always emit SQLx sqlx::query events at INFO, which become DB spans under traced requests. To turn them off, add a sqlx::query=off directive to RUST_LOG or SPACE_TRACE_FILTER.

Execution eligibility

The two node settings replace the former SPACE_ENV admission rule. Before upgrading a node that runs non-E0 work, explicitly allow every environment that must remain usable. Leaving settings absent revokes those bindings even on development or Kind nodes. See execution policy and rollout for cohort constraints and the separate permission for historical unbound work.

Observability and feature flags

Identity precedence and the full telemetry contract are in observability.