SPACE: Troubleshooting

First identify which stage failed: local setup, API authentication, a sandbox operation, or a command stream. An accepted request, a running sandbox, and a successful command are three different outcomes.

Local installation

This section covers the Kind installation. Use the explicit kind-space-local context so diagnostics target that local deployment, not another cluster. If you selected another cluster name, replace the context accordingly.

kubectl --context kind-space-local -n space get pods
kubectl --context kind-space-local -n space get events \
  --sort-by=.metadata.creationTimestamp
kubectl --context kind-space-local -n space logs job/space-seed-publish
kubectl --context kind-space-local -n space \
  logs job/space-db-migrate -c migrate
tilt logs provision-local-console --tail 50

For an individual pod, use kubectl --context kind-space-local -n space describe pod <pod-name> and kubectl --context kind-space-local -n space logs <pod-name> -c <container-name>. Use --previous to read the previous instance's logs after a container restart.

Sandbox pods (space-sbx-…) run in their own namespace, sandboxes, not in space: list them and their events with kubectl --context kind-space-local -n sandboxes get pods,events. NetworkPolicies there admit only the API and cluster-controller on port 9800; kubectl --context kind-space-local -n sandboxes get networkpolicies shows them.

For an empty reply, inspect the Tilt resource that holds the forward (space-ui for port 3000, space-object-store for port 8333) for port-forward errors, and wait for its pod to be ready again before retrying:

for resource in space-ui space-object-store; do
  printf '\n== %s ==\n' "$resource"
  tilt logs "$resource" 2>&1 | grep -i 'port.forward' | tail -n 20
done

An empty reply does not establish whether a mutation reached the server. Inspect its outcome and retain the original request body and idempotency key before retrying.

If a port forward is absent, start the one you need in a separate terminal and leave it running. Each command below runs in the foreground, and requires its local port to be free.

kubectl --context kind-space-local -n space \
  port-forward --address=127.0.0.1 svc/space-ui 3000:3000

kubectl --context kind-space-local -n space \
  port-forward --address=127.0.0.1 svc/object-store 8333:8333

Sandbox and command failures

Use the Python example's read-only inspect action from Your first sandbox, with RUN_DIR set to the exact saved run directory. Keep the same exported endpoint settings and a current SPACE_API_TOKEN.

python3 examples/first_sandbox.py inspect "${RUN_DIR:?Set the saved run directory}"

This prints sandbox listings and the saved creation/deletion operations, sandbox, and running-process list. Operations report lifecycle progress or failure; the sandbox reports its current state and conditions. If a request failed before returning IDs, inspect its HTTP error and the saved request in state.json, rather than constructing an empty-ID URL.

Inspect a running process with GET /v1/sandboxes/{sandbox_id}/processes/{process_id}. An empty process list is expected after a command exits; use the saved event transcript to check its result. Keep a failed transcript for diagnosis, but review it for secrets before sharing it.

Timeouts mean different things

A client deadline limits how long the client waits; it does not cancel a lifecycle operation or establish that a process stopped. Inspect the resource before retrying or issuing another command.

The Python example uses Linux signals to bound whole requests and stream reads. Run it as an executable on the documented Linux host, not in a background Python thread.

Retry without duplicating work

For endpoints that declare a required Idempotency-Key, keep the original request body and key until the mutation's outcome is known. The Python example saves them in state.json. For a retry of the same action, resend the same body with the same key; changing the body under the same key causes an idempotency_conflict.

Process kill and signal do not deduplicate requests with Idempotency-Key. Do not blindly retry an ambiguous outcome; inspect the process and its event stream first, because a repeat can apply the action again.

Check a returned operation ID before resending a lifecycle mutation. A new key represents a new request and can create another sandbox or command; do not use a new key merely because a connection timed out.

Read the API error's code, message, details, request_id, and retryable fields. Correct non-retryable errors first, and use bounded backoff for retryable errors without discarding the original key.

A file route's 404 (not_found) whose details include {"key": "resource", "value": "path"} means the sandbox is yours and the path inside it does not exist, so there is no need to look the sandbox up again. Without that detail, the sandbox itself was not found or is not yours; the two answer alike.

rate_limited is not the only 429. A POST /v1/sandboxes that would exceed the organization's or the user's sandbox limit gets 429 quota_exceeded with retryable: false and no Retry-After: resending it unchanged fails the same way until a sandbox is deleted or an in-flight create ends. See SPACE_NATIVE_ORG_SANDBOX_LIMIT in docs/configuration.md.

A command create's 400 (invalid_request) whose details include {"key": "field", "value": "pty"} means it asked for a pty with Accept: text/event-stream. Server-sent events cannot drive a terminal, so drop the pty, or create the command without that header and attach over WebSocket.

A sandbox that is at capacity, or shutting down, can refuse a request before doing any of its work: a retryable 429 (rate_limited) or 503 with Retry-After. That request did nothing, though an earlier attempt of the same action may have; wait at least that long and resend it with the same body and key, or reattach to the same session. SPACE does not retry these refusals for you. A timeout leaves the outcome unknown, and it is retryable only where the endpoint deduplicates by its key. Lifecycle operations and command creation deduplicate for 24 hours; file writes, batch writes, and renames only for 60 seconds after they complete, and not across a restart of the sandbox's agent, so retry them promptly or check the file first. File mkdir, remove, and change tracking do not deduplicate: their timeouts are not retryable, so check the sandbox's state before repeating them.

When reporting a problem, include the source revision, failing step, request ID, relevant sandbox/operation/process/session IDs, and sanitized error or event output. Exclude bearer tokens, passwords, private keys, and sensitive command environment values.