Spacelift in production: five failure patterns

Pattern 01 Terraform state lock held by a crashed Spacelift worker blocking all subsequent stack runs indefinitely because Spacelift provides no automatic lock-break or orphaned-run detection

When Spacelift dispatches a run to a worker, the worker acquires the Terraform state lock on the configured backend (S3+DynamoDB, GCS, Terraform Cloud, etc.) at the start of the plan phase. If the worker process crashes mid-run — due to an OOMKill, a spot instance preemption, a Docker daemon restart, or a node eviction — the state lock is never released. Terraform's locking mechanism is not aware that the worker is gone; it only sees an active lock entry with a LockID and a creation timestamp.

All subsequent runs on the same stack that attempt to acquire the lock receive Error acquiring the state lock and fail at the plan phase. Spacelift marks these runs as failed, and if the stack has notifications configured, every failed run pages the team. Spacelift's run UI shows the failed runs with the Terraform error message, but there is no visual indicator distinguishing a legitimate lock conflict from an orphaned lock.

Engineers often assume a concurrent run is still in progress and wait, lengthening the time the stack is blocked. In high-velocity environments where stacks are triggered frequently via stack dependencies or webhooks, dozens of failed runs can accumulate before anyone investigates the root cause. Diagnose by inspecting the DynamoDB lock table (for S3 backends): aws dynamodb scan --table-name <terraform-lock-table> --filter-expression "attribute_exists(LockID)".

Note the LockID and Info fields — the Info JSON contains the Operation, Who, and Created timestamp. If Created is older than your longest expected run duration, the lock is likely orphaned. For GCS backends: gsutil ls gs://<bucket>/<path>.tflock.

In the Spacelift UI, check the run history for the stack and look for a run that entered Applying or Planning state and never transitioned out. Fix the immediate outage by forcing lock release: for DynamoDB, aws dynamodb delete-item --table-name <terraform-lock-table> --key '{"LockID":{"S":"<lock-id>"}}'. For GCS, gsutil rm gs://<bucket>/<path>.tflock.

To prevent recurrence, configure your Spacelift worker pool with preemption-safe run handling by enabling runCleanupOnWorkerRestart: true in the worker pool launcher configuration. Additionally, add a DynamoDB lock age alarm in CloudWatch: alert when any lock item has a Created timestamp older than 30 minutes, which catches orphaned locks automatically. For spot-based worker pools, implement a Lambda function that scans the lock table and posts to a Slack channel when stale locks are detected.

Pattern 02 Stack dependencies failing to trigger downstream stacks when the upstream run produces no Terraform plan diff, even when the upstream stack's outputs have changed due to provider-side drift

Spacelift's stack dependency feature triggers downstream stacks when an upstream stack completes a tracked run. However, by default, Spacelift only propagates the trigger if the upstream run results in changes being applied — that is, if the Terraform plan is non-empty. If the upstream run produces a clean plan (no diff), Spacelift considers the upstream stack unchanged and does not enqueue the downstream stacks.

This is correct behaviour in most cases, but it breaks down when upstream outputs change due to reasons that Terraform does not detect as a diff: provider-side drift where the cloud resource was modified outside Terraform, a Terraform provider version upgrade that changes how outputs are computed, or a situation where the upstream stack's outputs block references a data source that refreshed with new values but the managed resources are unchanged. Downstream stacks that depend on those outputs — for example, a networking stack whose VPC CIDR is consumed by an application stack — then run with stale output values. The application stack runs successfully because it passes its own plan, but it's using the previous VPC CIDR.

The misconfiguration is not detected until a downstream resource fails to connect to the network or a security group rule rejects traffic. Diagnose by checking the trigger history in the Spacelift UI: navigate to the downstream stack, click on Runs, and look for a gap in the run timeline that corresponds to when the upstream stack ran with no diff. Compare the upstream stack's output values from two consecutive runs using spacectl stack output list --id <upstream-stack-id> before and after the no-diff run.

If the output values differ despite the no-diff result, you have encountered this pattern. Fix by configuring the upstream stack to always trigger downstream stacks regardless of diff status. In the Spacelift UI, edit the stack dependency and enable triggerAlways: true, or via Terraform using the Spacelift provider: resource "spacelift_stack_dependency" set trigger_always = true`.

For stacks where outputs are driven by data sources, add a terraform_data resource in the upstream stack that explicitly tracks the data source's output value as a trigger, forcing Terraform to detect the change as a resource update. Alternatively, schedule downstream stacks to run on a time-based trigger in addition to the dependency trigger, providing a backstop that catches stale output propagation within a bounded time window.

Pattern 03 Private worker pool workers losing connectivity to the Spacelift control plane causing runs to hang in Queued state until the worker health-check timeout expires, blocking the entire pool

Spacelift private worker pools communicate with the Spacelift control plane over an outbound HTTPS WebSocket connection. Workers poll for jobs and receive run assignments via this channel. When network connectivity between the worker and the Spacelift control plane is interrupted — a firewall rule change, a NAT gateway failure, a VPC endpoint misconfiguration, or an expired TLS certificate on the egress proxy — the worker's connection drops silently.

The worker process detects the disconnection after the TCP keepalive timeout (typically 60 to 120 seconds) and attempts to reconnect. During this window, any runs assigned to that worker are stuck in Queued state in the Spacelift UI. Spacelift does not immediately reassign queued runs to other workers because it does not distinguish between a worker that is actively processing a run and one that has lost connectivity but not yet been marked unhealthy.

If the worker pool has only one worker (common in smaller self-hosted setups), all runs queue indefinitely. If there are multiple workers, runs may eventually be picked up by healthy workers, but the Spacelift UI shows misleading Queued statuses that make it appear the pool is healthy when one or more workers are dark. Diagnose by SSHing into the worker instance and checking the launcher process logs: journalctl -u spacelift-worker --since="30 minutes ago" | grep -i "connect\|websocket\|disconnect\|error".

Also verify outbound connectivity from the worker: curl -v https://spacelift.io/api/health (or your self-hosted control plane URL). Check your VPC flow logs or firewall logs for dropped packets to the Spacelift control plane IP range. In the Spacelift UI, navigate to Worker Pools and inspect the last-seen timestamp for each worker — a stale timestamp indicates the worker is not actively connected.

Fix the connectivity issue by resolving the underlying network problem: update security group egress rules to permit HTTPS (443) to *.spacelift.io, ensure NAT gateway has an available Elastic IP, and verify that any egress proxy certificates are valid. For faster recovery, configure the Spacelift worker launcher with reconnectInterval: 5s and maxReconnectAttempts: 0 (unlimited) in /etc/spacelift/launcher.yaml. Add a CloudWatch alarm on the worker's custom metric for WebSocket connection state, or use Spacelift's Prometheus-compatible metrics endpoint (GET /metrics on the launcher) to monitor spacelift_worker_connected and alert when it drops to 0.

Pattern 04 Drift detection runs failing silently when cloud provider credentials expire between scheduled checks, with Spacelift marking the run as skipped rather than failed, hiding the credential outage

Spacelift's drift detection feature schedules periodic runs that execute terraform plan and compare the result against the last known state. These runs authenticate to cloud providers using credentials injected via Spacelift's cloud integrations (AWS OIDC, GCP Workload Identity, Azure federated credentials, or static API keys stored as environment variables). When the credential mechanism fails — an OIDC role trust policy is updated, a Workload Identity binding is deleted, a static secret key is rotated without updating Spacelift's environment variable — the drift detection run fails during provider initialisation.

Terraform exits with a non-zero code before producing any plan output. Spacelift interprets a pre-plan exit as an initialisation failure and marks the run as Skipped rather than Failed in some versions, or records it as failed but does not trigger stack failure notifications if the stack's notification policy only alerts on apply failures. The consequence is that real infrastructure drift accumulates undetected.

The drift detection schedule continues firing, each run fails silently, and the drift dashboard shows the last successful drift check timestamp advancing to days or weeks ago without any alert being raised. Engineers discover the gap only during an incident post-mortem when they check whether drift was monitored and find months of skipped runs. Diagnose by filtering drift detection runs in the Spacelift UI: navigate to the stack, click Runs, and filter by Type: DRIFT_DETECTION.

Look for runs with status Skipped, Failed, or Discarded. Click into a failed run and expand the Initialize phase logs, which will contain the provider authentication error. For AWS OIDC, the error typically reads Error: No valid credential sources found or failed to retrieve credentials from oidc.

Cross-check with AWS CloudTrail: aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRoleWithWebIdentity and look for AccessDenied events from Spacelift's OIDC issuer. Fix by configuring Spacelift notification policies to alert on drift detection failures regardless of failure phase: add a policy rule that triggers on run.state == "FAILED" AND run.type == "DRIFT_DETECTION". For AWS OIDC, set up a CloudWatch alarm on AssumeRoleWithWebIdentity access denied events from the Spacelift role.

Rotate credentials before expiry by adding a reminder task tied to the credential expiry date. For static credentials, migrate to OIDC-based authentication which uses short-lived tokens that cannot expire in the same way — configure aws_integration in the Spacelift provider with generate_credentials_in_worker: true.

Pattern 05 Environment variable secret injection order causing Terraform provider initialisation failures when dynamic secret references are resolved before the secret backend authentication context is available

Spacelift supports dynamic secrets through integrations with Vault, AWS Secrets Manager, and similar backends. When a stack is configured with both a cloud integration (for backend authentication) and dynamic secret references (for Terraform provider credentials pulled from Vault), Spacelift injects environment variables in a defined order during run initialisation. If a dynamic secret reference such as ${{ secrets.vault.AWS_ACCESS_KEY_ID }} is evaluated before the Vault token has been established — which can happen when the Vault integration depends on the same AWS role that the cloud integration is still in the process of assuming — Terraform receives an empty or malformed provider credential.

The terraform init step succeeds (it doesn't validate provider credentials), but terraform plan fails immediately with Error: No valid credential sources found for AWS provider even though the AWS integration is correctly configured. This ordering issue is non-deterministic: it depends on the initialisation sequence of Spacelift's run preamble, which can vary based on network latency to the secret backend and whether the cloud integration uses cached OIDC tokens. The run will succeed on retry most of the time, leading teams to classify it as an infrastructure flake and ignore it, when in reality it reflects a race condition in the initialisation sequence.

Diagnose by comparing two consecutive runs of the same stack where one succeeds and one fails. Expand the Initialize phase in both runs and look at the timestamp delta between the cloud integration credential injection log line and the Vault secret resolution log line. If the failing run shows Vault resolution completing before the AWS credential injection, you have confirmed the race.

Also check whether the issue is correlated with cold starts (new worker, no cached token) versus warm starts. Fix by restructuring the secret injection dependency so Vault authentication explicitly depends on the cloud credential being ready. In the Spacelift Terraform provider, use a before_init hook script that explicitly waits for the AWS credential to be available before the run proceeds: add a before_init script in stack.runner_image or the stack's hook configuration that runs aws sts get-caller-identity in a retry loop until it succeeds, ensuring the credential is usable before Vault attempts to authenticate with it.

Alternatively, restructure the Vault authentication to use a static Vault token stored directly in Spacelift's secret store rather than relying on AWS IAM for Vault auth, eliminating the circular dependency. For stacks where this is not possible, set runner_image to a custom image that includes a startup script encoding the correct initialisation order.