RisingWave in production: five failure patterns

Pattern 01 Materialized view lag spiking during barrier synchronization when a slow compute node holds the global barrier epoch

RisingWave's exactly-once processing guarantee is built on a barrier protocol modelled after Apache Flink's Chandy-Lamport checkpoint barriers. The Meta node periodically injects barrier messages into every source actor's input stream. Each barrier carries an epoch number and propagates downstream through the streaming actor graph — source actors, filter actors, join actors, aggregation actors — until it reaches sink actors. Every compute node must acknowledge its local actors' receipt of the barrier before the Meta node can advance to the next epoch. Because the barrier epoch is global and all actors across all compute nodes must participate, a single slow compute node — one that is CPU-bound from a hot aggregation, experiencing a JVM-style memory allocation pause (RisingWave is written in Rust but the OS memory allocator can still stall under pressure), or briefly network-partitioned from the Meta node — causes the global barrier epoch to stall. During the stall, no actor on any compute node can commit new output. Materialized view queries served by the Frontend node continue to return results, but the results freeze at the last committed epoch. From a user's perspective, a dashboard fed by a materialized view stops updating and query latency measured by the frontend appears normal while the data is hours stale.

Diagnose barrier stalls by inspecting the rw_streaming_jobs and rw_actor_state system tables from any psql-compatible client connected to the RisingWave frontend (default port 4566):

```sql -- Check current epoch lag across all streaming actors SELECT actor_id, node_id, fragment_id, status FROM rw_actor_state WHERE status != 'Running' ORDER BY node_id;

-- See all running materialized views and their upstream source lag SHOW STREAMING JOBS;

-- Inspect the barrier wait time exposed via the metrics endpoint -- RisingWave exposes Prometheus metrics on port 1250 of each compute node ```

From kubectl, identify which compute node pod is slow:

```bash # List compute node pods and their CPU/memory utilisation kubectl top pods -n risingwave -l component=compute

# Tail the compute node logs for barrier timeout warnings kubectl logs -n risingwave -l component=compute --tail=200 | grep -i 'barrier\|epoch\|timeout'

# Check if a compute node pod has been OOM-killed or restarted recently kubectl get pods -n risingwave -l component=compute \ -o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,STATUS:.status.phase' ```

The stream_barrier_wait_second Prometheus metric (exposed at http://<compute-pod>:1250/metrics) measures how long each compute node waits for the barrier to be injected. A sustained value above two seconds indicates a stall. Fix by increasing compute node CPU resources or by isolating the slow actor onto a dedicated compute node using RisingWave's streaming parallelism configuration:

``yaml # risingwave-compute values (Helm or raw manifest) resources: requests: cpu: "4" memory: "16Gi" limits: cpu: "8" memory: "32Gi" # Increase barrier interval to reduce checkpoint frequency under load # Set in the RisingWave system parameter (psql session) ``

```sql -- Increase barrier interval from the default 1 second to reduce checkpoint pressure -- Connect as superuser ALTER SYSTEM SET barrier_interval_ms = 2000;

-- Reduce streaming parallelism for the hot materialized view -- to limit the number of actors competing for compute resources ALTER MATERIALIZED VIEW mv_hot_aggregation SET PARALLELISM = 4; ```

If one compute node is chronically slow due to a hot shard (a single actor handling disproportionate data volume), use EXPLAIN STREAM to inspect the actor graph and identify the bottleneck operator, then restructure the query to distribute load — for example, by pre-partitioning on an upstream key before the bottleneck aggregation.

Pattern 02 Kafka source connector offset loss after Meta node restart when the last checkpoint was not fully persisted to object storage

RisingWave persists streaming job state — including Kafka source connector offsets — as part of its barrier checkpoint protocol. When a barrier epoch completes (all actors across all compute nodes acknowledge it), the compute nodes upload their local state snapshots to shared object storage (S3 or MinIO) and the Meta node records the epoch as durably committed. The Meta node maintains the mapping from streaming job IDs to their last committed epoch and the corresponding object storage paths. If the Meta node restarts during the brief window between a compute node uploading its state and the Meta node recording the epoch as committed, the Meta node's in-memory epoch pointer reverts to the last epoch it recorded as fully committed before the crash. The source connector's Kafka offset stored in the uncommitted epoch's snapshot on object storage is orphaned — the Meta node does not know about it. When streaming jobs resume after the Meta node restart, the source connector replays from the last fully committed Kafka offset rather than the actual high-water mark, delivering every Kafka message between the two offsets a second time to all downstream actors and materialized views.

Detect the problem by comparing the Kafka consumer group's committed offset (as tracked by the Kafka broker) against the offset RisingWave reports as its source state. RisingWave stores source split offsets in the rw_source_backfill_info and internal state backend tables:

```sql -- After a Meta node restart, check the source job state SELECT job_id, job_name, job_status, create_time FROM rw_streaming_jobs WHERE job_status IN ('Running', 'Creating');

-- Inspect source connector state SELECT * FROM rw_sources; ```

```bash # Check the Kafka consumer group offset for the RisingWave consumer group # RisingWave uses a consumer group named after the source connector ID kafka-consumer-groups.sh --bootstrap-server kafka:9092 \ --describe --group risingwave-source-<source_id>

# Compare against what RisingWave reports — look for lag growing immediately # after a Meta restart, which indicates replay from a stale offset

# Check Meta node restart history kubectl get pods -n risingwave -l component=meta \ -o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount'

# View Meta node logs around the restart window kubectl logs -n risingwave -l component=meta --previous | tail -100 ```

Prevent this by ensuring the Meta node's embedded etcd (or external etcd, if configured) has durable storage and that checkpoint completion is confirmed before the Meta node acknowledges a new epoch. Configure the Meta node with a meta_store_endpoint backed by a persistent volume rather than emptyDir:

``yaml # meta node PersistentVolumeClaim — do not use emptyDir for Meta state volumeMounts: - name: meta-store mountPath: /risingwave/data volumes: - name: meta-store persistentVolumeClaim: claimName: risingwave-meta-pvc ``

```sql -- Set a longer checkpoint interval to reduce the window during which -- a Meta restart can cause offset loss by widening the committed epoch gap ALTER SYSTEM SET checkpoint_frequency = 10;

-- After confirming duplicate delivery, drop and recreate the affected -- materialized view to reset its state from a clean source replay DROP MATERIALIZED VIEW mv_affected; CREATE MATERIALIZED VIEW mv_affected AS SELECT ... FROM source_topic ...; ```

For idempotent downstream systems, duplicate delivery is benign if the sink is an upsert destination keyed on the message's primary key. For append-only sinks (Kafka sink, JDBC sink to an append-only table), implement deduplication in the materialized view query using a TUMBLE window with a deduplication key before the sink writes.

Pattern 03 Compute node OOM from unbounded aggregation state under high-cardinality GROUP BY keys in a streaming aggregation

RisingWave's streaming aggregation operators (implementing SUM, COUNT, AVG, MIN, MAX over a GROUP BY clause) maintain their aggregation state in a hash table resident in the compute node's heap. For each distinct combination of GROUP BY key values, one entry is allocated in the hash table holding the running aggregate state. Under steady-state operation with a bounded key space (for example, GROUP BY region with a fixed set of regions), the hash table stabilises at a predictable size. The failure occurs when the GROUP BY key space is effectively unbounded: GROUP BY user_id, session_id where both dimensions grow with each new user session, GROUP BY device_id, event_type for an IoT source with millions of devices, or GROUP BY order_id on an orders stream without expiry. As new key combinations arrive, the hash table grows without bound. RisingWave does not automatically spill aggregation state to object storage during streaming (as of versions before 1.9). The compute node's heap fills until the Linux OOM killer terminates the process. The killed compute node pod restarts, rejoins the cluster, and replays state from the last committed checkpoint epoch — but if the aggregation state snapshot in object storage is itself large (because it was serialised before the OOM), the restore phase also runs out of memory and the pod enters an OOM crash loop.

Identify unbounded aggregation by explaining the streaming plan before creating the materialized view:

```sql -- Always explain the stream plan before creating a high-cardinality MV EXPLAIN (TYPE STREAM) SELECT user_id, session_id, COUNT(*) AS event_count FROM clickstream GROUP BY user_id, session_id;

-- The output shows a HashAgg node; note whether a state table is created -- for each GROUP BY key combination

-- After an OOM, check which materialized view was running on the killed node SELECT job_id, job_name, parallelism FROM rw_streaming_jobs WHERE job_status = 'Running'; ```

```bash # Confirm OOM kill on the compute node kubectl describe pod -n risingwave <compute-pod-name> | grep -A5 'OOMKilled\|Last State'

# Check current memory usage across compute nodes kubectl top pods -n risingwave -l component=compute

# Stream compute node logs to watch for memory pressure warnings before OOM kubectl logs -n risingwave <compute-pod-name> -f | grep -i 'memory\|oom\|heap' ```

For sessions and other time-bounded key spaces, use RisingWave's SESSION or TUMBLE windowing to automatically expire old aggregation state rather than maintaining it indefinitely:

```sql -- Replace an unbounded aggregation with a tumbling window aggregation -- so state for each window is released after the window closes CREATE MATERIALIZED VIEW mv_session_counts AS SELECT window_start, window_end, user_id, COUNT(*) AS event_count FROM TUMBLE(clickstream, event_time, INTERVAL '1 hour') GROUP BY window_start, window_end, user_id;

-- For truly unbounded GROUP BY, add a TTL via EMIT ON WINDOW CLOSE -- or pre-filter the key space to bounded cardinality CREATE MATERIALIZED VIEW mv_active_users AS SELECT user_id, COUNT(*) AS events_last_hour FROM clickstream WHERE event_time >= NOW() - INTERVAL '1 hour' GROUP BY user_id; ```

``yaml # Set a per-compute-node memory limit and configure RisingWave's # streaming memory quota to trigger backpressure before OOM env: - name: RW_TOTAL_MEMORY_BYTES value: "17179869184" # 16 GiB — must match the container memory limit - name: RW_MEMORY_USAGE_RATIO_LIMIT value: "0.8" # trigger backpressure at 80% to avoid OOM resources: limits: memory: "20Gi" requests: memory: "16Gi" ``

Pattern 04 Backfill job consuming all compute resources and starving live stream processing when a large materialized view is created over existing data

When CREATE MATERIALIZED VIEW is executed against a source that already contains historical data (a table populated via INSERT, a CDC source with an existing dataset, or a Kafka topic with non-zero retention), RisingWave must first backfill the materialized view with the historical data before it can switch to processing the live stream. The backfill is implemented as a special backfill executor actor that runs within the compute nodes' streaming actor framework. The backfill executor reads historical data in batches from the source's state backend or from the object storage snapshot, processes it through the materialized view's operator graph (joins, aggregations, filters), and writes the output to the materialized view's state table. In RisingWave's actor scheduling model, the backfill executor competes for the same CPU and memory resources as the live streaming actors that are maintaining all other running materialized views. The backfill executor is not rate-limited or prioritised below live streaming actors by default. A backfill over a 500 GB table with complex joins and aggregations can saturate all available CPU cores across all compute nodes for hours, because the batch read throughput from object storage is bounded only by available CPU for decompression and processing. During this window, live streaming actors that must process new Kafka messages and advance their materialized view state are CPU-starved. Their processing falls behind the source's produce rate, barrier epoch advancement slows (because the CPU-starved live actors take longer to acknowledge each barrier), and all materialized views — including unrelated ones — experience lag.

Monitor the backfill progress and identify its resource impact:

```sql -- See all streaming jobs including backfilling ones SHOW STREAMING JOBS;

-- Identify backfill actors specifically — they appear as Creating or Backfilling SELECT job_id, job_name, job_status, parallelism, create_time FROM rw_streaming_jobs WHERE job_status = 'Creating';

-- Check the streaming plan to understand backfill operator placement EXPLAIN (TYPE STREAM) SELECT ... FROM large_table ...; ```

```bash # Monitor CPU usage across compute nodes while the backfill runs kubectl top pods -n risingwave -l component=compute --sort-by=cpu

# Watch for lag increasing on live streaming jobs # (check source lag metrics on the Prometheus/Grafana dashboard) kubectl port-forward -n risingwave svc/risingwave-meta 1250:1250 & curl -s http://localhost:1250/metrics | grep 'source_latest_message_id\|barrier_latency'

# Check whether live streaming job barrier wait times are increasing kubectl logs -n risingwave -l component=compute --tail=500 | \ grep 'barrier_wait\|checkpoint_latency' | tail -30 ```

The primary mitigation is to run large backfills during off-peak hours and to throttle the backfill rate. RisingWave exposes a backfill_rate_limit system parameter:

```sql -- Throttle the backfill rate to leave CPU headroom for live streaming -- Value is in rows per second per actor ALTER SYSTEM SET backfill_rate_limit = 50000;

-- Alternatively, reduce the parallelism of the backfilling MV -- so it uses fewer actors and fewer CPU cores ALTER MATERIALIZED VIEW mv_new_large SET PARALLELISM = 2; ```

``yaml # If your cluster has multiple compute nodes, use Kubernetes node labels # and RisingWave's compute node group feature (1.8+) to pin backfill # actors to dedicated nodes that do not serve live streaming actors nodeSelector: risingwave.io/role: backfill tolerations: - key: risingwave.io/backfill operator: Exists effect: NoSchedule ``

For very large tables, consider pre-splitting the backfill across multiple smaller materialized view definitions that each cover a partition of the data (for example, by date range), union them in a final materialized view, and drop the partition views once backfill is complete. This approach limits the blast radius of any single backfill job stalling live processing.

Pattern 05 Sink connector write failure blocking barrier propagation and pausing the entire streaming job when the downstream system is unavailable

RisingWave's sink connectors — Kafka sink, JDBC sink, Iceberg sink, and others — run as sink actors within the compute nodes' streaming actor framework. A sink actor receives processed records from upstream actors and writes them to the downstream system. Sink actors participate in the barrier protocol: before a sink actor can acknowledge a barrier to the Meta node, it must durably flush all records from the current epoch to the downstream system. If the downstream system is unavailable — the Kafka cluster is in leader election, the JDBC target database is down for maintenance, or a network partition separates the RisingWave compute nodes from the downstream system — the sink actor's flush call fails. RisingWave's sink actor enters an exponential backoff retry loop: it retries the write with increasing delays (for example, 1 s, 2 s, 4 s, 8 s up to a configurable maximum). During the entire retry window the sink actor cannot acknowledge the current barrier. The Meta node waits for the barrier acknowledgement from the sink actor. Because no new barrier can be injected until the current barrier is fully acknowledged by all actors (including the stalled sink actor), barrier epoch advancement halts globally. All upstream actors — source actors reading from Kafka, join actors, aggregation actors — finish processing their current epoch's records but cannot commit them and cannot accept new records beyond the blocked barrier. The streaming job effectively pauses. If the downstream system does not recover within RisingWave's sink timeout, the streaming job transitions to a Failed state and must be manually restarted via ALTER SINK or by dropping and recreating the sink.

Identify a blocked sink actor by checking streaming job status and sink metrics:

```sql -- Check all sinks and their current status SELECT sink_id, sink_name, sink_type, connection_name FROM rw_sinks;

-- Check if any streaming jobs are in a non-Running state SELECT job_id, job_name, job_status, create_time FROM rw_streaming_jobs WHERE job_status != 'Running';

-- After recovery, resume a paused or failed streaming job -- For a sink failure that put the job in Failed state: ALTER SINK my_kafka_sink RESTART; ```

```bash # Look for sink retry error messages in compute node logs kubectl logs -n risingwave -l component=compute --tail=500 | \ grep -i 'sink\|retry\|downstream\|flush\|timeout'

# Check barrier wait metrics — a sustained high barrier wait indicates a blocked sink kubectl port-forward -n risingwave svc/risingwave-meta 1250:1250 & curl -s http://localhost:1250/metrics | \ grep -E 'barrier_wait|sink_commit_latency|sink_throughput'

# Check if the downstream Kafka cluster is healthy from the RisingWave network perspective kubectl exec -n risingwave <compute-pod> -- \ curl -s --connect-timeout 5 http://kafka-broker:9092 || echo 'Kafka unreachable' ```

Prevent indefinite blocking by configuring a sink timeout that causes the sink actor to fail fast rather than retrying indefinitely, and by decoupling the sink from the core streaming job using RisingWave's sink_decouple feature where available:

``sql -- Create a Kafka sink with explicit timeout and decouple settings CREATE SINK my_kafka_sink FROM mv_processed_events WITH ( connector = 'kafka', properties.bootstrap.server = 'kafka:9092', topic = 'processed-events', type = 'append-only', -- Decouple sink from barrier protocol (requires RisingWave 1.7+) -- Allows the streaming job to continue even if the sink is slow sink.decouple = 'enable' ); ``

``yaml # Set sink-related timeouts in the compute node environment # to prevent indefinite retry loops from blocking barrier progression env: - name: RW_SINK_TIMEOUT_SECONDS value: "60" - name: RW_MAX_SINK_PARALLELISM value: "4" ``

For JDBC sinks, ensure the target database is configured with connection pooling and has a health check endpoint. Use Kubernetes readiness probes on the downstream system's Service to prevent RisingWave from attempting sink writes to a pod that is not ready. For Kafka sinks, configure properties.request.timeout.ms and properties.delivery.timeout.ms to give the Kafka producer a bounded retry window rather than the default unbounded timeout. In all cases, instrument the rw_sink_metrics Prometheus gauge to alert when sink flush latency exceeds the barrier interval — an early warning that the sink is becoming a barrier bottleneck before it fully stalls the streaming job.