Databricks makes Real-Time Mode generally available for Spark Declarative Pipelines

Databricks made Real-Time Mode generally available for Spark Declarative Pipelines on October 29, bringing latency-sensitive operational streaming into the same managed, declarative framework that customers use for data pipelines.

The release follows the public preview that Databricks introduced on May 27. It expands Spark Declarative Pipelines beyond ETL and conventional streaming to uses including fraud scoring, live personalization, alerts, recommendations and real-time feature computation.

Databricks says putting those workloads in one framework can reduce duplicate code as well as the separate governance, observability, billing and operational systems that can come with running another low-latency engine.

Real-Time Mode changes how a streaming query moves through a pipeline. Databricks’ documentation says it keeps batches running and processes records as they become available. Query stages run at the same time, passing results through a streaming shuffle rather than waiting for an upstream stage to finish. The design is intended to reduce delays between stages.

Databricks claims end-to-end latency as low as five milliseconds, depending on the workload. The company also says its Feature Store uses the architecture to reach 200-millisecond p99 latency from Kafka to an online feature store. The reviewed evidence did not independently measure either figure.

In internal feature-computation benchmarks, Databricks says Spark Real-Time Mode was up to 92% faster than Apache Flink. The linked Databricks Solutions benchmark suite says the tests ran at 10,000 rows per second after a five-minute warm-up. It reports p99 latency of 3 milliseconds for both systems on a stateless transformation, 14 milliseconds for Spark versus 45 milliseconds for Flink on a stateful aggregation, and 9 milliseconds versus 95 milliseconds on a stream-static join. No independent reproduction was identified in the reviewed material.

Customers must run a continuous pipeline, set spark.databricks.streaming.realTimeMode.enabled, and use an update flow with pipelines.trigger set to RealTime to enable the feature. Databricks lists classic and serverless compute as supported.

Real-Time Mode itself became generally available for Apache Spark Structured Streaming in March; the October release applies that status to its Spark Declarative Pipelines integration. The pipeline documentation lists Kafka as a supported source and sink and supports a ForEach sink for custom per-row destinations. It does not support Delta as a source or sink, stream-to-stream joins, or forEachBatch. On classic compute, all query stages need task slots at the same time. Longer checkpoint intervals can lower overhead but increase recovery time after a failure.