Databricks adds on-demand state repartitioning for stateful streaming

Databricks has introduced on-demand state repartitioning in Public Preview. Supported Apache Spark Structured Streaming queries can now change their state partition count across a restart without discarding checkpoint state. Previously, the shuffle partition count was fixed when a checkpoint was created, so using a different count required a new checkpoint.

For teams running long-lived stateful pipelines, that means partitioning can be resized while accumulated state is retained. Databricks says users stop the query, set spark.sql.streaming.stateStore.partitions to the new count and restart it. For stateful queries, that setting takes precedence over spark.sql.shuffle.partitions.

Once restarted, the query completes the last planned microbatch and runs a repartition operation, redistributing state across the new number of partitions before processing resumes. The capability requires Databricks Runtime 18 LTS or later and a query using the RocksDB state store provider.

RocksDB is the default state store provider in Databricks Runtime 17.3 and later, but repartitioning still requires Runtime 18 LTS or later. Databricks’ stateful streaming documentation says the state-management scheme cannot be changed between restarts. Queries that are not already using RocksDB must therefore start from scratch with a new checkpoint location to change providers.

Operators can track the repartition duration in StreamingQueryProgress, under durationMs at controlBatch.REPARTITION. Databricks warns that repartitioning may take longer as state size increases.

DataPhoenix has separately covered Databricks’ recent funding and acquisition activity.