Databricks makes Apache Spark 4.3 available in Runtime 19

Databricks made Apache Spark 4.3 available in Databricks Runtime 19, bringing stateful workloads to Spark’s Real-Time Mode. Developers can now run streaming aggregations, duplicate removal and the JVM Scala/Java transformWithState operator through the existing Structured Streaming APIs.

Databricks said its tests of those three workloads produced P99 end-to-end latency of 136 milliseconds for dropDuplicates, 139 milliseconds for window aggregation and 141 milliseconds for transformWithState. The company limited those results to the tested workloads and said they were not a latency guarantee for every pipeline.

The stateful Real-Time Mode implementation uses concurrent stage scheduling and pipelined shuffles. Producer and consumer stages can run at the same time, allowing consumers to begin reading shuffle output before producers finish.

Spark 4.3 also adds Slowly Changing Dimension Type 2 support to Auto CDC through Python, SQL and Spark Connect. It preserves changes as versioned rows bounded by __START_AT and __END_AT metadata columns. Because the feature applies micro-batches with MERGE, its target catalog connector must support Data Source V2 row-level operations. Databricks cited Apache Iceberg as a common choice and said a built-in file format such as Parquet does not qualify on its own.

Project Feather adds two opt-in paths intended to cut fixed overhead on smaller workloads: shuffle-free single-task execution for eligible small, single-partition query fragments and Arrow-native Dataset caching. Other Spark 4.3 release changes include REST endpoints for inspecting Spark Connect sessions and operations, operation IDs for request attribution, expanded JDBC and API coverage, Python DataFrame.zip, SQL JSON_TABLE, VARIANT transformation functions and ASOF JOIN, plus preview nanosecond-precision timestamp types.

The November 5 announcement was not Runtime 19’s first release. Databricks’ Runtime 19 release notes list a Spark 4.3.0-powered beta on June 15 and general availability on July 23, while the Apache Spark project announced the upstream Spark 4.3.0 release on October 30. The opened sources do not explain that pre-release versioning relationship in detail. Apache said the upstream release addressed more than 1,100 Jira tickets with contributions from more than 210 people.