Databricks makes multi-statement transactions generally available

Databricks said multi-statement transactions are now generally available for Unity Catalog-managed Delta Lake and Apache Iceberg tables. The feature allows a sequence of changes across multiple tables to commit or roll back as one unit.

That expands Databricks beyond its default model, in which each SQL statement is an atomic transaction against one table. Its transaction documentation says every write target in a multi-statement transaction must be a Unity Catalog-managed Delta or Iceberg table with catalog commits enabled. Catalog commits are off by default, require the delta.feature.catalogManaged table property and are unavailable for external or foreign tables.

The rollout comes with an unresolved qualification. Databricks’ October 16 announcement describes transactions across both Delta Lake and Apache Iceberg tables as generally available. However, the transaction and ACID documentation observed October 17 still says writes to Unity Catalog-managed Iceberg tables are in Private Preview. The company has not reconciled those statements in the supplied materials.

For data pipelines, an ETL job can now update related tables in one transaction, preventing a failure from leaving only part of the workflow published. Databricks says developers can also combine stored procedures, temporary tables and SQL scripting in a single atomic workflow.

Databricks supports two modes. A non-interactive BEGIN ATOMIC ... END block commits automatically on success and rolls back automatically on error. An interactive BEGIN TRANSACTION session gives the user explicit COMMIT or ROLLBACK control. Non-interactive transactions can run on a SQL warehouse, serverless compute or a cluster using Databricks Runtime 18.0 or later; interactive transactions require a SQL warehouse.

Supported read and write paths include Delta Lake and Iceberg tables, including streaming tables. That lets a workflow publish a streaming-table update atomically with other changes. Databricks also says Unity Catalog attribute-based access controls, including row-level security and column masking, continue to govern reads and writes within a transaction.

Supported operations include INSERT, UPDATE, DELETE, MERGE, COPY INTO and selected inspection commands. Databricks uses optimistic concurrency control: non-interactive transactions can use row-level conflict detection when row-level concurrency is enabled, while interactive transactions generally detect conflicts at the table level. A transaction can read from or write to as many as 100 tables combined and read from as many as 100 views. DDL operations such as CREATE TABLE, ALTER TABLE and DROP TABLE remain unsupported inside a transaction.