Databricks AI Search adds direct indexing for streaming, materialized and Iceberg tables

Databricks has added native source support to AI Search for Streaming Tables, Materialized Views and managed tables using Iceberg v3 or later. The product already supported Delta Lake tables.

Previously, customers using those source types had to create an intermediate Delta representation and maintain custom pipelines to keep it synchronized with the original data, according to Databricks. The company says direct indexing removes both the extra table and the synchronization work. It did not quantify any reduction in cost, latency or maintenance.

The indexing workflow remains the same: customers create a Delta Sync Index and point it at a source table through the Databricks interface, Python SDK or REST API. Databricks documentation says this index type updates incrementally as the source changes.

Databricks says AI Search can index new records as they arrive in an append-only Streaming Table. For a Materialized View, changes to underlying data are incrementally maintained in the view and propagated to the search index, allowing searches over filtered, aggregated or joined state. Managed Iceberg tables can serve as sources without conversion to another format, although support is limited to managed tables using Iceberg v3 or later.

A source for a standard AI Search endpoint must still provide a change data feed, according to the AI Search overview. Databricks says row tracking supplies the feed automatically for Delta tables, materialized views with external metadata enabled and managed Iceberg v3-or-later tables. A Materialized View source must have external metadata, also called Smart Clone, enabled on the view or its defining pipeline. Databricks did not specify a release stage for the three new source integrations.