Databricks expands Lakeflow Designer with data-quality and output controls

Databricks has expanded Lakeflow Designer with no-code data cleaning, deduplication and quality checks, adding broader controls for joins, outputs and collaboration to its visual data-preparation product. The company said these features have shipped in Lakeflow Designer, which it describes as generally available across Databricks workspaces.

The new tools let data teams enforce checks and route results without writing code. The Prepare operator chains cleaning and derived-column steps. Unique removes duplicate rows across an entire record or by selected keys. Guardrails tests rows against configured conditions and separates passing, failing and error results. Teams can continue with passing rows, block the passed output or fail the workflow before later operators run. Databricks’ operator documentation describes those controls.

The Join operator now accepts three or more input tables. With two inputs, it produces matched, left-unmatched and right-unmatched outputs by default, while Filter creates included and excluded branches. Select can choose columns dynamically by data type, naming rules or metadata formulas, allowing selection logic to adjust when a schema changes.

Sources and outputs can now use parameters that teams override for individual schedules, including test and production runs. The Output operator can write managed Unity Catalog tables, publish materialized views that refresh with the visual data prep, or place CSV, Excel and JSON files in Unity Catalog volumes. Table outputs support overwrite, append and merge modes, with merge using specified keys to update or insert rows. Databricks documents the output configuration in its workflow guide.

Interface changes include a searchable table of contents, generated-code inspection and a version-history view that marks added, removed and changed operators. Databricks also says coworkers editing the same visual data-prep file can now see one another’s live cursor activity.

Four other capabilities are not part of this shipped release. Databricks describes integration with Databricks Apps, profiling across every row returned by an operator, a custom Report operator and Lakebridge-powered bulk migration tooling as forthcoming. The Apps integration is intended to let non-technical users configure inputs, run workflows and view results. The Report operator is planned to combine outputs and save reports to a workspace or send them by email. Current Lakeflow Designer documentation describes profiling selected output data in a details panel, rather than the forthcoming full-output profiling. Databricks gave no delivery dates for any of the four planned capabilities.