A lakehouse is a data lake that behaves like a database: ACID transactions, schema enforcement, fine-grained governance and fast SQL, on open files in cheap object storage. On Databricks that behaviour comes from a handful of mechanisms: the Delta transaction log, file layout and statistics, deletion vectors, Unity Catalog, and the ingestion and pipeline frameworks on top. Knowing how they work is the difference between a platform that stays fast and cheap and one that slowly fills with small files, slow merges and surprise bills. This post opens each one up.

How does a Delta table give you ACID on object storage?

A Delta table is a directory of Parquet data files plus a transaction log in _delta_log/. Each commit is a JSON file with a zero-padded version number (00000000000000000042.json) containing actions:

{"commitInfo": {"operation": "MERGE", "operationMetrics": {"numTargetRowsUpdated": "1203"}}}
{"remove": {"path": "part-00007-....parquet", "deletionTimestamp": 1760000000000, "dataChange": true}}
{"add": {"path": "part-00011-....parquet", "size": 104857600, "dataChange": true,
         "stats": "{\"numRecords\":812000,\"minValues\":{\"order_date\":\"2026-09-01\"},\"maxValues\":{\"order_date\":\"2026-09-30\"}}"}}

(Simplified.)

The current state of the table is the set of files added and not yet removed, replayed from the log. To avoid replaying thousands of JSON files, Delta periodically writes a checkpoint: a Parquet summary of the full table state. Readers start from the latest checkpoint and apply only the JSON commits after it. The other actions in the log hold the schema and table properties (metaData), the reader and writer feature requirements (protocol), and per-application transaction IDs (txn), which make streaming writes idempotent.

Atomicity comes from the commit itself. A writer writes its new data files first, invisible to readers, then tries to create the next log file. Exactly one writer can create version N. Readers only ever see whole commits, so they see the table before or after a transaction, never halfway through.

Concurrency is optimistic. A writer reads snapshot N, does its work, and attempts to commit N+1. If another writer got there first, Delta checks whether the two transactions logically conflict: did the other commit add or remove files that this transaction read or rewrote? Non-conflicting commits (two appends, or merges touching different files) are retried automatically on top of the new version. Real conflicts raise errors such as ConcurrentAppendException or ConcurrentDeleteReadException. Two practical consequences:

  • Make concurrent jobs touch disjoint data, and include the clustering or partition columns in MERGE and UPDATE predicates, so conflict detection can prove the writes are disjoint.
  • On Databricks, row-level concurrency (for unpartitioned tables, including liquid-clustered ones, with deletion vectors enabled, on recent runtimes) detects conflicts at row rather than file level, which removes most false conflicts between concurrent MERGEs.

You can inspect all of this directly:

DESCRIBE HISTORY sales.orders;   -- every commit: version, operation, metrics, user, job
DESCRIBE DETAIL  sales.orders;   -- file count, size, clustering columns, table features

How do time travel and retention work?

Because removed files are only logically removed, earlier versions remain queryable:

SELECT * FROM sales.orders VERSION AS OF 1182;
SELECT * FROM sales.orders TIMESTAMP AS OF '2026-10-10T06:00:00';
RESTORE TABLE sales.orders TO VERSION AS OF 1182;   -- undo a bad write in seconds

Two retention settings bound how far back you can go:

  • delta.logRetentionDuration (default 30 days): how long log entries are kept.
  • delta.deletedFileRetentionDuration (default 7 days): how long removed data files must be kept before VACUUM may delete them.

VACUUM physically deletes files that are no longer referenced and are older than the retention threshold. After it runs, time travel beyond that point fails. Never lower the threshold below the duration of your longest-running reads or streams; Delta refuses retention below 7 days unless you explicitly disable the safety check. Time travel is an operational safety net, not a long-term archive. Keep regulatory history in SCD tables or snapshots.

What are deletion vectors, and why do merges get faster?

Without deletion vectors, a DELETE, UPDATE or MERGE that touches a single row in a 1 GB file must rewrite the entire file (copy-on-write). With deletion vectors, Delta writes a compact bitmap that marks rows as deleted, and leaves the data file in place. Updated rows are written as new rows, with the old ones marked deleted. Readers filter marked rows at query time (merge-on-read).

The result is much cheaper and faster row-level changes, especially for CDC and GDPR-style deletes. The cost is a little extra read work until the files are compacted. OPTIMIZE drops the vectors for the files it rewrites, and REORG TABLE ... APPLY (PURGE) guarantees it when you need deleted data physically removed, for example for privacy requests. On recent Databricks runtimes, deletion vectors are enabled by default for new tables.

How should data be laid out: partitioning, Z-order or liquid clustering?

Query speed on a lakehouse is mostly about reading fewer files. Delta stores min/max statistics per file for the first 32 columns by default (adjustable with delta.dataSkippingNumIndexedCols or delta.dataSkippingStatsColumns). The engine skips every file whose range cannot match the filter. Data skipping only works if similar values are co-located in the same files.

  • Hive-style partitioning (PARTITIONED BY (date)) physically separates data into directories. It is effective for very large tables with a low-cardinality filter column. Over-partitioning, on high-cardinality columns or small tables, creates millions of small files. Databricks' long-standing guidance is not to partition tables smaller than about 1 TB.
  • Z-ordering (OPTIMIZE ... ZORDER BY (a, b)) interleaves several columns so that skipping works on each. It is not incremental: each run rewrites substantial data, and changing columns means rewriting everything.
  • Liquid clustering replaces both for new tables. You declare clustering keys, and Databricks clusters data incrementally: only newly written or unclustered data is reorganised on OPTIMIZE. You can change keys without rewriting history, which applies as data is rewritten.
CREATE TABLE sales.orders (
  order_id BIGINT, customer_id BIGINT, order_ts TIMESTAMP, region STRING, amount DECIMAL(18,2)
) CLUSTER BY (customer_id, order_ts);

ALTER TABLE sales.orders CLUSTER BY (region, order_ts);   -- change keys as query patterns change
ALTER TABLE sales.events CLUSTER BY AUTO;                  -- let Databricks pick keys from query history
OPTIMIZE sales.orders;                                      -- incremental clustering

Choose keys from the columns most often used in filters and joins: up to four, ordered by importance. A table cannot use liquid clustering together with partitioning or Z-order. With predictive optimization enabled for Unity Catalog managed tables, Databricks schedules OPTIMIZE, VACUUM and statistics collection itself, based on how each table is used.

How do you propagate only what changed?

Enable the Change Data Feed and Delta records row-level changes per commit:

ALTER TABLE silver.customers SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

SELECT customer_id, segment, _change_type, _commit_version, _commit_timestamp
FROM table_changes('silver.customers', 2045)        -- all changes since version 2045
WHERE _change_type IN ('insert', 'update_postimage', 'delete');

_change_type is one of insert, update_preimage, update_postimage or delete. Downstream jobs read the feed incrementally, in batch or as a stream with readChangeFeed. A gold aggregate then recomputes only the affected keys instead of rescanning the whole silver table. This is the backbone of incremental medallion pipelines; see our CDC post for the merge patterns that feed it.

What does Unity Catalog actually govern?

Unity Catalog (UC) is the governance layer for every data and AI asset in the account. Its object model is a three-level namespace, catalog.schema.object, where objects are tables, views, materialized views, volumes (governed file storage), functions and registered ML models.

What it gives you in practice:

  • One permission model. GRANT SELECT ON TABLE ..., GRANT USE CATALOG ..., inherited down the hierarchy and enforced on every compute type.
  • Managed vs external tables. Managed tables live in UC-managed storage, and get lifecycle management and predictive optimization. External tables point at paths you manage.
  • Automatic lineage down to column level, across notebooks, jobs, SQL and dashboards. You can answer "what breaks if I change this column?" before you change it.
  • Fine-grained access with row filters and column masks, defined as SQL functions:
-- gov.user_regions maps each user to the regions they may see
-- Name the parameter differently from any column: inside the subquery an unqualified
-- `region` would resolve to m.region (columns win over parameters), making the filter always true.
CREATE FUNCTION gov.row_region_filter(p_region STRING)
RETURN is_account_group_member('admins')
    OR EXISTS (SELECT 1 FROM gov.user_regions m
               WHERE m.user_email = current_user() AND m.region = p_region);

ALTER TABLE sales.orders SET ROW FILTER gov.row_region_filter ON (region);

CREATE FUNCTION gov.mask_email(email STRING)
RETURN CASE WHEN is_account_group_member('pii_readers') THEN email
            ELSE regexp_replace(email, '^[^@]+', '***') END;

ALTER TABLE crm.customers ALTER COLUMN email SET MASK gov.mask_email;
  • Auditing and cost visibility through system tables such as system.access.audit, system.access.table_lineage and system.billing.usage. These are ordinary tables you can query, alert on and build dashboards from.

We create one catalog per environment (dev, staging, prod), with schemas per domain and layer. Jobs run as service principals, never as people, and production write access is limited to deployment pipelines. That structure is what makes the productionisation workflow safe.

How should files be ingested?

Auto Loader (cloudFiles) incrementally ingests new files from cloud storage or volumes, with exactly-once tracking of which files it has processed:

(spark.readStream.format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", "/Volumes/prod/meta/schemas/orders")
    .option("cloudFiles.schemaEvolutionMode", "addNewColumns")   # evolve on new fields
    .option("cloudFiles.inferColumnTypes", "true")
    .load("/Volumes/prod/landing/orders/")
    .writeStream
    .option("checkpointLocation", "/Volumes/prod/checkpoints/bronze_orders")
    .trigger(availableNow=True)
    .toTable("prod.bronze.orders"))

Auto Loader infers and tracks the schema. In addNewColumns mode the stream stops when it detects new columns and picks them up on restart, so run it as a job with retries. Values that do not fit the schema go into a _rescued_data column instead of failing the stream or being silently dropped. For large directories, file notification mode uses cloud events rather than repeated directory listing. For SaaS and database sources, Lakeflow Connect's managed connectors remove much of the custom ingestion code.

When do we use Lakeflow Declarative Pipelines?

Lakeflow Declarative Pipelines (formerly Delta Live Tables) let you declare tables and their dependencies. The framework handles orchestration, incremental processing, retries, checkpoints and data quality:

CREATE OR REFRESH STREAMING TABLE silver_orders (
  CONSTRAINT valid_id     EXPECT (order_id IS NOT NULL) ON VIOLATION FAIL UPDATE,
  CONSTRAINT valid_amount EXPECT (amount >= 0)          ON VIOLATION DROP ROW,
  CONSTRAINT recent_ts    EXPECT (order_ts > '2020-01-01')               -- record, don't block
)
AS SELECT * FROM STREAM(prod.bronze.orders);

CREATE OR REFRESH MATERIALIZED VIEW gold_daily_revenue AS
SELECT date(order_ts) AS order_date, region, sum(amount) AS revenue
FROM silver_orders GROUP BY ALL;

Expectations are declarative data-quality rules with three behaviours: record violations and keep the rows (the default), drop the violating rows, or fail the update. Their pass and fail counts are captured as metrics automatically, which gives a data-quality history with no extra code. Streaming tables process append-only sources incrementally. Materialized views are refreshed incrementally where the engine can, and fully otherwise.

We choose declarative pipelines for standard medallion flows, CDC (via AUTO CDC) and teams that benefit from managed operations. We choose plain jobs with Spark code when we need fine control over each step, unusual sinks or complex custom logic.

What about performance and interoperability?

  • Photon, Databricks' vectorised C++ query engine, accelerates SQL and DataFrame workloads without code changes. It is on by default on SQL warehouses and serverless compute.
  • Serverless compute for jobs, notebooks, pipelines and SQL removes cluster sizing and start-up time. Combine it with budget policies and tags for cost attribution.
  • Open formats. Delta tables are open Parquet plus an open log. UniForm additionally writes Iceberg metadata, so Iceberg clients can read the same table, and Delta Sharing shares live tables across organisations and platforms without copying.

A maintenance checklist

  • Liquid clustering on large, frequently filtered tables; no new partitioning under ~1 TB
  • Predictive optimization enabled, or scheduled OPTIMIZE and VACUUM jobs
  • Retention set deliberately; VACUUM threshold above your longest read or stream
  • Deletion vectors on for CDC tables; purge where privacy requires physical deletion
  • Change Data Feed on silver tables that feed incremental gold jobs
  • One UC catalog per environment; service principals for jobs; row filters and masks for PII
  • Expectations on every silver table, with alerting on their metrics
  • Audit and billing system tables in a monitored dashboard