The medallion architecture organises a lakehouse into three layers — bronze, silver and gold — so that every dataset moves from raw to business-ready in clear, testable steps. On Databricks it has become the default pattern for data engineering, and for good reason: it makes data quality visible, keeps pipelines replayable and gives analytics and AI teams a stable place to build.

But "bronze, silver, gold" is a naming convention, not a design. This guide covers the decisions that actually determine whether a medallion lakehouse stays reliable once it holds hundreds of tables.

What each layer is for

Layer Purpose Typical contents Who uses it
Bronze Land data exactly as received Raw files and events, plus ingestion metadata Data engineers
Silver Make data correct and consistent Cleansed, deduplicated, typed, conformed entities Engineers, data scientists
Gold Make data useful for a decision Aggregates, dimensional models, feature tables Analysts, BI, ML and GenAI apps

The most useful rule of thumb: bronze answers "what did we receive?", silver answers "what is true?", gold answers "what do we need to know?"

Bronze: land everything, change nothing

Bronze exists so you can always replay history. If a transformation bug corrupts silver, bronze lets you rebuild it without going back to source systems.

Practical guidance:

  • Store data as received, in Delta tables, with minimal parsing. Keep the original payload (for example a JSON string column) alongside any extracted fields.
  • Add ingestion metadata to every row: source system, file name or offset, and ingestion timestamp. It makes debugging and incremental processing far easier.
  • Use incremental ingestion. Auto Loader for files and structured streaming for Kafka or event hubs avoid re-reading data you have already processed.
  • Be append-only. Avoid updates and deletes in bronze except for legal or privacy obligations, which you should handle with a documented process.
  • Keep schemas permissive. Let schema evolution capture new columns in bronze and decide how to handle them in silver, rather than failing ingestion.

Silver: where quality is enforced

Silver is where most engineering effort belongs. It turns raw records into trustworthy entities — customers, orders, devices, campaigns — that many gold tables can reuse.

  • Deduplicate and apply change data. Use MERGE for upserts, or declarative change-data-capture features, to maintain current state from CDC feeds.
  • Enforce types and constraints. Cast explicitly, standardise time zones and units, and use Delta constraints for rules that must never be broken.
  • Declare data quality expectations. Lakeflow Declarative Pipelines (formerly Delta Live Tables) let you declare expectations that warn, drop or fail on bad records. Whatever tooling you use, decide per rule whether a violation should stop the pipeline or quarantine the record.
  • Quarantine instead of silently dropping. Route failing records to a quarantine table with the reason attached, so issues can be fixed and replayed.
  • Conform shared entities once. A single silver customer entity, with keys resolved across systems, prevents every gold table from reinventing the same joins.

A common mistake is to push business logic into silver. Keep silver about correctness; keep business definitions — "active customer", "net revenue" — in gold or a semantic layer, where they can change without rewriting foundations.

Gold: design for the consumer

Gold tables exist for specific consumers, so design them from the question backwards.

  • Model for the workload. Star schemas suit BI tools; wide, denormalised tables suit ad-hoc analysis; feature tables suit machine learning; curated, well-described tables suit GenAI and natural-language analytics.
  • Name and document for humans. Clear table and column comments are no longer optional — AI assistants and text-to-SQL tools rely on them to generate correct queries.
  • Define metrics once. Put business metrics in a semantic layer or metric views so every dashboard and AI assistant reports the same numbers.
  • Keep gold rebuildable. Everything in gold should be derivable from silver, so you can change a definition and backfill with confidence.

Governance across every layer

Governance works best when it is part of the architecture rather than a later project. On Databricks, Unity Catalog provides the building blocks:

  • Organise by catalog and schema. A common pattern is one catalog per environment (dev, test, prod) with schemas per domain and layer, or catalogs per layer. Pick one convention and apply it everywhere.
  • Grant access by layer. Most users should only read gold; engineers need silver; very few people need bronze, which may hold unmasked personal data.
  • Classify and mask sensitive data with tags, row filters and column masks, rather than creating copies of tables for different audiences.
  • Use lineage. Table and column lineage shows which dashboards and models are affected by a change in an upstream table — invaluable for impact analysis.

Operating the lakehouse: quality, observability and cost

A medallion architecture that nobody monitors degrades quietly. Build operations in from the first pipeline:

  1. Monitor freshness, volume and schema for critical tables and alert when they drift from normal.
  2. Test transformations with unit tests for logic and data tests for expectations, run in CI before deployment.
  3. Deploy as code. Use Databricks Asset Bundles or your infrastructure-as-code tool so pipelines, jobs and permissions are versioned and reviewed.
  4. Manage file layout. Use liquid clustering or appropriate partitioning, and schedule table maintenance, so query performance does not decay as tables grow.
  5. Right-size compute. Separate ingestion, transformation and BI workloads, use job compute for pipelines and serverless where it fits, and review cost per pipeline regularly.

Common pitfalls

  • Too many layers. Adding "platinum" or "silver-2" layers usually signals unclear responsibilities. Fix the design instead of adding a tier.
  • Gold tables built directly from bronze. It saves a step once and costs you consistency forever.
  • Every team building its own silver. Shared entities should be owned once and reused.
  • Quality checks only in gold. By then, bad data has already spread. Enforce quality where it enters — in silver.
  • No ownership. Every table needs an owner and a description, or it becomes an orphan nobody trusts.

A starting checklist

  • Bronze is append-only, incremental and carries ingestion metadata
  • Silver enforces types, deduplication and declared expectations, with quarantine
  • Shared entities are conformed once in silver
  • Business metrics live in gold or a semantic layer, not scattered across dashboards
  • Unity Catalog permissions follow layers, with sensitive data masked
  • Freshness, volume and schema are monitored for critical tables
  • Pipelines, jobs and permissions are deployed as code

The medallion architecture is simple to describe and easy to get subtly wrong. Treat each layer as a contract — what it guarantees and to whom — and the lakehouse becomes a foundation that analytics, machine learning and generative AI can all build on with confidence.

Planning a lakehouse or modernising an existing one? Talk to our data platform team.