From raw to golden data: the journey your data actually takes

There are two dates in the life of any dataset. The day it lands in your platform, and the day someone finally trusts it enough to put it in front of a decision.

Ingestion projects celebrate the first date. The business runs on the second. And in most companies the distance between the two is measured in months of skilled, largely invisible, manual work — a journey almost no one budgets for, because it doesn’t have a name on the architecture diagram. The diagram shows sources, pipes, and a lakehouse. It ends exactly where the hard part begins.

I want to walk that road properly, station by station, because you can’t understand what agents change until you feel the weight of what’s there today.

The road from raw to trusted

Station one: finding out what you actually received

Nobody knows what landed. Not really. The source team says the customer table has clean emails; it has 14% nulls and three formats. The order table’s “status” column has eleven values, four of which appear nowhere in any documentation, because they belong to a process that was retired in 2019 and never cleaned up.

So the first real work is profiling. Row counts, null rates, distinct values, candidate keys, the columns that quietly carry national IDs. Done properly, per table, this takes an engineer hours. Multiply by a few hundred tables and it becomes the thing everyone agrees is essential and nobody finishes. Most teams profile the tables that burned them last quarter and hope about the rest.

Station two: cleansing and formatting

Now the scrubbing. Dates arrive in three formats. Phone numbers with and without country codes. “EGY”, “EG”, and “Egypt” in the same column. Trailing whitespace that breaks a join at 2am. None of this is hard. All of it is endless, and every rule someone writes lives in a script that only its author understands. When that person leaves, the rules stay behind like furniture nobody remembers buying.

Station three: business calculations

Here the work changes character. Cleansing has right answers; calculations have opinions. What counts as revenue? Is a cancelled-then-reinstated order one order or two? Which exchange rate, on which day? An engineer can’t answer these alone, so this station runs on meetings. The analyst explains, the engineer translates, something gets lost, and three weeks later finance asks why their number is different from marketing’s.

The definitions that come out of those meetings usually live in the worst possible place: inside the SQL, invisible to the people who argued for them.

Station four: designing the model

Somewhere along the way, someone has to decide what shape this data should take. Which business processes matter. What grain each fact table carries — one row per order, or per order line, or per line per day? Which dimensions are shared across processes so that “customer” means the same thing in sales and in support.

This is star schema design, and it is genuinely skilled work. Kimball wrote it down decades ago: pick the process, declare the grain, find the dimensions, find the facts. The method isn’t the problem. The problem is that under deadline pressure, teams skip it, and the warehouse becomes a mirror of the source systems with extra steps: a hundred tables that reproduce the operational mess at higher cost.

Station five: mastering the data

And then the hardest station of all. The same customer exists in your CRM, your billing system, and your support desk, spelled three ways, with two birthdays. Master data management is the discipline of resolving that: matching records that refer to the same real-world thing, merging them, and writing survivorship rules that decide which value wins when sources disagree.

The industry’s honest track record here is sobering. Enterprise MDM programs are scoped in years. Somewhere in your company right now there is probably a spreadsheet named something like customer_match_rules_v14.xlsx, and a steward queue of unresolved matches that grows faster than anyone reviews it.

The prize at the end is the golden record: one trusted version of each customer, each product, each supplier. Most organizations chase it for years and settle for bronze-plated.

The map that lies

One more thing sits alongside every station: the data catalog that’s supposed to describe all of it. In most organizations the catalog is written after the work, by whoever has time, which means it was accurate for about a month once. Ask a data engineer whether they trust their catalog and watch their face.

Even the naming system we use for the road itself is vaguer than it looks. Medallion architecture (bronze, silver, gold) tells you how refined a table is, but not what it does. Practitioners have been pointing this out for a while: silver in particular becomes a catch-all layer, and without shared definitions every team draws the lines differently. A table sitting in silver could be doing cleansing, joins, deduplication, or mastering. The medal says “somewhere in the middle.” That is not an answer an auditor, a new hire, or an AI agent can use.

So that’s the road. Five stations of skilled manual work, a catalog that drifts, and a labeling system that describes altitude but not purpose. This is where the cost of the modern data platform actually lives. Not in storage or compute, but in the months of human translation between “landed” and “trusted.”

The same road, walked by agents

Celvari’s starting position is different in one structural way, and everything else follows from it: the journey starts with a domain, not a pipeline.

You point the platform at an area of your business — customers, orders, claims — and the domain building agents take it from there. From that single starting point you get the profiling, the model, the data products, and the mastering, in one governed flow. Here is what each station looks like on the other side.

Profiling stops being a project and becomes a property. Every table in the domain is profiled and classified as it enters: row counts, nulls, key candidates, and (the part that saves careers) sensitive data flagged at column grain before it ever reaches a product. The profile isn’t a report someone ran in March. It’s live metadata the rest of the journey builds on.

Cleansing becomes a generated data product, not a pile of scripts. Start with where the rules come from. In Celvari they aren’t buried in code; they live in the domain’s knowledge base, drawn from the documents the agents have ingested (the data dictionary, the policy PDFs) and from what your team tells the platform directly. From those rules the agents generate the cleansing logic, and the output lands as a new, governed data product — classified silver, with its tests and lineage attached. The quality gate still stands in front of it, so bad data stops with a named violation instead of surfacing three weeks later in a dashboard. The rules that used to be folklore in a departed engineer’s scripts are now versioned inputs you can read, and the cleansed table is a product you can point at, test, and trust.

Business calculations follow the same path. Same sources — the knowledge base, your documents, your people — applied as data products too. This is the layer of honest, column-level work: the month extracted from a full date, a price multiplied by a quantity, a margin derived from two fields. Still within a single table or a simple join; the heavier multi-table modeling comes at the next station. What changes is where the definition lives. It’s no longer sealed inside the SQL where finance can’t see it. It sits in the catalog, linked to the exact product that implements it, and the three-week translation loop between analyst and engineer collapses into a conversation with a planning agent that asks the clarifying questions itself.

The model is designed, not mirrored. The design agents work the way a senior architect does. They start from your business questions, declare the grain of every process in plain language, verify it against the profiled data, and lay out conformed dimensions across processes. A separate critic agent challenges the design before a person ever sees it — its explicit job is to reject source-table mirrors. What reaches your steward for approval is a real dimensional model with real metric expressions, not a hundred renamed copies of the source.

Mastering becomes a session, not a program. This is the newest part of the platform, in beta with early users now. An MDM session runs inside a domain: agents propose matches across sources, apply survivorship logic, and produce candidate golden records. A human steward reviews and decides, which is exactly where human judgment belongs. The matching that used to live in customer_match_rules_v14.xlsx becomes governed, versioned platform logic. The queue that grew faster than anyone could review it becomes a session with a beginning and an end.

The catalog stops drifting because nobody writes it. This might be the deepest change of the whole journey. In Celvari, catalog metadata is a byproduct of the work itself. When an agent profiles a table, the profile lands in the catalog. When it builds a product, the lineage is recorded at column grain while the build happens. When a definition is extracted from your documents, it’s linked to its columns in the same motion. There is no “documentation phase” to skip, so there is nothing to fall out of date. The map is drawn by the territory.

And the medallion finally gets subtitles. On top of bronze, silver, and gold, we’re adding a role classification for every table in the flow: this one cleanses, this one joins, this one deduplicates, this one masters. The medal tells you how refined a table is; the role tells you what it does and why it exists. Your lakehouse becomes legible: to auditors, to new engineers, and to the agents themselves, which use those roles to reason about the pipeline the way a human architect would.

Two Tuesdays

Here’s the difference in the shape of a single day.

Before. A product manager asks for churn by customer segment. The request becomes a ticket. An engineer spends Tuesday morning figuring out which of four customer tables is the least wrong, finds the segment definitions in a wiki page last edited fourteen months ago, pings the analyst who wrote it, waits. Wednesday is cleansing. The following week is modeling and a meeting about what “churned” means. The product ships in five weeks, undocumented, and the catalog never hears about it.

After. The same request goes to the planning agent in plain language on Tuesday morning. The agent finds the sources through the domain’s live profile, asks two clarifying questions (should churn follow the glossary definition, and at what grain?) and drafts the spec. The quality gate runs, the model builds against the approved domain design, the tests pass, and by afternoon a human is looking at one approval screen: intent, generated logic, test results, quality findings, side by side. One decision. The product deploys with column-grain lineage and catalog entries already written, because they were written by the build.

The engineer, in this second Tuesday, has not been replaced. They’ve been promoted: from digging through the sediment of past decisions to making the decisions that actually need a human. Is this definition right, is this design sound, does this ship.

What stays yours

I want to be precise about the boundary, because this is where trust is won or lost. The agents do the work: profiling, cleansing, generation, testing, matching, documentation. Your team keeps the judgment: definitions of business terms, survivorship policy, design approval, and the final call before anything reaches production. Three hard gates enforce that split on every build — quality before code, tests before progress, one human approval before deploy. Nothing golden gets minted without a person deciding it deserves the name.

The road from landed to golden used to be measured in quarters and walked by hand. It’s now measured in sessions, and walked by agents that leave a paved, documented, governed path behind them.

The data landing was never the finish line. It was the starting gun.

MDM sessions are in beta today. If you’d like to see your own data walk this road, talk to us.