How We Build Data Programs

Many stalled data programs are waiting on ownership, definitions, validation, or coordination rather than on engineering. What follows are conclusions we have reached from building data warehouses since 1999, and how those conclusions shape the way we design the work. For the shorter account of what an engagement feels like from the client side, see working with Cloud Data →

Every responsibility needs a named owner

A data program needs ownership of priorities, business definitions, implementation, validation, and platform operation. One person often covers several of these, and we fill several of them ourselves. What matters is that each one is named.

If the gap is senior ownership—the person responsible for keeping the architecture aligned with the business—that is the role our fractional head of data engagement is designed to fill.

When a responsibility is unassigned, the work often continues on undocumented assumptions and deferred decisions. Those gaps surface later as unowned feeds, unresolved definitions, or validation delays. Identifying and filling the gap early is considerably less expensive than discovering it during validation. If it is not caught there, incorrect or incomplete data can reach users and erode trust in the platform.

See how these responsibilities divide during an engagement →

The first build is one high-value slice, at production standard

We start with a single subject area and build it end to end. Choosing the starting point belongs to the client; our role is to help structure the decision.

Together, we inventory the candidate subject areas or review the inventory the client already maintains. For each one, we consider expected return, risk, the state of the underlying data, and whether the people who know that part of the business have time to participate. A structured session or two with each department is usually enough to build that inventory, including areas that may never have been prioritized for analytics before.

Starting with one subject area also gives the team something concrete to evaluate. Requirements written entirely in the abstract tend to describe the report someone already has. Requirements developed in response to a working model are better able to reflect what the business actually needs.

A deliberately small first build surfaces important assumptions while corrections are still contained and inexpensive, before other models and reports depend on them. It also tests the delivery process end to end. The platform configuration, connectors and ingestion, transformation models, deployment automation, and BI layer all carry real data to real users. The practices that make later phases faster are proven on the first production slice.

The first subject area has a bounded scope, but it is built to the architecture, testing, security, and documentation standards intended for the larger program. It becomes the first production component rather than a prototype that needs to be rebuilt before anyone can rely on it.

The lasting asset is your data in a governed model, not the reports

A report answers a fixed set of questions. The lasting asset is your data in a governed model: integrated across source systems, standardized, and defined once. That model holds the business logic that reports, applications, and one-off analysis can reuse.

Calculations that need to mean the same thing everywhere—revenue, margin, customer status, comparable-period logic, and similar measures—belong in that shared model rather than inside individual reporting tools.

The practical effect is that a definition is built and reconciled once instead of in every dashboard that uses it. New reports come together faster because the modeling decisions behind them are already made, and a change to a shared definition can be made in one place.

The first subject area also begins to establish the reusable structure of the model. Conformed dimensions such as customer, product, and date can be shared by the fact tables introduced in later subject areas. That reuse is a major reason the second subject area can move faster than the first. For teams that use a Kimball bus matrix, this is the first row being filled in.

When business logic remains inside individual dashboards, each new report creates another implementation to maintain. Small differences accumulate, and reconciling those implementations becomes harder as the reporting environment grows. Different answers can begin appearing in reporting before the discrepancy itself is obvious.

Reconciliation, validation, and remediation are three different activities

These activities require different evidence, different owners, and different scope decisions. Treating them as one creates ambiguity about when a model is complete and who is responsible for problems found along the way.

Reconciliation establishes that data moved and transformed as intended. Records arrived, counts agree, and transformations ran as written. This is ours to own and is largely automatable.

Validation establishes that the result represents the business concept it is meant to represent. It begins with an agreed definition; without one, there is nothing to validate against.

A pipeline can run cleanly while the resulting measure still represents the wrong business concept. Successful execution does not establish business correctness. That requires someone who knows the business, which is why a measure is not considered validated until one of the client’s experts has tied a period of history back to the system of record and confirmed that the results agree.

Remediation is the correction of problems that reconciliation or validation finds in the source. Looking closely at data that has historically been used one report at a time often surfaces problems that were not visible before. Correcting those problems may require different owners, systems, and budget.

When data is incorrect in a source system, we can identify the problem, measure its effect, and apply an agreed treatment in the model where appropriate. Changing the source itself requires someone on the client side with the authority to do so. We document the problem and its impact, then scope any required remediation separately so that the original phase remains bounded.

Initial validation establishes confidence at a point in time. Automated monitoring and controlled deployment practices are then needed to preserve that confidence as sources and definitions change.

That includes checks on source availability, schema changes, volume, and the business relationships the model depends on. Changes to shared definitions also move through a controlled deployment process rather than being edited into individual dashboards.

Governance happens while the models are built

Definitions, naming, and ownership determine whether two reports using the same measure agree. When that work is deferred until after the platform is built, inconsistencies may already be encoded into the implementation and become more expensive to unwind.

Governance work is frequently deferred because it appears less urgent than visible delivery. In practice, it has a large influence on whether people trust and continue using the resulting platform.

For an important shared measure, practical governance records what it means, the grain at which it is calculated, what it includes and excludes, the source fields behind it, who owns and approved the definition, its known limitations, and when it took effect.

The core delivery team can capture those decisions as the model is built; it does not require a separate committee. These are also the questions people ask about a number six months later, and recording the answers when the decisions are made is much easier than reconstructing them afterward.

We can surface inconsistencies, recommend a definition, explain the downstream effect of each option, and document the decision. Final authority over what a measure means stays inside the client organization because the definition represents a business decision, not simply a technical one.

We architect, design, and build for total cost of ownership

Implementation is one component of a data platform’s total cost of ownership. Maintenance, change, staff time, and software costs continue for as long as the platform is in use. The warehouse, connectors, transformation tooling, and BI layer all create ongoing operating costs.

We consider those costs when making design decisions rather than evaluating only the effort required for the initial build.

That also means recommending against work when the expected upkeep is likely to exceed its value. A measure that has no clear use, does not fit the source system it would depend on, or requires substantial maintenance for marginal benefit creates an ongoing cost for the team that inherits it.

Platform spend belongs in the same calculation. Poorly tuned workloads create both performance and cost problems. On Snowflake, increasing warehouse size can improve runtime, but it also increases the rate of credit consumption. We tune workloads and monitor spend as part of operating the platform rather than relying on additional compute as the default response.

Tool choices require a similar tradeoff. A managed connector may cost more in software fees but substantially less in engineering time than a custom pipeline. We evaluate both sides of that equation.

We also evaluate legacy logic rule by rule rather than carrying it forward or discarding it wholesale. Some of what sits in a long-lived spreadsheet represents an intentional business rule that needs to be preserved. Other logic accumulated as a workaround for a limitation that no longer exists.

The first belongs in the new model deliberately. The second should not be reproduced simply because it is present in the old implementation. In some cases, the sponsor makes that distinction directly: preserve the approved measure, but do not recreate every exception that accumulated around it.

Rebuilding large bodies of legacy logic line by line is often slower than building the new model and reconciling it against the old reporting. The reconciliation provides evidence that the new model produces the expected result without automatically importing every historical implementation choice.

Technology choices follow the same reasoning. We consider the client team’s skills, the volume and latency the work actually requires, governance and security needs, integration complexity, operating cost, and who is expected to run the result. We evaluate those factors before settling on the stack so the platform reflects the client’s requirements and operating model rather than a default implementation pattern.

The platform also needs to be understandable by people other than the team that built it. Routine operation should not depend on knowledge held only by us. That means modular models another qualified engineer can understand and extend, tests that document expected behavior, and documentation written alongside the implementation rather than assembled at the end.

Training follows the same schedule as implementation. If it is deferred until the end, it becomes a separate handoff activity competing with the client team’s existing work. When training happens throughout the build, analysts can begin using each subject area as it is validated and take on more responsibility while we are still available to support them.

A governed model is what makes an AI answer checkable

Giving an AI assistant access to a warehouse does not resolve differences already present in the data. If revenue is defined differently across several tables or reports, the assistant needs an explicit basis for determining which definition applies. Without approved metadata or governed query logic, it can return a technically valid answer using a definition that does not match what the user intended.

Here, governance has both an engineering and a business dimension.

From an engineering perspective, the model is code. It is versioned in source control and changed through review, testing, and controlled deployment, so the logic that produced an answer can be identified.

From a business perspective, the model carries definitions approved by the people who own them and resolves those definitions against data that has been validated back to the system of record.

Both matter. Version control alone does not establish that a business definition is correct, and an approved definition that is never encoded into the model cannot control how the system behaves.

The model also records the grain of each measure, its inclusion and exclusion rules, its approved definition and owner, and its lineage back to source data. The same definitions that drive dashboards and recurring reports should also support conversational interfaces. Whether that interface is delivered through Snowflake, Omni, or another analytics platform, a shared business measure should resolve through the same underlying rules.

Governance is necessary, but it does not complete an AI implementation. Permissions, approved query paths, response labeling, evaluation against expected behavior, handling for questions the model was not built to answer, and ongoing monitoring still have to be designed. The governed model provides the standard those controls can evaluate generated answers against.

This is an area we are continuing to develop rather than one we consider settled. Natural-language access can expose unresolved definitions more quickly because a broader group of people can ask questions directly. It does not resolve those differences. Business owners still approve what a measure means, and generated answers remain identified as generated while their logic or validation is still provisional.

AI-assisted delivery follows the same engineering controls

We use AI coding assistants for repeatable implementation work, including drafting transformation code, tests, documentation, and deployment changes. These tools can reduce the time spent on mechanical development work and help us put a working model in front of users sooner.

The controls around that work remain the same. AI-assisted changes go through the same review, testing, and deployment process as manually written changes. Generated documentation is also checked against what was actually implemented.

Architecture, grain, business definitions, and validation remain with practitioners and the client’s experts. AI helps us iterate faster within the same engineering and governance process.

Who this approach serves

We work this way for organizations building a data capability intended to outlast the engagement: a governed model, definitions the business has approved, and a team prepared to keep the platform answering new questions as the business changes.

See what this looks like during an engagement →