dbt and the Modern Data Stack: How Data Engineering Changed in Three Years

When dbt (data build tool) appeared in 2016, it solved a specific but pervasive problem: analytics teams needed a software engineering workflow for their SQL transformations, and they didn’t have one.

What dbt Actually Does

dbt transforms raw data (already loaded into your warehouse) using SQL. It defines transformations as models (SELECT statements), handles dependency management between models, runs data quality tests, and generates documentation automatically.

The analytics engineering role — data work applied with software engineering discipline — emerged largely around dbt. Teams that adopted dbt stopped having “single analyst who knows how all the tables work” problems.

The Modern Data Stack

dbt sits in the transformation layer of a broader stack: Fivetran or Airbyte for extraction, Snowflake/BigQuery/Databricks as the warehouse, dbt for transformation, and Looker/Metabase for visualization.

The stack has matured. The main complexity now is governance: who owns what, who can change what, what are the data contracts between producers and consumers.

Where It’s Heading

Semantic layer: dbt’s semantic layer aims to create a single definition of business metrics, preventing the “different dashboards show different revenue numbers” problem.

AI in the data stack: Natural language to SQL tools are layering on top of the modern data stack. The accuracy on simple queries is high; complex multi-join queries remain challenging. The stack is becoming a foundation for AI-powered data products.

Why Testing Discipline Separates Mature dbt Implementations From Immature Ones

The single clearest differentiator between dbt implementations that genuinely improve data reliability and those that simply move the same fragile SQL into a new tool is testing discipline. dbt’s built-in testing framework — uniqueness tests, not-null tests, referential integrity tests, and custom business logic tests — is frequently underutilized, with teams writing models without corresponding tests and discovering data quality issues downstream in dashboards rather than catching them at the source. Mature dbt implementations treat test coverage with the same seriousness software engineering teams treat unit test coverage, often enforcing a minimum testing requirement for any model feeding business-critical reporting before it can be merged into the production data pipeline.

The Documentation Debt That Accumulates Silently

dbt’s automatic documentation generation from model definitions and tests is one of its most valuable but most underutilized features. Teams under delivery pressure frequently skip writing the column-level descriptions and model-level context that make dbt’s generated documentation genuinely useful, resulting in a documentation site that exists technically but provides little real value to anyone trying to understand what a given model represents or how it should be used. This documentation debt compounds over time in ways that are difficult to retroactively fix, since the people who understood the original business context for a model often move to other projects or leave the company before anyone circles back to add the missing documentation.

Orchestration Beyond dbt’s Native Scheduler

While dbt Cloud provides native scheduling for smaller-scale deployments, organizations with complex data pipeline dependencies spanning multiple systems beyond just SQL transformation increasingly orchestrate dbt runs through dedicated workflow orchestration tools like Airflow, Dagster, or Prefect. This becomes necessary once data pipelines need genuine cross-system dependency management — waiting for an upstream extraction job to complete successfully before triggering dbt transformation, then triggering downstream reverse-ETL or reporting refresh only after dbt completes successfully — coordination that dbt’s native scheduling alone doesn’t handle well.

This article is part of our ongoing coverage of Software & SaaS. For related reading, see API design best practices and open source software business models.

The Skills Gap Behind Analytics Engineering Adoption

Organizations adopting dbt and the broader analytics engineering practice frequently underestimate the skills transition required from traditional data analysts, who may have strong SQL skills but limited experience with version control, code review, and software engineering testing discipline. Successful dbt adoption typically requires either hiring specifically for the analytics engineering skill combination or investing meaningfully in upskilling existing analysts, and organizations that adopt the tooling without this parallel skills investment tend to see dbt used as a slightly more organized place to write the same undisciplined SQL that existed before, missing most of the actual reliability and maintainability benefit the tool is capable of delivering.