All stacks · Quality & modeling

Data Quality

Contracts, Great Expectations, Soda, dbt tests, and freshness SLAs.

Data Quality

Aug 24, 20265 min read

Data Quality: Fail the Job, Open a Ticket, or Ignore Red Forever

Four checks — freshness, volume, schema, uniqueness — plus a written fail-versus-ticket policy, why permanently red tests train people to ignore them, and dbt vs GX vs a vendor.

data-quality
observability

Related reading

Illustrated overview of Data Contracts for Pipeline Teams: Fail the Producer, Not Just the Test

Python

Aug 30, 202610 min read

Data Contracts for Pipeline Teams: Fail the Producer, Not Just the Test

Schema plus SLAs as a contract owned by the producer — fail the write path when the grain or freshness breaks, instead of discovering it in a warehouse test after the fact.

data-quality
python

Illustrated overview of Using LLMs Inside Data Pipelines: 4 Patterns That Actually Work in Production

AI & GenAI

Aug 30, 20268 min read

Using LLMs Inside Data Pipelines: 4 Patterns That Actually Work in Production

Beyond chatbots: how data engineers use LLMs for entity extraction, data classification, documentation generation, and pipeline triage — with cost controls.

ai
llm
python

Python

Aug 16, 20269 min read

Data Quality Checks in Python: Catch Bad Data Before Your Users Do

A practical framework for adding data quality checks to Python pipelines: what to check, where to check it, and when to fail loudly vs quarantine.

python
data-quality
tutorials

Illustrated overview of A dbt Testing Strategy That Catches Regressions, Not 4,000 Warnings

dbt

Apr 9, 20268 min read

A dbt Testing Strategy That Catches Regressions, Not 4,000 Warnings

How I structure dbt tests in production: generic tests on keys, unit tests on logic, dbt-expectations on shape, severity that means something.

dbt
quality

About Data Quality

Data quality is not a tool — it is a contract: freshness, volume, schema, and grain tests that fail the pipeline or open a ticket with an owner. Great Expectations, Soda, dbt tests, and Monte Carlo-class observability all sit on that contract.

We already write about data contracts and Python checks. This hub is the quality layer: when to block, when to warn, and why 400 redundant tests train people to ignore red.

What you'll learn here

Contracts at the producer vs tests at the warehouse
Freshness, volume, schema, and uniqueness — the four that matter first
Great Expectations / Soda vs dbt tests vs warehouse monitors
Fail the job vs ticket vs quarantine table

Frequently asked questions

Are dbt tests enough?

They are enough for grain and relationships on models you own. They do not watch source freshness or silent volume drops unless you add jobs for that. Pair dbt tests with freshness SLAs and a small set of anomaly monitors.

Should quality failures fail the pipeline?

Yes for grain, uniqueness, and PII contract breaks. No for a soft anomaly you do not understand yet — that should ticket, not hide yesterday’s gold. Write the policy down or every team will pick a different one.

Do I need Monte Carlo / a warehouse observability vendor?

If you have hundreds of tables and no owners, a vendor can find silent breakage faster than another YAML file. If you have twenty marts and good dbt CI, start with tests and freshness. Buy when the miss rate is a business incident.

New Data Quality posts, straight to your inbox

One email a week with our latest tutorials. No spam.

Newsletter signup is not live yet. Use the contact form if you want to be notified.