Great Expectations & Soda cheat sheet
Expectation suites, checkpoints, SodaCL checks, and where standalone DQ tools beat dbt tests.
Great Expectations core objects
context = gx.get_context(mode="file")- The Data Context is the project root holding datasources, suites, and checkpoints. File mode keeps it in version control.
ds = context.data_sources.add_snowflake(name="wh", connection_string=...)- Datasources describe where data lives. One per warehouse or lake location, reused across suites.
suite = context.suites.add(gx.ExpectationSuite(name="orders"))- A suite is a named collection of expectations — the equivalent of a dbt model's tests block.
suite.add_expectation(gxe.ExpectColumnValuesToNotBeNull(column="order_id"))- Expectations are typed objects in GX 1.x, not dictionaries. Autocomplete works, which is the main upgrade.
Common expectations
ExpectColumnValuesToBeBetween(column="amount", min_value=0, max_value=100000)- Range checks that catch unit errors — cents stored as dollars is the classic case.
ExpectColumnValuesToMatchRegex(column="email", regex=r"^[^@]+@[^@]+$")- Format validation at the boundary. Keep the regex loose; strict email regexes reject valid addresses.
ExpectTableRowCountToBeBetween(min_value=1000)- Volume anomaly detection. A pipeline that loads zero rows usually succeeds silently without this.
ExpectColumnPairValuesAToBeGreaterThanB(column_A="ended_at", column_B="started_at")- Cross-column invariants that single-column tests cannot express.
ExpectCompoundColumnsToBeUnique(column_list=["tenant_id", "external_id"])- Composite key uniqueness — the check that prevents duplicate multi-tenant ingestion.
Checkpoints and results
checkpoint = context.checkpoints.add( gx.Checkpoint(name="nightly", validation_definitions=[vd]) ) result = checkpoint.run()- Checkpoints bundle validation plus actions. This is what your orchestrator calls, not individual expectations.
result.success- Boolean gate for the pipeline. Raise on False so downstream tasks never consume unvalidated data.
actions=[UpdateDataDocsAction(name="docs")]- Regenerates Data Docs, the browsable HTML report. The main reason non-engineers tolerate GX.
SodaCL checks
checks for analytics.silver.orders: - row_count > 0 - missing_count(order_id) = 0 - duplicate_count(order_id) = 0- SodaCL is YAML-first and much terser than GX. Analysts can read and edit it without Python.
- freshness(ordered_at) < 1d- Freshness as a first-class check. Cleaner than the max-timestamp queries teams hand-roll.
- anomaly score for row_count < default- Learns a baseline from history and flags deviations — useful where a fixed threshold is impossible to pick.
- failed rows: fail query: | select * from orders where amount < 0- Custom SQL with failed-row capture for triage.
Running Soda
soda scan -d snowflake_prod -c configuration.yml checks.yml- The single CLI command. Exit code is nonzero on failure, so it drops straight into any orchestrator.
soda test-connection -d snowflake_prod -c configuration.yml- Verify credentials before debugging check logic. Saves an hour of confusion.
Choosing a tool
dbt tests- Right default when data is already in the warehouse and your team owns the models. Zero extra infrastructure.
Great Expectations- Choose it for pre-load validation of files and DataFrames, and when stakeholders want the Data Docs report.
Soda- Choose it when checks must be readable and owned by analysts, and for freshness and anomaly detection out of the box.
Run checks where the data lands- Validate at ingestion and at the gold boundary. Testing only the final mart means bad data has already propagated.
From DataLane — tutorials at/blog, practice SQL live in theplayground.