Reference

Data engineering glossary

52 terms in plain English — the vocabulary of pipelines, warehouses, and the AI stack.

ACID: Atomicity, Consistency, Isolation, Durability — the transaction guarantees that make concurrent writes safe. Delta Lake and Iceberg bring these to data lakes.
Airbyte: Open-source EL tool with a large connector catalog. The usual alternative when Fivetran’s price is the constraint.
Apache Flink: True stream processor with event-at-a-time state. Choose it over micro-batch Spark when you need sub-second, complex event time logic.
Apache Hudi: Lakehouse table format optimized for upserts and incremental pulls. Often compared with Iceberg and Delta when CDC lands on object storage.
Apache Iceberg: Open table format for analytic datasets: ACID, snapshots, hidden partitioning. The interoperability format Snowflake, Spark, Trino, and Athena increasingly share.
Backfill: Re-running a pipeline for historical dates, typically after fixing a bug or adding a new column. Idempotent tasks make backfills safe.
Batch processing: Processing data in scheduled chunks (hourly, daily) rather than continuously. Simpler to operate than streaming; right answer for most workloads.
CAP Theorem: A distributed store can fully guarantee only two of consistency, availability, and partition tolerance. Explains warehouse vs operational DB trade-offs.
CDC (Change Data Capture): Streaming inserts, updates, and deletes from a database's transaction log into downstream systems, instead of repeatedly copying full tables.
Columnar storage: Storing data by column instead of by row (Parquet, warehouse formats). Analytics read few columns of many rows, so this is dramatically faster and more compressible.
Data Catalog: Searchable inventory of datasets, owners, and definitions. Collibra, DataHub, Atlan, and warehouse-native catalogs all play this role.
Data contract: An explicit, versioned agreement about a dataset's schema, semantics, and SLAs between producers and consumers — schema changes become negotiations, not surprises.
Data Fabric: Architecture that uses metadata and (often) AI to discover, integrate, and serve data across silos without always physically copying it.
Data lake: Cheap object storage (S3, ADLS, GCS) holding raw files in open formats. Flexible and inexpensive, but without a table format it lacks transactions and governance.
Data lineage: The graph of where data comes from and what depends on it. Essential for impact analysis: "if I change this column, what breaks?"
Data mart: A curated, business-facing set of tables (often star schemas) serving one domain — the "gold layer" consumers actually query.
Data Mesh: Organizational model: domain teams own analytical data products, with federated governance instead of one central warehouse team owning every table.
Data Observability: Monitoring freshness, volume, schema, distribution, and lineage so you learn about bad data from a dashboard — not from a VP.
Data Vault: Modeling style (hubs, links, satellites) built for auditability and source-agnostic history. Heavier than a star schema; used in large regulated warehouses.
dbt: The standard tool for SQL transformation: models are SELECT statements, with dependency management, testing, and documentation built in.
Dead letter queue: A holding area for messages/rows that failed processing, so one poison record does not block the pipeline and nothing is silently dropped.
Delta Lake: Databricks-origin table format adding ACID and time travel on object storage. Default on Databricks; UniForm can expose Iceberg metadata.
Dimensional modeling: Organizing data into fact tables (events, measures) and dimension tables (who/what/where) — the star schema that BI tools expect.
ELT vs ETL: ETL transforms before loading; ELT loads raw data first and transforms inside the warehouse. Cheap warehouse compute made ELT the modern default.
Embedding: A vector of floats representing meaning; similar content produces nearby vectors. The foundation of semantic search and RAG.
Feature store: A system that computes ML features once and serves them consistently to both training and inference, with point-in-time correctness.
Fivetran: Managed EL: high connector quality, higher price, low maintenance. The “it just syncs” end of the ingest spectrum.
Grain: What one row represents in a table ("one row per order per day"). Getting the grain explicit and consistent is half of data modeling.
Great Expectations: Python data-quality framework: you declare expectations, run them in pipelines, and publish data docs. Heavier than a few asserts; worth it at scale.
Idempotency: Running the same task twice produces the same result. The single most important property of a reliable pipeline — enables retries and backfills.
Incremental model: A transformation that processes only new/changed rows instead of rebuilding the whole table, trading complexity for speed and cost.
Infrastructure as Code: Warehouses, IAM, and networks defined in Terraform/Pulumi so environments are reviewable and repeatable — not click-ops.
Lakehouse: Data lake storage plus warehouse guarantees, via table formats like Delta Lake and Iceberg — one copy of data, many engines.
Medallion architecture: Bronze (raw) → Silver (cleaned) → Gold (business-ready) layering. Each layer is rebuildable from the one below.
OLAP: Online analytical processing — column-oriented, scan-heavy queries over historical facts. Warehouses and DuckDB are OLAP; OLTP is the app database.
OneLake: Microsoft Fabric’s single logical lake. Shortcuts can point at ADLS or other clouds so Fabric items do not always copy data.
Orchestrator: The system that runs pipelines on schedule, manages dependencies and retries, and alerts on failure (Airflow, Dagster, Prefect).
Partitioning: Physically splitting a table by a column (usually date) so queries scan only relevant chunks. The #1 lever for warehouse cost.
Point-in-time join: Joining data as it existed at the event's time, not as it is now. Prevents leaking future information into ML training data.
RAG (Retrieval-Augmented Generation): Retrieving relevant documents (usually via vector search) and injecting them into an LLM prompt, grounding answers in your data.
Reverse ETL: Pushing warehouse-curated data back into operational SaaS (CRM, support) so the business acts on the same metrics you publish.
SCD (Slowly Changing Dimension): Strategies for dimensions that change over time. Type 1 overwrites; Type 2 keeps history with validity ranges — the common choice.
Schema drift: Upstream schema changes (new/renamed/retyped columns) arriving unannounced. Handled with contracts, schema tests, and tolerant ingestion.
Shuffle: Redistributing rows between workers so matching keys colocate (joins, groupBy). The expensive part of distributed processing.
Snowpark: Snowflake’s DataFrame/UDF runtime (Python, Java, Scala) that executes inside the warehouse instead of pulling data out to a Spark cluster.
Star Schema: A fact table surrounded by dimensions. The BI-friendly shape most semantic layers and Looker/Power BI models expect.
Streaming: Processing events continuously within seconds of arrival (Kafka, Flink, Spark Structured Streaming). Powerful and operationally expensive.
Unity Catalog: Databricks’ governance layer: identities, lineage, and a three-level namespace across workspaces. The control plane above Delta tables.
Vector database: A store optimized for k-nearest-neighbor search over embeddings (pgvector, Pinecone, Chroma) — the retrieval half of RAG.
Watermark: In streaming, how long to wait for late events before finalizing a window — the trade-off between completeness and latency.
XCom: Airflow's mechanism for passing small values between tasks. Fine for metadata; never for actual datasets.
Zero-copy clone: Instantly duplicating a table/database via metadata pointers instead of copying data (Snowflake, Delta). Perfect for testing against production-shaped data.