127 illustrated guides. Search or filter by stack.Write for us →
Latest
The MLOps landscape explained through a data engineering lens: what feature stores actually solve, why training pipelines are just DAGs, and the skills that transfer.
Read the guide →MLOpsThe MLOps landscape explained through a data engineering lens: what feature stores actually solve, why training pipelines are just DAGs, and the skills that transfer.
AirflowHow the triggerer frees workers: why classic sensors sleep on a slot, FileSensor vs deferrable waits, and the operational habits that keep deferred tasks from sitting forever.
ClickHouseORDER BY is the only index that matters, one-row inserts explode parts, mutations rewrite data, and ClickHouse is not a warehouse stand-in. Batch the writes.
DagsterSoftware-defined assets vs Airflow tasks, how partitions change backfills, asset checks versus dbt tests, and why warehouse credentials belong in resources.
PythonSchema plus SLAs as a contract owned by the producer — fail the write path when the grain or freshness breaks, instead of discovering it in a warehouse test after the fact.
dbtHow incremental dbt models actually write: unique_key as merge grain, is_incremental filters, lookback windows, and the late-arrival cases that silently drop rows.
DockerA Friday CI green on python:3.11-slim:latest became a Saturday outage when prod pulled a new digest. Pin the bytes, promote the same image, and stop treating Compose as Kafka.
AI & GenAIBeyond chatbots: how data engineers use LLMs for entity extraction, data classification, documentation generation, and pipeline triage — with cost controls.
AI & GenAIModel Context Protocol as a tool bus for warehouse and pipeline context — read-only roles, no DROP, and prompt injection that hides in table comments.
SnowflakeHow Dynamic Tables actually refresh: TARGET_LAG, the warehouse that pays for it, streams vs DT, and why I still keep dbt for gold.
SnowflakeWhat a micropartition is, how clustering and sargable filters prune, why wrapping columns and SELECT * blow the scan, and how to read partitions_scanned.
SnowflakeCREATE STREAM on a table, consume it in a Task MERGE, SHOW STREAMS, stale offsets, and the cases where Airflow still orchestrates dbt and everything outside Snowflake.
SnowflakeCREATE … CLONE shares micropartitions until someone writes. How storage grows, how to clone at a timestamp, how I use clones in CI, and why I never mutate gold through a clone.
ProductivityHow to actually use AI coding assistants for pipeline work — SQL generation, dbt refactoring, debugging, and documentation — plus the failure modes to watch for.
Microsoft FabricOneLake shortcuts are not a copy job, capacity units are shared, and a Power BI refresh can queue a Spark notebook. Fabric is not rebranded Synapse.
PolarsEager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.
PrefectA flow is Python. A deployment is what actually runs. Work pools are where. Prefect will call an empty extract a success unless you raise.
AI & GenAIAn honest comparison of vector storage options for AI workloads — and why the right answer is often the database you already run.
Apache NiFiFlowFiles pile up when a downstream processor stalls. Provenance has its own disk. Set queue limits, watch both repositories, and do not use NiFi as a Kafka bus.
ScalaA typed Dataset[T] job caught a column I had been silently dropping in PySpark. You still have to read Scala; you do not have to write every job in it.
ToolsA template sponsored post showing how to feature a partner product: a realistic case study structure with a clear sponsored label.
TrinoTrino stores nothing. A federated SELECT * on prod Postgres is a database incident. Use Iceberg for the lake, and keep interactive SQL off OLTP.
Apache FlinkProcessing time lies after a restart. Watermarks drop late events on purpose. Checkpoints recover; savepoints migrate. Exactly-once to Kafka does not upsert Postgres.
Apache HudiCopy-on-write versus merge-on-read, incremental pulls versus snapshots, compaction as a job you schedule, and why one table gets one writer and one format.
PostgreSQLA BI scan on the orders primary froze autovacuum, the replication slot kept WAL, and the disk hit 91%. Read the plan, manage bloat, and keep CDC from filling the volume.
AI & GenAIRetrieval-augmented generation is an ETL problem wearing an AI costume. How to build a production RAG ingestion pipeline with the skills you already have.
AirbyteGreen sync, empty destination, raw JSON versus typed silver, self-host versus Cloud, and why Airbyte does not replace dbt.
AzureA practical guide to the Azure data platform: when to use Data Factory, what happened to Synapse, and how Microsoft Fabric changes the picture.
Apache BeamWrite-once Beam is real. The Dataflow bill is vCPU-hours plus Shuffle plus Streaming Engine. Streaming jobs keep a worker all night. Spark is cheaper if the team already writes Spark.
MongoDBA nested `items` array hit gold twice and finance saw 1.9x revenue. Land the document raw, explode once in silver, and do not pretend Mongo is the warehouse.
CareerWe compared the most popular data engineering courses and certificates on price, depth, and job-readiness. Here are the five actually worth your money.
CassandraALLOW FILTERING timed out the coordinator and a "quick report" full-scanned the serving cluster. Model the query first, extract without a table scan, and keep Cassandra off the warehouse path.
FivetranMonthly active rows surprise you, warehouse writes have their own bill, a custom extractor can beat a connector, and Fivetran still does not replace dbt.
HadoopNameNode small-file pauses, the Hive metastore as the real lock-in, and why distcp-plus-EMR keeps the same tables. Iceberg is the exit, not a new cluster.
AWSS3, Glue, Kinesis, Redshift, Athena, EMR, Lambda — a map of the AWS data services, what each is actually for, and the combinations that work in practice.
Data QualityFour checks — freshness, volume, schema, uniqueness — plus a written fail-versus-ticket policy, why permanently red tests train people to ignore them, and dbt vs GX vs a vendor.
Apache PulsarBrokers are stateless. BookKeeper holds the log. Pulsar wins on tenant isolation and geo. A healthy Kafka cluster is not a fashion problem.
RedisWe trained on warehouse snapshots and served from Redis. TTL eviction and a stampede later, the model looked broken. Online features are a cache, not a fact table.
ElasticsearchDynamic mapping stored amount as text, a terms agg became the executive number, and nobody could replay the index. Idempotent ids, a reindex fallback, and leave warehouse math in the warehouse.
SnowflakeHow to use Cortex LLM functions inside pipelines — COMPLETE, CLASSIFY, EMBED — with cost controls, schema checks, retries, and a warehouse-native RAG shape.
SQLMeshA plan is a snapshot diff you apply. Virtual environments share unchanged physical tables. dbt defer is CI, not isolation. Do not migrate a healthy dbt estate. You still need an orchestrator.
TerraformClick-ops Snowflake grants will not exist in the next account. Modules for warehouse plus IAM, secrets out of state, UI to explore, Terraform for anything that must last.
ToolsAn honest comparison of the three major Python orchestrators: where each one shines, where each one hurts, and a simple decision rule.
KubernetesAirflow and Spark can spawn unbounded pods, requests and limits are the difference between busy and evicted, crash loops hide in operators, and MWAA or Databricks is sometimes the right way to not run K8s.
Amazon RedshiftCOMPOUND sort keys and DISTKEY still decide scans on RA3. VACUUM and ANALYZE are not optional. Spectrum SELECT * is an S3 bill. Concurrency scaling will surprise you.
BI & semantic layerQ2 revenue was $14.2M, $13.8M, and $15.1M depending on the workbook. LookML, MetricFlow, or a Power BI dataset can hold the metric — they do not replace dbt or hide extract lag.
SnowflakeA practical guide to Snowflake Time Travel and Fail-safe: retention, UNDROP, cloning at a timestamp, storage cost, and a restore runbook you can test.
AirflowExtract from an API, transform with pandas, load to a warehouse — scheduled in Airflow with retries, idempotency, and a local Docker setup.
SnowflakeWhen to use Snowflake-managed Iceberg vs native tables: external volumes, catalogs, deletion vectors, and the interoperability jobs Iceberg actually wins.
SQLDeduplication, running totals, sessionization, period-over-period comparisons, and top-N per group — plus frames, pitfalls, and live playground queries.
ToolsAn honest head-to-head: SQL warehousing vs Spark lakehouse, Iceberg interoperability, Cortex vs Mosaic, and when you actually need both.
PythonA practical framework for adding data quality checks to Python pipelines: what to check, where to check it, and when to fail loudly vs quarantine.
ToolsThe pricing models decide the architecture: warehouse-hours versus on-demand scans, plus multi-cloud vs GCP-native gravity.
DuckDBWhy DuckDB replaced pandas in many pipelines: query Parquet and CSV files directly with SQL, at speeds that embarrass much bigger tools.
KafkaWhen to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
BigQueryBigQuery charges by bytes scanned. Learn how partitioning and clustering cut scan costs by 90%+, and the query patterns that silently bypass them.
DatabricksTable-format choice in 2026: deletion vectors, catalogs, and which engine you are willing to lock in.
DatabricksWhat the lakehouse actually is, how Delta Lake adds ACID transactions to cheap object storage, and the medallion architecture in practice.
SnowflakeWarehouse sizing, auto-suspend, clustering, ACCOUNT_USAGE queries, and resource monitors — estimate the bill from usage before you change prod.
dbtSet up a dbt project from scratch: staging models, marts, tests, and documentation — with the project structure conventions used by real teams.
KafkaThe mental model every data engineer needs before touching Kafka: how topics, partitions, offsets, and consumer groups actually work together.
Apache SparkLearn the PySpark DataFrame API, understand lazy evaluation and partitions, and avoid the classic mistakes that make Spark jobs slow.
AirflowA four-layer testing setup for Airflow: parse checks, DAG structure tests, task unit tests with mocked hooks, and dag.test() end-to-end runs against local infrastructure.
PythonWhere the memory actually goes in a pandas job, how much dtype selection and Arrow-backed strings really save, and when to stop loading and start streaming.
SnowflakeA practical comparison of Snowpipe file ingestion and Snowpipe Streaming, covering real latency ranges, the credit math per million rows, and how to pick without over-engineering.
AirflowPractical patterns for expand, expand_kwargs, and partial in Airflow: how to size a fan-out, batch it safely, and avoid the mapped tasks that overwhelm your scheduler and metadata database.
IcebergThe feature lists have converged. What still differs is metadata structure, catalog architecture, and engine support, and those are what decide which format fits your stack.
PythonA layered testing strategy for Python data pipelines: pure transform tests, DuckDB as a warehouse fake, property-based invariants, and what belongs in CI versus production.
Data ModelingWorking SQL for every SCD type, the point-in-time join that most Type 2 implementations get wrong, and an honest account of which types are worth the maintenance.
SnowflakeHow to move from per-column masking policies to tag-based masking and mapping-table row access policies, including the performance traps and the audit queries that prove coverage.
AzureA component-by-component migration map from Synapse to Fabric, the T-SQL and workload management gaps that bite, and the order I move things in to keep risk low.
DebeziumA production walkthrough of Postgres CDC with Debezium: how the replication slot fills your disk, how to snapshot a large table without locking it, and how to apply changes so replays are safe.
AirflowHow to move cross-team DAG dependencies from padded cron offsets to dataset-driven triggers, including conditional scheduling, naming rules, and the failure modes nobody warns you about.
DatabricksWhat DLT genuinely buys you, how expectations and streaming tables behave in production, and the four situations where I still write plain Spark jobs instead.
GCPA production comparison of Dataflow and Dataproc on GCP: what Beam buys you, how the two billing models differ, and the team characteristics that decide the choice.
AzureHow to build one Data Factory pipeline that ingests 300 tables instead of 300 pipelines, and how to size integration runtimes so the bill and the SLA both work.
PythonA production migration guide: the expression API, lazy frames and query optimization, null versus NaN semantics, and the pandas patterns with no polars equivalent.
SnowflakeHow Snowflake secure data sharing works across accounts, regions, and clouds, when to use a reader account, and the governance patterns that keep a share from leaking more than you intended.
Data ModelingJoins are not the expense they were in 2015. Here is what a wide denormalized table really costs in storage, rebuild time, and history, and the hybrid I ship instead of picking a side.
AirflowThe Airflow 3 changes that break real DAGs: the Task SDK boundary, logical_date becoming optional, removed context keys, DAG versioning, and a pre-upgrade checklist.
KafkaThe Connect settings that decide whether your connectors survive contact with real data: converter choice, error tolerance with a DLQ, SMT limits, and how to size and scale workers.
DatabricksHow I take 30 percent off a Databricks bill in a week: kill all-purpose clusters for jobs, price spot properly, test Photon per workload, and rank spend with system.billing.
DuckDBReal timings and costs comparing single-node DuckDB against a Spark cluster, and why the crossover point is decided by shuffle volume rather than raw data size.
SnowflakeA production comparison of Snowpark Python and plain SQL in Snowflake, covering lazy DataFrames, UDF serialization costs, and the four workloads where Python genuinely earns its place.
SQLDISTINCT, GROUP BY, ROW_NUMBER, QUALIFY, MERGE, and stream-side dedup compared on cost and semantics, plus how to pick the survivor row deterministically.
dbtCutting dbt run time and warehouse spend on a large project: how to find the real critical path, when threads stop helping, and how to attribute cost to individual models.
AI & GenAITreating an LLM call as a transformation step in a batch pipeline. Content-hash idempotency, per-row cost budgets, retry semantics for a non-deterministic API, and how to test output you cannot diff.
AWSState transitions versus scheduler hours, execution history versus task logs, and the specific signals that tell you which orchestrator your pipeline has outgrown.
SnowflakeHow Time Travel retention, fail-safe, and zero-copy clones bill against storage, plus the migration and recovery patterns that make the storage cost worth paying.
Apache SparkThe four things that decide whether a Spark stream survives a year in production: trigger choice, checkpoint discipline, watermark math, and what exactly-once really guarantees.
SQLHow WITH RECURSIVE actually executes, patterns for hierarchy traversal and BOM explosion, and the cycle guards that keep a bad edge from running forever.
AWSLambda is excellent glue and a terrible framework. Here are the limits that shape every design, the idempotency you must build yourself, and the point to stop.
DuckDBWhere DuckDB belongs in a real pipeline, how far out-of-core execution gets you on one machine, and the concurrency and memory walls that decide when to stop.
KafkaAvro versus Protobuf, what each compatibility mode really allows, and the CI gate that stops a breaking schema change before it reaches a topic your consumers cannot read.
dbtWhich Jinja patterns are worth the indirection in a production dbt project, how to test macros, and the metaprogramming that quietly makes a project unmaintainable.
SQLConditional aggregation, native PIVOT syntax, unpivoting wide tables, and how to handle dynamic categories without generating SQL at runtime.
AI & GenAINative vector search in Snowflake, BigQuery, and pgvector has quietly become good enough for most RAG workloads. Here are the thresholds, the real costs, and when a dedicated store still wins.
AWSA production comparison of Redshift Serverless and Snowflake: how RPU-seconds differ from credits, where each one bends under concurrency, and what actually breaks in a migration.
SnowflakeHow to split functional roles from access roles, use future grants and managed access schemas, and build a Snowflake permission model that stays reviewable as teams change.
Apache SparkHow Spark executor memory is actually divided, why most OOMs are one oversized partition rather than a small cluster, and the six failures I can now diagnose from the error text alone.
BigQueryWhat BigQuery reads before it reads your data: partition metadata, Capacitor block statistics, and the specific query shapes that silently turn a pruned scan into a full one.
dbtChoosing between check and timestamp snapshot strategies, what happens when a snapshot run is missed, and the cases where a hand-rolled SCD2 model beats dbt snapshot.
KafkaIdempotent producers, transactions, and read_committed give you exactly-once inside Kafka. The moment you write to a database, you still need an idempotent sink. Here is where the boundary sits.
SnowflakeHow to pick a Snowflake warehouse size from evidence instead of intuition, using spilling, queueing, and scaling tests to decide between a bigger warehouse and more clusters.
Apache SparkOne task at 40 minutes while 399 finished in 20 seconds is skew. Here is how I find the offending key, and the four fixes ranked by how much they cost me.
SQLWhat actually happens to a CTE in Postgres, Snowflake, and BigQuery, when it blocks predicate pushdown, and when referencing it twice costs you two scans.
AI & GenAIThe failure modes that only show up after launch — join paths, metric drift, runaway cost — and the semantic layer, constrained generation, and eval harness that make text-to-SQL survivable.
AWSAthena bills you for bytes scanned. Here is the partition projection, Parquet layout, CTAS compaction, and workgroup guardrails I use to cut that number by 90 percent.
BigQueryHow I take a runaway BigQuery invoice apart: find the top jobs, fix pruning, cap bytes billed, use materialized views honestly, and decide when Editions beats on-demand.
dbtHow I lay out staging, intermediate, and marts so a dbt project still makes sense after two years: naming rules, ref boundaries, and the folders that always rot.
KafkaRebalances are the most common cause of Kafka consumer lag spikes. Here is the protocol, the four timeouts that matter, and the cooperative sticky config I ship by default.
SnowflakeA ranked playbook for reducing Snowflake spend: auto-suspend defaults, right-sizing warehouses, resource monitors that actually fire, and the ACCOUNT_USAGE queries that name your top ten offenders.
Apache SparkA shuffle is a disk write, a network fetch, and a sort you did not ask for. Here is what it costs, how to spot it in the Spark UI, and the query rewrites that delete it.
SQLHow window frames actually evaluate, why RANGE and ROWS give different answers on ties, and six window patterns that delete self-joins from your models.
AI & GenAIHow Cortex Analyst semantic models and Cortex Search hybrid retrieval fit together, what the YAML actually controls, and where the accuracy comes from.
AWSWhen Kinesis Data Streams earns its shards, when Firehose is enough, and the buffering and partitioning patterns that keep S3 landings queryable.
SQLThe join fan-out failure class: grain checks before and after every join, EXISTS for semi-joins, anti-joins done right, and the postmortem pattern.
dbtHow to get dbt CI from 50 minutes to 4 with state comparison and defer: manifest artifacts, GitHub Actions wiring, and the edge cases that bite.
DatabricksHow I structure Unity Catalog for real teams: three-level namespace design, group-based grants, lineage that works, and a hive_metastore migration that ships.
SnowflakeHow the search access path works, what it costs to build and maintain, which predicates it serves, and when clustering keys are the better buy.
SnowflakeClustering depth, what automatic reclustering actually costs, and a decision framework for clustering keys vs Search Optimization vs doing nothing.
AI & GenAIWhy raw text-to-SQL fails on real schemas, when retrieval over curated marts wins, and the semantic model and guardrails that make either safe.
AWSGlue vs EMR for production Spark: DPU pricing vs instance pricing, cold starts, library control, and the job size where teams switch.
SQLFive SQL habits that scan more than they should: SELECT *, functions on filter columns, DISTINCT as a bug fix, OR-joins, and per-row UDF calls.
dbtHow I structure dbt tests in production: generic tests on keys, unit tests on logic, dbt-expectations on shape, severity that means something.
DatabricksHow the _delta_log actually works: JSON commits, optimistic concurrency, checkpoints, deletion vectors, and the small-file problem nobody escapes.
SnowflakeHow I read the operator tree: TableScan bytes, partition pruning, local and remote spilling, and the exchange steps that quietly dominate runtime.