DataLane weekly digest

Week of Nov 2, 2026

Mon, 02 Nov 2026 00:00:00 GMT

Revised entries

AWS Redshift Cost Optimization Guide: Serverless RPUs and WLM: Corrected the Redshift Serverless minimum base capacity from 8 RPUs to 4 RPUs.
dbt Core vs dbt Cloud: What You Are Actually Buying, and When to Stop Building It Yourself: Noted that dbt Labs renamed dbt Cloud to the dbt platform in May 2025.
Flink vs Spark Structured Streaming: State, Latency, and the Team You Actually Have: Credited the improved Spark stateful API to Spark 4.0 transformWithState rather than Spark 3.5.
Docker Images: Pin Digests or Watch :latest Ship a Different Binary: Removed the ZooKeeper option from the local Kafka example; Kafka 4.0 is KRaft-only.
Kafka Fundamentals: Topics, Partitions, and Consumer Groups Explained: Dropped ZooKeeper mode from the local setup; Kafka 4.0 runs only in KRaft mode.
Declarative Pipelines in Databricks: Expectations, Streaming Tables, and When the Abstraction Fights You: Updated the product name to Lakeflow pipelines and noted the recommended pyspark.pipelines (dp) module.
Airflow 3 for Pipeline Authors: What Actually Changed and What to Fix First: Noted that Airflow 2 reached end of life on April 22, 2026.
Redshift Serverless vs Snowflake in 2026: Architecture, Concurrency, Cost Model, and Migration Friction: Corrected the Redshift Serverless minimum base capacity from 8 RPUs to 4 RPUs.
Snowflake Time Travel: How It Works, What It Costs, and When It Is Not a Backup: Snowflake Time Travel and Fail-safe in practice: retention, UNDROP, cloning at a timestamp, why storage follows churn, and a restore runbook you can test.
Snowflake Cost Optimization: 7 Techniques That Actually Cut the Bill: Warehouse sizing, auto-suspend, clustering, ACCOUNT_USAGE queries, and resource monitors — estimate the bill from usage before you change prod.
Iceberg vs Delta Lake in 2026: Metadata Design, Catalog Options, and How to Actually Choose: The feature lists have converged. What still differs is metadata structure, catalog architecture, and engine support, and those decide which format fits.

Week of Sep 21, 2026

Mon, 21 Sep 2026 00:00:00 GMT

New entries

AWS Redshift Cost Optimization Guide: Serverless RPUs and WLM: How to cut your AWS Redshift bill by optimizing Serverless base capacity, managing RPUs, leveraging RA3 instances, and tuning Workload Management (WLM).
Cloud Data Warehouse Cost Comparison 2026: Snowflake, Databricks, BigQuery, and Redshift: A comprehensive TCO guide: pricing equations, real-world workload cost models, and how to use our interactive data stack cost estimator.
Databricks vs Google BigQuery: Lakehouse vs Serverless Warehouse: A pragmatic comparison: Databricks for Lakehouse/Spark/ML vs BigQuery for Serverless/SQL-native analytical workflows on GCP.

Week of Aug 31, 2026

Mon, 31 Aug 2026 00:00:00 GMT

New entries

Airbyte vs Fivetran: Connector Coverage, the Bill You Cannot Predict, and Who Fixes It at 2 a.m.: The ELT decision as it plays out: MAR pricing versus infrastructure you operate, what happens when a connector breaks, and when to write a custom extractor.
ClickHouse vs Snowflake: Serving Latency, Concurrency, and Two Very Different Bills: Where a real-time OLAP engine beats a cloud warehouse and where it loses: latency at high concurrency, ingest patterns, joins at scale, and governance gaps.
dbt Core vs dbt Cloud: What You Are Actually Buying, and When to Stop Building It Yourself: The honest split between dbt Core and dbt Cloud: what Cloud adds over CI plus an orchestrator, what seats cost, and when self-hosting stops being cheaper.
Flink vs Spark Structured Streaming: State, Latency, and the Team You Actually Have: Flink vs Spark Structured Streaming in production: how each holds state, their real latency floors, and why your existing batch stack usually decides it.
Kafka vs Pulsar: Storage Architecture, Multi-Tenancy, and Why Most Teams Should Stay: The architecture differences that change operations: segment storage vs partition-as-log, tiered offload, tenant isolation, and when leaving Kafka is a mistake.
Parquet vs ORC vs Avro: Pick by Access Pattern, Not by Benchmark: What each file format is built for: columnar analytics, Hive-native storage, and row-oriented streaming, plus the compression and small-file realities.
polars vs pandas for Data Pipelines: Where the Speedup Is Real and Where It Is Not: A decision guide, not a benchmark: what the expression API and lazy engine buy you, which pandas jobs will not get faster, and the gaps that remain.
MLOps for Data Engineers: Feature Stores, Training Pipelines, and Where You Fit: MLOps through a data engineering lens: what feature stores actually solve, why training pipelines are just DAGs, and the skills that transfer.

Revised entries

ClickHouse MergeTree: ORDER BY Is the Index, Tiny Inserts Are the Outage: ORDER BY is the only index that matters, one-row inserts explode parts, mutations rewrite data, and ClickHouse is not a warehouse stand-in. Batch the writes.
Dagster Assets: Partitions, Checks, and Secrets That Do Not Live in Ops: Software-defined assets vs Airflow tasks, how partitions change backfills, asset checks versus dbt tests, and why warehouse credentials belong in resources.
Fabric OneLake and Capacity: Shortcuts First, Then Stop Starving Spark: OneLake shortcuts are not a copy job, capacity units are shared, and a Power BI refresh can queue a Spark notebook. Fabric is not rebranded Synapse.
Polars LazyFrames: The Speedup Is the Plan, Not the Syntax: Eager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.
Prefect Deployments: Flows, Work Pools, and the Green Run With Zero Rows: A flow is Python. A deployment is what actually runs. Work pools are where. Prefect will call an empty extract a success unless you raise.
NiFi Backpressure and Provenance: Unbounded Queues, Disk Fill, and When Kafka Connect Wins: FlowFiles pile up when a downstream processor stalls and provenance has its own disk. Set queue limits, watch both repositories, and skip NiFi as a Kafka bus.
Scala Datasets: Read the Hot Path Even If You Ship PySpark: A typed Dataset[T] job caught a column I had been silently dropping in PySpark. You still have to read Scala; you do not have to write every job in it.
Trino Federation: Query the Lake Without Melting Prod Postgres: Trino stores nothing. A federated SELECT * on prod Postgres is a database incident. Use Iceberg for the lake, and keep interactive SQL off OLTP.
Flink Event Time and Watermarks: Checkpoints, Savepoints, and the Sink Key You Still Need: Processing time lies after a restart and watermarks drop late events on purpose. Checkpoints recover, savepoints migrate, Kafka EOS does not upsert Postgres.
Hudi COW vs MOR: Incremental Queries Need Compaction, Not Another Format: Copy-on-write versus merge-on-read, incremental pulls versus snapshots, compaction as a job you schedule, and why one table gets one writer and one format.
Postgres: EXPLAIN ANALYZE, VACUUM, and Why BI Must Leave the Primary: A BI scan froze autovacuum, the replication slot kept WAL, and the disk hit 91%. Read the plan, manage bloat, and keep CDC from filling the volume.
Airbyte in Production: A Successful Sync of Zero Rows Is Still an Outage: Green sync, empty destination, raw JSON versus typed silver, self-host versus Cloud, and why Airbyte does not replace dbt.
Beam Runners and Dataflow Cost: Portability vs Workers That Never Scale to Zero: Write-once Beam is real. The Dataflow bill is vCPU-hours plus Shuffle plus Streaming Engine, and Spark is cheaper if the team already writes Spark.
MongoDB to Silver: Flatten Nested Arrays Before They Double Revenue: A nested `items` array hit gold twice and finance saw 1.9x revenue. Land the document raw, explode once in silver, and do not pretend Mongo is the warehouse.
Cassandra: A New Query Is a New Table, Not a New Index Hope: ALLOW FILTERING timed out the coordinator and a quick report full-scanned the cluster. Model the query first and keep Cassandra off the warehouse path.
Fivetran MAR: The Invoice That Doubled, and the Credits You Still Pay: Monthly active rows surprise you, warehouse writes have their own bill, a custom extractor can beat a connector, and Fivetran still does not replace dbt.
Hadoop Hive to Iceberg: Why Lift-and-Shift to S3 Fails and How Banks Actually Exit: NameNode small-file pauses, the Hive metastore as the real lock-in, and why distcp-plus-EMR keeps the same tables. Iceberg is the exit, not a new cluster.
Data Quality: Fail the Job, Open a Ticket, or Ignore Red Forever: Four checks (freshness, volume, schema, uniqueness), a written fail-versus-ticket policy, why always-red tests get ignored, and dbt vs GX vs a vendor.
Pulsar Multi-Tenant Messaging: Brokers, BookKeeper, and When Not to Leave Kafka: Brokers are stateless. BookKeeper holds the log. Pulsar wins on tenant isolation and geo. A healthy Kafka cluster is not a fashion problem.
Redis Features: Cache the Online Path, Never the Source of Truth: We trained on warehouse snapshots and served from Redis. After TTL eviction and a stampede the model looked broken. Online features are a cache.
Elasticsearch: Sync Documents, Stop Treating Aggregations as Facts: Dynamic mapping stored amount as text and a terms agg became the executive number. Idempotent ids, a reindex fallback, and warehouse math in the warehouse.
SQLMesh Plans and Virtual Environments: What dbt State Cannot Isolate, and When to Stay: A SQLMesh plan is a snapshot diff you apply and virtual environments share unchanged tables. dbt defer is CI, not isolation, and you still need an orchestrator.
Terraform for Data Platforms: Modules for Warehouses, Not Click-Ops Grants: Click-ops Snowflake grants will not exist in the next account. Modules for warehouse plus IAM, secrets out of state, and Terraform for anything that lasts.
Kubernetes Operators for Data Jobs: Unbounded Pods, Then the Cluster Dies: Airflow and Spark can spawn unbounded pods, requests and limits decide eviction, and MWAA or Databricks is sometimes the right way to not run K8s.
Redshift Sort Keys, WLM, and Spectrum: RA3 Tuning and the Concurrency Scaling Bill: COMPOUND sort keys and DISTKEY still decide scans on RA3, VACUUM and ANALYZE are not optional, and Spectrum SELECT * is an S3 bill.
Semantic Layer: One Revenue Definition, Not Three Tableau Workbooks: Q2 revenue had three values depending on the workbook. LookML, MetricFlow, or a Power BI dataset can hold the metric, but they do not replace dbt.
dbt Tutorial: Build Your First Transformation Project the Right Way: Set up a dbt project from scratch: staging models, marts, tests, and documentation — with the project structure conventions used by real teams.

Week of Aug 24, 2026

Mon, 24 Aug 2026 00:00:00 GMT

New entries

Airflow Deferrable Operators: Stop Burning Worker Slots on Sensors: How the triggerer frees workers: why classic sensors hold a slot, FileSensor vs deferrable waits, and the habits that keep deferred tasks from sitting forever.
ClickHouse MergeTree: ORDER BY Is the Index, Tiny Inserts Are the Outage: ORDER BY is the only index that matters, one-row inserts explode parts, mutations rewrite data, and ClickHouse is not a warehouse stand-in. Batch the writes.
Dagster Assets: Partitions, Checks, and Secrets That Do Not Live in Ops: Software-defined assets vs Airflow tasks, how partitions change backfills, asset checks versus dbt tests, and why warehouse credentials belong in resources.
Data Contracts for Pipeline Teams: Fail the Producer, Not Just the Test: Schema plus SLAs as a producer-owned contract: fail the write path when grain or freshness breaks, instead of finding out in a warehouse test afterwards.
dbt Incremental Models in Production: unique_key, Merge, and Late Arrivals: How incremental dbt models actually write: unique_key as merge grain, is_incremental filters, lookback windows, and late arrivals that silently drop rows.
Docker Images: Pin Digests or Watch :latest Ship a Different Binary: A green CI build on python:3.11-slim:latest became a Saturday outage when prod pulled a new digest. Pin the bytes and promote the same image.
Using LLMs Inside Data Pipelines: 4 Patterns That Actually Work in Production: Beyond chatbots: how data engineers use LLMs for entity extraction, data classification, documentation generation, and pipeline triage — with cost controls.
MCP for Data Engineers: A Tool Bus, Not a Warehouse Login: Model Context Protocol as a tool bus for warehouse and pipeline context — read-only roles, no DROP, and prompt injection that hides in table comments.
Snowflake Dynamic Tables: TARGET_LAG, Refresh Warehouses, and When to Keep dbt: How Dynamic Tables actually refresh: TARGET_LAG, the warehouse that pays for it, streams vs DT, and why I still keep dbt for gold.
Snowflake Micro-Partitions and Pruning: Why Your Filter Still Scans the Table: What a micropartition is, how clustering and sargable filters prune, why wrapping columns and SELECT * blow the scan, and how to read partitions_scanned.
Snowflake Streams and Tasks: Incremental Apply, Schedules, and When Airflow Still Owns the Graph: CREATE STREAM, consume it in a Task MERGE, handle stale offsets, and know when Airflow should still orchestrate dbt and everything outside Snowflake.
Snowflake Zero-Copy Clones: Storage After Writes, Time Travel, and CI Schemas: CREATE … CLONE shares micropartitions until someone writes. How storage grows, cloning at a timestamp, clones in CI, and why gold is never mutated via a clone.
The AI-Assisted Data Engineer: A Practical Daily Workflow: How to actually use AI coding assistants for pipeline work: SQL generation, dbt refactoring, debugging, and docs, plus the failure modes to watch for.
Fabric OneLake and Capacity: Shortcuts First, Then Stop Starving Spark: OneLake shortcuts are not a copy job, capacity units are shared, and a Power BI refresh can queue a Spark notebook. Fabric is not rebranded Synapse.
Polars LazyFrames: The Speedup Is the Plan, Not the Syntax: Eager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.
Prefect Deployments: Flows, Work Pools, and the Green Run With Zero Rows: A flow is Python. A deployment is what actually runs. Work pools are where. Prefect will call an empty extract a success unless you raise.
Vector Databases Compared: pgvector, Pinecone, Chroma, and When You Need None: An honest comparison of vector storage options for AI workloads — and why the right answer is often the database you already run.
NiFi Backpressure and Provenance: Unbounded Queues, Disk Fill, and When Kafka Connect Wins: FlowFiles pile up when a downstream processor stalls and provenance has its own disk. Set queue limits, watch both repositories, and skip NiFi as a Kafka bus.
Scala Datasets: Read the Hot Path Even If You Ship PySpark: A typed Dataset[T] job caught a column I had been silently dropping in PySpark. You still have to read Scala; you do not have to write every job in it.
Trino Federation: Query the Lake Without Melting Prod Postgres: Trino stores nothing. A federated SELECT * on prod Postgres is a database incident. Use Iceberg for the lake, and keep interactive SQL off OLTP.
Flink Event Time and Watermarks: Checkpoints, Savepoints, and the Sink Key You Still Need: Processing time lies after a restart and watermarks drop late events on purpose. Checkpoints recover, savepoints migrate, Kafka EOS does not upsert Postgres.
Hudi COW vs MOR: Incremental Queries Need Compaction, Not Another Format: Copy-on-write versus merge-on-read, incremental pulls versus snapshots, compaction as a job you schedule, and why one table gets one writer and one format.
Postgres: EXPLAIN ANALYZE, VACUUM, and Why BI Must Leave the Primary: A BI scan froze autovacuum, the replication slot kept WAL, and the disk hit 91%. Read the plan, manage bloat, and keep CDC from filling the volume.
RAG Pipelines for Data Engineers: You Already Know How to Build This: Retrieval-augmented generation is an ETL problem wearing an AI costume. How to build a production RAG ingestion pipeline with the skills you already have.
Airbyte in Production: A Successful Sync of Zero Rows Is Still an Outage: Green sync, empty destination, raw JSON versus typed silver, self-host versus Cloud, and why Airbyte does not replace dbt.
Azure Data Engineering in 2026: Data Factory, Synapse, and Where Fabric Fits: A practical guide to the Azure data platform: when to use Data Factory, what happened to Synapse, and how Microsoft Fabric changes the picture.
Beam Runners and Dataflow Cost: Portability vs Workers That Never Scale to Zero: Write-once Beam is real. The Dataflow bill is vCPU-hours plus Shuffle plus Streaming Engine, and Spark is cheaper if the team already writes Spark.
MongoDB to Silver: Flatten Nested Arrays Before They Double Revenue: A nested `items` array hit gold twice and finance saw 1.9x revenue. Land the document raw, explode once in silver, and do not pretend Mongo is the warehouse.
The 5 Best Data Engineering Courses in 2026 (Honest Review): We compared the most popular data engineering courses and certificates on price, depth, and job-readiness. Here are the five actually worth your money.
Cassandra: A New Query Is a New Table, Not a New Index Hope: ALLOW FILTERING timed out the coordinator and a quick report full-scanned the cluster. Model the query first and keep Cassandra off the warehouse path.
Fivetran MAR: The Invoice That Doubled, and the Credits You Still Pay: Monthly active rows surprise you, warehouse writes have their own bill, a custom extractor can beat a connector, and Fivetran still does not replace dbt.
Hadoop Hive to Iceberg: Why Lift-and-Shift to S3 Fails and How Banks Actually Exit: NameNode small-file pauses, the Hive metastore as the real lock-in, and why distcp-plus-EMR keeps the same tables. Iceberg is the exit, not a new cluster.
The AWS Data Engineering Stack: Which Service Does What (and What to Skip): S3, Glue, Kinesis, Redshift, Athena, EMR, Lambda — a map of the AWS data services, what each is actually for, and the combinations that work in practice.
Data Quality: Fail the Job, Open a Ticket, or Ignore Red Forever: Four checks (freshness, volume, schema, uniqueness), a written fail-versus-ticket policy, why always-red tests get ignored, and dbt vs GX vs a vendor.
Pulsar Multi-Tenant Messaging: Brokers, BookKeeper, and When Not to Leave Kafka: Brokers are stateless. BookKeeper holds the log. Pulsar wins on tenant isolation and geo. A healthy Kafka cluster is not a fashion problem.
Redis Features: Cache the Online Path, Never the Source of Truth: We trained on warehouse snapshots and served from Redis. After TTL eviction and a stampede the model looked broken. Online features are a cache.

Revised entries

The AI-Assisted Data Engineer: A Practical Daily Workflow: How to actually use AI coding assistants for pipeline work: SQL generation, dbt refactoring, debugging, and docs, plus the failure modes to watch for.
Vector Databases Compared: pgvector, Pinecone, Chroma, and When You Need None: An honest comparison of vector storage options for AI workloads — and why the right answer is often the database you already run.
RAG Pipelines for Data Engineers: You Already Know How to Build This: Retrieval-augmented generation is an ETL problem wearing an AI costume. How to build a production RAG ingestion pipeline with the skills you already have.
Azure Data Engineering in 2026: Data Factory, Synapse, and Where Fabric Fits: A practical guide to the Azure data platform: when to use Data Factory, what happened to Synapse, and how Microsoft Fabric changes the picture.
The 5 Best Data Engineering Courses in 2026 (Honest Review): We compared the most popular data engineering courses and certificates on price, depth, and job-readiness. Here are the five actually worth your money.
The AWS Data Engineering Stack: Which Service Does What (and What to Skip): S3, Glue, Kinesis, Redshift, Athena, EMR, Lambda — a map of the AWS data services, what each is actually for, and the combinations that work in practice.
Snowflake Cortex AI for Data Engineers: SQL Functions, Not Another Chatbot: How to use Cortex LLM functions inside pipelines — COMPLETE, CLASSIFY, EMBED — with cost controls, schema checks, retries, and a warehouse-native RAG shape.
Airflow vs Dagster vs Prefect in 2026: Which Orchestrator Should You Pick?: An honest comparison of the three major Python orchestrators: where each one shines, where each one hurts, and a simple decision rule.
Building Your First Data Pipeline with Python and Airflow: A beginner-friendly guide to your first ETL pipeline: extract from an API, transform with pandas, and load into a database, scheduled with Apache Airflow.
Snowflake-Managed Iceberg Tables: Open Storage without Losing Warehouse SQL: When to use Snowflake-managed Iceberg vs native tables: external volumes, catalogs, deletion vectors, and the interoperability jobs Iceberg actually wins.
SQL Window Functions: The 5 Patterns Every Data Engineer Uses Weekly: Deduplication, running totals, sessionization, period-over-period comparisons, and top-N per group — plus frames, pitfalls, and live playground queries.
Snowflake vs Databricks in 2026: Pick the Workload, Not the Logo: An honest head-to-head: SQL warehousing vs Spark lakehouse, Iceberg interoperability, Cortex vs Mosaic, and when you actually need both.
Data Quality Checks in Python: Catch Bad Data Before Your Users Do: A practical framework for adding data quality checks to Python pipelines: what to check, where to check it, and when to fail loudly vs quarantine.
Snowflake vs BigQuery: Credits vs Bytes Scanned: The pricing models decide the architecture: warehouse-hours versus on-demand scans, plus multi-cloud vs GCP-native gravity.
DuckDB: The Fastest Way to Build Local Data Pipelines in 2026: Why DuckDB replaced pandas in many pipelines: query Parquet and CSV files directly with SQL, at speeds that embarrass much bigger tools.
Kafka vs Amazon Kinesis: Control vs Less Ops: When to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
BigQuery Partitioning and Clustering: Stop Paying for Full Table Scans: BigQuery charges by bytes scanned. Learn how partitioning and clustering cut scan costs by 90%+, and the query patterns that silently bypass them.
Databricks and Delta Lake: A Practical Introduction to the Lakehouse: What the lakehouse actually is, how Delta Lake adds ACID transactions to cheap object storage, and the medallion architecture in practice.
PySpark for Data Engineers: From Zero to Your First Production Job: Learn the PySpark DataFrame API, understand lazy evaluation and partitions, and avoid the classic mistakes that make Spark jobs slow.

Week of Aug 17, 2026

Mon, 17 Aug 2026 00:00:00 GMT

New entries

Elasticsearch: Sync Documents, Stop Treating Aggregations as Facts: Dynamic mapping stored amount as text and a terms agg became the executive number. Idempotent ids, a reindex fallback, and warehouse math in the warehouse.
Snowflake Cortex AI for Data Engineers: SQL Functions, Not Another Chatbot: How to use Cortex LLM functions inside pipelines — COMPLETE, CLASSIFY, EMBED — with cost controls, schema checks, retries, and a warehouse-native RAG shape.
SQLMesh Plans and Virtual Environments: What dbt State Cannot Isolate, and When to Stay: A SQLMesh plan is a snapshot diff you apply and virtual environments share unchanged tables. dbt defer is CI, not isolation, and you still need an orchestrator.
Terraform for Data Platforms: Modules for Warehouses, Not Click-Ops Grants: Click-ops Snowflake grants will not exist in the next account. Modules for warehouse plus IAM, secrets out of state, and Terraform for anything that lasts.
Airflow vs Dagster vs Prefect in 2026: Which Orchestrator Should You Pick?: An honest comparison of the three major Python orchestrators: where each one shines, where each one hurts, and a simple decision rule.
Kubernetes Operators for Data Jobs: Unbounded Pods, Then the Cluster Dies: Airflow and Spark can spawn unbounded pods, requests and limits decide eviction, and MWAA or Databricks is sometimes the right way to not run K8s.
Redshift Sort Keys, WLM, and Spectrum: RA3 Tuning and the Concurrency Scaling Bill: COMPOUND sort keys and DISTKEY still decide scans on RA3, VACUUM and ANALYZE are not optional, and Spectrum SELECT * is an S3 bill.
Semantic Layer: One Revenue Definition, Not Three Tableau Workbooks: Q2 revenue had three values depending on the workbook. LookML, MetricFlow, or a Power BI dataset can hold the metric, but they do not replace dbt.
Snowflake Time Travel: How It Works, What It Costs, and When It Is Not a Backup: Snowflake Time Travel and Fail-safe in practice: retention, UNDROP, cloning at a timestamp, why storage follows churn, and a restore runbook you can test.
Building Your First Data Pipeline with Python and Airflow: A beginner-friendly guide to your first ETL pipeline: extract from an API, transform with pandas, and load into a database, scheduled with Apache Airflow.
Snowflake-Managed Iceberg Tables: Open Storage without Losing Warehouse SQL: When to use Snowflake-managed Iceberg vs native tables: external volumes, catalogs, deletion vectors, and the interoperability jobs Iceberg actually wins.
SQL Window Functions: The 5 Patterns Every Data Engineer Uses Weekly: Deduplication, running totals, sessionization, period-over-period comparisons, and top-N per group — plus frames, pitfalls, and live playground queries.
Snowflake vs Databricks in 2026: Pick the Workload, Not the Logo: An honest head-to-head: SQL warehousing vs Spark lakehouse, Iceberg interoperability, Cortex vs Mosaic, and when you actually need both.

Week of Aug 10, 2026

Mon, 10 Aug 2026 00:00:00 GMT

New entries

Data Quality Checks in Python: Catch Bad Data Before Your Users Do: A practical framework for adding data quality checks to Python pipelines: what to check, where to check it, and when to fail loudly vs quarantine.
Snowflake vs BigQuery: Credits vs Bytes Scanned: The pricing models decide the architecture: warehouse-hours versus on-demand scans, plus multi-cloud vs GCP-native gravity.
DuckDB: The Fastest Way to Build Local Data Pipelines in 2026: Why DuckDB replaced pandas in many pipelines: query Parquet and CSV files directly with SQL, at speeds that embarrass much bigger tools.
Kafka vs Amazon Kinesis: Control vs Less Ops: When to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
BigQuery Partitioning and Clustering: Stop Paying for Full Table Scans: BigQuery charges by bytes scanned. Learn how partitioning and clustering cut scan costs by 90%+, and the query patterns that silently bypass them.
Databricks and Delta Lake: A Practical Introduction to the Lakehouse: What the lakehouse actually is, how Delta Lake adds ACID transactions to cheap object storage, and the medallion architecture in practice.

Week of Aug 3, 2026

Mon, 03 Aug 2026 00:00:00 GMT

New entries

Snowflake Cost Optimization: 7 Techniques That Actually Cut the Bill: Warehouse sizing, auto-suspend, clustering, ACCOUNT_USAGE queries, and resource monitors — estimate the bill from usage before you change prod.
dbt Tutorial: Build Your First Transformation Project the Right Way: Set up a dbt project from scratch: staging models, marts, tests, and documentation — with the project structure conventions used by real teams.
Kafka Fundamentals: Topics, Partitions, and Consumer Groups Explained: The mental model every data engineer needs before touching Kafka: how topics, partitions, offsets, and consumer groups actually work together.

Week of Jul 27, 2026

Mon, 27 Jul 2026 00:00:00 GMT

New entries

PySpark for Data Engineers: From Zero to Your First Production Job: Learn the PySpark DataFrame API, understand lazy evaluation and partitions, and avoid the classic mistakes that make Spark jobs slow.

Week of May 18, 2026

Mon, 18 May 2026 00:00:00 GMT

New entries

Testing Airflow Locally: A Setup That Catches Bugs Before Production Does: A four-layer Airflow testing setup: parse checks, DAG structure tests, task unit tests with mocked hooks, and dag.test() runs against local infrastructure.
Cutting Memory in Python Data Jobs: Dtypes, Chunking, Arrow, and Streaming: Where the memory actually goes in a pandas job, how much dtype selection and Arrow-backed strings really save, and when to stop loading and start streaming.
Snowpipe vs Snowpipe Streaming: Latency, Cost Per Row, and Which One You Need: Snowpipe file ingestion vs Snowpipe Streaming: real latency ranges, the credit math per million rows, and how to pick without over-engineering.
Airflow Dynamic Task Mapping: Fan-Out That Works, and Fan-Out That Melts the Scheduler: Practical patterns for expand, expand_kwargs, and partial in Airflow: size and batch a fan-out safely without overwhelming the scheduler and metadata DB.
Iceberg vs Delta Lake in 2026: Metadata Design, Catalog Options, and How to Actually Choose: The feature lists have converged. What still differs is metadata structure, catalog architecture, and engine support, and those decide which format fits.
Testing Data Pipelines in Python: Fixtures, Warehouse Fakes, and Property-Based Transforms: Layered testing for Python data pipelines: pure transform tests, DuckDB as a warehouse fake, property-based invariants, and what runs in CI vs production.
Slowly Changing Dimensions: Types 1 Through 6, and the Two You Will Actually Use: Working SQL for every SCD type, the point-in-time join most Type 2 builds get wrong, and an honest account of which types are worth the maintenance.
Snowflake Masking and Row Access Policies: Tag-Based Governance That Scales: Moving from per-column masking to tag-based masking and mapping-table row access policies, with the performance traps and audit queries that prove coverage.
Migrating from Synapse to Microsoft Fabric: What Maps Cleanly, What Does Not, and How to Sequence It: A component-by-component map from Synapse to Fabric, the T-SQL and workload management gaps that bite, and the order I move things in to keep risk low.

Week of May 11, 2026

Mon, 11 May 2026 00:00:00 GMT

New entries

Debezium on Postgres: Replication Slots, Snapshots, and Applying Changes Idempotently: Postgres CDC with Debezium in production: how the replication slot fills your disk, snapshotting a large table without locks, and replay-safe change apply.
Airflow Datasets and Assets: Replacing Cron Guesswork With Data-Aware Scheduling: Move cross-team DAG dependencies from padded cron offsets to dataset triggers, with conditional scheduling, naming rules, and the failure modes to expect.
Declarative Pipelines in Databricks: Expectations, Streaming Tables, and When the Abstraction Fights You: What DLT genuinely buys you, how expectations and streaming tables behave in production, and the four situations where I still write plain Spark jobs instead.
Dataflow vs Dataproc: The Beam Model, the Cost Profiles, and Which One Your Team Can Actually Run: Dataflow vs Dataproc on GCP in production: what Beam buys you, how the two billing models differ, and the team traits that decide the choice.
ADF Pipeline Patterns That Scale: Parameterized Datasets, Metadata-Driven Ingestion, and Integration Runtime Sizing: How to build one Data Factory pipeline that ingests 300 tables instead of 300 pipelines, and how to size integration runtimes so the bill and the SLA both work.
Pandas to Polars: What Translates, What Does Not, and What Breaks Quietly: A production migration guide: the expression API, lazy frames and query optimization, null versus NaN semantics, and pandas patterns with no polars twin.
Snowflake Data Sharing: Secure Shares, Reader Accounts, and the Marketplace Without Shipping Copies: How Snowflake secure data sharing works across accounts, regions, and clouds, when to use a reader account, and governance that keeps a share from leaking.
Star Schema vs One Big Table: What Actually Costs Money on a Columnar Warehouse: Joins are not the expense they were in 2015. What a wide denormalized table really costs in storage, rebuild time, and history, and the hybrid I ship instead.
Airflow 3 for Pipeline Authors: What Actually Changed and What to Fix First: The Airflow 3 changes that break real DAGs: the Task SDK boundary, optional logical_date, removed context keys, DAG versioning, and an upgrade checklist.
Kafka Connect in Production: Converters, Dead Letter Queues, and Worker Sizing: The Connect settings that decide whether connectors survive real data: converter choice, error tolerance with a DLQ, SMT limits, and sizing workers.
Cutting a Databricks Bill: Job Clusters, Spot, Photon, and Finding the Top Offenders in System Tables: Taking 30 percent off a Databricks bill in a week: drop all-purpose clusters for jobs, price spot properly, test Photon per workload, rank system.billing.
DuckDB vs Spark on One Node: Where the Crossover Actually Happens: Real timings and costs comparing single-node DuckDB against a Spark cluster, and why the crossover point is decided by shuffle volume rather than raw data size.
Snowpark vs SQL: When Python Wins and When It Is Just a Slower Way to Write SQL: Snowpark Python vs plain SQL in Snowflake: lazy DataFrames, UDF serialization costs, and the four workloads where Python genuinely earns its place.
Six Ways to Deduplicate a Table: What Each Costs and Which Row Survives: DISTINCT, GROUP BY, ROW_NUMBER, QUALIFY, MERGE, and stream-side dedup compared on cost and semantics, plus how to pick the survivor row deterministically.
dbt Performance at Scale: Threads, Materializations, and Per-Model Cost Attribution: Cutting dbt run time and warehouse spend on a large project: find the real critical path, see when threads stop helping, and attribute cost per model.
Batch LLM Enrichment: Idempotency, Cost Per Row, and Knowing When the Output Is Wrong: An LLM call as a batch transformation step: content-hash idempotency, per-row cost budgets, retries for a non-deterministic API, and testing fuzzy output.
Step Functions vs Airflow for Data Orchestration: Cost, Observability, and the Point Where You Outgrow One: State transitions vs scheduler hours, execution history vs task logs, and the signals that tell you which orchestrator your pipeline has outgrown.
Structured Streaming in Production: Triggers, Checkpoints, Watermarks, and Exactly-Once Sinks: What decides whether a Spark stream survives a year in production: trigger choice, checkpoint discipline, watermark math, and what exactly-once guarantees.
Recursive CTEs in Production: Org Charts, Bill of Materials, and Cycle Protection: How WITH RECURSIVE actually executes, patterns for hierarchy traversal and BOM explosion, and the cycle guards that keep a bad edge from running forever.

Week of May 4, 2026

Mon, 04 May 2026 00:00:00 GMT

New entries

Where Lambda Belongs in a Data Pipeline and Where It Quietly Becomes a Distributed System You Cannot Debug: Lambda is excellent glue and a terrible framework. Here are the limits that shape every design, the idempotency you must build yourself, and the point to stop.
DuckDB in Production: The Jobs It Genuinely Wins and the Single-Node Limits You Will Hit: Where DuckDB belongs in a real pipeline, how far out-of-core execution gets you on one machine, and the concurrency and memory walls that decide when to stop.
Kafka Schema Registry: Evolving Event Schemas Without Breaking Consumers: Avro versus Protobuf, what each compatibility mode really allows, and the CI gate that stops a breaking schema change before it reaches a topic.
dbt Macros and Jinja: The Patterns That Earn Their Keep, and the Clever Ones That Do Not: Which Jinja patterns earn their indirection in a production dbt project, how to test macros, and the metaprogramming that makes a project unmaintainable.
Pivot and Unpivot in SQL: Patterns That Survive a Changing Column Set: Conditional aggregation, native PIVOT syntax, unpivoting wide tables, and how to handle dynamic categories without generating SQL at runtime.
Vector Search in the Warehouse: When You Do Not Need a Vector Database: Native vector search in Snowflake, BigQuery, and pgvector is now good enough for most RAG workloads. Thresholds, real costs, and when a dedicated store wins.
Redshift Serverless vs Snowflake in 2026: Architecture, Concurrency, Cost Model, and Migration Friction: Redshift Serverless vs Snowflake in production: how RPU-seconds differ from credits, where each bends under concurrency, and what breaks in a migration.
Snowflake RBAC Design: A Role Hierarchy That Survives an Audit and a Reorg: Split functional roles from access roles, use future grants and managed access schemas, and build a Snowflake permission model that stays reviewable.
Spark Executor Memory: Spill, GC, and the OOM Errors That Are Really Partition-Sizing Errors: How Spark executor memory is divided, why most OOMs are one oversized partition rather than a small cluster, and six failures you can read from the error.
How BigQuery Pruning Actually Works: Partition Metadata, Block Statistics, and the Shapes That Defeat Both: What BigQuery reads before your data: partition metadata, Capacitor block statistics, and the query shapes that silently turn a pruned scan into a full one.
dbt Snapshots for SCD Type 2: Check vs Timestamp, and When to Hand-Roll: Choosing check or timestamp snapshot strategies, what happens when a snapshot run is missed, and when a hand-rolled SCD2 model beats dbt snapshot.
Kafka Exactly-Once Semantics: What Transactions Actually Guarantee: Idempotent producers, transactions, and read_committed give exactly-once inside Kafka. Write to a database and you still need an idempotent sink.
Snowflake Warehouse Sizing: The Spill and Queue Signals That Tell You Up or Out: Pick a Snowflake warehouse size from evidence, not intuition, using spilling, queueing, and scaling tests to choose a bigger warehouse or more clusters.
Spark Data Skew: Diagnosing It Properly, Then Fixing It With Salting, AQE, and Broadcast: One task at 40 minutes while 399 finished in 20 seconds is skew. Here is how I find the offending key, and the four fixes ranked by how much they cost me.
CTE vs Subquery: When the WITH Clause Is a Materialization Barrier: What actually happens to a CTE in Postgres, Snowflake, and BigQuery, when it blocks predicate pushdown, and when referencing it twice costs you two scans.
Text-to-SQL in Production: Why the Demo Works and the Deployment Fails: The failure modes that appear after launch (join paths, metric drift, runaway cost) and the semantic layer, constraints, and evals that make text-to-SQL work.
Making Athena Fast and Cheap: Partition Projection, Parquet Layout, CTAS, and Workgroup Byte Limits: Athena bills for bytes scanned. The partition projection, Parquet layout, CTAS compaction, and workgroup guardrails I use to cut that number by 90 percent.
Cutting a BigQuery Bill: Partitioning, Byte Limits, Materialized Views, and On-Demand vs Editions: Taking a runaway BigQuery invoice apart: find the top jobs, fix pruning, cap bytes billed, use materialized views honestly, and know when Editions wins.
dbt Project Structure: A Layout That Survives Two Years and Forty Models: How I lay out staging, intermediate, and marts so a dbt project still makes sense after two years: naming rules, ref boundaries, and folders that rot.
Kafka Consumer Group Rebalancing: Why Your Consumers Stall and How to Stop It: Rebalances are the most common cause of Kafka consumer lag spikes. The protocol, the four timeouts that matter, and the cooperative sticky config I ship.
Spark Shuffle Explained: What It Costs, How to Read It in the UI, and the Rewrites That Remove It: A shuffle is a disk write, a network fetch, and a sort you did not ask for. What it costs, how to spot it in the Spark UI, and the rewrites that remove it.
Window Frames Deep Dive: RANGE vs ROWS and the Patterns That Replace Self-Joins: How window frames actually evaluate, why RANGE and ROWS give different answers on ties, and six window patterns that delete self-joins from your models.

Week of Apr 20, 2026

Mon, 20 Apr 2026 00:00:00 GMT

New entries

Cortex Analyst and Cortex Search: Shipping Chat With Your Data That Answers Correctly: How Cortex Analyst semantic models and Cortex Search hybrid retrieval fit together, what the YAML actually controls, and where the accuracy comes from.
Kinesis Data Streams vs Firehose: Shards, On-Demand, and Landing Streams in S3: When Kinesis Data Streams earns its shards, when Firehose is enough, and the buffering and partitioning patterns that keep S3 landings queryable.
Joins and the Fan-Out Bug: How Revenue Doubles Without an Error: The join fan-out failure class: grain checks before and after every join, EXISTS for semi-joins, anti-joins done right, and the postmortem pattern.
Slim CI in dbt: state:modified+ and --defer on a 1,000-Model Project: How to get dbt CI from 50 minutes to 4 with state comparison and defer: manifest artifacts, GitHub Actions wiring, and the edge cases that bite.
Unity Catalog in Practice: Metastore Layout, Grants That Scale, and Leaving hive_metastore: How I structure Unity Catalog for real teams: three-level namespace design, group-based grants, lineage that works, and a hive_metastore migration that ships.
Snowflake Search Optimization Service: Point Lookups Without the Full Scan: How the search access path works, what it costs to build and maintain, which predicates it serves, and when clustering keys are the better buy.