What changed this week

Feed: weekly digest (RSS)· every entry: RSS

1. 01

   Week of Nov 2, 20260 new · 11 revised

   • RevisedAWS Redshift Cost Optimization Guide: Serverless RPUs and WLMCorrected the Redshift Serverless minimum base capacity from 8 RPUs to 4 RPUs.
   • Reviseddbt Core vs dbt Cloud: What You Are Actually Buying, and When to Stop Building It YourselfNoted that dbt Labs renamed dbt Cloud to the dbt platform in May 2025.
   • RevisedFlink vs Spark Structured Streaming: State, Latency, and the Team You Actually HaveCredited the improved Spark stateful API to Spark 4.0 transformWithState rather than Spark 3.5.
   • RevisedDocker Images: Pin Digests or Watch :latest Ship a Different BinaryRemoved the ZooKeeper option from the local Kafka example; Kafka 4.0 is KRaft-only.
   • RevisedKafka Fundamentals: Topics, Partitions, and Consumer Groups ExplainedDropped ZooKeeper mode from the local setup; Kafka 4.0 runs only in KRaft mode.
   • RevisedDeclarative Pipelines in Databricks: Expectations, Streaming Tables, and When the Abstraction Fights YouUpdated the product name to Lakeflow pipelines and noted the recommended pyspark.pipelines (dp) module.
   • RevisedAirflow 3 for Pipeline Authors: What Actually Changed and What to Fix FirstNoted that Airflow 2 reached end of life on April 22, 2026.
   • RevisedRedshift Serverless vs Snowflake in 2026: Architecture, Concurrency, Cost Model, and Migration FrictionCorrected the Redshift Serverless minimum base capacity from 8 RPUs to 4 RPUs.
   • RevisedSnowflake Time Travel: How It Works, What It Costs, and When It Is Not a BackupSnowflake Time Travel and Fail-safe in practice: retention, UNDROP, cloning at a timestamp, why storage follows churn, and a restore runbook you can test.
   • RevisedSnowflake Cost Optimization: 7 Techniques That Actually Cut the BillWarehouse sizing, auto-suspend, clustering, ACCOUNT_USAGE queries, and resource monitors — estimate the bill from usage before you change prod.
   • RevisedIceberg vs Delta Lake in 2026: Metadata Design, Catalog Options, and How to Actually ChooseThe feature lists have converged. What still differs is metadata structure, catalog architecture, and engine support, and those decide which format fits.
2. 02

   Week of Sep 21, 20263 new · 0 revised

   • NewAWS Redshift Cost Optimization Guide: Serverless RPUs and WLMHow to cut your AWS Redshift bill by optimizing Serverless base capacity, managing RPUs, leveraging RA3 instances, and tuning Workload Management (WLM).
   • NewCloud Data Warehouse Cost Comparison 2026: Snowflake, Databricks, BigQuery, and RedshiftA comprehensive TCO guide: pricing equations, real-world workload cost models, and how to use our interactive data stack cost estimator.
   • NewDatabricks vs Google BigQuery: Lakehouse vs Serverless WarehouseA pragmatic comparison: Databricks for Lakehouse/Spark/ML vs BigQuery for Serverless/SQL-native analytical workflows on GCP.
3. 03

   Week of Aug 31, 20268 new · 27 revised

   • NewAirbyte vs Fivetran: Connector Coverage, the Bill You Cannot Predict, and Who Fixes It at 2 a.m.The ELT decision as it plays out: MAR pricing versus infrastructure you operate, what happens when a connector breaks, and when to write a custom extractor.
   • NewClickHouse vs Snowflake: Serving Latency, Concurrency, and Two Very Different BillsWhere a real-time OLAP engine beats a cloud warehouse and where it loses: latency at high concurrency, ingest patterns, joins at scale, and governance gaps.
   • Newdbt Core vs dbt Cloud: What You Are Actually Buying, and When to Stop Building It YourselfThe honest split between dbt Core and dbt Cloud: what Cloud adds over CI plus an orchestrator, what seats cost, and when self-hosting stops being cheaper.
   • NewFlink vs Spark Structured Streaming: State, Latency, and the Team You Actually HaveFlink vs Spark Structured Streaming in production: how each holds state, their real latency floors, and why your existing batch stack usually decides it.
   • NewKafka vs Pulsar: Storage Architecture, Multi-Tenancy, and Why Most Teams Should StayThe architecture differences that change operations: segment storage vs partition-as-log, tiered offload, tenant isolation, and when leaving Kafka is a mistake.
   • NewParquet vs ORC vs Avro: Pick by Access Pattern, Not by BenchmarkWhat each file format is built for: columnar analytics, Hive-native storage, and row-oriented streaming, plus the compression and small-file realities.
   • Newpolars vs pandas for Data Pipelines: Where the Speedup Is Real and Where It Is NotA decision guide, not a benchmark: what the expression API and lazy engine buy you, which pandas jobs will not get faster, and the gaps that remain.
   • NewMLOps for Data Engineers: Feature Stores, Training Pipelines, and Where You FitMLOps through a data engineering lens: what feature stores actually solve, why training pipelines are just DAGs, and the skills that transfer.
   • RevisedClickHouse MergeTree: ORDER BY Is the Index, Tiny Inserts Are the OutageORDER BY is the only index that matters, one-row inserts explode parts, mutations rewrite data, and ClickHouse is not a warehouse stand-in. Batch the writes.
   • RevisedDagster Assets: Partitions, Checks, and Secrets That Do Not Live in OpsSoftware-defined assets vs Airflow tasks, how partitions change backfills, asset checks versus dbt tests, and why warehouse credentials belong in resources.
   • RevisedFabric OneLake and Capacity: Shortcuts First, Then Stop Starving SparkOneLake shortcuts are not a copy job, capacity units are shared, and a Power BI refresh can queue a Spark notebook. Fabric is not rebranded Synapse.
   • RevisedPolars LazyFrames: The Speedup Is the Plan, Not the SyntaxEager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.
   • RevisedPrefect Deployments: Flows, Work Pools, and the Green Run With Zero RowsA flow is Python. A deployment is what actually runs. Work pools are where. Prefect will call an empty extract a success unless you raise.
   • RevisedNiFi Backpressure and Provenance: Unbounded Queues, Disk Fill, and When Kafka Connect WinsFlowFiles pile up when a downstream processor stalls and provenance has its own disk. Set queue limits, watch both repositories, and skip NiFi as a Kafka bus.
   • RevisedScala Datasets: Read the Hot Path Even If You Ship PySparkA typed Dataset[T] job caught a column I had been silently dropping in PySpark. You still have to read Scala; you do not have to write every job in it.
   • RevisedTrino Federation: Query the Lake Without Melting Prod PostgresTrino stores nothing. A federated SELECT * on prod Postgres is a database incident. Use Iceberg for the lake, and keep interactive SQL off OLTP.
   • RevisedFlink Event Time and Watermarks: Checkpoints, Savepoints, and the Sink Key You Still NeedProcessing time lies after a restart and watermarks drop late events on purpose. Checkpoints recover, savepoints migrate, Kafka EOS does not upsert Postgres.
   • RevisedHudi COW vs MOR: Incremental Queries Need Compaction, Not Another FormatCopy-on-write versus merge-on-read, incremental pulls versus snapshots, compaction as a job you schedule, and why one table gets one writer and one format.
   • RevisedPostgres: EXPLAIN ANALYZE, VACUUM, and Why BI Must Leave the PrimaryA BI scan froze autovacuum, the replication slot kept WAL, and the disk hit 91%. Read the plan, manage bloat, and keep CDC from filling the volume.
   • RevisedAirbyte in Production: A Successful Sync of Zero Rows Is Still an OutageGreen sync, empty destination, raw JSON versus typed silver, self-host versus Cloud, and why Airbyte does not replace dbt.
   • RevisedBeam Runners and Dataflow Cost: Portability vs Workers That Never Scale to ZeroWrite-once Beam is real. The Dataflow bill is vCPU-hours plus Shuffle plus Streaming Engine, and Spark is cheaper if the team already writes Spark.
   • RevisedMongoDB to Silver: Flatten Nested Arrays Before They Double RevenueA nested `items` array hit gold twice and finance saw 1.9x revenue. Land the document raw, explode once in silver, and do not pretend Mongo is the warehouse.
   • RevisedCassandra: A New Query Is a New Table, Not a New Index HopeALLOW FILTERING timed out the coordinator and a quick report full-scanned the cluster. Model the query first and keep Cassandra off the warehouse path.
   • RevisedFivetran MAR: The Invoice That Doubled, and the Credits You Still PayMonthly active rows surprise you, warehouse writes have their own bill, a custom extractor can beat a connector, and Fivetran still does not replace dbt.
   • RevisedHadoop Hive to Iceberg: Why Lift-and-Shift to S3 Fails and How Banks Actually ExitNameNode small-file pauses, the Hive metastore as the real lock-in, and why distcp-plus-EMR keeps the same tables. Iceberg is the exit, not a new cluster.
   • RevisedData Quality: Fail the Job, Open a Ticket, or Ignore Red ForeverFour checks (freshness, volume, schema, uniqueness), a written fail-versus-ticket policy, why always-red tests get ignored, and dbt vs GX vs a vendor.
   • RevisedPulsar Multi-Tenant Messaging: Brokers, BookKeeper, and When Not to Leave KafkaBrokers are stateless. BookKeeper holds the log. Pulsar wins on tenant isolation and geo. A healthy Kafka cluster is not a fashion problem.
   • RevisedRedis Features: Cache the Online Path, Never the Source of TruthWe trained on warehouse snapshots and served from Redis. After TTL eviction and a stampede the model looked broken. Online features are a cache.
   • RevisedElasticsearch: Sync Documents, Stop Treating Aggregations as FactsDynamic mapping stored amount as text and a terms agg became the executive number. Idempotent ids, a reindex fallback, and warehouse math in the warehouse.
   • RevisedSQLMesh Plans and Virtual Environments: What dbt State Cannot Isolate, and When to StayA SQLMesh plan is a snapshot diff you apply and virtual environments share unchanged tables. dbt defer is CI, not isolation, and you still need an orchestrator.
   • RevisedTerraform for Data Platforms: Modules for Warehouses, Not Click-Ops GrantsClick-ops Snowflake grants will not exist in the next account. Modules for warehouse plus IAM, secrets out of state, and Terraform for anything that lasts.
   • RevisedKubernetes Operators for Data Jobs: Unbounded Pods, Then the Cluster DiesAirflow and Spark can spawn unbounded pods, requests and limits decide eviction, and MWAA or Databricks is sometimes the right way to not run K8s.
   • RevisedRedshift Sort Keys, WLM, and Spectrum: RA3 Tuning and the Concurrency Scaling BillCOMPOUND sort keys and DISTKEY still decide scans on RA3, VACUUM and ANALYZE are not optional, and Spectrum SELECT * is an S3 bill.
   • RevisedSemantic Layer: One Revenue Definition, Not Three Tableau WorkbooksQ2 revenue had three values depending on the workbook. LookML, MetricFlow, or a Power BI dataset can hold the metric, but they do not replace dbt.
   • Reviseddbt Tutorial: Build Your First Transformation Project the Right WaySet up a dbt project from scratch: staging models, marts, tests, and documentation — with the project structure conventions used by real teams.
4. 04

   Week of Aug 24, 202636 new · 19 revised

   • NewAirflow Deferrable Operators: Stop Burning Worker Slots on SensorsHow the triggerer frees workers: why classic sensors hold a slot, FileSensor vs deferrable waits, and the habits that keep deferred tasks from sitting forever.
   • NewClickHouse MergeTree: ORDER BY Is the Index, Tiny Inserts Are the OutageORDER BY is the only index that matters, one-row inserts explode parts, mutations rewrite data, and ClickHouse is not a warehouse stand-in. Batch the writes.
   • NewDagster Assets: Partitions, Checks, and Secrets That Do Not Live in OpsSoftware-defined assets vs Airflow tasks, how partitions change backfills, asset checks versus dbt tests, and why warehouse credentials belong in resources.
   • NewData Contracts for Pipeline Teams: Fail the Producer, Not Just the TestSchema plus SLAs as a producer-owned contract: fail the write path when grain or freshness breaks, instead of finding out in a warehouse test afterwards.
   • Newdbt Incremental Models in Production: unique_key, Merge, and Late ArrivalsHow incremental dbt models actually write: unique_key as merge grain, is_incremental filters, lookback windows, and late arrivals that silently drop rows.
   • NewDocker Images: Pin Digests or Watch :latest Ship a Different BinaryA green CI build on python:3.11-slim:latest became a Saturday outage when prod pulled a new digest. Pin the bytes and promote the same image.
   • NewUsing LLMs Inside Data Pipelines: 4 Patterns That Actually Work in ProductionBeyond chatbots: how data engineers use LLMs for entity extraction, data classification, documentation generation, and pipeline triage — with cost controls.
   • NewMCP for Data Engineers: A Tool Bus, Not a Warehouse LoginModel Context Protocol as a tool bus for warehouse and pipeline context — read-only roles, no DROP, and prompt injection that hides in table comments.
   • NewSnowflake Dynamic Tables: TARGET_LAG, Refresh Warehouses, and When to Keep dbtHow Dynamic Tables actually refresh: TARGET_LAG, the warehouse that pays for it, streams vs DT, and why I still keep dbt for gold.
   • NewSnowflake Micro-Partitions and Pruning: Why Your Filter Still Scans the TableWhat a micropartition is, how clustering and sargable filters prune, why wrapping columns and SELECT * blow the scan, and how to read partitions_scanned.
   • NewSnowflake Streams and Tasks: Incremental Apply, Schedules, and When Airflow Still Owns the GraphCREATE STREAM, consume it in a Task MERGE, handle stale offsets, and know when Airflow should still orchestrate dbt and everything outside Snowflake.
   • NewSnowflake Zero-Copy Clones: Storage After Writes, Time Travel, and CI SchemasCREATE … CLONE shares micropartitions until someone writes. How storage grows, cloning at a timestamp, clones in CI, and why gold is never mutated via a clone.
   • NewThe AI-Assisted Data Engineer: A Practical Daily WorkflowHow to actually use AI coding assistants for pipeline work: SQL generation, dbt refactoring, debugging, and docs, plus the failure modes to watch for.
   • NewFabric OneLake and Capacity: Shortcuts First, Then Stop Starving SparkOneLake shortcuts are not a copy job, capacity units are shared, and a Power BI refresh can queue a Spark notebook. Fabric is not rebranded Synapse.
   • NewPolars LazyFrames: The Speedup Is the Plan, Not the SyntaxEager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.
   • NewPrefect Deployments: Flows, Work Pools, and the Green Run With Zero RowsA flow is Python. A deployment is what actually runs. Work pools are where. Prefect will call an empty extract a success unless you raise.
   • NewVector Databases Compared: pgvector, Pinecone, Chroma, and When You Need NoneAn honest comparison of vector storage options for AI workloads — and why the right answer is often the database you already run.
   • NewNiFi Backpressure and Provenance: Unbounded Queues, Disk Fill, and When Kafka Connect WinsFlowFiles pile up when a downstream processor stalls and provenance has its own disk. Set queue limits, watch both repositories, and skip NiFi as a Kafka bus.
   • NewScala Datasets: Read the Hot Path Even If You Ship PySparkA typed Dataset[T] job caught a column I had been silently dropping in PySpark. You still have to read Scala; you do not have to write every job in it.
   • NewTrino Federation: Query the Lake Without Melting Prod PostgresTrino stores nothing. A federated SELECT * on prod Postgres is a database incident. Use Iceberg for the lake, and keep interactive SQL off OLTP.
   • NewFlink Event Time and Watermarks: Checkpoints, Savepoints, and the Sink Key You Still NeedProcessing time lies after a restart and watermarks drop late events on purpose. Checkpoints recover, savepoints migrate, Kafka EOS does not upsert Postgres.
   • NewHudi COW vs MOR: Incremental Queries Need Compaction, Not Another FormatCopy-on-write versus merge-on-read, incremental pulls versus snapshots, compaction as a job you schedule, and why one table gets one writer and one format.
   • NewPostgres: EXPLAIN ANALYZE, VACUUM, and Why BI Must Leave the PrimaryA BI scan froze autovacuum, the replication slot kept WAL, and the disk hit 91%. Read the plan, manage bloat, and keep CDC from filling the volume.
   • NewRAG Pipelines for Data Engineers: You Already Know How to Build ThisRetrieval-augmented generation is an ETL problem wearing an AI costume. How to build a production RAG ingestion pipeline with the skills you already have.
   • NewAirbyte in Production: A Successful Sync of Zero Rows Is Still an OutageGreen sync, empty destination, raw JSON versus typed silver, self-host versus Cloud, and why Airbyte does not replace dbt.
   • NewAzure Data Engineering in 2026: Data Factory, Synapse, and Where Fabric FitsA practical guide to the Azure data platform: when to use Data Factory, what happened to Synapse, and how Microsoft Fabric changes the picture.
   • NewBeam Runners and Dataflow Cost: Portability vs Workers That Never Scale to ZeroWrite-once Beam is real. The Dataflow bill is vCPU-hours plus Shuffle plus Streaming Engine, and Spark is cheaper if the team already writes Spark.
   • NewMongoDB to Silver: Flatten Nested Arrays Before They Double RevenueA nested `items` array hit gold twice and finance saw 1.9x revenue. Land the document raw, explode once in silver, and do not pretend Mongo is the warehouse.
   • NewThe 5 Best Data Engineering Courses in 2026 (Honest Review)We compared the most popular data engineering courses and certificates on price, depth, and job-readiness. Here are the five actually worth your money.
   • NewCassandra: A New Query Is a New Table, Not a New Index HopeALLOW FILTERING timed out the coordinator and a quick report full-scanned the cluster. Model the query first and keep Cassandra off the warehouse path.
   • NewFivetran MAR: The Invoice That Doubled, and the Credits You Still PayMonthly active rows surprise you, warehouse writes have their own bill, a custom extractor can beat a connector, and Fivetran still does not replace dbt.
   • NewHadoop Hive to Iceberg: Why Lift-and-Shift to S3 Fails and How Banks Actually ExitNameNode small-file pauses, the Hive metastore as the real lock-in, and why distcp-plus-EMR keeps the same tables. Iceberg is the exit, not a new cluster.
   • NewThe AWS Data Engineering Stack: Which Service Does What (and What to Skip)S3, Glue, Kinesis, Redshift, Athena, EMR, Lambda — a map of the AWS data services, what each is actually for, and the combinations that work in practice.
   • NewData Quality: Fail the Job, Open a Ticket, or Ignore Red ForeverFour checks (freshness, volume, schema, uniqueness), a written fail-versus-ticket policy, why always-red tests get ignored, and dbt vs GX vs a vendor.
   • NewPulsar Multi-Tenant Messaging: Brokers, BookKeeper, and When Not to Leave KafkaBrokers are stateless. BookKeeper holds the log. Pulsar wins on tenant isolation and geo. A healthy Kafka cluster is not a fashion problem.
   • NewRedis Features: Cache the Online Path, Never the Source of TruthWe trained on warehouse snapshots and served from Redis. After TTL eviction and a stampede the model looked broken. Online features are a cache.
   • RevisedThe AI-Assisted Data Engineer: A Practical Daily WorkflowHow to actually use AI coding assistants for pipeline work: SQL generation, dbt refactoring, debugging, and docs, plus the failure modes to watch for.
   • RevisedVector Databases Compared: pgvector, Pinecone, Chroma, and When You Need NoneAn honest comparison of vector storage options for AI workloads — and why the right answer is often the database you already run.
   • RevisedRAG Pipelines for Data Engineers: You Already Know How to Build ThisRetrieval-augmented generation is an ETL problem wearing an AI costume. How to build a production RAG ingestion pipeline with the skills you already have.
   • RevisedAzure Data Engineering in 2026: Data Factory, Synapse, and Where Fabric FitsA practical guide to the Azure data platform: when to use Data Factory, what happened to Synapse, and how Microsoft Fabric changes the picture.
   • RevisedThe 5 Best Data Engineering Courses in 2026 (Honest Review)We compared the most popular data engineering courses and certificates on price, depth, and job-readiness. Here are the five actually worth your money.
   • RevisedThe AWS Data Engineering Stack: Which Service Does What (and What to Skip)S3, Glue, Kinesis, Redshift, Athena, EMR, Lambda — a map of the AWS data services, what each is actually for, and the combinations that work in practice.
   • RevisedSnowflake Cortex AI for Data Engineers: SQL Functions, Not Another ChatbotHow to use Cortex LLM functions inside pipelines — COMPLETE, CLASSIFY, EMBED — with cost controls, schema checks, retries, and a warehouse-native RAG shape.
   • RevisedAirflow vs Dagster vs Prefect in 2026: Which Orchestrator Should You Pick?An honest comparison of the three major Python orchestrators: where each one shines, where each one hurts, and a simple decision rule.
   • RevisedBuilding Your First Data Pipeline with Python and AirflowA beginner-friendly guide to your first ETL pipeline: extract from an API, transform with pandas, and load into a database, scheduled with Apache Airflow.
   • RevisedSnowflake-Managed Iceberg Tables: Open Storage without Losing Warehouse SQLWhen to use Snowflake-managed Iceberg vs native tables: external volumes, catalogs, deletion vectors, and the interoperability jobs Iceberg actually wins.
   • RevisedSQL Window Functions: The 5 Patterns Every Data Engineer Uses WeeklyDeduplication, running totals, sessionization, period-over-period comparisons, and top-N per group — plus frames, pitfalls, and live playground queries.
   • RevisedSnowflake vs Databricks in 2026: Pick the Workload, Not the LogoAn honest head-to-head: SQL warehousing vs Spark lakehouse, Iceberg interoperability, Cortex vs Mosaic, and when you actually need both.
   • RevisedData Quality Checks in Python: Catch Bad Data Before Your Users DoA practical framework for adding data quality checks to Python pipelines: what to check, where to check it, and when to fail loudly vs quarantine.
   • RevisedSnowflake vs BigQuery: Credits vs Bytes ScannedThe pricing models decide the architecture: warehouse-hours versus on-demand scans, plus multi-cloud vs GCP-native gravity.
   • RevisedDuckDB: The Fastest Way to Build Local Data Pipelines in 2026Why DuckDB replaced pandas in many pipelines: query Parquet and CSV files directly with SQL, at speeds that embarrass much bigger tools.
   • RevisedKafka vs Amazon Kinesis: Control vs Less OpsWhen to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
   • RevisedBigQuery Partitioning and Clustering: Stop Paying for Full Table ScansBigQuery charges by bytes scanned. Learn how partitioning and clustering cut scan costs by 90%+, and the query patterns that silently bypass them.
   • RevisedDatabricks and Delta Lake: A Practical Introduction to the LakehouseWhat the lakehouse actually is, how Delta Lake adds ACID transactions to cheap object storage, and the medallion architecture in practice.
   • RevisedPySpark for Data Engineers: From Zero to Your First Production JobLearn the PySpark DataFrame API, understand lazy evaluation and partitions, and avoid the classic mistakes that make Spark jobs slow.
5. 05

   Week of Aug 17, 202613 new · 0 revised

   • NewElasticsearch: Sync Documents, Stop Treating Aggregations as FactsDynamic mapping stored amount as text and a terms agg became the executive number. Idempotent ids, a reindex fallback, and warehouse math in the warehouse.
   • NewSnowflake Cortex AI for Data Engineers: SQL Functions, Not Another ChatbotHow to use Cortex LLM functions inside pipelines — COMPLETE, CLASSIFY, EMBED — with cost controls, schema checks, retries, and a warehouse-native RAG shape.
   • NewSQLMesh Plans and Virtual Environments: What dbt State Cannot Isolate, and When to StayA SQLMesh plan is a snapshot diff you apply and virtual environments share unchanged tables. dbt defer is CI, not isolation, and you still need an orchestrator.
   • NewTerraform for Data Platforms: Modules for Warehouses, Not Click-Ops GrantsClick-ops Snowflake grants will not exist in the next account. Modules for warehouse plus IAM, secrets out of state, and Terraform for anything that lasts.
   • NewAirflow vs Dagster vs Prefect in 2026: Which Orchestrator Should You Pick?An honest comparison of the three major Python orchestrators: where each one shines, where each one hurts, and a simple decision rule.
   • NewKubernetes Operators for Data Jobs: Unbounded Pods, Then the Cluster DiesAirflow and Spark can spawn unbounded pods, requests and limits decide eviction, and MWAA or Databricks is sometimes the right way to not run K8s.
   • NewRedshift Sort Keys, WLM, and Spectrum: RA3 Tuning and the Concurrency Scaling BillCOMPOUND sort keys and DISTKEY still decide scans on RA3, VACUUM and ANALYZE are not optional, and Spectrum SELECT * is an S3 bill.
   • NewSemantic Layer: One Revenue Definition, Not Three Tableau WorkbooksQ2 revenue had three values depending on the workbook. LookML, MetricFlow, or a Power BI dataset can hold the metric, but they do not replace dbt.
   • NewSnowflake Time Travel: How It Works, What It Costs, and When It Is Not a BackupSnowflake Time Travel and Fail-safe in practice: retention, UNDROP, cloning at a timestamp, why storage follows churn, and a restore runbook you can test.
   • NewBuilding Your First Data Pipeline with Python and AirflowA beginner-friendly guide to your first ETL pipeline: extract from an API, transform with pandas, and load into a database, scheduled with Apache Airflow.
   • NewSnowflake-Managed Iceberg Tables: Open Storage without Losing Warehouse SQLWhen to use Snowflake-managed Iceberg vs native tables: external volumes, catalogs, deletion vectors, and the interoperability jobs Iceberg actually wins.
   • NewSQL Window Functions: The 5 Patterns Every Data Engineer Uses WeeklyDeduplication, running totals, sessionization, period-over-period comparisons, and top-N per group — plus frames, pitfalls, and live playground queries.
   • NewSnowflake vs Databricks in 2026: Pick the Workload, Not the LogoAn honest head-to-head: SQL warehousing vs Spark lakehouse, Iceberg interoperability, Cortex vs Mosaic, and when you actually need both.
6. 06

   Week of Aug 10, 20266 new · 0 revised

   • NewData Quality Checks in Python: Catch Bad Data Before Your Users DoA practical framework for adding data quality checks to Python pipelines: what to check, where to check it, and when to fail loudly vs quarantine.
   • NewSnowflake vs BigQuery: Credits vs Bytes ScannedThe pricing models decide the architecture: warehouse-hours versus on-demand scans, plus multi-cloud vs GCP-native gravity.
   • NewDuckDB: The Fastest Way to Build Local Data Pipelines in 2026Why DuckDB replaced pandas in many pipelines: query Parquet and CSV files directly with SQL, at speeds that embarrass much bigger tools.
   • NewKafka vs Amazon Kinesis: Control vs Less OpsWhen to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
   • NewBigQuery Partitioning and Clustering: Stop Paying for Full Table ScansBigQuery charges by bytes scanned. Learn how partitioning and clustering cut scan costs by 90%+, and the query patterns that silently bypass them.
   • NewDatabricks and Delta Lake: A Practical Introduction to the LakehouseWhat the lakehouse actually is, how Delta Lake adds ACID transactions to cheap object storage, and the medallion architecture in practice.
7. 07

   Week of Aug 3, 20263 new · 0 revised

   • NewSnowflake Cost Optimization: 7 Techniques That Actually Cut the BillWarehouse sizing, auto-suspend, clustering, ACCOUNT_USAGE queries, and resource monitors — estimate the bill from usage before you change prod.
   • Newdbt Tutorial: Build Your First Transformation Project the Right WaySet up a dbt project from scratch: staging models, marts, tests, and documentation — with the project structure conventions used by real teams.
   • NewKafka Fundamentals: Topics, Partitions, and Consumer Groups ExplainedThe mental model every data engineer needs before touching Kafka: how topics, partitions, offsets, and consumer groups actually work together.
8. 08

   Week of Jul 27, 20261 new · 0 revised

   • NewPySpark for Data Engineers: From Zero to Your First Production JobLearn the PySpark DataFrame API, understand lazy evaluation and partitions, and avoid the classic mistakes that make Spark jobs slow.
9. 09

   Week of May 18, 20269 new · 0 revised

   • NewTesting Airflow Locally: A Setup That Catches Bugs Before Production DoesA four-layer Airflow testing setup: parse checks, DAG structure tests, task unit tests with mocked hooks, and dag.test() runs against local infrastructure.
   • NewCutting Memory in Python Data Jobs: Dtypes, Chunking, Arrow, and StreamingWhere the memory actually goes in a pandas job, how much dtype selection and Arrow-backed strings really save, and when to stop loading and start streaming.
   • NewSnowpipe vs Snowpipe Streaming: Latency, Cost Per Row, and Which One You NeedSnowpipe file ingestion vs Snowpipe Streaming: real latency ranges, the credit math per million rows, and how to pick without over-engineering.
   • NewAirflow Dynamic Task Mapping: Fan-Out That Works, and Fan-Out That Melts the SchedulerPractical patterns for expand, expand_kwargs, and partial in Airflow: size and batch a fan-out safely without overwhelming the scheduler and metadata DB.
   • NewIceberg vs Delta Lake in 2026: Metadata Design, Catalog Options, and How to Actually ChooseThe feature lists have converged. What still differs is metadata structure, catalog architecture, and engine support, and those decide which format fits.
   • NewTesting Data Pipelines in Python: Fixtures, Warehouse Fakes, and Property-Based TransformsLayered testing for Python data pipelines: pure transform tests, DuckDB as a warehouse fake, property-based invariants, and what runs in CI vs production.
   • NewSlowly Changing Dimensions: Types 1 Through 6, and the Two You Will Actually UseWorking SQL for every SCD type, the point-in-time join most Type 2 builds get wrong, and an honest account of which types are worth the maintenance.
   • NewSnowflake Masking and Row Access Policies: Tag-Based Governance That ScalesMoving from per-column masking to tag-based masking and mapping-table row access policies, with the performance traps and audit queries that prove coverage.
   • NewMigrating from Synapse to Microsoft Fabric: What Maps Cleanly, What Does Not, and How to Sequence ItA component-by-component map from Synapse to Fabric, the T-SQL and workload management gaps that bite, and the order I move things in to keep risk low.
10. 10

    Week of May 11, 202619 new · 0 revised

    • NewDebezium on Postgres: Replication Slots, Snapshots, and Applying Changes IdempotentlyPostgres CDC with Debezium in production: how the replication slot fills your disk, snapshotting a large table without locks, and replay-safe change apply.
    • NewAirflow Datasets and Assets: Replacing Cron Guesswork With Data-Aware SchedulingMove cross-team DAG dependencies from padded cron offsets to dataset triggers, with conditional scheduling, naming rules, and the failure modes to expect.
    • NewDeclarative Pipelines in Databricks: Expectations, Streaming Tables, and When the Abstraction Fights YouWhat DLT genuinely buys you, how expectations and streaming tables behave in production, and the four situations where I still write plain Spark jobs instead.
    • NewDataflow vs Dataproc: The Beam Model, the Cost Profiles, and Which One Your Team Can Actually RunDataflow vs Dataproc on GCP in production: what Beam buys you, how the two billing models differ, and the team traits that decide the choice.
    • NewADF Pipeline Patterns That Scale: Parameterized Datasets, Metadata-Driven Ingestion, and Integration Runtime SizingHow to build one Data Factory pipeline that ingests 300 tables instead of 300 pipelines, and how to size integration runtimes so the bill and the SLA both work.
    • NewPandas to Polars: What Translates, What Does Not, and What Breaks QuietlyA production migration guide: the expression API, lazy frames and query optimization, null versus NaN semantics, and pandas patterns with no polars twin.
    • NewSnowflake Data Sharing: Secure Shares, Reader Accounts, and the Marketplace Without Shipping CopiesHow Snowflake secure data sharing works across accounts, regions, and clouds, when to use a reader account, and governance that keeps a share from leaking.
    • NewStar Schema vs One Big Table: What Actually Costs Money on a Columnar WarehouseJoins are not the expense they were in 2015. What a wide denormalized table really costs in storage, rebuild time, and history, and the hybrid I ship instead.
    • NewAirflow 3 for Pipeline Authors: What Actually Changed and What to Fix FirstThe Airflow 3 changes that break real DAGs: the Task SDK boundary, optional logical_date, removed context keys, DAG versioning, and an upgrade checklist.
    • NewKafka Connect in Production: Converters, Dead Letter Queues, and Worker SizingThe Connect settings that decide whether connectors survive real data: converter choice, error tolerance with a DLQ, SMT limits, and sizing workers.
    • NewCutting a Databricks Bill: Job Clusters, Spot, Photon, and Finding the Top Offenders in System TablesTaking 30 percent off a Databricks bill in a week: drop all-purpose clusters for jobs, price spot properly, test Photon per workload, rank system.billing.
    • NewDuckDB vs Spark on One Node: Where the Crossover Actually HappensReal timings and costs comparing single-node DuckDB against a Spark cluster, and why the crossover point is decided by shuffle volume rather than raw data size.
    • NewSnowpark vs SQL: When Python Wins and When It Is Just a Slower Way to Write SQLSnowpark Python vs plain SQL in Snowflake: lazy DataFrames, UDF serialization costs, and the four workloads where Python genuinely earns its place.
    • NewSix Ways to Deduplicate a Table: What Each Costs and Which Row SurvivesDISTINCT, GROUP BY, ROW_NUMBER, QUALIFY, MERGE, and stream-side dedup compared on cost and semantics, plus how to pick the survivor row deterministically.
    • Newdbt Performance at Scale: Threads, Materializations, and Per-Model Cost AttributionCutting dbt run time and warehouse spend on a large project: find the real critical path, see when threads stop helping, and attribute cost per model.
    • NewBatch LLM Enrichment: Idempotency, Cost Per Row, and Knowing When the Output Is WrongAn LLM call as a batch transformation step: content-hash idempotency, per-row cost budgets, retries for a non-deterministic API, and testing fuzzy output.
    • NewStep Functions vs Airflow for Data Orchestration: Cost, Observability, and the Point Where You Outgrow OneState transitions vs scheduler hours, execution history vs task logs, and the signals that tell you which orchestrator your pipeline has outgrown.
    • NewStructured Streaming in Production: Triggers, Checkpoints, Watermarks, and Exactly-Once SinksWhat decides whether a Spark stream survives a year in production: trigger choice, checkpoint discipline, watermark math, and what exactly-once guarantees.
    • NewRecursive CTEs in Production: Org Charts, Bill of Materials, and Cycle ProtectionHow WITH RECURSIVE actually executes, patterns for hierarchy traversal and BOM explosion, and the cycle guards that keep a bad edge from running forever.
11. 11

    Week of May 4, 202622 new · 0 revised

    • NewWhere Lambda Belongs in a Data Pipeline and Where It Quietly Becomes a Distributed System You Cannot DebugLambda is excellent glue and a terrible framework. Here are the limits that shape every design, the idempotency you must build yourself, and the point to stop.
    • NewDuckDB in Production: The Jobs It Genuinely Wins and the Single-Node Limits You Will HitWhere DuckDB belongs in a real pipeline, how far out-of-core execution gets you on one machine, and the concurrency and memory walls that decide when to stop.
    • NewKafka Schema Registry: Evolving Event Schemas Without Breaking ConsumersAvro versus Protobuf, what each compatibility mode really allows, and the CI gate that stops a breaking schema change before it reaches a topic.
    • Newdbt Macros and Jinja: The Patterns That Earn Their Keep, and the Clever Ones That Do NotWhich Jinja patterns earn their indirection in a production dbt project, how to test macros, and the metaprogramming that makes a project unmaintainable.
    • NewPivot and Unpivot in SQL: Patterns That Survive a Changing Column SetConditional aggregation, native PIVOT syntax, unpivoting wide tables, and how to handle dynamic categories without generating SQL at runtime.
    • NewVector Search in the Warehouse: When You Do Not Need a Vector DatabaseNative vector search in Snowflake, BigQuery, and pgvector is now good enough for most RAG workloads. Thresholds, real costs, and when a dedicated store wins.
    • NewRedshift Serverless vs Snowflake in 2026: Architecture, Concurrency, Cost Model, and Migration FrictionRedshift Serverless vs Snowflake in production: how RPU-seconds differ from credits, where each bends under concurrency, and what breaks in a migration.
    • NewSnowflake RBAC Design: A Role Hierarchy That Survives an Audit and a ReorgSplit functional roles from access roles, use future grants and managed access schemas, and build a Snowflake permission model that stays reviewable.
    • NewSpark Executor Memory: Spill, GC, and the OOM Errors That Are Really Partition-Sizing ErrorsHow Spark executor memory is divided, why most OOMs are one oversized partition rather than a small cluster, and six failures you can read from the error.
    • NewHow BigQuery Pruning Actually Works: Partition Metadata, Block Statistics, and the Shapes That Defeat BothWhat BigQuery reads before your data: partition metadata, Capacitor block statistics, and the query shapes that silently turn a pruned scan into a full one.
    • Newdbt Snapshots for SCD Type 2: Check vs Timestamp, and When to Hand-RollChoosing check or timestamp snapshot strategies, what happens when a snapshot run is missed, and when a hand-rolled SCD2 model beats dbt snapshot.
    • NewKafka Exactly-Once Semantics: What Transactions Actually GuaranteeIdempotent producers, transactions, and read_committed give exactly-once inside Kafka. Write to a database and you still need an idempotent sink.
    • NewSnowflake Warehouse Sizing: The Spill and Queue Signals That Tell You Up or OutPick a Snowflake warehouse size from evidence, not intuition, using spilling, queueing, and scaling tests to choose a bigger warehouse or more clusters.
    • NewSpark Data Skew: Diagnosing It Properly, Then Fixing It With Salting, AQE, and BroadcastOne task at 40 minutes while 399 finished in 20 seconds is skew. Here is how I find the offending key, and the four fixes ranked by how much they cost me.
    • NewCTE vs Subquery: When the WITH Clause Is a Materialization BarrierWhat actually happens to a CTE in Postgres, Snowflake, and BigQuery, when it blocks predicate pushdown, and when referencing it twice costs you two scans.
    • NewText-to-SQL in Production: Why the Demo Works and the Deployment FailsThe failure modes that appear after launch (join paths, metric drift, runaway cost) and the semantic layer, constraints, and evals that make text-to-SQL work.
    • NewMaking Athena Fast and Cheap: Partition Projection, Parquet Layout, CTAS, and Workgroup Byte LimitsAthena bills for bytes scanned. The partition projection, Parquet layout, CTAS compaction, and workgroup guardrails I use to cut that number by 90 percent.
    • NewCutting a BigQuery Bill: Partitioning, Byte Limits, Materialized Views, and On-Demand vs EditionsTaking a runaway BigQuery invoice apart: find the top jobs, fix pruning, cap bytes billed, use materialized views honestly, and know when Editions wins.
    • Newdbt Project Structure: A Layout That Survives Two Years and Forty ModelsHow I lay out staging, intermediate, and marts so a dbt project still makes sense after two years: naming rules, ref boundaries, and folders that rot.
    • NewKafka Consumer Group Rebalancing: Why Your Consumers Stall and How to Stop ItRebalances are the most common cause of Kafka consumer lag spikes. The protocol, the four timeouts that matter, and the cooperative sticky config I ship.
    • NewSpark Shuffle Explained: What It Costs, How to Read It in the UI, and the Rewrites That Remove ItA shuffle is a disk write, a network fetch, and a sort you did not ask for. What it costs, how to spot it in the Spark UI, and the rewrites that remove it.
    • NewWindow Frames Deep Dive: RANGE vs ROWS and the Patterns That Replace Self-JoinsHow window frames actually evaluate, why RANGE and ROWS give different answers on ties, and six window patterns that delete self-joins from your models.
12. 12

    Week of Apr 20, 20266 new · 0 revised

    • NewCortex Analyst and Cortex Search: Shipping Chat With Your Data That Answers CorrectlyHow Cortex Analyst semantic models and Cortex Search hybrid retrieval fit together, what the YAML actually controls, and where the accuracy comes from.
    • NewKinesis Data Streams vs Firehose: Shards, On-Demand, and Landing Streams in S3When Kinesis Data Streams earns its shards, when Firehose is enough, and the buffering and partitioning patterns that keep S3 landings queryable.
    • NewJoins and the Fan-Out Bug: How Revenue Doubles Without an ErrorThe join fan-out failure class: grain checks before and after every join, EXISTS for semi-joins, anti-joins done right, and the postmortem pattern.
    • NewSlim CI in dbt: state:modified+ and --defer on a 1,000-Model ProjectHow to get dbt CI from 50 minutes to 4 with state comparison and defer: manifest artifacts, GitHub Actions wiring, and the edge cases that bite.
    • NewUnity Catalog in Practice: Metastore Layout, Grants That Scale, and Leaving hive_metastoreHow I structure Unity Catalog for real teams: three-level namespace design, group-based grants, lineage that works, and a hive_metastore migration that ships.
    • NewSnowflake Search Optimization Service: Point Lookups Without the Full ScanHow the search access path works, what it costs to build and maintain, which predicates it serves, and when clustering keys are the better buy.
13. 13

    Week of Apr 13, 20261 new · 0 revised

    • NewSnowflake Clustering Keys: When They Pay for Themselves and When They Burn CreditsClustering depth, what automatic reclustering actually costs, and a decision framework for clustering keys vs Search Optimization vs doing nothing.
14. 14

    Week of Apr 6, 20266 new · 0 revised

    • NewRAG Over Warehouse Data: Text-to-SQL, Curated Marts, and the Guardrails Between ThemWhy raw text-to-SQL fails on real schemas, when retrieval over curated marts wins, and the semantic model and guardrails that make either safe.
    • NewAWS Glue vs EMR for Spark: The Cost Model, the Cold Starts, and the Migration PointGlue vs EMR for production Spark: DPU pricing vs instance pricing, cold starts, library control, and the job size where teams switch.
    • NewSQL Anti-Patterns That Quietly Multiply Your Warehouse BillFive SQL habits that scan more than they should: SELECT *, functions on filter columns, DISTINCT as a bug fix, OR-joins, and per-row UDF calls.
    • NewA dbt Testing Strategy That Catches Regressions, Not 4,000 WarningsHow I structure dbt tests in production: generic tests on keys, unit tests on logic, dbt-expectations on shape, severity that means something.
    • NewDelta Lake Internals: The Transaction Log, Checkpoints, and Why Small Files HappenHow the _delta_log actually works: JSON commits, optimistic concurrency, checkpoints, deletion vectors, and the small-file problem nobody escapes.
    • NewReading the Snowflake Query Profile: Where the Credits Actually GoHow I read the operator tree: TableScan bytes, partition pruning, local and remote spilling, and the exchange steps that quietly dominate runtime.