Topic

#python

MLOps

Aug 31, 20269 min read

MLOps for Data Engineers: Feature Stores, Training Pipelines, and Where You Fit

The MLOps landscape explained through a data engineering lens: what feature stores actually solve, why training pipelines are just DAGs, and the skills that transfer.

mlops
ai
python

Illustrated overview of Data Contracts for Pipeline Teams: Fail the Producer, Not Just the Test

Python

Aug 30, 202610 min read

Data Contracts for Pipeline Teams: Fail the Producer, Not Just the Test

Schema plus SLAs as a contract owned by the producer — fail the write path when the grain or freshness breaks, instead of discovering it in a warehouse test after the fact.

data-quality
python

Illustrated overview of Using LLMs Inside Data Pipelines: 4 Patterns That Actually Work in Production

AI & GenAI

Aug 30, 20268 min read

Using LLMs Inside Data Pipelines: 4 Patterns That Actually Work in Production

Beyond chatbots: how data engineers use LLMs for entity extraction, data classification, documentation generation, and pipeline triage — with cost controls.

ai
llm
python

Polars

Aug 29, 20264 min read

Polars LazyFrames: The Speedup Is the Plan, Not the Syntax

Eager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.

polars
python

Illustrated overview of RAG Pipelines for Data Engineers: You Already Know How to Build This

AI & GenAI

Aug 27, 20269 min read

RAG Pipelines for Data Engineers: You Already Know How to Build This

Retrieval-augmented generation is an ETL problem wearing an AI costume. How to build a production RAG ingestion pipeline with the skills you already have.

Airflow

Aug 20, 20265 min read

Building Your First Data Pipeline with Python and Airflow

Extract from an API, transform with pandas, load to a warehouse — scheduled in Airflow with retries, idempotency, and a local Docker setup.

tutorials
airflow
python

Python

Aug 16, 20269 min read

Data Quality Checks in Python: Catch Bad Data Before Your Users Do

A practical framework for adding data quality checks to Python pipelines: what to check, where to check it, and when to fail loudly vs quarantine.

python
data-quality
tutorials

DuckDB

Aug 14, 20269 min read

DuckDB: The Fastest Way to Build Local Data Pipelines in 2026

Why DuckDB replaced pandas in many pipelines: query Parquet and CSV files directly with SQL, at speeds that embarrass much bigger tools.

duckdb
sql
python

Illustrated overview of PySpark for Data Engineers: From Zero to Your First Production Job

Apache Spark

Aug 1, 20269 min read

PySpark for Data Engineers: From Zero to Your First Production Job

Learn the PySpark DataFrame API, understand lazy evaluation and partitions, and avoid the classic mistakes that make Spark jobs slow.

spark
python
tutorials

Airflow

May 21, 202610 min read

Testing Airflow Locally: A Setup That Catches Bugs Before Production Does

A four-layer testing setup for Airflow: parse checks, DAG structure tests, task unit tests with mocked hooks, and dag.test() end-to-end runs against local infrastructure.

airflow
python
best-practices

Python

May 20, 202612 min read

Cutting Memory in Python Data Jobs: Dtypes, Chunking, Arrow, and Streaming

Where the memory actually goes in a pandas job, how much dtype selection and Arrow-backed strings really save, and when to stop loading and start streaming.

python
performance

Python

May 18, 202612 min read

Testing Data Pipelines in Python: Fixtures, Warehouse Fakes, and Property-Based Transforms

A layered testing strategy for Python data pipelines: pure transform tests, DuckDB as a warehouse fake, property-based invariants, and what belongs in CI versus production.

python

Illustrated overview of Pandas to Polars: What Translates, What Does Not, and What Breaks Quietly

Python

May 15, 202612 min read

Pandas to Polars: What Translates, What Does Not, and What Breaks Quietly

A production migration guide: the expression API, lazy frames and query optimization, null versus NaN semantics, and the pandas patterns with no polars equivalent.

python
performance

Illustrated overview of Snowpark vs SQL: When Python Wins and When It Is Just a Slower Way to Write SQL

Snowflake

May 13, 202610 min read

Snowpark vs SQL: When Python Wins and When It Is Just a Slower Way to Write SQL

A production comparison of Snowpark Python and plain SQL in Snowflake, covering lazy DataFrames, UDF serialization costs, and the four workloads where Python genuinely earns its place.

snowflake
python
performance

Illustrated overview of Batch LLM Enrichment: Idempotency, Cost Per Row, and Knowing When the Output Is Wrong

AI & GenAI

May 12, 202612 min read

Batch LLM Enrichment: Idempotency, Cost Per Row, and Knowing When the Output Is Wrong

Treating an LLM call as a transformation step in a batch pipeline. Content-hash idempotency, per-row cost budgets, retry semantics for a non-deterministic API, and how to test output you cannot diff.

ai
python

DuckDB

May 10, 202613 min read

DuckDB in Production: The Jobs It Genuinely Wins and the Single-Node Limits You Will Hit

Where DuckDB belongs in a real pipeline, how far out-of-core execution gets you on one machine, and the concurrency and memory walls that decide when to stop.

duckdb
python