Topic
17 posts ·view all posts
Polars🟡IntermediateA decision guide rather than a benchmark: what the expression API and lazy engine actually buy you, which pandas jobs will not get faster, and the ecosystem gaps that still send work back to pandas.
MLOps🟡IntermediateThe MLOps landscape explained through a data engineering lens: what feature stores actually solve, why training pipelines are just DAGs, and the skills that transfer.
Python🟡IntermediateSchema plus SLAs as a contract owned by the producer — fail the write path when the grain or freshness breaks, instead of discovering it in a warehouse test after the fact.
AI & GenAI🟡IntermediateBeyond chatbots: how data engineers use LLMs for entity extraction, data classification, documentation generation, and pipeline triage — with cost controls.
Polars🟢BeginnerEager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.
AI & GenAI🟡IntermediateRetrieval-augmented generation is an ETL problem wearing an AI costume. How to build a production RAG ingestion pipeline with the skills you already have.
Airflow🟡IntermediateA beginner-friendly, step-by-step guide to building your first ETL pipeline — extract data from an API, transform it with pandas, and load it into a database, all scheduled and automated with Apache Airflow.
Python🟡IntermediateA practical framework for adding data quality checks to Python pipelines: what to check, where to check it, and when to fail loudly vs quarantine.
DuckDB🟡IntermediateWhy DuckDB replaced pandas in many pipelines: query Parquet and CSV files directly with SQL, at speeds that embarrass much bigger tools.
Apache Spark🟡IntermediateLearn the PySpark DataFrame API, understand lazy evaluation and partitions, and avoid the classic mistakes that make Spark jobs slow.
Airflow🟡IntermediateA four-layer testing setup for Airflow: parse checks, DAG structure tests, task unit tests with mocked hooks, and dag.test() end-to-end runs against local infrastructure.
Python🟡IntermediateWhere the memory actually goes in a pandas job, how much dtype selection and Arrow-backed strings really save, and when to stop loading and start streaming.
Python🟡IntermediateA layered testing strategy for Python data pipelines: pure transform tests, DuckDB as a warehouse fake, property-based invariants, and what belongs in CI versus production.
Python🟡IntermediateA production migration guide: the expression API, lazy frames and query optimization, null versus NaN semantics, and the pandas patterns with no polars equivalent.
Snowflake🟡IntermediateA production comparison of Snowpark Python and plain SQL in Snowflake, covering lazy DataFrames, UDF serialization costs, and the four workloads where Python genuinely earns its place.
AI & GenAI🟡IntermediateTreating an LLM call as a transformation step in a batch pipeline. Content-hash idempotency, per-row cost budgets, retry semantics for a non-deterministic API, and how to test output you cannot diff.
DuckDB🟡IntermediateWhere DuckDB belongs in a real pipeline, how far out-of-core execution gets you on one machine, and the concurrency and memory walls that decide when to stop.
