DuckDB9 min read
DuckDB: The Fastest Way to Build Local Data Pipelines in 2026
Why DuckDB replaced pandas in many pipelines: query Parquet and CSV files directly with SQL, at speeds that embarrass much bigger tools.
- duckdb
- sql
- python
All stacks · Warehouses & analytics
DuckDB
In-process analytics and lightweight local pipelines.
DuckDB
DuckDB13 min read
DuckDB vs Spark on One Node: Where the Crossover Actually Happens
Real timings and costs comparing single-node DuckDB against a Spark cluster, and why the crossover point is decided by shuffle volume rather than raw data size.
- duckdb
- spark
DuckDB13 min read
DuckDB in Production: The Jobs It Genuinely Wins and the Single-Node Limits You Will Hit
Where DuckDB belongs in a real pipeline, how far out-of-core execution gets you on one machine, and the concurrency and memory walls that decide when to stop.
- duckdb
- python
Related reading
Polars4 min read
Polars LazyFrames: The Speedup Is the Plan, Not the Syntax
Eager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.
- polars
- python
Python12 min read
Testing Data Pipelines in Python: Fixtures, Warehouse Fakes, and Property-Based Transforms
A layered testing strategy for Python data pipelines: pure transform tests, DuckDB as a warehouse fake, property-based invariants, and what belongs in CI versus production.
- python
About DuckDB
DuckDB is SQLite for analytics: a full columnar OLAP engine that runs in-process, installs with pip, and reads Parquet, CSV, and Iceberg straight off disk or S3. It has quietly become the fastest way to develop, test, and often run pipelines that do not need a cluster.
These guides cover DuckDB as a serious pipeline tool — local development against production Parquet, replacing pandas in transformation scripts, powering tests in CI without warehouse credentials, and knowing exactly where its single-node ceiling sits.
What you'll learn here
- Querying Parquet, CSV, and S3 data lakes directly with zero loading
- DuckDB as a pandas replacement in Python pipelines
- Testing dbt and SQL logic locally and in CI without a warehouse
- Extensions: httpfs, Iceberg, and the growing ecosystem
- When to graduate to a warehouse or MotherDuck — and when not to
Frequently asked questions
Is DuckDB production-ready or just a toy?
Production-ready for the single-node niche: pipeline transforms, CI, embedded analytics, and datasets into the hundreds of GB. It is not a concurrent multi-user warehouse and does not pretend to be one.
How much data can DuckDB handle?
More than fits in RAM — it spills to disk for larger-than-memory workloads. Practical comfort zone on a decent machine is tens to a few hundred GB per query. Past that, latency and operational patience decide whether you shard the work or move to a distributed engine.
Why use DuckDB instead of pandas?
SQL expressiveness, multi-core execution, larger-than-memory support, and direct Parquet/S3 reads without loading everything first. A groupBy-join workload that stalls pandas at a few GB is routine for DuckDB, and you can hand the same SQL to a warehouse later.
What is MotherDuck?
A managed cloud service built on DuckDB that adds shared storage, collaboration, and hybrid execution — your laptop and the cloud splitting the same query. It is the graduation path when local DuckDB needs sharing and persistence.
Can DuckDB read my lakehouse tables?
Yes — the Iceberg and Delta extensions read open-format tables, and httpfs reads S3/GCS/Azure directly. That makes DuckDB an excellent local query head over the same files your Spark and warehouse jobs use.
New DuckDB posts, straight to your inbox
One email a week with our latest tutorials. No spam.
Newsletter signup is not live yet. Use the contact form if you want to be notified.