All stacks · Platform & IaC

Docker

Local stacks, image discipline, and the unit of pipeline compute.

Docker cover

About Docker

Docker is the unit of local and CI compute for pipelines: the image that runs the DAG, the dbt job, and the Playwright-less SQL test. Image discipline — pin bases, do not copy secrets, keep layers small — is the production skill, not “I ran postgres in compose.”

What you'll learn here

  • Images vs containers vs Compose for a local stack
  • Pinning digests and why `latest` is an incident
  • Multi-stage builds for Python/Java jobs
  • Compose for Kafka + Postgres + a warehouse emulator — and when it lies

Frequently asked questions

Is Docker enough without Kubernetes?

For local and many batch jobs on ECS/Cloud Run/batch, yes. Kubernetes is orchestration of containers, not a requirement to use images.

Should every DAG be a custom image?

A small set of blessed images beats 40 snowflake Dockerfiles. Add a package when the team needs it, not per-DAG art.

Why is CI using a different Python than prod?

Because someone ran it on the runner’s Python. The job should run the same image in CI and prod. That is the whole point.

Search DataLane

↑↓ navigate↵ openesc close