Docker for Data Engineers cheat sheet
Dockerfiles for Python pipelines, layer caching, compose stacks for local Postgres and Kafka, and image slimming.
A pipeline Dockerfile
FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY src/ src/ CMD ["python", "-m", "src.pipeline"]- Dependencies before source is the whole caching trick — code edits then rebuild in seconds instead of minutes.
FROM python:3.12-slim AS builder ... FROM python:3.12-slim- Multi-stage build. Compile wheels in the builder, copy only the installed packages, and drop the toolchain.
RUN useradd -m app && chown -R app /app; USER app- Never run pipelines as root. Many managed runtimes reject root containers outright.
ENV PYTHONUNBUFFERED=1- Without it, logs are buffered and a crashed container loses the output that explains the crash.
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv- uv installs dependencies far faster than pip in CI, and uv sync --frozen enforces the lockfile.
Slimming images
.dockerignore with .git, target/, tests/, *.duckdb- Build context size directly affects build time. Shipping .git into the image is the most common waste.
RUN apt-get install -y --no-install-recommends gcc && rm -rf /var/lib/apt/lists/*- One RUN layer that cleans up after itself. Cleaning in a later layer does not shrink the image.
python:3.12-slim over python:3.12- Roughly 700 MB saved for most pipelines. Reach for alpine only if you accept musl wheel-compilation pain.
docker image history myimage- Shows which layer is bloating the image. Usually an uncleaned apt cache or a copied virtualenv.
Compose for local stacks
services: postgres: image: postgres:16 environment: POSTGRES_PASSWORD: local ports: ["5432:5432"] volumes: ["pgdata:/var/lib/postgresql/data"]- A named volume keeps data across restarts. Without it, every docker compose down wipes your test tables.
healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 5s- Healthchecks plus depends_on with condition service_healthy stop the pipeline from racing the database.
docker compose up -d --build- Rebuild and start detached. Add --wait to block until healthchecks pass, which makes CI scripts deterministic.
docker compose logs -f pipeline- Follow one service's logs instead of the interleaved firehose.
docker compose run --rm pipeline pytest- One-off command in the service environment. --rm keeps stopped containers from accumulating.
Debugging containers
docker exec -it <container> bash- Shell into a running container. On slim images use sh; bash may not be installed.
docker run --rm -it --entrypoint bash myimage- Inspect an image whose entrypoint crashes immediately, before it can exit.
docker logs --tail 100 <container>- First stop for a container that exited. Pair with docker inspect for the exit code.
docker stats- Live memory and CPU. A pipeline killed with exit code 137 hit the memory limit, not a bug in your code.
Registry and CI
docker build -t myrepo/pipeline:1.4.0 -t myrepo/pipeline:latest .- Always tag an immutable version alongside latest. Deploying latest makes rollbacks guesswork.
docker buildx build --platform linux/amd64 --push- Required when building on Apple Silicon for x86 cloud runtimes, or the container fails to start on deploy.
cache-from: type=gha- GitHub Actions layer caching. Turns a five-minute image build into under a minute on unchanged dependencies.
docker scout cves myimage- Vulnerability scan before push. Base-image updates fix most findings without touching your code.
From DataLane — tutorials at/blog, practice SQL live in theplayground.