Kubernetes Operators for Data Jobs: Unbounded Pods, Then the Cluster Dies

Airflow KubernetesExecutor plus dynamic task mapping expanded a backfill to 2,400 pods. The node pool scaled to 180 nodes. CoreDNS fell over. The cluster API started timing out. Overnight compute was $6,200. The DAG was “just catching up March.” MWAA would have queued the tasks and billed a boring environment fee. We had chosen Kubernetes so we could scale. We scaled.

I do not think every data team should run a cluster. I do think everyone who already has one should treat Airflow, Spark, and Flink operators like production apps: requests, limits, quotas, and an alert on crash loops. Orchestration comparisons live in Step Functions vs Airflow. This post is the K8s-shaped failure: unbounded pods.

Operators spawn jobs. Jobs spawn pods.

An operator (Airflow, Spark, Flink, Strimzi) watches custom resources and creates Jobs, pods, or StatefulSets. Your DAG is not “a container.” It is a factory. Dynamic mapping, SparkApplication executor counts, and a mis-set parallelism are load generators aimed at the API server.

flowchart TD
  dag["Airflow DAG / SparkApplication"] --> op["Operator"]
  op --> pods["Executor / task pods"]
  pods --> q{"Quota + requests?"}
  q -->|no| scale["Node pool + API + DNS"]
  scale --> out["Cluster brownout"]
  q -->|yes| cap["Queued or rejected"]

Without a quota, “retry March” is a cluster-wide event. With one, it is a slow DAG.

# Namespace quota I wish we had before the 2,400-pod night.
apiVersion: v1
kind: ResourceQuota
metadata:
  name: airflow-tasks
  namespace: airflow
spec:
  hard:
    pods: "200"
    requests.cpu: "80"
    requests.memory: 160Gi
    limits.cpu: "160"
    limits.memory: 320Gi
---
# Every task pod. No limits = BestEffort or a noisy neighbor.
resources:
  requests:
    cpu: "500m"
    memory: 2Gi
  limits:
    cpu: "2"
    memory: 4Gi

Requests are what the scheduler believes. Limits are the ceiling. I have watched Spark executors without memory limits get OOM-killed in a crash loop while the driver reported “lost executor” for two hours. I have also watched a Python task with no request sit BestEffort and get evicted the moment the node filled. Both look like “Airflow is flaky.”

Crash loops are the outage

CrashLoopBackOff on the Airflow scheduler, the Spark operator, or the Kafka controller is not a cosmetic. The factory stopped. Tasks queue or vanish. I alert on restart count, not on “pod not ready for 30 seconds” during a rolling deploy.

# kubectl get pods -n airflow -o json | this filter in CI / a probe
def crash_looping(pods: list[dict]) -> list[str]:
    bad = []
    for p in pods:
        name = p["metadata"]["name"]
        for cs in p.get("status", {}).get("containerStatuses", []):
            wait = (cs.get("state") or {}).get("waiting") or {}
            if wait.get("reason") == "CrashLoopBackOff":
                bad.append(name)
            if cs.get("restartCount", 0) >= 5 and not cs.get("ready"):
                bad.append(name)
    return sorted(set(bad))

Read kubectl describe pod once. ImagePullBackOff is credentials. OOMKilled is the limit. CreateContainerConfigError is a missing secret. You do not need CKA. You need to stop guessing from the Airflow UI.

When not to run Kubernetes

If the company does not staff a platform team, I will argue for MWAA, Composer, or Cloud Composer-shaped Airflow, or for Databricks / EMR instead of Spark-on-K8s. You are buying someone else’s node pool, etcd, and upgrade train. The APIs look similar. The on-call does not. Spark-on-K8s is cheaper on paper until you count the people who understand PersistentVolumes and the night the node pool policy was max: 200.

# Spark operator: cap executors the way you cap Airflow mapping.
# A "temporary" executor count of 400 is the same incident.
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: nightly-orders
  namespace: spark
spec:
  type: Python
  sparkVersion: "3.5.1"
  driver:
    cores: 1
    memory: 4g
  executor:
    cores: 2
    memory: 8g
    instances: 20   # not 400

I also cap max_active_tasks / pool slots in Airflow so mapping cannot outrun the quota. The quota is the last line. The DAG should not try 2,400 in the first place.

FAQ

Do I need CKA to run Airflow on K8s? You need to read a pod spec, a quota, and a crash loop. CKA is optional. Unbounded mapping is not.

When should I use MWAA / Composer instead? When you do not staff etcd, upgrades, and node pools. A boring environment fee is cheaper than a $6,200 night.

Spark operator or Databricks jobs? If the org already pays Databricks, do not reinvent executors on a cluster you barely own. If you already run Spark on K8s well, stay.

Hub: Kubernetes. Related: Airflow dynamic mapping.

Pitfalls

No ResourceQuota on airflow and spark. The node pool max becomes the quota. Finance sees it on the invoice. DNS sees it first.

Default pod spec without requests. BestEffort tasks get evicted. The retry storm makes it worse. Pin a pod template on the executor.

Ignoring CrashLoopBackOff on the operator. The UI still shows DAGs. Nothing schedules. Page the operator, not the task.

Running the metadata database on a 20 GB disk. Scheduler crash loops from a full PVC look like “Airflow is down.” Size the metadata store like an app database.

Choosing K8s to save a Databricks bill with no platform rotation. You will spend the savings on the first upgrade weekend. Buy the product until you staff the cluster.

What I put in the namespace before the first DAG

A ResourceQuota on pods, CPU, and memory. A LimitRange so a forgotten pod spec cannot be BestEffort. A default pod template on the Airflow executor with requests and limits. A page if the operator deployment is not Available. A cap on mapped tasks (max_map_length or an Airflow pool) so March cannot become 2,400 pods.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: airflow
  namespace: airflow
spec:
  hard:
    pods: "200"
    requests.cpu: "80"
    requests.memory: 160Gi
    limits.cpu: "160"
    limits.memory: 320Gi

200 pods is a slow backfill, not a cluster incident. The node pool max is not a quota — it is a credit card. Related: Airflow dynamic mapping and Docker pinned images.

I still run operators when the org already has a cluster, quotas, and a platform rotation. Then the data team’s job is: default pod templates with requests/limits, a namespace quota, mapping caps (max_map_length / pool slots), and a page on crash loops. The March backfill should have been 200 pods and a four-hour DAG. It was 2,400 pods and a cluster incident. Scale was never free. It was unmetered.