NiFi Backpressure and Provenance: Unbounded Queues, Disk Fill, and When Kafka Connect Wins

The PutDatabaseRecord processor hung on a lock at 02:14. NiFi did not fail. It queued. By 06:40 the content repository and the provenance repository had eaten 480 GB of a 500 GB volume. The flow file repository started throwing No space left on device. Ingest stopped for every process group on the node, including the ones that had nothing to do with that database. We found it because the FTP landing directory had 14,000 files and the downstream warehouse was still on Friday.

I had set heap and I had set nifi.content.repository.archive.max.retention.period. I had not set connection backpressure below the defaults, and I had not paged on provenance disk. The stall was one processor. The outage was the node.

flowchart LR
  src[FTP / HTTP / file] --> ff[FlowFile]
  ff --> q[Connection queue]
  q -->|under limit| proc[Next processor]
  q -->|object or byte cap| bp[Backpressure]
  bp --> src
  proc --> dest[Sink]
  proc --> prov[Provenance repo]
  q --> content[Content repo]

Backpressure must close the source. Provenance writes on every hop even when the sink is healthy.

FlowFiles are not Kafka records

A FlowFile is a pointer: attributes in the FlowFile repository, bytes in the content repository, and an event in provenance for each CREATE, DROP, CLONE, CONTENTMODIFIED, and SEND. Clone a FlowFile to three destinations and you pay three provenance events and, depending on the processors, three content claims.

That is why NiFi feels magical at the edge and expensive in the middle. It can fork, route, and retry a single file with a lineage graph you can click. It will also write that graph to disk forever if you let the default provenance window stand.

import os
import requests

NIFI = os.environ["NIFI_API"].rstrip("/")  # https://nifi:8443/nifi-api
VERIFY = os.environ.get("NIFI_CA", "true").lower() != "false"


def session() -> requests.Session:
    s = requests.Session()
    s.auth = (os.environ["NIFI_USER"], os.environ["NIFI_PASSWORD"])
    s.verify = VERIFY
    s.headers["Accept"] = "application/json"
    return s


def assert_repos_have_headroom(s: requests.Session, min_free_pct: float = 25.0) -> None:
    diag = s.get(f"{NIFI}/system-diagnostics", timeout=20)
    diag.raise_for_status()
    snap = diag.json()["systemDiagnostics"]["aggregateSnapshot"]
    for name, key in (
        ("content", "contentRepositoryStorageUsage"),
        ("provenance", "provenanceRepositoryStorageUsage"),
        ("flowfile", "flowFileRepositoryStorageUsage"),
    ):
        usage = snap[key]
        # NiFi returns a list when there are multiple repo paths.
        items = usage if isinstance(usage, list) else [usage]
        for u in items:
            free = 100.0 - float(u["utilization"].rstrip("%"))
            if free < min_free_pct:
                raise RuntimeError(
                    f"{name} repo {u.get('identifier', '?')} "
                    f"only {free:.1f}% free (used {u['usedSpace']})"
                )

Run that from the same box that pages disk, not from a dashboard someone looks at on Mondays. Content and provenance go independently. I have seen content at 18% and provenance at 96% because a retry loop cloned FlowFiles faster than the sink acked them.

Unbounded queues are a configuration error

Every connection has two caps: object threshold and data-size threshold. Hit either and the upstream processor is told to stop offering. If you leave both at the canvas defaults, a stuck sink becomes a disk-fill.

I set edge ingest connections to 500 FlowFiles or 256 MB, whichever hits first, and I set the source processors (GetSFTP, ListenHTTP) so they cannot pull when the next queue is pressured. A full landing directory is a visible backlog. A full NiFi disk is an invisible one that takes the node with it.

Swap the numbers for your file size. A 200 MB EDI dump needs a different byte cap than 4 KB JSON. The failure mode is the same: the queue is a buffer you own, not a courtesy the UI draws.

Provenance disk is a product decision

Provenance answers “where did this file go?” It is the reason auditors like NiFi. It is also a write-ahead log of every event. Indexing it will OOM a node that looks fine on heap charts if you keep weeks of events on a busy flow.

Keep provenance on its own volume. Cap retention in nifi.properties (nifi.provenance.repository.max.storage.time and max.storage.size) to something you can actually query — I use 48 hours on high-volume ingest, seven days on the regulated FTP flow. If you need a year of lineage, copy the events you care about to object storage on a schedule. Do not ask the on-node repo to be the archive.

Edge and FTP versus Kafka Connect

NiFi is the right tool when the source is a protocol: SFTP from a vendor, a file share, HTTP posts from a plant, a scanner that cannot run a consumer. Guaranteed delivery at the edge, attribute-based routing, and a human-readable lineage graph are the product.

If the data is already on Kafka and the sink is JDBC, S3, or a warehouse, Kafka Connect is a smaller system: workers, tasks, a DLQ, converters. You do not get a canvas. You also do not get a third disk that fills because a processor retried. Exactly-once still ends at the sink key — that boundary is in Kafka exactly-once, and NiFi does not repeal it. processing.guarantee on a Kafka processor does not make PutSQL idempotent.

I do not put NiFi in the middle of two Kafka clusters. That is a very expensive identity function with provenance.

Failure modes

Default backpressure. 10,000 FlowFiles is not a safety rail on a 50 MB file.

One volume for all three repos. Content, FlowFile, and provenance contend and die together.

Provenance queries in the hot path. A UI search that scans a week will stall the node you are trying to debug.

Retry without a destination. Infinite loops clone FlowFiles. Cap retries. Route failures to a dead-letter process group with its own backpressure.

NiFi as the enterprise bus. Use it to land. Hand off to Kafka or object storage. Do not grow the canvas across teams.

What I page on

Heap charts lie when the disk is the incident. I page on content-repo free percent, provenance-repo free percent, and FlowFile-repo free percent — the same three system-diagnostics keys in the check above. I also page if any connection sits at backpressure for more than ten minutes. That is the stuck sink, not the full node yet.

Hub: NiFi. If the next hop is already a topic, prefer Kafka Connect. Exactly-once still ends at the sink key: Kafka exactly-once.

FAQ

Why did the whole node die from one processor? Default queues filled the content and provenance repositories. One volume, three repos, one full disk. Every process group on that node stopped.

What backpressure should I set? Start at 500 FlowFiles or 256 MB on edge ingest, then swap for your file size. A 200 MB EDI dump is not a 4 KB JSON event. Hit either cap and stop the source.

How long do I keep provenance? 48 hours on high-volume ingest, seven days on the regulated FTP flow. Archive events you must keep to object storage. Do not ask the on-node repo for a year.

NiFi or Kafka Connect? NiFi when the source is a protocol (SFTP, file share, plant HTTP). Connect when the data is already on Kafka and the sink is JDBC, S3, or a warehouse.

Does processing.guarantee make PutSQL safe? No. Replay still needs a destination key. NiFi does not repeal the sink-key rule.

What to do Monday

Set object and byte caps on every connection that can grow. Put provenance on its own disk and cap its retention. Page on repository free space, not on JVM heap alone. Keep NiFi at the edge where a protocol must be spoken. If the next hop is already Kafka, stop drawing processors and run a connector.