Pulsar Multi-Tenant Messaging: Brokers, BookKeeper, and When Not to Leave Kafka

A platform team replaced Kafka because a conference talk said Pulsar was multi-tenant. Seven months later a BookKeeper ensemble filled a data disk. The broker layer looked fine — CPU idle, connections accepted — and producers got PersistenceException from a bookie that had been at 98% for eleven days. One tenant’s unbounded topic had compacted nothing and retained everything. The other twelve tenants shared the same bookie set. Isolation in the name was not isolation on the disk.

Kafka was not on fire when they started. Consumer lag was boring. They wanted namespaces. They got a second distributed log to learn, and an outage class their runbooks did not have.

flowchart LR
  prod[Producer] --> br[Broker]
  br --> bk[BookKeeper ledger]
  bk --> disk[Bookie disks]
  cons[Consumer] --> br
  ns[Tenant / namespace] --> quota[Quotas and policies]
  quota --> br
  disk -->|full| outage[All namespaces on that ensemble]

The broker is a front door. The outage is almost always storage. Namespace quotas have to hit the bookies, not just the API.

Brokers versus BookKeeper

A broker is stateless serving: produce, consume, dispatch, schema. You can add brokers to take load. Losing one should be a reconnect.

BookKeeper (bookies) stores segments (ledgers). That is the durable log. Replication, disk, and ensemble placement live here. If you only monitor broker JVM heap, you are watching the wrong process.

ZooKeeper or the metadata store sits under both. Treat it as production data, not as “the thing the install docs mentioned.” I have lost a namespace map because someone snapshot-restored ZK from Tuesday.

import os

import pulsar

client = pulsar.Client(
    os.environ["PULSAR_URL"],
    authentication=pulsar.AuthenticationToken(os.environ["PULSAR_TOKEN"]),
    operation_timeout_seconds=15,
)

producer = client.create_producer(
    "persistent://payments/live/authorizations",
    batching_enabled=True,
    batching_max_publish_delay_ms=10,
    block_if_queue_full=True,
    # Partitioned topic. One partition is a hotspot, not a design.
)

def publish(payment_id: str, body: bytes) -> None:
    producer.send(
        body,
        partition_key=payment_id,
        event_timestamp=int(os.environ.get("NOW_MS", "0")) or None,
    )


consumer = client.subscribe(
    "persistent://payments/live/authorizations",
    subscription_name="ledger-writer",
    consumer_type=pulsar.ConsumerType.KeyShared,
    negative_ack_redelivery_delay_ms=30_000,
)

while True:
    msg = consumer.receive(timeout_millis=5000)
    try:
        upsert_ledger(msg.partition_key(), msg.data())
        consumer.acknowledge(msg)
    except TemporaryStoreError:
        consumer.negative_acknowledge(msg)

persistent://tenant/namespace/topic is the address. Put quotas on the namespace — backlog size, publish rate, retention — before the first noisy tenant arrives. The Bookie fill I opened with was a missing backlog quota, not a missing “multi-tenant” feature.

Do not replace healthy Kafka for fashion

If Kafka is meeting SLOs, your people know consumer groups, and the pain is “we wish we had nicer names,” stay. Prefixes and quota plugins are ugly. They are cheaper than a migration that rewrites every producer, every Connect connector, and every exactly-once story.

KoP (Kafka-on-Pulsar) looks like an escape hatch. It is a compatibility layer with its own lag and its own bugs. I will not bet a payments topic on it so that a slide can say “no client changes.”

The delivery boundary does not move. Pulsar transactions and Kafka transactions both stop at the log. The database INSERT still duplicates on replay. That write-up is Kafka exactly-once, and it applies here with different class names.

When Pulsar wins

Many tenants with hard isolation. A namespace per team or per customer, with storage and rate quotas that can actually evict, is a first-class model. Kafka can approximate it. Pulsar is built as it.

Geo. Built-in geo-replication at the namespace, not a MirrorMaker estate you pretend is not a product. If you need active-active across regions and you are willing to own BookKeeper in both, Pulsar is a reason.

Huge topic counts. Topic-per-entity patterns that make Kafka brokers sad are closer to Pulsar’s design. Still put a bound on it. “Per entity” becomes “per bookie disk” without retention.

If you have one cluster, one team, and a few hundred topics, Kafka is the default I hire for. The comparison that includes orchestrators is unrelated; the comparison that includes “should we move the log” is this paragraph.

Quotas I set before the second tenant

A namespace without a backlog cap is a shared disk with etiquette. I set these on every new namespace, including prod:

backlog size          — hard cap, not "we will watch it"
publish rate          — noisy tenant cannot starve others
retention / TTL       — debug topics die on a clock
compaction threshold  — compacted topics actually compact

The Bookie fill was an unbounded persistent:// topic with infinite retention and no backlog quota. Twelve other namespaces were innocent and still went dark.

Compare the log choice, not the conference talk: Kafka vs Pulsar. Hub: Pulsar.

FAQ

Is Pulsar “better Kafka”? It is a different storage split. Better for many tenants and geo if you will operate BookKeeper. Worse if your hire market and connectors are Kafka-shaped.

Do I need BookKeeper expertise? Yes. If you only want to think about brokers, you will miss the disk page. Budget bookie disks like you budget Kafka log dirs.

Can I run KoP and call the migration done? You can run it. I would not call payments done. Two protocols, one on-call, extra lag modes.

What about Pulsar Functions vs Flink? Functions are light transforms. Flink is a stateful engine. Do not replace a watermark job with a Function because the namespace looks tidy.

How do I page this? Bookie disk %, write latency, namespace backlog, metadata-store health. Broker CPU is a vanity chart until those are green.

Failure modes

Broker-only dashboards. Bookie disk and write latency are the pages.

Shared ensemble, no namespace quota. Multi-tenant in the URL, single-tenant on the disk.

Kafka replacement as a platform goal. Fashion. Require an SLO the current cluster misses.

KoP as the migration. Two protocols, one on-call.

Retention = infinite on a debug topic. That is the disk-fill. Set TTL on every namespace, including prod.

What to do Monday

If Kafka is boring, leave it. If you already run Pulsar, page on bookie disk and namespace backlog, not on broker CPU. Put publish and backlog quotas on every namespace before you add a tenant. Use Pulsar when isolation or geo is the requirement you can name. Do not use it because the old cluster is unfashionable.