Kafka vs Pulsar: Storage Architecture, Multi-Tenancy, and Why Most Teams Should Stay

I like Pulsar’s architecture more than Kafka’s. I have also talked several teams out of migrating to it. Both of those positions are correct, and the gap between them is the whole point of this post.

The comparison is usually framed as a feature list. It is really a question about what your operational pain is today, and whether that pain is the specific kind that Pulsar’s storage design removes.

The architectural difference in one diagram

flowchart TD
  subgraph Kafka
    kp["Producer"] --> kb["Broker owns partition<br/>leader + local disk"]
    kb --> kd[("Local log segments")]
  end
  subgraph Pulsar
    pp["Producer"] --> pb["Broker: stateless serving"]
    pb --> bk["BookKeeper ensemble"]
    bk --> bd[("Segments spread<br/>across bookies")]
  end

In Kafka the broker is the storage. In Pulsar the broker is a router in front of storage.

In Kafka, a partition is a log that lives on specific brokers. The leader owns it, followers replicate it, and the data is on those machines’ disks. Everything follows from this: adding a broker means reassigning partitions and copying data across the network; a partition cannot be larger than a broker’s disk; and rebalancing is an operation you schedule carefully rather than something that just happens.

In Pulsar, a topic is served by a stateless broker and stored as segments in BookKeeper. Segments are distributed across bookies independently of which broker serves the topic. Adding a broker adds serving capacity instantly, because there is no data to move. Adding a bookie adds storage capacity, and new segments start using it immediately. A single topic can exceed any one node’s disk.

This is a genuinely better design for elasticity. It is also two systems to operate instead of one, plus ZooKeeper in most deployments, which is the tax you pay for it.

Where Pulsar’s design pays off in practice

Multi-tenancy. Pulsar has tenants and namespaces as first-class concepts, with quotas, retention, backlog limits, and auth policies per namespace. If you are running a platform where twenty teams share one cluster and one team’s runaway producer must not affect the others, this is not a feature you can approximate in Kafka with naming conventions and hope. I covered the operational shape in Pulsar multi-tenant messaging.

Geo-replication. Built into the broker, configured per namespace, with async replication between clusters as a configuration rather than a separate product. Kafka’s answer is MirrorMaker 2 or a commercial equivalent, and everyone who has operated MirrorMaker 2 has an opinion about it.

Queueing and streaming in one system. Pulsar’s shared subscription gives you competing-consumer queue semantics with per-message acknowledgement, alongside exclusive and failover subscriptions for log semantics. If you currently run Kafka and RabbitMQ or SQS because Kafka’s consumer-group model does not fit task distribution, Pulsar collapses that into one system.

Tiered storage as a normal thing. Offloading old segments to S3 is a namespace policy. Kafka has tiered storage now too, and it is newer and less battle-tested in the open-source distribution.

Where Kafka wins, which is most places

The ecosystem, and it is not close. Kafka Connect has hundreds of production-grade connectors — see Kafka Connect in production. Debezium emits Kafka natively. Schema Registry with Avro and Protobuf compatibility modes is mature and covered in the schema evolution post. Every warehouse, every ELT vendor, and every observability tool has a Kafka integration. Pulsar has adapters and a Kafka-compatible protocol layer, and “compatible” always develops an asterisk at the worst moment.

Hiring and operational knowledge. You can hire someone who has run Kafka. The failure modes are documented in a thousand postmortems. When your Pulsar cluster has a BookKeeper ledger issue at 3 a.m., the search results are thinner.

Managed options everywhere. MSK, Confluent Cloud, Redpanda, Aiven. Pulsar has StreamNative and a few others. Fewer choices means less pricing pressure and fewer exits.

KRaft removed the ZooKeeper argument. A lot of the “Kafka is operationally complex” case was really “ZooKeeper is operationally complex”. Modern Kafka does not need it, which quietly deleted one of Pulsar’s strongest talking points.

The consumption model difference that affects your code

This is where a migration actually costs you.

Kafka consumer groups assign whole partitions to consumers. Parallelism is capped at the partition count, ordering is per partition, and offsets are a single number per partition. Rebalancing is a stop-the-world event you tune around — the timeouts and cooperative sticky config are in consumer group rebalancing.

Pulsar acknowledges individual messages and tracks a cursor with per-message state. A shared subscription can have more consumers than partitions, because messages are dispatched individually rather than by partition ownership. There is a real negative-acknowledgement path and a built-in dead letter policy per subscription.

For a task-distribution workload, that is a much better fit. For a stream-processing workload where ordering per key matters, both work and Kafka’s model is simpler to reason about.

The migration cost is that every consumer’s error handling, offset management, and parallelism assumption is written against one model. This is not a client library swap.

The honest decision

flowchart TD
  q1{"Running Kafka today and it is healthy?"} -->|yes| stay["Stay. Fix the actual pain."]
  q1 -->|no / greenfield| q2{"Many tenants sharing one platform?"}
  q2 -->|yes| pulsar[Pulsar]
  q2 -->|no| q3{"Need queue semantics AND streaming?"}
  q3 -->|yes| pulsar
  q3 -->|no| q4{"Active-active across regions as a core requirement?"}
  q4 -->|yes| pulsar
  q4 -->|no| kafka["Kafka — ecosystem and hiring win"]

Most paths end at Kafka. That is not a criticism of Pulsar.

If you are running Kafka and the pain is rebalancing storms, partition-count planning, or an expensive MirrorMaker 2 topology, Pulsar addresses those directly and the evaluation is legitimate.

If the pain is consumer lag, badly sized partitions, or a connector that keeps dying, you have a Kafka tuning problem and migrating platforms will carry it with you.

Pitfalls

Migrating for architecture rather than a problem. “Stateless brokers” is not a business outcome. Write down the specific incident you are trying to stop having.

Underestimating BookKeeper. It is a distributed system with its own tuning: ensemble size, write quorum, ack quorum, journal disk separate from ledger disk. Running it on one shared disk will disappoint you.

Assuming the Kafka protocol handler is a drop-in. KoP covers a useful subset. Transactions, some admin operations, and specific client behaviors are where it thins out. Test your actual clients, not a hello-world producer.

Running both during a long migration. Dual-writing to two brokers doubles the failure surface and creates ordering questions between them. Migrate topic families completely, one at a time.

Ignoring backlog quotas in Pulsar. A namespace without a backlog limit and a slow consumer will fill BookKeeper. The controls exist; defaults are not your policy.

Treating either as a database. Neither is a source of truth for queries. Infinite retention plus tiered storage still is not a warehouse — land it in Iceberg or Delta for analytics.

FAQ

Is Pulsar faster than Kafka? Published benchmarks go both directions depending on who ran them and which durability settings were used. At the throughput most data teams operate at, both saturate the network or the disk before the software matters. Do not choose on this.

What about Redpanda? If your complaint is Kafka’s operational overhead rather than its data model, Redpanda is usually the more relevant evaluation: one binary, no JVM, no ZooKeeper, wire-compatible with Kafka clients. You keep the ecosystem, which is the expensive thing to give up.

Does Pulsar Functions replace Flink? For simple per-message transforms, routing, and enrichment, yes, and it saves you a cluster. For windowing, joins, and large keyed state, no — that is still a Flink or Spark job.

Can I run Pulsar without ZooKeeper? Newer versions support alternative metadata backends, but the ZooKeeper-free story is less mature than Kafka’s KRaft. Check the current state for your version before you plan around it.

We are on MSK. Should we look at Pulsar? Almost certainly not. Managed Kafka on AWS removes most of the operational burden Pulsar’s architecture solves, and you would be taking on a self-managed two-tier system to fix a problem your provider is already handling.

What this means for data engineers

Kafka is the correct default and will stay that way, mostly because the ecosystem and the hiring pool are the parts you cannot rebuild. The Kafka CLI sheet and streaming architecture patterns are worth more to your platform than any broker swap.

Pulsar earns its place in three specific situations: a genuinely multi-tenant platform, an active-active geo requirement, or a stack that runs both a queue and a log and wants one system. If you are in one of those, the architecture is not marketing — it is the answer. If you are not, the best streaming decision available to you is tuning what you already run.