Kafka cost is driven by five things: broker compute, storage and retention, network (especially cross-zone replication and client traffic), partition count, and the connectors around the cluster. Partitions are the hidden multiplier — each one adds metadata, open files, replication work, rebalance time and producer memory, and the producer side multiplies again by the number of application instances. Size partitions from measured throughput and consumer parallelism, not habit; tune producers to fill batches; adopt the new consumer group protocol to shrink rebalances; use tiered storage and rack-aware fetching to cut storage and cross-zone bills; and retire the topics nobody reads. You cannot shrink a topic’s partition count in place, so reducing it is a migration to a new topic — we show the pattern. A 30-day plan and a checklist close the post.
Where Kafka cost actually comes from
Whether you run Apache Kafka yourself, use a managed service, or pay a usage-based cloud bill, the money goes to the same five places. The proportions differ by deployment model, but the drivers do not.
| Cost driver | What drives it | Typical lever |
|---|---|---|
| Broker compute | Peak throughput, request rate, partition count, TLS, compression done on brokers | Right-size brokers, batch better, consolidate clusters |
| Storage & retention | Ingress × retention × replication factor | Retention by topic class, tiered storage, compaction where it fits |
| Network | Cross-zone replication, producers and consumers talking to leaders in other zones, egress | Rack-aware placement, follower fetching, compression |
| Partitions | Topic count × partitions per topic × replication factor | Partition right-sizing, topic cleanup |
| Connectors & clients | Connect workers, stream processors, idle consumers, over-provisioned pods | Task right-sizing, autoscaling, retiring unused pipelines |
Two patterns show up again and again. First, clusters are sized for their partition count rather than their traffic: brokers are added because the partition count per broker crept past a comfortable limit, not because throughput required it. Second, network is the line item nobody modeled. Replication between zones and consumers fetching from leaders in other zones can cost as much as the brokers themselves on some cloud bills.
The partition tax: why too many partitions hurt
Partitions are Kafka’s unit of parallelism, and adding them feels free. It is not. Each partition carries a fixed cost on the brokers, the controller and every client that touches it, and that cost is paid whether or not traffic flows.
- Metadata and the controller. Every partition and replica is state the controller tracks and propagates. KRaft has raised the practical ceiling a long way compared with ZooKeeper-based clusters, but leadership changes, broker restarts and metadata propagation still scale with partition count.
- Files and memory on brokers. Each replica has log segments and index files on disk, and the open file handles and page cache pressure that come with them.
- Replication work. Followers fetch every partition they replicate. More partitions means more fetch requests and more cross-zone traffic for the same bytes.
- Rebalances and recovery. Consumer group rebalances, broker restarts and partition reassignments all take longer as the number of partitions grows.
- Producer efficiency. Producers batch per partition. Spread the same traffic over more partitions and each batch fills more slowly, so you send more, smaller requests — more CPU on brokers, worse compression, and higher latency for the same throughput.
The producer memory math
The producer side is where the tax gets multiplied. A Kafka producer keeps an in-memory buffer (buffer.memory, 32 MB by default) and builds one batch per partition it is writing to, up to batch.size bytes. Each application instance has its own producer. So the memory the fleet reserves, and the number of tiny batches it sends, depend on three numbers multiplied together: partitions written, instances, and batch size.
The practical consequences are familiar to anyone who has scaled a large fleet: producers that block on buffer.memory under load, request rates on brokers that climb faster than bytes, and the instinct to add more pods — which makes it worse, because every new pod opens its own set of half-empty batches. When producers are the bottleneck at scale, the fix is usually fewer, fuller batches, not more instances.
A sizing method that starts from measurements
Partition counts are usually chosen once, by habit (“we always use 50”) or by fear (“make it 200 so we never have to change it”). We size them from three measured inputs.
Peak throughput, not average
Measure bytes in and bytes out per topic at the busiest realistic period — month-end, a sale, market open — and add headroom for growth over the topic’s planned life. Averages hide the hour that matters.
Throughput per partition, measured on your hardware
Benchmark what one partition sustains for your record size, compression and acks setting, for both producing and consuming. The consumer side is usually the binding constraint, because consumers do real work per record.
Consumer parallelism you actually need
A consumer group cannot use more consumers than there are partitions. Size for the maximum parallelism the slowest consumer needs at peak, not for an imagined future of thousands of consumers.
The partition count is then the larger of two numbers: peak throughput divided by per-partition throughput, and the parallelism the slowest consumer needs. Add modest headroom and stop.
# Illustrative sizing for an "orders" topic peak ingress = 40 MB/s # measured at month-end, plus 50% growth per-partition (consume) = 5 MB/s # benchmarked with real processing by throughput = 40 / 5 = 8 partitions slowest consumer needs = 12 parallel workers at peak choose = max(8, 12) + headroom = 16 partitions # The topic was created with 120. Same traffic, 7.5x the partition tax.
Keys matter more than counts. If ordering is keyed, a hot key pins traffic to one partition no matter how many you add. Check key skew before raising partition counts; the fix may be a better key, not more partitions.
Producer tuning: fill the batch
Most producer “performance problems” are batching problems. These are the settings we review first. None of them require changing application logic, which is why they are often the fastest win available.
# Producer baseline we start from, then tune with measurements acks=all enable.idempotence=true # default on in modern clients; safe retries without duplicates linger.ms=10 # wait briefly so batches fill; raise for throughput-heavy topics batch.size=131072 # 128 KB; useful only if linger lets batches fill compression.type=zstd # or lz4 for lowest CPU; compression works per batch buffer.memory=67108864 # 64 MB; size against partitions written per instance max.in.flight.requests.per.connection=5 # ordering preserved with idempotence partitioner: default sticky behavior for unkeyed records
| Setting | What it does | Watch for |
|---|---|---|
linger.ms | Lets batches fill before sending | A default of 0 or very low values on high-volume topics sends many tiny requests (recent clients default to 5 ms) |
batch.size | Upper bound per partition batch | Raising it does nothing if traffic per partition is too thin to fill it |
compression.type | Compresses each batch | Bigger batches compress far better; zstd for ratio, lz4 for CPU |
acks / idempotence | Durability and duplicate-free retries | Do not trade durability for speed; fix batching instead |
buffer.memory | Total producer memory | Blocking sends under load; size it from partitions written, not a guess |
A useful diagnostic is to compare record rate with request rate on the brokers. If requests per second are close to records per second, producers are effectively sending one record at a time. Looking at what clients actually send on the wire — request sizes, metadata refresh storms, reconnect loops, consumers committing after every record — finds anti-patterns that dashboards built on broker metrics alone miss.
Consumers and rebalances
On the consumer side, the cost of wide topics shows up as rebalance time and idle capacity.
- Fetch sizes. Raise
fetch.min.bytesand set a sensiblefetch.max.wait.msso consumers pull fewer, larger responses instead of polling constantly for small ones. - The new consumer group protocol. Kafka’s next-generation rebalance protocol (KIP-848, generally available on the consumer side in Apache Kafka 4.0) moves assignment to the broker-side group coordinator and makes rebalances incremental. Members no longer all stop and rejoin when one changes, which removes much of the “stop the world” pain on large groups. Check your client library and platform support, then opt in with
group.protocol=consumer. - Static membership. For the classic protocol,
group.instance.idavoids rebalances on rolling restarts of the same instances. - Idle consumers. Consumers above the partition count sit idle. Autoscalers that scale consumer pods past the partition count add cost and nothing else.
Storage, retention and tiered storage
Storage cost is ingress multiplied by retention multiplied by replication factor. The most common waste is a single retention setting applied to every topic.
- Classify topics by retention need. Operational events may need hours; integration topics a few days; audit and replay topics much longer. Set retention per class, not per cluster.
- Tiered storage. Keeping recent segments on broker disks and older segments in object storage (KIP-405 in Apache Kafka, and the tiered storage features of the managed platforms) makes long retention affordable without growing brokers to hold it.
- Compaction where it fits. Topics that represent the latest state per key, such as reference data and changelogs, should be compacted rather than retained by time.
- Object-storage-first topics. A newer class of designs writes topic data directly to object storage instead of replicating it across broker disks, trading some latency for far less cross-zone replication traffic and much cheaper storage. Several platforms and open proposals now offer variants of these “diskless” topics. They suit high-volume, latency-tolerant streams such as logs, telemetry and lakehouse feeds; we evaluate them per workload rather than as a cluster-wide default.
Network: stop paying for cross-zone traffic
In a three-zone cluster, every byte produced is replicated to two other zones, and consumers by default read from the partition leader wherever it lives. On clouds that charge for inter-zone traffic, that adds up quickly.
- Rack awareness. Set
broker.rackso replicas spread across zones deliberately. - Follower fetching. With KIP-392, consumers set
client.rackand read from a replica in their own zone instead of crossing to the leader. For read-heavy topics with several consumer groups, this is often the single largest network saving. - Compression end to end. Bytes compressed by the producer stay compressed through replication and fetch, so batching and compression reduce network cost as well as storage.
Topic cleanup and cluster consolidation
Every estate we assess has topics with no producers, topics with no consumers, and clusters that exist because a team once needed isolation and nobody revisited it. Cleaning these up is the least technical and often the most valuable step.
- Find the dead. Topics with no produce traffic and no consumer group offsets over a long window are candidates for deletion after an owner confirms. Measure it; do not ask in a meeting.
- Assign owners. A topic without an owner cannot be right-sized, because nobody can say what its consumers need.
- Use quotas, not clusters, for isolation. Produce, fetch and request quotas per client or team stop one tenant from starving others, which is usually the reason separate clusters were created.
Reducing partitions: you cannot shrink in place
Kafka lets you add partitions to a topic but not remove them. Adding partitions also changes which partition a key maps to, which breaks per-key ordering for keyed topics at the moment of the change. So both directions need care, and reducing partitions is always a migration.
Create the right-sized topic
Same configuration, the new partition count, and a versioned name such as orders.v2.
Bridge with a copy job
A small stream processor or replication flow copies records from the old topic to the new one, re-partitioning by key, while producers still write to the old topic.
Move consumers, then producers
Consumers switch to the new topic at a known position (by timestamp or a recorded offset mapping), and producers switch once consumers are reading it. For keyed topics, drain the old topic before producers switch so ordering per key holds.
Retire the old topic
Stop the bridge, keep the old topic read-only for an agreed window, then delete it.
Topic names that are treated as APIs — versioned, owned, with a published schema — make this migration routine. Topics that every service hard-codes make it a cross-team project.
Handling surges without over-provisioning
The reason clusters are oversized is usually fear of the peak. There are cheaper ways to survive it than running at peak capacity all year.
- Let Kafka absorb it. Kafka is a buffer. A consumer that falls behind during a burst and catches up afterwards is often the correct design, as long as retention covers the backlog and downstream SLAs tolerate the delay.
- Backpressure in producers. Producers that block or shed low-priority traffic when the buffer fills behave better than producers that retry aggressively and amplify the surge.
- Quotas. Broker quotas keep one surging tenant from taking down everyone else on a shared cluster.
- Autoscale consumers on lag, up to the partition count. Scaling on consumer lag rather than CPU tracks the real backlog; scaling beyond the partition count adds nothing.
Decision table
| Situation | Do this | Not this |
|---|---|---|
| Producers blocking under load | Raise linger, size buffer to partitions written, reduce partitions per pod | Add more producer pods |
| Broker CPU high, bytes modest | Fix batching and compression; check request rate | Add brokers |
| Long, frequent rebalances | New consumer protocol, static membership, fewer partitions | Raise session timeouts indefinitely |
| Storage growing faster than traffic | Retention classes, tiered storage, compaction | Bigger disks everywhere |
| Large inter-zone network bill | Follower fetching, rack-aware clients, compression | Single-zone clusters for production |
| Topic with far more partitions than needed | Migrate to a right-sized v2 topic | Leave it, or try to shrink in place |
| Many lightly used clusters | Consolidate with quotas and standards | One cluster per team by default |
Failure modes we plan for
1. Right-sizing without owners. Partitions are reduced on a topic whose consumer needed the parallelism at month-end, and the first close after the change runs late.
2. Adding partitions to a keyed topic in production. Key-to-partition mapping changes, and consumers that relied on per-key ordering process events out of order.
3. Cutting retention below the replay need. Storage drops, and the next reprocessing request discovers the data is gone.
4. Tuning on averages. Settings that look fine on an average day fail at the peak that justified the cluster in the first place.
5. Saving money by weakening durability. Dropping replication factor or acks to cut cost trades a visible bill for an invisible data-loss risk.
A 30-day right-sizing plan
Week 1 — Measure
Inventory clusters, topics, partitions, retention, owners, producers and consumer groups. Capture peak throughput, request rates, lag and the network bill by zone. Our InsightKafka health checks automate most of this assessment.
Week 2 — Quick wins without code changes
Producer batching and compression settings, consumer fetch settings, follower fetching, retention classes, and deleting confirmed-dead topics.
Week 3 — Structural changes
Tiered storage, new consumer protocol rollout, quotas, and the first right-sized v2 topic migrations for the widest topics.
Week 4 — Consolidate and lock it in
Plan cluster consolidation, add partition and retention standards to topic provisioning through GitOps, and set alerts so the tax does not creep back.
Checklist
| Area | Healthy when |
|---|---|
| Ownership | Every topic has an owner and a documented consumer need |
| Partitions | Counts are derived from measured peak throughput and consumer parallelism |
| Producers | Request rate is well below record rate; batches fill; compression is on |
| Consumers | Rebalances are short and rare; no consumers idle above the partition count |
| Retention | Set by topic class; long retention uses tiered storage |
| Network | Consumers fetch from replicas in their own zone |
| Isolation | Quotas protect tenants on shared clusters |
| Standards | Topic creation goes through GitOps with partition and retention defaults |
The bottom line
Kafka rarely gets expensive because the business sends too much data. It gets expensive because partitions, topics, retention and cross-zone traffic grow by habit while nobody measures what they cost. Right-sizing is not a rebuild: measure the real peaks, fill the batches, fetch locally, tier the storage, retire what nobody reads, and migrate the widest topics to right-sized versions one at a time.
Done that way, the cluster gets cheaper and faster at the same time — fewer requests, fuller batches, shorter rebalances and less to recover when something fails. That is the rare optimization where the finance team and the on-call engineer want the same thing.