Kafka Disaster Recovery That Actually Works: MirrorMaker 2, Cluster Linking and Multi-Region Patterns

Start from RPO and RTO, per data class

The most common DR mistake we see is a single answer for the whole cluster. “We replicate everything to the other region” sounds complete, but it hides the real decisions. A payment instruction, a clickstream event and a cache-invalidation message do not deserve the same recovery design, and paying for the strictest design everywhere is how streaming DR becomes the most expensive line in the platform budget.

We start every engagement by classifying topics into a small number of data classes and agreeing two numbers for each:

• RPO — recovery point objective. How many seconds of acknowledged writes the business can afford to lose if a region disappears. Zero is possible, but only with synchronous replication, and synchronous replication has a latency cost on every write.
• RTO — recovery time objective. How long producers and consumers can be unable to do their work. This is dominated not by Kafka, but by how quickly clients are redirected and how quickly consumers find the right place to resume.

Two things usually fall out of this exercise. First, Tier 0 is a small fraction of the topics, which makes a zero-RPO design affordable for the data that needs it. Second, a surprising number of topics are Tier 3 — they can be recomputed from upstream data, so replicating them only adds cost and failure modes.

Three topologies, and what each one costs

With the numbers agreed, the topology choice is mostly mechanical. There are three patterns worth considering, and most enterprises end up running two of them side by side.

Stretched cluster: zero RPO, paid for in latency

A single Kafka cluster whose brokers are spread across three availability zones — or, with care, across nearby regions. Rack awareness (broker.rack) places each partition’s replicas in different zones, and with replication.factor=3, min.insync.replicas=2 and producers using acks=all, every acknowledged write exists in at least two zones before the producer hears back. Lose a zone and the cluster keeps serving from the remaining replicas: no client redirection, no offset translation, no data loss.

The price is that the slowest in-sync replica sets the write latency for every message. Across availability zones in one region that is usually a few milliseconds and well worth it. Across regions hundreds of kilometres apart it becomes tens of milliseconds per write, and throughput for latency-sensitive producers suffers. Stretching across regions also needs a third site for the controller quorum, because a two-site cluster cannot tell a partition from a failure. That is why we reserve cross-region stretching for small Tier 0 clusters, and use a three-zone, single-region stretch as the default building block for everything.

Active-passive: the workhorse

Producers and consumers use a primary cluster in one region. A replication process copies topics asynchronously to a standby cluster in another region. On a regional disaster, clients are pointed at the standby and resume. RPO equals the replication lag at the moment of failure — usually seconds, if the link is monitored and sized. This is the right design for most Tier 1 and Tier 2 data, and it keeps the application model simple: at any moment, there is one place to write.

Active-active: each region owns its writes

Both regions accept writes, but each writes to its own topics. Replication copies each region’s topics to the other under a distinguishing name, and consumers that need a global view read the local topic plus the remote copy (or an aggregate topic built from both). It gives near-zero RTO for producers — local writers never need to move — but the application must accept that events have two origins and that ordering is only guaranteed within one origin. Never let two regions write the same topic and replicate it in both directions under the same name; that creates loops and conflicting histories.

Replication tools: MirrorMaker 2, cluster linking and what is coming

Once the topology is chosen, the replication tool decides how hard failover will be. The differences that matter are not throughput — all of them can move a lot of data — but what happens to offsets, topic names and metadata.

MirrorMaker 2

MirrorMaker 2 is the open-source replication tool that ships with Apache Kafka. It runs on Kafka Connect as a set of connectors: a source connector that consumes from the source cluster and produces to the target, a checkpoint connector that records how consumer group offsets in the source map to offsets in the target, and a heartbeat connector used for monitoring and topology discovery.

• Topics are renamed by default. The default replication policy prefixes the source cluster alias, so orders becomes primary.orders on the target. That is exactly what active-active needs, and exactly what surprises active-passive teams on failover day, when every consumer subscribed to orders finds nothing. An identity replication policy keeps names unchanged; choose deliberately.
• Offsets are translated, not preserved. The target cluster writes the replicated records at its own offsets, which rarely match the source. Consumers resume using the checkpoint mapping, which is periodic and approximate, so a failover typically reprocesses a small window of messages.
• It is portable. It works between any Kafka-compatible clusters — self-managed, managed cloud services and vendor platforms — which makes it the default choice for heterogeneous estates and migrations.
• It is another system to operate. A Connect cluster with its own capacity, tasks, monitoring and upgrades. Under-sized replication tasks are a common cause of growing lag during traffic surges — which is precisely when you might need the standby.

Cluster linking

Confluent’s cluster linking replicates from inside the destination brokers rather than through an external Connect cluster. The destination cluster creates mirror topics that are byte-for-byte copies of the source partitions, with the same offsets. Mirror topics are read-only while linked; on failover they are promoted (or failed over) to become normal writable topics. Consumer group offsets and ACLs can be synchronised over the link as well.

• Offsets are preserved, so a consumer that read up to offset 1,000 on the source resumes at offset 1,000 on the destination. Duplicates after failover are bounded by consumer-offset sync frequency rather than by translation accuracy.
• No extra infrastructure to run and size, because replication is part of the broker.
• It requires Confluent — Confluent Platform or Confluent Cloud on the destination side — so it is a platform decision as much as a DR decision.

Broker-native mirroring in Apache Kafka

The Apache Kafka community has been discussing replication built into the broker itself, so that open-source clusters could get offset-preserving mirrors without an external Connect deployment. At the time of writing we treat this as emerging work to watch, not something to design a production DR plan around. If you are planning a DR architecture this quarter, choose between MirrorMaker 2 and cluster linking, and design the runbook so the replication layer can be swapped later.

Consumer offsets: where failovers actually go wrong

Replicating records is the easy part. The question that decides your RTO is: when a consumer group starts reading in the other region, where does it start? Get it wrong in one direction and you skip messages. Get it wrong in the other and you reprocess hours of data — or, with auto.offset.reset=earliest on a long-retention topic, the entire history.

The practical rules we apply:

• Sync committed offsets to the standby continuously, not as a step in the runbook. With MirrorMaker 2, enable group offset sync so target groups are pre-positioned; with cluster linking, enable consumer offset sync on the link.
• Make consumers idempotent. Assume a failover will replay some messages. Upserts keyed by business ID, deduplication on an event ID, or transactional sinks turn replay from a data problem into a non-event.
• Never rely on auto.offset.reset during failover. If a group has no committed offset on the standby, the runbook should say exactly where it starts — by timestamp, using offsets-for-times — not leave it to a default.
• Kafka Streams and Flink need extra care. Their state is derived from changelog topics and checkpoints. Either replicate the changelogs and checkpoint storage, or plan to rebuild state in the standby region and measure how long that takes — it is often the longest item in the RTO.

Producers and clients: how traffic actually moves

Kafka clients connect to a bootstrap address and then talk directly to brokers. Moving them to another cluster is a client concern, and it needs to be designed, not improvised:

• Indirect bootstrap addresses. Clients use a DNS name you control (for example kafka-orders.internal), never a broker host name. Failover changes the record, with a short TTL. For critical applications, a configuration service or feature flag that switches the cluster without redeploying is even better.
• Restart, don’t hope. Running clients cache broker metadata and will keep trying the old cluster. The runbook restarts or reinitialises producers and consumers after the switch, in a defined order.
• Idempotent producers with acks=all. Idempotence removes duplicates caused by retries within a cluster. It does not span clusters, so a producer that retried into the old region and then into the new one can still create a duplicate — another reason for idempotent consumers.
• Credentials and quotas exist on both sides. Service accounts, API keys, certificates and client quotas must already be valid on the standby. Discovering an expired certificate in the DR region during an outage is a classic way to turn minutes into hours.
• Stop writes to the old primary. If the old region comes back half-alive while clients are in the new one, you have two histories. Fence it: revoke producer ACLs, or take it out of DNS and keep it out until failback.

Schemas, ACLs and configuration: the metadata that must follow the data

A standby with all the bytes but none of the metadata cannot serve production. Four things have to be replicated or reproducible:

Failover and failback, step by step

Failover is the part everyone draws. Failback is the part that hurts, because by then the standby has accepted writes that the old primary has never seen. A good runbook covers both, with an owner and a time budget for each step.

The step most often skipped is the first one: who decides. Automatic failover of Kafka across regions sounds attractive, but a flapping network can trigger it while the primary is healthy, producing exactly the split-brain you were trying to avoid. For cross-region failover we recommend a human decision against clear criteria — with everything after the decision automated.

Failback deserves the same rigour. Records that were written to the old primary but never replicated before the disaster now conflict with the standby’s history. Those records need to be extracted, reviewed and, if still valid, re-published — not silently merged. Only then is the old primary rebuilt as a follower of the new one, caught up, and switched back during a quiet window.

For a cluster-linking estate, the core of the runbook is small. A sketch with placeholder names:

# Before: region B mirrors region A over a link named "a-to-b"
kafka-mirrors --describe --link a-to-b --bootstrap-server $B
# Check replication lag per mirror topic is within the RPO budget

# Disaster: A is unreachable. Make B's mirrors writable.
kafka-mirrors --failover --link a-to-b --topics orders,payments --bootstrap-server $B
# (use --promote instead when A is healthy and you want zero loss)

# Redirect clients, then restart producers before consumers
update-dns kafka-orders.internal --target $B --ttl 30

# MirrorMaker 2 equivalent: keep names stable and offsets synced
replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy
emit.checkpoints.interval.seconds = 10
sync.group.offsets.enabled = true
sync.group.offsets.interval.seconds = 10

Command names and flags vary by version and distribution; the point is that each step is a single, scripted, rehearsed action with a known outcome.

Choosing a design: the decision table

Rehearse it: game days, chaos and a replication-lag SLO

An unrehearsed DR plan is a hypothesis. The only way to know your RTO is to measure it, and the only way to keep it is to measure it regularly, because applications, topics and people change every month.

• Replication lag is an SLO, not a dashboard. Alert when lag for each data class exceeds its RPO budget, and track the percentage of time you were inside it. If lag regularly breaches the budget during traffic peaks, your RPO on paper is not your RPO in practice.
• Game days on a calendar. At least twice a year per critical domain, fail over for real in a controlled window: fence, promote, redirect, restart, verify, then fail back. Time every step against the runbook.
• Smaller chaos, more often. Kill a broker, a zone, a replication task or a Connect worker in a pre-production environment monthly. Inject network latency between regions and watch what replication lag does.
• Test the surge, not just the outage. Run a load test at peak traffic with the replication link active. Disasters and traffic spikes are correlated more often than anyone would like.
• Verify data, not just uptime. After each exercise, reconcile counts and sample records between regions, check consumers resumed where expected, and record the replay window.

The failure modes we plan for

DR readiness checklist

The bottom line

Kafka disaster recovery fails for predictable reasons: one design for all data, renamed topics, missing offsets, unreplicated schemas, clients that cannot be redirected and a failback nobody planned. None of these are solved by the replication tool alone. They are solved by agreeing RPO and RTO per data class, choosing the topology each class can afford, picking the replication tool whose offset and naming behaviour matches that topology, and treating the standby as code.

Then comes the part that separates a diagram from a plan: rehearsal. A failover that has been run, timed and improved twice is a failover you can trust at three in the morning. One that has only been drawn is a hope.