Decoupling the Mainframe: Real-Time DB2 Offload to the Cloud with Kafka

Why offload, and why now

The mainframe is usually the most reliable system in the building. The problem is not that it fails; it is that every new consumer of its data makes it more expensive. A mobile app that checks balances, a fraud model that scores every card swipe, a customer-360 view, a regulator’s report, an AI assistant answering “where is my payment?” — each of these is mostly reads, and on a mainframe reads cost money in exactly the same currency as the transactions that pay the bills.

Four pressures tend to arrive together:

• MIPS cost. Software licensing on z/OS is commonly tied to peak consumption. Read-heavy workloads from digital channels push the peak up, and the bill follows the peak, not the average.
• Digital channels. Customers expect balances, statuses and histories in milliseconds, around the clock, at volumes the original batch-and-CICS design never anticipated.
• Analytics and AI. Models and agents need history, joins across domains and fresh context. Nightly unload files are a day old before anyone reads them.
• Skills and change speed. New features take longer when every one of them needs a COBOL change, a CICS transaction and a mainframe release window.

Getting changes out of DB2: the realistic options

Everything downstream depends on how changes leave DB2 for z/OS. There are three families of approach, and most large programmes end up using more than one.

Two things matter more than the product name. First, read the log, not the tables. Polling tables with timestamps misses deletes, misses intermediate states, and adds exactly the read load you are trying to remove. Second, keep transaction boundaries. A funds transfer that updates two rows must appear downstream as one unit, or you will briefly show a customer money that left one account and never arrived in the other.

A custom Connect source earns its place when off-the-shelf tooling cannot keep up with an initial load or a very large daily change volume, or when the mainframe team already produces change files that need to be streamed rather than batch-loaded. In that case we design it so that one logical source fans out to many tasks by key range or table, each task commits offsets only after Kafka acknowledges the batch, and the connector can resume from a known log position after any restart.

The reference architecture

The architecture separates four concerns that teams often blur together: capturing changes, decoding mainframe data, building a clean canonical model, and serving it to the systems that need it.

Raw topics: one per table, keyed by primary key

Raw topics mirror the source faithfully. One topic per source table, the record key is the table’s primary key, and the value carries the before image, the after image, the operation type, the commit timestamp, the log position and a transaction identifier. Keying by primary key is not a detail: it guarantees that every change to the same row lands in the same partition and is consumed in commit order. Use log compaction or a long retention on raw topics so they can be replayed to rebuild any downstream store.

Decoding: EBCDIC, packed decimal and copybooks

Mainframe data does not look like the data cloud systems expect. Character columns are EBCDIC in a specific code page, numbers may be packed decimal (COMP-3) or zoned decimal, dates arrive in a dozen formats including the classic 0001-01-01 placeholder, and VSAM or IMS-derived structures are described by COBOL copybooks with REDEFINES and OCCURS clauses. Many CDC tools convert DB2 column types for you; data that originates outside DB2, or arrives as extracts, usually needs copybook-driven decoding.

We generate schemas from the copybooks and DB2 catalog rather than writing them by hand, register them in Schema Registry, and keep the original field names in the raw layer so anyone can trace a value back to its source column.

// Copybook-derived Avro schema for the raw layer (abridged)
// 05 ACCT-NO        PIC X(12).
// 05 ACCT-BAL       PIC S9(13)V99 COMP-3.
// 05 ACCT-OPEN-DT   PIC X(10).
{
  "type": "record", "name": "ACCT_MASTER", "namespace": "mf.db2.core",
  "fields": [
    { "name": "ACCT_NO",      "type": "string" },
    { "name": "ACCT_BAL",     "type": { "type": "bytes", "logicalType": "decimal", "precision": 15, "scale": 2 } },
    { "name": "ACCT_OPEN_DT", "type": ["null", { "type": "int", "logicalType": "date" }], "default": null },
    { "name": "_op",          "type": { "type": "enum", "name": "Op", "symbols": ["I","U","D"] } },
    { "name": "_commit_ts",   "type": { "type": "long", "logicalType": "timestamp-micros" } },
    { "name": "_txn_id",      "type": "string" },
    { "name": "_log_pos",     "type": "string" }
  ]
}

Two rules keep decoding safe. Map packed decimals to Avro decimals, never to floating point; a rounding error on a balance is a production incident. And make code-page conversion explicit and tested, including the characters that differ between code pages, because names and addresses are exactly where silent corruption hides.

Canonicalization: from tables to business entities

Raw topics describe DB2 tables. Consumers want business entities: an account with its holder, product and current balance, not five normalized tables with cryptic column names. A Flink or Kafka Streams layer joins related raw topics, renames fields into a business vocabulary, applies PII masking, and emits curated topics that carry a data contract: a schema with documented meaning, an owner, a compatibility policy and freshness expectations.

-- Flink SQL: canonical account entity from raw CDC topics (simplified)
INSERT INTO curated_account
SELECT
  a.ACCT_NO                         AS account_id,
  c.CUST_ID                         AS customer_id,
  p.PROD_DESC                       AS product_name,
  a.ACCT_BAL                        AS current_balance,
  a.ACCT_OPEN_DT                    AS opened_on,
  a._commit_ts                      AS source_commit_ts
FROM raw_acct_master a
JOIN raw_acct_holder c ON a.ACCT_NO = c.ACCT_NO
JOIN raw_product     p ON a.PROD_CD = p.PROD_CD
WHERE a._op <> 'D';

Serving: operational store, lakehouse and events

Curated topics feed three kinds of consumer. An operational store (a relational database, document store or cache) answers API reads in milliseconds and is what actually takes load off the mainframe. An Iceberg lakehouse keeps full history for analytics, reporting and model training, written by a sink that commits in small, frequent snapshots. And event consumers — fraud scoring, notifications, AI agents that need fresh context — subscribe directly to the curated topics.

Strangler fig: reads first, writes much later

The safest offload follows the strangler-fig pattern: build the new path alongside the old one and move traffic over one capability at a time. The order matters. Reads move first because a cloud copy that is a few seconds behind is acceptable for almost every read. Writes move last, and only for capabilities where the business is ready to make the cloud the system of record.

In phase 4, when a domain’s writes move to the cloud, the direction of replication for that domain reverses: the cloud becomes the system of record, and changes flow back to the mainframe only for the batch jobs and partners that still need them. That reversal is a deliberate, rehearsed cutover for one domain, never a gradual drift where both sides accept writes.

Ordering and transactional consistency

Kafka guarantees order within a partition, not across a topic. That is enough for most of the problem, if the design respects it.

• Key by primary key. All changes to a row go to one partition, in commit order. Changing the key or the partition count later breaks this guarantee for in-flight data, so size partitions for growth at the start.
• Carry the transaction. Each change record carries its transaction identifier and commit log position. Where a consumer must see a multi-row transaction atomically — the two legs of a transfer — the canonicalization layer buffers by transaction and emits the entity only when the whole transaction has arrived.
• Make sinks idempotent. Restarts and rebalances will replay some records. Sinks upsert by key and ignore changes with an older log position than the one already applied, so replays cannot move data backwards.
• Use exactly-once processing where it pays. Kafka transactions and Flink checkpoints give end-to-end exactly-once between Kafka topics; for external sinks, idempotent upserts are usually simpler and just as safe.

Initial load and the CDC handoff

CDC captures changes from the moment it starts. The tables already hold years of data, so every offload needs an initial load, and the handoff between the load and the change stream is where data is most often lost or duplicated.

1. Start CDC first and record the log position where capture began.
2. Take the initial load from a consistent point: an image copy, an unload from a quiesced or point-in-time copy, or the CDC tool’s own refresh. For very large tables, split by key range and load in parallel.
3. Replay buffered changes from the recorded position with idempotent upserts. Rows changed during the load are overwritten with their latest committed state, and the log position check stops older changes from winning.
4. Reconcile before anyone consumes. Counts and checksums must match the source for a defined window before the target is opened to traffic.

Raw topic retention has to cover the full duration of the load plus a margin. If a load takes eighteen hours and the topic keeps twelve, the handoff fails silently and the gap only appears in reconciliation — if you are reconciling.

Reconciliation and latency SLOs

Offload succeeds or fails on trust. The first time a branch employee sees a different balance in the new app than on the green screen, the programme loses a year of credibility. Reconciliation is how you find differences before they do.

Set latency SLOs per consumer, not per platform. An inquiry API might need p99 under five seconds from mainframe commit to cloud read; a regulatory report might accept fifteen minutes; a fraud model might need sub-second for one table and not care about the rest. Measuring end-to-end lag from the source commit timestamp, not from when Kafka received the record, is what makes these numbers honest.

Schema evolution and data contracts

Mainframe schemas change rarely, but they do change, and when they do the change usually arrives as a copybook update in a release nobody on the cloud side was told about.

• Raw layer: mirror the source. Register schema changes automatically from the DB2 catalog or copybook repository, with backward compatibility so existing consumers keep working.
• Curated layer: contracts with owners. Curated topics change through review. A new source column does not appear in a contract until someone decides what it means.
• Fail loudly, not silently. A record that does not match its schema goes to a dead-letter topic with an alert. It never gets dropped or half-parsed.
• Treat topics as APIs. Version curated topics, publish them in a catalog with owner, meaning and SLO, and deprecate old versions on a schedule.

Security: identities, encryption and PII

Mainframe security is mature, and offload must not be the moment it gets weaker. Data that was protected by RACF (or ACF2 or Top Secret) profiles on the mainframe should be at least as protected in Kafka and the cloud.

• Dedicated capture identities. The CDC agent runs under its own mainframe user with the minimum authority to read the log and the tables in scope, separate from application and batch IDs.
• Encryption everywhere. TLS from the capture agent to Kafka, mTLS or OAuth between clients and brokers, and encryption at rest on brokers, tiered storage and every target.
• Least-privilege topics. Raw topics are readable only by the canonicalization service. Everyone else reads curated topics, which is also where masking happens.
• Mask PII in motion. Account numbers, national IDs and card data are tokenized or masked in the processing layer, so curated topics and downstream stores never hold more than each consumer needs.
• Audit end to end. Every read of a curated topic and every target store is attributable to a service identity, and the mapping from mainframe profile to cloud grant is documented.

The cost model: where MIPS actually go down

Offload pays for itself only if the mainframe does less work. It is worth being precise about where that reduction comes from, and what it costs.

Measure before and after with the mainframe team’s own capacity reports, at the peak hour that drives licensing. A dashboard that shows “reads moved to cloud” next to “peak consumption this month” is the slide that funds phase three.

CDC, batch unload or dual-write?

Why dual-write is a trap. Asking the application to write to DB2 and publish to Kafka looks simple. But the two writes are not atomic: if DB2 commits and the publish fails, or the publish succeeds and DB2 rolls back, the systems diverge and nothing records that they did. It also requires changing every COBOL program that writes the data, which is exactly the change velocity problem offload is meant to solve. Reading the database log gives you every committed change, in order, without touching application code.

A phased roadmap

The failure modes we plan for

Readiness checklist

The bottom line

Real-time mainframe offload is one of the few modernization projects that can pay for itself while it is still running. The mainframe keeps committing transactions; Kafka carries every committed change out within seconds; the cloud answers the reads, the reports, the models and the agents that used to drive the peak.

It works when it is treated as an engineering discipline: read the log, keep transactions intact, decode with copybook-driven schemas, put contracts between raw and curated data, load and hand off without a gap, reconcile everything, and move reads long before you even discuss writes. Done that way, every domain you add is cheaper than the last, and the mainframe gets quieter every quarter.