The Lost Order Mystery: Dual-Writes vs. Transactional Outbox
Simulate silent data loss under network failure with naive dual-writes vs guaranteed consistency with the Transactional Outbox pattern.
The Lost Order: Dual-Writes vs. Transactional Outbox
Imagine buying a laptop at a store register. Step 1: The cashier charges your card \$1,500. Step 2: The cashier phones the warehouse across town to dispatch your box. If the phone line dies right between step 1 and step 2, your card was billed, but the warehouse never got the memo! The Transactional Outbox prints a carbon copy into the local register drawer (the database transaction). Even if the phone line cuts out for an hour, the courier delivers the copy as soon as the line reconnects.
End-to-End Order & Message Lifecycle
INSERT INTO orders (id, amount, status)✓ SUCCESS (100)INSERT INTO outbox_events✓ ATOMIC COMMITorders.events.v1100 / 100 Messages ReceivedBy saving the order and the message event inside the exact same local database transaction, it is mathematically impossible for one to commit without the other. Even if the network cable to Kafka drops for 10 minutes, the background relay safely retries when connectivity returns. Zero ghost orders, 100% reliable fulfillment!
PostgreSQL Write-Ahead Log (WAL) streams committed events straight into Kafka without issuing heavy polling queries or causing table lock contention.
Engineering Defenses Against Dual-Write Data Corruption
Never perform network RPCs across different systems inside web handlers. Write the business state change and the event payload into the same database transaction. Either both commit, or neither does.
Deploy Debezium or inspect PostgreSQL WAL directly (pgoutput). The relay reads committed WAL records and publishes them to Kafka with at-least-once delivery, bypassing application code completely.
Because distributed relays guarantee at-least-once delivery, downstream consumers use idempotency tables (INSERT ... ON CONFLICT (event_id) DO NOTHING) to guarantee exactly-once processing effects.
Deep-Dive Technical Reading
Explore the engineering theory, mathematical foundations, and production Go implementations behind this simulator.
Experiencing architectural bottlenecks in production?
From connection starvation and p99 latency spikes to Kafka consumer lag, index optimization, and algorithm profiling.