Atomicity and Consistency Across Databases and Messaging Systems by Andrei Shakirin

The outbox pattern helps achieve atomicity between database updates and message queue operations, ensuring both succeed or fail together

Key challenges when sending messages and updating databases:

• Messages can be duplicated even in success scenarios
• Transaction failures can lead to inconsistency
• Message ordering can be broken
• System scalability can be impacted

Traditional solutions and their limitations:

• Direct message sending is unreliable
• Distributed transactions (XA) are complex and don’t scale well
• Simple retries don’t guarantee consistency

Recommended implementation approach:

• Store events in an outbox table within the same database transaction
• Use a separate background job to reliably process and send messages
• Deploy the processing job as a separate process for better scalability
• Consider using timestamps instead of incremental IDs for ordering

Best practices for handling edge cases:

• Implement idempotency on the receiver side
• Skip very recent events to avoid duplicates
• Group multiple events into batches when possible
• Use broker-specific features for ordering when available

Trade-offs to consider:

• Events are sent with some delay (eventual consistency)
• Additional infrastructure needed for background job
• Duplicate messages still possible
• Need to maintain and monitor additional components

The pattern works best when:

• Strong consistency is truly required by business needs
• System can tolerate some delay in message delivery
• Application can handle duplicate messages
• Horizontal scaling is needed