Messaging: The fine line between awesome and awful - Laila Bougria - NDC Oslo 2024

Use queues instead of HTTP APIs for system boundaries to achieve better decoupling and resilience

Make idempotency a key design pattern when building distributed systems to handle duplicate messages and retries safely

Carefully choose between synchronous and asynchronous communication patterns based on business needs:

• Use synchronous for immediate responses
• Use asynchronous for eventual consistency and decoupling
• Consider combining both approaches when needed

Invest in proper testing of distributed scenarios:

• Test out-of-order message delivery
• Test duplicate message handling
• Test compensation/rollback flows
• Test timeout and failure scenarios

Implement proper observability using telemetry, logs and metrics to troubleshoot distributed system issues

Use the saga pattern for complex workflows requiring coordination across multiple services while maintaining eventual consistency

Define and enforce SLAs (Service Level Agreements) for asynchronous operations to set proper expectations

Consider using delayed/scheduled messages to enforce business rules and handle temporary failures

Find proper service boundaries based on data dependencies and change patterns rather than arbitrary splits

Avoid distributed transactions across service boundaries - if you need strong consistency, keep the data together in one service