A Kafkaesque Series of Events • Lily Mara • YOW! 2022

Kafka handles 8 billion push notifications daily with at-least-once delivery, ensuring scalability, fault tolerance, and high performance. Learn how.

Key takeaways

To achieve at-least-once delivery, a Kafka topic with 4 partitions was created for customer subscriptions.
A commit buffer was used to store in-process commits and ensure that everything is processed.
Subpartition processing was used to process messages concurrently, allowing for a fair distribution of messages on processors.
Offset IDs were used to ensure that messages are processed in order.
To control concurrency, Kafka partitions were used to assign messages to workers.
To limit the number of messages in the topic, a property was set to 10.
sns GraphQL API was used to generate and send updates.
Kafka was used to handle the load and provide at-least-once delivery.
The system was able to process 8 billion push notifications per day.
The solution was able to decrease latency and improve performance.
The benefits of using Kafka included increased scalability, fault tolerance, and high performance.
To achieve exactly-once delivery, commit batching was used to group updates together before committing them.
Partitioning was used to distribute messages across different processors.
To improve observability, metrics were collected and stored in a centralized logging platform.
To reduce latency, the solution used a caching layer to store frequently accessed data.
To improve concurrency, multiple workers were used to process messages concurrently.
The system was able to handle high traffic and low latency simultaneous requests.

More talks