We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
ElixirConf 2023 - Jeffery Utter - Scaling Teams with Kafka on the BEAM
Learn how to scale your teams and applications with Kafka on the BEAM, enabling real-time processing, parallel processing, and large data volumes, while simplifying communication paths and improving scalability and efficiency.
- Scaling teams with Kafka on the BEAM requires decoupling teams to reduce complexity.
- Kafka allows for independent development and deployment of services, enabling teams to work autonomously.
- Kafka partitions messages to enable parallel processing and handling large volumes of data.
- Producer-consumer relationships are crucial for message flow and can be managed with consumer groups.
- Streaming data enables real-time processing and reduces the need for batch processing.
- Kafka has built-in dead-letter queues, which can be used to handle message failures and retries.
- Schema Registry ensures data consistency and compatibility across services.
- Datadex, a service developed at The Score, helps ingest and process large volumes of data from various sources.
- Kafka and Elixir combine to provide a scalable and efficient platform for building real-time applications.
- Scalability can be achieved by using multiple instances of services and Load Balancing.
- Connecting Elixir apps to Kafka can be done using the Kafka-Elixir library.
- The Score uses Kafka to handle high volumes of message traffic, with over 125,000 messages processed per second.
- Kafka’s scalability is due to its distributed and partitioned log architecture, allowing it to handle large amounts of data.
- The speaker emphasizes the importance of simplifying communication paths between services and teams.
- Using Kafka as a distributed log enables real-time processing and reduces the need for batch processing.
- The speaker highlights the benefits of using Elixir with Kafka, including improved scalability and efficiency.
- Kafka’s low latency and high throughput make it suitable for real-time applications.
- The speaker shares his personal experience of using Kafka to build a scalable and efficient platform at The Score.
- Decoupling services and teams is crucial for building scalable applications.
- Using Kafka’s built-in features, such as partitions and consumer groups, enables efficient and scalable message processing.