We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
ElixirConf 2023 - Anthony Accomazzo - Building recursive workflows with Broadway
Learn how to build recursive workflows with Broadway, a library that provides a simple and high-level abstraction for work processes, ensuring easy management of state and concurrency.
- Broadway is a library that helps build recursive workflows in Elixir, making it easier to handle complex workflows.
- The library aims to provide a simple and high-level abstraction for work processes, making it easier to manage state and concurrency.
- The author’s experience with building a data pipeline for Sequin is used to illustrate the benefits of using Broadway.
- The data pipeline was built using Broadway and features a producer, manager, and workers, which handle tasks concurrently.
- Makers should be designed with a “do one thing well” paradigm, whereas Managers should have a variety of tasks.
- Acknowledgments are used to handle job failures and retries.
- The use of Broadway allowed the authors to scale up the pipeline to handle millions of jobs, while also reducing the memory footprint.
- The library provides a rate limiter to handle API request limits.
- The pipeline uses a process key value store to share data between processes.
- The author suggests that the limiting factor in API polling is usually CPU, not memory.
- Simple optimizations, such as handling buffer copies and avoiding loads, can greatly improve performance.
- Broadway allows for easy management of state and concurrency, making it a great choice for complex workflows.
- Some Elixir libraries can make it easy to do complex things, such as recursion, without needing to write low-level code.