RustConf 2023 - Async building blocks: A streaming Data Drama in Three Acts

Developing a system for real-time data processing requires careful planning.

Key takeaways

Key findings
- Developing a system to process messages in real-time can be complex and require careful planning
- Buffering messages can help manage order and prevent duplicate messages
- Paralleling execution of tasks can improve efficiency and speed
- Coordinating access to buffers is crucial to prevent conflicts and ensure data integrity
Technical concepts
- Streams and channels can be used to manage data flow and ordering
- Arc-mutex combinations can help synchronize access to buffers
- Futures can execute concurrently on a single thread
- async/await syntax can be used for asynchronous programming
Design considerations
- Factors to consider when designing a system for processing messages include message ordering, parallelization, and buffer management
- Buffering messages can help manage volume and prevent overload
- Careful planning and execution are necessary to ensure the system works properly
Possible solutions
- Using streams and channels can help manage data flow and ordering
- Implementing arc-mutex combinations can help synchronize access to buffers
- futures can execute concurrently on a single thread
- Using async/await syntax can be used for asynchronous programming

More talks