Elijah ben Izzy & Stefan Krawczyk - Bridging Classic ML Pipelines with the World of LLMs

Testing

Bridging the gap between classic machine learning pipelines and Large Language Models (LLMs) using Directed Acyclic Graphs (DAGs) and the Hamilton tool, exploring modularity, reuse, and orchestration for efficient software engineering practices.

Key takeaways

DAGs (Directed Acyclic Graphs): Model ML and LLM pipelines as DAGs to represent complex computations.
Hamilton: A tool that helps structure software engineering practices and enables easy testing, modularity, and reuse.
LLMs (Large Language Models): Can be used to develop applications quickly, but require similar engineering efforts as classic ML.
Micro orchestration: Handles internal computations within steps.
Macro orchestration: Handles orchestration at a higher level.
Agent space: Hamilton helps structure agent capabilities.
Software engineering best practices: Hamilton offers a way to apply software engineering best practices to ML and LLM pipelines.
Modularity and reuse: Hamilton’s functions can be reused and composed to create new workflows.
Testing and debugging: Hamilton’s modular structure makes it easier to test and debug pipelines.
Productionization: Hamilton helps simplify the productionization process by providing a modular and reusable structure.
LLM pipelines: Can be developed quickly, but still require software engineering efforts.
Classic ML vs. LLMs: The two can be equated from a pipeline perspective, with similar structural similarities.

Elijah ben Izzy & Stefan Krawczyk - Bridging Classic ML Pipelines with the World of LLMs

More talks