We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
Ville Tuulos - Compute anything with Metaflow | PyData Global 2023
Explore the evolving landscape of compute power, from vertical to horizontal scaling, and discover how Metaflow can help you build a robust machine learning infrastructure that meets the needs of AI and LLMs.
- Compute power has increased exponentially over the years, with a massive increase in processing power and relatively stable cost.
- The changing landscape of compute has led to a shift from vertical scaling to horizontal scaling.
- The growth in cloud computing has provided more options for compute resources, with providers like AWS, Azure, and Google Cloud.
- The importance of considering the whole stack of machine learning infrastructure, including data, compute, and orchestration.
- The rise of AI and LLMs has led to a new generation of compute needs.
- The need for more diverse landscape of compute options, with different tools and frameworks for different use cases.
- The value of Python as a compute language, with frameworks like Pandas, NumPy, and Dask.
- The importance of considering the capabilities of the hardware, such as GPUs and multi-core processors.
- The growth of distributed computing, with frameworks like Apache Spark and Hadoop.
- The importance of considering the sustainability and cost-effectiveness of compute resources.
- Metaflow as a compute framework that focuses on the full stack of machine learning infrastructure.
- The importance of considering the organizational and human concerns of compute resources.