Python-based ML and HPC workflows in the Cloud for science and engineering I PyData Chicago 2022

Parcel, a Python-based parallel scripting library, simplifies complex workflows in science and engineering by orchestrating applications, managing resources, and tracking progress in the cloud.

Key takeaways
  • Parcel is a Python-based parallel scripting library that can be used to orchestrate complex workflows involving multiple applications and data sources.
  • Parcel provides a simple and intuitive interface for defining and managing computational resources, such as cloud instances and storage.
  • Parcel can be used to run applications in parallel on multiple workers, either on a local cluster or in the cloud.
  • Parcel can be used to manage data dependencies between applications, ensuring that each application has the data it needs to run.
  • Parcel can be used to track the progress of workflows and to monitor the status of individual applications.
  • Parcel can be used to orchestrate workflows that involve both Python and non-Python applications.
  • Parcel can be used to integrate machine learning models into workflows, allowing for the automation of complex tasks.
  • Parcel is open source and available under the Apache License 2.0.
  • Parcel is supported by a team of developers and contributors who are committed to making it the best possible tool for orchestrating complex workflows.
  • Parcel is used by a variety of organizations, including universities, research institutions, and companies, to orchestrate complex workflows in a variety of domains, including weather forecasting, ocean modeling, and drug discovery.