Soham Butala- Prefect Workflows for Scaling Acoustic Fisheries Survey Pipelines | PyData Global 2023

Learn about EcoFlow, a Python package that streamlines acoustic fisheries survey pipeline workflows using Prefect, featuring customization, scalability, and a user-friendly interface.

Key takeaways
  • EcoFlow is a package developed using Prefect orchestration tool, designed to streamline the data processing workflow for acoustic fisheries survey pipelines.
  • It can be effortlessly deployed on various platforms, including virtual machines, EC2 instances, and local machines.
  • EcoFlow takes raw files and YAML configurations as input, and can process data using various modules, including Echo pipe, Echo shader, and Echo regions.
  • The package is designed to be flexible and adaptable, allowing users to define custom processing steps and add models to the YAML configuration.
  • EcoFlow uses Prefect’s concepts of flows and tasks, with flows being broader concepts that can have multiple tasks or subflows.
  • The package is customizable, allowing research groups to tailor it to their specific needs.
  • EcoFlow offers many use cases, including oceanography studies, identifying species, and tracking their populations.
  • It can be used to process large datasets, including those from the National Ocean and Atmospheric Administration (NOAA).
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a command-line interface (CLI) capabilities, allowing users to execute commands and set up environments.
  • The package is still in its early stages, and users need to clone the repo and install necessary dependencies to use it.
  • EcoFlow is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • The package is designed to be extensible, allowing users to add new modules and processing steps as needed.
  • EcoFlow provides a dashboard to monitor flow runs, display failures and successes, and provide centralized error handling.
  • The package is designed to be compatible with various cloud technologies, including AWS, Azure, and GCP.
  • EcoFlow uses Prefect’s concepts of blocks, which makes deployment and execution of workflows easy and flexible.
  • The package is designed to be easy to use, with a simple and intuitive interface.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go

More talks

Kourlitis et al. - Scientific Python ecosystem helps answering fundamental questions of the Universe

Privacy Detective: Sniffing Out Your Data Leaks for Android

Integrated AI and Enterprise Intelligence with LLMs and RAG by Felix Huchzermeyer

Kalyan Prasad - Python-Driven Portfolios: Bridging Theory and Practice for Efficient Investments

Meent & Tomaszewski - Fusing Machine Learning and Mixed Integer Linear Programming

Hugo Bowne-Anderson - Full-stack Machine Learning and Generative AI for Data Scientists

Deep dive into Go's build system - Daniel Marti

Building a Lightning Fast Firewall with Java & eBPF by Mohammed Aboullaite, Johannes Bechberger