Soham Butala- Prefect Workflows for Scaling Acoustic Fisheries Survey Pipelines | PyData Global 2023

Learn about EcoFlow, a Python package that streamlines acoustic fisheries survey pipeline workflows using Prefect, featuring customization, scalability, and a user-friendly interface.

Key takeaways
  • EcoFlow is a package developed using Prefect orchestration tool, designed to streamline the data processing workflow for acoustic fisheries survey pipelines.
  • It can be effortlessly deployed on various platforms, including virtual machines, EC2 instances, and local machines.
  • EcoFlow takes raw files and YAML configurations as input, and can process data using various modules, including Echo pipe, Echo shader, and Echo regions.
  • The package is designed to be flexible and adaptable, allowing users to define custom processing steps and add models to the YAML configuration.
  • EcoFlow uses Prefect’s concepts of flows and tasks, with flows being broader concepts that can have multiple tasks or subflows.
  • The package is customizable, allowing research groups to tailor it to their specific needs.
  • EcoFlow offers many use cases, including oceanography studies, identifying species, and tracking their populations.
  • It can be used to process large datasets, including those from the National Ocean and Atmospheric Administration (NOAA).
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a command-line interface (CLI) capabilities, allowing users to execute commands and set up environments.
  • The package is still in its early stages, and users need to clone the repo and install necessary dependencies to use it.
  • EcoFlow is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • The package is designed to be extensible, allowing users to add new modules and processing steps as needed.
  • EcoFlow provides a dashboard to monitor flow runs, display failures and successes, and provide centralized error handling.
  • The package is designed to be compatible with various cloud technologies, including AWS, Azure, and GCP.
  • EcoFlow uses Prefect’s concepts of blocks, which makes deployment and execution of workflows easy and flexible.
  • The package is designed to be easy to use, with a simple and intuitive interface.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go.
  • The package is designed to be scalable, allowing users to process large datasets and execute complex workflows.
  • EcoFlow provides a set of packages that can help users process and analyze large datasets.
  • The package is designed to be customizable, allowing users to tailor it to their specific needs.
  • EcoFlow provides a set of modules that can help users process and analyze large datasets.
  • The package is designed to be easy to use, with a Pythonic nature and ease of integrating with standard Python code.
  • EcoFlow provides a set of tools that can help users define and modify pipelines on the go

More talks

Catching Commits to Secure Infrastructure as Code • Rosemary Wang • GOTO 2023

Ionut Barbu & Tim Brakenhoff - How to build production-ready data science pipelines with Kedro

Juan Nunez-Iglesias - View, annotate, and analyze multi-dimensional images in Python with napari

Cedi: Modern Observability - Scalable Observability with the LGTM Stack: Harnessing the Power of Lok

Building an ML Factory | Adi Hirschtein, Iguazio (May 2022)

Stefan Kahl & Josef Haupt - AI-powered bioacoustic monitoring with BirdNET [PyData Prague]

Input Output + Syslog (iO+S): Obtaining Data From Locked iOS Devices via Live Monitoring

Deno 2.0 – Ryan Dahl, Node Congress 2023