How to Schedule Tasks with Celery and Django

- Andrew Holmes
Django

Learn how to use Celery and Django to execute tasks asynchronously, scale your app, and manage complex workflows while preventing data loss and job failures.

Key takeaways
  • Use Celery to execute tasks asynchronously and scale your application horizontally.
  • Set up a result store to keep track of task results and prevent data loss.
  • Use Celery beat for scheduled tasks, but only run one process per queue to avoid duplicate tasks.
  • Minimize the size of task arguments and results to reduce message overhead.
  • Use the star function to group tasks and run them in parallel.
  • Define a task with the delay function to schedule tasks at a later time.
  • Use Redis as a result store and message broker, but be aware of its limitations.
  • Configure Celery to hold a task that exceeds its visibility timeout to prevent job failure.
  • Monitor Celery tasks and results through the Django admin interface.
  • Utilize Celery’s built-in features, such as group and chain, to manage complex workflows.

More talks

DjangoCon Europe 2024 | Django, SQLite, and Production Eve Freedman The Django Jigsaw Puzzle: Aligning All the Pieces with Will Vincent - DjangoCon US 2022 Will Vincent Inside Fiber: the In-Depth Overview You Wanted a TLDR for – Matheus Albuquerque, React Summit 2022 Matheus Albuquerque GraphQL Caching Demystified - Matteo Collina, GraphQL Galaxy 2021 Matteo Collina The (big) picture of debt - DPC2022: Scaling: from 0 to 20 million users Ilya Etingof Vue + Django: Combining Django Templates and Vue Single File Components without compromise Mike Fearing Scaling Papaya Global Past $100M ARR -