Rosa Gutiérrez - Solid Queue internals, externals and all the things in between - Rails World 2024

Solid Queue is a new database-backed ActiveJob implementation that will be default in Rails 8, focused on simplicity and built-in support for MySQL, PostgreSQL and SQLite

Core design uses multiple tables to separate concerns:

• Jobs table for storing actual job data
• Separate tables for scheduled, ready, and claimed jobs
• Process registry table for worker heartbeats
• Additional tables for paused queues and concurrency controls

Key features include:

• Bulk operations support
• Concurrency controls via semaphores
• Sequential job processing
• Scheduled/delayed jobs
• Efficient polling using SELECT FOR UPDATE SKIP LOCK
• Support for multiple queues with priorities
• Built-in dashboard (Mission Control)

Performance optimizations:

• Minimal polling queries (1 query per queue)
• Optimized indexes for queue polling
• Separate database recommended for high-volume scenarios
• Claims ~400 jobs/second in production environments

Production-tested at scale:

• Powers Hey’s email/calendar service
• Handles ~20 million jobs per day
• Runs across 74 VMs with 800 workers
• No significant impact on application response times
• Each feature tested in production before release

Design priorities:

• Simplicity over complexity
• Leverage existing database capabilities
• Minimal dependencies
• Battle-tested in real production environments
• Active community involvement in development