Rosa Gutiérrez - Solid Queue internals, externals and all the things in between - Rails World 2024

SolidQueue is a new database-backed job processing system designed to be the default backend for ActiveJob in Rails 8

Key features include:

• Support for multiple databases (MySQL, PostgreSQL, SQLite)
• Built-in concurrency control
• Queue prioritization and ordering
• Scheduled/delayed jobs
• Bulk operations
• Pause/resume queues
• Process monitoring and job supervision

Performance metrics:

• Handles 90+ million jobs per day
• Takes ~110 microseconds per polling query
• Runs 4,900 polling queries per second
• Processes 400 jobs per second
• Operates with 800 workers across 74 VMs

Architecture highlights:

• Uses separate tables for different job states (ready, claimed, scheduled)
• Implements process registry for worker management
• Employs database locking (SELECT FOR UPDATE SKIP LOCKED) to prevent job duplication
• Keeps polling table small for performance
• Performs limit checking at enqueue time rather than poll time

Key improvements over existing solutions:

• Simpler architecture compared to Redis-based systems
• Built-in sequential job processing
• No additional infrastructure requirements
• Better handling of scheduled jobs
• Improved bulk operation performance
• Native database integration

Originally extracted from production use at Hey.com, where it successfully replaced Resque and addressed scaling challenges