Talks - Anthony Shaw: Unlocking the Parallel Universe: Subinterpreters and Free-Threading in...

Python 3.12 introduced the ability to run multiple interpreters using a new API, with each having its own GIL (Global Interpreter Lock)

In Python 3.13 (beta), you can disable the GIL completely and run true parallel threads within a single interpreter

Subinterpreters provide parallelism with less overhead compared to multiprocessing, making them ideal for CPU-intensive tasks that can be segmented

Key execution models in Python:

• Coroutines (concurrent, smallest overhead)
• Threads (parallel with GIL disabled)
• Subinterpreters (fully parallel)
• Multiprocessing (fully parallel, largest overhead)

C extensions present the biggest challenge for GIL removal:

• Many assume GIL exists for thread safety
• Need significant updates to work without GIL
• Pure Python code works more readily with these changes

Web application architecture considerations:

• I/O operations dominate most web apps
• Subinterpreters work well for segmented tasks
• Can share memory between subinterpreters
• Good for horizontal scaling

Performance implications:

• Small tasks may see overhead exceed benefits
• Parallel execution benefits increase with CPU-intensive workloads
• Need to carefully consider task segmentation

Practical limitations:

• Not all libraries are compatible yet (e.g., Django)
• Requires special builds of CPython
• Still experimental in Python 3.13

Best use cases for subinterpreters:

• Long-running workers
• CPU-intensive tasks
• Workloads that can be clearly segmented
• Web applications with multiple workers