Mike Kraus - Polishing Python: Preventing Performance Corrosion with Rust | PyData Amsterdam 2024

Python and Rust have complementary strengths - Python excels at ease of use and rapid development, while Rust offers high performance, memory safety, and concurrency

Bridging Python to Rust requires significant investment:

• Need to understand Rust ownership model and memory management
• No REPL for quick prototyping
• Requires compile-time checking and dealing with bindings
• Translation overhead between languages

Performance gains from Rust integration are most notable with:

• CPU-bound computations
• Large data volumes/batch processing
• Tasks that can leverage concurrency
• Custom use cases not covered by existing optimized packages

For many data science/ML workflows, existing optimized packages (NumPy, Polars, etc.) are better first choices than custom Rust integration

Key benefits of Rust:

• Zero-cost abstractions
• Thread safety through ownership model
• No garbage collector overhead
• Compiled performance
• Memory safety guarantees

Development considerations:

• Must compile in release mode for optimizations
• Need to make collections thread-safe for concurrent access
• Requires understanding of scopes and ownership
• More verbose than Python equivalents
• Steeper learning curve

Real-world adoption should focus on:

• Specific performance bottlenecks
• Areas not well-served by existing packages
• Cases where memory safety/concurrency are critical
• Projects that can justify the development overhead