Sponsor Presentation - Python Profiling State of the World

Key takeaways
  1. Sampling vs. Deterministic Profilers:

    • Sampling profilers are less accurate but have lower overhead.
    • Deterministic profilers are more accurate but have higher overhead.
  2. Flame Graphs vs. Flame Charts:

    • Flame graphs show the frequency of function calls.
    • Flame charts show the duration of function calls.
  3. Overhead:

    • CProfile has 15% overhead, which is considered low.
    • PySpy has 9% overhead with a 100 Hz sampling frequency.
    • Deterministic profilers can have significantly higher overhead, depending on the workload.
  4. Continuous Profiling:

    • Continuous profiling runs the profiler while your code is running, allowing you to capture performance data over time.
    • This is in contrast to traditional profiling, which only captures data for a single run of your code.
  5. Visualizing Profiling Data:

    • There are a variety of tools available for visualizing profiling data, including:
      • gprof2dot (converts CProfile data to a flame graph)
      • PySpy (a Python-specific profiling tool that can generate flame graphs and flame charts)
      • Speed Scope (a web-based tool for visualizing profiling data)
  6. Production Profiling:

    • When profiling in production, it is important to use a profiler with low overhead, such as a sampling profiler.
    • It is also important to consider the impact of profiling on your application’s performance.
  7. Memory Profilers:

    • Memory profilers are used to identify memory leaks and other memory-related issues.
    • They are not covered in this talk, but are an important tool for performance analysis.
  8. Distributed Tracing:

    • Distributed tracing is used to track the flow of requests through a distributed system.
    • It can be used to identify performance bottlenecks and other issues.
  9. Sentry:

    • Sentry is a company that provides a cloud-based error monitoring and performance monitoring platform.
    • They have developed a number of profiling tools, including a Python profiler that is used in production by thousands of customers.
  10. Future of Profiling:

    • There is a growing interest in using AI and machine learning to improve the accuracy and efficiency of profiling.
    • This could lead to the development of new profiling tools that are more effective and easier to use.