Cheuk Ting Ho Driving down the Memray lane Profiling your data science work | JupyterCon 2023

Python

Learn how to optimize your data science code and prevent memory errors using memory profiling with Python's `memory_profiler` library and `mprof` tool, and discover how to visualize memory usage with frame graphs.

Key takeaways

Memory profiling is crucial for data scientists to optimize their code and prevent memory errors.
Python’s memory allocation is complex, with different types of memory (stack, heap, and virtual memory) and garbage collection.
A frame graph is a visual representation of memory usage, showing the relationships between objects and functions.
Memory profiling tools can help identify memory leaks, optimize code, and prevent crashes.
The speaker recommends using the memory_profiler library to profile memory usage.
The memory_profiler library can be used to profile Python code and generate a frame graph.
The frame graph can be used to identify memory leaks and optimize code.
The speaker also recommends using the mprof command-line tool to profile memory usage.
The mprof tool can be used to generate a frame graph and identify memory leaks.
The speaker emphasizes the importance of understanding memory allocation and garbage collection in Python.
The speaker also emphasizes the importance of using memory profiling tools to optimize code and prevent memory errors.
The speaker recommends using the memory_profiler library and mprof tool to profile memory usage and generate a frame graph.
The speaker also recommends using the pandas library to work with large datasets and optimize memory usage.
The speaker emphasizes the importance of understanding the trade-offs between memory usage and performance.
The speaker also emphasizes the importance of using memory profiling tools to identify memory leaks and optimize code.
The speaker recommends using the memory_profiler library and mprof tool to profile memory usage and generate a frame graph.

Cheuk Ting Ho Driving down the Memray lane Profiling your data science work | JupyterCon 2023

More talks