Matt Harrison - Idiomatic Pandas | SciPy 2023

Unlock the full potential of pandas and simplify your data manipulation workflows with this talk on idiomatic pandas usage.

Key takeaways
  • Chaining operations in pandas can be confusing and may not be necessary.
  • The pipe method can be used to simplify code and make it easier to read.
  • When working with missing values, it’s important to understand why they are missing and how to handle them.
  • Aggregations are powerful and can be used to summarize data, but the syntax can be tricky.
  • The groupby method can be used to group data by one or more columns.
  • The describe method can be used to get summary statistics for a dataset.
  • The loc method can be used to select rows and columns from a dataframe.
  • The set_index method can be used to set the index of a dataframe.
  • The unstack method can be used to unpivot a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The query method can be used to query a dataframe using a SQL-like syntax.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The fillna method can be used to fill missing values in a dataframe.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used

More talks

Ashwin Srinath - cudf.pandas: The Zero Code Change GPU Accelerator for Pandas | PyData Global 2023

Talks - Bradley Dice: Hacking `import` for speed: how we wrote a GPU accelerator for pandas

Martin Trautmann - pydiverse pipedag: A library for data pipeline orchestration

Prawiroatmodjo et al. - How to bootstrap a Data Warehouse with DuckDB | SciPy 2024

Tutorials - Leah Berg, Ray: Feature Engineering is for Everyone!

Shaurya Agarwal - All Them Data Engines: Data Munging with Python circa 2023 | PyData Global 2023

Miki Tebeka - IPython: The Productivity Booster

Richard Iannone - Great Tables for Everyone | SciPy 2024