Matt Harrison - Idiomatic Pandas | SciPy 2023

Unlock the full potential of pandas and simplify your data manipulation workflows with this talk on idiomatic pandas usage.

Key takeaways
  • Chaining operations in pandas can be confusing and may not be necessary.
  • The pipe method can be used to simplify code and make it easier to read.
  • When working with missing values, it’s important to understand why they are missing and how to handle them.
  • Aggregations are powerful and can be used to summarize data, but the syntax can be tricky.
  • The groupby method can be used to group data by one or more columns.
  • The describe method can be used to get summary statistics for a dataset.
  • The loc method can be used to select rows and columns from a dataframe.
  • The set_index method can be used to set the index of a dataframe.
  • The unstack method can be used to unpivot a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The query method can be used to query a dataframe using a SQL-like syntax.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The fillna method can be used to fill missing values in a dataframe.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used to get the count of unique values in a column.
  • The astype method can be used to convert the data type of a column.
  • The dtypes attribute can be used to get the data type of a column.
  • The shape attribute can be used to get the shape of a dataframe.
  • The info method can be used to get information about a dataframe.
  • The describe method can be used to get summary statistics for a dataframe.
  • The head method can be used to get the first few rows of a dataframe.
  • The tail method can be used to get the last few rows of a dataframe.
  • The sample method can be used to sample a dataframe.
  • The dropna method can be used to drop rows with missing values.
  • The fillna method can be used to fill missing values in a dataframe.
  • The pivot_table method can be used to create a pivot table.
  • The groupby method can be used to group data by one or more columns.
  • The agg method can be used to apply aggregations to a dataframe.
  • The value_counts method can be used

More talks

The PyArrow revolution in Pandas — Reuven M. Lerner

Ian Ozsvald & Giles Weaver - Pandas 2, Dask or Polars? Tackling larger data on a single machine

Talks - Jeff Glass: Making Your Documentation Interactive with PyScript

Tutorial Boost your Data Science skills with the new Python in Excel [PyCon DE & PyData Berlin 2024]

Matt Harrison - An Introduction to Pandas 2, Polars, and DuckDB | PyData Global 2023

Hadi Abdi Khojasteh - Pandas Roadmap and Beyond [PyData Prague #19]

Shaurya Agarwal - All Them Data Engines: Data Munging with Python circa 2023 | PyData Global 2023

Monahan et al. - In Process Analytical Data Management with DuckDB | SciPy 2023