pandas
pandas from Series and DataFrames to selection, missing data, transforms, groupby, reshaping, merges, text, dates, file input and output, and a full analysis.
- Level
- Intermediate
- Time
- About 28 hours
- Modules
- 5
- Lessons
- 15
- Exercises
- 45
pandas is the everyday tool of data analysis in Python: it loads tables, cleans them, combines them and summarises them in a few lines. This course teaches its core objects and operations, the rules behind them, and the traps that produce silently wrong numbers.
pandas cannot run in this site's Python runner, so every lesson shows real pandas code with results written as plain lists and numbers, and every runnable example and exercise is a plain-Python model of what pandas does. Implementing the rules yourself makes the real library predictable when you use it on your own computer.
What you will learn
- Create and inspect Series and DataFrames and select data with loc, iloc and masks
- Handle missing data and transform, sort and rank columns
- Summarise with groupby and reshape with pivot_table and melt
- Merge and stack tables safely, with validation
- Work with text and date columns and read and write files correctly
- Write clear, fast analyses as method chains from question to report
Skills you will gain
- Series and DataFrames
- Selection and filtering
- Missing data
- groupby and pivot tables
- Merging tables
- Text and date columns
Before you start
- The Python course: lists, dictionaries, loops, functions and string methods
- The NumPy course helps but is not required
- Nothing to install for the course; install pandas with pip later to use it on your own computer
Where it is used
- Data analysis and reporting
- Data cleaning pipelines
- Preparing data for machine learning
- Business and research analytics
Modules
Module 1: pandas foundations
Why pandas exists, Series with their index and alignment, and DataFrames with their shape, types and summaries.
- 1
Why pandas, and how this course runs it Intermediate
What pandas is for, its two structures (Series and DataFrame), how a DataFrame stores a table as labelled columns, and why the exercises here model pandas in plain Python.
- 2
Series: one labelled column Intermediate
Creating a Series, its values and index, selecting by label and by position, vectorised arithmetic, and alignment: why adding two Series lines values up by label and gives NaN where a label is missing.
- 3
DataFrames: tables of labelled columns Intermediate
Creating a DataFrame, its shape, columns, index and dtypes, looking at the first rows, selecting one or several columns, and summarising the numeric columns with describe.
Module 2: Selecting, cleaning and transforming
Selecting with loc, iloc and masks, finding and handling missing values, and creating and recoding columns.
- 1
Selecting rows and columns: loc, iloc and masks Intermediate
Choosing parts of a DataFrame by label with loc, by position with iloc, and by condition with boolean masks combined with &, | and ~, plus assigning values through a selection.
- 2
Missing data: finding, dropping and filling Intermediate
How pandas marks missing values with NaN and None, counting them with isna, how statistics skip them, and the two ways to deal with them: dropping rows with dropna or filling with fillna, ffill and bfill.
- 3
Transforming columns Intermediate
Creating and changing columns: arithmetic between columns, assign, map with a dictionary, apply with a function, astype, rename and drop, and binning numbers into categories with cut.
Module 3: Summarising and reshaping
Sorting and ranking, value counts, grouping with aggregation and transform, and reshaping with pivot_table, melt and crosstab.
- 1
Sorting, ranking and counting Intermediate
Ordering rows with sort_values and sort_index, sorting by several columns in different directions, where missing values go, ranking with ties, the top rows with nlargest, and counting values with value_counts.
- 2
groupby: split, apply, combine Intermediate
Grouping rows by one or more keys and summarising each group with sum, mean, count, size and agg, the difference between count and size, transform for per-row results, and filter for keeping whole groups.
- 3
Reshaping: pivot_table, melt and crosstab Advanced
Turning long data into a wide grid with pivot and pivot_table (and why duplicates need an aggregation), turning wide data back into long rows with melt, and counting combinations with crosstab, with margins for totals.
Module 4: Combining tables, text and dates
Merging and stacking tables with checks, cleaning and extracting text with the .str accessor, and parsing and resampling dates.
- 1
Merging and joining DataFrames Intermediate
Combining tables in pandas: pd.merge with inner, left, right and outer joins, keys with different names, suffixes, the indicator column, validate for catching duplicate keys, concat for stacking, and join on the index.
- 2
Working with text columns Intermediate
Cleaning and analysing text in pandas with the .str accessor: case, spaces and replacements, contains and startswith with case and missing values, split and expand, and pulling structured fields out of codes with str.extract and regular expressions.
- 3
Dates and times in pandas Intermediate
Working with dates in pandas: parsing with to_datetime and errors="coerce", the .dt accessor for years, months and day names, date differences as Timedeltas, and resampling a series to monthly totals with empty months filled.
Module 5: pandas in practice
Reading and writing files correctly, method chains and fast vectorised code, and a complete analysis from question to report.
- 1
Reading and writing files Intermediate
Getting data in and out of pandas: read_csv with separators, missing-value markers, column selection, types, dates and chunks; writing with to_csv and its quoting rules; and the other formats pandas reads and writes.
- 2
Method chaining, apply and performance Advanced
Writing clear, fast pandas: method chains with assign, query and pipe, when apply with axis=1 is acceptable and how to replace it with vectorised code, np.where and np.select, and avoiding the SettingWithCopyWarning.
- 3
A complete analysis with pandas Advanced
One analysis from start to finish: framing the question, loading and inspecting, cleaning, merging, grouping, ranking with nlargest, computing shares and per-customer summaries, checking the numbers and presenting the result.
Projects
- A monthly sales report A shop exports raw sales, products and customers as CSV files. Build a pandas script that cleans them, merges them with checks and produces a monthly report by category, region and customer.
- A survey data cleaner Clean a messy survey export with free-text answers, inconsistent categories, invalid dates and duplicates, and produce a tidy long table ready for analysis.
What to learn next
- Machine Learning Machine learning from scratch in plain Python: regression, gradient descent, regularization, k-NN, logistic regression, metrics, trees, ensembles and k-means.
- Database Management Systems Tables and SQL: queries, joins, grouping, keys, subqueries, ER design, normalization, relational algebra, indexes and transactions, on a real database in your browser.