Introduction
Before cleaning or analysing a table, you look at it: how big is it, what columns does it have, what kind of values are in each, and what do the first few rows look like? pandas makes those questions one-liners, and this lesson covers the ones you will type at the start of every piece of work, along with how to pick columns out of a table.
How it works
A DataFrame is usually created from a file, but it can also be built from a dictionary whose keys are column names and whose values are equal-length lists:
import pandas as pd
df = pd.DataFrame({
"name": ["Ada", "Rahim", "Mei", "Omar", "Lena"],
"age": [36, 24, 29, 41, 33],
"city": ["London", "Dhaka", "Tokyo", "Cairo", "Berlin"],
})
df.shape # (5, 3)
df.columns.tolist() # ['name', 'age', 'city']
df.index # RangeIndex(start=0, stop=5, step=1)
With no labels given, rows get a RangeIndex, 0 to 4. Each column has a dtype: here age is int64, while name and city are object, pandas' type for text and mixed values. df.dtypes lists them, and df.info() prints the dtypes together with how many values in each column are not missing.
df.head() shows the first five rows and df.head(2) the first two; df.tail() shows the last ones. df.describe() summarises every numeric column: the count of values that are not missing, the mean, the standard deviation, the minimum, the 25%, 50% (median) and 75% quantiles, and the maximum. For our ages, the count is 5, the mean 32.6, the minimum 24, the median 33 and the maximum 41.
Selecting columns uses square brackets. One name gives one column as a Series: df["age"]. A list of names gives a smaller DataFrame: df[["name", "age"]]. Note the double brackets: the outer pair selects, the inner pair is a Python list. Selecting rows by label or position, and by conditions, is the next lesson.
Most methods return a new object and leave the original alone. That makes it easy to explore safely: df.head(2) does not shorten df.
Example
# A DataFrame as columns: shape, columns, head and a per-column summary.
df = {
"name": ["Ada", "Rahim", "Mei", "Omar", "Lena"],
"age": [36, 24, 29, 41, 33],
"city": ["London", "Dhaka", "Tokyo", "Cairo", "Berlin"],
}
n = len(df["name"])
print("shape:", (n, len(df))) # df.shape
print("columns:", list(df)) # df.columns.tolist()
print("head(2):") # df.head(2)
for i in range(2):
print(" ", i, [df[col][i] for col in df])
ages = df["age"] # df["age"].describe()
print("age count:", len(ages))
print("age mean:", sum(ages) / len(ages))
print("age min/max:", min(ages), max(ages))
middle = sorted(ages)[len(ages) // 2]
print("age median:", middle)Line by line
dfis a dictionary of five-value columns, like the DataFrame above.- The shape counts rows (the length of a column) and columns (the number of keys).
head(2)takes the first two positions of every column.- The summary works on the
agecolumn only, asdescribereports numeric columns. - The median is the middle value after sorting, the 50% line of
describe.
Expected result
shape: (5, 3)
columns: ['name', 'age', 'city']
head(2):
0 ['Ada', 36, 'London']
1 ['Rahim', 24, 'Dhaka']
age count: 5
age mean: 32.6
age min/max: 24 41
age median: 33
In the real world
Data work starts with these checks, and so do debugging sessions: a shape with more rows than expected points to duplicated records, an object dtype on a number column points to text such as "n/a" mixed into the numbers, and a count in describe lower than the number of rows shows missing values. Experienced analysts look at df.head() and df.info() after every step that changes a table.
Common mistakes
- Writing df["name", "age"] with single brackets for two columns; it needs a list:
df[["name", "age"]]. - Building a DataFrame from lists of different lengths, which raises an error.
- Ignoring an object dtype on a numeric column, which makes sums join text and means fail.
- Reading describe's count as the number of rows: it counts values that are not missing.
Best practices
- Run shape, head, dtypes and describe on every new table.
- Select columns by name rather than by position, so code survives reordered files.
- Give columns short, lower-case names without spaces, so
df["unit_price"]is easy to type.
Summary
A DataFrame is a table of equal-length columns sharing an index, created from a dictionary or a file. shape, columns, dtypes, head, info and describe show its size, structure and contents. One column name gives a Series and a list of names gives a DataFrame, and methods return new objects rather than changing the original.
Quick check
Key takeaways
- A DataFrame is a dictionary of equal-length columns sharing an index.
- shape, columns, dtypes, head and describe are the first things to look at.
- df["col"] gives a Series; df[["a", "b"]] gives a DataFrame.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.