Introduction
Real data has holes: a sensor that failed for an hour, a survey question left blank, a price not entered yet. pandas marks every hole the same way, and gives you tools to find them, to work around them and to fill them. Handling missing values is a decision, not a technicality, because dropping or filling changes your results. This lesson shows the tools and the effect of each.
How it works
pandas marks a missing number with NaN ("not a number"), a special floating-point value. In text columns it also accepts Python's None. Because NaN is a float, a column of whole numbers with a gap becomes float64.
df = pd.DataFrame({
"name": ["Ada", "Rahim", None, "Omar", "Lena"],
"age": [36, None, 29, None, 41],
})
df["age"].dtype # float64: the gaps made the whole numbers floats
df.isna().sum().tolist() # [1, 2]: one missing name, two missing ages
df["age"].count() # 3: only the values that are there
df["age"].mean() # the mean of 36, 29 and 41: the gaps are skipped
isna() gives a mask that is True where a value is missing, and .sum() counts the Trues per column. Never test with == NaN: NaN is not equal to anything, not even itself.
Statistics skip missing values by default. count() counts the values present, mean() averages them, sum() adds them (and gives 0 for a column with nothing in it). That is usually what you want, but it means a mean can be based on far fewer rows than the table has.
There are two ways to deal with gaps. Dropping removes rows: df.dropna() drops every row with any missing value; dropna(subset=["age"]) looks only at the age column; dropna(how="all") drops only rows where everything is missing; dropna(thresh=2) keeps rows with at least two values present. Filling replaces gaps: fillna(0) uses a constant, fillna(df["age"].mean()) the column's mean, and fillna({"name": "unknown", "age": 0}) a value per column. For data ordered in time, forward fill, ffill(), repeats the last known value, and bfill() uses the next one.
len(df.dropna()) # 2: only Ada and Lena are complete
df["age"].fillna(0).tolist() # [36.0, 0.0, 29.0, 0.0, 41.0]
df["age"].ffill().tolist() # [36.0, 36.0, 29.0, 29.0, 41.0]
Example
# Missing values: None plays the part of NaN.
names = ["Ada", "Rahim", None, "Omar", "Lena"]
ages = [36, None, 29, None, 41]
print("isna:", [a is None for a in ages]) # df["age"].isna()
print("missing per column:", {"name": names.count(None), "age": ages.count(None)})
present = [a for a in ages if a is not None]
print("mean skips NaN:", round(sum(present) / len(present), 2)) # df["age"].mean()
print("count:", len(present)) # df["age"].count()
rows = list(zip(names, ages))
print("dropna:", [r for r in rows if None not in r]) # df.dropna()
print("fillna(0):", [0 if a is None else a for a in ages]) # df["age"].fillna(0)
filled, last = [], None # df["age"].ffill()
for a in ages:
last = a if a is not None else last
filled.append(last)
print("ffill:", filled)Line by line
- None stands in for NaN in the two columns.
isnais a mask; counting the Nones per column isisna().sum().- The mean and the count use only the three ages that are present.
dropnakeeps the rows with no missing value at all: Ada and Lena.- Filling with 0 and forward filling give two very different columns from the same data.
Expected result
isna: [False, True, False, True, False]
missing per column: {'name': 1, 'age': 2}
mean skips NaN: 35.33
count: 3
dropna: [('Ada', 36), ('Lena', 41)]
fillna(0): [36, 0, 29, 0, 41]
ffill: [36, 36, 29, 29, 41]
In the real world
Analysts report how much data is missing before any result, because a mean of three ages out of five rows is a weaker claim than a mean of five. Time series of prices and sensor readings are usually forward filled, since the last known value is the best guess. Machine learning pipelines fill gaps with a column's mean or median, and add a column saying the value was filled, so the model can tell.
Common mistakes
- Testing for missing values with == NaN, which is always False; use
isna(). - Dropping every row with any gap, which can throw away most of a table because of one sparse column.
- Filling with 0 when 0 is a real value, such as a temperature or a balance, which invents data.
- Forgetting that fillna and dropna return new tables, so the result must be assigned.
Best practices
- Count missing values per column before deciding anything.
- Drop by subset and fill per column, choosing a rule that makes sense for each.
- Note in your report how many values were dropped or filled.
Summary
pandas marks missing values with NaN (and None in text), and a gap turns an integer column into floats. isna().sum() counts gaps per column, and statistics skip them by default. dropna removes rows, with subset, how and thresh controlling which, and fillna, ffill and bfill replace gaps with a value, a statistic, or a neighbouring value.
Quick check
Key takeaways
- isna() marks missing values; isna().sum() counts them per column.
- Statistics skip NaN by default: count, sum and mean use only the values that are there.
- Decide per column whether to drop or fill, and record that decision.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.