Introduction
Once you know the individual pandas operations, the next skill is combining them into code that is easy to read and fast to run. Two habits help most: writing steps as a method chain, and replacing slow row-by-row Python with whole-column operations. This lesson also explains the most confusing pandas warning.
How it works
Method chains. Each method returns a new DataFrame, so the steps of an analysis can be written in order:
report = (
sales
.assign(total=lambda d: d["price"] * d["qty"])
.query("total > 10")
.sort_values("total", ascending=False, kind="stable")
.head(5)
)
assign adds columns (the lambda receives the DataFrame at that point of the chain), query filters with a readable expression, and pipe(f) inserts your own function: .pipe(remove_outliers, column="total"). Each line is one step that can be commented out while debugging. Note kind="stable": the default sort does not promise to keep tied rows in their original order.
apply. df.apply(f, axis=1) calls a Python function for every row. It handles any rule, but it runs Python code per row, which is often many times slower than column operations on large tables. Series.map(dict) replaces values through a dictionary and is fast.
Vectorised alternatives. Most row rules can be written on whole columns:
df["cost"] = df["zone"].map({"A": 5, "B": 8, "C": 12}) + 2 * (df["weight"] - 1).clip(lower=0)
df["size"] = np.where(df["total"] >= 25, "big", "small")
df["band"] = np.select([df["age"] < 18, df["age"] < 65], ["child", "adult"], default="senior")
np.where(condition, a, b) chooses between two values, np.select between several (the first true condition wins), and clip limits values to a range.
Chained assignment. df[df["age"] > 60]["band"] = "senior" may change a temporary copy and leave df unchanged, which is what the SettingWithCopyWarning warns about. Write df.loc[df["age"] > 60, "band"] = "senior" instead. Under Copy-on-Write, which becomes the default in pandas 3.0, chained assignment never changes the original, so the .loc form is the only correct one.
Example
# A method chain modelled step by step: assign, query, sort_values and a row-wise apply.
rows = [
{"item": "pen", "price": 1.5, "qty": 4},
{"item": "book", "price": 12.0, "qty": 2},
{"item": "bag", "price": 30.0, "qty": 1},
]
step = [dict(r, total=r["price"] * r["qty"]) for r in rows] # .assign(total=...)
step = [r for r in step if r["total"] > 10] # .query("total > 10")
step = sorted(step, key=lambda r: r["total"], reverse=True) # .sort_values("total", ascending=False)
for r in step:
size = "big" if r["total"] >= 25 else "small" # .apply(rule, axis=1)
print(r["item"], r["total"], size)Line by line
- The first step adds a total to each row, like
assign. - The second keeps rows with a total over 10, like
query. - The third sorts by total, largest first.
- The loop applies a rule to each row, like
apply(axis=1); in pandas,np.wherewould do it for the whole column at once.
Expected result
bag 30.0 big
book 24.0 small
In the real world
Analysis notebooks in industry are often a handful of long chains, each producing one table for a report. Replacing a row-wise apply with vectorised code is one of the most common speed-ups in data pipelines, turning minutes into seconds on millions of rows.
Common mistakes
- Using apply(axis=1) for simple arithmetic that column operations do faster.
- Writing df[mask]["col"] = value, which may not change df.
- Building very long chains without checking intermediate results.
- Relying on the default sort to keep ties in order.
Best practices
- Write one step per line in a chain, and use pipe for steps you reuse.
- Reach for column arithmetic, map, np.where and np.select before apply.
- Use .loc for every assignment to a subset of rows.
Summary
Method chains with assign, query, sort_values and pipe express an analysis step by step. apply(axis=1) runs Python per row and is slow; column arithmetic, map, clip, np.where and np.select do the same work on whole columns. Assign to subsets with .loc to avoid the SettingWithCopyWarning.
Quick check
Key takeaways
- Chains with assign, query, sort_values and pipe read like the steps of the analysis.
- Prefer vectorised column arithmetic, np.where and np.select to apply(axis=1).
- Assign with df.loc[mask, "col"] = value, never df[mask]["col"] = value.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.