Introduction
Before modelling data you describe it, and NumPy's statistics functions do that for whole arrays and along any axis. They are fast, but they hide a few choices that change the answers, such as which formula the standard deviation uses and how a percentile is computed between two values. This lesson makes those choices visible.
How it works
Centre and spread. np.mean(a), np.median(a), np.std(a) and np.var(a) work on the whole array or with axis=. The trap is ddof: np.std divides by n − ddof, and the default ddof is 0, the population formula. The sample standard deviation divides by n − 1, so pass ddof=1. pandas uses ddof=1 by default, so the same data can give two different "standard deviations":
a = np.array([2, 4, 4, 4, 5, 5, 7, 9])
print(np.std(a)) # 2.0
print(np.std(a, ddof=1)) # about 2.138
Percentiles. np.percentile(a, q) sorts the data and looks at position (n − 1) × q ÷ 100. When that position falls between two values, the default linear method interpolates: for [1, 2, 3, 4], the 25th percentile is at position 0.75, three quarters of the way from 1 to 2, which is 1.75. np.median(a) is the 50th percentile. Other methods exist (method="nearest", for example), and spreadsheets may use yet another, so small differences are normal.
Histograms. counts, edges = np.histogram(a, bins=k) splits the range from the minimum to the maximum into k equal bins. Each bin includes its left edge; the last bin also includes its right edge, so the maximum is counted. If all values are equal, NumPy widens the range by 0.5 on each side.
Correlation. np.corrcoef(x, y) returns a 2 × 2 matrix whose off-diagonal entry is Pearson's r between x and y.
Standardising. With a data matrix X (rows are observations, columns are features), broadcasting standardises every column at once:
Z = (X - X.mean(axis=0)) / X.std(axis=0)
A column whose standard deviation is 0 would divide by zero and give nan, so such columns are usually dropped or left at 0.
Example
# NumPy's statistics, modelled: median, percentile (linear), std with ddof, histogram counts.
data = [2, 4, 4, 4, 5, 5, 7, 9]
n = len(data)
mean = sum(data) / n
print("mean:", mean)
print("std (ddof=0):", round((sum((x - mean) ** 2 for x in data) / n) ** 0.5, 12))
print("std (ddof=1):", round((sum((x - mean) ** 2 for x in data) / (n - 1)) ** 0.5, 3))
s = sorted(data)
pos = (n - 1) * 25 / 100
low = int(pos)
print("percentile 25:", s[low] + (pos - low) * (s[low + 1] - s[low]))
lo, hi, bins = min(data), max(data), 3
width = (hi - lo) / bins
counts = [0] * bins
for x in data:
counts[min(int((x - lo) / width), bins - 1)] += 1
print("histogram counts:", counts, "edges:", [lo + i * width for i in range(bins + 1)])Line by line
- The mean and both standard deviations show the effect of ddof: 2.0 with n, about 2.138 with n − 1.
- The 25th percentile is at position 7 × 0.25 = 1.75 in the sorted data, between 4 and 4.
- The histogram uses 3 equal bins from 2 to 9; the maximum 9 is placed in the last bin rather than past it.
Expected result
mean: 5.0
std (ddof=0): 2.0
std (ddof=1): 2.138
percentile 25: 4.0
histogram counts: [4, 2, 2] edges: [2.0, 4.333333333333334, 6.666666666666667, 9.0]
In the real world
Data scientists standardise features before many machine learning methods, and compute percentiles for reports such as "the 95th percentile response time". Histograms are the first look at any numeric variable, and np.corrcoef gives a quick correlation matrix of many features.
Common mistakes
- Mixing NumPy's and pandas' standard deviations without noticing the different ddof.
- Expecting percentiles to be values from the data, when linear interpolation creates values between them.
- Forgetting that the last histogram bin is closed when checking counts by hand.
- Standardising test data with its own mean and std instead of the training data's.
Best practices
- State ddof whenever you report a standard deviation.
- Check
np.percentilewith a small example before trusting it on real data. - Keep the column means and stds from training to standardise new data the same way.
Summary
NumPy computes means, medians, variances and standard deviations along any axis, with ddof=0 by default. Percentiles interpolate linearly between sorted values, np.histogram counts values in equal-width bins with a closed last bin, np.corrcoef gives correlations, and broadcasting standardises every column in one expression.
Quick check
Key takeaways
- np.std uses ddof=0; pass ddof=1 for the sample standard deviation, which pandas uses by default.
- np.percentile(a, q) interpolates between sorted values at position (n − 1) × q ÷ 100.
- Reductions with axis=0 plus broadcasting standardise every column in one line.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.