Data, statistics and features

A dataset is a table: each row is an example, each column a feature (or the label). Before any model, look at the numbers.

Summing up a column

marks = [72, 85, 64, 90, 78, 0, 81, 69]      # one student was absent

n = len(marks)
mean = sum(marks) / n
ordered = sorted(marks)
median = (ordered[n // 2 - 1] + ordered[n // 2]) / 2 if n % 2 == 0 else ordered[n // 2]
variance = sum((x - mean) ** 2 for x in marks) / n
std = variance ** 0.5
print(f"with the 0:    mean {mean:.2f}, median {median}, std {std:.2f}")

present = [x for x in marks if x > 0]
mean2 = sum(present) / len(present)
print(f"without the 0: mean {mean2:.2f}")

One student's 0 (absent from the exam) pulls the mean down by almost 10 marks, but barely moves the median. Extreme values like that are outliers: check whether they are real, or mistakes in the data.

This is the population standard deviation (divide by n). Statistics books often divide by n − 1 for a sample; Python's statistics.stdev does that, and statistics.pstdev divides by n.

Scaling features

Features often have very different ranges: a house's area (500 to 3000 square feet) and its number of rooms (1 to 6). Methods that measure distances (k-nearest neighbours, k-means) would then care almost only about area, and gradient descent would take tiny steps in one direction and huge ones in another. So we scale each feature first:

• Min-max scaling: x′ = (x − min) / (max − min), which puts every value between 0 and 1.
• Standardization (z-score): z = (x − x̄) / σ, which gives the feature mean 0 and standard deviation 1. A z of 2 means "2 standard deviations above average".

area = [650, 1200, 900, 2400, 1500]      # square feet
rooms = [1, 3, 2, 5, 3]

def min_max(values):
    lo, hi = min(values), max(values)
    return [round((v - lo) / (hi - lo), 2) for v in values]

def z_scores(values):
    mean = sum(values) / len(values)
    std = (sum((v - mean) ** 2 for v in values) / len(values)) ** 0.5
    return [round((v - mean) / std, 2) for v in values]

print("area, min-max:", min_max(area))
print("rooms, min-max:", min_max(rooms))
print("area, z-scores:", z_scores(area))

cities = ["Dhaka", "Sylhet", "Dhaka", "Khulna"]
names = sorted(set(cities))
one_hot = [[1 if c == name else 0 for name in names] for c in cities]
print(names, one_hot)

Compute the min, max, mean and σ on the training data only, then use those same numbers to scale the test data and future data.

Categories and gaps

• Models need numbers. A category such as city ∈ {Dhaka, Sylhet, Khulna} becomes one-hot columns: Dhaka = [1, 0, 0], Sylhet = [0, 1, 0], Khulna = [0, 0, 1]. Numbering them 1, 2, 3 would wrongly suggest that Khulna is "three times" Dhaka.
• Missing values: drop the rows (if few), or fill each gap with the column's mean or median.

Mark lesson complete