Linear regression

Linear regression predicts a number with a straight line:

ŷ = w × x + b

x is the feature, ŷ ("y hat") the prediction, w the slope (the weight) and b where the line crosses the y-axis (the bias). Training means finding the w and b that fit the examples best.

How wrong is a line?

For each example, the residual (error) is ŷ − y: the prediction minus the truth. One number for the whole dataset is the mean squared error:

MSE = mean of (ŷᵢ − yᵢ)²

Squaring makes every error count as positive, and punishes big errors much more than small ones. The best line is the one with the smallest MSE: the least-squares line.

The least-squares formulas

For one feature, calculus gives the best line directly:

w = Σ (xᵢ − x̄)(yᵢ − ȳ) / Σ (xᵢ − x̄)²
b = ȳ − w × x̄

where x̄ and ȳ are the means. The best line always passes through the point (x̄, ȳ).

size = [600, 800, 1000, 1200, 1500]     # square feet
price = [70, 95, 118, 140, 178]          # lakh taka

n = len(size)
x_mean = sum(size) / n
y_mean = sum(price) / n
w = sum((x - x_mean) * (y - y_mean) for x, y in zip(size, price)) / sum((x - x_mean) ** 2 for x in size)
b = y_mean - w * x_mean
print(f"price = {w:.4f} × size {'+' if b >= 0 else '-'} {abs(b):.2f}")

predicted = [w * x + b for x in size]
mse = sum((p - y) ** 2 for p, y in zip(predicted, price)) / n
baseline = sum((y_mean - y) ** 2 for y in price) / n
print(f"MSE {mse:.2f}, R² {1 - mse / baseline:.4f}")

for x in [700, 2000]:
    print(f"{x} sq ft -> {w * x + b:.1f} lakh")

The model learns that each extra 100 square feet adds about 12 lakh taka, and predicts prices for new sizes. The R² score compares the model's MSE with that of always guessing the average: 1 is a perfect fit, and 0 is no better than the average.

More features

With several features, ŷ = w₁x₁ + w₂x₂ + ... + wₖxₖ + b: one weight per feature. There is still a formula for the best weights (the normal equations, solved with matrices), but for many features or huge datasets it is slow, and gradient descent, the next lesson, is used instead.

Reading the result

• The weight says how much ŷ changes when that feature grows by 1 and the others stay the same.
• A good fit shows correlation, not causation: ice-cream sales and drownings rise together because both rise in summer, not because one causes the other.
• A line can only fit a straight-line pattern. A curve needs new features (x², log x) or another model.

Mark lesson complete