Put the course together: impute, standardize and predict, with every statistic taken from the training data only, so nothing leaks from the test set.
What to build
A program that fits the preparation steps on training data and applies them, with a given logistic model, to test data.
Requirements
- Training median: the median of the known training values (mean of the two middle ones for an even count). Fill missing training values (
?) with it, then compute the mean and the population σ of the filled values. - Print
train median M, mean A, std Swith 2 decimal places. - For each test row: fill a missing value with the training median, compute z (0 if σ is 0) and p = 1 / (1 + e^(−(w z + b))), and print
z Z -> p P -> C, with Z to 2 decimals, P to 3 and C the predicted class (1 when p ≥ 0.5). - Finally print
test accuracy Awith 3 decimal places.
Input
Line 1: n, then n training lines x label. Then m, then m test lines x label. Then a line w b. Any x may be ?.
Output
m + 2 lines.
Constraints
1 ≤ n, m ≤ 500; at least one training x is known.
Example
Input:
5
10 0
20 0
? 1
40 1
50 1
3
35 1
? 0
12 0
2 0
Output:
train median 30.00, mean 30.00, std 14.14
z 0.35 -> p 0.670 -> 1
z 0.00 -> p 0.500 -> 1
z -1.27 -> p 0.073 -> 0
test accuracy 0.667
Why it works
Fitting the median, mean and σ on the training data only means the test rows are treated exactly like future data, so the test accuracy is an honest estimate.
Examples
5 10 0 20 0 ? 1 40 1 50 1 3 35 1 ? 0 12 0 2 0
train median 30.00, mean 30.00, std 14.14 z 0.35 -> p 0.670 -> 1 z 0.00 -> p 0.500 -> 1 z -1.27 -> p 0.073 -> 0 test accuracy 0.667
2 1 0 3 1 1 5 1 1 0
train median 2.00, mean 2.00, std 1.00 z 3.00 -> p 0.953 -> 1 test accuracy 1.000
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
Compute the median of the known training values, fill the training gaps with it, then the mean and σ of the filled training values. Never look at the test rows for these.
Hint 2
For each test row: fill a gap with the training median, z = (x − mean) / σ (0 if σ is 0), p = σ(w z + b).
Hint 3
Predict 1 when p ≥ 0.5.
Solution
Show one correct solution (try it yourself first)
import math
n = int(input())
train = [input().split() for _ in range(n)]
m = int(input())
test = [input().split() for _ in range(m)]
w, b = map(float, input().split())
known = sorted(float(x) for x, _ in train if x != "?")
k = len(known)
median = known[k // 2] if k % 2 else (known[k // 2 - 1] + known[k // 2]) / 2
filled = [median if x == "?" else float(x) for x, _ in train]
mean = sum(filled) / n
std = (sum((v - mean) ** 2 for v in filled) / n) ** 0.5
print(f"train median {median:.2f}, mean {mean:.2f}, std {std:.2f}")
right = 0
for x, label in test:
value = median if x == "?" else float(x)
z = (value - mean) / std if std > 0 else 0.0
p = 1 / (1 + math.exp(-(w * z + b)))
c = 1 if p >= 0.5 else 0
right += c == int(label)
print(f"z {z:.2f} -> p {p:.3f} -> {c}")
print(f"test accuracy {right / m:.3f}")