Data science: Standardise without leakage

Scaling must use training statistics only, or the test set leaks into the model. Standardise test values with the training mean and standard deviation.

What to build

A program that standardises test values using statistics of the training values.

Requirements

  • Print train mean M sd S, with the population standard deviation, both to two decimals.
  • Then print test z: followed by each test value's z score, to two decimals, separated by spaces.

Input

Two lines: the training values, then the test values.

Output

Two lines.

Constraints

2 to 1000 training values, not all equal; 1 to 1000 test values.

Example

Input:

2 4 4 4 5 5 7 9
5 9 1

Output:

train mean 5.00 sd 2.00
test z: 0.00 2.00 -2.00

Why it works

The model only ever sees data scaled with the training numbers, so applying the same numbers to the test values keeps the test set unseen, exactly as new data will be in real use.

Examples

Example 1 · Input
2 4 4 4 5 5 7 9
5 9 1
Expected output
train mean 5.00 sd 2.00
test z: 0.00 2.00 -2.00
Example 2 · Input
10 20
15 30
Expected output
train mean 15.00 sd 5.00
test z: 0.00 3.00

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Compute the mean and the population standard deviation from the training values only.

Hint 2

Apply the same two numbers to every test value: z = (x − mean) ÷ sd.

Solution

Show one correct solution (try it yourself first)
train = list(map(float, input().split()))
test = list(map(float, input().split()))
mean = sum(train) / len(train)
sd = (sum((x - mean) ** 2 for x in train) / len(train)) ** 0.5
print(f"train mean {mean:.2f} sd {sd:.2f}")
print("test z:", " ".join(f"{(x - mean) / sd:.2f}" for x in test))

Lesson: The data science workflow →