Compute the weight scales that keep a deep network trainable.
What to build
A program that computes the Xavier and He initialization scales for a layer.
Requirements
- Inputs: fan-in (number of inputs) and fan-out (number of neurons).
- Print
xavier uniform limit X,xavier normal std Yandhe normal std Z, each with four decimal places.
Input
One line: fan-in and fan-out.
Output
Three lines.
Constraints
Both between 1 and 100,000.
Example
Input:
2 2
Output:
xavier uniform limit 1.2247
xavier normal std 0.7071
he normal std 1.0000
Why it works
These scales make the variance of a layer's outputs, and of its gradients, about equal to that of its inputs, so signals keep their size through many layers.
Examples
Example 1 · Input
2 2
Expected output
xavier uniform limit 1.2247 xavier normal std 0.7071 he normal std 1.0000
Example 2 · Input
784 128
Expected output
xavier uniform limit 0.0811 xavier normal std 0.0468 he normal std 0.0505
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
Xavier uniform limit √(6 / (in + out)); Xavier normal std √(2 / (in + out)); He normal std √(2 / in).
Hint 2
Use math.sqrt.
Solution
Show one correct solution (try it yourself first)
import math
fan_in, fan_out = map(int, input().split())
print(f"xavier uniform limit {math.sqrt(6 / (fan_in + fan_out)):.4f}")
print(f"xavier normal std {math.sqrt(2 / (fan_in + fan_out)):.4f}")
print(f"he normal std {math.sqrt(2 / fan_in):.4f}")