Deep learning: Initialization scales

Compute the weight scales that keep a deep network trainable.

What to build

A program that computes the Xavier and He initialization scales for a layer.

Requirements

  • Inputs: fan-in (number of inputs) and fan-out (number of neurons).
  • Print xavier uniform limit X, xavier normal std Y and he normal std Z, each with four decimal places.

Input

One line: fan-in and fan-out.

Output

Three lines.

Constraints

Both between 1 and 100,000.

Example

Input:

2 2

Output:

xavier uniform limit 1.2247
xavier normal std 0.7071
he normal std 1.0000

Why it works

These scales make the variance of a layer's outputs, and of its gradients, about equal to that of its inputs, so signals keep their size through many layers.

Examples

Example 1 · Input
2 2
Expected output
xavier uniform limit 1.2247
xavier normal std 0.7071
he normal std 1.0000
Example 2 · Input
784 128
Expected output
xavier uniform limit 0.0811
xavier normal std 0.0468
he normal std 0.0505

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Xavier uniform limit √(6 / (in + out)); Xavier normal std √(2 / (in + out)); He normal std √(2 / in).

Hint 2

Use math.sqrt.

Solution

Show one correct solution (try it yourself first)
import math

fan_in, fan_out = map(int, input().split())
print(f"xavier uniform limit {math.sqrt(6 / (fan_in + fan_out)):.4f}")
print(f"xavier normal std {math.sqrt(2 / (fan_in + fan_out)):.4f}")
print(f"he normal std {math.sqrt(2 / fan_in):.4f}")

Lesson: Initialization and vanishing gradients →