Deep learning: Inverted dropout

Apply dropout exactly as frameworks do, in both directions.

What to build

A program that applies inverted dropout to a layer's activations and to the gradient flowing back.

Requirements

  • The mask has 1 for a kept unit and 0 for a dropped one; keep is the probability of keeping a unit. In train mode use the mask and scale by 1/keep; in test mode pass everything through unchanged.
  • Print forward: and the outputs, then backward: and the input gradients, all with four decimal places.

Input

Line 1: the mode and keep. Line 2: the activations. Line 3: the mask. Line 4: the gradients arriving from the next layer.

Output

Two lines.

Constraints

At most 1,000 units; 0 < keep ≤ 1.

Example

Input:

train 0.5
1 2 3 4
1 0 1 0
0.1 0.2 0.3 0.4

Output:

forward: 2.0000 0.0000 6.0000 0.0000
backward: 0.2000 0.0000 0.6000 0.0000

Why it works

Dividing by keep makes each unit's expected output the same as without dropout, so the network can run at inference with no dropout and no rescaling.

Examples

Example 1 · Input
train 0.5
1 2 3 4
1 0 1 0
0.1 0.2 0.3 0.4
Expected output
forward: 2.0000 0.0000 6.0000 0.0000
backward: 0.2000 0.0000 0.6000 0.0000
Example 2 · Input
test 0.5
1 2 3 4
1 0 1 0
0.1 0.2 0.3 0.4
Expected output
forward: 1.0000 2.0000 3.0000 4.0000
backward: 0.1000 0.2000 0.3000 0.4000

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Forward: output = activation × mask / keep, so dropped units become 0.

Hint 2

Backward: gradient in = gradient out × mask / keep, in training; at inference both pass unchanged.

Solution

Show one correct solution (try it yourself first)
mode, keep = input().split()
keep = float(keep)
acts = list(map(float, input().split()))
mask = list(map(float, input().split()))
grads = list(map(float, input().split()))
if mode == "test":
    mask, keep = [1.0] * len(acts), 1.0
print("forward: " + " ".join(f"{a * m / keep + 0.0:.4f}" for a, m in zip(acts, mask)))
print("backward: " + " ".join(f"{g * m / keep + 0.0:.4f}" for g, m in zip(grads, mask)))

Lesson: Overfitting and regularization →