Deep learning: Gradients of one neuron

Backpropagate through a neuron with a sigmoid and a squared error.

What to build

A program that computes the loss of a sigmoid neuron on one example and the gradients of the loss.

Requirements

  • The neuron computes a = sigmoid(w·x + b); the loss is (a − y)².
  • Print loss L, then dw: and the gradient for each weight, then db: and the gradient for the bias, all with four decimal places.

Input

Line 1: the inputs. Line 2: the weights. Line 3: the bias and the target y.

Output

Three lines.

Constraints

At most 100 inputs.

Example

Input:

1 2
0.5 -0.25
0 1

Output:

loss 0.2500
dw: -0.2500 -0.5000
db: -0.2500

Why it works

The chain rule multiplies the loss's sensitivity to the output, the sigmoid's slope, and each input, giving the effect of every weight on the loss.

Examples

Example 1 · Input
1 2
0.5 -0.25
0 1
Expected output
loss 0.2500
dw: -0.2500 -0.5000
db: -0.2500
Example 2 · Input
1
0
0 0
Expected output
loss 0.2500
dw: 0.2500
db: 0.2500

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Forward: z = w·x + b, a = sigmoid(z), loss = (a − y)².

Hint 2

Backward: dL/da = 2(a − y), da/dz = a(1 − a), so dL/dz = dL/da × a(1 − a); then dL/dwᵢ = dL/dz × xᵢ and dL/db = dL/dz.

Solution

Show one correct solution (try it yourself first)
import math

xs = list(map(float, input().split()))
ws = list(map(float, input().split()))
b, y = map(float, input().split())
z = sum(x * w for x, w in zip(xs, ws)) + b
a = 1 / (1 + math.exp(-z))
dz = 2 * (a - y) * a * (1 - a)
print(f"loss {(a - y) ** 2:.4f}")
print("dw: " + " ".join(f"{dz * x:.4f}" for x in xs))
print(f"db: {dz:.4f}")

Lesson: Backpropagation →