Part 4 · Compute a prediction

Forward propagation is an ordered chain of calculations.

Data moves from left to right. At each layer we calculate scores, apply an activation, and pass the result onward. No parameter changes yet.

Calculate first

For x = [1.0, 0.5], which shape should emerge after multiplying by a 2 × 2 hidden-weight matrix?

A 1 × 2 row: one pre-activation score for each of the two hidden neurons.

Step 0: choose one row and initial parameters

Think of the two standardized features as usage frequency and account age. Suppose this row belongs to class 1.

\[x=\begin{bmatrix}1.0&0.5\end{bmatrix},\qquad y=1\]\[W^{[1]}=\begin{bmatrix}0.4&-0.2\\0.1&0.3\end{bmatrix},\quad b^{[1]}=\begin{bmatrix}0&0\end{bmatrix}\]\[W^{[2]}=\begin{bmatrix}0.7\\-0.5\end{bmatrix},\qquad b^{[2]}=0\]

Step 1: hidden weighted sums

\[Z^{[1]}=xW^{[1]}+b^{[1]}\]\[=\begin{bmatrix}1.0&0.5\end{bmatrix}\begin{bmatrix}0.4&-0.2\\0.1&0.3\end{bmatrix}+\begin{bmatrix}0&0\end{bmatrix}\]\[=\begin{bmatrix}1.0(0.4)+0.5(0.1)&1.0(-0.2)+0.5(0.3)\end{bmatrix}=\begin{bmatrix}0.45&-0.05\end{bmatrix}\]

The first hidden neuron sees positive evidence; the second receives a score just below zero.

Step 2: hidden activation

Apply tanh element by element:

\[A^{[1]}=\tanh(Z^{[1]})=\begin{bmatrix}\tanh(0.45)&\tanh(-0.05)\end{bmatrix}\approx\begin{bmatrix}0.4219&-0.0500\end{bmatrix}\]

Representation view: the original row [1.0, 0.5] has been re-expressed as two learned feature values [0.4219, -0.0500].

Step 3: output score and probability

\[Z^{[2]}=A^{[1]}W^{[2]}+b^{[2]}\]\[=0.4219(0.7)+(-0.0500)(-0.5)=0.3203\]\[A^{[2]}=\sigma(0.3203)=\frac{1}{1+e^{-0.3203}}\approx0.5794\]

The network currently assigns a probability of about 57.94% to class 1. During training we keep this probability. A threshold such as 0.5 is only needed later when converting probabilities to class decisions.

Cache the evidence for the return journey

Backpropagation will need intermediate values, especially Z[1], A[1], Z[2], and A[2]. A forward function therefore returns the prediction and stores a cache.

Z1 = X @ W1 + b1 A1 = np.tanh(Z1) Z2 = A1 @ W2 + b2 A2 = sigmoid(Z2) cache = {"Z1": Z1, "A1": A1, "Z2": Z2, "A2": A2}

Forward pass question: “What does the network predict with its current parameters?” The loss asks “How bad is that prediction?” Backpropagation asks “Which parameters were responsible, and by how much?”