RNN Forward Propagation

This note writes out the RNN forward pass completely, with the exact formulas, then works through a small numerical example across several time steps by hand.

The Complete Formulas

\(\tanh\) is the traditional choice of non-linearity for the hidden-state update (see Tanh Function) — its zero-centered, bounded range helps keep the hidden state's magnitude from growing unboundedly across many time steps. The output equation \(\mathbf{y}_t\) is a plain linear projection (or followed by an appropriate activation, like softmax, depending on the task).

Numerical Example — Three Time Steps

Scalar simplification for clarity: \(W_{xh}=0.5\), \(W_{hh}=0.8\), \(b_h=0\), \(h_0=0\). Inputs: \(x_1=1, x_2=0.5, x_3=-1\).

Notice how \(h_3\) reflects a blend of the current input (\(-1\), pulling it negative) and the accumulated memory from \(h_2\) (positive, pulling it back up) — exactly the "combining new information with carried memory" behavior described conceptually in Hidden State.

Code — Verifying With PyTorch

import torch
import torch.nn as nn

rnn_cell = nn.RNNCell(input_size=1, hidden_size=1, nonlinearity='tanh')
with torch.no_grad():
    rnn_cell.weight_ih.fill_(0.5)   # W_xh
    rnn_cell.weight_hh.fill_(0.8)   # W_hh
    rnn_cell.bias_ih.fill_(0.0)
    rnn_cell.bias_hh.fill_(0.0)

h = torch.zeros(1, 1)
for x_val in [1.0, 0.5, -1.0]:
    x = torch.tensor([[x_val]])
    h = rnn_cell(x, h)
    print(h.item())
# approximately: 0.4621, 0.5507, -0.0593 -- matches the hand-worked example

Processing a Full Sequence — The Complete Loop

import torch.nn as nn

rnn = nn.RNN(input_size=10, hidden_size=20, batch_first=True)
sequence = torch.randn(2, 6, 10)   # batch of 2, 6 time steps, 10 features

all_hidden_states, final_hidden_state = rnn(sequence)
print(all_hidden_states.shape)   # (2, 6, 20) -- h_t for every one of the 6 time steps
print(final_hidden_state.shape)   # (1, 2, 20) -- only h_6, the final hidden state

Common Mistakes

• Forgetting the bias terms when computing forward propagation by hand — easy to drop, and they shift where the tanh non-linearity's sensitive region falls.
• Applying the wrong non-linearity (e.g. ReLU instead of tanh) without realizing plain RNNs traditionally use tanh specifically — while ReLU RNNs exist, tanh remains the more standard default for the basic architecture covered here.

Interview Relevance

Q: "Write out the forward-pass equations for a basic RNN cell." \(\mathbf{h}_t = \tanh(\mathbf{W}_{xh}\mathbf{x}_t + \mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{b}_h)\), followed by \(\mathbf{y}_t = \mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y\) if an output is needed at this step. A strong answer also notes that all weight matrices are shared across every time step, and that \(\mathbf{h}_0\) is typically initialized to zero.

Practice Question

Using the same weights as the worked example (\(W_{xh}=0.5, W_{hh}=0.8, b_h=0\)), continue the computation for a fourth input \(x_4=2\), given \(h_3\approx-0.0593\).

Related DL Notes

• RNN Architecture
• Tanh Function
• Backpropagation Through Time