Recurrent Neural Networks & LSTM

The Birth of Recurrent Networks

The idea of feeding a network's output back into itself dates to the earliest days of neural network research. In 1982, John Hopfield introduced the Hopfield Network, a fully connected recurrent system that could store and recall patterns through energy minimization. While not designed for sequences, it proved that recurrence could endow a network with a form of memory.

The modern era of sequence modeling began with Jeffrey Elman, who in 1990 published "Finding Structure in Time." Elman proposed a simple recurrent network (SRN) where hidden-state activations at time \( t-1 \) are copied into special "context units" and fed back as additional input at time \( t \). This elegant trick gave the network a short-term memory of its recent past, enabling it to learn temporal patterns in language and other sequential data.

Elman's architecture was remarkably simple: a standard feedforward network augmented with a single recurrent connection. Yet it was powerful enough to learn grammar-like rules from raw character sequences, sparking an explosion of interest in recurrent approaches to language modeling, speech recognition, and time-series prediction.

The LSTM Revolution

Despite Elman's success, vanilla RNNs suffered from a crippling limitation: they could not learn long-range dependencies. When the gap between relevant information and the point where it is needed grows large, gradients either vanish or explode during training, making learning impossible.

In 1997, Sepp Hochreiter and Jurgen Schmidhuber published their landmark paper introducing the Long Short-Term Memory (LSTM) network. The key innovation was the cell state -- a dedicated memory highway that runs through the entire sequence, regulated by learned gates that control what information to store, forget, and output. This architecture solved the vanishing gradient problem by allowing gradients to flow unchanged through the cell state.

The LSTM was refined over the following decade. In 2000, Felix Gers added the forget gate, which became essential for allowing the cell to reset its memory. By the mid-2010s, LSTMs had become the dominant architecture for virtually every sequence modeling task, from machine translation to speech synthesis.

The Timeline of Recurrent Architectures

Why Sequences Are Different

Standard feedforward neural networks treat each input independently. They have no notion of order, context, or history. But many real-world problems are inherently sequential: the meaning of a word depends on the words before it; tomorrow's stock price depends on today's; the next note in a melody depends on the preceding phrase.

Consider predicting the next word in the sentence "The clouds are dark and it is likely to ___." A feedforward network seeing only the word "to" has no information to work with. But a recurrent network that has processed the entire sentence so far carries a compressed summary of "clouds," "dark," and "likely" in its hidden state, enabling it to predict "rain" with confidence.

The key insight is that sequence modeling requires memory. The network must maintain an internal state that summarizes everything it has seen so far, updating this state at each timestep as new information arrives. This is exactly what the hidden state \( h_t \) provides in a recurrent neural network.

The Hidden State as Compressed Memory

At each timestep \( t \), the RNN receives two inputs: the current observation \( x_t \) and the previous hidden state \( h_{t-1} \). It combines these to produce a new hidden state \( h_t \) that encodes both the current input and the relevant history:

The hidden state \( h_t \) is a fixed-size vector that must compress all relevant information from the entire history \( (x_1, x_2, \ldots, x_t) \) into a single representation. This is both the power and the limitation of RNNs: the fixed-size bottleneck forces the network to learn what to remember and what to forget, but it also means that very long-range dependencies can be difficult to capture.

The output \( y_t \) at each timestep is then computed from the hidden state:

This simple formulation -- state update followed by output -- is the fundamental building block of all recurrent architectures, from vanilla RNNs to LSTMs and GRUs.

Types of Sequence Tasks

The Vanilla RNN Equations

A vanilla RNN computes its hidden state at time \( t \) using the following recurrence:

Where \( W_h \in \mathbb{R}^{d_h \times d_h} \) is the hidden-to-hidden weight matrix, \( W_x \in \mathbb{R}^{d_h \times d_x} \) is the input-to-hidden weight matrix, and \( b \in \mathbb{R}^{d_h} \) is a bias vector. The \( \tanh \) activation squashes the result to the range \([-1, 1]\).

The output at each timestep is computed as:

Where \( W_y \in \mathbb{R}^{d_y \times d_h} \) maps the hidden state to the output space. For classification tasks, a softmax is applied: \( \hat{y}_t = \text{softmax}(y_t) \).

The critical feature is weight sharing: the same matrices \( W_h \), \( W_x \), and \( W_y \) are used at every timestep. This means an RNN can process sequences of any length with a fixed number of parameters, and patterns learned at one position in the sequence can be applied at any other position.

Unrolling Through Time

To understand how an RNN processes a sequence, we "unroll" it through time. The single recurrent cell is replicated \( T \) times (once per timestep), with the hidden state flowing from left to right. This unrolled view reveals the RNN as a very deep feedforward network with shared weights at each layer.

Interactive: RNN Unrolled Through Time

Visualize how an RNN cell is unrolled across timesteps. Input \( x_t \) enters from below, hidden state \( h_t \) flows to the right, and output \( y_t \) exits above. Use the slider to control the number of timesteps, and click Animate to watch data pulse through the chain.

Unrolling and the Chain Rule

Backpropagation Through Time (BPTT) is simply the standard backpropagation algorithm applied to the unrolled RNN graph. Because the RNN is unrolled into \( T \) copies of the same cell, BPTT must sum the gradient contributions from every timestep.

The total loss over a sequence of length \( T \) is the sum of per-timestep losses:

To compute the gradient of \( L \) with respect to the shared weight matrix \( W_h \), we apply the chain rule through every timestep where \( W_h \) is used:

The critical term is the product \( \prod_{j=k+1}^{t} \frac{\partial h_j}{\partial h_{j-1}} \). Each factor in this product involves the Jacobian of the recurrent transition function. When this product spans many timesteps (large \( t - k \)), its magnitude can either explode or vanish, depending on the spectral properties of \( W_h \).

The Jacobian Chain

For the vanilla RNN with \( h_t = \tanh(W_h h_{t-1} + W_x x_t + b) \), each Jacobian factor is:

Where \( \text{diag}(1 - h_t^2) \) is a diagonal matrix of \( \tanh \) derivatives (values between 0 and 1). The product of \( (t - k) \) such Jacobians behaves roughly as:

Where \( \gamma < 1 \) accounts for the \( \tanh \) derivative suppression. If the largest singular value of \( W_h \) multiplied by \( \gamma \) is less than 1, the gradient vanishes exponentially. If it exceeds 1, the gradient explodes. This is the fundamental instability of vanilla RNN training.

Truncated BPTT

In practice, computing gradients across the entire sequence is both memory-intensive and numerically unstable. Truncated BPTT limits the backward pass to a fixed window of \( \tau \) timesteps, computing gradients only through the most recent \( \tau \) steps. This is an approximation that trades long-range gradient accuracy for computational efficiency.

The Exponential Decay Problem

The gradient of the loss at time \( t \) with respect to the hidden state at an earlier time \( k \) involves a product of Jacobians spanning \( (t-k) \) timesteps. For a simplified scalar RNN \( h_t = w \cdot h_{t-1} \), this product reduces to:

Three regimes emerge:

• \( |w| < 1 \): The gradient vanishes exponentially. After 50 timesteps with \( w = 0.9 \), the gradient is \( 0.9^{50} \approx 0.005 \) -- essentially zero. The network cannot learn from distant past events.
• \( |w| = 1 \): The gradient is stable, neither growing nor shrinking. This is the ideal regime, and it is exactly what LSTM achieves through its cell state.
• \( |w| > 1 \): The gradient explodes exponentially. With \( w = 1.1 \) over 50 steps, the gradient is \( 1.1^{50} \approx 117 \), causing numerical overflow and divergent training.

Consequences for Learning

Vanishing gradients mean the network is effectively blind to long-range dependencies. If a word at position 5 is crucial for predicting the output at position 50, the gradient signal connecting them is astronomically small. The weights responsible for maintaining that memory receive essentially zero updates, making it impossible for the network to learn the dependency.

Exploding gradients are easier to detect (the loss suddenly becomes NaN) and easier to fix. The standard remedy is gradient clipping:

Where \( \theta \) is a threshold (typically 1.0 or 5.0). Gradient clipping rescales the gradient vector when its norm exceeds \( \theta \), preventing parameter updates from becoming destructively large while preserving the gradient direction.

Interactive: Gradient Flow Visualization

Watch how gradient magnitude changes across timesteps for different weight values. When \( w < 1 \), gradients vanish (blue). When \( w > 1 \), they explode (red). When \( w \approx 1 \), gradients remain stable (green).

Gradient Decay/Growth Over 50 Steps

The chart below compares gradient magnitude over 50 timesteps for four different weight values, illustrating how even small deviations from \( w = 1 \) cause dramatic exponential effects.

The Cell State Highway

The central innovation of LSTM is the cell state \( C_t \) -- a dedicated memory channel that runs through the entire sequence like a conveyor belt. Information can flow along the cell state unchanged, with only minor linear interactions controlled by gates. This is why gradients can propagate through hundreds of timesteps without vanishing.

The cell state is updated at each timestep through a two-step process: first, selectively forget old information; then, selectively add new information. Both operations are controlled by learned sigmoid gates that output values between 0 and 1.

The Four Components of an LSTM Cell

An LSTM cell contains four interacting components, each with its own weight matrices and bias:

1. Forget Gate -- decides what information to discard from the cell state:

2. Input Gate -- decides what new information to store in the cell state:

3. Cell State Update -- computes candidate values and updates the cell:

4. Output Gate -- decides what to output based on the filtered cell state:

Where \( \sigma \) is the sigmoid function, \( \odot \) denotes element-wise multiplication, and \( [\cdot, \cdot] \) denotes concatenation. Note that the cell state update \( C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t \) is a linear operation on \( C_{t-1} \), which is why gradients flow through it without vanishing.

Interactive: LSTM Gate Visualization

Explore how the forget and input gates control the LSTM cell state. The cell state highway runs along the top. The forget gate controls how much old memory to keep, while the input gate controls how much new information to add.

Gated Recurrent Unit Equations

The Gated Recurrent Unit (GRU), introduced by Cho et al. in 2014, simplifies the LSTM by combining the forget and input gates into a single update gate and merging the cell state and hidden state into one. This reduces the number of parameters while maintaining most of the LSTM's ability to capture long-range dependencies.

Reset Gate -- decides how much past information to forget:

Update Gate -- decides how much of the new candidate to use vs keeping old state:

Candidate Hidden State:

Final Hidden State -- interpolates between old and new:

When \( z_t \approx 0 \), the hidden state is copied forward unchanged (memory preservation). When \( z_t \approx 1 \), the hidden state is completely replaced by the candidate. This linear interpolation provides a gradient highway similar to the LSTM cell state.

LSTM vs GRU Comparison

Why Two Directions?

A standard (unidirectional) RNN processes the sequence from left to right, so the hidden state at time \( t \) only contains information from past inputs \( (x_1, \ldots, x_t) \). But in many tasks, context from future inputs is equally important. For example, in the sentence "He said the bank was steep," the meaning of "bank" depends on the word "steep" that comes after it.

A Bidirectional RNN (BiRNN) solves this by running two separate RNNs in parallel: one processing the sequence left-to-right (forward), and another processing it right-to-left (backward). The outputs of both are then combined at each timestep.

BiRNN Formulation

The forward hidden state captures past context:

The backward hidden state captures future context:

The final representation at each timestep concatenates both directions:

This doubles the representation size at each timestep, providing the network with complete bidirectional context. Bidirectional LSTMs (BiLSTMs) apply this same principle using LSTM cells instead of vanilla RNN cells, and they are the standard architecture for tasks like named entity recognition and machine reading comprehension.

Hidden Size

The hidden size \( d_h \) determines the capacity of the recurrent layer -- how much information the hidden state can encode. Larger hidden sizes can model more complex patterns but increase computation quadratically (since \( W_h \in \mathbb{R}^{d_h \times d_h} \)).

• Small tasks (sentiment, short sequences): 64-128
• Medium tasks (NER, POS tagging): 256-512
• Large tasks (machine translation, speech): 512-1024
• Key insight: Doubling hidden size quadruples the recurrent weight matrix size

Number of Layers (Stacking)

Stacking multiple RNN layers creates a deep RNN where the hidden state of layer \( l \) becomes the input to layer \( l+1 \):

• 1 layer: Sufficient for most tasks with sequences under 200 steps
• 2-3 layers: Standard for production NLP and speech models
• 4+ layers: Diminishing returns; requires residual connections to train effectively

Gradient Clipping

Essential for stable RNN training. Clips the global gradient norm before the optimizer step:

• Typical threshold \( \theta \): 1.0 to 5.0
• Always use gradient clipping with RNNs. It prevents exploding gradients without any downside.
• In PyTorch: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Dropout in RNNs

Applying dropout to recurrent connections is tricky because naive dropout at every timestep destroys the hidden state signal. The standard approach is variational dropout (Gal & Ghahramani, 2016): use the same dropout mask across all timesteps within a sequence, but different masks across sequences in a batch.

Where RNNs and LSTMs Excel

Sequence Prediction Playground

Type a short string and see a simple character-level frequency-based prediction for the next character. The canvas visualizes characters flowing through RNN cells.

Architecture Comparison on Sequence Tasks

The chart below compares Vanilla RNN, LSTM, GRU, and BiLSTM across four common sequence tasks, showing accuracy or F1 score.

PyTorch LSTM for Sentiment Classification

A complete LSTM model for classifying text sequences using PyTorch.

import torch
import torch.nn as nn

class LSTMClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim,
                 output_dim, n_layers=2, dropout=0.3):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(
            embed_dim, hidden_dim, num_layers=n_layers,
            batch_first=True, dropout=dropout,
            bidirectional=True
        )
        self.fc = nn.Linear(hidden_dim * 2, output_dim)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        embedded = self.dropout(self.embedding(x))
        output, (hidden, cell) = self.lstm(embedded)
        # Concatenate final forward and backward hidden states
        hidden = torch.cat((hidden[-2], hidden[-1]), dim=1)
        return self.fc(self.dropout(hidden))

# Usage
model = LSTMClassifier(
    vocab_size=10000, embed_dim=128,
    hidden_dim=256, output_dim=2
)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

# Training loop
for epoch in range(10):
    model.train()
    for batch_x, batch_y in train_loader:
        optimizer.zero_grad()
        predictions = model(batch_x)
        loss = criterion(predictions, batch_y)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()

Keras LSTM for Time Series Forecasting

Using Keras to build a stacked LSTM for predicting the next value in a time series.

import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Dropout

# Generate synthetic sine wave data
t = np.linspace(0, 100, 2000)
data = np.sin(t) + 0.1 * np.random.randn(len(t))

# Create sequences of length 50
seq_len = 50
X, y = [], []
for i in range(len(data) - seq_len):
    X.append(data[i:i+seq_len])
    y.append(data[i+seq_len])
X = np.array(X).reshape(-1, seq_len, 1)
y = np.array(y)

# Split data
split = int(0.8 * len(X))
X_train, X_test = X[:split], X[split:]
y_train, y_test = y[:split], y[split:]

# Build stacked LSTM model
model = Sequential([
    LSTM(64, return_sequences=True,
         input_shape=(seq_len, 1)),
    Dropout(0.2),
    LSTM(32, return_sequences=False),
    Dropout(0.2),
    Dense(1)
])

model.compile(optimizer='adam', loss='mse')
model.summary()

# Train
history = model.fit(
    X_train, y_train, epochs=20, batch_size=32,
    validation_data=(X_test, y_test)
)

# Evaluate
mse = model.evaluate(X_test, y_test)
print(f"Test MSE: {mse:.6f}")

PyTorch GRU for Character-Level Language Model

import torch
import torch.nn as nn

class CharGRU(nn.Module):
    def __init__(self, vocab_size, embed_dim=64,
                 hidden_dim=128, n_layers=1):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.gru = nn.GRU(
            embed_dim, hidden_dim, n_layers,
            batch_first=True
        )
        self.fc = nn.Linear(hidden_dim, vocab_size)

    def forward(self, x, hidden=None):
        emb = self.embedding(x)
        out, hidden = self.gru(emb, hidden)
        logits = self.fc(out)
        return logits, hidden

# Build vocabulary from text
text = "hello world this is a character level model"
chars = sorted(set(text))
char2idx = {c: i for i, c in enumerate(chars)}
idx2char = {i: c for c, i in char2idx.items()}
vocab_size = len(chars)

# Encode text
encoded = [char2idx[c] for c in text]
X = torch.tensor(encoded[:-1]).unsqueeze(0)
y = torch.tensor(encoded[1:]).unsqueeze(0)

# Train
model = CharGRU(vocab_size)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
criterion = nn.CrossEntropyLoss()

for epoch in range(200):
    logits, _ = model(X)
    loss = criterion(logits.view(-1, vocab_size), y.view(-1))
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    if (epoch + 1) % 50 == 0:
        print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

Bidirectional LSTM for Named Entity Recognition

import torch
import torch.nn as nn

class BiLSTM_NER(nn.Module):
    def __init__(self, vocab_size, tagset_size,
                 embed_dim=100, hidden_dim=128):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(
            embed_dim, hidden_dim, num_layers=2,
            batch_first=True, bidirectional=True,
            dropout=0.3
        )
        self.fc = nn.Linear(hidden_dim * 2, tagset_size)

    def forward(self, x):
        emb = self.embedding(x)
        lstm_out, _ = self.lstm(emb)
        logits = self.fc(lstm_out)
        return logits

# Example: predict a tag for every token
model = BiLSTM_NER(vocab_size=5000, tagset_size=9)
sample_input = torch.randint(0, 5000, (4, 30))  # batch=4, seq=30
output = model(sample_input)
print(f"Output shape: {output.shape}")  # (4, 30, 9)