Sequential Data

Every architecture covered so far in this hub — MLPs, and the convolutional networks from earlier categories — assumes each training example is independent, with no meaningful order between examples. Sequential data breaks that assumption entirely: order isn't incidental, it's the whole point.

What Makes Data "Sequential"

In sequential data, each element's meaning depends on what came before it (and sometimes after it). Shuffling the elements destroys the information — "dog bites man" and "man bites dog" contain the identical set of words, but mean completely different things because of order alone.

Common Examples

Why Standard Feedforward Networks Struggle With This

An MLP or CNN (as covered in the CNN Fundamentals category) expects a fixed-size input — one specific number of input features, always in the same "positions." Two problems immediately arise for sequential data: (1) variable length — a sentence might be 5 words or 50 words, and a fixed-size input can't naturally accommodate both; (2) no built-in notion of order-dependence across arbitrary positions — even if you padded every sequence to the same fixed length and fed it to an MLP, the network would need to learn completely separate weights for "the pattern that matters at position 3" versus "the same pattern at position 30," rather than recognizing it's the same pattern regardless of where in the sequence it occurs.

The Core Requirement a Sequence Model Needs

Whatever architecture processes sequential data needs to: (1) handle variable-length input naturally, (2) share the same learned pattern-detection logic across every position in the sequence (not learn a separate copy per position), and (3) maintain some form of "memory" of earlier elements while processing later ones. This is exactly the specification the Recurrent Neural Network, introduced in the next note, was designed to satisfy.

Code — Representing a Sequence as a Tensor

import torch

# A batch of sequences: (batch_size, sequence_length, feature_size)
# e.g. 4 sentences, each up to 10 words, each word represented by a 300-dim embedding
sequences = torch.randn(4, 10, 300)
print(sequences.shape)   # torch.Size([4, 10, 300])

# Contrast with a typical MLP/CNN input: (batch_size, feature_size) or (batch_size, C, H, W)
# -- neither has a dedicated "sequence length" dimension representing ordered steps

Common Mistakes

• Treating a sequence's elements as independent, identically distributed samples (the assumption behind standard train/test splitting and most non-sequential loss functions) — this can silently break evaluation for time-series data specifically, as flagged in Dataset Train/Val/Test Split.
• Assuming any data with multiple "features per example" is automatically sequential — sequential data specifically requires that the order of elements carries meaning, not just that there are multiple related values.

Interview Relevance

Q: "Why can't you just feed a sentence, padded to a fixed length, directly into a standard MLP?" An MLP would need to learn entirely separate weights for detecting the same pattern at each different position in the sequence, since it has no mechanism for recognizing "this same pattern, wherever it occurs." It also can't naturally generalize to sequences longer than whatever fixed length it was trained on. A model needs a way to share pattern-detection logic across positions and process sequences of varying length — exactly what RNNs (and later, attention-based models) were built for.

Practice Question

Is a dataset of house prices with features like square footage, number of bedrooms, and location sequential data? Explain why or why not.

Related DL Notes

• Why RNN
• RNN Architecture
• Tokenization