Expected Value

The expected value of a random variable is its long-run average — the weighted average of every possible outcome, weighted by how likely each one is. It's the formal definition behind terms like "average loss" that you'll see in every training log.

Formula

Numerical Example

A fair six-sided die: \(\mathbb{E}[X] = \sum_{k=1}^{6} k \cdot \frac{1}{6} = \frac{1+2+3+4+5+6}{6} = 3.5\). Notice 3.5 isn't even a possible outcome of a single roll — the expected value is the long-run average across many rolls, not a predicted single result.

A Deep Learning Example

The training loss reported per epoch, \(\frac{1}{N}\sum_{i=1}^N L(y_i, \hat{y}_i)\), is exactly an empirical expected value — the average loss across your training samples, which approximates the true expected loss over the full (unobserved) data distribution: \(\mathbb{E}_{(x,y)\sim\mathcal{D}}[L(y, f(x))]\). This is precisely what training is trying to minimize — not the loss on any one example, but its expectation across the whole data distribution.

Code

import numpy as np

# Empirical expected value: average loss across a batch
losses = np.array([0.8, 1.2, 0.3, 0.9, 1.5])
expected_loss = np.mean(losses)
print(expected_loss)   # 0.94 -- this IS an expected value estimate

import torch
import torch.nn as nn

loss_fn = nn.MSELoss()   # by default, reduction='mean' -- averages over the batch,
                            # i.e. computes an empirical expected value of the per-sample loss

Linearity of Expectation

This holds regardless of whether \(X\) and \(Y\) are independent — a useful fact for reasoning about how averaging affects noisy quantities like gradient estimates in mini-batch training.

Common Mistakes

• Assuming the expected value must be a value the random variable can actually take (like assuming a die "should" roll 3.5) — it's a long-run average, not a most-likely single outcome.
• Forgetting that a reported training loss is only an estimate of the true expected loss over the entire data distribution, based on a finite sample — this is exactly why a held-out validation set exists, to check whether that estimate generalizes.

Interview Relevance

Q: "What is a neural network's loss function actually minimizing?" In the idealized sense, it minimizes the expected loss over the true (unknown) data distribution. In practice, training minimizes the empirical average loss over the training set as a stand-in — which is why generalization to unseen data (measured via a validation/test set) is a separate concern from training loss alone.

Practice Question

A biased coin lands heads (worth 1) with probability 0.7 and tails (worth 0) with probability 0.3. Compute \(\mathbb{E}[X]\).

Related DL Notes

• Probability Distributions
• Variance & Standard Deviation
• Mean Squared Error