Autograd

This note is the practical, hands-on reference for autograd — PyTorch's automatic differentiation engine — covering every piece of syntax used throughout this hub's backpropagation examples: requires_grad, .backward(), .grad, and the contexts that disable gradient tracking.

The Core API

import torch

x = torch.tensor(3.0, requires_grad=True)   # tells autograd to TRACK operations on x
y = x ** 2 + 2 * x

y.backward()   # computes dy/dx and stores it in x.grad
print(x.grad)   # tensor(8.) -- matches 2x+2 evaluated at x=3

requires_grad — Which Tensors Get Tracked

a = torch.tensor(2.0, requires_grad=True)    # tracked
b = torch.tensor(3.0)                          # NOT tracked by default
c = a * b
print(c.requires_grad)   # True -- if ANY input requires grad, the output does too

Model parameters created via nn.Parameter or standard layers (nn.Linear, etc.) have requires_grad=True by default; raw input data typically doesn't need it unless you're specifically computing gradients with respect to the input itself (e.g. for adversarial examples or visualization techniques).

Disabling Gradient Tracking

with torch.no_grad():
    y = model(x)   # no computational graph is built -- saves memory, used for inference/validation

x.requires_grad_(False)   # or explicitly turn off tracking for a specific tensor
y = x.detach()              # creates a new tensor sharing data but detached from the graph entirely

This is exactly the mechanism behind Validation Loop's torch.no_grad() block — no gradient tracking means no unnecessary memory spent caching values a backward pass will never use.

Gradient Accumulation — Why zero_grad() Matters

x = torch.tensor(2.0, requires_grad=True)

y1 = x ** 2
y1.backward()
print(x.grad)   # tensor(4.)

y2 = x ** 2
y2.backward()
print(x.grad)   # tensor(8.) -- ACCUMULATED, not overwritten! 4 + 4 = 8

x.grad.zero_()   # must explicitly reset before the next independent computation

Common Mistakes

• Forgetting that gradients accumulate by default across multiple .backward() calls — always call optimizer.zero_grad() (or tensor.grad.zero_() directly) between independent computations.
• Calling .backward() on a non-scalar tensor without specifying a gradient argument — .backward() by default expects a scalar output; for a vector/tensor output, you must either sum/reduce it to a scalar first, or pass an explicit gradient tensor matching its shape.

Interview Relevance

Q: "Why does PyTorch accumulate gradients by default across multiple .backward() calls, instead of overwriting them?" This design supports use cases like gradient accumulation over multiple small batches (simulating a larger effective batch size when memory is limited) and computing gradients from multiple loss terms across separate backward passes. It does mean, however, that optimizer.zero_grad() must be called explicitly before each new, independent gradient computation, or stale gradients from previous steps will silently corrupt the current update.

Practice Question

What is the difference between torch.no_grad() and calling .detach() on a specific tensor?

Related DL Notes

• Backpropagation (Intro)
• Computational Graphs (PyTorch)
• PyTorch Training Loop