TensorFlow Tensors

This note covers TensorFlow's tensor types — including a genuinely TensorFlow-specific distinction PyTorch doesn't have in the same form: tf.constant versus tf.Variable.

tf.constant — Immutable

import tensorflow as tf

x = tf.constant([1.0, 2.0, 3.0])
print(x.shape, x.dtype)   # (3,) float32

# x[0] = 5.0   # would raise an error -- tf.constant tensors CANNOT be modified in place

tf.Variable — Mutable, Used for Trainable Weights

w = tf.Variable([1.0, 2.0, 3.0])
w.assign([4.0, 5.0, 6.0])   # explicitly reassign its value
w.assign_add([1.0, 1.0, 1.0])   # in-place-style update

print(w)   # tf.Variable now holds [5.0, 6.0, 7.0]

This distinction has no direct equivalent in PyTorch, where a regular tensor with requires_grad=True serves both roles. In TensorFlow, model weights are always represented as tf.Variable objects specifically, since they need to be mutable (updated every training step) — tf.constant is reserved for fixed, unchanging values like input data or hyperparameter constants.

Gradient Computation With GradientTape

x = tf.Variable(3.0)

with tf.GradientTape() as tape:
    y = x ** 2 + 2 * x

gradient = tape.gradient(y, x)
print(gradient)   # tf.Tensor(8.0, ...) -- matches 2x+2 at x=3, same as the PyTorch autograd example

GradientTape is TensorFlow's direct equivalent of PyTorch's autograd — it "records" operations performed on tf.Variable objects within its context, then computes gradients on demand via .gradient(), structurally parallel to PyTorch's .backward() from Autograd.

Common Mistakes

• Using tf.constant for model weights — since it's immutable, it can never actually be updated during training; trainable parameters must always be tf.Variable.
• Forgetting that GradientTape only tracks operations on tf.Variable objects by default (not tf.constant) — computing a gradient with respect to a constant requires explicitly calling tape.watch() on it first.

Interview Relevance

Q: "Why does TensorFlow distinguish between tf.constant and tf.Variable, when PyTorch doesn't have an equivalent split?" TensorFlow's design explicitly separates immutable values (inputs, fixed constants) from mutable, trainable state (model weights) at the type level — tf.Variable exists specifically to represent values that need to be updated in place during training. PyTorch instead uses one unified tensor type, distinguishing trainable from non-trainable purely via the requires_grad flag, without a separate mutability distinction at the object type level.

Practice Question

Why would attempting to directly modify an element of a tf.constant tensor raise an error, while the same operation on a tf.Variable works fine (via .assign())?

Related DL Notes

• TensorFlow Basics
• Autograd
• Keras Overview