KNN's entire prediction hinges on one question: how do you measure "closeness" between two points? The distance metric you choose directly determines which neighbors get consulted — and therefore what the model predicts.
Euclidean Distance — The Default
This is the ordinary straight-line distance — the Pythagorean theorem, extended to \(n\) dimensions. For 2D points \(x=(2,3)\) and \(y=(6,6)\):
Geometric Intuition
Euclidean distance is exactly the hypotenuse of the right triangle formed by each feature's difference.
Manhattan Distance — The Alternative
Instead of a straight diagonal line, Manhattan distance sums the absolute differences along each axis — like navigating city blocks instead of flying directly. For the same points: \(d = |6-2|+|6-3| = 4+3 = 7\).
Minkowski Distance — The General Formula
This is a family that includes both special cases: \(p=2\) gives Euclidean distance, \(p=1\) gives Manhattan distance. scikit-learn's KNN implementation defaults to Minkowski with \(p=2\) — i.e., ordinary Euclidean distance.
Python Implementation
import numpy as np
from scipy.spatial import distance
a = np.array([2, 3])
b = np.array([6, 6])
print(distance.euclidean(a, b)) # 5.0
print(distance.cityblock(a, b)) # 7.0 -- Manhattan/cityblock distance
print(distance.minkowski(a, b, p=2)) # 5.0 -- same as Euclidean when p=2
print(distance.minkowski(a, b, p=1)) # 7.0 -- same as Manhattan when p=1Why the Choice of Metric Matters
Euclidean distance is sensitive to the overall magnitude of differences across all dimensions simultaneously; Manhattan distance treats each dimension's contribution independently and can be more robust to outliers in a single dimension, since it doesn't square (and thereby amplify) large individual differences. For high-dimensional data, both distances tend to become less discriminating — see the curse of dimensionality.
Why Scaling Is Non-Negotiable for Any Distance Metric
# WITHOUT scaling: income (thousands) completely dominates age (tens)
person_a = [25, 40000] # age, income
person_b = [55, 41000]
import numpy as np
d = np.sqrt((25-55)**2 + (40000-41000)**2)
print(d) # ~1000.4 -- the 30-year age gap barely registers next to the 1000-unit income gapThis is precisely why feature scaling is described as non-negotiable for KNN — without it, whichever feature happens to have the largest numeric range silently dominates every distance calculation, regardless of its actual real-world importance.
Common Mistakes
- Using Euclidean distance on unscaled features, letting one large-range feature dominate.
- Applying Euclidean/Manhattan distance to categorical data without proper encoding first — raw category codes have no meaningful numeric distance between them.
Interview Relevance
Q: "Why does KNN require feature scaling, but a decision tree doesn't?" KNN's predictions come directly from distance calculations across all features simultaneously, so an unscaled feature with a large numeric range dominates the distance; a decision tree splits on one feature's threshold at a time, so relative scale between different features never affects its split decisions.
Practice Question
Compute both the Euclidean and Manhattan distance between points \((0,0)\) and \((3,4)\) by hand.
