Understand the four core assumptions of linear regression to ensure your model’s predictions are unbiased and statistically valid.
What it is
Linear regression assumes a specific relationship between independent variables (X) and a dependent variable (y). For Ordinary Least Squares (OLS) estimates to be Best Linear Unbiased Estimators (BLUE), four key assumptions must hold: Linearity, Independence, Homoscedasticity, and Normality of residuals. Residuals are the differences between observed values and predicted values (e = y - ŷ). Violating these assumptions can lead to misleading p-values, confidence intervals, and predictions.
Why it matters
- Valid Inference: Ensures hypothesis tests (t-tests, F-tests) accurately reflect statistical significance.
- Reliable Predictions: Prevents systematic bias in forecasted values across different ranges of input data.
- Efficient Estimates: Guarantees that the regression coefficients have the smallest possible variance among all unbiased estimators.
- Diagnostics: Helps identify when to transform data or switch to non-linear models.
Syntax or steps
To check assumptions, you typically fit a model and analyze its residuals. The standard workflow involves: 1. Fit the linear model using least squares. 2. Calculate residuals for each observation. 3. Visualize residuals against fitted values (for Linearity and Homoscedasticity). 4. Check residual distribution (for Normality).
Example
import numpy as np
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt
# Generate synthetic data satisfying assumptions
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X.flatten() + np.random.randn(100)
# Fit model
model = LinearRegression().fit(X, y)
y_pred = model.predict(X)
residuals = y - y_pred
# Plot Residuals vs Fitted Values
plt.scatter(y_pred, residuals, alpha=0.6)
plt.axhline(0, color='red', linestyle='--')
plt.xlabel('Fitted Values')
plt.ylabel('Residuals')
plt.title('Check: Linearity & Homoscedasticity')
plt.show()This code generates data where the true relationship is linear with constant variance noise. The scatter plot shows residuals randomly distributed around zero with no funnel shape, indicating satisfied assumptions.
Common mistakes
- Ignoring Outliers: A single extreme point can skew residuals and violate normality. Use robust regression or remove outliers if justified.
- Assuming Causality: Regression shows correlation. Independence assumption refers to error terms, not necessarily causal independence of variables.
- Overlooking Heteroscedasticity: If residual spread increases with fitted values (funnel shape), standard errors are biased. Fix by transforming
y(e.g., log) or using weighted least squares. - Small Sample Size: Normality of residuals is less critical with large samples due to the Central Limit Theorem, but vital for small datasets.
When to use it
| Scenario | Use Linear Regression? | Alternative |
|---|---|---|
| Relationship is straight-line | Yes | - |
| Residuals show curved pattern | No | Polynomial Regression |
| Variance changes with X | No | Weighted Least Squares |
| Data is time-series | Carefully | ARIMA / GARCH |
Practice
Guided Exercise: Modify the example above to introduce heteroscedasticity by multiplying the noise term by X. Observe how the residual plot changes.
Challenge: Write a function that calculates the Breusch-Pagan test statistic manually using residuals and fitted values to detect heteroscedasticity.
Quick check
Q: What does a "funnel shape" in a residuals vs. fitted values plot indicate?
A: It indicates heteroscedasticity, meaning the variance of the residuals is not constant across all levels of the independent variables.
Summary
Linear regression is powerful only when its underlying assumptions about linearity, independence, homoscedasticity, and normality are met. Always inspect residual plots before trusting model conclusions, as visual diagnostics reveal violations that summary statistics might miss.
