By the end of this lesson, you will be able to perform simple linear regression in Python using scikit-learn to model the relationship between a single independent variable and a continuous dependent variable.
What it is
Regression analysis is a statistical method used to estimate the relationships among variables. In data analytics, it primarily helps predict a continuous outcome (dependent variable) based on one or more input features (independent variables). The most common form is Linear Regression, which assumes a straight-line relationship between inputs and outputs. Key terms include coefficients (the slope of the line), intercept (where the line crosses the y-axis), and R-squared (a metric indicating how well the model explains variance).Why it matters
- Prediction: Forecast future values, such as sales revenue based on advertising spend.
- Relationship Quantification: Determine how much a change in one variable affects another (e.g., price elasticity).
- Trend Analysis: Identify underlying trends in time-series data.
- Baseline Modeling: Provide a simple benchmark against which complex models can be compared.
Syntax or steps
The standard workflow for regression in Python involves: 1. Importing necessary libraries (pandas, sklearn).
2. Loading and preparing data (handling missing values, encoding categorical variables if needed).
3. Splitting data into training and testing sets.
4. Initializing the LinearRegression model.
5. Fitting the model to the training data.
6. Making predictions and evaluating performance.
Example
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
# 1. Create dummy dataset
data = {
'ad_spend': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100],
'sales': [15, 25, 35, 45, 55, 65, 75, 85, 95, 105]
}
df = pd.DataFrame(data)
# 2. Define features (X) and target (y)
X = df[['ad_spend']]
y = df['sales']
# 3. Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# 4. Initialize and train the model
model = LinearRegression()
model.fit(X_train, y_train)
# 5. Make predictions
predictions = model.predict(X_test)
# 6. Evaluate
print(f"Coefficient: {model.coef_[0]}")
print(f"Intercept: {model.intercept_}")
print(f"R-Squared: {r2_score(y_test, predictions)}")
ad_spend predicts sales. We split the data so the model learns from 80% and tests on 20%. The fit method calculates the best-fit line. Finally, we print the coefficient (slope), intercept, and R-squared score to evaluate accuracy.
Common mistakes
- Ignoring Data Scaling: While not strictly required for linear regression coefficients, scaling helps when comparing feature importance or using gradient descent-based implementations.
- Assuming Causation: A high correlation does not prove that changing
Xcauses changes iny; confounding variables may exist. - Overfitting with Too Many Features: Adding irrelevant variables increases noise and reduces generalization power.
- Not Checking Residuals: Always plot residuals to ensure they are randomly distributed; patterns indicate non-linear relationships or heteroscedasticity.
When to use it
Use linear regression when the relationship appears roughly linear and interpretability is key. Use decision trees or random forests when relationships are non-linear or interactions are complex.| Feature | Linear Regression | Decision Tree Regressor |
|---|---|---|
| Interpretability | High (clear coefficients) | Moderate (visualizable splits) |
| Non-linearity | Poor (unless transformed) | Good |
| Outlier Sensitivity | High | Low |
| Training Speed | Very Fast | Fast |
Practice
Guided Exercise: Modify the example above to include a second feature,social_media_posts, with values proportional to ad spend. Observe how the coefficients change.
Challenge: Implement Ridge Regression (sklearn.linear_model.Ridge) instead of Linear Regression. Compare the R-squared scores. Hint: Ridge adds a penalty to large coefficients to prevent overfitting.
