By the end of this lesson, you will be able to transform categorical variables into numerical formats suitable for machine learning models and understand how to validate these transformations.
What it is
Categorical data represents discrete groups or labels (e.g., "Red", "Blue", "Small", "Medium"). Most statistical algorithms and machine learning models require numerical input. Encoding is the process of converting these text-based categories into numbers. The two most common methods are Ordinal Encoding (assigning integers based on rank) and One-Hot Encoding (creating binary columns for each category).
Related terms include dummy variables, feature engineering, and high-cardinality (when a column has too many unique values for one-hot encoding to be efficient).
Why it matters
- Model Compatibility: Algorithms like Linear Regression, SVM, and Neural Networks cannot process strings directly.
- Avoiding False Order: One-hot encoding prevents models from assuming that category "3" is greater than category "1" when no such order exists.
- Data Integrity: Proper encoding ensures that missing categories in test data do not crash the model during prediction.
- Performance Optimization: Choosing the right encoder reduces memory usage and training time.
Syntax or steps
- Identify columns with non-numeric data types.
- Determine if the categories have an inherent order (ordinal) or are nominal (unordered).
- Apply
LabelEncoderfor ordinal features orOneHotEncoderfor nominal features. - Ensure the transformation is fitted only on training data to prevent data leakage.
Example
import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
# Sample Data
data = {
'size': ['S', 'M', 'L', 'XL'],
'color': ['Red', 'Blue', 'Green', 'Red']
}
df = pd.DataFrame(data)
# 1. Ordinal Encoding (Size has order: S < M < L < XL)
le_size = LabelEncoder()
df['size_encoded'] = le_size.fit_transform(df['size'])
# 2. One-Hot Encoding (Color has no order)
ohe_color = OneHotEncoder(sparse_output=False) # sparse_output=False returns numpy array
color_encoded = ohe_color.fit_transform(df[['color']])
color_df = pd.DataFrame(color_encoded, columns=ohe_color.get_feature_names_out(['color']))
# Combine results
final_df = pd.concat([df.drop(columns=['size', 'color']), color_df], axis=1)
print(final_df)This code first converts sizes to integers (0, 1, 2, 3). Then, it creates separate binary columns for each color. Note that sparse_output=False is used here for readability; in production, keep it sparse to save memory.
Common mistakes
- Using One-Hot on High Cardinality: If a column has 1,000 unique cities, creating 1,000 columns causes the "curse of dimensionality." Use Target Encoding or Hashing instead.
- Fitting on Test Data: Always call
.fit()on training data only. Calling.fit_transform()on test data leaks information about unseen categories. - Ignoring Missing Values: Encoders may fail if NaNs exist. Impute missing categories before encoding.
- Assuming Ordinality: Treating "Male/Female" as 0/1 implies Male > Female numerically. Use One-Hot for nominal data.
When to use it
| Method | Best For | Risk |
|---|---|---|
| Ordinal Encoding | Categories with logical order (Low/Med/High) | Implies linear relationship where none exists |
| One-Hot Encoding | Nominal categories with low cardinality (<10) | Explodes feature space with high cardinality |
| Target Encoding | High cardinality nominal data | Prone to overfitting if not smoothed |
Practice
Guided Exercise: Take a dataset with a column "Education Level" containing ["HS", "Bachelor", "Master", "PhD"]. Encode this using LabelEncoder. Verify that "PhD" gets the highest integer value.
Challenge: Create a pipeline that handles a new category "Associate" appearing in the test set but not in the training set. How does OneHotEncoder handle unknown categories? (Hint: Look at the handle_unknown parameter).
Quick check
Q: Why should you avoid using One-Hot Encoding on a column with 500 unique product IDs?
A: It would create 500 new columns, drastically increasing memory usage and computational cost while adding little predictive power due to sparsity.
Summary
Categorical encoding bridges the gap between human-readable labels and machine-readable numbers. Selecting the correct method—ordinal for ranked data and one-hot for unordered data—is critical for model accuracy and efficiency. Always fit your encoders on training data to maintain rigorous validation standards.
