Feature engineering

Why this matters

Models only see numbers. The work of turning raw columns (text categories, dates, prices in different units, free-text notes) into useful numeric inputs is called feature engineering, and on tabular data it often matters more than the choice of algorithm. A k-nearest-neighbours model can go from coin-flip to strong just by encoding and scaling the same information properly.

Encoding categories

A column like neighbourhood = north | center | south must become numbers.

Ordinal (integer) encoding maps each category to an integer: center=0, north=1, south=2. It is compact, but it invents an order and distances that do not exist: the model now thinks "south" is twice as far from "center" as "north" is. Use it only when the categories really are ordered (small < medium < large, education levels), or with tree models, which can cope with arbitrary codes.

One-hot encoding creates one 0/1 column per category:

neighbourhood   is_center  is_north  is_south
center              1          0         0
south               0          0         1
north               0          1         0

Every category is now equally far from every other, which is what distance-based and linear models need.

import numpy as np

def one_hot(values, categories):
    values = np.asarray(values)
    return np.column_stack([(values == c).astype(float) for c in categories])

Practical points:

• Fix the category list on the training data. A category seen only at prediction time should map to all zeros (or an explicit "other" column), not crash the pipeline.
• High-cardinality columns (ZIP codes, product IDs with thousands of values) explode into thousands of columns. Group rare values into "other", or use target encoding (replace each category by the average label in training data, computed inside cross-validation folds to avoid leakage).
• For linear models, drop one column per feature (drop="first") if you need unique coefficients; regularised models usually do not need this.

Scaling numeric features

Distance-based models (k-NN, k-means, SVMs), PCA and anything trained by gradient descent with regularisation are sensitive to feature scale. A price in dollars (around 1,500 ± 400) will completely swamp a distance in kilometres (0–10) in a Euclidean distance.

• Standardisation: (x - mean) / sd, giving mean 0 and sd 1. The usual default.
• Min-max scaling: (x - min) / (max - min), giving 0 to 1. Sensitive to outliers.
• Log transform: for skewed positive values such as income, counts or prices, log(1 + x) often helps both linear and distance-based models.

As in Lesson 2, compute the means, standard deviations and category lists on the training data only. Tree models do not need scaling.

k-nearest neighbours: a scale-sensitive model

k-NN is the simplest classifier: to predict a new row, find the k closest training rows and take a majority vote of their labels.

d = ((X_query[:, None, :] - X_train[None, :, :]) ** 2).sum(axis=2)  # all distances
nearest = np.argsort(d, axis=1)[:, :k]
pred = (y_train[nearest].mean(axis=1) >= 0.5).astype(int)

There is no training step, so everything depends on what "close" means, which is exactly what feature engineering controls.

Worked example

Listings are booked mostly when they are close to the centre, and the centre neighbourhood is especially popular; price is irrelevant in this dataset. With raw features [distance_km, price, neighbourhood_code], the nearest neighbours are simply the listings with similar prices, so accuracy sits near the 50% you would get by guessing. After standardising distance and price and one-hot encoding the neighbourhood, distance and location drive the neighbours, and accuracy rises to around 90%. Same data, same model, different features.

Other common features

• Dates: extract day of week, month, hour, "is holiday", and time since an event (days since signup). Never use the raw timestamp as a number.
• Ratios and interactions: debt / income, price per square metre, clicks / impressions.
• Aggregates: a customer's order count in the last 30 days (computed only from data before the prediction time).
• Missing values: impute (median for numbers, "missing" category for categories) and add an "was missing" indicator, since missingness itself can be informative.

In scikit-learn

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline

pre = ColumnTransformer([
    ("num", StandardScaler(), ["distance_km", "price"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["neighbourhood"]),
])
model = Pipeline([("pre", pre), ("knn", KNeighborsClassifier(n_neighbors=7))])
model.fit(X_train_df, y_train)          # every step is fitted on training data only

Putting preprocessing inside a Pipeline is the reliable way to avoid leakage, especially during cross-validation (Lesson 12).

Common mistakes

• Integer-encoding unordered categories for linear or distance-based models.
• Scaling with statistics from the full dataset.
• Features built with information from after the prediction time.
• Crashing on unseen categories in production. Use handle_unknown="ignore" or an "other" bucket.
• Engineering hundreds of features without validation, which invites overfitting.

Quick recap

• One-hot encode unordered categories; ordinal-encode only real orders.
• Scale numeric features for distance-, gradient- and regularisation-based models; trees do not need it.
• Fit encoders and scalers on training data, inside a pipeline.
• Good features often beat a fancier model.

Try it

On the Do tab, write one-hot encoding, a training-only standardiser and k-NN, and compare accuracy with raw versus engineered features.