Gradient boosting

Why this matters

Neural networks win on images, audio and text. On tabular data (rows and columns in a database: customers, transactions, sensor logs, insurance claims) the usual winner is gradient boosting: libraries such as XGBoost, LightGBM and CatBoost, or scikit-learn's HistGradientBoostingClassifier. Boosted trees handle mixed feature types, missing values and non-linear interactions with little preprocessing, and they train in seconds on a laptop. If you work with business data, this is likely the model you will ship most often.

The core idea: fix the previous mistakes

A random forest (from Machine Learning Foundations) trains many deep trees independently and averages them. Boosting trains many small trees one after another, each one correcting the errors of the ensemble so far.

For regression with squared error, the algorithm is:

F0(x) = mean(y)                               start with a constant
for m = 1..M:
    r = y - F(x)                              residuals: what is still wrong
    fit a small tree h_m to predict r
    F(x) = F(x) + learning_rate * h_m(x)      take a small step toward fixing it

The residual y - F(x) is the negative gradient of the squared-error loss with respect to the prediction. That is where the "gradient" in gradient boosting comes from: each tree is a step of gradient descent, taken in function space. For other losses (log loss for classification, absolute error for robust regression) you fit the tree to that loss's negative gradient instead.

Worked example by hand

Three houses, one feature (size), target price in thousands:

F0 is the mean, 200, and the mean squared error is (100² + 20² + 120²)/3 ≈ 8,267. The best single split for the residuals is at size 100: left mean -60, right mean +120. With learning rate 0.5:

house 1: 200 + 0.5 * (-60)  = 170
house 2: 200 + 0.5 * (-60)  = 170
house 3: 200 + 0.5 * (+120) = 260

The mean squared error fell to (70² + 10² + 60²)/3 ≈ 2,867. The next tree is fitted to the new residuals (-70, +10, +60), and so on. Each round removes part of the remaining error, and the small learning rate stops any single tree from overreacting.

The settings that matter

• n_estimators (number of trees) and learning_rate work together. A smaller rate (0.05–0.1) needs more trees but usually generalises better. Use early stopping on a validation set to pick the number of trees.
• Tree size (max_depth 3–8, or num_leaves in LightGBM): depth 1 trees (stumps) model each feature separately; deeper trees capture interactions between features.
• Subsampling (subsample, colsample_bytree around 0.8): train each tree on a random part of the rows or columns. This adds randomness that reduces overfitting.
• Regularisation (min_child_weight, reg_lambda): stop leaves from fitting a handful of rows.

In code

scikit-learn ships a fast histogram-based implementation:

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score

model = HistGradientBoostingClassifier(
    learning_rate=0.05, max_iter=500, max_depth=6,
    early_stopping=True, validation_fraction=0.15, random_state=0,
)
model.fit(X_train, y_train)
print(roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]))

XGBoost has a similar interface:

import xgboost as xgb

model = xgb.XGBClassifier(n_estimators=2000, learning_rate=0.05, max_depth=6,
                          subsample=0.8, colsample_bytree=0.8,
                          early_stopping_rounds=50, eval_metric="auc")
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
print(model.best_iteration)

LightGBM and CatBoost follow the same pattern. CatBoost handles categorical columns natively; LightGBM is often the fastest on large data. Install them with pip; all three work in free Colab and Kaggle notebooks.

Interpreting a boosted model

• Feature importance (gain or split counts) is a quick overview but can mislead with correlated features.
• Permutation importance (sklearn.inspection.permutation_importance) measures how much the validation score drops when one column is shuffled.
• SHAP values (the shap library) explain individual predictions: how much each feature pushed this row's prediction up or down.

Common mistakes

• Too many trees without early stopping. Boosting keeps reducing training error forever and eventually overfits.
• Learning rate too high (for example 1.0) with deep trees: fast to overfit.
• Leaky features. Trees exploit any column that encodes the answer, such as a "cancellation date" when predicting cancellations. Ask of each feature: would I know this at prediction time?
• One-hot encoding high-cardinality IDs into thousands of columns. Use native categorical support or target encoding inside cross-validation.
• Skipping the baseline. Compare with logistic regression and with predicting the mean.

Quick recap

• Boosting adds small trees one at a time, each fitted to the current residuals (the negative gradient).
• The learning rate shrinks each step; tune it together with the number of trees, using early stopping.
• On tabular data, boosted trees are the default strong model; XGBoost, LightGBM, CatBoost and scikit-learn all implement it.
• Explain models with permutation importance or SHAP, and watch for leaky features.

Try it

Open the Do tab. You will implement gradient boosting with decision stumps in NumPy, confirm that the training error falls every round, and beat a predict-the-mean baseline five times over.