meandering.ai
← All tutorials

04 / TREE ENSEMBLES

Boost trees
one step at a time.

Train a gradient-boosted decision-tree classifier on a small nonlinear dataset. Explore estimator settings, fit the model, and tune it with CoolBox in Python, Go, Rust, or C++.

20 MININTERMEDIATE

How boosting works

Gradient boosting builds an ensemble in rounds. Each added tree helps reduce the current prediction error; the learning rate controls how much each tree contributes. CoolBox provides boosted-tree regression and a classifier for binary or multiclass labels.

This walkthrough uses BoostTreeClassifier. Its trees are regression learners trained against class-probability residuals, with softmax probabilities for multiclass classification.

1. Prepare a nonlinear dataset

The XOR-shaped labels make a useful small classification example: the class changes by region rather than following one straight boundary. Keep the test rows separate so you can inspect predictions on samples the model did not fit.

2. Configure and fit the ensemble

num_estimators sets the boosting rounds, learning_rate scales each tree's contribution, and max_depth limits tree complexity. Start with depth one deliberately: individual stumps cannot represent XOR's feature interaction, so this underfit baseline gives the search a meaningful depth choice to improve.

3. Predict and check accuracy

Use the fitted model to classify each held-out row. The classifier also returns class probabilities; their order matches the sorted labels returned by classes(). Results can vary with implementation changes, so treat this tiny example as a learning exercise rather than a benchmark.

Choose a language for a complete baseline example. Current CoolBox bindings expose boosted-tree classifiers in Python, Go, and Rust, alongside the native C++ API. Rebuild older bindings if these APIs are missing.

COOLBOX BOOSTED-TREE CLASSIFIER
from ml_toolbox import decision_tree as trees

# XOR-like regions: the class cannot be separated by one straight line.
X_train = [
    [0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0],
    [0.1, 0.1], [0.1, 0.9], [0.9, 0.1], [0.9, 0.9],
]
y_train = [0, 1, 1, 0, 0, 1, 1, 0]

X_test = [[0.05, 0.8], [0.8, 0.05], [0.05, 0.05], [0.8, 0.8]]
y_test = [1, 1, 0, 0]

params = trees.BoostTreeParameters()
params.num_estimators = 60
params.learning_rate = 0.1
params.max_depth = 1

model = trees.BoostTreeClassifier(params)
model.fit(X_train, y_train)

classes = model.classes()
correct = 0
for sample, expected in zip(X_test, y_test):
    predicted = model.predict(sample)
    probabilities = model.predict_proba(sample)
    correct += predicted == expected
    print(
        f"{sample} -> class {predicted}; "
        f"probabilities {dict(zip(classes, probabilities))}"
    )

print(f"Test accuracy: {correct / len(y_test):.0%}")
Actual shallow CoolBox baseline predictions: depth-one trees produce an uninformative surface, validation log loss 0.6931, and 50 percent test accuracy. Circles show training labels and diamonds mark test locations.
Measured baseline from CoolBox's native C++ classifier: 60 trees, learning rate 0.1, depth one. These are the same baseline predictions, markers, axes, and confidence scale used in step 5, not hand-drawn class regions.

In C++, build CoolBox with the C++ setup guide and link the exported CoolBox::decision_tree target. The Python binding can be installed with the Python setup guide.

5. Draw the selected decision surface

Measured CoolBox surfaces before and after genetic search. Depth-one baseline: validation loss 0.6931, test accuracy 50 percent. Selected depth-three model: validation loss 0.1991, test accuracy 100 percent. Both panels use the same axes, training circles, test diamonds, and color scale.
Actual native CoolBox predictions, seed 42. Baseline: depth 1, 60 trees, rate 0.1. Selected: depth 3, 99 trees, rate 0.133765. Validation log loss fell from 0.6931 to 0.1991; accuracy on the four untouched test points rose from 50% to 100%. This small teaching example is not a benchmark.

Run the downloaded Python file with a current CoolBox build that exposes opt.Optimizer. It writes boosted-tree-decision-surface.svg, boosted-tree-search-preview.svg, and the raw prediction JSON in your working directory. Both images use one shared renderer, so the baseline panel matches the earlier plot exactly apart from size.

The exporter evaluates each fitted model on the same 60-by-60 grid. Blue and orange indicate the predicted class; darker cells indicate more confident probabilities. It uses Python's standard library to write SVG, so no Matplotlib or additional plotting package is required.

Why did depth help? The shallow baseline underfits XOR. Allowing depth up to four lets the search choose trees that capture interactions between features; it selected depth three in this run. The improvement is measured, not painted in. More depth is not always better, and the four-point test set is far too small to establish general performance.

The displayed experiment compiled CoolBox's native BoostTreeClassifier and opt::GeneticSearch sources from checkout d64618ba48ce28a1704c0018b9cdf3b72d09c328. Both plots are SVG exports of its predictions, not screenshots of a plotting application. Optimizer results may vary across compiler/platform implementations.

To reproduce these exact images without a compatible CoolBox Python extension, download the saved native predictions alongside the Python script, then run:

TERMINAL · RENDER SAVED PREDICTIONS
python boosted-tree-genetic-search.py --results boosted-tree-search-results.json

Keep exploring

Try the PCA tutorial to reduce feature dimensions, or compare the boosted ensemble with the neural-network tutorial. For regression, CoolBox also provides a BoostTree estimator with a batch prediction API.