[slievens]

Fixes #496

Follow-up to #492. This PR is based on the use-criterion branch. Merge that PR first. Before then, the diff here also shows its commits.

Checklist

• [] My branch is up-to-date with main branch. --> based on the code in use-criterion
• Everything works and tested on latest stable Rust.
• Coverage and Linting have been applied

Current behaviour

The regression trees and forests do not give feature importances. The tree nodes keep a split_score (the weighted decrease of the squared error for the split), but no public method uses it.

New expected behaviour

A new compute_feature_importances method calculates the mean decrease in impurity (MDI), as in scikit-learn (feature_importances_).

• DecisionTreeRegressor::compute_feature_importances(normalize: bool) -> Result<Vec<f64>, Failed>
  • For each feature, add the split_score of all the nodes that split on that feature.
  • normalize = false: divide by the total sample mass at the root. The value of a feature is then the decrease of the weighted MSE that comes from its splits. If the tree grows to full depth and all leaves are pure, the sum is the weighted variance of y.
  • normalize = true: divide by the sum, so the importances add up to 1. If no split decreases the impurity, all values stay 0.
• RandomForestRegressor::compute_feature_importances() and ExtraTreesRegressor::compute_feature_importances()
  • Normalize the importances of each tree, calculate the mean over the trees, then normalize again (the same as scikit-learn).
• If the model is not fitted, the methods return Failed::predict.

BaseTreeRegressor::compute_feature_importances(normalize, &mut [f64]) writes into a slice that the caller supplies. The forest uses one scratch buffer for all its trees, so it does not allocate a vector for each tree.

To support this, the fitted models store some new fields:

• BaseTreeRegressor.root_mass: the total sample mass at the root (the sum of bootstrap counts × sample weights).
• DecisionTreeRegressor.num_features and BaseForestRegressor.num_features: the number of columns in the training data.

Breaking change (serde)

Because of the new fields, models that were serialized with an earlier version cannot be deserialized. Fit these models again. CHANGELOG.md records this.

Tests

• Known-answer test: a single split on [1, 2, 5, 6] gives importance (17 − 0.5 − 0.5) / 4 = 4, and 1 when normalized.
• At full depth, the sum of the importances that are not normalized equals the weighted variance of y (Best and Random splitter, with and without weights).
• If you multiply all sample weights by a constant, the importances do not change.
• Forests: the importances add up to 1, a constant column gets 0, and the informative column gets the highest importance.
• If no split improves the impurity (max_depth = 1 or constant y), all importances are 0.
• The same seed gives the same importances.
• After a serde round trip, the importances are the same. This is an indirect check that root_mass and num_features are restored.
• Doctests for the three public methods.

Change logs

Added

• tree, ensemble: DecisionTreeRegressor, RandomForestRegressor and ExtraTreesRegressor get a compute_feature_importances method.

Changed

• tree, ensemble: the fitted DecisionTreeRegressor, RandomForestRegressor and ExtraTreesRegressor store new fields that the feature importances use. Models serialized with an earlier version cannot be deserialized.

Text of PR was written by an LLM.

[codecov]

Codecov Report

❌ Patch coverage is 96.40719% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.64%. Comparing base (9eaae9e) to head (c34b214).
⚠️ Report is 203 commits behind head on main.

Files with missing lines	Patch %	Lines
src/ensemble/base_forest_regressor.rs	85.00%	3 Missing ⚠️
src/tree/decision_tree_regressor.rs	66.66%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #497       +/-   ##
===========================================
+ Coverage   43.97%   85.64%   +41.67%     
===========================================
  Files          85       97       +12     
  Lines        7281     9456     +2175     
===========================================
+ Hits         3202     8099     +4897     
+ Misses       4079     1357     -2722     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.