Repository navigation
Merged
Conversation
The pass that gives prune_components() its components (family_columns()) started every family's intercept at 0 and skipped the family's null tree. A family that made no split holds its whole fit in that tree's one leaf (fit_null_leaf()), so its value was missing from the pruned model's intercept: with some families without a split, rule "D" at tau = 0 was not the forest. purify() starts the intercept from the null tree's value; the pass now does the same, with the same function. Forests whose families all split are unchanged.
The OOB path tracked every family's predictions from the score 0 its rows start from, and added the leaf values of each split. A family that has made no split predicts its null leaf's fit (fit_null_leaf(): the mean of its rows, or for logit and exponential the log-odds of its sample), so the path held 0 where the forest with that many splits predicts that fit; for logit and exponential a probability of 1/2. With a rare class and few rows many families make no split in their first rounds, and the OOB log-loss of few splits came out too high, so that the adaptive choice of n_splits could grow far past the best number of splits. path_start() now computes the null leaf's fit for every tracked family (the adaptive fit did so already for its rebuild), path_split() marks a family that has split, and path_snapshot() gives a family without a split at its fit at every checkpoint, for the fixed fit's path and the adaptive one. Families with a split are tracked as before, so forests whose families all split by the first checkpoint are unchanged.
A forest with loss = "logit" or "exponential" averages its tree families on the score scale. Where the families disagree, the probability of their mean score lies below the mean of their probabilities for a rare class (the probability is convex in the score below 1/2), the more so the more splits they make: the forest predicted a rare class too rarely. After the fit the forest's score now gets one constant per score column, its level: the maximum-likelihood intercept of the log-loss given the forest's out-of-bag scores, whose equations are the balance property (every class's mean out-of-bag probability is its frequency), with the second-order term of the out-of-bag correction for an out-of-bag mean of fewer families than the forest. classif_level() solves them by Newton's method with step halving (3 to 6 steps). The level is added to every family's null leaf (shift_level()), so predict(), purify(), the pruning's components and saved models see it as part of the intercept; $level holds it and print() shows it. deterministic = TRUE balances the forest's own predictions. The adaptive choice of n_splits scores every checkpoint at its own level and the forest is shifted by the chosen checkpoint's; a fit with fixed splits tracks its OOB path at its last split for the level. A forest with a level is purified after the shift; the pruning's cross-validation refits each fold with its own level. The level is on by default for both losses, binary and multiclass; options(randomPlantedForest.level = FALSE) fits without it, bit for bit as before. The setting is stored in $params$level, and a fit without it is refitted without a level. Regression and the L1 / L2 classification losses are unchanged. The tests of the losses' steps and of the OOB path's growth run without the level; the pinned digests of the logit and exponential fits and the print() snapshot of a deterministic logit fit are re-pinned. The classification article gains a short section on the level, and its table of a pure leaf's steps is fitted without it.
classif_level() built its Newton step with crossprod() and solve(), i.e. BLAS and LAPACK, and summed with colSums(), rowSums() and sum(), which accumulate in long double. The step's last bits then depend on the library: OpenBLAS's triangular solve multiplies by the reciprocal of the pivot where the reference BLAS divides by it. With R's reference BLAS (CRAN's macOS R) the level of two pinned fits, cls_cur_trees_2 and loss3_exponential, came out a few ulps away from OpenBLAS's (Homebrew R), so test-fit-digests.R failed on GitHub's macOS runner. The level's sums are now pairwise sums in double arithmetic (level_colsums(), level_sum()) or column by column (level_rowsums()), the Jacobian is summed term by term, and the step is solved by Gaussian elimination with the diagonal as pivots (level_solve(); the Jacobian is positive definite). Nothing in the level calls BLAS or LAPACK any more, and its bits depend only on its inputs and on exp() and log(). The levels move at rounding level (at most 5e-15 in the 52 digests' fits, the same number of Newton steps), so the 13 digests of the logit and exponential fits are re-pinned.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The level (third commit)
A classification forest averages its tree families on the score (log-odds) scale. Where the families disagree, the
probability of the mean score is not the mean of their probabilities, so the forest predicts a rare class too
rarely, and more so the more splits it makes.
After the fit, the forest's score now gets one constant per score column, its level, chosen so that each class's mean
out-of-bag probability equals the class's frequency in the training data. It is part of the forest's intercept, so
predict(),predict_components(),purify(), the pruning and saved models all see it;$levelholds it.loss = "logit"and"exponential", binary and multiclass. Classification fits thereforechange.
options(randomPlantedForest.level = FALSE)turns it off, and every fit is then the same as before, bitfor bit.
falls, most where a class is rare. The default pruned model changes little, since its refit already sets its own
intercepts.
Two bug fixes (first two commits)
Both concern a tree family that made no split.
purify().rows, few splits then looked too bad out of bag, and the out-of-bag choice of
n_splitscould take far moresplits than it should. Fixed for every loss.
Each fix has its own test and changes no fit whose families all split.
Checks
The test suite passes on macOS and Linux. With the level turned off, all fit digests are unchanged.