Regression in `RandomForestClassifier` predictions for onnxruntime 1.26.0

[betatim]

Describe the issue

I have a converted scikit-learn RandomForestClassifier where the predictions don't match with the "native" scikit-learn model. This worked in onnxruntime 1.25.1 but started failing in 1.26.0. I think this is a regression that might have been caused by the fix for #27533

I'm using these versions:

onnxruntime:  1.26.0
scikit-learn: 1.8.0
skl2onnx:     1.20.0

To reproduce

import numpy as np
import onnxruntime as ort
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType

X, y = make_classification(
    n_samples=500, n_features=10, n_informative=5, random_state=42
)
X = X.astype(np.float32)
X_train, X_test, y_train, _ = train_test_split(
    X, y, test_size=0.2, random_state=42
)

clf = RandomForestClassifier(n_estimators=20, max_depth=8, random_state=42)
clf.fit(X_train, y_train)

sk_preds = clf.predict(X_test)

onnx_model = convert_sklearn(
    clf,
    initial_types=[("float_input", FloatTensorType([None, 10]))],
    options={"zipmap": False},
)

sess = ort.InferenceSession(onnx_model.SerializeToString())
onnx_labels, onnx_probs = sess.run(None, {"float_input": X_test})

match_rate = np.mean(sk_preds == onnx_labels)
prob_row_sums = onnx_probs.sum(axis=1)

print(f"Prediction match rate: {match_rate:.4f} (expect 1.0)")
print(f"Probability row sums:  min={prob_row_sums.min():.4f} max={prob_row_sums.max():.4f} (expect 1.0)")
print(f"Sample probs[0]:       {onnx_probs[0]} (expect non-negative, sum to 1)")
print(f"Sample probs[4]:       {onnx_probs[4]} (expect non-negative, sum to 1)")

Urgency

This is a regression, something that used to work stopped working. There is no exception, so users have to have their own tests/checks to notice this.

There is a workaround available (pin to older version).

Platform

Linux

OS Version

6.17.0-23-generic

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.26.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[betatim]

I put AI to work by asking it to reproduce the issue, read the linked issue, the PR linked to it and then look at the onnxruntime code. It came up with the following. The reasoning in the text sounds plausible, but I've not tried out the patch. Maybe it is useful to help find out the real solution.

---

Fix binary TreeEnsembleClassifier with post_transform=NONE after #27552

PR #27552 removed weights_are_all_positive_ and forced binary classifiers
to always use add_second_class=2/3. For post_transform=LOGISTIC this is
correct (sigmoid(-x) + sigmoid(x) = 1). But for post_transform=NONE the
code computes [-score, score] (negation) instead of [1-score, score]
(complement), producing invalid probabilities that don't sum to 1.

This breaks skl2onnx-converted RandomForestClassifier models which use
post_transform=NONE with all-positive class_weights (probability fractions).

The fix changes case 2/3 to use complement for non-LOGISTIC transforms,
matching the semantics of the old case 0/1 path.

--- a/onnxruntime/core/providers/cpu/ml/ml_common.h
+++ b/onnxruntime/core/providers/cpu/ml/ml_common.h
@@ write_scores function, binary case (scores.size() == 1), case 2/3:
 
       case 2:
       case 3: // 2 = mixed weights, winning class is positive
         if (post_transform == POST_EVAL_TRANSFORM::LOGISTIC) {
           scores.resize(2);
           scores[1] = static_cast<T>(ComputeLogistic(static_cast<float>(scores[0])));
           scores[0] = static_cast<T>(ComputeLogistic(static_cast<float>(-scores[0])));
         } else {
           scores.push_back(scores[0]);
-          scores[0] = -scores[0];
+          scores[0] = 1 - scores[0];
         }
         *Z = static_cast<T>(scores[0]);
         *(Z + 1) = static_cast<T>(scores[1]);
         break;

--- a/onnxruntime/core/providers/cpu/ml/ml_common.h
+++ b/onnxruntime/core/providers/cpu/ml/ml_common.h
@@ batched_update_scores_inplace function, binary case, case 2/3:
 
       case 2: // 2 = mixed weights, winning class is positive
       case 3: // 3 = mixed weights, winning class is negative
         if (post_transform == POST_EVAL_TRANSFORM::LOGISTIC) {
           update_scores = [](const float score, float* output) {
             *output++ = ComputeLogistic(-score);
             *output = ComputeLogistic(score);
           };
         } else {
           update_scores = [](const float score, float* output) {
-            *output++ = -score;
+            *output++ = 1.f - score;
             *output = score;
           };
         }
         break;

[hariharans29]

@xadupre - Can you please take a look ? It looks like PR #27552 caused a regression in 1.26.0 ?