[Raashish Aggarwal (raashish1601)]

Why are these changes needed?

BaseEstimator.score(X, y, metric=...) (also used by AutoML.score) always used predict(), even for metrics that need class probabilities. As a result:

• roc_auc / ap on a binary task were computed from hard 0/1 labels and gave a silently wrong value (e.g. 0.922 instead of 0.996 for an LGBM model on breast cancer);
• log_loss, roc_auc_ovr etc. on a multiclass task raised errors (y_prob contains values greater than 1, y_score needs to be of shape (n_samples, n_classes)).

The search itself already picks between predict and predict_proba with get_y_pred() in flaml/automl/ml.py. score() now uses the same function. To keep passing extra keyword arguments through, get_y_pred() gets an optional **pred_kwargs that it forwards to predict/predict_proba. Existing callers are unchanged.

Related issue number

None. Found while checking AutoML.score with the metrics listed in its docstring.

Checks

• I've used pre-commit to lint the changes in this PR (note the same in integrated in our CI checks).
• I've included any doc changes needed for https://microsoft.github.io/FLAML/. No doc change needed; the docstring already lists these metrics.
• I've added tests (if relevant) corresponding to the changes introduced in this PR. test/automl/test_estimator_score_metrics.py covers binary roc_auc and multiclass log_loss/roc_auc_ovr. All three fail on main and pass with this change. test/automl/test_score.py classification tests pass locally.
• I've made sure all auto checks have passed.

[Li Jiang (thinkall)]

Overall review of the complete PR: approved.

Using the existing get_y_pred() metric dispatch in estimator scoring selects predict_proba() for probability metrics while retaining predict() for label metrics. Forwarding prediction kwargs preserves the existing score API, and the regression tests compare binary AUC and multiclass probability metrics with scikit-learn. No blocking issues found; the build matrix was still pending at review time.

Posted by thinkall-agent-auto-reviewer

[Copilot]

🟡 Changes recommended

Binary ROC-AUC variants can receive invalid 2-D predictions, and the new test has a lint-breaking unused import.

2 open findings

• Remove unused numpy import from test module · New
• Pass positive-class scores for binary ROC-AUC metrics · New

What changed in this PR

Updates estimator scoring to use probabilities when required by classification metrics.

Changes:

• Routes scoring predictions through get_y_pred.
• Forwards prediction keyword arguments.
• Adds binary and multiclass metric tests.

File	Description
flaml/​automl/​model.py	Selects predictions based on metric requirements.
flaml/​automl/​ml.py	Adds prediction-argument forwarding.
test/​automl/​test_estimator_score_metrics.py	Tests probability-based scoring.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Li Jiang (thinkall)]

Overall review of the complete updated PR: approved.

The post-review source commit corrects get_y_pred() so binary roc_auc_ovr, roc_auc_ovo and weighted variants receive positive-class probabilities instead of a two-column array, matching sklearn's binary ROC-AUC input. New regression cases cover all five binary variants while retaining multiclass probability checks. The original estimator.score() dispatch continues to use probabilities for probability metrics and labels for label metrics; the base refresh changes mape weighting separately and introduces no conflicting change to this dispatch. No blocking issue found in the updated PR.

Posted by thinkall-agent-auto-reviewer

[Copilot]

🟡 Changes recommended

Standalone estimators using generic classification still route binary probability metrics incorrectly.

1 open issue

• Generic classification task breaks binary ROC AUC scoring · New

2 resolved since last review

• Remove unused numpy import from test module
• Pass positive-class scores for binary ROC-AUC metrics

_{Review effort: Balanced}

[Li Jiang (thinkall)]

This should be addressed.

[Copilot]

🟢 Approval recommended

The focused fix has regression coverage and no unresolved findings; tests were not executed during this review.

1 open issue

• Generic classification task breaks binary ROC AUC scoring

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Standalone-estimator task compatibility remains unverified and needs human review.

1 open issue

• Generic classification task breaks binary ROC AUC scoring

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The focused changes address the reported scoring failures with regression coverage and no remaining actionable findings.

1 open issue

• Generic classification task breaks binary ROC AUC scoring

_{Review effort: Balanced}

[Li Jiang (thinkall)]

This should be addressed.

[Copilot]

🟡 Changes recommended

String tasks retained by directly fitted Spark classifiers cause probability-based scoring to fail.

2 open issues

• Normalize Spark estimator task before prediction and scoring · New
• Generic classification task breaks binary ROC AUC scoring

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The previously reported probability-metric scoring issue for directly fitted estimators using task="classification" remains unresolved.

2 open issues

• Normalize Spark estimator task before prediction and scoring
• Generic classification task breaks binary ROC AUC scoring

_{Review effort: Balanced}

[Li Jiang (thinkall)]

Re-reviewed commits f507063f1d84ec43a4647d6d31c32a251a4da5d3 -> 304bb93119c0a317619fdde5b120c10067d1af82. The only delta is main's website/yarn.lock update of follow-redirects 1.16.0 -> 1.16.1. One explicitly head-bound review-code Low execution (actual runtime gpt-6.1-sol) found no new issue in this identical inherited patch. Controller Git comparisons verify all non-lockfile tree entries are unchanged. This is a scoped merge-delta review, not a fresh clean signoff for unchanged feature code.

Version/URL/caret/SRI shape checks pass. No independent local registry/tarball verification, package smoke test or website rebuild is claimed. Previous feature findings and runtime validation limits remain preserved.

At the 2026-10-26T12:13:44.943946+00:00 gate snapshot, actual current-head workflows were: docs completed/success; Build in_progress/no conclusion; Code formatting completed/success. Copilot has Balanced review without an Approval recommended verdict; 2 review thread(s) remain unresolved. Gate approval is not successful CI completion, and no new PR approval is being submitted while required gates remain unmet.

Posted by thinkall-agent-auto-reviewer