Repository navigation
feat: support learning-to-rank tuning in TuneHyperparameters - #2778
Rana Singh (ranadeepsingh) merged 7 commits into
Conversation
Add a ranking-only route with whole-query cross-validation and graded NDCG, then refit the selected ranker on the full input. Keep existing classifier and regressor tuning unchanged. Clear copied reference datasets so a previously fitted ranker does not reuse unrelated binning. Cover Scala, native LightGBM and the generated Python API, and add a website example. Refs microsoft#934
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Hey Lenin Mookiah (@leninworld) 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
There was a problem hiding this comment.
🟡 Changes recommended
Quoted gain keys bypass validation, allowing training gains to differ silently from the tuning metric.
1 open issue
What changed in this PR
Adds learning-to-rank support to TuneHyperparameters, addressing the tuning and Python-example portion of #934 while preserving classification and regression behavior.
Changes:
- Adds query-preserving folds, graded NDCG evaluation, and best-model refitting.
- Adds ranking validation and failure cleanup.
- Adds Scala/Python tests and a documented Python example.
| File | Description |
|---|---|
| lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split1/VerifyLightGBMRankingTuning.scala | Tests native ranker tuning and gain validation. |
| lightgbm/src/test/python/synapsemltest/lightgbm/test_ranking_tuning.py | Tests the generated Python tuning API. |
| docs/Quick Examples/estimators/core/_AutoML.md | Documents ranking usage and limitations. |
| core/src/test/scala/com/microsoft/azure/synapse/ml/automl/VerifyRankingTuning.scala | Tests folds, metrics, validation, and cleanup. |
| core/src/main/scala/com/microsoft/azure/synapse/ml/automl/TuneHyperparameters.scala | Routes rankers to ranking-specific tuning. |
| core/src/main/scala/com/microsoft/azure/synapse/ml/automl/RankingTuning.scala | Implements ranking evaluation and model selection. |
Review effort: Balanced
Normalize pass-through gain keys consistently with LightGBM, including sampled parameters. Cancel Spark jobs while allowing sibling driver cleanup to complete before bounded forced shutdown. Add quoted-key and interruption regressions. Docker validation passes: 16 core and four native LightGBM tests, affected Scala compilation/style, and both cleanup tests in three fresh JVM runs. Refs microsoft#934
## Summary Merge master at e8b71d8 into the existing learning-to-rank tuning contribution. Preserve all five contributor commits and leave their production code, tests and example unchanged. ## Prompting Intent Prepare microsoft#2778 using the external-contributor review and PR-readiness workflows, retaining the contributor's ownership and running validation on the existing pull request. ## Linked Sources - Pull request: microsoft#2778 - Feature request: microsoft#934 - Integrated target: microsoft@e8b71d8 - Contributor head: microsoft@e2708b2 ## Rationale Use a history-preserving merge instead of assuming permission to rewrite an external contributor's commits. This permits an ordinary fast-forward push to the existing branch. The merge only incorporates dependency and workflow updates already accepted on master; it does not change the ranking implementation or expand the contribution's scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a191824-df94-4aa3-b2a0-391782cfaa15
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
## Summary Reject inconsistent effective prediction columns across all ranking candidates and sampled runs before training. Add two rejection regressions, a shared-column winner control, and document the candidate-column contract. ## Prompting Intent Make microsoft#2778 ready using the external-contributor review and PR-loop safeguards. Preserve the contributor's commits and implementation intent, address verified review findings, and prove public behavior before pushing a focused maintainer follow-up to the existing branch. ## Linked Sources - Contribution: microsoft#2778 - Original request: microsoft#934 - Schema review: microsoft#2778 (comment) ## Rationale The tuner advertises the first estimator's output schema but returns the winning model. A later candidate with a different prediction column could therefore break downstream pipeline stages. Validate the common column in the existing ranking plan instead of changing the shared classification/regression route or renaming the winner's output after training. The regressions cover both an unsampled incompatible candidate and a better-scoring second candidate. They assert that rejection performs no training. The compatible control checks the winner, schema shape and materialized prediction values without conflating Spark's model-only metadata with an output-column mismatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a191824-df94-4aa3-b2a0-391782cfaa15
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
🔵 Needs a closer look
Supplied evidence reports hosted CI as pending and does not establish distributed multi-executor sibling-cancellation behavior, so final maintainer validation is needed.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
|
Thanks Lenin Mookiah (@leninworld) for adding query-preserving ranking tuning and the Python example! This makes I preserved all five of your commits and added two focused maintainer follow-ups:
The original unsupported-ranker regression fails against the base implementation and passes with your change. The two new column-rejection regressions likewise fail before the guard and pass afterward; the shared-column winner control passes both before and after. Azure build 242160268, triggered with GitHub checks are green, and the current-head automated review has no open or suppressed findings. Native tests do not establish distributed multi-executor sibling-cancellation behavior; the documented estimator-interruption responsibility remains unchanged. Please review these follow-ups and sign off if they fit your intent. If they do not fit your vision for the PR, feel free to revert my commits, or let me know and I will revert them. Contributor sign-off and the required human maintainer approval remain; this comment is not a merge approval. #934 retains its separate input-format and external-optimizer example scope. |
LGTM. Thank you! |
71346cf
into
microsoft:master
TuneHyperparameterscurrently rejects rankers, so Python users cannot use it to tuneLightGBMRanker. This PR adds query-preserving cross-validation, graded NDCG evaluation, and a Python example while leaving classification and regression tuning unchanged.Related Issues/PRs
#934. This covers learning-to-rank tuning and its Python example; additional input-format examples and external optimization-library integrations remain outside this PR.
What changes are proposed in this pull request?
TuneHyperparameterscurrently rejects rankers during evaluation. Add a ranking-only route that keeps each query group in one fold, evaluates graded NDCG (ndcgorndcg@k), selects the best run and refits it on the full input. The classifier and regressor route is unchanged.The route rejects mixed estimator kinds, inconsistent query/label/prediction columns, structural search-space changes and custom label gains. Fresh estimator copies discard prior reference-dataset binning. On a failed parallel run, sibling Spark job groups are cancelled and running futures are interrupted. The caller waits for actual worker termination, even beyond the shutdown wait interval, before releasing the tuner’s cached data and exposing the original cause. Estimators remain responsible for their own interruption handling and internal cleanup; an estimator that ignores interrupts can delay return.
The website includes a Python LightGBMRanker example and the metric/scaling limits. The metric uses LightGBM's default gain and discount and zero-IDCG convention, but does not promise identical native evaluation: it uses pessimistic prediction ties and equal query weights. Labels must be integer relevance values in [0, 30]. Fold assignment sorts distinct query groups in one partition.
getBestModelInfo()remains unsupported for rankers.How is this patch tested?
Maintainer validation
The history-preserving integration commit
9f55e5deb2bc8b9beb052407815eecd0b75b6d02merges current mastere8b71d877fa9e5517d6e5a4e384a3994a179b315. All five contributor commits remain intact; that integration preserved the six-file contribution patch byte-for-byte.The focused follow-up
b1a37c48976e845e841a1d1e9d961501c82d1383addresses the subsequent prediction-column review: candidates must agree on their effective prediction column before training, so a different winner cannot invalidate the tuner's advertised output column. It adds three regressions and clarifies the documented contract; classification/regression behavior is untouched.Local validation used WSL Linux, JDK 11, Spark 3.5.0 and Scala 2.12.17:
sbt core/compile core/Test/compile lightgbm/compile lightgbm/Test/compile— passed on the final follow-up source.sbt "core/testOnly com.microsoft.azure.synapse.ml.automl.VerifyRankingTuning"— passed all 26 tests on the final follow-up source, with no canceled, ignored, or pending tests.sbt "lightgbm/testOnly com.microsoft.azure.synapse.ml.lightgbm.split1.VerifyLightGBMRankingTuning"— passed all six native tests on the final follow-up source, with no canceled, ignored, or pending tests.sbt core/scalastyle core/Test/scalastyle lightgbm/Test/scalastyle— passed on the final follow-up source with zero errors or warnings.sbt 'core/testOnly com.microsoft.azure.synapse.ml.automl.VerifyRankingTuning -- -z "prediction column"'— before the guard, both incompatible-column regressions failed because fitting incorrectly succeeded; the compatible-column winner control passed. All three pass with the guard as part of the full 26-test suite. The negative tests verify zero training calls, including an incompatible candidate not selected bynumRuns.python3 -m black --check lightgbm/src/test/python/synapsemltest/lightgbm/test_ranking_tuning.py— passed.sbt core/codegen lightgbm/codegen— passed on the unchanged contributor source.python3 pr2778_baseline_replay.py— the baseTuneHyperparameters.scalaimplementation fails the public fit regression withModel type not supported for evaluation; the same regression passes against the PR implementation. Baseline classes were compiled separately; repository source was not mutated.python3 pr2778_python_smoke.py— passed the contributor's generated-Python regression with JVM class provenance verified against this checkout. Additional real LightGBM fitting produced 24 finite predictions and NDCG 1.0; model save/load preserved predictions and the metric.The two Python replay scripts and logs are retained in the maintainer's session evidence, not committed as new product tests. Native local tests use single-task training; they do not establish distributed multi-executor sibling-cancellation behavior.
Full Azure validation build 242160268 passed all 65 jobs for the follow-up after
/azp run(reason=pullRequest, source4191bc66722effc89ae586d282571990d9dd6df0, with verified current-target and final-head parents).Published individual results confirm 26/26 core ranking tests (including all three prediction-column regressions), 6/6 native ranking tests, and
test_tune_ranker_from_pythonin the 16/16 LightGBM Python suite. All 111 website code-block tests passed, including all three_AutoML.mdblocks. The existingVerifyTuneHyperparameterssuite also passed 13/13 tests. Relevant run IDs are1183508397,1183512101,1183527263,1183519282and1183508399, respectively.The first
UnitTests nnattempt passed all 40 tests but failed coverage publication because of a TLS certificate-name mismatch. Only that failed job was retried, on the same source SHA; its second attempt passed tests and publication. That retry changed no source, workflow, dependency pin or check policy. Three unrelated deep-learning tests retain their explicit pre-existing skips; no ranking or website scenario was skipped.GitHub validation passed, including compilation/style, Python formatting, both CodeQL language analyses, dependency review, website build and link checking. Website deployment is intentionally skipped for PRs; the separate CodeQL aggregate is skipped while both language analyses pass. The final-head automated review has no open or suppressed findings, and all review threads are resolved. Contributor sign-off and the required human approval remain outstanding.
Earlier integration build 242154399 published 23 passing core and six passing native ranking tests. Its
RTests corejob failed obtaining an OIDC token because ofERR_TLS_CERT_ALTNAME_INVALID; the R test task was skipped and coverage was consequently absent. That infrastructure failure is not counted as a passing check; this older run was superseded when the follow-up was pushed.Contributor's original and historical validation evidence
The test-teardown follow-up always interrupts the tuner caller in
finally, before releasing the worker cleanup latch and joining. Production code is unchanged. A controlled startup-assertion failure left the caller alive with the preceding teardown; the validation harness then rescued it. With the corrected teardown, the caller had already stopped. Both injected runs intentionally report the assertion failure and exit after the observation/rescue checks; they are not counted as passing test suites.On the exact restored one-line patch, fresh Docker validation passes 23 core ranking tests, six native LightGBM tests, affected compilation and three style lanes. Focused interruption/failure/cleanup cases pass in three fresh JVMs per module. Temporary fault-injection variants were restored before these checks.
The preceding worker-interruption correction keeps worker and caller interruption separate. A worker-thrown
InterruptedExceptionpropagates as the exact original exception without setting the caller's interrupt flag. Actual caller interruptions are restored after cleanup; caller-sideExecutionExceptionvalues are not accidentally unwrapped.Fresh Docker validation on that two-file correction passed 23 core ranking tests, six native LightGBM tests, affected main/test compilation and three Scala style lanes. Focused interruption/failure/cleanup cases pass in three fresh JVMs per module. The new worker-only regression fails on published predecessor
91c720b8(0/1), then passes with this patch. It verifies exception identity, a clear caller interrupt flag, a subsequent blocking call and cached-RDD cleanup. A real interrupted caller also waits for worker cleanup and returns with its interrupt flag restored.Worker-interruption validation from the preceding production revision (rendered summary of retained Docker logs; not a terminal capture):
Prior cancellation/cleanup revision (historical validation):
The cancellation/cleanup revision interrupted running sibling futures, releases ranking-score caches on the caller, and waits for actual worker termination before releasing shared folds. Caller interruption is restored after cleanup, and an existing worker failure remains the primary exception. Estimators remain responsible for their own internal cleanup under interruption.
On that revision, 18 core ranking tests and six native LightGBM tests passed, along with affected Scala compilation and style checks. Failure/cleanup cases pass in three fresh JVMs per affected module. The interruption regression fails on the preceding published source; the slow-cleanup tests reject a one-wait shutdown mutant. Native failure controls run a single failing LightGBM run (default transfer without validation data, and bulk transfer with validation data) and check cached RDD cleanup; they do not exercise sibling interruption inside LightGBM or audit broadcast cleanup.
One initial isolated native failure-only JVM stalled after its passing assertions and was stopped; the published-code control and an unchanged-candidate repeat both exited normally. The full suite and the three required fresh-JVM runs exited successfully.
The quoted
label_gainguards from the earlier review remain covered. The broader validation and two original feature images below are historical; Python, documentation and classifier/regressor source is unchanged.Docker on Linux AMD64, Spark 3.5.0, Java 11.0.32, Scala 2.12, Python 3.11.8 and LightGBM 3.3.510:
Hosted CI and the full all-module fuzzing suite have not run locally.
Latest cancellation/cleanup validation (rendered summary of retained Docker logs; not a terminal capture):
Original feature validation summaries (historical typed summaries of retained Docker logs; not terminal captures):
Does this PR change any dependencies?
Does this PR add a new feature? If so, have you added samples on website?
The Python learning-to-rank example is in
docs/Quick Examples/estimators/core/_AutoML.md, with the website test marker and existing API table.website/docs/documentationfolder.Make sure you choose the correct class
estimators/transformersand namespace.DocTablepoints to correct API link.npm startto make sure the website renders correctly.<!--pytest-codeblocks:cont-->before each python code blocks to enable auto-tests for python samples.WebsiteSamplesTestsjob pass in the pipeline.