From 43912cf1437554a0ebd33605787239990f48daed Mon Sep 17 00:00:00 2001 From: svonava Date: Fri, 25 Sep 2026 08:08:27 +0000 Subject: [PATCH 1/2] perf(models): load three GLiClass models with bucketed CUDA graphs gliclass-base-v1.0, gliclass-large-v1.0 and opir-multitask-large-v1.0 now load with adapter_options.loadtime.cuda_graphs: bucketed. On CUDA their forwards replay as CUDA graphs over sequence lengths padded to buckets. This changes their scores. Probabilities can differ from eager execution by up to 0.0049 (gliclass-base-v1.0), 0.0056 (gliclass-large-v1.0) and 0.0144 (opir-multitask-large-v1.0). The top label was unchanged in all 11,538 answers per model: - 384 CVE descriptions from examples/typed-decisions, three questions each, asked one at a time, as separate groups and as joint groups; - 65 long documents; - each input sent three times, long documents twice. Usage and billing do not change. On an L4, median latency of one call, eager -> graphs: | Model | flat call | separate-groups call | |---|---|---| | gliclass-base-v1.0 | 17.7 -> 3.2 ms | 21.2 -> 8.9 ms | | gliclass-large-v1.0 | 32.5 -> 8.4 ms | 36.2 -> 20.9 ms | | opir-multitask-large-v1.0 | 32.2 -> 8.7 ms | 35.7 -> 20.3 ms | The three models' graphs held about 1.4 GB together on one L4. Each model's graphs are capped at 4% of device memory, so about 2.7 GB for the three at most. To revert a model to eager execution, set cuda_graphs: off in its profile (quoted or not). A request can also send options={"cuda_graphs": "off"}. The other DeBERTa-v3 GLiClass models stay eager: gliclass-small-v1.0, gliclass-base-v3.0, gliclass-large-v3.0 and the instruct models each changed a near-tie top label on the same inputs. So do the mDeBERTa and ModernBERT models. Co-Authored-By: Claude Opus 5.5 (1M context) --- packages/sie_sdk/README.md | 12 ++++---- packages/sie_server/README.md | 29 ++++++++++++------- .../knowledgator__gliclass-base-v1.0.yaml | 3 ++ .../knowledgator__gliclass-large-v1.0.yaml | 3 ++ ...owledgator__opir-multitask-large-v1.0.yaml | 3 ++ .../adapters/test_gliclass_cuda_graphs.py | 29 ++++++++++++++++++- 6 files changed, 62 insertions(+), 17 deletions(-) diff --git a/packages/sie_sdk/README.md b/packages/sie_sdk/README.md index 8b11c182c..697b39dfc 100644 --- a/packages/sie_sdk/README.md +++ b/packages/sie_sdk/README.md @@ -152,11 +152,13 @@ billed. The other items still succeed. On a CUDA server, the operator can load the DeBERTa-based GLiClass models with CUDA graphs, which cut the CPU time spent launching kernels. With `bucketed` graphs, sequence lengths are padded to buckets, which moves probabilities -slightly, as batching requests together does: by up to 0.006 on -`gliclass-large-v1.0` and 0.023 on `gliclass-multilang-mini` in our tests. -Send `options={"cuda_graphs": "off"}` to run a request eagerly; a request -cannot turn graphs on. See the server README for the supported models and -memory use. +slightly, as batching requests together does. `gliclass-base-v1.0`, +`gliclass-large-v1.0` and `opir-multitask-large-v1.0` load with `bucketed` +graphs by default: their probabilities differed from eager execution by up to +0.014 in our tests, with no top label changed. Send +`options={"cuda_graphs": "off"}` to run a request eagerly; a request cannot +turn graphs on. See the server README for each model's measurements and the +memory graphs use. ## Generation prompts and guard verdicts diff --git a/packages/sie_server/README.md b/packages/sie_server/README.md index 61d4e1a3c..c0b1c4e17 100644 --- a/packages/sie_server/README.md +++ b/packages/sie_server/README.md @@ -101,20 +101,27 @@ documents twice), so that graphs were recorded and then replayed: 11,538 answers per model. A small change can still flip a near tie between the top two labels: -| Model | Largest probability change | Top label changed | -|--|--|--| -| `gliclass-small-v1.0` | 0.004 | 3 answers | -| `gliclass-base-v1.0` | 0.005 | none | -| `gliclass-large-v1.0` | 0.006 | none | -| `gliclass-base-v3.0` | 0.005 | 3 | -| `gliclass-large-v3.0` | 0.009 | 3 | -| `gliclass-instruct-base-v1.0` | 0.008 | 12 | -| `gliclass-instruct-large-v1.0` | 0.010 | 18 | -| `opir-multitask-large-v1.0` | 0.014 | none | -| `gliclass-multilang-mini` (100 descriptions, no joint groups: 2,016 answers) | 0.023 | 3 | +| Model | Largest probability change | Top label changed | Shipped profile | +|--|--|--|--| +| `gliclass-small-v1.0` | 0.004 | 3 answers | `off` | +| `gliclass-base-v1.0` | 0.005 | none | `bucketed` | +| `gliclass-large-v1.0` | 0.006 | none | `bucketed` | +| `gliclass-base-v3.0` | 0.005 | 3 | `off` | +| `gliclass-large-v3.0` | 0.009 | 3 | `off` | +| `gliclass-instruct-base-v1.0` | 0.008 | 12 | `off` | +| `gliclass-instruct-large-v1.0` | 0.010 | 18 | `off` | +| `opir-multitask-large-v1.0` | 0.014 | none | `bucketed` | +| `gliclass-multilang-mini` (100 descriptions, no joint groups: 2,016 answers) | 0.023 | 3 | `off` | `exact` changed nothing. +The shipped profiles load with `bucketed` graphs only where no top label +changed: `gliclass-base-v1.0`, `gliclass-large-v1.0` and +`opir-multitask-large-v1.0`. Their probabilities can differ from eager +execution by up to the amounts above. To run one of them eagerly, set +`cuda_graphs: off` in its profile, or send `options={"cuda_graphs": "off"}` with +a request. The other models load with `off`. + Graphs apply on CUDA to the DeBERTa-based GLiClass models: the v1.0 models, `gliclass-base-v3.0` and `gliclass-large-v3.0`, the base and large instruct models, the Opir multitask models and `gliclass-multilang-mini`. The diff --git a/packages/sie_server/models/knowledgator__gliclass-base-v1.0.yaml b/packages/sie_server/models/knowledgator__gliclass-base-v1.0.yaml index 5099ea39e..3858fb038 100644 --- a/packages/sie_server/models/knowledgator__gliclass-base-v1.0.yaml +++ b/packages/sie_server/models/knowledgator__gliclass-base-v1.0.yaml @@ -19,5 +19,8 @@ profiles: adapter_options: loadtime: classification_type: single-label + # Replay forwards as CUDA graphs over bucketed lengths; "off" runs them eagerly. + # See "GLiClass CUDA graphs" in packages/sie_server/README.md. + cuda_graphs: bucketed runtime: threshold: 0.0 diff --git a/packages/sie_server/models/knowledgator__gliclass-large-v1.0.yaml b/packages/sie_server/models/knowledgator__gliclass-large-v1.0.yaml index bf4a13f99..5c1916c8c 100644 --- a/packages/sie_server/models/knowledgator__gliclass-large-v1.0.yaml +++ b/packages/sie_server/models/knowledgator__gliclass-large-v1.0.yaml @@ -19,5 +19,8 @@ profiles: adapter_options: loadtime: classification_type: single-label + # Replay forwards as CUDA graphs over bucketed lengths; "off" runs them eagerly. + # See "GLiClass CUDA graphs" in packages/sie_server/README.md. + cuda_graphs: bucketed runtime: threshold: 0.0 diff --git a/packages/sie_server/models/knowledgator__opir-multitask-large-v1.0.yaml b/packages/sie_server/models/knowledgator__opir-multitask-large-v1.0.yaml index 4cc3dcaa8..4a1ffa25e 100644 --- a/packages/sie_server/models/knowledgator__opir-multitask-large-v1.0.yaml +++ b/packages/sie_server/models/knowledgator__opir-multitask-large-v1.0.yaml @@ -19,6 +19,9 @@ profiles: adapter_options: loadtime: classification_type: single-label + # Replay forwards as CUDA graphs over bucketed lengths; "off" runs them eagerly. + # See "GLiClass CUDA graphs" in packages/sie_server/README.md. + cuda_graphs: bucketed runtime: threshold: 0.0 multi-label: diff --git a/packages/sie_server/tests/adapters/test_gliclass_cuda_graphs.py b/packages/sie_server/tests/adapters/test_gliclass_cuda_graphs.py index 43018d303..21b1b1fa1 100644 --- a/packages/sie_server/tests/adapters/test_gliclass_cuda_graphs.py +++ b/packages/sie_server/tests/adapters/test_gliclass_cuda_graphs.py @@ -7,6 +7,7 @@ from __future__ import annotations +from pathlib import Path from types import SimpleNamespace from typing import Any @@ -22,9 +23,18 @@ segment_ids, unsupported_reason, ) -from sie_server.core.loader import reject_unknown_loadtime_options +from sie_server.core.loader import _build_adapter_kwargs, load_model_configs, reject_unknown_loadtime_options from sie_server.types.inputs import InvalidInputError +_MODELS_DIR = Path(__file__).resolve().parents[2] / "models" +# The shipped profiles that load with graphs: DeBERTa-v3 models whose bucketed +# scores changed no top label against eager execution (see the server README). +_BUCKETED_BY_DEFAULT = { + "knowledgator/gliclass-base-v1.0", + "knowledgator/gliclass-large-v1.0", + "knowledgator/opir-multitask-large-v1.0", +} + class _Encoder: def __init__(self) -> None: @@ -542,3 +552,20 @@ def test_a_request_that_opts_out_runs_eagerly(self) -> None: _adapter_with(runner)._forward(_Pipe(), _inputs(1, 100), ["a", "b"], same_labels=True, graphs="off") assert runner.replayed == [] + + +def test_shipped_profiles_enable_bucketed_graphs_only_where_measured() -> None: + configs = load_model_configs(_MODELS_DIR) + # Named profiles (``model:profile``) inherit the default profile's load-time options. + modes = { + name.split(":")[0]: config.resolve_profile("default").loadtime.get("cuda_graphs", "off") + for name, config in configs.items() + if config.resolve_profile("default").adapter_path.endswith(":GLiClassAdapter") + } + + assert {name for name, mode in modes.items() if mode != "off"} == _BUCKETED_BY_DEFAULT + assert {modes[name] for name in _BUCKETED_BY_DEFAULT} == {"bucketed"} + for name in _BUCKETED_BY_DEFAULT: + adapter = GLiClassAdapter(**_build_adapter_kwargs(configs[name], "float16")) + assert adapter._cuda_graphs == "bucketed" + assert adapter._request_cuda_graphs({"cuda_graphs": "off"}) == "off" From 6657c27071d8dd4e042dac8378c40f74d33601b5 Mon Sep 17 00:00:00 2001 From: svonava Date: Fri, 25 Sep 2026 09:31:06 +0000 Subject: [PATCH 2/2] docs: give GLiClass bucketed-graph maxima to four decimals The README table rounded each model's largest probability change to three decimals, so 0.0144 on opir-multitask-large-v1.0 read as 0.014 and was no longer an upper bound. Both READMEs now give the measured values. Co-Authored-By: Claude Opus 5.5 (1M context) --- packages/sie_sdk/README.md | 2 +- packages/sie_server/README.md | 18 +++++++++--------- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/packages/sie_sdk/README.md b/packages/sie_sdk/README.md index 697b39dfc..206b97cea 100644 --- a/packages/sie_sdk/README.md +++ b/packages/sie_sdk/README.md @@ -155,7 +155,7 @@ graphs, sequence lengths are padded to buckets, which moves probabilities slightly, as batching requests together does. `gliclass-base-v1.0`, `gliclass-large-v1.0` and `opir-multitask-large-v1.0` load with `bucketed` graphs by default: their probabilities differed from eager execution by up to -0.014 in our tests, with no top label changed. Send +0.0144 in our tests, with no top label changed. Send `options={"cuda_graphs": "off"}` to run a request eagerly; a request cannot turn graphs on. See the server README for each model's measurements and the memory graphs use. diff --git a/packages/sie_server/README.md b/packages/sie_server/README.md index c0b1c4e17..8e087f07a 100644 --- a/packages/sie_server/README.md +++ b/packages/sie_server/README.md @@ -103,15 +103,15 @@ two labels: | Model | Largest probability change | Top label changed | Shipped profile | |--|--|--|--| -| `gliclass-small-v1.0` | 0.004 | 3 answers | `off` | -| `gliclass-base-v1.0` | 0.005 | none | `bucketed` | -| `gliclass-large-v1.0` | 0.006 | none | `bucketed` | -| `gliclass-base-v3.0` | 0.005 | 3 | `off` | -| `gliclass-large-v3.0` | 0.009 | 3 | `off` | -| `gliclass-instruct-base-v1.0` | 0.008 | 12 | `off` | -| `gliclass-instruct-large-v1.0` | 0.010 | 18 | `off` | -| `opir-multitask-large-v1.0` | 0.014 | none | `bucketed` | -| `gliclass-multilang-mini` (100 descriptions, no joint groups: 2,016 answers) | 0.023 | 3 | `off` | +| `gliclass-small-v1.0` | 0.0039 | 3 answers | `off` | +| `gliclass-base-v1.0` | 0.0049 | none | `bucketed` | +| `gliclass-large-v1.0` | 0.0056 | none | `bucketed` | +| `gliclass-base-v3.0` | 0.0054 | 3 | `off` | +| `gliclass-large-v3.0` | 0.0093 | 3 | `off` | +| `gliclass-instruct-base-v1.0` | 0.0076 | 12 | `off` | +| `gliclass-instruct-large-v1.0` | 0.0098 | 18 | `off` | +| `opir-multitask-large-v1.0` | 0.0144 | none | `bucketed` | +| `gliclass-multilang-mini` (100 descriptions, no joint groups: 2,016 answers) | 0.0227 | 3 | `off` | `exact` changed nothing.