diff --git a/README.md b/README.md index 017dbe7bc..db9c6bc27 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ One SIE cluster runs the inference behind a whole agent. Each task is a handful | **Search** | Embed, match, and rerank to retrieve the right context. | [`bge-m3`](packages/sie_server/models/BAAI__bge-m3.yaml), [`splade-v3`](packages/sie_server/models/naver__splade-v3.yaml), [`colbertv2`](packages/sie_server/models/colbert-ir__colbertv2.0.yaml), [`qwen3-reranker`](packages/sie_server/models/Qwen__Qwen3-Reranker-4B.yaml) | | **Document to markdown** | PDFs, Office files, and scans become clean markdown. | [`lightonocr`](packages/sie_server/models/lightonai__LightOnOCR-2-1B.yaml), [`glm-ocr`](packages/sie_server/models/zai-org__GLM-OCR.yaml), [`mineru`](packages/sie_server/models/opendatalab__MinerU2.5-Pro-2604-1.2B.yaml), [`paddleocr-vl`](packages/sie_server/models/PaddlePaddle__PaddleOCR-VL-1.5.yaml), [`docling`](packages/sie_server/models/docling.yaml) | | **Structured output** | Schema-valid JSON, extracted or generated. | [`gliner2`](packages/sie_server/models/fastino__gliner2-large-v1.yaml), [`gliner-relex`](packages/sie_server/models/knowledgator__gliner-relex-large-v1.0.yaml), [`gliformer`](packages/sie_server/models/knowledgator__gliformer-large-v1.yaml), [`nuner-zero`](packages/sie_server/models/numind__NuNER_Zero.yaml), [`qwen3.8-27b`](packages/sie_server/models/Qwen__Qwen3.8-27B-FP8.yaml), [`qwen3.6-27b`](packages/sie_server/models/Qwen__Qwen3.6-27B.yaml) | -| **Decide** | Choice, yes/no, and score answers with probabilities to typed questions about a text or JSON state. | [`laya`](packages/sie_server/models/convaiinnovations__laya.yaml), [`laya-multilingual`](packages/sie_server/models/convaiinnovations__laya-multilingual.yaml), [`laya-typed-decisions`](packages/sie_server/models/convaiinnovations__laya-typed-decisions.yaml) | +| **Decide** | Choice, yes/no, and score answers with probabilities to typed questions about a text or JSON state. | [`laya`](packages/sie_server/models/convaiinnovations__laya.yaml), [`laya-multilingual`](packages/sie_server/models/convaiinnovations__laya-multilingual.yaml), [`laya-typed-decisions`](packages/sie_server/models/convaiinnovations__laya-typed-decisions.yaml), [`gliner2.5-decide`](packages/sie_server/models/fastino__GLiNER2.5-Decide.yaml), [`gliner2.5-multi-decide`](packages/sie_server/models/fastino__GLiNER2.5-multi-Decide.yaml), [`gliner2.5-decide-1b`](packages/sie_server/models/fastino__GLiNER2.5-Decide-1B.yaml) | | **Classify** | Zero-shot labels, with several label groups answered in one call. The instruct models also follow a task instruction and few-shot examples. | [`gliclass-large-v3`](packages/sie_server/models/knowledgator__gliclass-large-v3.0.yaml), [`gliclass-instruct-large`](packages/sie_server/models/knowledgator__gliclass-instruct-large-v1.0.yaml), [`gliclass-multilang-mini`](packages/sie_server/models/knowledgator__gliclass-multilang-mini.yaml) | | **Guard content** | A safety verdict: Yes/No with the threshold set in the model config, or safe/unsafe and policy-label scores with the threshold chosen per request. | [`granite-guardian-2b`](packages/sie_server/models/ibm-granite__granite-guardian-3.0-2b.yaml), [`opir-multitask-large`](packages/sie_server/models/knowledgator__opir-multitask-large-v1.0.yaml), [`opir-edge`](packages/sie_server/models/knowledgator__opir-edge-v1.0.yaml) | | **Run the agent loop** | Plan steps and call tools with an open LLM, streaming included. | [`qwen3.8-27b`](packages/sie_server/models/Qwen__Qwen3.8-27B-FP8.yaml), [`qwen3.6-27b`](packages/sie_server/models/Qwen__Qwen3.6-27B.yaml) | @@ -69,7 +69,7 @@ docker run --gpus all -p 8080:8080 \ -v sie-hf-cache:/app/.cache/huggingface \ ghcr.io/superlinked/sie-server:latest-cuda12-default -# Linux, NVIDIA GPU: Transformers 5 OCR models (LightOnOCR and GLM-OCR) +# Linux, NVIDIA GPU: Transformers 5 models (LightOnOCR, GLM-OCR, and the GLiNER2.5-Decide models) docker run --gpus all -p 8080:8080 \ -v sie-hf-cache:/app/.cache/huggingface \ ghcr.io/superlinked/sie-server:latest-cuda12-transformers5 @@ -87,7 +87,8 @@ docker run -p 8080:8080 \ Docker images are bundle-specific so dependency-incompatible model families stay isolated. Use the `sglang-vision-extract` image for LightOnOCR, GLM-OCR, and PaddleOCR-VL, or the `transformers5` image for the -`:transformers` profiles of LightOnOCR and GLM-OCR; the `default` image intentionally does not advertise them. +`:transformers` profiles of LightOnOCR and GLM-OCR and for the GLiNER2.5-Decide models; the `default` image +intentionally does not advertise them. ```bash # in a second terminal @@ -182,6 +183,30 @@ print(result["data"]["department"]) # values are illustrative and rounded # {'type': 'choice', 'choice': 'billing', 'probabilities': {'billing': 0.987, 'technical': 0.013}, 'confidence': 0.9} ``` +The GLiNER2.5-Decide models (`fastino/GLiNER2.5-Decide` for English, `GLiNER2.5-multi-Decide`, and +`GLiNER2.5-Decide-1B`), served by the `transformers5` image from step 1, take the same questions, GLiClass-style `options={"label_groups": {...}}`, or plain `labels`, +and return every option's probability. They read all of a call's questions next to the document in one row per item: +one forward pass answers them all, and each question's probabilities depend on the other questions sent with it. A +`score` question is read as the ordinal labels `"0"` to `"k-1"`, each described by its criterion; a `noul` question as +`"yes"`/`"no"`. `usage.input_tokens` counts the document tokens the model reads plus the questions' instructions and +criteria text; question ids and label names are not counted. + +```python +result = client.extract( + "fastino/GLiNER2.5-Decide", + Item(text="Guest in room 1408 says the AC has been out since yesterday and wants to move rooms tonight."), + output_schema={ + "intent": {"type": "choice", "instructions": "What does the guest want?", + "criteria": {"room_change": "move to another room", "maintenance": "fix something", "checkout": None}}, + "needs_human": {"type": "noul", "instructions": "Must a person act on this?"}, + "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["low", "normal", "high", "urgent"]}, + }, +) +print(result["data"]["intent"]) # values are illustrative and rounded +# {'type': 'choice', 'choice': 'room_change', +# 'probabilities': {'room_change': 0.871, 'maintenance': 0.085, 'checkout': 0.045}, 'confidence': 0.57} +``` + Text generation runs on the GPU generation image; stop the first server, then start this one on the same port: ```bash diff --git a/packages/sie_sdk/README.md b/packages/sie_sdk/README.md index 206b97cea..62590e066 100644 --- a/packages/sie_sdk/README.md +++ b/packages/sie_sdk/README.md @@ -160,6 +160,45 @@ graphs by default: their probabilities differed from eager execution by up to turn graphs on. See the server README for each model's measurements and the memory graphs use. +## Typed decisions + +Typed-decision models answer typed questions about each item and return a +probability for every option. The Laya models and the GLiNER2.5-Decide models +take the same question mapping as `output_schema`: `choice` (pick one of the +criteria), `noul` (yes or no), and `score` (an ordinal scale, index 0 first). +Answers come back in `data`, keyed by question id. + +```python +result = client.extract( + "fastino/GLiNER2.5-Decide", + Item(text="This is the third time I have explained the same missing refund. Get me a person."), + output_schema={ + "intent": { + "type": "choice", + "instructions": "What does the customer want?", + "criteria": {"refund_request": "money back", "cancel": None, "complaint": "unhappy with service"}, + }, + "handoff": {"type": "noul", "instructions": "Should a person take over?"}, + "frustration": {"type": "score", "instructions": "How frustrated is the customer?", + "criteria": ["calm", "annoyed", "angry"]}, + }, +) +answers = result["data"] +print(answers["intent"]["choice"], answers["intent"]["probabilities"]) +# e.g. refund_request {'refund_request': 0.59, 'cancel': 0.07, 'complaint': 0.34} +print(answers["handoff"]["answer"], answers["handoff"]["noul"]) # True 0.99 (the probability of yes) +print(answers["frustration"]["score"]) # e.g. 1.13, the expected level from 0 to 2 +``` + +`choice` and `score` answers carry `probabilities` and `confidence` +(`1 - entropy / log(number of options)`); a `noul` answer carries `noul`, the +probability of yes, `answer`, and `confidence` (`max(p, 1 - p)`). The +GLiNER2.5-Decide models also accept `options={"label_groups": {...}}` with +`"classification_type": "multi-label"` for independent per-label scores, and +plain `labels`, which return every label in `classifications`. They read all +of a call's questions in one row per item, so each question's probabilities +depend on the other questions sent with it. + ## Generation prompts and guard verdicts `generate` and `stream_generate` treat text-only prompts as raw continuation diff --git a/packages/sie_server/README.md b/packages/sie_server/README.md index 8e087f07a..07f6807a0 100644 --- a/packages/sie_server/README.md +++ b/packages/sie_server/README.md @@ -26,10 +26,15 @@ pip install sie-server pip install sie-server "transformers<5" ``` -- **OCR-VLM bundle** (LightOnOCR, GLM-OCR) — requires `transformers` 5.x, and is served with `-b transformers5`: +- **Transformers 5 bundle** (LightOnOCR, GLM-OCR, GLiGuard, and the GLiNER2.5-Decide models) — requires + `transformers` 5.x, and is served with `-b transformers5`. The GLiNER2.5-Decide models also need `gliner2` + 2.x. `sie-server` itself asks for `gliner2<2`, which the default bundle's GLiNER2 models need, so pip + reports that conflict when the second command below installs 2.x; the transformers5 bundle's GLiNER2 + models are verified on 2.0.0: ```bash pip install sie-server "transformers>=5,<6" + pip install "gliner2==2.0.0" # only for the GLiNER2.5-Decide models sie-server serve -b transformers5 ``` @@ -169,6 +174,38 @@ largest shape on `gliclass-large-v1.0`. On a GPU shared with other models, leave memory headroom, or enable graphs only where the model has the GPU to itself. Usage and billing do not change. +### GLiNER2.5-Decide usage and limits + +The GLiNER2.5-Decide models (`fastino/GLiNER2.5-Decide`, `GLiNER2.5-multi-Decide`, +`GLiNER2.5-Decide-1B`) run on `gliner2` 2.x, which the transformers5 bundle +pins (the `transformers5` image, or a native install as described above). Each +item is one encoder row: every question's (or label group's) name, +instruction, and labels, then the document. One forward pass answers them all, +so the questions of a request are not independent: adding or changing one can +change another's probabilities and score. `usage.input_tokens` counts the +document tokens the model reads plus the tokens of the instructions and label +descriptions (criteria) sent with the item, as Laya and GLiClass count +instructions and criteria. Question ids, group names, and label names are not +counted, and an item that returns an error counts nothing. + +A request takes at most 64 questions or label groups, 64 options per question, +and 1,024 options in total. Question ids and group names may have 128 +characters, labels 256, and each instruction or description 2,048, with 65,536 +characters in all. Strings that contain one of the model's prompt markers +(`[L]`, `[P]`, `[DESCRIPTION]`, ...) are refused. The questions may take at most +512 tokens, or half the model's window when that is less: 256 of +`GLiNER2.5-Decide`'s 512 tokens, 512 of the others' 2,048. This bounds the +uncounted question and label tokens read with each item; a request needing more +fails with `INPUT_TOO_LONG`. The +document is read up to the whole words that fit in the rest of the window; a +word longer than 4,096 characters, text past 64 characters per token of the +window, or 4 words per token of the window also ends what is read. Words are +split as gliner2 splits them, in linear time. A conversation (a list state) is +read from its newest turn back; a run of more than 4,096 characters without a +space is read only in its last 4,096 characters, and reading stops there. An item none of whose words fits, or that does +not fit whole with `options={"overflow_policy": "error"}`, returns a per-item +`INPUT_TOO_LONG` error while the other items succeed. + ## Configuration `sie-server` reads its config from `SIE_*` environment variables (Pydantic diff --git a/packages/sie_server/bundles/default.yaml b/packages/sie_server/bundles/default.yaml index 3be358ec3..d0d4d4f3f 100644 --- a/packages/sie_server/bundles/default.yaml +++ b/packages/sie_server/bundles/default.yaml @@ -76,7 +76,10 @@ deps: requests: '' # gliner, gliner_bi (0.2.26: first release that loads the relex checkpoints) gliner: '>=0.2.26,<1' - # gliner2 + # gliner2. Held at 1.x: gliner2 2.0 changes gliner2-base/large-v1 outputs + # (every repeated NER mention is returned, and structured extraction is + # rescored). The GLiNER2.5-Decide models, which need 2.x, are served from + # the transformers5 bundle. gliner2: '>=1.3.1,<2' # glirel glirel: '>=1.0,<2' diff --git a/packages/sie_server/bundles/transformers5.yaml b/packages/sie_server/bundles/transformers5.yaml index 35167784b..640340965 100644 --- a/packages/sie_server/bundles/transformers5.yaml +++ b/packages/sie_server/bundles/transformers5.yaml @@ -7,6 +7,7 @@ adapters: - sie_server.adapters.lighton_ocr.adapter - sie_server.adapters.glm_ocr - sie_server.adapters.gliner2.classification +- sie_server.adapters.gliner2.decide - sie_server.adapters.st_sparse_vision.adapter deps: # Transformers 5.x for models requiring newer architecture support @@ -30,5 +31,10 @@ deps: # unsatisfiable against that pin and broke `uv run` resolution. # Let transformers drive the version. pillow: '' - # gliguard-LLMGuardrails-300M uses GLiNER2 classification APIs. - gliner2: '>=1.3.1,<2' + # gliguard-LLMGuardrails-300M uses GLiNER2 classification APIs, and the + # GLiNER2.5-Decide models need gliner2 2.x (AutoExtractor, boundary + # architecture). Exact: the Decide adapter builds the task prompt with the + # package's processor. GLiGuard's outputs are unchanged from 1.3.2. gliner2 + # 2.0 declares transformers<5 only for its `local` extra, which is not used; + # the multilingual and 1B Decide checkpoints were saved with transformers 5. + gliner2: '==2.0.0' diff --git a/packages/sie_server/models/fastino__GLiNER2.5-Decide-1B.yaml b/packages/sie_server/models/fastino__GLiNER2.5-Decide-1B.yaml new file mode 100644 index 000000000..aecd7d659 --- /dev/null +++ b/packages/sie_server/models/fastino__GLiNER2.5-Decide-1B.yaml @@ -0,0 +1,38 @@ +# GLiNER2.5-Decide-1B: typed-decision classifier on the Ettin 1B encoder +# (jhu-clsp/ettin-enc-from-dec-1b, ModernBERT; span architecture), loaded through +# gliner2 2.x AutoExtractor. Served from the transformers5 bundle, which pins gliner2 2.x. +# +# Checkpoint caveat: the weights at this revision are the step-285000 checkpoint +# (upload commit eac5ac1e). The 59.6% fast-decisions score on the model card was +# measured on an earlier step-195000 checkpoint (stated in that revision's card); +# the uploaded weights had not been re-evaluated. +# +# The encoder config stores its RoPE bases only in the transformers-5 +# rope_parameters form (160000 for full and sliding-window layers). The adapter +# checks the bases the loaded encoder actually uses, and under transformers 4 +# writes them where that version reads them (it would otherwise run the +# sliding-window layers at 10000). +sie_id: fastino/GLiNER2.5-Decide-1B +hf_id: fastino/GLiNER2.5-Decide-1B +hf_revision: 52c94d3b698bf6d2619df9d898bdc1523ea3f1ca +inputs: + text: true + image: false + audio: false + video: false +tasks: + encode: null + score: null + extract: {} +# Tokens per item: every question's prompt and labels, then the document. The +# Ettin encoder config sets max_position_embeddings 7999 and the tokenizer +# model_max_length 8192; 2048 tokens stays within both. +max_sequence_length: 2048 +profiles: + default: + max_batch_tokens: 16384 + compute_precision: float16 + adapter_path: sie_server.adapters.gliner2.decide:GLiNER2DecideAdapter + adapter_options: + loadtime: {} + runtime: {} diff --git a/packages/sie_server/models/fastino__GLiNER2.5-Decide.yaml b/packages/sie_server/models/fastino__GLiNER2.5-Decide.yaml new file mode 100644 index 000000000..c5f69b696 --- /dev/null +++ b/packages/sie_server/models/fastino__GLiNER2.5-Decide.yaml @@ -0,0 +1,28 @@ +# GLiNER2.5-Decide: English typed-decision classifier (DeBERTa-v3-large encoder, +# span architecture), loaded through gliner2 2.x AutoExtractor. Served from the +# transformers5 bundle, which pins gliner2 2.x. +sie_id: fastino/GLiNER2.5-Decide +hf_id: fastino/GLiNER2.5-Decide +hf_revision: 7ee5da4c2415e32259bcdc0b1a7367c32ce8d6f6 +inputs: + text: true + image: false + audio: false + video: false +tasks: + encode: null + score: null + extract: {} +# Tokens per item: every question's prompt and labels, then the document. 512, +# as for fastino/gliner2-large-v1, this model's base: its DeBERTa-v3-large encoder +# was pretrained on 512 positions (max_position_embeddings), and neither the model +# card nor the checkpoint config (max_len null) gives a longer length. +max_sequence_length: 512 +profiles: + default: + max_batch_tokens: 16384 + compute_precision: float16 + adapter_path: sie_server.adapters.gliner2.decide:GLiNER2DecideAdapter + adapter_options: + loadtime: {} + runtime: {} diff --git a/packages/sie_server/models/fastino__GLiNER2.5-multi-Decide.yaml b/packages/sie_server/models/fastino__GLiNER2.5-multi-Decide.yaml new file mode 100644 index 000000000..3887e0707 --- /dev/null +++ b/packages/sie_server/models/fastino__GLiNER2.5-multi-Decide.yaml @@ -0,0 +1,27 @@ +# GLiNER2.5-multi-Decide: multilingual typed-decision classifier (mDeBERTa-v3-base +# encoder, boundary architecture; trained with max_len 4096 words), loaded through +# gliner2 2.x AutoExtractor. Served from the transformers5 bundle, which pins gliner2 2.x. +sie_id: fastino/GLiNER2.5-multi-Decide +hf_id: fastino/GLiNER2.5-multi-Decide +hf_revision: 6bc1d43d201b0691e733626389af8c57eea3ea68 +inputs: + text: true + image: false + audio: false + video: false +tasks: + encode: null + score: null + extract: {} +# Tokens per item: every question's prompt and labels, then the document. The +# checkpoint config sets max_len 4096 (gliner2 counts it in words), and the +# tokenizer sets no length limit; 2048 tokens stays within both. +max_sequence_length: 2048 +profiles: + default: + max_batch_tokens: 16384 + compute_precision: float16 + adapter_path: sie_server.adapters.gliner2.decide:GLiNER2DecideAdapter + adapter_options: + loadtime: {} + runtime: {} diff --git a/packages/sie_server/pyproject.toml b/packages/sie_server/pyproject.toml index 2a54f634a..29f8ae871 100644 --- a/packages/sie_server/pyproject.toml +++ b/packages/sie_server/pyproject.toml @@ -46,6 +46,8 @@ dependencies = [ # never shipped, killing three extract cells at model load (#2873). # 0.2.26 is the first release that loads the joint entity-relation checkpoints. "gliner>=0.2.26,<1", + # gliner2 stays on 1.x with the default bundle (2.0 changes gliner2-base/large + # outputs); the transformers5 bundle overlays the 2.x pin its GLiNER2.5 models need. "gliner2>=1.3.1,<2", "glirel>=1.0,<2", # 0.1.17 adds the cross-attention scorer and pass-through pooling used by diff --git a/packages/sie_server/scripts/generate_gliner2_decide_goldens.py b/packages/sie_server/scripts/generate_gliner2_decide_goldens.py new file mode 100644 index 000000000..6e07ceb21 --- /dev/null +++ b/packages/sie_server/scripts/generate_gliner2_decide_goldens.py @@ -0,0 +1,262 @@ +#!/usr/bin/env python3 +"""Generate reference outputs for the GLiNER2.5-Decide adapter parity tests. + +Companion to ``packages/sie_server/tests/adapters/test_gliner2_decide_parity.py``. + +The script runs the ``gliner2`` package directly, with no SIE code involved: +``AutoExtractor`` loads the pinned checkpoint and ``gliner2.classification`` +scores it in float32 on the CPU. Each case records the SIE ``extract`` request +under test next to the ``ClassificationSchema`` that request must translate to, +and stores every label's probability (``ClassificationScores.probability``) and +a hash of the encoder row the package builds for each text. + +A long document is read only as far as the model window: the words whose tokens +fit after the task prompt (``window`` minus the prompt, as the package's +processor builds and tokenizes both). The script computes that word count with +the package's own processor, passes it as ``ClassificationConfig(max_len=...)``, +and records it. + +Run it with the stack the models are served on (the transformers5 bundle); the +output records that stack under ``generated_with``: + + uv run --no-project --python 3.12 --with gliner2==2.0.0 --with transformers==5.17.0 \\ + --with torch==2.9.1 python packages/sie_server/scripts/generate_gliner2_decide_goldens.py \\ + --model fastino/GLiNER2.5-Decide --revision 7ee5da4c2415e32259bcdc0b1a7367c32ce8d6f6 --window 512 \\ + --out packages/sie_server/tests/adapters/goldens/gliner2_decide/fastino__GLiNER2.5-Decide.json +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from importlib.metadata import version +from pathlib import Path +from typing import Any + +import torch +from gliner2 import AutoExtractor +from gliner2.classification import ClassificationConfig, ClassificationSchema, Classifier +from huggingface_hub import snapshot_download + +CHECKPOINT_FILES = ["config.json", "encoder_config/*", "tokenizer*", "special_tokens_map.json", "model.safetensors"] + +TEXTS = [ + "My subscription renewed on April 15 for 5,400 yen after the service was already down. " + "Can I get that charge refunded?", + "Guest in room 1408 says the AC has been out since yesterday and they want to move tonight or leave. " + "They also asked for the incidentals hold to be released.", + "Das Paket kam beschädigt an, bitte schicken Sie Ersatz. Ich brauche es bis Freitag, sonst storniere ich " + "die Bestellung!", + "请帮我取消订单，我不想要了。退款什么时候到账？", + "The treaty was signed in Paris in 1992. It entered into force the following year, after the last signatory " + "ratified it.", +] +_REPORT = [ + "Quarterly operations report for the northern warehouse, week {n}.", + "The team shipped {orders} orders and handled {returns} returns; late deliveries fell to {late} percent.", + "Carrier contract {n} renewed without changes, and the forklift inspection passed on schedule.", + "Payroll ran on time for {staff} staff members and no overtime disputes were filed.", +] +LONG_TEXT = " ".join( + sentence.format(n=n, orders=12000 + 37 * n, returns=300 + n, late=max(1, 9 - n % 8), staff=40 + n % 5) + for n in range(60) + for sentence in _REPORT +) + ( + " In the last week the payroll system failed during the month-end close and salaries for 40 staff are " + "delayed; finance needs a fix before the Friday 5pm cutoff" +) + +INTENTS = { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": None, + "other": "", +} +QUESTIONS = { + "intent": {"type": "choice", "instructions": "What does the customer want?", "criteria": INTENTS}, + "urgency": { + "type": "score", + "instructions": "How urgent is this request?", + "criteria": ["not urgent", "within a week", "today", "blocking or past a deadline"], + }, + "needs_human": { + "type": "noul", + "instructions": "Must a person act on this?", + "criteria": {"true": "automation cannot resolve it"}, + }, + "rating": { + "type": "score", + "instructions": "Rate the severity from 0 to 10.", + "criteria": [str(i) for i in range(11)], + }, +} +PASSAGE_QUESTION = {"answer": {"type": "noul", "instructions": "Did the treaty enter into force in 1992?"}} +GROUPS = { + "intent": ["refund_request", "cancel_order", "room_change", "maintenance", "other"], + "priority": ["low", "normal", "high", "urgent"], + "needs_human": ["yes", "no"], +} +MULTI_GROUPS = {"topics": ["billing", "hvac", "shipping", "payroll", "account"], "channel": ["email", "chat", "phone"]} +LABELS = ["billing", "technical", "hospitality", "logistics", "other"] + + +def _single(name: str, labels: Any, instruction: str | None = None) -> dict[str, Any]: + return {"name": name, "kind": "single", "labels": labels, "instruction": instruction} + + +def _multi(name: str, labels: Any, instruction: str | None = None) -> dict[str, Any]: + return {"name": name, "kind": "multi", "labels": labels, "instruction": instruction} + + +def _described(criteria: dict[str, str | None]) -> dict[str, str | None]: + return {label: description or None for label, description in criteria.items()} + + +# Each case: the SIE request, the documents it reads, and the gliner2 tasks it must become. +CASES: list[dict[str, Any]] = [ + { + "name": "questions", + "request": {"output_schema": QUESTIONS}, + "documents": "texts", + "tasks": [ + _single("intent", _described(INTENTS), "What does the customer want?"), + _single( + "urgency", + {"0": "not urgent", "1": "within a week", "2": "today", "3": "blocking or past a deadline"}, + "How urgent is this request?", + ), + _single("needs_human", {"yes": "automation cannot resolve it", "no": None}, "Must a person act on this?"), + _single("rating", [str(i) for i in range(11)], "Rate the severity from 0 to 10."), + ], + }, + { + "name": "question_over_passage", + "request": {"output_schema": PASSAGE_QUESTION}, + "documents": "texts", + "tasks": [_single("answer", ["yes", "no"], "Did the treaty enter into force in 1992?")], + }, + { + "name": "label_groups", + "request": {"instruction": "Triage the message.", "options": {"label_groups": GROUPS}}, + "documents": "texts", + "tasks": [_single(name, labels, "Triage the message.") for name, labels in GROUPS.items()], + }, + { + "name": "label_groups_multi_label", + "request": {"options": {"label_groups": MULTI_GROUPS, "classification_type": "multi-label"}}, + "documents": "texts", + "tasks": [_multi(name, labels) for name, labels in MULTI_GROUPS.items()], + }, + { + "name": "labels", + "request": {"labels": LABELS, "instruction": "Which team should handle this?"}, + "documents": "texts", + "tasks": [_single("label", LABELS, "Which team should handle this?")], + }, + { + "name": "long_document", + "request": {"output_schema": QUESTIONS}, + "documents": "long", + "tasks": None, # same tasks as "questions" + }, +] + + +def build_schema(tasks: list[dict[str, Any]]) -> ClassificationSchema: + schema = ClassificationSchema() + for task in tasks: + add = schema.single if task["kind"] == "single" else schema.multi + add(task["name"], task["labels"], instruction=task["instruction"]) + return schema + + +def readable_words(processor: Any, model_schema: dict[str, Any], text: str, window: int) -> int | None: + """Words of ``text`` the model reads after the task prompt, or None when all of them fit.""" + prefix = processor.transform_and_format(".", model_schema).text_word_first_positions[0] + room = window - prefix + if not text.endswith((".", "!", "?")): + text += "." # as the processor's collate does + used = 0 + for count, (word, _, _) in enumerate(processor.word_splitter(text, lower=True)): + tokens = len(processor.tokenizer.tokenize(word)) + if used + tokens > room: + return count + used += tokens + return None + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--model", required=True) + parser.add_argument("--revision", required=True) + parser.add_argument("--window", type=int, required=True, help="the model config's max_sequence_length") + parser.add_argument("--out", type=Path, required=True) + args = parser.parse_args() + + torch.manual_seed(0) + path = snapshot_download(args.model, revision=args.revision, allow_patterns=CHECKPOINT_FILES) + model = AutoExtractor.from_pretrained(path, map_location="cpu") + model.eval() + classifier = Classifier(model) + processor = model.processor + + cases = [] + for case in CASES: + tasks = case["tasks"] if case["tasks"] is not None else CASES[0]["tasks"] + compiled = classifier.compile_schema(build_schema(tasks)) + model_schema = compiled.build() + documents = TEXTS if case["documents"] == "texts" else [LONG_TEXT] + results = [] + for text in documents: + max_len = readable_words(processor, model_schema, text, args.window) + batch = processor.collate_fn_inference([(text, model_schema)], max_len=max_len) + row = batch.input_ids[0, : batch.original_lengths[0]].tolist() + assert len(row) <= args.window, "reference row exceeds the window" + with torch.inference_mode(): + scores = classifier.score(text, compiled, config=ClassificationConfig(max_len=max_len)) + probabilities = { + task["name"]: {label: scores.probability(task["name"], label) for label in scores.tasks[task["name"]]} + for task in tasks + } + results.append( + { + "max_len": max_len, + "row_length": len(row), + "row_sha256": hashlib.sha256(json.dumps(row).encode()).hexdigest(), + "probabilities": probabilities, + } + ) + cases.append( + { + "name": case["name"], + "request": case["request"], + "documents": case["documents"], + "reference_call": {"tasks": tasks}, + "results": results, + } + ) + + golden = { + "model": args.model, + "revision": args.revision, + "generated_with": { + "gliner2": version("gliner2"), + "transformers": version("transformers"), + "torch": torch.__version__, + "device": "cpu", + "dtype": "float32", + "window": args.window, + }, + "texts": TEXTS, + "long_text": LONG_TEXT, + "cases": cases, + } + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(json.dumps(golden, indent=1, ensure_ascii=False) + "\n", encoding="utf-8") + print(f"wrote {args.out}") + + +if __name__ == "__main__": + main() diff --git a/packages/sie_server/src/sie_server/adapters/gliner2/decide.py b/packages/sie_server/src/sie_server/adapters/gliner2/decide.py new file mode 100644 index 000000000..5a38f92c6 --- /dev/null +++ b/packages/sie_server/src/sie_server/adapters/gliner2/decide.py @@ -0,0 +1,739 @@ +"""GLiNER2.5-Decide typed-decision adapter (``extract``). + +Serves Fastino's GLiNER2.5-Decide classifiers through the ``gliner2`` 2.x +package: ``AutoExtractor`` loads the checkpoint (span or boundary architecture), +and the package's own processor builds the task prompt. Every label of every +question gets a probability. + +Request contract (see ``decisions.py``): Laya's typed question mapping in +``output_schema`` (answers in ``data[question_id]``), GLiClass's +``options.label_groups`` (answers in ``data[group]`` and ``"group.label"`` +``classifications``), or plain ``labels`` (every label in ``classifications``). +Items are texts, or Laya states in ``metadata.state`` (a string, a JSON object, +or a list of conversation turns, of which the newest are kept). + +One row per item. The model reads all of a request's tasks and the document in +one sequence, ``(task prompts and labels) [SEP_TEXT] document``, and scores every +label from its ``[L]`` marker in a single forward pass, as +``gliner2.classification.Classifier`` does. A task's probabilities therefore +depend on the other tasks sent with it. The task prompt is built once per +request by the package's processor; each document's words are appended the way +the processor appends them (lowercased words, each tokenized on its own). + +Window: the task prompt may take at most ``max_prompt_tokens`` (default 512, or +half of ``max_seq_length`` if less), and the document is cut to the whole words +that fit in the rest. Words are read lazily, so work stops at the window: a word +longer than ``_MAX_WORD_CHARS`` characters, or reaching past +``_MAX_CHARS_PER_TOKEN`` characters per token of the window, also ends the part +the model reads. An item none of whose words fits returns a per-item +``INPUT_TOO_LONG`` error, as does any item that does not fit whole when +``options.overflow_policy`` is ``"error"``. + +Usage: an item's input tokens are its document tokens the model reads plus the +tokens of the caller's free text encoded with it (question instructions and +label descriptions), as the Laya and GLiClass adapters count instructions and +criteria. Task names and label names are not counted. Errored items count +nothing. +""" + +from __future__ import annotations + +import json +import logging +import re +import tempfile +from collections.abc import Iterator, Mapping +from contextlib import contextmanager +from dataclasses import dataclass +from functools import lru_cache +from pathlib import Path +from typing import Any, ClassVar + +import numpy as np +import torch +import transformers +from huggingface_hub import snapshot_download + +from sie_server.adapters._base_adapter import BaseAdapter +from sie_server.adapters._spec import AdapterSpec +from sie_server.adapters._types import ERR_NOT_LOADED, ComputePrecision +from sie_server.adapters.errors import InputTooLongError +from sie_server.adapters.gliner2.decisions import ( + DecisionRequest, + answer, + approx_json_chars, + classifications, + parse_request, + probabilities, + schema_chars, + split_logits, +) +from sie_server.adapters.gliner2.words import linear_equivalent +from sie_server.adapters.laya.questions import STATE_BYTES_PER_TOKEN, serialize_state +from sie_server.core.inference_output import ExtractItemError, ExtractOutput +from sie_server.types.inputs import InvalidInputError, Item +from sie_server.types.overflow_policy import DEFAULT_OVERFLOW_POLICY, VALID_OVERFLOW_POLICIES +from sie_server.types.responses import Classification, ErrorCode + +logger = logging.getLogger(__name__) + +# Files a GLiNER2.5-Decide checkpoint needs; the Hub repos also carry a banner image and an agent skill file. +_CHECKPOINT_FILES = ("config.json", "encoder_config/*", "tokenizer*", "special_tokens_map.json", "model.safetensors") +_DEFAULT_WINDOW = 512 +# Tokens the task prompt may take by default (at most half the window). Task +# and label names are not billed, so this bounds the unbilled work a request +# attaches to each item, as GLiFormer's max_prompt_tokens does. +_DEFAULT_MAX_PROMPT_TOKENS = 512 +# Padded tokens per forward pass; larger requests run in several passes. +_DEFAULT_INFERENCE_BATCH_TOKENS = 16384 +# A document word longer than this ends the part of the document the model reads. +_MAX_WORD_CHARS = 4096 +# ...as does reaching past this many characters per token of the window. Prose +# spends about 5; text laid out with runs of spaces about 40. +_MAX_CHARS_PER_TOKEN = 64 +# ...or reading this many words per token of the window: a word of characters +# the tokenizer drops takes no room. +_MAX_WORDS_PER_TOKEN = 4 +# A run of non-space characters. gliner2's words never cross a space. +_RUN = re.compile(r"\S+") +# Batching-cost estimate: characters per window token. +_COST_CHARS_PER_TOKEN = 4 +# Token ids of recent words, kept across requests. Only words of at most +# _CACHED_WORD_CHARS characters are kept, so the cache holds a few tens of MB at +# most; a longer word (a hash, an id, a base64 run) is tokenized each time. +_WORD_CACHE_SIZE = 16384 +_CACHED_WORD_CHARS = 32 +_SENTENCE_END = (".", "!", "?") + +_ERR_ITEM = ( + "GLiNER2.5-Decide items need a non-blank text, or metadata.state holding a string, a JSON object, " + "or a list of turns (not both)" +) +_ERR_STATE_JSON = "GLiNER2.5-Decide metadata.state must be JSON-serializable" +_ERR_NO_WORD_FITS = ( + "No word of the document fits in the {room} tokens the questions leave in the model's {window}-token window " + "(a single word longer than {max_word} characters is not read)" +) +_ERR_DOES_NOT_FIT = ( + "The document does not fit in the {room} tokens the questions leave in the model's {window}-token window " + "(overflow_policy 'error')" +) +_ERR_NON_FINITE = "GLiNER2.5-Decide returned non-finite scores for this item" + +# ModernBERT RoPE bases: transformers 5 writes them per layer type in +# ``rope_parameters``; transformers 4 reads only these flat fields, with these defaults. +_ROPE_FIELDS = {"full_attention": "global_rope_theta", "sliding_attention": "local_rope_theta"} +_TRANSFORMERS4_ROPE_DEFAULTS = {"full_attention": 160000.0, "sliding_attention": 10000.0} +_ROPE_RELATIVE_TOLERANCE = 1e-3 +_LAYER_INDEX = re.compile(r"(?:^|\.)layers\.(\d+)\.") + + +# --------------------------------------------------------------------------- +# Checkpoint compatibility +# --------------------------------------------------------------------------- + + +def _transformers_major() -> int: + return int(transformers.__version__.split(".", 1)[0]) + + +def declared_rope_thetas(encoder_config: Mapping[str, Any]) -> dict[str, float] | None: + """The RoPE base a ModernBERT checkpoint declares for each layer type, or None for other encoders. + + Raises: + ValueError: The checkpoint declares RoPE parameters this adapter cannot verify. + """ + if encoder_config.get("model_type") != "modernbert": + return None + rope = encoder_config.get("rope_parameters") + if rope is None: + return { + layer_type: float(encoder_config.get(field, _TRANSFORMERS4_ROPE_DEFAULTS[layer_type])) + for layer_type, field in _ROPE_FIELDS.items() + } + if not isinstance(rope, Mapping) or not set(rope) <= set(_ROPE_FIELDS): + raise ValueError(f"Unsupported ModernBERT rope_parameters {rope!r}") + thetas: dict[str, float] = {} + for layer_type in _ROPE_FIELDS: + params = rope.get(layer_type) + if params is None: + continue + theta = params.get("rope_theta") if isinstance(params, Mapping) else None + if ( + not isinstance(params, Mapping) + or params.get("rope_type", "default") != "default" + or isinstance(theta, bool) + or not isinstance(theta, (int, float)) + ): + raise ValueError(f"Unsupported ModernBERT {layer_type} RoPE parameters {params!r}") + thetas[layer_type] = float(theta) + return thetas + + +def transformers4_encoder_config(encoder_config: Mapping[str, Any]) -> dict[str, Any] | None: + """The encoder config with its RoPE bases where transformers 4 reads them, or None when nothing changes. + + transformers 5 saves ModernBERT's RoPE bases only in ``rope_parameters``; + transformers 4 ignores that and runs sliding-window layers at its default + base of 10000. GLiNER2.5-Decide-1B's Ettin encoder uses 160000 for both. + + Raises: + ValueError: The layer layout cannot be expressed in transformers 4. + """ + if _transformers_major() >= 5 or encoder_config.get("rope_parameters") is None: + return None + thetas = declared_rope_thetas(encoder_config) + if not thetas: + return None + every = encoder_config.get("global_attn_every_n_layers", 3) + layer_types = encoder_config.get("layer_types") + if layer_types is not None: + expected = [ + "full_attention" if index % every == 0 else "sliding_attention" for index in range(len(layer_types)) + ] + if list(layer_types) != expected: + raise ValueError( + "ModernBERT layer_types do not follow global_attn_every_n_layers; transformers 4 cannot load them" + ) + patched = dict(encoder_config) + for layer_type, theta in thetas.items(): + patched[_ROPE_FIELDS[layer_type]] = theta + return patched if patched != dict(encoder_config) else None + + +def transformers4_tokenizer_config(tokenizer_config: Mapping[str, Any]) -> dict[str, Any] | None: + """The tokenizer config naming a class transformers 4 has, or None when nothing changes. + + transformers 5 records a generic fast tokenizer as ``TokenizersBackend``; its + ``tokenizer.json`` loads unchanged as a ``PreTrainedTokenizerFast``. + """ + if _transformers_major() >= 5 or tokenizer_config.get("tokenizer_class") != "TokenizersBackend": + return None + return {**tokenizer_config, "tokenizer_class": "PreTrainedTokenizerFast"} + + +def _read_json(path: Path) -> dict[str, Any]: + with path.open(encoding="utf-8") as f: + return json.load(f) + + +@contextmanager +def loadable_checkpoint(checkpoint: Path) -> Iterator[Path]: + """``checkpoint``, or a temporary overlay of it with the configs transformers 4 needs. + + The overlay links every file and rewrites only the patched JSON files, so + the cached checkpoint is never modified. + """ + patches: dict[str, dict[str, Any]] = {} + encoder = transformers4_encoder_config(_read_json(checkpoint / "encoder_config" / "config.json")) + if encoder is not None: + patches["encoder_config/config.json"] = encoder + tokenizer_path = checkpoint / "tokenizer_config.json" + if tokenizer_path.is_file(): + tokenizer = transformers4_tokenizer_config(_read_json(tokenizer_path)) + if tokenizer is not None: + patches["tokenizer_config.json"] = tokenizer + if not patches: + yield checkpoint + return + with tempfile.TemporaryDirectory(prefix="sie-gliner2-decide-") as tmp: + root = Path(tmp) + for path in checkpoint.rglob("*"): + if path.is_dir(): + continue + relative = path.relative_to(checkpoint).as_posix() + target = root / relative + target.parent.mkdir(parents=True, exist_ok=True) + if relative in patches: + target.write_text(json.dumps(patches[relative], indent=2), encoding="utf-8") + else: + target.symlink_to(path.resolve()) + logger.info("Loading %s with transformers 4 compatible %s", checkpoint, ", ".join(sorted(patches))) + yield root + + +def loaded_rope_thetas(encoder: torch.nn.Module, every: int) -> dict[str, list[float]]: + """The RoPE base each rotary table of a loaded encoder encodes, by layer type. + + ``inv_freq[i] = theta ** (-i / n)`` for a table of ``n`` frequencies, so + ``theta = inv_freq[1] ** -n``. Layers are typed by name (transformers 5) or + by index (transformers 4: every ``every``-th layer is full attention). + """ + observed: dict[str, list[float]] = {} + for name, buffer in encoder.named_buffers(): + if not name.endswith("inv_freq") or name.endswith("original_inv_freq") or buffer.numel() < 2: + continue + if "full_attention" in name: + layer_type = "full_attention" + elif "sliding_attention" in name: + layer_type = "sliding_attention" + elif (match := _LAYER_INDEX.search(name)) is not None: + layer_type = "full_attention" if int(match.group(1)) % every == 0 else "sliding_attention" + else: + continue + theta = float(buffer[1].double().item() ** -buffer.numel()) + observed.setdefault(layer_type, []).append(theta) + return observed + + +def verify_encoder_rope(encoder: torch.nn.Module, encoder_config: Mapping[str, Any]) -> dict[str, float] | None: + """Check that a loaded ModernBERT encoder runs the RoPE bases its checkpoint declares. + + Run in float32, before any precision cast. Returns the verified bases, or + None for encoders without rotary tables. + + Raises: + RuntimeError: A layer type's rotary tables are missing or encode another base. + """ + expected = declared_rope_thetas(encoder_config) + if not expected: + return None + observed = loaded_rope_thetas(encoder, int(encoder_config.get("global_attn_every_n_layers", 3))) + for layer_type, theta in expected.items(): + values = observed.get(layer_type) + if not values: + raise RuntimeError(f"Cannot find the {layer_type} RoPE tables of the loaded encoder to verify them") + wrong = [value for value in values if abs(value - theta) > _ROPE_RELATIVE_TOLERANCE * theta] + if wrong: + raise RuntimeError( + f"The loaded encoder's {layer_type} layers run RoPE base {wrong[0]:.1f}, but the checkpoint " + f"declares {theta:.1f}" + ) + return expected + + +# --------------------------------------------------------------------------- +# Adapter +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True, slots=True) +class _Prefix: + """A request's task prompt: token ids up to and including ``[SEP_TEXT]``, and each label's marker.""" + + ids: list[int] + label_positions: list[int] + billed_tokens: int + + +@dataclass(frozen=True, slots=True) +class _Document: + """The part of an item's document the model reads.""" + + ids: list[int] + billed_tokens: int + complete: bool + + +class GLiNER2DecideAdapter(BaseAdapter): + """Typed decisions with GLiNER2.5-Decide checkpoints (``gliner2`` 2.x). + + Reference models: + - fastino/GLiNER2.5-Decide + - fastino/GLiNER2.5-multi-Decide + - fastino/GLiNER2.5-Decide-1B + """ + + spec: ClassVar[AdapterSpec] = AdapterSpec( + inputs=("text",), + outputs=("json",), + unload_fields=("_model", "_processor", "_tokenizer", "_word_splitter", "_word_ids", "_word_cache"), + ) + + def __init__( + self, + model_name_or_path: str | Path, + *, + max_seq_length: int | None = None, + max_prompt_tokens: int | None = None, + inference_batch_tokens: int = _DEFAULT_INFERENCE_BATCH_TOKENS, + compute_precision: ComputePrecision = "float16", + revision: str | None = None, + **kwargs: Any, + ) -> None: + """Initialize the adapter. + + Args: + model_name_or_path: Hugging Face repo id or local checkpoint directory. + max_seq_length: Tokens per row: task prompt, ``[SEP_TEXT]``, and document. + max_prompt_tokens: Most tokens the task prompt may take (default: 512, + or half of ``max_seq_length`` when that is less); a larger request + fails with ``INPUT_TOO_LONG``. + inference_batch_tokens: Padded tokens per forward pass. + compute_precision: Accelerator precision (``float16`` is the package's + ``quantize=True``). CPU always runs float32. + revision: Hugging Face revision (commit SHA) to pin. + **kwargs: Additional loader arguments (ignored). + """ + _ = kwargs + window = _DEFAULT_WINDOW if max_seq_length is None else int(max_seq_length) + prompt = min(window // 2, _DEFAULT_MAX_PROMPT_TOKENS) if max_prompt_tokens is None else int(max_prompt_tokens) + if window < 2 or not 0 < prompt < window: + raise ValueError("GLiNER2.5-Decide needs 0 < max_prompt_tokens < max_seq_length") + if int(inference_batch_tokens) < window: + raise ValueError("GLiNER2.5-Decide inference_batch_tokens must hold at least one full row") + self._model_name_or_path = str(model_name_or_path) + self._window = window + self._max_prompt_tokens = prompt + self._inference_batch_tokens = int(inference_batch_tokens) + self._compute_precision = compute_precision + self._revision = revision + + self._model: Any = None + self._processor: Any = None + self._tokenizer: Any = None + self._word_splitter: Any = None + self._word_ids: Any = None + self._word_cache: Any = None + self._device: str | None = None + + # ------------------------------------------------------------------ loading + + def _checkpoint_dir(self) -> Path: + path = Path(self._model_name_or_path) + if path.is_dir(): + return path + return Path( + snapshot_download( + repo_id=self._model_name_or_path, + revision=self._revision, + allow_patterns=list(_CHECKPOINT_FILES), + ) + ) + + def load(self, device: str) -> None: + """Load the checkpoint with ``gliner2.AutoExtractor`` onto ``device``.""" + try: + from gliner2 import AutoExtractor # ty:ignore[unresolved-import] + except ImportError as exc: + raise RuntimeError("GLiNER2.5-Decide needs gliner2 2.x (AutoExtractor)") from exc + + checkpoint = self._checkpoint_dir() + encoder_config = _read_json(checkpoint / "encoder_config" / "config.json") + with loadable_checkpoint(checkpoint) as path: + model = AutoExtractor.from_pretrained(str(path), map_location=device) + if not hasattr(model, "classifier") or not hasattr(model, "encoder"): + raise RuntimeError(f"{self._model_name_or_path} has no classification head") + rope = verify_encoder_rope(model.encoder, encoder_config) + model.eval() + processor = model.processor + processor.change_mode(is_training=False) + + dtype = self._resolve_dtype() if device.startswith("cuda") else torch.float32 + if dtype == torch.float16: + model.quantize() # the package's own half-precision path + elif dtype == torch.bfloat16: + model.to(dtype=torch.bfloat16) + + self._attach(model, processor, device) + logger.info( + "Loaded GLiNER2.5-Decide %s (%s) on %s: %s, window %d tokens, prompt at most %d, RoPE %s", + self._model_name_or_path, + type(model).__name__, + device, + dtype, + self._window, + self._max_prompt_tokens, + rope, + ) + + def _attach(self, model: Any, processor: Any, device: str) -> None: + """Use ``model`` (encoder and classification head) and ``processor`` (the task prompt builder). + + Raises: + RuntimeError: The processor splits words with a splitter this adapter + has no linear-time equivalent for. + """ + splitter = linear_equivalent(processor.word_splitter) + if splitter is None: + raise RuntimeError( + f"GLiNER2.5-Decide has no linear-time equivalent of {type(processor.word_splitter).__name__}; " + "gliner2's word splitter changed" + ) + processor.word_splitter = splitter + tokenizer = processor.tokenizer + + def tokenize(word: str) -> tuple[int, ...]: + return tuple(tokenizer.convert_tokens_to_ids(tokenizer.tokenize(word))) + + cached = lru_cache(maxsize=_WORD_CACHE_SIZE)(tokenize) + + def word_ids(word: str) -> tuple[int, ...]: + return cached(word) if len(word) <= _CACHED_WORD_CHARS else tokenize(word) + + self._model = model + self._processor = processor + self._tokenizer = tokenizer + self._word_splitter = splitter + self._word_ids = word_ids + self._word_cache = cached + self._device = device + + def warmup(self) -> None: + """Run one tiny decision to initialize kernels.""" + self.extract([Item(text="warmup")], labels=["yes", "no"]) + + # ------------------------------------------------------------------ extract + + def extract( + self, + items: list[Item], + *, + labels: list[str] | None = None, + output_schema: dict[str, Any] | None = None, + instruction: str | None = None, + options: dict[str, Any] | None = None, + prepared_items: list[Any] | None = None, + ) -> ExtractOutput: + """Answer every task for every item. See the module docstring for the contract.""" + _ = prepared_items + if self._model is None: + raise RuntimeError(ERR_NOT_LOADED) + opts = options or {} + request = parse_request(labels=labels, output_schema=output_schema, instruction=instruction, options=opts) + overflow_policy = opts.get("overflow_policy", DEFAULT_OVERFLOW_POLICY) + if not isinstance(overflow_policy, str) or overflow_policy not in VALID_OVERFLOW_POLICIES: + raise InvalidInputError( + f"GLiNER2.5-Decide overflow_policy must be one of {sorted(VALID_OVERFLOW_POLICIES)}" + ) + + with self._tokenizer_guard(): + prefix = self._prefix(request) + room = self._window - len(prefix.ids) + documents: list[_Document | None] = [] + errors: list[ExtractItemError | None] = [] + for item in items: + document, error = self._document(item, room, overflow_policy) + documents.append(document) + errors.append(error) + + kept = [(index, document) for index, document in enumerate(documents) if document is not None] + rows = [prefix.ids + document.ids for _, document in kept] + logits = self._score(rows, prefix.label_positions) if rows else np.empty((0, 0), dtype=np.float32) + + data: list[dict[str, Any]] = [{} for _ in items] + item_classifications: list[list[Classification]] = [[] for _ in items] + token_counts = [0] * len(items) + for row, (index, document) in enumerate(kept): + if not np.isfinite(logits[row]).all(): + errors[index] = ExtractItemError(code=ErrorCode.INFERENCE_ERROR.value, message=_ERR_NON_FINITE) + continue + per_task = [ + probabilities(task, part) + for task, part in zip(request.tasks, split_logits(request, logits[row]), strict=True) + ] + if request.mode != "labels": + data[index] = { + task.key: answer(task, p, threshold=request.threshold) + for task, p in zip(request.tasks, per_task, strict=True) + } + if request.mode != "questions": + item_classifications[index] = classifications(request, per_task, grouped=request.mode == "groups") + token_counts[index] = document.billed_tokens + prefix.billed_tokens + + return ExtractOutput( + entities=[[] for _ in items], + classifications=None if request.mode == "questions" else item_classifications, + data=None if request.mode == "labels" else data, + errors=errors if any(error is not None for error in errors) else None, + input_token_counts=token_counts, + ) + + def _prefix(self, request: DecisionRequest) -> _Prefix: + """The task prompt as the processor builds it, once per request. + + Raises: + InputTooLongError: The task prompt takes more than ``max_prompt_tokens``. + """ + record = self._processor.transform_and_format(".", request.model_schema()) + # The processor keeps every string it tokenizes in an LRU of 50,000 + # entries; task prompts with descriptions run to tens of KB, so drop + # them rather than keep each request's prompt for the adapter's lifetime. + cache_clear = getattr(getattr(self._processor, "_tokenize_cached", None), "cache_clear", None) + if cache_clear is not None: + cache_clear() + text_start = record.text_word_first_positions[0] + ids = list(record.input_ids[:text_start]) + if len(ids) > self._max_prompt_tokens: + raise InputTooLongError( + f"GLiNER2.5-Decide questions and labels take {len(ids)} tokens; this model takes at most " + f"{self._max_prompt_tokens} of its {self._window}-token window. Send fewer or shorter questions, " + "labels, instructions, or descriptions." + ) + positions: list[list[int]] = record.schema_special_positions + if len(positions) != len(request.tasks) or any( + len(task_positions) != len(task.labels) + 1 + for task_positions, task in zip(positions, request.tasks, strict=True) + ): + raise RuntimeError("GLiNER2.5-Decide prompt markers do not match the request's labels") + billed = sum(len(self._tokenizer.tokenize(text)) for task in request.tasks for text in task.free_texts()) + return _Prefix( + ids=ids, + label_positions=[position for task_positions in positions for position in task_positions[1:]], + billed_tokens=billed, + ) + + def _document( + self, item: Item, room: int, overflow_policy: str + ) -> tuple[_Document | None, ExtractItemError | None]: + """The words of an item the model reads, or the item's error.""" + try: + text, from_end, whole = self._item_text(item, room) + except InvalidInputError as exc: + return None, ExtractItemError(code=ErrorCode.INVALID_INPUT.value, message=str(exc)) + document = self._read(text, room, from_end=from_end, whole=whole) + if not document.ids: + message = _ERR_NO_WORD_FITS.format(room=room, window=self._window, max_word=_MAX_WORD_CHARS) + return None, ExtractItemError(code=ErrorCode.INPUT_TOO_LONG.value, message=message) + if not document.complete and overflow_policy == "error": + message = _ERR_DOES_NOT_FIT.format(room=room, window=self._window) + return None, ExtractItemError(code=ErrorCode.INPUT_TOO_LONG.value, message=message) + return document, None + + @staticmethod + def _item_text(item: Item, room: int) -> tuple[str, bool, bool]: + """(text, read from the end, text is the whole document) for an item. + + A state is rendered only as far as a row can read: its first (for a list + of turns, its last) ``STATE_BYTES_PER_TOKEN`` UTF-8 bytes per window token. + + Raises: + InvalidInputError: The item has no usable text or state. + """ + metadata = item.metadata or {} + has_state = "state" in metadata + if has_state == (item.text is not None): + raise InvalidInputError(_ERR_ITEM) + if not has_state: + text = item.text or "" + from_end, whole = False, True + else: + state = metadata["state"] + if not isinstance(state, (str, Mapping, list)): + raise InvalidInputError(_ERR_ITEM) + from_end = isinstance(state, list) + limit = room * STATE_BYTES_PER_TOKEN + try: + text = serialize_state(state, limit, from_end=from_end) + except Exception as exc: # caller data that cannot be rendered fails only its item + raise InvalidInputError(_ERR_STATE_JSON) from exc + whole = len(text.encode("utf-8", "surrogatepass")) < limit + if not text or text.isspace(): + raise InvalidInputError(_ERR_ITEM) + return text, from_end, whole + + def _read(self, text: str, room: int, *, from_end: bool, whole: bool) -> _Document: + """The whole words of ``text`` whose tokens fit in ``room``, as the processor tokenizes them. + + The processor ends a text that does not end a sentence with ``"."`` and + reads it as one more word; that word is read when it fits but not billed. + Words come from a linear-time splitter, read lazily from the start; from + the end (conversation turns), run by run of non-space characters, each + split on its own (no word crosses a space), keeping only the last + ``_MAX_WORD_CHARS`` characters of a longer run and stopping after it. + """ + normalized = text if text.endswith(_SENTENCE_END) else text + "." + char_limit = room * _MAX_CHARS_PER_TOKEN + max_words = room * _MAX_WORDS_PER_TOKEN + words = self._words_from_end(normalized) if from_end else self._word_splitter(normalized, lower=True) + kept: list[tuple[tuple[int, ...], int]] = [] + used = 0 + complete = whole + for word, start, end in words: + if not word: # _words_from_end cut an overlong run here: older text is not read + complete = False + break + reach = len(normalized) - start if from_end else end + if end - start > _MAX_WORD_CHARS or reach > char_limit or len(kept) >= max_words: + complete = False + break + ids = self._word_ids(word) + if used + len(ids) > room: + # Missing only the processor's "." still reads the caller's whole text. + complete = complete and not from_end and start >= len(text) + break + kept.append((ids, start)) + used += len(ids) + if from_end: + kept.reverse() + return _Document( + ids=[token for ids, _ in kept for token in ids], + billed_tokens=sum(len(ids) for ids, start in kept if start < len(text)), + complete=complete, + ) + + def _words_from_end(self, text: str) -> Iterator[tuple[str, int, int]]: + """The words of ``text``, last first. + + A run longer than ``_MAX_WORD_CHARS`` yields only its tail's words, then + an empty word marking the cut (the splitter never yields an empty word). + """ + for run in reversed([match.span() for match in _RUN.finditer(text)]): + begin, end = run + cut = max(begin, end - _MAX_WORD_CHARS) + words = list(self._word_splitter(text[cut:end], lower=True)) + for word, start, stop in reversed(words): + yield word, cut + start, cut + stop + if cut > begin: + yield "", cut, cut + return + + def _score(self, rows: list[list[int]], label_positions: list[int]) -> np.ndarray: + """Every row's label logits, ``[rows, labels]``, in padded chunks of similar length.""" + model, device = self._model, self._device + if model is None or device is None: + raise RuntimeError(ERR_NOT_LOADED) + order = sorted(range(len(rows)), key=lambda row: len(rows[row]), reverse=True) + out = np.empty((len(rows), len(label_positions)), dtype=np.float32) + markers = torch.tensor(label_positions, dtype=torch.long, device=device) + start = 0 + while start < len(order): + longest = len(rows[order[start]]) + count = max(1, min(len(order) - start, self._inference_batch_tokens // longest)) + chunk = order[start : start + count] + input_ids = torch.zeros((count, longest), dtype=torch.long) + attention_mask = torch.zeros((count, longest), dtype=torch.long) + for j, row in enumerate(chunk): + input_ids[j, : len(rows[row])] = torch.tensor(rows[row], dtype=torch.long) + attention_mask[j, : len(rows[row])] = 1 + with torch.inference_mode(): + hidden = model.encoder( + input_ids=input_ids.to(device), attention_mask=attention_mask.to(device) + ).last_hidden_state + logits = model.classifier(hidden[:, markers, :]).squeeze(-1) + out[chunk] = logits.float().cpu().numpy() + start += count + return out + + # ------------------------------------------------------------------ cost + + def extract_item_costs( + self, + items: list[Item], + *, + labels: list[str] | None = None, + output_schema: dict[str, Any] | None = None, + instruction: str | None = None, + options: dict[str, Any] | None = None, + ) -> list[int] | None: + """Batching cost per item: the characters of its one row, the task prompt included. + + A document is read only as far as the window, so its cost is capped + there, and the task prompt is encoded again with every item. Runs before + batching and validation; best-effort, never raises. + """ + try: + limit = self._window * _COST_CHARS_PER_TOKEN + prompt = schema_chars(output_schema, labels, options or {}, limit) + prompt = min(limit, prompt + (len(instruction) if isinstance(instruction, str) else 0)) + return [_item_chars(item, limit) + prompt for item in items] + except Exception: # noqa: BLE001 -- a cost estimate must never fail a request + return None + + +def _item_chars(item: Item, limit: int) -> int: + state = (item.metadata or {}).get("state") + if state is not None and not isinstance(state, str): + return approx_json_chars(state, limit) + text = state if isinstance(state, str) else item.text + return min(len(text or ""), limit) diff --git a/packages/sie_server/src/sie_server/adapters/gliner2/decisions.py b/packages/sie_server/src/sie_server/adapters/gliner2/decisions.py new file mode 100644 index 000000000..b42bd30bb --- /dev/null +++ b/packages/sie_server/src/sie_server/adapters/gliner2/decisions.py @@ -0,0 +1,601 @@ +"""Typed decisions for GLiNER2.5-Decide: request parsing, the model's task schema, and answers. + +GLiNER2.5-Decide reads a *classification task* as a name, an optional prompt, +and a label set, where each label may carry a description. This module maps +the three request shapes the other SIE decision models accept onto such tasks, +and turns the model's per-label logits back into their answers: + +* ``output_schema``: Laya's typed question mapping ``{question_id: {"type", + "instructions", "criteria", "labels"?}}``, validated by Laya's own rules. The + question id is the task name and ``instructions`` its prompt. + + - ``choice``: one label per criterion; a criterion's description (dict form) + becomes the label's description. + - ``score``: levels ``"0"`` .. ``"k-1"`` (the ordinal labels the model reads), + each described by its criterion. + - ``noul``: the labels ``"yes"`` and ``"no"`` (or the question's ``labels`` + wording), described by ``criteria["true"]`` / ``criteria["false"]``. + +* ``options.label_groups``: GLiClass's ``{group: [label, ...]}``; the group name is + the task name and a request ``instruction`` the prompt of every group. + ``options.classification_type`` ("single-label" or "multi-label") applies to + every group. +* ``labels``: one task named ``options.classification_task`` (default + ``"label"``), with ``instruction`` as its prompt. + +Single-label tasks are a softmax over their labels and multi-label tasks an +independent sigmoid per label, as ``gliner2.classification`` computes them. +Nothing here imports torch or gliner2. +""" + +from __future__ import annotations + +import math +from collections.abc import Iterator, Mapping, Sequence +from dataclasses import dataclass +from typing import Any, Literal + +import numpy as np + +from sie_server.adapters.laya.questions import ( + Question, + confidence_from_probs, + parse_questions, + render_criterion, + resolve_noul_labels, + shown, +) +from sie_server.types.inputs import InvalidInputError +from sie_server.types.responses import Classification + +TaskKind = Literal["choice", "score", "noul", "group", "labels"] +RequestMode = Literal["questions", "groups", "labels"] + +# Tasks (questions or label groups) per request. Every task is encoded in each +# item's single row, so the row's token budget binds long before this does. +MAX_TASKS = 64 +# Labels (answer options) per task: gliner2.classification's default candidate +# cap per task (ClassificationConfig.max_candidates_per_task). +MAX_LABELS_PER_TASK = 64 +# Labels across all tasks of a request. +MAX_LABELS = 1024 +# Characters in a task name (question id, group name) and in a label. +MAX_NAME_CHARS = 128 +MAX_LABEL_CHARS = 256 +# Characters in one free-text field: a prompt (instructions) or a label description. +MAX_TEXT_CHARS = 2048 +# Characters of every string the model reads in its task schema together. All +# of them are checked against these bounds before anything is tokenized. +MAX_SCHEMA_CHARS = 65_536 + +DEFAULT_LABELS_TASK = "label" +NOUL_WORDS = ("no", "yes") # (false, true) labels the model reads for a noul question +DEFAULT_MULTI_LABEL_THRESHOLD = 0.5 +GROUP_LABEL_SEPARATOR = "." # "group.label" classifications, as the GLiClass adapter names them + +# Structural tokens of the GLiNER2 prompt. A task name, label, prompt, or +# description containing one would be read as structure, so they are refused. +# (The document cannot inject them: it is split into lowercased words first.) +MARKERS = ( + "[P]", + "[L]", + "[C]", + "[E]", + "[R]", + "[DESCRIPTION]", + "[EXAMPLE]", + "[OUTPUT]", + "[SEP_STRUCT]", + "[SEP_TEXT]", +) + +_ERR_NO_TASK = ( + "GLiNER2.5-Decide requires typed questions in output_schema ({question_id: {type, instructions, criteria}}), " + "options.label_groups, or labels" +) +_ERR_SCHEMA_AND_LABELS = "GLiNER2.5-Decide takes either output_schema (typed questions) or labels, not both" +_ERR_SCHEMA_AND_GROUPS = ( + "GLiNER2.5-Decide takes either output_schema (typed questions) or options.label_groups, not both" +) +_ERR_GROUPS_AND_LABELS = "GLiNER2.5-Decide takes either labels or options.label_groups, not both" +_ERR_SCHEMA_AND_INSTRUCTION = ( + "GLiNER2.5-Decide reads each question's own 'instructions' from output_schema; " + "the request-level instruction applies only to labels and label_groups" +) + + +@dataclass(frozen=True, slots=True) +class DecisionTask: + """One classification task the model reads, and how its answer is reported. + + Attributes: + key: The answer's key in ``data`` (question id or group name) or, in + labels mode, the task name. + name: Task name the model reads. + kind: ``choice``/``score``/``noul`` (output_schema), ``group`` + (label_groups), or ``labels``. + labels: Label names the model reads, in order. + keys: Output key of each label: the caller's label, or the level index + of a score question. + descriptions: Each label's description, or None. + prompt: The task prompt, or None. + multi_label: Independent sigmoid per label instead of a softmax. + legend: A score question's criteria, index 0 first. + true_index: Position of the "true" label of a noul question. + """ + + key: str + name: str + kind: TaskKind + labels: tuple[str, ...] + keys: tuple[str, ...] + descriptions: tuple[str | None, ...] + prompt: str | None = None + multi_label: bool = False + legend: tuple[Any, ...] = () + true_index: int = 0 + + def model_entry(self) -> dict[str, Any]: + """This task as a ``classifications`` entry of a gliner2 schema dict. + + The same entry ``gliner2.classification.compile_schema`` emits for a + task with these labels, descriptions, and instruction. + """ + entry: dict[str, Any] = { + "task": self.name, + "labels": list(self.labels), + "true_label": ["N/A"], # gliner2 2.0 reads it unconditionally + "multi_label": self.multi_label, + "cls_threshold": 0.5, + "class_act": "auto", + } + if self.prompt: + entry["prompt"] = self.prompt + descriptions = { + label: description for label, description in zip(self.labels, self.descriptions, strict=True) if description + } + if descriptions: + entry["label_descriptions"] = descriptions + return entry + + def free_texts(self) -> Iterator[str]: + """The caller's free text this task puts in front of every document (billed).""" + if self.prompt: + yield self.prompt + yield from (description for description in self.descriptions if description) + + +@dataclass(frozen=True, slots=True) +class DecisionRequest: + """A validated request: its tasks, in order, and how answers are reported.""" + + mode: RequestMode + tasks: tuple[DecisionTask, ...] + threshold: float = 0.0 + + def model_schema(self) -> dict[str, Any]: + """The gliner2 schema dict for every task (classifications only).""" + return { + "json_structures": [], + "classifications": [task.model_entry() for task in self.tasks], + "entities": {}, + "relations": [], + "json_descriptions": {}, + "entity_descriptions": {}, + } + + @property + def label_count(self) -> int: + return sum(len(task.labels) for task in self.tasks) + + +# --------------------------------------------------------------------------- +# Request parsing +# --------------------------------------------------------------------------- + + +def parse_request( + *, + labels: list[str] | None, + output_schema: dict[str, Any] | None, + instruction: str | None, + options: Mapping[str, Any], +) -> DecisionRequest: + """Validate a request and build its tasks. Every bound is checked before anything is tokenized. + + Raises: + InvalidInputError: For any malformed or out-of-bounds request. + """ + groups = options.get("label_groups") + _check_unsupported_options(options) + if output_schema is not None: + if labels: + raise InvalidInputError(_ERR_SCHEMA_AND_LABELS) + if groups is not None: + raise InvalidInputError(_ERR_SCHEMA_AND_GROUPS) + if instruction is not None: + raise InvalidInputError(_ERR_SCHEMA_AND_INSTRUCTION) + request = DecisionRequest(mode="questions", tasks=_question_tasks(output_schema)) + else: + prompt = _free_text(instruction, "instruction") + multi_label = multi_label_option(options) + threshold = threshold_option(options) + if groups is not None: + if labels: + raise InvalidInputError(_ERR_GROUPS_AND_LABELS) + tasks = _group_tasks(groups, prompt=prompt, multi_label=multi_label) + request = DecisionRequest(mode="groups", tasks=tasks, threshold=threshold) + elif labels: + task = _labels_task(labels, options.get("classification_task"), prompt=prompt, multi_label=multi_label) + request = DecisionRequest(mode="labels", tasks=(task,), threshold=threshold) + else: + raise InvalidInputError(_ERR_NO_TASK) + _check_totals(request) + return request + + +def _check_unsupported_options(options: Mapping[str, Any]) -> None: + encoding = options.get("group_encoding") + if encoding is not None and encoding != "joint": + raise InvalidInputError( + "GLiNER2.5-Decide scores every question and label group of an item in one encoder row " + "(group_encoding 'joint'); other encodings are not supported" + ) + if options.get("examples") is not None: + raise InvalidInputError("GLiNER2.5-Decide does not take few-shot examples") + + +def _question_tasks(output_schema: Any) -> tuple[DecisionTask, ...]: + if not isinstance(output_schema, Mapping): + raise InvalidInputError("GLiNER2.5-Decide output_schema must map question ids to question definitions") + if not output_schema: + raise InvalidInputError(_ERR_NO_TASK) + if len(output_schema) > MAX_TASKS: + raise InvalidInputError( + f"GLiNER2.5-Decide accepts at most {MAX_TASKS} questions per request, got {len(output_schema)}" + ) + for qid, qdef in output_schema.items(): + declared = _declared_options(qdef) + if declared > MAX_LABELS_PER_TASK: + raise InvalidInputError( + f"question {shown(qid)}: GLiNER2.5-Decide accepts at most {MAX_LABELS_PER_TASK} options " + f"per question, got {declared}" + ) + questions = parse_questions(output_schema) + tasks = tuple(_question_task(q) for q in questions) + _check_unique_names(tasks) + return tasks + + +def _declared_options(qdef: Any) -> int: + """Options a raw question definition declares, counted without walking its criteria.""" + if not isinstance(qdef, Mapping) or qdef.get("type") == "noul": + return 0 + criteria = qdef.get("criteria") + return len(criteria) if isinstance(criteria, (Mapping, list)) else 0 + + +def _question_task(q: Question) -> DecisionTask: + where = f"question {shown(q.qid)}" + name = _name(q.qid, f"{where}: the question id", MAX_NAME_CHARS) + prompt = _free_text(q.instructions, f"{where}: instructions") + if q.qtype == "choice": + criteria: dict[str, Any] = q.criteria + keys = tuple(criteria) + labels = _labels([str(key) for key in keys], where) + descriptions = tuple(_description(value, where) for value in criteria.values()) + return DecisionTask( + key=q.qid, name=name, kind="choice", labels=labels, keys=keys, descriptions=descriptions, prompt=prompt + ) + if q.qtype == "score": + levels: list[Any] = q.criteria + labels = tuple(str(i) for i in range(len(levels))) + descriptions = tuple( + None if _is_label_itself(level, label) else _description(level, where) + for label, level in zip(labels, levels, strict=True) + ) + return DecisionTask( + key=q.qid, + name=name, + kind="score", + labels=labels, + keys=labels, + descriptions=descriptions, + prompt=prompt, + legend=tuple(levels), + ) + false_word, true_word = resolve_noul_labels(q.labels) if q.labels is not None else NOUL_WORDS + labels = _labels([true_word, false_word], where) + criteria = q.criteria or {} + descriptions = (_description(criteria.get("true"), where), _description(criteria.get("false"), where)) + return DecisionTask( + key=q.qid, + name=name, + kind="noul", + labels=labels, + keys=("true", "false"), + descriptions=descriptions, + prompt=prompt, + true_index=0, + ) + + +def _is_label_itself(level: Any, label: str) -> bool: + """A score level that just repeats its own index ("0", "1", ...) needs no description.""" + return level in (None, "") or (isinstance(level, str) and level.strip() == label) + + +def _group_tasks(groups: Any, *, prompt: str | None, multi_label: bool) -> tuple[DecisionTask, ...]: + if not isinstance(groups, Mapping) or not groups: + raise InvalidInputError( + "GLiNER2.5-Decide label_groups must be a non-empty object mapping group names to label lists" + ) + if len(groups) > MAX_TASKS: + raise InvalidInputError( + f"GLiNER2.5-Decide accepts at most {MAX_TASKS} label groups per request, got {len(groups)}" + ) + tasks: list[DecisionTask] = [] + for group, group_labels in groups.items(): + where = f"label_groups[{shown(group)}]" + name = _name(group, f"{where}: the group name", MAX_NAME_CHARS) + if not isinstance(group_labels, list) or not group_labels: + raise InvalidInputError(f"GLiNER2.5-Decide {where} must be a non-empty list of labels") + if len(group_labels) > MAX_LABELS_PER_TASK: + raise InvalidInputError( + f"GLiNER2.5-Decide {where} may have at most {MAX_LABELS_PER_TASK} labels, got {len(group_labels)}" + ) + labels = _labels(group_labels, where) + tasks.append( + DecisionTask( + key=group, + name=name, + kind="group", + labels=labels, + keys=tuple(group_labels), + descriptions=(None,) * len(labels), + prompt=prompt, + multi_label=multi_label, + ) + ) + _check_unique_names(tasks) + return tuple(tasks) + + +def _labels_task(labels: Any, task_name: Any, *, prompt: str | None, multi_label: bool) -> DecisionTask: + if not isinstance(labels, list): + raise InvalidInputError("GLiNER2.5-Decide labels must be a list of non-empty strings") + if len(labels) > MAX_LABELS_PER_TASK: + raise InvalidInputError( + f"GLiNER2.5-Decide accepts at most {MAX_LABELS_PER_TASK} labels per task, got {len(labels)}" + ) + name = DEFAULT_LABELS_TASK if task_name is None else _name(task_name, "classification_task", MAX_NAME_CHARS) + return DecisionTask( + key=name, + name=name, + kind="labels", + labels=_labels(labels, "labels"), + keys=tuple(labels), + descriptions=(None,) * len(labels), + prompt=prompt, + multi_label=multi_label, + ) + + +def _labels(values: Sequence[Any], where: str) -> tuple[str, ...]: + labels = tuple(_name(value, f"{where}: a label", MAX_LABEL_CHARS) for value in values) + if len(set(labels)) != len(labels): + raise InvalidInputError( + f"GLiNER2.5-Decide {where}: labels must be unique once surrounding whitespace is removed" + ) + return labels + + +def _check_unique_names(tasks: Sequence[DecisionTask]) -> None: + names = [task.name for task in tasks] + if len(set(names)) != len(names): + raise InvalidInputError( + "GLiNER2.5-Decide question ids and label group names must be unique once surrounding whitespace is removed" + ) + + +def _name(value: Any, what: str, limit: int) -> str: + """A task name or label as the model reads it: stripped, non-empty, bounded, no markers.""" + if not isinstance(value, str) or not value.strip(): + raise InvalidInputError(f"GLiNER2.5-Decide {what} must be a non-empty string") + if len(value) > limit: + raise InvalidInputError(f"GLiNER2.5-Decide {what} may have at most {limit} characters, got {len(value)}") + _check_markers(value, what) + return value.strip() + + +def _free_text(value: Any, what: str) -> str | None: + """A prompt as the model reads it: None when empty, else bounded and without markers.""" + if value is None: + return None + if not isinstance(value, str): + raise InvalidInputError(f"GLiNER2.5-Decide {what} must be a string") + if len(value) > MAX_TEXT_CHARS: + raise InvalidInputError( + f"GLiNER2.5-Decide {what} may have at most {MAX_TEXT_CHARS} characters, got {len(value)}" + ) + _check_markers(value, what) + return value.strip() or None + + +def _description(value: Any, where: str) -> str | None: + """A criterion as a label description: strings as given, other values as compact JSON (as Laya renders them).""" + if value is None or value == "": + return None + what = f"{where}: a criterion" + try: + text = render_criterion(value, limit=4 * MAX_TEXT_CHARS + 4) + except (TypeError, ValueError) as exc: + raise InvalidInputError(f"GLiNER2.5-Decide {what} must be a string or JSON-serializable") from exc + return _free_text(text, what) + + +def _check_markers(value: str, what: str) -> None: + for marker in MARKERS: + if marker in value: + raise InvalidInputError( + f"GLiNER2.5-Decide {what} may not contain {marker!r}, a structural token of the model's prompt" + ) + + +def _check_totals(request: DecisionRequest) -> None: + if request.label_count > MAX_LABELS: + raise InvalidInputError( + f"GLiNER2.5-Decide accepts at most {MAX_LABELS} labels or answer options per request, " + f"got {request.label_count}" + ) + chars = 0 + for task in request.tasks: + chars += len(task.name) + sum(len(label) for label in task.labels) + chars += sum(len(text) for text in task.free_texts()) + if chars > MAX_SCHEMA_CHARS: + raise InvalidInputError( + f"GLiNER2.5-Decide questions, labels, prompts, and descriptions may total at most {MAX_SCHEMA_CHARS} " + f"characters, got {chars}" + ) + + +def multi_label_option(options: Mapping[str, Any]) -> bool: + """``classification_type`` (GLiClass spelling) or ``multi_label`` (GLiNER2 spelling); default single-label.""" + classification_type = options.get("classification_type") + multi_label = options.get("multi_label") + if multi_label is not None and not isinstance(multi_label, bool): + raise InvalidInputError("GLiNER2.5-Decide multi_label must be a boolean") + if classification_type is None: + return bool(multi_label) + if classification_type not in ("single-label", "multi-label"): + raise InvalidInputError("GLiNER2.5-Decide classification_type must be 'single-label' or 'multi-label'") + resolved = classification_type == "multi-label" + if multi_label is not None and multi_label != resolved: + raise InvalidInputError("GLiNER2.5-Decide multi_label contradicts classification_type") + return resolved + + +def threshold_option(options: Mapping[str, Any]) -> float: + value = options.get("threshold", 0.0) + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise InvalidInputError("GLiNER2.5-Decide threshold must be a finite number between 0 and 1") + try: + threshold = float(value) + except OverflowError: # an int too large for a float + threshold = math.inf + if not math.isfinite(threshold) or not 0.0 <= threshold <= 1.0: + raise InvalidInputError("GLiNER2.5-Decide threshold must be a finite number between 0 and 1") + return threshold + + +# --------------------------------------------------------------------------- +# Answers +# --------------------------------------------------------------------------- + + +def probabilities(task: DecisionTask, logits: np.ndarray) -> np.ndarray: + """Softmax over a single-label task's logits, or a sigmoid per label of a multi-label task (float64).""" + z = np.asarray(logits, dtype=np.float64) + if task.multi_label: + return np.exp(-np.logaddexp(0.0, -z)) + e = np.exp(z - z.max()) + return e / e.sum() + + +def answer(task: DecisionTask, p: np.ndarray, *, threshold: float = 0.0) -> dict[str, Any]: + """One task's answer in the shape the Laya adapter (questions) or GLiClass adapter (groups) returns. + + ``confidence`` is ``1 - H(p) / log(k)`` for choice, score, and single-label + groups, and ``max(p, 1 - p)`` for noul questions (Laya's definitions). + """ + k = len(task.labels) + probs = {key: float(value) for key, value in zip(task.keys, p, strict=True)} + if task.kind == "score": + return { + "type": "score", + "score": float(np.dot(np.arange(k, dtype=np.float64), p)), + "legend": {str(i): level for i, level in enumerate(task.legend)}, + "probabilities": probs, + "confidence": confidence_from_probs(p, k), + } + if task.kind == "noul": + p_true = float(p[task.true_index]) + p_false = float(p[1 - task.true_index]) + return {"type": "noul", "noul": p_true, "answer": p_true > p_false, "confidence": max(p_true, 1.0 - p_true)} + if task.multi_label: + selection = threshold if threshold > 0.0 else DEFAULT_MULTI_LABEL_THRESHOLD + return {"labels": [key for key, value in probs.items() if value >= selection], "probabilities": probs} + return { + "type": "choice", + "choice": task.keys[int(np.argmax(p))], + "probabilities": probs, + "confidence": confidence_from_probs(p, k), + } + + +def classifications(request: DecisionRequest, per_task: Sequence[np.ndarray], *, grouped: bool) -> list[Classification]: + """Every label with its probability, highest first, filtered by the request threshold. + + Label groups name their labels ``"group.label"``, as the GLiClass adapter does. + """ + result: list[Classification] = [] + for task, p in zip(request.tasks, per_task, strict=True): + for key, value in zip(task.keys, p, strict=True): + label = f"{task.key}{GROUP_LABEL_SEPARATOR}{key}" if grouped else key + if float(value) >= request.threshold: + result.append(Classification(label=label, score=float(value))) + result.sort(key=lambda c: c["score"], reverse=True) + return result + + +def split_logits(request: DecisionRequest, logits: np.ndarray) -> list[np.ndarray]: + """A row's label logits, one array per task (tasks' labels are consecutive in the row).""" + parts: list[np.ndarray] = [] + offset = 0 + for task in request.tasks: + parts.append(logits[offset : offset + len(task.labels)]) + offset += len(task.labels) + return parts + + +def schema_chars(output_schema: Any, labels: Any, options: Mapping[str, Any], limit: int) -> int: + """Approximate characters of a request's task schema, capped at ``limit``; never raises (batching cost).""" + try: + if output_schema is not None: + return approx_json_chars(output_schema, limit) + total = approx_json_chars(options.get("label_groups"), limit) if options.get("label_groups") else 0 + if isinstance(labels, list): + total += approx_json_chars(labels, limit) + return min(total, limit) + except Exception: # noqa: BLE001 -- a cost estimate must never fail a request + return 0 + + +def approx_json_chars(value: Any, limit: int) -> int: + """Approximate ``len(json.dumps(value))``, visiting at most ``limit`` characters' worth of values.""" + total = 0 + stack: list[Iterator[Any]] = [iter((value,))] + while stack and total < limit: + node = next(stack[-1], _END) + if node is _END: + stack.pop() + elif isinstance(node, str): + total += len(node) + 4 + elif isinstance(node, Mapping): + total += 2 + stack.append(_flatten_items(node)) + elif isinstance(node, (list, tuple)): + total += 2 + stack.append(iter(node)) + else: + total += 6 # a number, boolean, or null + return min(total, limit) + + +def _flatten_items(mapping: Mapping[Any, Any]) -> Iterator[Any]: + for key, value in mapping.items(): + yield key + yield value + + +_END = object() diff --git a/packages/sie_server/src/sie_server/adapters/gliner2/words.py b/packages/sie_server/src/sie_server/adapters/gliner2/words.py new file mode 100644 index 000000000..33dddaaba --- /dev/null +++ b/packages/sie_server/src/sie_server/adapters/gliner2/words.py @@ -0,0 +1,136 @@ +r"""A linear-time equivalent of gliner2's whitespace word splitter. + +gliner2 splits a document into words with one regular expression +(``WhitespaceTokenSplitter``), matched case-insensitively:: + + https?://[^\s]+ | www\.[^\s]+ a URL + [a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,} an e-mail address + @[a-z0-9_]+ a mention + \w+(?:[-_]\w+)* a word + \S any other character + +The e-mail alternative is tried at every word. Its local part runs over every +e-mail character ahead of the word before failing on the missing ``@``, so a run +of n such characters (``"...."``, ``"a.a.a."``) costs O(n^2): splitting 64 KiB +of ``"."`` takes about 17 seconds, on the thread that serves every request. + +``word_spans`` yields the same matches in linear time. It tries the alternatives +in the same order and decides the e-mail alternative without rescanning: a local +part can only end at the first non-e-mail character after the word start, which +is the end of the maximal run of e-mail characters holding it, so each such run +is scanned once, and the domain after an ``@`` is matched once. +""" + +from __future__ import annotations + +import re +from collections.abc import Callable, Iterator +from typing import Any + +# gliner2's pattern (WhitespaceTokenSplitter._PATTERN), with its flags. +PACKAGE_PATTERN = re.compile( + r"""(?:https?://[^\s]+|www\.[^\s]+) + |[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,} + |@[a-z0-9_]+ + |\w+(?:[-_]\w+)* + |\S""", + re.VERBOSE | re.IGNORECASE, +) + +_START = re.compile(r"\S") +_URL = re.compile(r"https?://[^\s]+|www\.[^\s]+", re.IGNORECASE) +_LOCAL = re.compile(r"[a-z0-9._%+-]+", re.IGNORECASE) +_DOMAIN = re.compile(r"[a-z0-9.-]+\.[a-z]{2,}", re.IGNORECASE) +_OTHER = re.compile(r"@[a-z0-9_]+|\w+(?:[-_]\w+)*|\S", re.IGNORECASE) + +WordSplitter = Callable[..., Iterator[tuple[str, int, int]]] + + +def word_spans(text: str) -> Iterator[tuple[int, int]]: + """``(start, end)`` of each match of ``PACKAGE_PATTERN.finditer(text)``, in linear time.""" + size = len(text) + position = 0 + local_end = 0 # end of the run of e-mail characters holding the current word start + email_end: int | None = None # where an e-mail address starting in that run ends, if one does + while (start := _START.search(text, position)) is not None: + begin = start.start() + if (url := _URL.match(text, begin)) is not None: + end = url.end() + else: + if begin >= local_end: + local = _LOCAL.match(text, begin) + local_end = begin if local is None else local.end() + email_end = None + if local_end > begin and local_end < size and text[local_end] == "@": + domain = _DOMAIN.match(text, local_end + 1) + email_end = None if domain is None else domain.end() + if begin < local_end and email_end is not None: + end = email_end + else: + other = _OTHER.match(text, begin) + end = other.end() if other is not None else begin + 1 + yield begin, end + position = end + + +class LinearWordSplitter: + """Drop-in for gliner2's ``WhitespaceTokenSplitter``: the same words, in linear time. + + gliner2 1.x lowercases the text and then splits it; gliner2 2.x splits the + text as given and lowercases each word (offsets then always index the text). + ``lower_text_first`` selects the 1.x behavior. + """ + + __slots__ = ("lower_text_first",) + + def __init__(self, *, lower_text_first: bool) -> None: + self.lower_text_first = lower_text_first + + def __call__(self, text: str, lower: bool = True) -> Iterator[tuple[str, int, int]]: + if lower and self.lower_text_first: + text = text.lower() + for begin, end in word_spans(text): + yield text[begin:end], begin, end + return + for begin, end in word_spans(text): + word = text[begin:end] + yield (word.lower() if lower else word), begin, end + + +# Texts on which the two lowercasing behaviors (and a different pattern) disagree. +_PROBES = ( + "\u0130stanbul and \u03a3\u038a\u03a3\u03a5\u03a6\u039f\u03a3 met at https://Example.com/x?y=1, " + "mail A.B@Example.CO.uk @Team_1 x-y_z 3.14 ...@ a@b", + "Stra\u00dfe \u01c5emal \ufb01ne KELVIN\u212a na\u00efve", +) + + +def linear_equivalent(splitter: Any) -> LinearWordSplitter | None: + """A ``LinearWordSplitter`` that yields exactly what gliner2's ``splitter`` yields, or None. + + None unless ``splitter`` is gliner2's ``WhitespaceTokenSplitter`` with the + pattern above, and one of the two lowercasing behaviors reproduces it on + probe texts. + """ + pattern = getattr(type(splitter), "_PATTERN", None) + if ( + type(splitter).__name__ != "WhitespaceTokenSplitter" + or not isinstance(pattern, re.Pattern) + or _normalized(pattern.pattern) != _normalized(PACKAGE_PATTERN.pattern) + or pattern.flags != PACKAGE_PATTERN.flags + ): + return None + for lower_text_first in (False, True): + candidate = LinearWordSplitter(lower_text_first=lower_text_first) + if all( + list(candidate(probe, lower)) == list(splitter(probe, lower)) + for probe in _PROBES + for lower in (True, False) + ): + return candidate + return None + + +def _normalized(pattern: str) -> str: + """A verbose pattern without its layout whitespace (this pattern has no escaped spaces).""" + return "".join(pattern.split()) diff --git a/packages/sie_server/src/sie_server/api/options.py b/packages/sie_server/src/sie_server/api/options.py index 153ffff33..2c46edf32 100644 --- a/packages/sie_server/src/sie_server/api/options.py +++ b/packages/sie_server/src/sie_server/api/options.py @@ -3,8 +3,7 @@ from fastapi import HTTPException, status from sie_server.config.model import ModelConfig, ResolvedProfile -from sie_server.core.runtime_options import merge_runtime_options_with_profile -from sie_server.types.overflow_policy import VALID_OVERFLOW_POLICIES +from sie_server.core.runtime_options import InvalidOverflowPolicyError, merge_runtime_options_with_profile from sie_server.types.responses import ErrorCode @@ -43,7 +42,9 @@ def resolve_runtime_options_with_profile( try: merged, resolved_profile = merge_runtime_options_with_profile(config, request_options) except ValueError as e: - span.set_attribute("error", "invalid_profile") + # The same check the queue worker applies (core.runtime_options). + invalid = "invalid_overflow_policy" if isinstance(e, InvalidOverflowPolicyError) else "invalid_profile" + span.set_attribute("error", invalid) raise HTTPException( status_code=status.HTTP_400_BAD_REQUEST, detail={ @@ -52,19 +53,4 @@ def resolve_runtime_options_with_profile( }, ) from e - overflow_policy = merged.get("overflow_policy") - if overflow_policy is not None and ( - not isinstance(overflow_policy, str) or overflow_policy not in VALID_OVERFLOW_POLICIES - ): - span.set_attribute("error", "invalid_overflow_policy") - raise HTTPException( - status_code=status.HTTP_400_BAD_REQUEST, - detail={ - "code": ErrorCode.INVALID_INPUT.value, - "message": ( - f"Invalid overflow_policy: {overflow_policy!r}. Must be one of {sorted(VALID_OVERFLOW_POLICIES)}." - ), - }, - ) - return merged, resolved_profile diff --git a/packages/sie_server/src/sie_server/core/runtime_options.py b/packages/sie_server/src/sie_server/core/runtime_options.py index 8493c5b42..5b1368c03 100644 --- a/packages/sie_server/src/sie_server/core/runtime_options.py +++ b/packages/sie_server/src/sie_server/core/runtime_options.py @@ -25,6 +25,7 @@ from typing import TYPE_CHECKING, Any from sie_server.types.inputs import InvalidInputError +from sie_server.types.overflow_policy import VALID_OVERFLOW_POLICIES if TYPE_CHECKING: from sie_server.config.model import ModelConfig, ResolvedProfile @@ -44,15 +45,40 @@ def _resolve_profile_or_raise( raise InvalidInputError(str(exc)) from exc +class InvalidOverflowPolicyError(InvalidInputError): + """``options.overflow_policy`` is not one of ``VALID_OVERFLOW_POLICIES``.""" + + +def check_overflow_policy(options: dict[str, Any]) -> None: + """Reject an ``overflow_policy`` that is not one of the valid policy names. + + Raises: + InvalidOverflowPolicyError: The value is not a string naming a valid policy. + """ + overflow_policy = options.get("overflow_policy") + if overflow_policy is not None and ( + not isinstance(overflow_policy, str) or overflow_policy not in VALID_OVERFLOW_POLICIES + ): + raise InvalidOverflowPolicyError( + f"Invalid overflow_policy: {overflow_policy!r}. Must be one of {sorted(VALID_OVERFLOW_POLICIES)}." + ) + + def merge_runtime_options_with_profile( config: ModelConfig, request_options: dict[str, Any] | None, ) -> tuple[dict[str, Any], ResolvedProfile]: - """Return merged adapter options and the profile used to derive them.""" + """Return merged adapter options and the profile used to derive them. + + Raises: + InvalidInputError: The request selects an unknown or malformed profile, or + the merged ``overflow_policy`` is not a valid policy name. + """ resolved = _resolve_profile_or_raise(config, request_options) merged: dict[str, Any] = dict(resolved.runtime) if request_options: merged |= {key: value for key, value in request_options.items() if key != "profile"} + check_overflow_policy(merged) return merged, resolved @@ -76,7 +102,7 @@ def merge_runtime_options( Raises: InvalidInputError: If ``request_options`` selects a malformed or - unknown profile. + unknown profile, or the merged ``overflow_policy`` is invalid. """ merged, _ = merge_runtime_options_with_profile(config, request_options) return merged diff --git a/packages/sie_server/tests/adapters/goldens/gliner2_decide/fastino__GLiNER2.5-Decide-1B.json b/packages/sie_server/tests/adapters/goldens/gliner2_decide/fastino__GLiNER2.5-Decide-1B.json new file mode 100644 index 000000000..28f64363f --- /dev/null +++ b/packages/sie_server/tests/adapters/goldens/gliner2_decide/fastino__GLiNER2.5-Decide-1B.json @@ -0,0 +1,978 @@ +{ + "model": "fastino/GLiNER2.5-Decide-1B", + "revision": "52c94d3b698bf6d2619df9d898bdc1523ea3f1ca", + "generated_with": { + "gliner2": "2.0.0", + "transformers": "5.17.0", + "torch": "2.9.1+cu128", + "device": "cpu", + "dtype": "float32", + "window": 2048 + }, + "texts": [ + "My subscription renewed on April 15 for 5,400 yen after the service was already down. Can I get that charge refunded?", + "Guest in room 1408 says the AC has been out since yesterday and they want to move tonight or leave. They also asked for the incidentals hold to be released.", + "Das Paket kam beschädigt an, bitte schicken Sie Ersatz. Ich brauche es bis Freitag, sonst storniere ich die Bestellung!", + "请帮我取消订单，我不想要了。退款什么时候到账？", + "The treaty was signed in Paris in 1992. It entered into force the following year, after the last signatory ratified it." + ], + "long_text": "Quarterly operations report for the northern warehouse, week 0. The team shipped 12000 orders and handled 300 returns; late deliveries fell to 9 percent. Carrier contract 0 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 1. The team shipped 12037 orders and handled 301 returns; late deliveries fell to 8 percent. Carrier contract 1 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 2. The team shipped 12074 orders and handled 302 returns; late deliveries fell to 7 percent. Carrier contract 2 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 3. The team shipped 12111 orders and handled 303 returns; late deliveries fell to 6 percent. Carrier contract 3 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 4. The team shipped 12148 orders and handled 304 returns; late deliveries fell to 5 percent. Carrier contract 4 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 5. The team shipped 12185 orders and handled 305 returns; late deliveries fell to 4 percent. Carrier contract 5 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 6. The team shipped 12222 orders and handled 306 returns; late deliveries fell to 3 percent. Carrier contract 6 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 7. The team shipped 12259 orders and handled 307 returns; late deliveries fell to 2 percent. Carrier contract 7 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 8. The team shipped 12296 orders and handled 308 returns; late deliveries fell to 9 percent. Carrier contract 8 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 9. The team shipped 12333 orders and handled 309 returns; late deliveries fell to 8 percent. Carrier contract 9 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 10. The team shipped 12370 orders and handled 310 returns; late deliveries fell to 7 percent. Carrier contract 10 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 11. The team shipped 12407 orders and handled 311 returns; late deliveries fell to 6 percent. Carrier contract 11 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 12. The team shipped 12444 orders and handled 312 returns; late deliveries fell to 5 percent. Carrier contract 12 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 13. The team shipped 12481 orders and handled 313 returns; late deliveries fell to 4 percent. Carrier contract 13 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 14. The team shipped 12518 orders and handled 314 returns; late deliveries fell to 3 percent. Carrier contract 14 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 15. The team shipped 12555 orders and handled 315 returns; late deliveries fell to 2 percent. Carrier contract 15 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 16. The team shipped 12592 orders and handled 316 returns; late deliveries fell to 9 percent. Carrier contract 16 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 17. The team shipped 12629 orders and handled 317 returns; late deliveries fell to 8 percent. Carrier contract 17 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 18. The team shipped 12666 orders and handled 318 returns; late deliveries fell to 7 percent. Carrier contract 18 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 19. The team shipped 12703 orders and handled 319 returns; late deliveries fell to 6 percent. Carrier contract 19 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 20. The team shipped 12740 orders and handled 320 returns; late deliveries fell to 5 percent. Carrier contract 20 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 21. The team shipped 12777 orders and handled 321 returns; late deliveries fell to 4 percent. Carrier contract 21 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 22. The team shipped 12814 orders and handled 322 returns; late deliveries fell to 3 percent. Carrier contract 22 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 23. The team shipped 12851 orders and handled 323 returns; late deliveries fell to 2 percent. Carrier contract 23 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 24. The team shipped 12888 orders and handled 324 returns; late deliveries fell to 9 percent. Carrier contract 24 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 25. The team shipped 12925 orders and handled 325 returns; late deliveries fell to 8 percent. Carrier contract 25 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 26. The team shipped 12962 orders and handled 326 returns; late deliveries fell to 7 percent. Carrier contract 26 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 27. The team shipped 12999 orders and handled 327 returns; late deliveries fell to 6 percent. Carrier contract 27 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 28. The team shipped 13036 orders and handled 328 returns; late deliveries fell to 5 percent. Carrier contract 28 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 29. The team shipped 13073 orders and handled 329 returns; late deliveries fell to 4 percent. Carrier contract 29 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 30. The team shipped 13110 orders and handled 330 returns; late deliveries fell to 3 percent. Carrier contract 30 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 31. The team shipped 13147 orders and handled 331 returns; late deliveries fell to 2 percent. Carrier contract 31 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 32. The team shipped 13184 orders and handled 332 returns; late deliveries fell to 9 percent. Carrier contract 32 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 33. The team shipped 13221 orders and handled 333 returns; late deliveries fell to 8 percent. Carrier contract 33 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 34. The team shipped 13258 orders and handled 334 returns; late deliveries fell to 7 percent. Carrier contract 34 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 35. The team shipped 13295 orders and handled 335 returns; late deliveries fell to 6 percent. Carrier contract 35 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 36. The team shipped 13332 orders and handled 336 returns; late deliveries fell to 5 percent. Carrier contract 36 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 37. The team shipped 13369 orders and handled 337 returns; late deliveries fell to 4 percent. Carrier contract 37 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 38. The team shipped 13406 orders and handled 338 returns; late deliveries fell to 3 percent. Carrier contract 38 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 39. The team shipped 13443 orders and handled 339 returns; late deliveries fell to 2 percent. Carrier contract 39 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 40. The team shipped 13480 orders and handled 340 returns; late deliveries fell to 9 percent. Carrier contract 40 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 41. The team shipped 13517 orders and handled 341 returns; late deliveries fell to 8 percent. Carrier contract 41 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 42. The team shipped 13554 orders and handled 342 returns; late deliveries fell to 7 percent. Carrier contract 42 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 43. The team shipped 13591 orders and handled 343 returns; late deliveries fell to 6 percent. Carrier contract 43 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 44. The team shipped 13628 orders and handled 344 returns; late deliveries fell to 5 percent. Carrier contract 44 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 45. The team shipped 13665 orders and handled 345 returns; late deliveries fell to 4 percent. Carrier contract 45 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 46. The team shipped 13702 orders and handled 346 returns; late deliveries fell to 3 percent. Carrier contract 46 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 47. The team shipped 13739 orders and handled 347 returns; late deliveries fell to 2 percent. Carrier contract 47 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 48. The team shipped 13776 orders and handled 348 returns; late deliveries fell to 9 percent. Carrier contract 48 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 49. The team shipped 13813 orders and handled 349 returns; late deliveries fell to 8 percent. Carrier contract 49 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 50. The team shipped 13850 orders and handled 350 returns; late deliveries fell to 7 percent. Carrier contract 50 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 51. The team shipped 13887 orders and handled 351 returns; late deliveries fell to 6 percent. Carrier contract 51 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 52. The team shipped 13924 orders and handled 352 returns; late deliveries fell to 5 percent. Carrier contract 52 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 53. The team shipped 13961 orders and handled 353 returns; late deliveries fell to 4 percent. Carrier contract 53 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 54. The team shipped 13998 orders and handled 354 returns; late deliveries fell to 3 percent. Carrier contract 54 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 55. The team shipped 14035 orders and handled 355 returns; late deliveries fell to 2 percent. Carrier contract 55 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 56. The team shipped 14072 orders and handled 356 returns; late deliveries fell to 9 percent. Carrier contract 56 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 57. The team shipped 14109 orders and handled 357 returns; late deliveries fell to 8 percent. Carrier contract 57 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 58. The team shipped 14146 orders and handled 358 returns; late deliveries fell to 7 percent. Carrier contract 58 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 59. The team shipped 14183 orders and handled 359 returns; late deliveries fell to 6 percent. Carrier contract 59 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. In the last week the payroll system failed during the month-end close and salaries for 40 staff are delayed; finance needs a fix before the Friday 5pm cutoff", + "cases": [ + { + "name": "questions", + "request": { + "output_schema": { + "intent": { + "type": "choice", + "instructions": "What does the customer want?", + "criteria": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": "" + } + }, + "urgency": { + "type": "score", + "instructions": "How urgent is this request?", + "criteria": [ + "not urgent", + "within a week", + "today", + "blocking or past a deadline" + ] + }, + "needs_human": { + "type": "noul", + "instructions": "Must a person act on this?", + "criteria": { + "true": "automation cannot resolve it" + } + }, + "rating": { + "type": "score", + "instructions": "Rate the severity from 0 to 10.", + "criteria": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ] + } + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "intent", + "kind": "single", + "labels": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": null + }, + "instruction": "What does the customer want?" + }, + { + "name": "urgency", + "kind": "single", + "labels": { + "0": "not urgent", + "1": "within a week", + "2": "today", + "3": "blocking or past a deadline" + }, + "instruction": "How urgent is this request?" + }, + { + "name": "needs_human", + "kind": "single", + "labels": { + "yes": "automation cannot resolve it", + "no": null + }, + "instruction": "Must a person act on this?" + }, + { + "name": "rating", + "kind": "single", + "labels": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ], + "instruction": "Rate the severity from 0 to 10." + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 217, + "row_sha256": "b17a5400de16f4d7fb7c7920523d7468d1af2bc81ce86f5ab33b438e1ff11f53", + "probabilities": { + "intent": { + "refund_request": 0.8010653731600689, + "cancel_order": 0.07469228758777705, + "room_change": 0.02259114600468769, + "maintenance": 0.0384947019627022, + "other": 0.06315649128476412 + }, + "urgency": { + "0": 0.30362083205108553, + "1": 0.24988522048889183, + "2": 0.23198067694069355, + "3": 0.21451327051932906 + }, + "needs_human": { + "yes": 0.36402350501851766, + "no": 0.6359764949814823 + }, + "rating": { + "0": 0.10897192912277597, + "1": 0.10461417072119812, + "2": 0.1037650692067597, + "3": 0.10495203021421258, + "4": 0.10257595253970324, + "5": 0.09701108589604848, + "6": 0.08780869884799078, + "7": 0.08086493361627362, + "8": 0.07660293694553529, + "9": 0.0717258662969906, + "10": 0.06110732659251161 + } + } + }, + { + "max_len": null, + "row_length": 227, + "row_sha256": "3f8edc7dda65efa089e62449ccdf083ff7fe4c30c5482cff086169d34f57cf7d", + "probabilities": { + "intent": { + "refund_request": 0.012228097326302229, + "cancel_order": 0.02106592153836021, + "room_change": 0.15379027148069782, + "maintenance": 0.6979721870742723, + "other": 0.11494352258036743 + }, + "urgency": { + "0": 0.26969235028636906, + "1": 0.23007562736187961, + "2": 0.2600929385958381, + "3": 0.24013908375591322 + }, + "needs_human": { + "yes": 0.5389770539262533, + "no": 0.4610229460737467 + }, + "rating": { + "0": 0.09082264439262412, + "1": 0.0923239194838256, + "2": 0.09898734901776567, + "3": 0.10238851380255051, + "4": 0.09942257440260478, + "5": 0.10048439192132845, + "6": 0.09325194504202873, + "7": 0.08882068876010286, + "8": 0.08523787760552841, + "9": 0.08003759716641319, + "10": 0.06822249840522766 + } + } + }, + { + "max_len": null, + "row_length": 228, + "row_sha256": "e5e925d46f25fb51d733f9f5e38f3968bec597fdc35cc15d5c5bd5ca203c662d", + "probabilities": { + "intent": { + "refund_request": 0.14015047701593003, + "cancel_order": 0.6780489083838533, + "room_change": 0.03713635650933235, + "maintenance": 0.06534549729887337, + "other": 0.07931876079201086 + }, + "urgency": { + "0": 0.14132577996462378, + "1": 0.22753703254690405, + "2": 0.29805223603427383, + "3": 0.33308495145419825 + }, + "needs_human": { + "yes": 0.7581265392865992, + "no": 0.2418734607134008 + }, + "rating": { + "0": 0.07239392517798311, + "1": 0.08776138085211085, + "2": 0.09368789418947525, + "3": 0.09364166814265595, + "4": 0.09373986430836663, + "5": 0.09235294682016999, + "6": 0.09262920895145006, + "7": 0.0935149873718151, + "8": 0.09773562294575652, + "9": 0.09536281560904186, + "10": 0.08717968563117459 + } + } + }, + { + "max_len": null, + "row_length": 218, + "row_sha256": "dc294d571322457188948e05ded2abde18b42c8d7bb93785a6b5ca7f0f83c017", + "probabilities": { + "intent": { + "refund_request": 0.13580192632202132, + "cancel_order": 0.5750195487896889, + "room_change": 0.03935414548190969, + "maintenance": 0.043760013830487714, + "other": 0.20606436557589242 + }, + "urgency": { + "0": 0.3401854788936401, + "1": 0.23525413003329482, + "2": 0.2131711812059737, + "3": 0.2113892098670914 + }, + "needs_human": { + "yes": 0.5125864842165553, + "no": 0.48741351578344466 + }, + "rating": { + "0": 0.12132999538089477, + "1": 0.11013249894539567, + "2": 0.10170448932016747, + "3": 0.10119625939504467, + "4": 0.09721070820495314, + "5": 0.09259744984401207, + "6": 0.08293663913016602, + "7": 0.0779146600340303, + "8": 0.07390400184161015, + "9": 0.07377867083384593, + "10": 0.06729462706987975 + } + } + }, + { + "max_len": null, + "row_length": 214, + "row_sha256": "df4e7e1ff032176642ef7ca82b5628428870c94493b03ededf718e2746de5734", + "probabilities": { + "intent": { + "refund_request": 0.16104482800501035, + "cancel_order": 0.15017369931278604, + "room_change": 0.11238415451497373, + "maintenance": 0.13847991483519248, + "other": 0.4379174033320373 + }, + "urgency": { + "0": 0.34584683267541283, + "1": 0.23007415586417315, + "2": 0.21340419637312605, + "3": 0.21067481508728803 + }, + "needs_human": { + "yes": 0.4587280509047346, + "no": 0.5412719490952653 + }, + "rating": { + "0": 0.11501374315993007, + "1": 0.09303563505400793, + "2": 0.08561614959330362, + "3": 0.08766950909319454, + "4": 0.08856789396846435, + "5": 0.08874254937249208, + "6": 0.08365277568302953, + "7": 0.08374964172366488, + "8": 0.08520395559662372, + "9": 0.08796656938081664, + "10": 0.10078157737447264 + } + } + } + ] + }, + { + "name": "question_over_passage", + "request": { + "output_schema": { + "answer": { + "type": "noul", + "instructions": "Did the treaty enter into force in 1992?" + } + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "answer", + "kind": "single", + "labels": [ + "yes", + "no" + ], + "instruction": "Did the treaty enter into force in 1992?" + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 52, + "row_sha256": "3c035442f6c2b7615f6d1dc520faf2bbca97e1c9bdc5f8e46c0cc72b5392229c", + "probabilities": { + "answer": { + "yes": 0.30084693186843525, + "no": 0.6991530681315649 + } + } + }, + { + "max_len": null, + "row_length": 62, + "row_sha256": "ae85d4dc529b284a7ba4c0c7b17de03a7390b67bac64a643e326144232d5b509", + "probabilities": { + "answer": { + "yes": 0.34437007280609494, + "no": 0.6556299271939051 + } + } + }, + { + "max_len": null, + "row_length": 63, + "row_sha256": "e16358c0722cbc0719a90306c959bccda5bb01be51be0a0a415d86e3e10c4b2e", + "probabilities": { + "answer": { + "yes": 0.27834142035030984, + "no": 0.7216585796496902 + } + } + }, + { + "max_len": null, + "row_length": 53, + "row_sha256": "955bd5a7e3b49ba8970b608cdebe5c3121ff417a60b15c4571ec39faaae4f89d", + "probabilities": { + "answer": { + "yes": 0.21836896557019128, + "no": 0.7816310344298087 + } + } + }, + { + "max_len": null, + "row_length": 49, + "row_sha256": "bb2c6f3ecf906bd5dce25538f1592317a6173eecbae3fc5965850ce13cc91ab6", + "probabilities": { + "answer": { + "yes": 0.993166662003856, + "no": 0.006833337996144008 + } + } + } + ] + }, + { + "name": "label_groups", + "request": { + "instruction": "Triage the message.", + "options": { + "label_groups": { + "intent": [ + "refund_request", + "cancel_order", + "room_change", + "maintenance", + "other" + ], + "priority": [ + "low", + "normal", + "high", + "urgent" + ], + "needs_human": [ + "yes", + "no" + ] + } + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "intent", + "kind": "single", + "labels": [ + "refund_request", + "cancel_order", + "room_change", + "maintenance", + "other" + ], + "instruction": "Triage the message." + }, + { + "name": "priority", + "kind": "single", + "labels": [ + "low", + "normal", + "high", + "urgent" + ], + "instruction": "Triage the message." + }, + { + "name": "needs_human", + "kind": "single", + "labels": [ + "yes", + "no" + ], + "instruction": "Triage the message." + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 103, + "row_sha256": "d44ace352326e622b3fedb21d1432dcdfa1b4b486a221842b6a525e9573e1946", + "probabilities": { + "intent": { + "refund_request": 0.9566478273343769, + "cancel_order": 0.008551281448888169, + "room_change": 0.004837760738954607, + "maintenance": 0.010275450770498277, + "other": 0.019687679707282038 + }, + "priority": { + "low": 0.32485284134321674, + "normal": 0.32269457229749243, + "high": 0.18291611301326566, + "urgent": 0.1695364733460252 + }, + "needs_human": { + "yes": 0.29937443349108434, + "no": 0.7006255665089157 + } + } + }, + { + "max_len": null, + "row_length": 113, + "row_sha256": "c2e3181a40416fad35854de848da1d1c04a1795d1ec9564df45023b8f94f7c34", + "probabilities": { + "intent": { + "refund_request": 0.008916215520265338, + "cancel_order": 0.03202051126313205, + "room_change": 0.46126995167513873, + "maintenance": 0.3767196461411649, + "other": 0.12107367540029898 + }, + "priority": { + "low": 0.297756031595477, + "normal": 0.29993290766270925, + "high": 0.20246181313640663, + "urgent": 0.19984924760540715 + }, + "needs_human": { + "yes": 0.4116039844314372, + "no": 0.5883960155685627 + } + } + }, + { + "max_len": null, + "row_length": 114, + "row_sha256": "6584a94b0e9d97eb7375044672ccdc4c0a6eec040bc59b6ab74006a457d81b08", + "probabilities": { + "intent": { + "refund_request": 0.11605921700661267, + "cancel_order": 0.7145490813574593, + "room_change": 0.04011157895039521, + "maintenance": 0.06378187449027146, + "other": 0.06549824819526133 + }, + "priority": { + "low": 0.14773416106667, + "normal": 0.13971591056003527, + "high": 0.31275855295961735, + "urgent": 0.39979137541367743 + }, + "needs_human": { + "yes": 0.7926373003908037, + "no": 0.20736269960919626 + } + } + }, + { + "max_len": null, + "row_length": 104, + "row_sha256": "72edb218e7e31dbe7b5017f505a9f809bacbbd900b523e598126e0e8d5a1f6d0", + "probabilities": { + "intent": { + "refund_request": 0.011141326007375817, + "cancel_order": 0.9497464089753589, + "room_change": 0.0034789021705911856, + "maintenance": 0.004324219957867206, + "other": 0.03130914288880675 + }, + "priority": { + "low": 0.3358787938651861, + "normal": 0.2777234213999378, + "high": 0.18724882373848342, + "urgent": 0.19914896099639276 + }, + "needs_human": { + "yes": 0.4494587464645528, + "no": 0.5505412535354473 + } + } + }, + { + "max_len": null, + "row_length": 100, + "row_sha256": "f58e38024ec3487434331a336dcbcb8b2e04f3a3f8ed4a4129a0e75d1e62d838", + "probabilities": { + "intent": { + "refund_request": 0.12728508318008155, + "cancel_order": 0.12248810071493149, + "room_change": 0.15867497160283547, + "maintenance": 0.15840487988668453, + "other": 0.4331469646154669 + }, + "priority": { + "low": 0.2676208393858538, + "normal": 0.31571265142830907, + "high": 0.21615857634614322, + "urgent": 0.20050793283969387 + }, + "needs_human": { + "yes": 0.31056317287729596, + "no": 0.6894368271227042 + } + } + } + ] + }, + { + "name": "label_groups_multi_label", + "request": { + "options": { + "label_groups": { + "topics": [ + "billing", + "hvac", + "shipping", + "payroll", + "account" + ], + "channel": [ + "email", + "chat", + "phone" + ] + }, + "classification_type": "multi-label" + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "topics", + "kind": "multi", + "labels": [ + "billing", + "hvac", + "shipping", + "payroll", + "account" + ], + "instruction": null + }, + { + "name": "channel", + "kind": "multi", + "labels": [ + "email", + "chat", + "phone" + ], + "instruction": null + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 66, + "row_sha256": "2680b449d6abeb125703f7d5e2ed6ac73b199fba48cc156364195df50f21a4dd", + "probabilities": { + "topics": { + "billing": 0.6416319915755686, + "hvac": 0.0882600855011472, + "shipping": 0.0735680225623457, + "payroll": 0.14623319051081268, + "account": 0.3147806998864284 + }, + "channel": { + "email": 0.43045308896362755, + "chat": 0.38332163952686493, + "phone": 0.2501468759749388 + } + } + }, + { + "max_len": null, + "row_length": 76, + "row_sha256": "6f71af286d4ab3e988ef425d81092073bc5b203e32b073d773c927f531ac9e6b", + "probabilities": { + "topics": { + "billing": 0.017712506111758678, + "hvac": 0.944989419686647, + "shipping": 0.07358077951763287, + "payroll": 0.04777493582324472, + "account": 0.1685462301030868 + }, + "channel": { + "email": 0.23220887866389917, + "chat": 0.665498256703042, + "phone": 0.22108351714502808 + } + } + }, + { + "max_len": null, + "row_length": 77, + "row_sha256": "1e8a2db490476461edb7a2837122de3e3b085e8ece75cd717262b9921f5b7cff", + "probabilities": { + "topics": { + "billing": 0.15004572929430768, + "hvac": 0.10191936995347431, + "shipping": 0.6472975803421689, + "payroll": 0.11279554943797881, + "account": 0.19865343262148402 + }, + "channel": { + "email": 0.4226878971090922, + "chat": 0.3867389474534807, + "phone": 0.23131778592466928 + } + } + }, + { + "max_len": null, + "row_length": 67, + "row_sha256": "d343acb14da3e5967aeac539fbedda59d46a329c84ef9b4f0626f34725550ba0", + "probabilities": { + "topics": { + "billing": 0.1685900372459668, + "hvac": 0.07682372625674679, + "shipping": 0.6614131484561885, + "payroll": 0.11555529379979552, + "account": 0.24600949159093294 + }, + "channel": { + "email": 0.32224852165893436, + "chat": 0.4105203258786524, + "phone": 0.2764912085240724 + } + } + }, + { + "max_len": null, + "row_length": 63, + "row_sha256": "4409d5e753bd2df9babb995bb8125d5936086c72cb06d8ec81cd44750c13ce0b", + "probabilities": { + "topics": { + "billing": 0.15368615750732142, + "hvac": 0.17753931685799718, + "shipping": 0.2033869752732013, + "payroll": 0.21487835470050054, + "account": 0.42934832448636034 + }, + "channel": { + "email": 0.39453303388797195, + "chat": 0.25249484021498514, + "phone": 0.2939829186547611 + } + } + } + ] + }, + { + "name": "labels", + "request": { + "labels": [ + "billing", + "technical", + "hospitality", + "logistics", + "other" + ], + "instruction": "Which team should handle this?" + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "label", + "kind": "single", + "labels": [ + "billing", + "technical", + "hospitality", + "logistics", + "other" + ], + "instruction": "Which team should handle this?" + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 59, + "row_sha256": "870392c7cc28f19baeeae73c3ccceb5a268fa62f819f283cba664c3c7fb82f54", + "probabilities": { + "label": { + "billing": 0.5514985694867639, + "technical": 0.09372265550362922, + "hospitality": 0.10261645035749171, + "logistics": 0.06636174841463639, + "other": 0.18580057623747875 + } + } + }, + { + "max_len": null, + "row_length": 69, + "row_sha256": "2502ce5f5bc07706a62e091fae564d54a751850d79503ce9aad932de647997fc", + "probabilities": { + "label": { + "billing": 0.026132864401067053, + "technical": 0.07778808403763922, + "hospitality": 0.4600355436867869, + "logistics": 0.16941801606947862, + "other": 0.2666254918050283 + } + } + }, + { + "max_len": null, + "row_length": 70, + "row_sha256": "005b87f70fdd7fd2b634d8882f2975f3a2ffb440cde67aadf480ee65f20ec44f", + "probabilities": { + "label": { + "billing": 0.15612624966880803, + "technical": 0.11420660882350125, + "hospitality": 0.19085045954678986, + "logistics": 0.3363211840690228, + "other": 0.2024954978918781 + } + } + }, + { + "max_len": null, + "row_length": 60, + "row_sha256": "cb2ed34055fdc627e922acd4e7d2a009ef56b975cedfa783f2e7475ca4a61ec1", + "probabilities": { + "label": { + "billing": 0.1445198363405052, + "technical": 0.07370607869861916, + "hospitality": 0.1838798144824622, + "logistics": 0.21094237652329204, + "other": 0.3869518939551213 + } + } + }, + { + "max_len": null, + "row_length": 56, + "row_sha256": "f5557c978150f760232299b4e33fcf9501b5494b663ea0f5f2186ec48bef0a57", + "probabilities": { + "label": { + "billing": 0.10985101446355566, + "technical": 0.1716885293890112, + "hospitality": 0.12870966210052184, + "logistics": 0.13776654860538948, + "other": 0.4519842454415218 + } + } + } + ] + }, + { + "name": "long_document", + "request": { + "output_schema": { + "intent": { + "type": "choice", + "instructions": "What does the customer want?", + "criteria": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": "" + } + }, + "urgency": { + "type": "score", + "instructions": "How urgent is this request?", + "criteria": [ + "not urgent", + "within a week", + "today", + "blocking or past a deadline" + ] + }, + "needs_human": { + "type": "noul", + "instructions": "Must a person act on this?", + "criteria": { + "true": "automation cannot resolve it" + } + }, + "rating": { + "type": "score", + "instructions": "Rate the severity from 0 to 10.", + "criteria": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ] + } + } + }, + "documents": "long", + "reference_call": { + "tasks": [ + { + "name": "intent", + "kind": "single", + "labels": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": null + }, + "instruction": "What does the customer want?" + }, + { + "name": "urgency", + "kind": "single", + "labels": { + "0": "not urgent", + "1": "within a week", + "2": "today", + "3": "blocking or past a deadline" + }, + "instruction": "How urgent is this request?" + }, + { + "name": "needs_human", + "kind": "single", + "labels": { + "yes": "automation cannot resolve it", + "no": null + }, + "instruction": "Must a person act on this?" + }, + { + "name": "rating", + "kind": "single", + "labels": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ], + "instruction": "Rate the severity from 0 to 10." + } + ] + }, + "results": [ + { + "max_len": 1459, + "row_length": 2048, + "row_sha256": "12ead9e8546d36e87e72f6d143f7be1590ff562d235b89a03d6391d292601da1", + "probabilities": { + "intent": { + "refund_request": 0.21151807815922868, + "cancel_order": 0.14954527545004506, + "room_change": 0.15089548638457193, + "maintenance": 0.2658831389379384, + "other": 0.22215802106821586 + }, + "urgency": { + "0": 0.45403828246288297, + "1": 0.2055955922976189, + "2": 0.17165729095748355, + "3": 0.16870883428201458 + }, + "needs_human": { + "yes": 0.20926345200274268, + "no": 0.7907365479972572 + }, + "rating": { + "0": 0.14418349686774384, + "1": 0.09275335334433255, + "2": 0.08978627683037574, + "3": 0.09469647106423965, + "4": 0.08967755377697186, + "5": 0.08246900860389268, + "6": 0.07271769602859025, + "7": 0.07741888002042417, + "8": 0.08760383052157766, + "9": 0.09061254443533848, + "10": 0.07808088850651315 + } + } + } + ] + } + ] +} diff --git a/packages/sie_server/tests/adapters/goldens/gliner2_decide/fastino__GLiNER2.5-Decide.json b/packages/sie_server/tests/adapters/goldens/gliner2_decide/fastino__GLiNER2.5-Decide.json new file mode 100644 index 000000000..ced90d3a2 --- /dev/null +++ b/packages/sie_server/tests/adapters/goldens/gliner2_decide/fastino__GLiNER2.5-Decide.json @@ -0,0 +1,978 @@ +{ + "model": "fastino/GLiNER2.5-Decide", + "revision": "7ee5da4c2415e32259bcdc0b1a7367c32ce8d6f6", + "generated_with": { + "gliner2": "2.0.0", + "transformers": "5.17.0", + "torch": "2.9.1+cu128", + "device": "cpu", + "dtype": "float32", + "window": 512 + }, + "texts": [ + "My subscription renewed on April 15 for 5,400 yen after the service was already down. Can I get that charge refunded?", + "Guest in room 1408 says the AC has been out since yesterday and they want to move tonight or leave. They also asked for the incidentals hold to be released.", + "Das Paket kam beschädigt an, bitte schicken Sie Ersatz. Ich brauche es bis Freitag, sonst storniere ich die Bestellung!", + "请帮我取消订单，我不想要了。退款什么时候到账？", + "The treaty was signed in Paris in 1992. It entered into force the following year, after the last signatory ratified it." + ], + "long_text": "Quarterly operations report for the northern warehouse, week 0. The team shipped 12000 orders and handled 300 returns; late deliveries fell to 9 percent. Carrier contract 0 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 1. The team shipped 12037 orders and handled 301 returns; late deliveries fell to 8 percent. Carrier contract 1 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 2. The team shipped 12074 orders and handled 302 returns; late deliveries fell to 7 percent. Carrier contract 2 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 3. The team shipped 12111 orders and handled 303 returns; late deliveries fell to 6 percent. Carrier contract 3 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 4. The team shipped 12148 orders and handled 304 returns; late deliveries fell to 5 percent. Carrier contract 4 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 5. The team shipped 12185 orders and handled 305 returns; late deliveries fell to 4 percent. Carrier contract 5 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 6. The team shipped 12222 orders and handled 306 returns; late deliveries fell to 3 percent. Carrier contract 6 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 7. The team shipped 12259 orders and handled 307 returns; late deliveries fell to 2 percent. Carrier contract 7 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 8. The team shipped 12296 orders and handled 308 returns; late deliveries fell to 9 percent. Carrier contract 8 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 9. The team shipped 12333 orders and handled 309 returns; late deliveries fell to 8 percent. Carrier contract 9 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 10. The team shipped 12370 orders and handled 310 returns; late deliveries fell to 7 percent. Carrier contract 10 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 11. The team shipped 12407 orders and handled 311 returns; late deliveries fell to 6 percent. Carrier contract 11 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 12. The team shipped 12444 orders and handled 312 returns; late deliveries fell to 5 percent. Carrier contract 12 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 13. The team shipped 12481 orders and handled 313 returns; late deliveries fell to 4 percent. Carrier contract 13 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 14. The team shipped 12518 orders and handled 314 returns; late deliveries fell to 3 percent. Carrier contract 14 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 15. The team shipped 12555 orders and handled 315 returns; late deliveries fell to 2 percent. Carrier contract 15 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 16. The team shipped 12592 orders and handled 316 returns; late deliveries fell to 9 percent. Carrier contract 16 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 17. The team shipped 12629 orders and handled 317 returns; late deliveries fell to 8 percent. Carrier contract 17 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 18. The team shipped 12666 orders and handled 318 returns; late deliveries fell to 7 percent. Carrier contract 18 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 19. The team shipped 12703 orders and handled 319 returns; late deliveries fell to 6 percent. Carrier contract 19 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 20. The team shipped 12740 orders and handled 320 returns; late deliveries fell to 5 percent. Carrier contract 20 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 21. The team shipped 12777 orders and handled 321 returns; late deliveries fell to 4 percent. Carrier contract 21 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 22. The team shipped 12814 orders and handled 322 returns; late deliveries fell to 3 percent. Carrier contract 22 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 23. The team shipped 12851 orders and handled 323 returns; late deliveries fell to 2 percent. Carrier contract 23 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 24. The team shipped 12888 orders and handled 324 returns; late deliveries fell to 9 percent. Carrier contract 24 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 25. The team shipped 12925 orders and handled 325 returns; late deliveries fell to 8 percent. Carrier contract 25 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 26. The team shipped 12962 orders and handled 326 returns; late deliveries fell to 7 percent. Carrier contract 26 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 27. The team shipped 12999 orders and handled 327 returns; late deliveries fell to 6 percent. Carrier contract 27 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 28. The team shipped 13036 orders and handled 328 returns; late deliveries fell to 5 percent. Carrier contract 28 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 29. The team shipped 13073 orders and handled 329 returns; late deliveries fell to 4 percent. Carrier contract 29 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 30. The team shipped 13110 orders and handled 330 returns; late deliveries fell to 3 percent. Carrier contract 30 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 31. The team shipped 13147 orders and handled 331 returns; late deliveries fell to 2 percent. Carrier contract 31 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 32. The team shipped 13184 orders and handled 332 returns; late deliveries fell to 9 percent. Carrier contract 32 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 33. The team shipped 13221 orders and handled 333 returns; late deliveries fell to 8 percent. Carrier contract 33 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 34. The team shipped 13258 orders and handled 334 returns; late deliveries fell to 7 percent. Carrier contract 34 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 35. The team shipped 13295 orders and handled 335 returns; late deliveries fell to 6 percent. Carrier contract 35 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 36. The team shipped 13332 orders and handled 336 returns; late deliveries fell to 5 percent. Carrier contract 36 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 37. The team shipped 13369 orders and handled 337 returns; late deliveries fell to 4 percent. Carrier contract 37 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 38. The team shipped 13406 orders and handled 338 returns; late deliveries fell to 3 percent. Carrier contract 38 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 39. The team shipped 13443 orders and handled 339 returns; late deliveries fell to 2 percent. Carrier contract 39 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 40. The team shipped 13480 orders and handled 340 returns; late deliveries fell to 9 percent. Carrier contract 40 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 41. The team shipped 13517 orders and handled 341 returns; late deliveries fell to 8 percent. Carrier contract 41 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 42. The team shipped 13554 orders and handled 342 returns; late deliveries fell to 7 percent. Carrier contract 42 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 43. The team shipped 13591 orders and handled 343 returns; late deliveries fell to 6 percent. Carrier contract 43 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 44. The team shipped 13628 orders and handled 344 returns; late deliveries fell to 5 percent. Carrier contract 44 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 45. The team shipped 13665 orders and handled 345 returns; late deliveries fell to 4 percent. Carrier contract 45 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 46. The team shipped 13702 orders and handled 346 returns; late deliveries fell to 3 percent. Carrier contract 46 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 47. The team shipped 13739 orders and handled 347 returns; late deliveries fell to 2 percent. Carrier contract 47 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 48. The team shipped 13776 orders and handled 348 returns; late deliveries fell to 9 percent. Carrier contract 48 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 49. The team shipped 13813 orders and handled 349 returns; late deliveries fell to 8 percent. Carrier contract 49 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 50. The team shipped 13850 orders and handled 350 returns; late deliveries fell to 7 percent. Carrier contract 50 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 51. The team shipped 13887 orders and handled 351 returns; late deliveries fell to 6 percent. Carrier contract 51 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 52. The team shipped 13924 orders and handled 352 returns; late deliveries fell to 5 percent. Carrier contract 52 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 53. The team shipped 13961 orders and handled 353 returns; late deliveries fell to 4 percent. Carrier contract 53 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 54. The team shipped 13998 orders and handled 354 returns; late deliveries fell to 3 percent. Carrier contract 54 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 55. The team shipped 14035 orders and handled 355 returns; late deliveries fell to 2 percent. Carrier contract 55 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 56. The team shipped 14072 orders and handled 356 returns; late deliveries fell to 9 percent. Carrier contract 56 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 57. The team shipped 14109 orders and handled 357 returns; late deliveries fell to 8 percent. Carrier contract 57 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 58. The team shipped 14146 orders and handled 358 returns; late deliveries fell to 7 percent. Carrier contract 58 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 59. The team shipped 14183 orders and handled 359 returns; late deliveries fell to 6 percent. Carrier contract 59 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. In the last week the payroll system failed during the month-end close and salaries for 40 staff are delayed; finance needs a fix before the Friday 5pm cutoff", + "cases": [ + { + "name": "questions", + "request": { + "output_schema": { + "intent": { + "type": "choice", + "instructions": "What does the customer want?", + "criteria": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": "" + } + }, + "urgency": { + "type": "score", + "instructions": "How urgent is this request?", + "criteria": [ + "not urgent", + "within a week", + "today", + "blocking or past a deadline" + ] + }, + "needs_human": { + "type": "noul", + "instructions": "Must a person act on this?", + "criteria": { + "true": "automation cannot resolve it" + } + }, + "rating": { + "type": "score", + "instructions": "Rate the severity from 0 to 10.", + "criteria": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ] + } + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "intent", + "kind": "single", + "labels": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": null + }, + "instruction": "What does the customer want?" + }, + { + "name": "urgency", + "kind": "single", + "labels": { + "0": "not urgent", + "1": "within a week", + "2": "today", + "3": "blocking or past a deadline" + }, + "instruction": "How urgent is this request?" + }, + { + "name": "needs_human", + "kind": "single", + "labels": { + "yes": "automation cannot resolve it", + "no": null + }, + "instruction": "Must a person act on this?" + }, + { + "name": "rating", + "kind": "single", + "labels": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ], + "instruction": "Rate the severity from 0 to 10." + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 200, + "row_sha256": "6919d38ae1082478886241e227d1aa0c80dc5365cbab17d47759d6261258da0f", + "probabilities": { + "intent": { + "refund_request": 0.8377580469759476, + "cancel_order": 0.03852756093973993, + "room_change": 0.024106327678254623, + "maintenance": 0.04407026570295509, + "other": 0.055537798703102804 + }, + "urgency": { + "0": 0.29109837822070345, + "1": 0.2572241862276529, + "2": 0.22657261090515418, + "3": 0.22510482464648943 + }, + "needs_human": { + "yes": 0.41316766746022937, + "no": 0.5868323325397707 + }, + "rating": { + "0": 0.09141969335093744, + "1": 0.09857008430962419, + "2": 0.10069676406609644, + "3": 0.09509464909003124, + "4": 0.09377955898929459, + "5": 0.0936186914806251, + "6": 0.09112537516223466, + "7": 0.08825349993027677, + "8": 0.08611186275442732, + "9": 0.08330339921726368, + "10": 0.07802642164918852 + } + } + }, + { + "max_len": null, + "row_length": 209, + "row_sha256": "378784753766786bb22bda1f93e19f1d273e96d8e5f201e36796f75704e49475", + "probabilities": { + "intent": { + "refund_request": 0.029723083247197354, + "cancel_order": 0.037174491439999846, + "room_change": 0.484220701573441, + "maintenance": 0.3757058998891551, + "other": 0.07317582385020666 + }, + "urgency": { + "0": 0.18856515474142382, + "1": 0.2032561988854108, + "2": 0.34399317756475617, + "3": 0.26418546880840915 + }, + "needs_human": { + "yes": 0.506815326158382, + "no": 0.4931846738416181 + }, + "rating": { + "0": 0.07729608006338132, + "1": 0.08398762061438621, + "2": 0.09059413593157098, + "3": 0.08995247839427273, + "4": 0.0904505926598452, + "5": 0.09065379130097416, + "6": 0.09153285604096285, + "7": 0.09536671224226967, + "8": 0.09727355391755875, + "9": 0.09893423621963636, + "10": 0.09395794261514183 + } + } + }, + { + "max_len": null, + "row_length": 210, + "row_sha256": "4245a966b66633b480ad55681e5848d60502f28186ad01cb369a21eecc19d554", + "probabilities": { + "intent": { + "refund_request": 0.08768791772066176, + "cancel_order": 0.4032474546384302, + "room_change": 0.0676108286320946, + "maintenance": 0.2843711044258883, + "other": 0.15708269458292515 + }, + "urgency": { + "0": 0.10023291535387574, + "1": 0.5274737318781414, + "2": 0.13828811542678004, + "3": 0.2340052373412029 + }, + "needs_human": { + "yes": 0.457295319107406, + "no": 0.5427046808925939 + }, + "rating": { + "0": 0.0770856873928381, + "1": 0.08909462235547622, + "2": 0.08633420052808946, + "3": 0.09244801312751426, + "4": 0.09230195057410978, + "5": 0.09233886285323793, + "6": 0.0919677721004221, + "7": 0.09431976767464116, + "8": 0.09618620413658666, + "9": 0.09589263446779182, + "10": 0.09203028478929243 + } + } + }, + { + "max_len": null, + "row_length": 204, + "row_sha256": "1a868bfc49e4d3ed764ae98247b20a819babff637bae8bebe630a53b657a5f4e", + "probabilities": { + "intent": { + "refund_request": 0.11429798043925134, + "cancel_order": 0.7616150279997952, + "room_change": 0.0224081675801431, + "maintenance": 0.03578719682673999, + "other": 0.0658916271540705 + }, + "urgency": { + "0": 0.24640737458214684, + "1": 0.29261238712466064, + "2": 0.23867246959440616, + "3": 0.2223077686987864 + }, + "needs_human": { + "yes": 0.49068747410767855, + "no": 0.5093125258923216 + }, + "rating": { + "0": 0.09480157073853049, + "1": 0.0997239212313904, + "2": 0.0946457420047139, + "3": 0.09011679334095365, + "4": 0.08841782394523968, + "5": 0.08879611856197256, + "6": 0.08826032476943929, + "7": 0.08946298738763951, + "8": 0.08962682086336436, + "9": 0.08962400023944397, + "10": 0.08652389691731212 + } + } + }, + { + "max_len": null, + "row_length": 199, + "row_sha256": "f67efea6dbb9560105ec32188db5cc84e4388fac0e38819d3104ab26aa1bf854", + "probabilities": { + "intent": { + "refund_request": 0.14503878048346744, + "cancel_order": 0.12423839906773355, + "room_change": 0.13170078778888086, + "maintenance": 0.19524724683521613, + "other": 0.403774785824702 + }, + "urgency": { + "0": 0.3260858224890541, + "1": 0.23958053300314197, + "2": 0.21763053282523342, + "3": 0.21670311168257056 + }, + "needs_human": { + "yes": 0.3841253070792061, + "no": 0.6158746929207939 + }, + "rating": { + "0": 0.10043559765332136, + "1": 0.09672647001745906, + "2": 0.09266307684679315, + "3": 0.09027974074608079, + "4": 0.0884167321785384, + "5": 0.08841725918490037, + "6": 0.08749726438918604, + "7": 0.08815854026538808, + "8": 0.08858270454978726, + "9": 0.08930160822711092, + "10": 0.08952100594143463 + } + } + } + ] + }, + { + "name": "question_over_passage", + "request": { + "output_schema": { + "answer": { + "type": "noul", + "instructions": "Did the treaty enter into force in 1992?" + } + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "answer", + "kind": "single", + "labels": [ + "yes", + "no" + ], + "instruction": "Did the treaty enter into force in 1992?" + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 46, + "row_sha256": "f1b6924bfe5f6581cb1891d51df04dbb66661037e3a6865242ab9027d7bb6ecc", + "probabilities": { + "answer": { + "yes": 0.19415866896556594, + "no": 0.8058413310344341 + } + } + }, + { + "max_len": null, + "row_length": 55, + "row_sha256": "139edd959f483d6429cc098de5c07476ad0cb69f7f809721ff866686aee15a6e", + "probabilities": { + "answer": { + "yes": 0.25255249443937705, + "no": 0.7474475055606229 + } + } + }, + { + "max_len": null, + "row_length": 56, + "row_sha256": "b1ec6d006d5970701c2e6b1090112b5900e3edd744db266b74b1180e39a6b5c5", + "probabilities": { + "answer": { + "yes": 0.21395905286206765, + "no": 0.7860409471379324 + } + } + }, + { + "max_len": null, + "row_length": 50, + "row_sha256": "15b244a1380486c145862fe431fe173636709c1f9817d06186c4d02555698fd3", + "probabilities": { + "answer": { + "yes": 0.241598028729905, + "no": 0.758401971270095 + } + } + }, + { + "max_len": null, + "row_length": 45, + "row_sha256": "69b797baba0d3fc0ef68b7ae15bca9877166b7ba0a0b4094c366a9b0b77f786a", + "probabilities": { + "answer": { + "yes": 0.9962633614014068, + "no": 0.0037366385985932228 + } + } + } + ] + }, + { + "name": "label_groups", + "request": { + "instruction": "Triage the message.", + "options": { + "label_groups": { + "intent": [ + "refund_request", + "cancel_order", + "room_change", + "maintenance", + "other" + ], + "priority": [ + "low", + "normal", + "high", + "urgent" + ], + "needs_human": [ + "yes", + "no" + ] + } + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "intent", + "kind": "single", + "labels": [ + "refund_request", + "cancel_order", + "room_change", + "maintenance", + "other" + ], + "instruction": "Triage the message." + }, + { + "name": "priority", + "kind": "single", + "labels": [ + "low", + "normal", + "high", + "urgent" + ], + "instruction": "Triage the message." + }, + { + "name": "needs_human", + "kind": "single", + "labels": [ + "yes", + "no" + ], + "instruction": "Triage the message." + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 94, + "row_sha256": "1516623c51f5876d43b94374382bfcbab3e708d635842f8a8d368a6f0f7c3611", + "probabilities": { + "intent": { + "refund_request": 0.9404910799385175, + "cancel_order": 0.009240010702618115, + "room_change": 0.008283562972581414, + "maintenance": 0.018541945269809724, + "other": 0.02344340111647326 + }, + "priority": { + "low": 0.26495476715974753, + "normal": 0.2919947411897471, + "high": 0.24162719831130725, + "urgent": 0.20142329333919817 + }, + "needs_human": { + "yes": 0.5203135336938709, + "no": 0.47968646630612904 + } + } + }, + { + "max_len": null, + "row_length": 103, + "row_sha256": "7f060ee66d77fffe7bb89b96b9285a3209018df575ee59b9f9f05782a6e21dca", + "probabilities": { + "intent": { + "refund_request": 0.030929660479936883, + "cancel_order": 0.04474916201660331, + "room_change": 0.5473463846151558, + "maintenance": 0.2983445818400503, + "other": 0.07863021104825363 + }, + "priority": { + "low": 0.17118780397141312, + "normal": 0.19198126000013602, + "high": 0.2803799321514195, + "urgent": 0.3564510038770313 + }, + "needs_human": { + "yes": 0.5837333486974632, + "no": 0.41626665130253676 + } + } + }, + { + "max_len": null, + "row_length": 104, + "row_sha256": "a8c0fd35229b1940ef1e91b5c1fa3c0d988de968dec5c4ab58d52c4fe215f315", + "probabilities": { + "intent": { + "refund_request": 0.11260952205275208, + "cancel_order": 0.4788022995943985, + "room_change": 0.080814041498909, + "maintenance": 0.1633771286954337, + "other": 0.16439700815850672 + }, + "priority": { + "low": 0.19900531991790843, + "normal": 0.23440890944115378, + "high": 0.28657841698934516, + "urgent": 0.2800073536515927 + }, + "needs_human": { + "yes": 0.5132498679436681, + "no": 0.4867501320563318 + } + } + }, + { + "max_len": null, + "row_length": 98, + "row_sha256": "65fb03d566f3b7d496e8ce8f8af10784dcc96ac8f571035a126526a52ee9f272", + "probabilities": { + "intent": { + "refund_request": 0.05054963066425079, + "cancel_order": 0.8874969348495433, + "room_change": 0.013372391244517381, + "maintenance": 0.016881204609475727, + "other": 0.03169983863221282 + }, + "priority": { + "low": 0.2712209877820008, + "normal": 0.2648551098603429, + "high": 0.2396694497841234, + "urgent": 0.22425445257353285 + }, + "needs_human": { + "yes": 0.5222328267118117, + "no": 0.4777671732881883 + } + } + }, + { + "max_len": null, + "row_length": 93, + "row_sha256": "9b8ef8ca9e36afb8315c9ec36b034676a964fd2fcc8288ac47b4e22e498595a6", + "probabilities": { + "intent": { + "refund_request": 0.14538371652892604, + "cancel_order": 0.13456491122073228, + "room_change": 0.15548323147140303, + "maintenance": 0.1894045893383865, + "other": 0.37516355144055213 + }, + "priority": { + "low": 0.28316705552295274, + "normal": 0.3073815814715746, + "high": 0.21288837005748373, + "urgent": 0.19656299294798893 + }, + "needs_human": { + "yes": 0.399748480944129, + "no": 0.600251519055871 + } + } + } + ] + }, + { + "name": "label_groups_multi_label", + "request": { + "options": { + "label_groups": { + "topics": [ + "billing", + "hvac", + "shipping", + "payroll", + "account" + ], + "channel": [ + "email", + "chat", + "phone" + ] + }, + "classification_type": "multi-label" + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "topics", + "kind": "multi", + "labels": [ + "billing", + "hvac", + "shipping", + "payroll", + "account" + ], + "instruction": null + }, + { + "name": "channel", + "kind": "multi", + "labels": [ + "email", + "chat", + "phone" + ], + "instruction": null + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 55, + "row_sha256": "40fdca3925de75a1edaad7eaef130d70a461e2b8374c635ea7e6cf2078a5a06b", + "probabilities": { + "topics": { + "billing": 0.6100859383725932, + "hvac": 0.08558381119942395, + "shipping": 0.109851818451355, + "payroll": 0.11790444306145235, + "account": 0.49009397998234994 + }, + "channel": { + "email": 0.4393256542805312, + "chat": 0.31808374999366923, + "phone": 0.31821417538168867 + } + } + }, + { + "max_len": null, + "row_length": 64, + "row_sha256": "ce3ff4745162ef8c08dc825fdc697459def6694f6269cd47bf1cfe3dfb7ea030", + "probabilities": { + "topics": { + "billing": 0.11756958579524555, + "hvac": 0.9855033092083002, + "shipping": 0.08603492691661602, + "payroll": 0.0351185321173825, + "account": 0.431877318721589 + }, + "channel": { + "email": 0.3101318742107335, + "chat": 0.29052328241819636, + "phone": 0.39510966219520427 + } + } + }, + { + "max_len": null, + "row_length": 65, + "row_sha256": "7615406c2521c7dcc9465633f98f4f01449e9a46bab3dd86aa478efcef53a83d", + "probabilities": { + "topics": { + "billing": 0.12995622663470716, + "hvac": 0.04470228709947839, + "shipping": 0.8983644130975078, + "payroll": 0.06824170920505014, + "account": 0.16763747414911004 + }, + "channel": { + "email": 0.37851471882062054, + "chat": 0.3402033652870692, + "phone": 0.31909859097854637 + } + } + }, + { + "max_len": null, + "row_length": 59, + "row_sha256": "e1d5aebcdba249e2c48c1f3873dc80604fc7bf7e488b714d2e3cf9c08ccee21a", + "probabilities": { + "topics": { + "billing": 0.3892914368709659, + "hvac": 0.07744815922650088, + "shipping": 0.09598050547766758, + "payroll": 0.24709012825623203, + "account": 0.41630377598517604 + }, + "channel": { + "email": 0.4025773417673173, + "chat": 0.3252571956034508, + "phone": 0.30325939363798843 + } + } + }, + { + "max_len": null, + "row_length": 54, + "row_sha256": "b7549a384bf57bdc4fcf39e7bc8f870c75a6aa715238fca8b7e75407b49222fe", + "probabilities": { + "topics": { + "billing": 0.16974911168220216, + "hvac": 0.20776213522335127, + "shipping": 0.20308826816848574, + "payroll": 0.20579542231832443, + "account": 0.28705295255159574 + }, + "channel": { + "email": 0.3235962941510402, + "chat": 0.2508169011844059, + "phone": 0.31341666365861925 + } + } + } + ] + }, + { + "name": "labels", + "request": { + "labels": [ + "billing", + "technical", + "hospitality", + "logistics", + "other" + ], + "instruction": "Which team should handle this?" + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "label", + "kind": "single", + "labels": [ + "billing", + "technical", + "hospitality", + "logistics", + "other" + ], + "instruction": "Which team should handle this?" + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 49, + "row_sha256": "6050c2323efaed82c4ed250201774d8da2da00066a625b421cc88df838b81c9a", + "probabilities": { + "label": { + "billing": 0.6348261266194452, + "technical": 0.09682806381754619, + "hospitality": 0.07568381152723842, + "logistics": 0.06603000290198079, + "other": 0.12663199513378953 + } + } + }, + { + "max_len": null, + "row_length": 58, + "row_sha256": "3ebdc2e3e48fd7c40579636fd1233ceeb789121942701e648d7d27347ffd87ce", + "probabilities": { + "label": { + "billing": 0.0881362498721967, + "technical": 0.29876494621662836, + "hospitality": 0.40527814226592346, + "logistics": 0.08172020574489486, + "other": 0.1261004559003567 + } + } + }, + { + "max_len": null, + "row_length": 59, + "row_sha256": "79d4f61af6f822c563793cf9b5db9d46bf3bb93ee6c82aef9804d4b26836239f", + "probabilities": { + "label": { + "billing": 0.018794803304148906, + "technical": 0.025086530286738917, + "hospitality": 0.02809187713598207, + "logistics": 0.8866642786883767, + "other": 0.041362510584753445 + } + } + }, + { + "max_len": null, + "row_length": 53, + "row_sha256": "9c624e647a80ae999866a150d2b43bf11954eac8dbade4e9dc997595d3bedddc", + "probabilities": { + "label": { + "billing": 0.5141262686527034, + "technical": 0.09120221888214329, + "hospitality": 0.11322964128868573, + "logistics": 0.09006075757884066, + "other": 0.19138111359762694 + } + } + }, + { + "max_len": null, + "row_length": 48, + "row_sha256": "735a68a58ef0b4c2533b686cc4210b99ef10d724f9273c3d74fd3359fba3fd8b", + "probabilities": { + "label": { + "billing": 0.13911313340897377, + "technical": 0.17158245067372063, + "hospitality": 0.1949804517250651, + "logistics": 0.1784517861599918, + "other": 0.3158721780322487 + } + } + } + ] + }, + { + "name": "long_document", + "request": { + "output_schema": { + "intent": { + "type": "choice", + "instructions": "What does the customer want?", + "criteria": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": "" + } + }, + "urgency": { + "type": "score", + "instructions": "How urgent is this request?", + "criteria": [ + "not urgent", + "within a week", + "today", + "blocking or past a deadline" + ] + }, + "needs_human": { + "type": "noul", + "instructions": "Must a person act on this?", + "criteria": { + "true": "automation cannot resolve it" + } + }, + "rating": { + "type": "score", + "instructions": "Rate the severity from 0 to 10.", + "criteria": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ] + } + } + }, + "documents": "long", + "reference_call": { + "tasks": [ + { + "name": "intent", + "kind": "single", + "labels": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": null + }, + "instruction": "What does the customer want?" + }, + { + "name": "urgency", + "kind": "single", + "labels": { + "0": "not urgent", + "1": "within a week", + "2": "today", + "3": "blocking or past a deadline" + }, + "instruction": "How urgent is this request?" + }, + { + "name": "needs_human", + "kind": "single", + "labels": { + "yes": "automation cannot resolve it", + "no": null + }, + "instruction": "Must a person act on this?" + }, + { + "name": "rating", + "kind": "single", + "labels": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ], + "instruction": "Rate the severity from 0 to 10." + } + ] + }, + "results": [ + { + "max_len": 331, + "row_length": 512, + "row_sha256": "8baa0cc1a339dfaac8aee60aa4e2f32257c532897b4c20588ad98c67e11a97ed", + "probabilities": { + "intent": { + "refund_request": 0.12487164177499598, + "cancel_order": 0.12840286891749098, + "room_change": 0.11701781893671682, + "maintenance": 0.3565272392590684, + "other": 0.27318043111172785 + }, + "urgency": { + "0": 0.41092748904249427, + "1": 0.21650117441683117, + "2": 0.19534805155525328, + "3": 0.1772232849854213 + }, + "needs_human": { + "yes": 0.22369402482552722, + "no": 0.7763059751744729 + }, + "rating": { + "0": 0.1291215580428145, + "1": 0.11323780328809659, + "2": 0.10058878104286714, + "3": 0.09157749817625348, + "4": 0.08967113868462538, + "5": 0.08585546384144833, + "6": 0.08070856652077378, + "7": 0.07735948525841996, + "8": 0.07736450217237832, + "9": 0.07685877361003457, + "10": 0.07765642936228795 + } + } + } + ] + } + ] +} diff --git a/packages/sie_server/tests/adapters/goldens/gliner2_decide/fastino__GLiNER2.5-multi-Decide.json b/packages/sie_server/tests/adapters/goldens/gliner2_decide/fastino__GLiNER2.5-multi-Decide.json new file mode 100644 index 000000000..5742b2bff --- /dev/null +++ b/packages/sie_server/tests/adapters/goldens/gliner2_decide/fastino__GLiNER2.5-multi-Decide.json @@ -0,0 +1,978 @@ +{ + "model": "fastino/GLiNER2.5-multi-Decide", + "revision": "6bc1d43d201b0691e733626389af8c57eea3ea68", + "generated_with": { + "gliner2": "2.0.0", + "transformers": "5.17.0", + "torch": "2.9.1+cu128", + "device": "cpu", + "dtype": "float32", + "window": 2048 + }, + "texts": [ + "My subscription renewed on April 15 for 5,400 yen after the service was already down. Can I get that charge refunded?", + "Guest in room 1408 says the AC has been out since yesterday and they want to move tonight or leave. They also asked for the incidentals hold to be released.", + "Das Paket kam beschädigt an, bitte schicken Sie Ersatz. Ich brauche es bis Freitag, sonst storniere ich die Bestellung!", + "请帮我取消订单，我不想要了。退款什么时候到账？", + "The treaty was signed in Paris in 1992. It entered into force the following year, after the last signatory ratified it." + ], + "long_text": "Quarterly operations report for the northern warehouse, week 0. The team shipped 12000 orders and handled 300 returns; late deliveries fell to 9 percent. Carrier contract 0 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 1. The team shipped 12037 orders and handled 301 returns; late deliveries fell to 8 percent. Carrier contract 1 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 2. The team shipped 12074 orders and handled 302 returns; late deliveries fell to 7 percent. Carrier contract 2 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 3. The team shipped 12111 orders and handled 303 returns; late deliveries fell to 6 percent. Carrier contract 3 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 4. The team shipped 12148 orders and handled 304 returns; late deliveries fell to 5 percent. Carrier contract 4 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 5. The team shipped 12185 orders and handled 305 returns; late deliveries fell to 4 percent. Carrier contract 5 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 6. The team shipped 12222 orders and handled 306 returns; late deliveries fell to 3 percent. Carrier contract 6 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 7. The team shipped 12259 orders and handled 307 returns; late deliveries fell to 2 percent. Carrier contract 7 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 8. The team shipped 12296 orders and handled 308 returns; late deliveries fell to 9 percent. Carrier contract 8 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 9. The team shipped 12333 orders and handled 309 returns; late deliveries fell to 8 percent. Carrier contract 9 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 10. The team shipped 12370 orders and handled 310 returns; late deliveries fell to 7 percent. Carrier contract 10 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 11. The team shipped 12407 orders and handled 311 returns; late deliveries fell to 6 percent. Carrier contract 11 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 12. The team shipped 12444 orders and handled 312 returns; late deliveries fell to 5 percent. Carrier contract 12 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 13. The team shipped 12481 orders and handled 313 returns; late deliveries fell to 4 percent. Carrier contract 13 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 14. The team shipped 12518 orders and handled 314 returns; late deliveries fell to 3 percent. Carrier contract 14 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 15. The team shipped 12555 orders and handled 315 returns; late deliveries fell to 2 percent. Carrier contract 15 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 16. The team shipped 12592 orders and handled 316 returns; late deliveries fell to 9 percent. Carrier contract 16 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 17. The team shipped 12629 orders and handled 317 returns; late deliveries fell to 8 percent. Carrier contract 17 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 18. The team shipped 12666 orders and handled 318 returns; late deliveries fell to 7 percent. Carrier contract 18 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 19. The team shipped 12703 orders and handled 319 returns; late deliveries fell to 6 percent. Carrier contract 19 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 20. The team shipped 12740 orders and handled 320 returns; late deliveries fell to 5 percent. Carrier contract 20 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 21. The team shipped 12777 orders and handled 321 returns; late deliveries fell to 4 percent. Carrier contract 21 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 22. The team shipped 12814 orders and handled 322 returns; late deliveries fell to 3 percent. Carrier contract 22 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 23. The team shipped 12851 orders and handled 323 returns; late deliveries fell to 2 percent. Carrier contract 23 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 24. The team shipped 12888 orders and handled 324 returns; late deliveries fell to 9 percent. Carrier contract 24 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 25. The team shipped 12925 orders and handled 325 returns; late deliveries fell to 8 percent. Carrier contract 25 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 26. The team shipped 12962 orders and handled 326 returns; late deliveries fell to 7 percent. Carrier contract 26 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 27. The team shipped 12999 orders and handled 327 returns; late deliveries fell to 6 percent. Carrier contract 27 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 28. The team shipped 13036 orders and handled 328 returns; late deliveries fell to 5 percent. Carrier contract 28 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 29. The team shipped 13073 orders and handled 329 returns; late deliveries fell to 4 percent. Carrier contract 29 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 30. The team shipped 13110 orders and handled 330 returns; late deliveries fell to 3 percent. Carrier contract 30 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 31. The team shipped 13147 orders and handled 331 returns; late deliveries fell to 2 percent. Carrier contract 31 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 32. The team shipped 13184 orders and handled 332 returns; late deliveries fell to 9 percent. Carrier contract 32 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 33. The team shipped 13221 orders and handled 333 returns; late deliveries fell to 8 percent. Carrier contract 33 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 34. The team shipped 13258 orders and handled 334 returns; late deliveries fell to 7 percent. Carrier contract 34 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 35. The team shipped 13295 orders and handled 335 returns; late deliveries fell to 6 percent. Carrier contract 35 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 36. The team shipped 13332 orders and handled 336 returns; late deliveries fell to 5 percent. Carrier contract 36 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 37. The team shipped 13369 orders and handled 337 returns; late deliveries fell to 4 percent. Carrier contract 37 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 38. The team shipped 13406 orders and handled 338 returns; late deliveries fell to 3 percent. Carrier contract 38 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 39. The team shipped 13443 orders and handled 339 returns; late deliveries fell to 2 percent. Carrier contract 39 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 40. The team shipped 13480 orders and handled 340 returns; late deliveries fell to 9 percent. Carrier contract 40 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 41. The team shipped 13517 orders and handled 341 returns; late deliveries fell to 8 percent. Carrier contract 41 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 42. The team shipped 13554 orders and handled 342 returns; late deliveries fell to 7 percent. Carrier contract 42 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 43. The team shipped 13591 orders and handled 343 returns; late deliveries fell to 6 percent. Carrier contract 43 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 44. The team shipped 13628 orders and handled 344 returns; late deliveries fell to 5 percent. Carrier contract 44 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 45. The team shipped 13665 orders and handled 345 returns; late deliveries fell to 4 percent. Carrier contract 45 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 46. The team shipped 13702 orders and handled 346 returns; late deliveries fell to 3 percent. Carrier contract 46 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 47. The team shipped 13739 orders and handled 347 returns; late deliveries fell to 2 percent. Carrier contract 47 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 48. The team shipped 13776 orders and handled 348 returns; late deliveries fell to 9 percent. Carrier contract 48 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 49. The team shipped 13813 orders and handled 349 returns; late deliveries fell to 8 percent. Carrier contract 49 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 50. The team shipped 13850 orders and handled 350 returns; late deliveries fell to 7 percent. Carrier contract 50 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 51. The team shipped 13887 orders and handled 351 returns; late deliveries fell to 6 percent. Carrier contract 51 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 52. The team shipped 13924 orders and handled 352 returns; late deliveries fell to 5 percent. Carrier contract 52 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 53. The team shipped 13961 orders and handled 353 returns; late deliveries fell to 4 percent. Carrier contract 53 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 54. The team shipped 13998 orders and handled 354 returns; late deliveries fell to 3 percent. Carrier contract 54 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 55. The team shipped 14035 orders and handled 355 returns; late deliveries fell to 2 percent. Carrier contract 55 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 40 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 56. The team shipped 14072 orders and handled 356 returns; late deliveries fell to 9 percent. Carrier contract 56 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 41 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 57. The team shipped 14109 orders and handled 357 returns; late deliveries fell to 8 percent. Carrier contract 57 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 42 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 58. The team shipped 14146 orders and handled 358 returns; late deliveries fell to 7 percent. Carrier contract 58 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 43 staff members and no overtime disputes were filed. Quarterly operations report for the northern warehouse, week 59. The team shipped 14183 orders and handled 359 returns; late deliveries fell to 6 percent. Carrier contract 59 renewed without changes, and the forklift inspection passed on schedule. Payroll ran on time for 44 staff members and no overtime disputes were filed. In the last week the payroll system failed during the month-end close and salaries for 40 staff are delayed; finance needs a fix before the Friday 5pm cutoff", + "cases": [ + { + "name": "questions", + "request": { + "output_schema": { + "intent": { + "type": "choice", + "instructions": "What does the customer want?", + "criteria": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": "" + } + }, + "urgency": { + "type": "score", + "instructions": "How urgent is this request?", + "criteria": [ + "not urgent", + "within a week", + "today", + "blocking or past a deadline" + ] + }, + "needs_human": { + "type": "noul", + "instructions": "Must a person act on this?", + "criteria": { + "true": "automation cannot resolve it" + } + }, + "rating": { + "type": "score", + "instructions": "Rate the severity from 0 to 10.", + "criteria": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ] + } + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "intent", + "kind": "single", + "labels": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": null + }, + "instruction": "What does the customer want?" + }, + { + "name": "urgency", + "kind": "single", + "labels": { + "0": "not urgent", + "1": "within a week", + "2": "today", + "3": "blocking or past a deadline" + }, + "instruction": "How urgent is this request?" + }, + { + "name": "needs_human", + "kind": "single", + "labels": { + "yes": "automation cannot resolve it", + "no": null + }, + "instruction": "Must a person act on this?" + }, + { + "name": "rating", + "kind": "single", + "labels": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ], + "instruction": "Rate the severity from 0 to 10." + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 234, + "row_sha256": "c300f4f1eedb7a932f9a995375e43c1140c4120432ee5f3f1ebea57a5306b397", + "probabilities": { + "intent": { + "refund_request": 0.8653422372199283, + "cancel_order": 0.03450084898476019, + "room_change": 0.012078067428275139, + "maintenance": 0.029505144996428447, + "other": 0.05857370137060786 + }, + "urgency": { + "0": 0.20284874594982216, + "1": 0.24311020085065396, + "2": 0.2659260870293298, + "3": 0.2881149661701941 + }, + "needs_human": { + "yes": 0.500748931801502, + "no": 0.49925106819849796 + }, + "rating": { + "0": 0.08046718993580655, + "1": 0.08906680895442307, + "2": 0.0934985956331202, + "3": 0.09595803061044571, + "4": 0.0957161247774777, + "5": 0.0976250881698765, + "6": 0.09257131565461933, + "7": 0.09211546742826163, + "8": 0.09119083759620662, + "9": 0.08866964362517778, + "10": 0.083120897614585 + } + } + }, + { + "max_len": null, + "row_length": 243, + "row_sha256": "39ec24fee33e88ea4f9e2be4a383bc7723dd6fce3f43bde73b7ade6fe56b218e", + "probabilities": { + "intent": { + "refund_request": 0.06817533353987852, + "cancel_order": 0.05605657413946528, + "room_change": 0.5351599176271147, + "maintenance": 0.19887152477280506, + "other": 0.14173664992073645 + }, + "urgency": { + "0": 0.17551762855334838, + "1": 0.21198615910952678, + "2": 0.2739309404720854, + "3": 0.33856527186503954 + }, + "needs_human": { + "yes": 0.5406472818180786, + "no": 0.45935271818192136 + }, + "rating": { + "0": 0.0770764216234726, + "1": 0.08235757125042457, + "2": 0.09340344438522002, + "3": 0.09967735617468781, + "4": 0.09744752606449507, + "5": 0.0978724809345962, + "6": 0.0926880414527889, + "7": 0.09324157698242443, + "8": 0.09109992451052977, + "9": 0.09078488015884718, + "10": 0.08435077646251349 + } + } + }, + { + "max_len": null, + "row_length": 237, + "row_sha256": "121a23ba29a87c358181480531e031042e80d8acfa5a7b3c605fb8c1bd45206f", + "probabilities": { + "intent": { + "refund_request": 0.054698441080062894, + "cancel_order": 0.7882612180903259, + "room_change": 0.019162545386786023, + "maintenance": 0.05998062902608238, + "other": 0.07789716641674277 + }, + "urgency": { + "0": 0.12770230550857256, + "1": 0.20154298257347203, + "2": 0.2764879624104309, + "3": 0.39426674950752444 + }, + "needs_human": { + "yes": 0.5644253730127116, + "no": 0.4355746269872884 + }, + "rating": { + "0": 0.07846375877537351, + "1": 0.08677583333018656, + "2": 0.09442011105084717, + "3": 0.09676355595498756, + "4": 0.09465799668051146, + "5": 0.09653925072154865, + "6": 0.09162503377236929, + "7": 0.09173296845120982, + "8": 0.09197188995942916, + "9": 0.09351012946865679, + "10": 0.08353947183488011 + } + } + }, + { + "max_len": null, + "row_length": 224, + "row_sha256": "0a8cc71303a6335a1674fd7fbfc90555dee59c3dd5b5ba33872d53839ec5610c", + "probabilities": { + "intent": { + "refund_request": 0.02263417836653408, + "cancel_order": 0.9552144548941393, + "room_change": 0.002488009099897836, + "maintenance": 0.003006055144091452, + "other": 0.016657302495337405 + }, + "urgency": { + "0": 0.23647786467295173, + "1": 0.24199428143202062, + "2": 0.25164947670141785, + "3": 0.26987837719360985 + }, + "needs_human": { + "yes": 0.5171507442346706, + "no": 0.4828492557653295 + }, + "rating": { + "0": 0.10010890234534435, + "1": 0.0932930938019073, + "2": 0.09464095392118523, + "3": 0.09350876429119166, + "4": 0.09227005902154302, + "5": 0.0938133078064512, + "6": 0.09067714763074344, + "7": 0.08938593115721644, + "8": 0.08804881981659674, + "9": 0.08525644177910455, + "10": 0.07899657842871609 + } + } + }, + { + "max_len": null, + "row_length": 233, + "row_sha256": "285cd7b230c2b36c2d0ef306302385c1412320d7f85216f12378614927423f11", + "probabilities": { + "intent": { + "refund_request": 0.13159847656083726, + "cancel_order": 0.15907767845788198, + "room_change": 0.09347238339833705, + "maintenance": 0.07421214863069613, + "other": 0.5416393129522475 + }, + "urgency": { + "0": 0.2489354429613785, + "1": 0.2423315371594498, + "2": 0.2519829155606013, + "3": 0.25675010431857037 + }, + "needs_human": { + "yes": 0.42528726425413543, + "no": 0.5747127357458645 + }, + "rating": { + "0": 0.07942884194267318, + "1": 0.08228779855000765, + "2": 0.0893583237073672, + "3": 0.0958273822239652, + "4": 0.09632633065105693, + "5": 0.09711759475509496, + "6": 0.09276840697923255, + "7": 0.09207804067792887, + "8": 0.09271766071540209, + "9": 0.0927527047329241, + "10": 0.08933691506434724 + } + } + } + ] + }, + { + "name": "question_over_passage", + "request": { + "output_schema": { + "answer": { + "type": "noul", + "instructions": "Did the treaty enter into force in 1992?" + } + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "answer", + "kind": "single", + "labels": [ + "yes", + "no" + ], + "instruction": "Did the treaty enter into force in 1992?" + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 58, + "row_sha256": "32deb0d1390ba395f566ad2a640a7d236eca61bd81ec8c6ebf2d329d341e8b6e", + "probabilities": { + "answer": { + "yes": 0.24841801885062556, + "no": 0.7515819811493744 + } + } + }, + { + "max_len": null, + "row_length": 67, + "row_sha256": "328bc4cd434a55d2ed2fe5b24054e4c414a2bf48f6bad833e7a438f85aaabc37", + "probabilities": { + "answer": { + "yes": 0.31462890543725375, + "no": 0.6853710945627463 + } + } + }, + { + "max_len": null, + "row_length": 61, + "row_sha256": "31649e942cad9062fbd50f2b14e281a7fce9b8b62f530d7a7a9dd36d55763fae", + "probabilities": { + "answer": { + "yes": 0.2669829496234064, + "no": 0.7330170503765936 + } + } + }, + { + "max_len": null, + "row_length": 48, + "row_sha256": "3e95581460471d68706aa973a93e636b66610219ddf79683eb1aacc425c3bcce", + "probabilities": { + "answer": { + "yes": 0.3410693353132902, + "no": 0.6589306646867097 + } + } + }, + { + "max_len": null, + "row_length": 57, + "row_sha256": "773a90ebab74562f25be0486fd665611a623f7762c52e0e797751271384bcb7b", + "probabilities": { + "answer": { + "yes": 0.9933019022020662, + "no": 0.00669809779793398 + } + } + } + ] + }, + { + "name": "label_groups", + "request": { + "instruction": "Triage the message.", + "options": { + "label_groups": { + "intent": [ + "refund_request", + "cancel_order", + "room_change", + "maintenance", + "other" + ], + "priority": [ + "low", + "normal", + "high", + "urgent" + ], + "needs_human": [ + "yes", + "no" + ] + } + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "intent", + "kind": "single", + "labels": [ + "refund_request", + "cancel_order", + "room_change", + "maintenance", + "other" + ], + "instruction": "Triage the message." + }, + { + "name": "priority", + "kind": "single", + "labels": [ + "low", + "normal", + "high", + "urgent" + ], + "instruction": "Triage the message." + }, + { + "name": "needs_human", + "kind": "single", + "labels": [ + "yes", + "no" + ], + "instruction": "Triage the message." + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 112, + "row_sha256": "5a2a8d3f334af6ac555598bfd44ed60cb74e5032319d30d965b680f070548adc", + "probabilities": { + "intent": { + "refund_request": 0.992542480604977, + "cancel_order": 0.0015130109425440358, + "room_change": 0.0004970936067399901, + "maintenance": 0.0012871165936307303, + "other": 0.004160298252108347 + }, + "priority": { + "low": 0.23502648533841594, + "normal": 0.3075152367534908, + "high": 0.2532555011019328, + "urgent": 0.20420277680616045 + }, + "needs_human": { + "yes": 0.5703937415542173, + "no": 0.4296062584457827 + } + } + }, + { + "max_len": null, + "row_length": 121, + "row_sha256": "71beb2ce681925aa29e38a95a75a680d3cef9ad783046b72cd4b2dfbe3a7d4ac", + "probabilities": { + "intent": { + "refund_request": 0.07405230650129703, + "cancel_order": 0.05687348320725224, + "room_change": 0.5682816286210692, + "maintenance": 0.15886877456777443, + "other": 0.14192380710260705 + }, + "priority": { + "low": 0.19033413382300554, + "normal": 0.2255110625686171, + "high": 0.29515538575499806, + "urgent": 0.2889994178533793 + }, + "needs_human": { + "yes": 0.563724071157929, + "no": 0.436275928842071 + } + } + }, + { + "max_len": null, + "row_length": 115, + "row_sha256": "cbc1dfa364f50f8f00ff04a455e91e149fa1fbca6fd5e49de9c5cc370a517223", + "probabilities": { + "intent": { + "refund_request": 0.007306926753793017, + "cancel_order": 0.97607903480116, + "room_change": 0.0010640839719275488, + "maintenance": 0.005471416745275971, + "other": 0.01007853772784352 + }, + "priority": { + "low": 0.14616614050872026, + "normal": 0.17338279070055723, + "high": 0.3309454389654734, + "urgent": 0.34950562982524913 + }, + "needs_human": { + "yes": 0.7350032631246142, + "no": 0.2649967368753859 + } + } + }, + { + "max_len": null, + "row_length": 102, + "row_sha256": "92872f4675af6101236de0cc43f82b18aa26deca7ebb6acd8f223cde93193fd3", + "probabilities": { + "intent": { + "refund_request": 0.004258707140121063, + "cancel_order": 0.9921012540057011, + "room_change": 0.00017867828707934327, + "maintenance": 0.0003654525245295128, + "other": 0.0030959080425688795 + }, + "priority": { + "low": 0.2552381896105709, + "normal": 0.2844838899978587, + "high": 0.23983462222405222, + "urgent": 0.22044329816751823 + }, + "needs_human": { + "yes": 0.5460669637573503, + "no": 0.45393303624264975 + } + } + }, + { + "max_len": null, + "row_length": 111, + "row_sha256": "b097ed6172b8be1913758b32312bf2ca229a90d73dbf996c2be215cbcb2bf7db", + "probabilities": { + "intent": { + "refund_request": 0.06827418052943238, + "cancel_order": 0.06822474637756669, + "room_change": 0.07629117247267378, + "maintenance": 0.09473030082254938, + "other": 0.6924795997977778 + }, + "priority": { + "low": 0.264424202772207, + "normal": 0.3183966632491285, + "high": 0.2334700147435491, + "urgent": 0.18370911923511532 + }, + "needs_human": { + "yes": 0.3931246565934047, + "no": 0.6068753434065953 + } + } + } + ] + }, + { + "name": "label_groups_multi_label", + "request": { + "options": { + "label_groups": { + "topics": [ + "billing", + "hvac", + "shipping", + "payroll", + "account" + ], + "channel": [ + "email", + "chat", + "phone" + ] + }, + "classification_type": "multi-label" + } + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "topics", + "kind": "multi", + "labels": [ + "billing", + "hvac", + "shipping", + "payroll", + "account" + ], + "instruction": null + }, + { + "name": "channel", + "kind": "multi", + "labels": [ + "email", + "chat", + "phone" + ], + "instruction": null + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 72, + "row_sha256": "92d54c4c5ac9ae763e2ffc5570c840e34465ca981233a13073f92d84ebceaf6d", + "probabilities": { + "topics": { + "billing": 0.5509768171557046, + "hvac": 0.07499519564837895, + "shipping": 0.08699790332555553, + "payroll": 0.11328731372337252, + "account": 0.3470624839410997 + }, + "channel": { + "email": 0.3987747473918513, + "chat": 0.3347985877362211, + "phone": 0.275626540050097 + } + } + }, + { + "max_len": null, + "row_length": 81, + "row_sha256": "1511e4f3a3793f110ba274dc310069c470f6abc4258e2966af848bac64d2acd4", + "probabilities": { + "topics": { + "billing": 0.07819253958706025, + "hvac": 0.9805449305318654, + "shipping": 0.03347003740569826, + "payroll": 0.0408345366688793, + "account": 0.11962147215203459 + }, + "channel": { + "email": 0.3606983606307541, + "chat": 0.4068665501006504, + "phone": 0.3268092842455029 + } + } + }, + { + "max_len": null, + "row_length": 75, + "row_sha256": "f989cedda054fd6185e2c5d136f49ee1a9a680df9d57cad8c5dab543fc0eb284", + "probabilities": { + "topics": { + "billing": 0.15392406419471802, + "hvac": 0.03414020747831923, + "shipping": 0.8269061050822606, + "payroll": 0.05657939544203529, + "account": 0.1321888298186983 + }, + "channel": { + "email": 0.4712439390275077, + "chat": 0.31231505739645, + "phone": 0.27883299564587416 + } + } + }, + { + "max_len": null, + "row_length": 62, + "row_sha256": "bf03e67374a61279d69ed8790d8e04201bec15bc5a27e8a6011b7892bec84918", + "probabilities": { + "topics": { + "billing": 0.3442248473191424, + "hvac": 0.06183041387514028, + "shipping": 0.11847002656640461, + "payroll": 0.11350796002646542, + "account": 0.37715684724456217 + }, + "channel": { + "email": 0.39040485924501206, + "chat": 0.3058516407099851, + "phone": 0.27263776543028034 + } + } + }, + { + "max_len": null, + "row_length": 71, + "row_sha256": "9e0f7393d1b49c38f5b17b2fef4273dba68f0013fbf970189c2947e976480cb7", + "probabilities": { + "topics": { + "billing": 0.2174290483561565, + "hvac": 0.1341628062406503, + "shipping": 0.1953933412460402, + "payroll": 0.17123657217548122, + "account": 0.3065704799775977 + }, + "channel": { + "email": 0.39398790070918305, + "chat": 0.3866055564111748, + "phone": 0.253894336638229 + } + } + } + ] + }, + { + "name": "labels", + "request": { + "labels": [ + "billing", + "technical", + "hospitality", + "logistics", + "other" + ], + "instruction": "Which team should handle this?" + }, + "documents": "texts", + "reference_call": { + "tasks": [ + { + "name": "label", + "kind": "single", + "labels": [ + "billing", + "technical", + "hospitality", + "logistics", + "other" + ], + "instruction": "Which team should handle this?" + } + ] + }, + "results": [ + { + "max_len": null, + "row_length": 65, + "row_sha256": "fbc9c1cb8f4aabdcc007f3dfc0b2909853393e4ab83e6d0149de6c7e7325fafc", + "probabilities": { + "label": { + "billing": 0.7527314858653925, + "technical": 0.05528599264429413, + "hospitality": 0.02421007214910886, + "logistics": 0.049428431209613224, + "other": 0.11834401813159132 + } + } + }, + { + "max_len": null, + "row_length": 74, + "row_sha256": "dead6a01dfc53a5df07c0942fba8eb9eee02e9e030b36e026f64db85885a48b7", + "probabilities": { + "label": { + "billing": 0.032633745329729194, + "technical": 0.273623599144904, + "hospitality": 0.23137119907614176, + "logistics": 0.2570411402470624, + "other": 0.20533031620216266 + } + } + }, + { + "max_len": null, + "row_length": 68, + "row_sha256": "e892b2e197c71e238856ab1e55dd35e4e4591ea35ce837bb3b3afad3d0cb9c18", + "probabilities": { + "label": { + "billing": 0.14592706724966928, + "technical": 0.04231747595556333, + "hospitality": 0.06710331778655262, + "logistics": 0.506304832285875, + "other": 0.23834730672233984 + } + } + }, + { + "max_len": null, + "row_length": 55, + "row_sha256": "7609c07dd66570f5d0caecce204d9ba33948a38045e3b4ea902294a04d5c31f5", + "probabilities": { + "label": { + "billing": 0.36843164901097974, + "technical": 0.0768108273785425, + "hospitality": 0.0523916241851089, + "logistics": 0.15808914013863787, + "other": 0.34427675928673107 + } + } + }, + { + "max_len": null, + "row_length": 64, + "row_sha256": "32d4854b8380ce1dbff93dfd6da5d665d6ff4a3c84d38c92e84f343e2ff4ee22", + "probabilities": { + "label": { + "billing": 0.05232635251077149, + "technical": 0.07854168291958191, + "hospitality": 0.06159038105904645, + "logistics": 0.0684019802532734, + "other": 0.7391396032573269 + } + } + } + ] + }, + { + "name": "long_document", + "request": { + "output_schema": { + "intent": { + "type": "choice", + "instructions": "What does the customer want?", + "criteria": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": "" + } + }, + "urgency": { + "type": "score", + "instructions": "How urgent is this request?", + "criteria": [ + "not urgent", + "within a week", + "today", + "blocking or past a deadline" + ] + }, + "needs_human": { + "type": "noul", + "instructions": "Must a person act on this?", + "criteria": { + "true": "automation cannot resolve it" + } + }, + "rating": { + "type": "score", + "instructions": "Rate the severity from 0 to 10.", + "criteria": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ] + } + } + }, + "documents": "long", + "reference_call": { + "tasks": [ + { + "name": "intent", + "kind": "single", + "labels": { + "refund_request": "the customer wants money back", + "cancel_order": "the customer wants to cancel an order or subscription", + "room_change": "a guest wants another room", + "maintenance": null, + "other": null + }, + "instruction": "What does the customer want?" + }, + { + "name": "urgency", + "kind": "single", + "labels": { + "0": "not urgent", + "1": "within a week", + "2": "today", + "3": "blocking or past a deadline" + }, + "instruction": "How urgent is this request?" + }, + { + "name": "needs_human", + "kind": "single", + "labels": { + "yes": "automation cannot resolve it", + "no": null + }, + "instruction": "Must a person act on this?" + }, + { + "name": "rating", + "kind": "single", + "labels": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + "10" + ], + "instruction": "Rate the severity from 0 to 10." + } + ] + }, + "results": [ + { + "max_len": 1218, + "row_length": 2047, + "row_sha256": "c8f870c279becee339680e16b3092fb008c94047609bfb1c836b95f8e3972a27", + "probabilities": { + "intent": { + "refund_request": 0.22137453640299068, + "cancel_order": 0.13554206829667656, + "room_change": 0.06737549277112384, + "maintenance": 0.1203290706636644, + "other": 0.4553788318655445 + }, + "urgency": { + "0": 0.2876334328606865, + "1": 0.23819383547916143, + "2": 0.25049020716699416, + "3": 0.22368252449315795 + }, + "needs_human": { + "yes": 0.3122318917744955, + "no": 0.6877681082255046 + }, + "rating": { + "0": 0.07948976483071504, + "1": 0.0747515657317049, + "2": 0.07955062351211512, + "3": 0.0850301649087393, + "4": 0.08933833893246318, + "5": 0.09274387834160848, + "6": 0.09014150542966329, + "7": 0.08957841282184897, + "8": 0.0841316921623689, + "9": 0.08915373182557224, + "10": 0.14609032150320053 + } + } + } + ] + } + ] +} diff --git a/packages/sie_server/tests/adapters/test_gliner2.py b/packages/sie_server/tests/adapters/test_gliner2.py index 13294e191..06cc83c14 100644 --- a/packages/sie_server/tests/adapters/test_gliner2.py +++ b/packages/sie_server/tests/adapters/test_gliner2.py @@ -199,7 +199,8 @@ def test_transformers5_bundle_carries_gliner2_classification_runtime() -> None: assert "sie_server.adapters.gliner2.classification" not in default["adapters"] assert "sie_server.adapters.gliner2.classification" in transformers5["adapters"] assert "sie_server.adapters.gliner2.adapter" not in transformers5["adapters"] - assert transformers5["deps"]["gliner2"] == ">=1.3.1,<2" + # The GLiNER2.5-Decide models share this bundle and need gliner2 2.x. + assert transformers5["deps"]["gliner2"] == "==2.0.0" profile = model["profiles"]["default"] assert profile["adapter_path"].endswith("gliner2.classification:GLiNER2ClassificationAdapter") assert profile["adapter_options"] == { diff --git a/packages/sie_server/tests/adapters/test_gliner2_decide.py b/packages/sie_server/tests/adapters/test_gliner2_decide.py new file mode 100644 index 000000000..bcd73b30d --- /dev/null +++ b/packages/sie_server/tests/adapters/test_gliner2_decide.py @@ -0,0 +1,895 @@ +"""GLiNER2.5-Decide adapter: request contract, bounds, errors, metering, and the ModernBERT RoPE check. + +The adapter's model and processor are replaced by fakes that follow gliner2 +2.0's prompt layout; the real processor and weights are covered by the +``model``-marked parity tests in ``test_gliner2_decide_parity.py``. +""" + +from __future__ import annotations + +import json +import math +import re +import time +from collections.abc import Iterator +from functools import lru_cache +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import numpy as np +import pytest +import torch +import transformers +import yaml +from sie_server.adapters.errors import InputTooLongError +from sie_server.adapters.gliner2 import decide as decide_module +from sie_server.adapters.gliner2.decide import ( + GLiNER2DecideAdapter, + declared_rope_thetas, + loadable_checkpoint, + loaded_rope_thetas, + transformers4_encoder_config, + transformers4_tokenizer_config, + verify_encoder_rope, +) +from sie_server.adapters.gliner2.decisions import ( + MAX_LABELS, + MAX_LABELS_PER_TASK, + MAX_SCHEMA_CHARS, + MAX_TASKS, + answer, + parse_request, + probabilities, +) +from sie_server.adapters.gliner2.words import PACKAGE_PATTERN, LinearWordSplitter +from sie_server.types.inputs import InvalidInputError, Item + +SIE_SERVER = Path(__file__).resolve().parents[2] +SPECIAL = ("[SEP_STRUCT]", "[SEP_TEXT]", "[P]", "[C]", "[E]", "[R]", "[L]", "[EXAMPLE]", "[OUTPUT]", "[DESCRIPTION]") + +QUESTIONS = { + "intent": { + "type": "choice", + "instructions": "What does the customer want?", + "criteria": {"refund": "wants money back", "cancel": None, "other": ""}, + }, + "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["low", "medium", "high"]}, + "needs_human": {"type": "noul", "instructions": "Must a person act?", "criteria": {"true": "escalate"}}, +} + + +# --------------------------------------------------------------------------- +# Fakes following gliner2 2.0's processor layout +# --------------------------------------------------------------------------- + + +class FakeTokenizer: + """Pieces of at most three characters (markers stay whole); ids are stable per piece.""" + + def __init__(self) -> None: + self.vocab: dict[str, int] = {token: 1000 + i for i, token in enumerate(SPECIAL)} + self.tokenized_chars = 0 + + def tokenize(self, text: str) -> list[str]: + self.tokenized_chars += len(text) + pieces: list[str] = [] + for part in re.split(r"(\[[A-Z_]+\])", text): + if part in self.vocab and part in SPECIAL: + pieces.append(part) + else: + pieces.extend(part[i : i + 3] for i in range(0, len(part), 3) if part[i : i + 3].strip()) + return pieces + + def convert_tokens_to_ids(self, tokens: list[str]) -> list[int]: + return [self.vocab.setdefault(token, 2000 + len(self.vocab)) for token in tokens] + + +class WhitespaceTokenSplitter: + """gliner2 2.0.0's word splitter (the package's regex, each word lowercased): the reference rows use it.""" + + _PATTERN = PACKAGE_PATTERN + + def __call__(self, text: str, lower: bool = True) -> Iterator[tuple[str, int, int]]: + for match in self._PATTERN.finditer(text): + word = match.group() + yield (word.lower() if lower else word), match.start(), match.end() + + +class FakeProcessor: + """``transform_and_format`` and ``collate`` (one row) as gliner2 2.0's ``SchemaTransformer`` builds them.""" + + def __init__(self) -> None: + self.tokenizer = FakeTokenizer() + self.word_splitter = WhitespaceTokenSplitter() + self.reference_splitter = WhitespaceTokenSplitter() + self._tokenize_cached = lru_cache(maxsize=100)(self.tokenizer.tokenize) + + def change_mode(self, is_training: bool) -> None: + assert not is_training + + @staticmethod + def _schemas(schema: dict[str, Any]) -> list[list[str]]: + schemas = [] + for entry in schema["classifications"]: + prompt = entry["task"] + (f": {entry['prompt']}" if entry.get("prompt") else "") + for label, description in entry.get("label_descriptions", {}).items(): + prompt += f" [DESCRIPTION] {label}: {description}" + labels = [token for label in entry["labels"] for token in ("[L]", label)] + schemas.append(["(", "[P]", prompt, "(", *labels, ")", ")"]) + return schemas + + def transform_and_format(self, text: str, schema: dict[str, Any]) -> SimpleNamespace: + words = [word for word, _, _ in self.reference_splitter(text, lower=True)] + return self._format(self._schemas(schema), words) + + def collate_row(self, text: str, schema: dict[str, Any], max_len: int | None) -> list[int]: + """``collate_fn_inference([(text, schema)], max_len)``'s input ids.""" + if not text.endswith((".", "!", "?")): + text += "." + words = [word for word, _, _ in self.reference_splitter(text, lower=True)] + if max_len is not None: + words = words[:max_len] + return self._format(self._schemas(schema), words).input_ids + + def _format(self, schemas: list[list[str]], words: list[str]) -> SimpleNamespace: + combined: list[str] = [] + for struct in schemas: + combined.extend([*struct, "[SEP_STRUCT]"]) + combined.pop() + combined.append("[SEP_TEXT]") + combined.extend(words) + markers: set[int] = set() + offset = 0 + for struct in schemas: + markers.add(offset + 1) + markers.update(offset + index for index in range(4, len(struct) - 2, 2)) + offset += len(struct) + 1 + subwords: list[str] = [] + positions: list[list[int]] = [[] for _ in schemas] + first_positions: list[int] = [] + schema_index, in_text = 0, False + for index, element in enumerate(combined): + position = len(subwords) + subwords.extend(self._tokenize_cached(element)) + if element == "[SEP_TEXT]": + in_text = True + elif in_text: + first_positions.append(position) + elif element == "[SEP_STRUCT]": + schema_index += 1 + elif index in markers: + positions[schema_index].append(position) + return SimpleNamespace( + input_ids=self.tokenizer.convert_tokens_to_ids(subwords), + text_word_first_positions=first_positions, + schema_special_positions=positions, + ) + + +class FakeEncoder(torch.nn.Module): + """Hidden state = (position, sum of the row's ids): label logits depend on marker and row.""" + + def forward(self, input_ids: torch.Tensor, attention_mask: torch.Tensor) -> SimpleNamespace: + position = torch.arange(input_ids.shape[1]).expand_as(input_ids).float() + total = (input_ids * attention_mask).sum(dim=1, keepdim=True).float().expand_as(position) + return SimpleNamespace(last_hidden_state=torch.stack([position, total], dim=-1)) + + +def fake_classifier(hidden: torch.Tensor) -> torch.Tensor: + return hidden[..., :1] * 0.01 + hidden[..., 1:2] * 1e-7 + + +def expected_logits(row: list[int], positions: list[int]) -> list[float]: + return [position * 0.01 + sum(row) * 1e-7 for position in positions] + + +def make_adapter(window: int = 256, **kwargs: Any) -> tuple[GLiNER2DecideAdapter, FakeProcessor]: + adapter = GLiNER2DecideAdapter("fake/decide", max_seq_length=window, **kwargs) + processor = FakeProcessor() + adapter._attach(SimpleNamespace(encoder=FakeEncoder(), classifier=fake_classifier), processor, "cpu") + return adapter, processor + + +def spy_rows(adapter: GLiNER2DecideAdapter, monkeypatch: pytest.MonkeyPatch) -> list[list[int]]: + """Record the rows the adapter scores.""" + seen: list[list[int]] = [] + original = adapter._score + + def score(rows: list[list[int]], positions: list[int]) -> np.ndarray: + seen.extend(rows) + return original(rows, positions) + + monkeypatch.setattr(adapter, "_score", score) + return seen + + +# --------------------------------------------------------------------------- +# Request contract +# --------------------------------------------------------------------------- + + +def test_laya_questions_map_to_decide_tasks() -> None: + request = parse_request(labels=None, output_schema=QUESTIONS, instruction=None, options={}) + + intent, urgency, noul = request.tasks + assert request.mode == "questions" + assert intent.model_entry() == { + "task": "intent", + "labels": ["refund", "cancel", "other"], + "true_label": ["N/A"], + "multi_label": False, + "cls_threshold": 0.5, + "class_act": "auto", + "prompt": "What does the customer want?", + "label_descriptions": {"refund": "wants money back"}, + } + assert urgency.labels == ("0", "1", "2") + assert urgency.model_entry()["label_descriptions"] == {"0": "low", "1": "medium", "2": "high"} + assert noul.labels == ("yes", "no") + assert noul.model_entry()["label_descriptions"] == {"yes": "escalate"} + assert request.model_schema()["classifications"] == [task.model_entry() for task in request.tasks] + + +def test_noul_labels_wording_and_numeric_score_levels() -> None: + request = parse_request( + labels=None, + output_schema={ + "handoff": {"type": "noul", "instructions": "Hand off?", "labels": {"false": "keep", "true": "handoff"}}, + "rating": {"type": "score", "instructions": "Rate it", "criteria": [str(i) for i in range(11)]}, + "tone": {"type": "score", "instructions": "", "criteria": [{"calm": True}, "angry"]}, + }, + instruction=None, + options={}, + ) + handoff, rating, tone = request.tasks + assert handoff.labels == ("handoff", "keep") + assert "label_descriptions" not in rating.model_entry() # levels that repeat their index + assert tone.prompt is None + assert tone.descriptions == ('{"calm": true}', "angry") # structured criteria render as Laya renders them + + +def test_label_groups_and_labels_modes() -> None: + groups = parse_request( + labels=None, + output_schema=None, + instruction="Triage the ticket.", + options={"label_groups": {"topic": ["billing", "bug"], "urgency": [" low ", "high"]}}, + ) + assert groups.mode == "groups" + assert [task.prompt for task in groups.tasks] == ["Triage the ticket.", "Triage the ticket."] + assert groups.tasks[1].labels == ("low", "high") # the model reads stripped labels... + assert groups.tasks[1].keys == (" low ", "high") # ...answers echo the caller's + + labels = parse_request( + labels=["spam", "ham"], + output_schema=None, + instruction=None, + options={"classification_type": "multi-label", "classification_task": "verdict"}, + ) + assert labels.mode == "labels" + assert labels.tasks[0].name == "verdict" + assert labels.tasks[0].multi_label + default = parse_request(labels=["a", "b"], output_schema=None, instruction=None, options={}) + assert default.tasks[0].name == "label" + assert default.tasks[0].prompt is None + + +@pytest.mark.parametrize( + ("kwargs", "message"), + [ + ({"labels": ["a"], "output_schema": QUESTIONS}, "either output_schema"), + ({"output_schema": QUESTIONS, "options": {"label_groups": {"g": ["a"]}}}, "either output_schema"), + ({"output_schema": QUESTIONS, "instruction": "x"}, "own 'instructions'"), + ({"labels": ["a"], "options": {"label_groups": {"g": ["a"]}}}, "either labels or options.label_groups"), + ({}, "requires typed questions"), + ({"output_schema": {}}, "requires typed questions"), + ({"output_schema": ["x"]}, "must map question ids"), + ({"output_schema": {"q": {"type": "maybe", "instructions": "x"}}}, "unknown type"), + ({"output_schema": {"q": {"type": "choice", "instructions": "x", "criteria": []}}}, "at least one criterion"), + ({"output_schema": {"q": {"type": "choice", "instructions": "x", "criteria": ["a", " a"]}}}, "unique"), + ({"output_schema": {"q": {"type": "choice", "instructions": "x", "criteria": ["a", " "]}}}, "non-empty"), + ({"output_schema": {" ": {"type": "noul", "instructions": "x"}}}, "question id must be a non-empty"), + ( + { + "output_schema": { + "a": {"type": "noul", "instructions": "x"}, + " a": {"type": "noul", "instructions": "y"}, + } + }, + "must be unique", + ), + ({"labels": ["a", "b"], "options": {"group_encoding": "separate"}}, "group_encoding"), + ({"labels": ["a", "b"], "options": {"examples": [{"text": "x", "labels": ["a"]}]}}, "few-shot"), + ({"labels": ["a", "b"], "options": {"threshold": 1.5}}, "threshold"), + ({"labels": ["a", "b"], "options": {"threshold": True}}, "threshold"), + ({"labels": ["a", "b"], "options": {"classification_type": "multi"}}, "classification_type"), + ( + {"labels": ["a", "b"], "options": {"classification_type": "multi-label", "multi_label": False}}, + "contradicts", + ), + ({"labels": "a"}, "must be a list"), + ({"labels": ["a", 3]}, "non-empty string"), + ({"options": {"label_groups": {"g": []}}}, "non-empty list"), + ({"options": {"label_groups": []}}, "non-empty object"), + ({"labels": ["a"], "options": {"classification_task": " "}}, "classification_task"), + ], +) +def test_malformed_requests_are_invalid_input(kwargs: dict[str, Any], message: str) -> None: + arguments = {"labels": None, "output_schema": None, "instruction": None, "options": {}, **kwargs} + with pytest.raises(InvalidInputError, match=re.escape(message)): + parse_request(**arguments) + + +@pytest.mark.parametrize( + "kwargs", + [ + {"labels": ["a", "b[L]"]}, + {"labels": ["a", "b"], "instruction": "Pick [DESCRIPTION] one"}, + {"output_schema": {"q[P]": {"type": "noul", "instructions": "x"}}}, + {"output_schema": {"q": {"type": "choice", "instructions": "x", "criteria": {"a": "[SEP_TEXT] b"}}}}, + {"options": {"label_groups": {"g[E]": ["a", "b"]}}}, + ], +) +def test_prompt_markers_are_refused(kwargs: dict[str, Any]) -> None: + arguments = {"labels": None, "output_schema": None, "instruction": None, "options": {}, **kwargs} + with pytest.raises(InvalidInputError, match="structural token"): + parse_request(**arguments) + + +def test_bounds_are_checked_before_tokenizing() -> None: + def parse(**kwargs: Any) -> None: + parse_request(**{"labels": None, "output_schema": None, "instruction": None, "options": {}, **kwargs}) + + many = {f"q{i}": {"type": "noul", "instructions": "x"} for i in range(MAX_TASKS + 1)} + with pytest.raises(InvalidInputError, match=f"at most {MAX_TASKS} questions"): + parse(output_schema=many) + wide = {"q": {"type": "choice", "instructions": "x", "criteria": [f"l{i}" for i in range(MAX_LABELS_PER_TASK + 1)]}} + with pytest.raises(InvalidInputError, match=f"at most {MAX_LABELS_PER_TASK} options"): + parse(output_schema=wide) + with pytest.raises(InvalidInputError, match=f"at most {MAX_LABELS_PER_TASK} labels"): + parse(labels=[f"l{i}" for i in range(MAX_LABELS_PER_TASK + 1)]) + groups = {f"g{i}": [f"l{j}" for j in range(40)] for i in range(MAX_LABELS // 40 + 1)} + with pytest.raises(InvalidInputError, match=f"at most {MAX_LABELS} labels"): + parse(options={"label_groups": groups}) + with pytest.raises(InvalidInputError, match="at most 256 characters"): + parse(labels=["a", "b" * 257]) + with pytest.raises(InvalidInputError, match="at most 2048 characters"): + parse(labels=["a", "b"], instruction="x" * 2049) + long_criteria = { + f"q{i}": {"type": "choice", "instructions": "x" * 2000, "criteria": {"a": "d" * 2000}} for i in range(17) + } + with pytest.raises(InvalidInputError, match=f"total at most {MAX_SCHEMA_CHARS}"): + parse(output_schema=long_criteria) + + +# --------------------------------------------------------------------------- +# Answers +# --------------------------------------------------------------------------- + + +def test_answers_have_every_probability_and_laya_confidence() -> None: + request = parse_request(labels=None, output_schema=QUESTIONS, instruction=None, options={}) + intent, urgency, noul = request.tasks + + p = probabilities(intent, np.array([2.0, 1.0, 0.0], dtype=np.float32)) + choice = answer(intent, p) + assert choice["choice"] == "refund" + assert list(choice["probabilities"]) == ["refund", "cancel", "other"] + assert sum(choice["probabilities"].values()) == pytest.approx(1.0) + entropy = -sum(v * math.log(v) for v in p) + assert choice["confidence"] == pytest.approx(1 - entropy / math.log(3)) + + score = answer(urgency, probabilities(urgency, np.array([0.0, 0.0, 0.0]))) + assert score["score"] == pytest.approx(1.0) + assert score["legend"] == {"0": "low", "1": "medium", "2": "high"} + assert score["probabilities"] == pytest.approx({"0": 1 / 3, "1": 1 / 3, "2": 1 / 3}) + assert score["confidence"] == pytest.approx(0.0) + + verdict = answer(noul, probabilities(noul, np.array([0.0, math.log(3.0)]))) # labels ("yes", "no") + assert verdict == {"type": "noul", "noul": pytest.approx(0.25), "answer": False, "confidence": pytest.approx(0.75)} + + +def test_multi_label_groups_use_sigmoid_and_threshold() -> None: + request = parse_request( + labels=None, + output_schema=None, + instruction=None, + options={"label_groups": {"topics": ["hvac", "billing"]}, "classification_type": "multi-label"}, + ) + task = request.tasks[0] + p = probabilities(task, np.array([2.0, -1.0])) + assert p.tolist() == pytest.approx([1 / (1 + math.exp(-2.0)), 1 / (1 + math.exp(1.0))]) + assert answer(task, p) == {"labels": ["hvac"], "probabilities": {"hvac": p[0], "billing": p[1]}} + assert answer(task, p, threshold=0.2)["labels"] == ["hvac", "billing"] + + +# --------------------------------------------------------------------------- +# Adapter: rows, outputs, metering, errors +# --------------------------------------------------------------------------- + + +def test_rows_equal_the_processor_rows_and_logits_come_from_label_markers(monkeypatch: pytest.MonkeyPatch) -> None: + adapter, processor = make_adapter() + rows = spy_rows(adapter, monkeypatch) + texts = ["Hi, we were billed twice for March. Refund please", "Login broken since update!"] + + output = adapter.extract([Item(text=text) for text in texts], output_schema=QUESTIONS) + + request = parse_request(labels=None, output_schema=QUESTIONS, instruction=None, options={}) + schema = request.model_schema() + assert rows == [processor.collate_row(text, schema, None) for text in texts] + prefix = processor.transform_and_format(".", schema) + positions = [p for task in prefix.schema_special_positions for p in task[1:]] + for row, data in zip(rows, output.data or [], strict=True): + logits = expected_logits(row, positions) + p = probabilities(request.tasks[0], np.array(logits[:3], dtype=np.float32)) + assert [data["intent"]["probabilities"][key] for key in ("refund", "cancel", "other")] == pytest.approx(p) + assert output.classifications is None + assert output.errors is None + assert list((output.data or [{}])[0]) == ["intent", "urgency", "needs_human"] + + +def test_metering_bills_document_and_free_text_not_labels(monkeypatch: pytest.MonkeyPatch) -> None: + adapter, processor = make_adapter() + tokenizer = processor.tokenizer + text = "the payroll file is late" + + output = adapter.extract([Item(text=text), Item(text=None)], output_schema=QUESTIONS) + + document = sum(len(tokenizer.tokenize(word)) for word, _, _ in WhitespaceTokenSplitter()(text)) + free_text = sum( + len(tokenizer.tokenize(value)) + for value in ( + "What does the customer want?", + "wants money back", + "How urgent is this?", + "low", + "medium", + "high", + "Must a person act?", + "escalate", + ) + ) + assert output.input_token_counts == [document + free_text, 0] # the processor's appended "." is not billed + assert output.errors is not None + assert output.errors[1] is not None + assert output.errors[1].code == "INVALID_INPUT" + assert output.data is not None + assert output.data[1] == {} + + labels_only = adapter.extract([Item(text=text)], labels=["a very long label name", "b"]) + assert labels_only.input_token_counts == [document] + + +def test_long_documents_are_cut_to_whole_words_in_the_window(monkeypatch: pytest.MonkeyPatch) -> None: + adapter, processor = make_adapter(window=96) + rows = spy_rows(adapter, monkeypatch) + text = " ".join(f"word{i:04d}" for i in range(500)) # 3 pieces per word + + output = adapter.extract([Item(text=text)], labels=["yes", "no"]) + + schema = parse_request(labels=["yes", "no"], output_schema=None, instruction=None, options={}).model_schema() + prefix_len = processor.transform_and_format(".", schema).text_word_first_positions[0] + kept = (96 - prefix_len) // 3 + assert rows == [processor.collate_row(text, schema, kept)] + assert len(rows[0]) <= 96 + assert output.input_token_counts == [3 * kept] + # Work stops at the window: only the kept words and the one that did not fit were tokenized. + assert processor.tokenizer.tokenized_chars < 20 * 96 + + +def test_caches_keep_neither_long_words_nor_task_prompts() -> None: + adapter, processor = make_adapter(window=256) + long_words = [f"{i:04d}" + "x" * 60 for i in range(20)] + + output = adapter.extract([Item(text=" ".join(["short", "words", *long_words]))], labels=["yes", "no"]) + + assert output.input_token_counts is not None + assert output.input_token_counts[0] > 100 # long words were read... + assert adapter._word_cache is not None + assert adapter._word_cache.cache_info().currsize == 2 # ...but only "short" and "words" were kept + assert processor._tokenize_cached.cache_info().currsize == 0 + + +def test_overflow_policy_error_and_unreadable_items_fail_alone() -> None: + adapter, _ = make_adapter(window=64) + items = [ + Item(text="short text"), + Item(text=" ".join(["word"] * 200)), + Item(text="x" * 5000), # one word longer than the adapter reads + Item(text=" "), + Item(text="a", metadata={"state": "b"}), + Item(metadata={"state": 3}), + ] + + output = adapter.extract(items, labels=["yes", "no"], options={"overflow_policy": "error"}) + + codes = [None if error is None else error.code for error in output.errors or []] + assert codes == [None, "INPUT_TOO_LONG", "INPUT_TOO_LONG", "INVALID_INPUT", "INVALID_INPUT", "INVALID_INPUT"] + assert output.input_token_counts is not None + assert output.input_token_counts[0] > 0 + assert output.input_token_counts[1:] == [0, 0, 0, 0, 0] + assert output.classifications is not None + assert output.classifications[0] + assert not any(output.classifications[1:]) + + truncated = adapter.extract(items[:2], labels=["yes", "no"], options={"overflow_policy": "truncate_text"}) + assert truncated.errors is None + + +def test_a_document_missing_only_the_processors_period_counts_as_whole() -> None: + adapter, processor = make_adapter(window=64) + schema = parse_request(labels=["yes", "no"], output_schema=None, instruction=None, options={}).model_schema() + room = 64 - processor.transform_and_format(".", schema).text_word_first_positions[0] + text = " ".join(["abc"] * room) # one token per word: the words fill the room exactly + + output = adapter.extract([Item(text=text)], labels=["yes", "no"], options={"overflow_policy": "error"}) + + assert output.errors is None + assert output.input_token_counts == [room] + longer = adapter.extract([Item(text=text + " abc")], labels=["yes", "no"], options={"overflow_policy": "error"}) + assert longer.errors is not None + assert longer.errors[0] is not None + assert longer.errors[0].code == "INPUT_TOO_LONG" + + +def test_states_render_like_laya_and_conversations_keep_the_newest_turns(monkeypatch: pytest.MonkeyPatch) -> None: + adapter, processor = make_adapter(window=64) + rows = spy_rows(adapter, monkeypatch) + turns = [{"role": "user", "content": f"turn {i} " + "blah " * 10} for i in range(30)] + + output = adapter.extract( + [Item(metadata={"state": {"subject": "Login", "body": "Error 500"}}), Item(metadata={"state": turns})], + labels=["yes", "no"], + ) + + assert output.errors is None + schema = parse_request(labels=["yes", "no"], output_schema=None, instruction=None, options={}).model_schema() + assert rows[0] == processor.collate_row(json.dumps({"subject": "Login", "body": "Error 500"}), schema, None) + inverse = {token_id: token for token, token_id in processor.tokenizer.vocab.items()} + prefix_len = processor.transform_and_format(".", schema).text_word_first_positions[0] + document = [inverse[token_id] for token_id in rows[1][prefix_len:]] + assert "29" in document # the newest turn is read... + assert "0" not in document # ...the oldest is not + assert document[-1] == "." + assert len(rows[1]) <= 64 + + +def test_the_adapter_splits_words_in_linear_time_like_the_package() -> None: + adapter, processor = make_adapter() + assert isinstance(processor.word_splitter, LinearWordSplitter) + assert adapter._word_splitter is processor.word_splitter + goldens = sorted((Path(__file__).parent / "goldens" / "gliner2_decide").glob("*.json")) + assert goldens + texts = [ + text + for path in goldens + for text in (*json.loads(path.read_text())["texts"], json.loads(path.read_text())["long_text"]) + ] + for text in texts: + assert list(adapter._word_splitter(text)) == list(processor.reference_splitter(text)) + + +def test_an_unknown_word_splitter_is_refused() -> None: + adapter = GLiNER2DecideAdapter("fake/decide", max_seq_length=256) + processor = FakeProcessor() + processor.word_splitter = lambda text, lower=True: iter(()) + with pytest.raises(RuntimeError, match="linear-time equivalent"): + adapter._attach(SimpleNamespace(encoder=FakeEncoder(), classifier=fake_classifier), processor, "cpu") + + +@pytest.mark.parametrize( + "item", + [ + Item(text="." * (2 * 1024 * 1024)), + Item(text="a." * (1024 * 1024)), + Item(text="hello world. " * (2 * 1024 * 1024 // 13)), + Item(metadata={"state": ["." * (64 * 1024)]}), + Item(metadata={"state": [{"role": "user", "content": "a." * (32 * 1024)}] * 3}), + ], + ids=["dots-2MiB", "a-dots-2MiB", "prose-2MiB", "dots-state-64KiB", "a-dots-turns"], +) +def test_pathological_documents_are_read_in_bounded_time(item: Item) -> None: + adapter, _ = make_adapter(window=2048) + adapter.extract([Item(text="warm up")], labels=["yes", "no"]) + started = time.perf_counter() + output = adapter.extract([item], labels=["yes", "no"]) + elapsed = time.perf_counter() - started + assert output.errors is None + assert output.input_token_counts is not None + assert 0 < output.input_token_counts[0] <= 2048 + # gliner2's own splitter takes 15 to 30 seconds on these; the adapter takes tens of milliseconds. + assert elapsed < 1.0 + + +def test_reading_from_the_end_keeps_the_tail_of_an_overlong_run(monkeypatch: pytest.MonkeyPatch) -> None: + adapter, processor = make_adapter(window=64) + rows = spy_rows(adapter, monkeypatch) + turns = ["older turn " * 20, "x" + "." * 10_000 + "tail"] + + output = adapter.extract([Item(metadata={"state": turns})], labels=["yes", "no"]) + + assert output.errors is None + inverse = {token_id: token for token, token_id in processor.tokenizer.vocab.items()} + prefix_len = processor.transform_and_format( + ".", parse_request(labels=["yes", "no"], output_schema=None, instruction=None, options={}).model_schema() + ).text_word_first_positions[0] + document = [inverse[token_id] for token_id in rows[0][prefix_len:]] + # The newest turn's run ('"x....tail"]' and the processor's ".") is longer than the + # adapter reads, so only its tail is read, and nothing of the older turn. + assert document[-5:] == ["tai", "l", '"', "]", "."] + assert set(document[:-5]) == {"."} + assert "old" not in document + + +def test_a_conversation_cut_inside_an_overlong_run_is_not_whole() -> None: + adapter, _ = make_adapter(window=2048) + item = Item(metadata={"state": ["older turn", "x" * 5000]}) # the newest run's tail fits in the room + + read = adapter.extract([item], labels=["yes", "no"]) + strict = adapter.extract([item], labels=["yes", "no"], options={"overflow_policy": "error"}) + + assert read.errors is None + assert strict.errors is not None + assert strict.errors[0] is not None + assert strict.errors[0].code == "INPUT_TOO_LONG" + + +def test_label_groups_answer_like_gliclass(monkeypatch: pytest.MonkeyPatch) -> None: + adapter, _ = make_adapter() + output = adapter.extract( + [Item(text="AC broken, move me tonight")], + options={"label_groups": {"intent": ["room_change", "billing"], "priority": ["low", "high"]}, "threshold": 0.0}, + ) + data = (output.data or [{}])[0] + assert set(data) == {"intent", "priority"} + assert data["intent"]["type"] == "choice" + assert set(data["intent"]["probabilities"]) == {"room_change", "billing"} + labels = [c["label"] for c in (output.classifications or [[]])[0]] + assert sorted(labels) == ["intent.billing", "intent.room_change", "priority.high", "priority.low"] + scores = [c["score"] for c in (output.classifications or [[]])[0]] + assert scores == sorted(scores, reverse=True) + + +def test_labels_mode_returns_every_label_sorted_and_threshold_filters() -> None: + adapter, _ = make_adapter() + output = adapter.extract([Item(text="hello there")], labels=["a", "b", "c"]) + assert output.data is None + assert output.classifications is not None + ranked = output.classifications[0] + assert [c["label"] for c in ranked] == ["c", "b", "a"] # later markers get higher fake logits + assert sum(c["score"] for c in ranked) == pytest.approx(1.0) + filtered = adapter.extract([Item(text="hello there")], labels=["a", "b", "c"], options={"threshold": 0.335}) + assert [c["label"] for c in (filtered.classifications or [[]])[0]] == ["c"] + + +def test_prompt_over_budget_is_input_too_long() -> None: + adapter, _ = make_adapter(window=64, max_prompt_tokens=32) + with pytest.raises(InputTooLongError, match="takes at most 32"): + adapter.extract([Item(text="x")], labels=[f"label number {i}" for i in range(10)]) + + +def test_scoring_chunks_preserve_row_order() -> None: + adapter, _ = make_adapter(window=64, inference_batch_tokens=64) + rows = [[5] * n for n in (3, 40, 7, 64, 1)] + positions = [0, 1, 2] + out = adapter._score(rows, positions) + np.testing.assert_allclose(out, np.array([expected_logits(row, positions) for row in rows]), rtol=1e-5, atol=1e-9) + + +def test_non_finite_scores_fail_their_item(monkeypatch: pytest.MonkeyPatch) -> None: + adapter, _ = make_adapter() + + def score(rows: list[list[int]], positions: list[int]) -> np.ndarray: + out = np.zeros((len(rows), len(positions)), dtype=np.float32) + out[1, 0] = np.nan + return out + + monkeypatch.setattr(adapter, "_score", score) + output = adapter.extract([Item(text="a"), Item(text="b")], labels=["x", "y"]) + assert output.errors is not None + assert output.errors[1] is not None + assert output.errors[1].code == "INFERENCE_ERROR" + assert output.input_token_counts is not None + assert output.input_token_counts[1] == 0 + + +def test_batching_cost_is_capped_at_the_window() -> None: + adapter, _ = make_adapter(window=100) + costs = adapter.extract_item_costs( + [Item(text="x" * 10_000), Item(text="short"), Item(metadata={"state": {"a": "b"}})], + labels=["yes", "no"], + ) + assert costs is not None + assert costs[0] <= 2 * 100 * 4 + assert costs[1] < costs[0] + assert adapter.extract_item_costs([Item(text="x")], output_schema={"q": object()}) is not None + + +@pytest.mark.parametrize("policy", ["drop", [], {"a": 1}, 3]) +def test_invalid_overflow_policy_is_invalid_input(policy: object) -> None: + adapter, _ = make_adapter() + with pytest.raises(InvalidInputError, match="overflow_policy"): + adapter.extract([Item(text="x")], labels=["a", "b"], options={"overflow_policy": policy}) + + +def test_constructor_bounds() -> None: + assert GLiNER2DecideAdapter("x", max_seq_length=2048)._max_prompt_tokens == 512 + assert GLiNER2DecideAdapter("x", max_seq_length=256)._max_prompt_tokens == 128 + assert GLiNER2DecideAdapter("x", max_seq_length=2048, max_prompt_tokens=1024)._max_prompt_tokens == 1024 + with pytest.raises(ValueError, match="max_prompt_tokens"): + GLiNER2DecideAdapter("x", max_seq_length=64, max_prompt_tokens=64) + with pytest.raises(ValueError, match="inference_batch_tokens"): + GLiNER2DecideAdapter("x", max_seq_length=64, inference_batch_tokens=32) + + +# --------------------------------------------------------------------------- +# ModernBERT RoPE (GLiNER2.5-Decide-1B) and transformers 4 compatibility +# --------------------------------------------------------------------------- + +ETTIN_ROPE = { + "full_attention": {"rope_theta": 160000.0, "rope_type": "default"}, + "sliding_attention": {"rope_theta": 160000.0, "rope_type": "default"}, +} + + +def tiny_ettin_config() -> dict[str, Any]: + """GLiNER2.5-Decide-1B's encoder config (transformers 5 form), shrunk to a few small layers.""" + return { + "model_type": "modernbert", + "architectures": ["ModernBertForMaskedLM"], + "hidden_size": 64, + "intermediate_size": 96, + "num_attention_heads": 2, + "num_hidden_layers": 3, + "global_attn_every_n_layers": 3, + "layer_types": ["full_attention", "sliding_attention", "sliding_attention"], + "local_attention": 16, + "max_position_embeddings": 128, + "vocab_size": 128, + "pad_token_id": 0, + "bos_token_id": 1, + "eos_token_id": 2, + "cls_token_id": 1, + "sep_token_id": 2, + "rope_parameters": ETTIN_ROPE, + } + + +def build_encoder(config: dict[str, Any]) -> torch.nn.Module: + fields = {key: value for key, value in config.items() if key not in ("model_type", "architectures")} + return transformers.AutoModel.from_config(transformers.ModernBertConfig(**fields), attn_implementation="eager") + + +def test_declared_rope_thetas() -> None: + assert declared_rope_thetas({"model_type": "deberta-v2"}) is None + assert declared_rope_thetas({"model_type": "modernbert", "rope_parameters": ETTIN_ROPE}) == { + "full_attention": 160000.0, + "sliding_attention": 160000.0, + } + assert declared_rope_thetas({"model_type": "modernbert", "local_rope_theta": 5000.0}) == { + "full_attention": 160000.0, + "sliding_attention": 5000.0, + } + with pytest.raises(ValueError, match="Unsupported"): + declared_rope_thetas( + {"model_type": "modernbert", "rope_parameters": {"full_attention": {"rope_type": "yarn", "rope_theta": 1}}} + ) + + +def test_transformers4_silently_misreads_ettin_rope_and_the_fix_restores_it() -> None: + config = tiny_ettin_config() + if int(transformers.__version__.split(".")[0]) >= 5: + assert transformers4_encoder_config(config) is None + assert verify_encoder_rope(build_encoder(config), config) == { + "full_attention": 160000.0, + "sliding_attention": 160000.0, + } + return + # The trap: transformers 4 ignores rope_parameters and runs sliding layers at 10000. + unpatched = build_encoder(config) + observed = loaded_rope_thetas(unpatched, 3) + assert observed["sliding_attention"] == pytest.approx([10000.0, 10000.0], rel=1e-3) + with pytest.raises(RuntimeError, match=r"sliding_attention layers run RoPE base 10000\.0.*declares 160000\.0"): + verify_encoder_rope(unpatched, config) + # The fix: write the bases where transformers 4 reads them. + patched = transformers4_encoder_config(config) + assert patched is not None + assert (patched["global_rope_theta"], patched["local_rope_theta"]) == (160000.0, 160000.0) + fixed = build_encoder(patched) + assert verify_encoder_rope(fixed, config) == {"full_attention": 160000.0, "sliding_attention": 160000.0} + assert loaded_rope_thetas(fixed, 3)["sliding_attention"] == pytest.approx([160000.0, 160000.0], rel=1e-3) + + +def test_rope_tables_are_read_by_layer_type_in_both_transformers_layouts() -> None: + def inv_freq(theta: float, n: int = 8) -> torch.Tensor: + return 1.0 / (theta ** (torch.arange(0, 2 * n, 2).float() / (2 * n))) + + v5 = torch.nn.Module() + v5.rotary_emb = torch.nn.Module() + v5.rotary_emb.register_buffer("full_attention_inv_freq", inv_freq(160000.0)) + v5.rotary_emb.register_buffer("sliding_attention_inv_freq", inv_freq(10000.0)) + v5.rotary_emb.register_buffer("sliding_attention_original_inv_freq", inv_freq(1.0)) + assert loaded_rope_thetas(v5, 3) == { + "full_attention": [pytest.approx(160000.0, rel=1e-4)], + "sliding_attention": [pytest.approx(10000.0, rel=1e-4)], + } + with pytest.raises(RuntimeError, match="sliding_attention"): + verify_encoder_rope(v5, {"model_type": "modernbert", "rope_parameters": ETTIN_ROPE}) + with pytest.raises(RuntimeError, match="Cannot find"): + verify_encoder_rope(torch.nn.Linear(2, 2), {"model_type": "modernbert", "rope_parameters": ETTIN_ROPE}) + assert verify_encoder_rope(torch.nn.Linear(2, 2), {"model_type": "deberta-v2"}) is None + + +def test_transformers4_tokenizer_config(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(decide_module, "_transformers_major", lambda: 4) + config = {"tokenizer_class": "TokenizersBackend", "extra_special_tokens": ["[P]"]} + assert transformers4_tokenizer_config(config) == {**config, "tokenizer_class": "PreTrainedTokenizerFast"} + assert transformers4_tokenizer_config({"tokenizer_class": "DebertaV2Tokenizer"}) is None + monkeypatch.setattr(decide_module, "_transformers_major", lambda: 5) + assert transformers4_tokenizer_config(config) is None + + +def test_transformers4_layer_layout_must_follow_global_attention_period(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(decide_module, "_transformers_major", lambda: 4) + config = {**tiny_ettin_config(), "layer_types": ["sliding_attention", "full_attention", "sliding_attention"]} + with pytest.raises(ValueError, match="layer_types"): + transformers4_encoder_config(config) + + +def test_loadable_checkpoint_overlays_only_the_patched_configs(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + monkeypatch.setattr(decide_module, "_transformers_major", lambda: 4) + (tmp_path / "encoder_config").mkdir() + encoder = tiny_ettin_config() + (tmp_path / "encoder_config" / "config.json").write_text(json.dumps(encoder)) + (tmp_path / "tokenizer_config.json").write_text(json.dumps({"tokenizer_class": "TokenizersBackend"})) + (tmp_path / "config.json").write_text("{}") + (tmp_path / "model.safetensors").write_bytes(b"weights") + + with loadable_checkpoint(tmp_path) as overlay: + assert overlay != tmp_path + assert (overlay / "model.safetensors").is_symlink() + assert (overlay / "model.safetensors").read_bytes() == b"weights" + assert json.loads((overlay / "encoder_config" / "config.json").read_text())["local_rope_theta"] == 160000.0 + assert ( + json.loads((overlay / "tokenizer_config.json").read_text())["tokenizer_class"] == "PreTrainedTokenizerFast" + ) + assert not overlay.exists() + assert json.loads((tmp_path / "encoder_config" / "config.json").read_text()) == encoder # cache untouched + + monkeypatch.setattr(decide_module, "_transformers_major", lambda: 5) + with loadable_checkpoint(tmp_path) as path: + assert path == tmp_path + + +# --------------------------------------------------------------------------- +# Catalog and bundle +# --------------------------------------------------------------------------- + +DECIDE_MODELS = { + "fastino/GLiNER2.5-Decide": ("7ee5da4c2415e32259bcdc0b1a7367c32ce8d6f6", 512), + "fastino/GLiNER2.5-multi-Decide": ("6bc1d43d201b0691e733626389af8c57eea3ea68", 2048), + "fastino/GLiNER2.5-Decide-1B": ("52c94d3b698bf6d2619df9d898bdc1523ea3f1ca", 2048), +} + + +@pytest.mark.parametrize(("sie_id", "pinned"), DECIDE_MODELS.items()) +def test_decide_model_configs(sie_id: str, pinned: tuple[str, int]) -> None: + path = SIE_SERVER / "models" / f"{sie_id.replace('/', '__')}.yaml" + config = yaml.safe_load(path.read_text()) + assert config["sie_id"] == config["hf_id"] == sie_id + assert (config["hf_revision"], config["max_sequence_length"]) == pinned + profile = config["profiles"]["default"] + assert profile["adapter_path"] == "sie_server.adapters.gliner2.decide:GLiNER2DecideAdapter" + assert profile["compute_precision"] == "float16" + + +def test_decide_is_served_from_the_bundle_that_pins_gliner2_2() -> None: + bundles = SIE_SERVER / "bundles" + transformers5 = yaml.safe_load((bundles / "transformers5.yaml").read_text()) + default = yaml.safe_load((bundles / "default.yaml").read_text()) + assert "sie_server.adapters.gliner2.decide" in transformers5["adapters"] + assert transformers5["deps"]["gliner2"] == "==2.0.0" + assert "sie_server.adapters.gliner2.decide" not in default["adapters"] + assert default["deps"]["gliner2"] == ">=1.3.1,<2" diff --git a/packages/sie_server/tests/adapters/test_gliner2_decide_parity.py b/packages/sie_server/tests/adapters/test_gliner2_decide_parity.py new file mode 100644 index 000000000..f271055b1 --- /dev/null +++ b/packages/sie_server/tests/adapters/test_gliner2_decide_parity.py @@ -0,0 +1,213 @@ +r"""Parity between the GLiNER2.5-Decide adapter and the ``gliner2`` 2.x package. + +The goldens under ``goldens/gliner2_decide`` hold the package's own float32 CPU +probabilities for fixed texts, generated by +``packages/sie_server/scripts/generate_gliner2_decide_goldens.py`` without any +SIE code, on the transformers5 bundle's stack (``generated_with``). Each case +pairs an SIE ``extract`` request (Laya questions, label groups, multi-label +groups, plain labels, and a document longer than the window) with the +``gliner2.classification`` schema it must translate to. + +* CPU float32: every label's probability within 1e-4 of the golden, and every + encoder row byte-identical to the package's (hashes in the golden). +* CUDA, at the catalog precision (float16, the package's ``quantize=True``): + every probability within 5e-3 of the package's own ``Classifier`` on the same + weights, and within 1e-2 of the float32 golden, with the same top answers. + The adapter pads a request's rows into one batch where the package scores one + text at a time, so float16 rounding differs; on an L4 the largest difference + was 2.3e-3 (the multilingual model), and 1.1e-3 or less for the other two. +* GLiNER2.5-Decide-1B: the loaded encoder runs RoPE base 160000 in every layer. + +These tests need ``gliner2`` 2.x (the transformers5 bundle); run them with the +bundle's requirements: + + python -m sie_server.cli resolve-deps --bundle transformers5 > /tmp/t5.txt + uv run --no-sync --with-requirements /tmp/t5.txt pytest -c pyproject.toml -m model \\ + packages/sie_server/tests/adapters/test_gliner2_decide_parity.py +""" + +from __future__ import annotations + +import hashlib +import json +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import numpy as np +import pytest +import torch +from sie_server.adapters.gliner2.decide import GLiNER2DecideAdapter, loaded_rope_thetas +from sie_server.core.loader import load_adapter, load_model_configs +from sie_server.types.inputs import Item + +pytestmark = pytest.mark.model + +_GOLDENS = sorted((Path(__file__).parent / "goldens" / "gliner2_decide").glob("*.json")) +_MODELS_DIR = Path(__file__).resolve().parents[2] / "models" +# The goldens are float32 CPU outputs; only BLAS scheduling differs between hosts. +_CPU_TOLERANCE = 1e-4 +# float16 on CUDA: against the package on the same weights (batched rows vs one +# text at a time), and against float32. +_GPU_PACKAGE_TOLERANCE = 5e-3 +_GPU_FLOAT32_TOLERANCE = 1e-2 + + +def _golden(path: Path) -> dict[str, Any]: + return json.loads(path.read_text(encoding="utf-8")) + + +def _adapter(golden: dict[str, Any], device: str) -> GLiNER2DecideAdapter: + pytest.importorskip("gliner2.classification", reason="needs gliner2 2.x (the transformers5 bundle)") + config = load_model_configs(_MODELS_DIR)[golden["model"]] + assert config.hf_revision == golden["revision"], "golden was generated for a different checkpoint revision" + assert config.max_sequence_length == golden["generated_with"]["window"] + adapter = load_adapter(config, _MODELS_DIR, device=device) + adapter.load(device) + assert isinstance(adapter, GLiNER2DecideAdapter) + return adapter + + +def _documents(golden: dict[str, Any], case: dict[str, Any]) -> list[str]: + return golden["texts"] if case["documents"] == "texts" else [golden["long_text"]] + + +def _request(case: dict[str, Any]) -> dict[str, Any]: + request = case["request"] + return { + "labels": request.get("labels"), + "output_schema": request.get("output_schema"), + "instruction": request.get("instruction"), + "options": request.get("options"), + } + + +def _answers(case: dict[str, Any], output: Any, index: int) -> dict[str, dict[str, float]]: + """The adapter's probabilities for one item, keyed like the golden's (task -> label -> p).""" + tasks = case["reference_call"]["tasks"] + if "labels" in case["request"]: + return {tasks[0]["name"]: {c["label"]: c["score"] for c in output.classifications[index]}} + data = output.data[index] + answers: dict[str, dict[str, float]] = {} + for key, answer in data.items(): + if answer.get("type") == "noul": + answers[key] = {"yes": answer["noul"], "no": 1.0 - answer["noul"]} + else: + answers[key] = answer["probabilities"] + return answers + + +def _run(adapter: GLiNER2DecideAdapter, case: dict[str, Any], texts: list[str]) -> tuple[Any, list[list[int]]]: + rows: list[list[int]] = [] + score = adapter._score + + def spy(batch: list[list[int]], positions: list[int]) -> np.ndarray: + rows.extend(batch) + return score(batch, positions) + + adapter._score = spy # ty:ignore[invalid-assignment] + try: + output = adapter.extract([Item(text=text) for text in texts], **_request(case)) + finally: + del adapter._score + return output, rows + + +def _max_difference(actual: dict[str, dict[str, float]], expected: dict[str, dict[str, float]]) -> float: + assert list(actual) == list(expected) + worst = 0.0 + for task, probabilities in expected.items(): + assert set(actual[task]) == set(probabilities), task + worst = max(worst, *(abs(actual[task][label] - p) for label, p in probabilities.items())) + return worst + + +def _same_top_answers(actual: dict[str, dict[str, float]], expected: dict[str, dict[str, float]]) -> bool: + """Same highest-probability label per task, wherever the expected top label leads by more than 0.01.""" + for task, probabilities in expected.items(): + ranked = sorted(probabilities.values(), reverse=True) + top = max(probabilities, key=probabilities.__getitem__) + if len(ranked) > 1 and ranked[0] - ranked[1] <= 0.01: + continue + if max(actual[task], key=actual[task].__getitem__) != top: + return False + return True + + +@pytest.fixture(scope="module", params=_GOLDENS, ids=lambda path: path.stem) +def cpu_adapter(request: pytest.FixtureRequest) -> Iterator[tuple[dict[str, Any], GLiNER2DecideAdapter]]: + golden = _golden(request.param) + adapter = _adapter(golden, "cpu") + yield golden, adapter + adapter.unload() + + +def test_cpu_float32_matches_the_package(cpu_adapter: tuple[dict[str, Any], GLiNER2DecideAdapter]) -> None: + golden, adapter = cpu_adapter + for case in golden["cases"]: + texts = _documents(golden, case) + output, rows = _run(adapter, case, texts) + assert output.errors is None, case["name"] + for index, (row, expected) in enumerate(zip(rows, case["results"], strict=True)): + assert len(row) == expected["row_length"], (case["name"], index) + assert hashlib.sha256(json.dumps(row).encode()).hexdigest() == expected["row_sha256"], (case["name"], index) + worst = _max_difference(_answers(case, output, index), expected["probabilities"]) + assert worst <= _CPU_TOLERANCE, (case["name"], index, worst) + + +def test_long_documents_are_read_to_the_window(cpu_adapter: tuple[dict[str, Any], GLiNER2DecideAdapter]) -> None: + golden, adapter = cpu_adapter + case = next(case for case in golden["cases"] if case["name"] == "long_document") + assert case["results"][0]["max_len"] is not None, "the long document must not fit the window" + output, rows = _run(adapter, case, [golden["long_text"]]) + assert len(rows[0]) <= golden["generated_with"]["window"] + assert output.input_token_counts is not None + assert 0 < output.input_token_counts[0] <= golden["generated_with"]["window"] + + +def test_decide_1b_runs_its_declared_rope_base(cpu_adapter: tuple[dict[str, Any], GLiNER2DecideAdapter]) -> None: + golden, adapter = cpu_adapter + if not golden["model"].endswith("Decide-1B"): + pytest.skip("only the Ettin (ModernBERT) encoder has rotary tables") + thetas = loaded_rope_thetas(adapter._model.encoder, 3) + assert set(thetas) == {"full_attention", "sliding_attention"} + for values in thetas.values(): + assert values == pytest.approx([160000.0] * len(values), rel=1e-3) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA") +@pytest.mark.parametrize("path", _GOLDENS, ids=lambda path: path.stem) +def test_cuda_matches_the_package_on_the_same_weights(path: Path) -> None: + from gliner2.classification import ( # ty:ignore[unresolved-import] + ClassificationConfig, + ClassificationSchema, + Classifier, + ) + + golden = _golden(path) + adapter = _adapter(golden, "cuda:0") + try: + classifier = Classifier(adapter._model) + for case in golden["cases"]: + texts = _documents(golden, case) + output, _ = _run(adapter, case, texts) + assert output.errors is None, case["name"] + schema = ClassificationSchema() + for task in case["reference_call"]["tasks"]: + add = schema.single if task["kind"] == "single" else schema.multi + add(task["name"], task["labels"], instruction=task["instruction"]) + for index, (text, expected) in enumerate(zip(texts, case["results"], strict=True)): + actual = _answers(case, output, index) + scores = classifier.score(text, schema, config=ClassificationConfig(max_len=expected["max_len"])) + package = { + task: {label: scores.probability(task, label) for label in labels} + for task, labels in expected["probabilities"].items() + } + assert _max_difference(actual, package) <= _GPU_PACKAGE_TOLERANCE, (case["name"], index) + assert _max_difference(actual, expected["probabilities"]) <= _GPU_FLOAT32_TOLERANCE, ( + case["name"], + index, + ) + assert _same_top_answers(actual, package), (case["name"], index) + finally: + adapter.unload() diff --git a/packages/sie_server/tests/adapters/test_gliner2_words.py b/packages/sie_server/tests/adapters/test_gliner2_words.py new file mode 100644 index 000000000..c4a74bf66 --- /dev/null +++ b/packages/sie_server/tests/adapters/test_gliner2_words.py @@ -0,0 +1,175 @@ +"""The linear-time word splitter yields exactly what gliner2's ``WhitespaceTokenSplitter`` yields. + +The reference splitters below are gliner2's own code (1.3.2 ``gliner2.processor`` +and 2.0.0 ``gliner2.processing.word_splitter``), copied verbatim; the installed +package's splitter is checked too. +""" + +from __future__ import annotations + +import random +import re +import time +from collections.abc import Iterator + +import pytest +from sie_server.adapters.gliner2.words import PACKAGE_PATTERN, LinearWordSplitter, linear_equivalent, word_spans + + +class WhitespaceTokenSplitter: + """gliner2 2.0.0's splitter: match the text, lowercase each word.""" + + __slots__ = () + _PATTERN = re.compile( + r"""(?:https?://[^\s]+|www\.[^\s]+) + |[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,} + |@[a-z0-9_]+ + |\w+(?:[-_]\w+)* + |\S""", + re.VERBOSE | re.IGNORECASE, + ) + + def __call__(self, text: str, lower: bool = True) -> Iterator[tuple[str, int, int]]: + for m in self._PATTERN.finditer(text): + token = m.group() + yield (token.lower() if lower else token), m.start(), m.end() + + +class Gliner2V1Splitter(WhitespaceTokenSplitter): + """gliner2 1.3.2's splitter: lowercase the text, then match it.""" + + __slots__ = () + + def __call__(self, text: str, lower: bool = True) -> Iterator[tuple[str, int, int]]: + if lower: + text = text.lower() + for m in self._PATTERN.finditer(text): + yield m.group(), m.start(), m.end() + + +Gliner2V1Splitter.__name__ = "WhitespaceTokenSplitter" + +CORPUS = [ + "", + " ", + "My subscription renewed on April 15 for 5,400 yen after the service was already down. Can I get a refund?", + "Guest in room 1408 says the AC has been out since yesterday; they want to move tonight or leave!", + ("Das Paket kam besch\u00e4digt an, bitte schicken Sie Ersatz. Ich brauche es bis Freitag, sonst storniere ich!"), + ( + "\u8bf7\u5e2e\u6211\u53d6\u6d88\u8ba2\u5355\uff0c\u6211\u4e0d\u60f3\u8981\u4e86\u3002\u9000\u6b3e\u4ec0\u4e48\u65f6\u5019\u5230\u8d26\uff1f" + ), + ("Visit https://example.com/a?b=c or www.Example.org/x, mail Jane.Doe+tag@Mail.Example.co.uk, ping @support_team!"), + "user@host a@b.c a@b.co x.y@z.com. ....@ ...@x.io @@@ a@@b.cc @a_b-c foo@bar", + "state-of-the-art under_score e-mail co-op x--y a_-b -lead trail- __init__ 3.14 1,000,000 v2.0.1", + "Tabs\tand\nnewlines\r\nand\u00a0no-break\u2003em spaces\u200bzero-width", + ( + "\u0130stanbul \u03a3\u038a\u03a3\u03a5\u03a6\u039f\u03a3 \u039f\u0394\u039f\u03a3'\u0391 Stra\u00dfe " + "\u01c5emal \ufb01ne KELVIN\u212a na\u00efve \u017fun" + ), + ( + "emoji \U0001f600\U0001f44d\U0001f3fd and flags \U0001f1eb\U0001f1f7, arrows \u2192, math \u2211\u222b," + " CJK \u6f22\u5b57\u304b\u306a\u30ab\u30ca\ud55c\uad6d\uc5b4" + ), + '{"subject": "Login broken", "body": "Error 500 since the update.", "tags": ["urgent", "web"]}', + ( + "a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a." + " ..................................................................................................." + ". %+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_%+-_@xxxxxxxxxx" + ), + "HTTP://UPPER.CASE/Path WWW.UPPER.ORG http:// https:/ www. wwwx.org", +] + + +def _spans(splitter: object, text: str, lower: bool) -> list[tuple[str, int, int]]: + return list(splitter(text, lower)) # ty:ignore[call-non-callable] + + +@pytest.mark.parametrize("text", CORPUS) +def test_word_spans_match_the_package_pattern(text: str) -> None: + assert list(word_spans(text)) == [(m.start(), m.end()) for m in PACKAGE_PATTERN.finditer(text)] + + +@pytest.mark.parametrize("text", CORPUS) +@pytest.mark.parametrize("lower", [True, False]) +def test_splitters_match_both_gliner2_versions(text: str, lower: bool) -> None: + v2 = linear_equivalent(WhitespaceTokenSplitter()) + v1 = linear_equivalent(Gliner2V1Splitter()) + assert v2 is not None + assert v1 is not None + assert not v2.lower_text_first + assert v1.lower_text_first + assert _spans(v2, text, lower) == _spans(WhitespaceTokenSplitter(), text, lower) + assert _spans(v1, text, lower) == _spans(Gliner2V1Splitter(), text, lower) + + +def test_fuzzed_texts_split_identically() -> None: + alphabet = [*"aZ9._%+-@:/ \t\nwhtps_", "http://", "https://", "www.", ".com", "@x.io", "\u0130", "\u017f"] + alphabet += ["\u212a", "\u00e9", "\u4e2d", "\uff0c", "--", "\u00a0", "\u200b", "\U0001f600", "\u01c5"] + alphabet += ["\u00df", "\u03a3", "'"] + rng = random.Random(0) # noqa: S311 -- deterministic test data + for _ in range(20_000): + text = "".join(rng.choice(alphabet) for _ in range(rng.randint(0, 40))) + assert list(word_spans(text)) == [(m.start(), m.end()) for m in PACKAGE_PATTERN.finditer(text)], text + + +def test_the_installed_gliner2_splitter_has_a_linear_equivalent() -> None: + gliner2 = pytest.importorskip("gliner2") + major = int(gliner2.__version__.split(".")[0]) + if major >= 2: + from gliner2.processing.word_splitter import ( + WhitespaceTokenSplitter as Installed, # ty:ignore[unresolved-import] + ) + else: + from gliner2.processor import WhitespaceTokenSplitter as Installed # ty:ignore[unresolved-import] + installed = Installed() + linear = linear_equivalent(installed) + assert linear is not None + assert linear.lower_text_first == (major < 2) + for text in CORPUS: + for lower in (True, False): + assert _spans(linear, text, lower) == _spans(installed, text, lower) + + +def test_other_splitters_have_no_linear_equivalent() -> None: + class CharLevelSplitter: + _PATTERN = re.compile(r"[A-Za-z0-9@._\-+]+|\S") + + def __call__(self, text: str, lower: bool = True) -> Iterator[tuple[str, int, int]]: + yield from () + + class WhitespaceTokenSplitterLookalike(WhitespaceTokenSplitter): + _PATTERN = re.compile(r"\S+", re.VERBOSE | re.IGNORECASE) + + WhitespaceTokenSplitterLookalike.__name__ = "WhitespaceTokenSplitter" + assert linear_equivalent(CharLevelSplitter()) is None + assert linear_equivalent(WhitespaceTokenSplitterLookalike()) is None + assert linear_equivalent(lambda text, lower=True: iter(())) is None + + +@pytest.mark.parametrize( + "text", + [ + "." * (2 * 1024 * 1024), + "a." * (1024 * 1024), + "%" * (2 * 1024 * 1024 - 2) + "@x", + "a@" * (1024 * 1024), + ". " * (1024 * 1024), + ], + ids=["dots", "a-dots", "local-run-then-at", "at-chain", "spaced-dots"], +) +def test_splitting_is_linear(text: str) -> None: + started = time.perf_counter() + first = [span for _, span in zip(range(2048), word_spans(text), strict=False)] + assert len(first) == 2048 + assert time.perf_counter() - started < 1.0 # the package's regex takes minutes to hours on these + + started = time.perf_counter() + count = sum(1 for _ in word_spans(text[: 256 * 1024])) + assert count > 0 + assert time.perf_counter() - started < 2.0 + + +def test_lower_text_first_offsets_index_the_lowered_text() -> None: + text = "İstanbul x" + assert list(LinearWordSplitter(lower_text_first=True)(text)) == list(Gliner2V1Splitter()(text)) + assert list(LinearWordSplitter(lower_text_first=False)(text)) == list(WhitespaceTokenSplitter()(text)) diff --git a/packages/sie_server/tests/api/test_option.py b/packages/sie_server/tests/api/test_option.py index 489a03f82..e8cc61737 100644 --- a/packages/sie_server/tests/api/test_option.py +++ b/packages/sie_server/tests/api/test_option.py @@ -32,3 +32,5 @@ def test_invalid_overflow_policy_type_returns_400() -> None: assert exc_info.value.status_code == 400 assert exc_info.value.detail["code"] == "INVALID_INPUT" + assert exc_info.value.detail["message"].startswith("Invalid overflow_policy: ['truncate_text']") + span.set_attribute.assert_called_once_with("error", "invalid_overflow_policy") diff --git a/packages/sie_server/tests/core/test_runtime_options.py b/packages/sie_server/tests/core/test_runtime_options.py index f095319d8..f0d290b98 100644 --- a/packages/sie_server/tests/core/test_runtime_options.py +++ b/packages/sie_server/tests/core/test_runtime_options.py @@ -164,6 +164,21 @@ def test_malformed_profile_selector_raises_invalid_input(profile: object) -> Non merge_runtime_options(config, {"profile": profile}) +@pytest.mark.parametrize("policy", [[], ["truncate_text"], {}, {"a": 1}, 0, True, "drop", ""]) +def test_invalid_overflow_policy_raises_invalid_input_on_both_ingress_paths(policy: object) -> None: + """The queue worker merges options here too, so an invalid policy is a 400, not an inference error.""" + config = _embedder_config() + + with pytest.raises(InvalidInputError, match="Invalid overflow_policy"): + merge_runtime_options(config, {"overflow_policy": policy}) + + +@pytest.mark.parametrize("policy", ["default", "truncate_text", "error", None]) +def test_valid_overflow_policies_pass_through(policy: str | None) -> None: + merged = merge_runtime_options(_embedder_config(), {"overflow_policy": policy}) + assert merged["overflow_policy"] == policy + + def _generation_config() -> ModelConfig: return ModelConfig.model_validate( { diff --git a/packages/sie_server/tests/test_all_models.py b/packages/sie_server/tests/test_all_models.py index 6969df5d4..07a6decc0 100644 --- a/packages/sie_server/tests/test_all_models.py +++ b/packages/sie_server/tests/test_all_models.py @@ -1319,6 +1319,55 @@ def test_convaiinnovations_laya_typed_decisions_extract() -> None: ) +# ============================================================================= +# Extract models (text input - GLiNER2.5-Decide typed decisions) +# The same typed questions as Laya. These models need gliner2 2.x, which the +# transformers5 bundle carries: run them with that bundle's requirements, e.g. +# python -m sie_server.cli resolve-deps --bundle transformers5 > /tmp/t5.txt +# uv run --no-sync --with-requirements /tmp/t5.txt pytest -c pyproject.toml -m model \ +# packages/sie_server/tests/test_all_models.py -k decide +# ============================================================================= + + +def _check_gliner2_decide(adapter: Any, model_name: str, expected: tuple[str, bool, str] | None) -> None: + """Typed answers for one state: (intent choice, refund noul answer, top department label).""" + output = adapter.extract([Item(text=_LAYA_STATE)], output_schema=_LAYA_QUESTIONS) + assert output.data is not None + answers = output.data[0] + assert list(answers) == list(_LAYA_QUESTIONS) + assert sum(answers["intent"]["probabilities"].values()) == pytest.approx(1.0, abs=1e-5) + assert sum(answers["frustration"]["probabilities"].values()) == pytest.approx(1.0, abs=1e-5) + assert 0.0 <= answers["frustration"]["score"] <= 3.0 + labels = adapter.extract([Item(text=_LAYA_STATE)], labels=["billing", "technical", "sales"]) + assert labels.classifications is not None + actual = (answers["intent"]["choice"], answers["refund_requested"]["answer"], labels.classifications[0][0]["label"]) + if expected is None: + msg = f"FILL: {model_name} decide = {actual}" + raise AssertionError(msg) + assert actual == expected + + +def test_fastino_gliner2_5_decide_extract() -> None: + pytest.importorskip("gliner2.classification", reason="needs gliner2 2.x (the transformers5 bundle)") + _check_gliner2_decide( + _get_adapter("fastino/GLiNER2.5-Decide"), "fastino/GLiNER2.5-Decide", ("refund", True, "billing") + ) + + +def test_fastino_gliner2_5_multi_decide_extract() -> None: + pytest.importorskip("gliner2.classification", reason="needs gliner2 2.x (the transformers5 bundle)") + _check_gliner2_decide( + _get_adapter("fastino/GLiNER2.5-multi-Decide"), "fastino/GLiNER2.5-multi-Decide", ("refund", True, "billing") + ) + + +def test_fastino_gliner2_5_decide_1b_extract() -> None: + pytest.importorskip("gliner2.classification", reason="needs gliner2 2.x (the transformers5 bundle)") + _check_gliner2_decide( + _get_adapter("fastino/GLiNER2.5-Decide-1B"), "fastino/GLiNER2.5-Decide-1B", ("refund", True, "billing") + ) + + # ============================================================================= # Extract models (image input - Florence-2, Donut) # These require image input, skipping for now as they need special handling