fix(server): limit the label prompt of GLiNER-family extract requests - #381
Merged
Merged
Conversation
GLiNER, GLiNER2 and GLiREL encode a request's labels, relation types, class labels and schema fields with every document, and bill only the document. Validate that prompt before any model work, as GLiFormer and GLiNER2.5-Decide do, and reject a request with INVALID_INPUT when: - a label, relation type, class label, task name, field name or choice has more than 128 characters (checked before anything is tokenized), or - the prompt takes more than max_prompt_tokens tokens, counted with the model's tokenizer: 1024 for GLiNER and GLiREL, 2048 for GLiNER2, whose schemas carry field descriptions. The option is configurable per model. The defaults sit well above real label sets: a 60-type PII list takes about 230 tokens and a 50-field described schema about 1,300. GLiREL also limits an item to 256 supplied entities, since it scores every pair of them, and reports missing or malformed input (labels, text, entities, entity offsets) as INVALID_INPUT, checked for every item before any is run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (6)
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 5 remain after this review. 📝 WalkthroughWalkthroughThe adapters now validate label lengths and prompt-token counts. GLiNER2 also uses prompt estimates in row sizing. GLiREL rejects malformed inputs and items with more than 256 entities. Tests cover prompt limits, input validation, and long-text row estimates. ChangesPrompt Limits Across GLiNER Adapters
Sequence Diagram(s)sequenceDiagram
participant Caller
participant GLiNERAdapter
participant PromptLimit
participant GLiNERProcessor
Caller->>GLiNERAdapter: extract with labels and relation labels
GLiNERAdapter->>PromptLimit: check prompt and cache key
PromptLimit->>GLiNERProcessor: count prepared prompt when uncached
GLiNERProcessor-->>PromptLimit: return token count
PromptLimit-->>GLiNERAdapter: return count or reject with InvalidInputError
Suggested reviewers: Priority: ➖ Normal Merge Risk: 🔵 Low · up to The prompt-limit change has no established functional failure, but the GLiNER package layout still needs to meet the contributor requirement. This is a bounded issue for the owner to address before merge or explicitly accept. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GLiNER, GLiNER2 and GLiREL encode a request's labels, relation types, class
labels and schema fields (the task prompt) with every document, and bill only
the document's tokens. Apart from the count of labels (at most 1000), the
prompt had no documented size limit. GLiFormer and GLiNER2.5-Decide already
limit theirs (512 prompt tokens).
Change
A shared module,
sie_server/adapters/_prompt_limit.py, validates theprompt before any model work. A request is rejected with HTTP 400
INVALID_INPUTwhen either of these holds:output_schemaproperty name or choice has more than 128 characters (thelimit GLiFormer applies). This is checked first, before anything is
tokenized.
tokenizer, and the whole prompt's characters are bounded before counting.
The limit is:
processor (entity types and, for relex models, relation types). GLiREL
counts
[REL] type ... [SEP].covers entity, relation and classification labels, the task name, and
output_schemafields and choices. It includes the classifier route usedby GLiGuard (
GLiNER2ClassificationAdapter, gliner2 2.x).The limit is a
max_prompt_tokensload-time option, so a deployment canchange it per model. The size of each distinct prompt is cached, so repeated
requests with the same labels do not tokenize them again.
The defaults sit well above real label sets. Measured with the models'
tokenizers:
output_schemawith descriptions (GLiNER2)GLiREL also:
metadata.entities. GLiREL scores every pair of them.INVALID_INPUT. This covers missing labels, textor entities, a non-list
entities, and invalid entity offsets. Before, mostof these were reported as an inference error. Every item is checked before
any is run.
GLiNER2 row estimates for batch planning (from #380) now come from the same
prompt count.
Compatibility
No API or wire change. Requests over the documented limits, which the models
could not use well in any case, now get a 400 with a message naming the limit
instead of running. The limits are documented in the adapters' docstrings.
The classic GLiNER limit of 1024 prompt tokens admits about 340 one-word or
200–250 two-word labels, fewer than the 1000 labels a request may carry;
deployments that need more can raise it with the
max_prompt_tokensoption.Tests
tests/adapters/test_prompt_limit.pycovers the following:anything is tokenized (the test asserts the counter is never called),
including a 4 MiB label. It is enforced for GLiNER relation types, GLiNER2
labels, the task name, field names and choices, and GLiREL relation types.
GLiNER2 (labels, long descriptions, many choices, classification) and
GLiREL. Rejection happens before any inference, and the limit is
configurable.
classifier route.
as
INVALID_INPUTbefore any model work.On
main, the adapter-level rejection tests fail because no 400 is raised.The GLiNER-family adapter tests pass with gliner2 1.3.2 and 2.0.0.
Validation
mise run lint,mise run typecheck: pass.Dockerfile.cuda12,defaultandtransformers5bundles), served on an L4 and probed over HTTP forgliner2-large, gliner_multi-v2.1, NuNER Zero, gliner-relex-large, GLiREL
large and GLiGuard:
schemas, 257 GLiREL entities, and bad offsets get 400
INVALID_INPUTinabout 5–20 ms.
🤖 Generated with Claude Code
Summary by CodeRabbit