Repository navigation
feat(decisions): image assets and Box questions for Jev - #155
Merged
Merged
Conversation
EvaluationRequest now carries an assets map of base64-encoded images keyed by the reference string used in state or question instructions, plus a with_image builder and a needs_image_opt_in check that also covers box questions. A new Box question variant lets callers ask where something is in an attached image, and empty assets are skipped on the wire so text-only requests serialise as before. Auto-committed-on: cuttlefish Co-authored-by: Medulla <medulla@tinyhumans.ai>
Requests can now carry base64-encoded image assets referenced from state or question text, plus a new Box question type that asks for bounding boxes. Validation rejects unreferenced or oversized assets, and the client sends the provider's image opt-in header whenever a request includes an image or Box question. Auto-committed-on: cuttlefish Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds a client test asserting the image opt-in header and assets payload appear only when an image is attached, and exposes the image module to the crate so tests can build PNG fixtures. Also reformats a few existing assertions. Auto-committed-on: cuttlefish Co-authored-by: Medulla <medulla@tinyhumans.ai>
…stions Adds a guide section covering image attachments and Box questions, including a usage example, the danger-zone header the client sends, and the validation limits applied before transport. The request module README now also mentions the image asset builder and the renamed test file. Auto-committed-on: cuttlefish Co-authored-by: Medulla <medulla@tinyhumans.ai>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a16bfdf4a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This was referenced Nov 14, 2026
Merged
…json values Asset reference validation now walks the decoded JSON values and object keys instead of searching the escaped serialized text, so keys containing backslashes, quotes or control characters are matched correctly. Auto-committed-on: cuttlefish Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a test that rekeys an asset to paths containing backslashes, quotes and tabs, confirming validation matches against the decoded state rather than the raw JSON text. Auto-committed-on: cuttlefish Co-authored-by: Medulla <medulla@tinyhumans.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
TypeSafe's System One API now accepts images for Jev, and Jev 1.14 (
jev-preview) adds aboxquestion that returns where a thing is in an image. This adds both totinyinference-decisions.EvaluationRequest.assets: a map of images keyed by the reference string the state uses.EvaluationRequest::with_image(key, ImageAsset::png(&bytes))/ImageAsset::jpeg(&bytes)encode plain padded base64 with nodata:prefix. When the map is empty it is left off the wire, so text requests serialise byte-for-byte as before.Question::Box(BoxQuestion { instructions }), andAnswer::Box(BoxAnswer { edges: BoxEdges { top, bottom, left, right }, confidence })withBoxEdges::centre(). Each edge is a 0–1 fraction measured from the top-left.The client sends
TypeSafe-Danger-Zone-Images: trueon any request that has an image or a Box question; the provider returns 400 without it.validate()checks the provider's documented image limits before sending:It reads dimensions from the file header, with no image decoder dependency.
Response validation for Box: edges within 0–1 and correctly ordered, confidence within 0–1.
API / behaviour changes
New public types:
Asset,ImageAsset,ImageFormat,BoxQuestion,BoxAnswer,BoxEdges. Breaking for exhaustive matches:QuestionandAnswereach gain aBoxvariant. Also breaking: code that buildsEvaluationRequestwith a struct literal now has to setassets. Requests without images behave exactly as before.Why
TinyComputer will use this to solve visual captchas inside browser tasks: Box to find the checkbox and Verify button, and Noul per tile for image grids. The follow-up PR is in tinycomputer.
Validation
cargo fmt --all -- --checkcargo clippy -p tinyinference-decisions --all-targets --all-features -- -D warningscargo build --all-targets --all-featurescargo test -p tinyinference-decisions: 73 unit tests pass, 13 of them new (image header reading, limits, the opt-in header, the Box wire shape and validation)RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p tinyinference-decisionsNot exercised against the live API in this PR. That happens in the tinycomputer captcha PR with a real key.
Summary by CodeRabbit