Skip to content

feat(decisions): image assets and Box questions for Jev - #155

Merged
senamakel merged 6 commits into
mainfrom
captcha-fighter
Nov 14, 2026
Merged

senamakel merged 6 commits into
mainfrom
captcha-fighter

Conversation

@senamakel

@senamakel senamakel commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

What

TypeSafe's System One API now accepts images for Jev, and Jev 1.14 (jev-preview) adds a box question that returns where a thing is in an image. This adds both to tinyinference-decisions.

  • EvaluationRequest.assets: a map of images keyed by the reference string the state uses. EvaluationRequest::with_image(key, ImageAsset::png(&bytes)) / ImageAsset::jpeg(&bytes) encode plain padded base64 with no data: prefix. When the map is empty it is left off the wire, so text requests serialise byte-for-byte as before.

  • Question::Box(BoxQuestion { instructions }), and Answer::Box(BoxAnswer { edges: BoxEdges { top, bottom, left, right }, confidence }) with BoxEdges::centre(). Each edge is a 0–1 fraction measured from the top-left.

  • The client sends TypeSafe-Danger-Zone-Images: true on any request that has an image or a Box question; the provider returns 400 without it.

  • validate() checks the provider's documented image limits before sending:

    • PNG or JPEG, with a header that matches the declared format
    • 280–1568 px per side
    • at most 16 MiB per image and 32 MiB in total
    • every asset key is referenced by the state or a question
    • non-empty references of at most 1024 bytes

    It reads dimensions from the file header, with no image decoder dependency.

  • Response validation for Box: edges within 0–1 and correctly ordered, confidence within 0–1.

API / behaviour changes

New public types: Asset, ImageAsset, ImageFormat, BoxQuestion, BoxAnswer, BoxEdges. Breaking for exhaustive matches: Question and Answer each gain a Box variant. Also breaking: code that builds EvaluationRequest with a struct literal now has to set assets. Requests without images behave exactly as before.

Why

TinyComputer will use this to solve visual captchas inside browser tasks: Box to find the checkbox and Verify button, and Noul per tile for image grids. The follow-up PR is in tinycomputer.

Validation

  • cargo fmt --all -- --check
  • cargo clippy -p tinyinference-decisions --all-targets --all-features -- -D warnings
  • cargo build --all-targets --all-features
  • cargo test -p tinyinference-decisions: 73 unit tests pass, 13 of them new (image header reading, limits, the opt-in header, the Box wire shape and validation)
  • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p tinyinference-decisions

Not exercised against the live API in this PR. That happens in the tinycomputer captcha PR with a real key.

Summary by CodeRabbit

  • New Features
    • Added support for attaching PNG and JPEG images to evaluation requests and asking questions about image regions.
    • Added bounding-box responses with normalized coordinates, confidence values, and a helper to calculate the box center.
    • Image requests include the required opt-in header. Image attachments are checked for supported dimensions, valid references, and individual and combined size limits.
  • Bug Fixes
    • Invalid box responses, including out-of-range or inverted coordinates, are rejected.
  • Documentation
    • Added guidance and examples for image attachments and box questions, including validation limits.

senamakel and others added 4 commits November 14, 2026 11:52
EvaluationRequest now carries an assets map of base64-encoded images keyed by the reference string used in state or question instructions, plus a with_image builder and a needs_image_opt_in check that also covers box questions. A new Box question variant lets callers ask where something is in an attached image, and empty assets are skipped on the wire so text-only requests serialise as before.

Auto-committed-on: cuttlefish
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Requests can now carry base64-encoded image assets referenced from state or question text, plus a new Box question type that asks for bounding boxes. Validation rejects unreferenced or oversized assets, and the client sends the provider's image opt-in header whenever a request includes an image or Box question.

Auto-committed-on: cuttlefish
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds a client test asserting the image opt-in header and assets payload appear only when an image is attached, and exposes the image module to the crate so tests can build PNG fixtures. Also reformats a few existing assertions.

Auto-committed-on: cuttlefish
Co-authored-by: Medulla <medulla@tinyhumans.ai>
…stions

Adds a guide section covering image attachments and Box questions, including a usage example, the danger-zone header the client sends, and the validation limits applied before transport. The request module README now also mentions the image asset builder and the renamed test file.

Auto-committed-on: cuttlefish
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-14T14:36:24.205620Z 67b2797 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The decisions crate adds image assets and Box questions to evaluation requests. It validates image data and references, adds an opt-in header to applicable client requests, and supports validated Box answers.

Changes: Image and Box support

Layer / File(s) Summary
Request types and image validation
crates/tinyinference-decisions/Cargo.toml, crates/tinyinference-decisions/src/request/..., crates/tinyinference-decisions/src/lib.rs, docs/tinyinference-decisions.md
Requests can include PNG or JPEG assets and Box questions. Validation checks image format, dimensions, size limits, asset references, and Box instructions. Tests cover serialization and validation rules.
Client image opt-in header
crates/tinyinference-decisions/src/client/...
The client adds the image opt-in header when a request contains an image asset or a Box question. Tests compare requests with and without an image.
Box response types and validation
crates/tinyinference-decisions/src/response/..., crates/tinyinference-decisions/src/lib.rs
Responses can include Box edges and confidence. Validation checks bounds and edge ordering. Tests cover valid and invalid answers and the calculated center.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant EvaluationRequest
  participant Client
  participant Provider
  Caller->>EvaluationRequest: Build request with image asset or Box question
  Client->>EvaluationRequest: Check whether opt-in is required
  Client->>Provider: Send evaluation request with opt-in header when required
Loading
Merge Risk: Minimal · up to 67b27

This adds image assets and Box questions to the decisions crate, with validation and an opt-in header only for requests that use them. Text-only requests are unchanged. No concrete merge-blocking issue was found. The public API change for exhaustive matches is intended and documented.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "feat(decisions): image assets and Box questions for Jev" matches the main change. The PR adds image assets (PNG/JPEG inline base64 with validation) and Box question/answer types to tinyinfe…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "feat(decisions): image assets and Box questions for Jev" matches the main change. The PR adds image assets (PNG/JPEG inline base64 with validation) and Box question/answer types to tinyinference-decisions, which the objectives and file summaries confirm. The title is specific and concise, and a teammate scanning history would understand the primary change. The conventional-commit scope names the affected crate.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit packs a picture tight,
With edges measured left and right.
The request carries its key,
The header joins when needed be.
Box answers mark a center true,
Then off the rabbit hops anew.

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 81d7ab3f-57d6-47de-bd76-12fe15f05f20
Commits

Reviewing files that changed from the base of the PR and between 8a16bfd and 67b2797.

Files selected for processing: 2

  • crates/tinyinference-decisions/src/request/mod.rs
  • crates/tinyinference-decisions/src/request/mod_tests.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8a16bfdf4a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

senamakel and others added 2 commits November 14, 2026 14:33
…json values

Asset reference validation now walks the decoded JSON values and object keys instead of searching the escaped serialized text, so keys containing backslashes, quotes or control characters are matched correctly.

Auto-committed-on: cuttlefish
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a test that rekeys an asset to paths containing backslashes, quotes and tabs, confirming validation matches against the decoded state rather than the raw JSON text.

Auto-committed-on: cuttlefish
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@senamakel
senamakel merged commit cf82933 into main Nov 14, 2026
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant