[audreyt]

Stacked on #120. This branch contains #120's commits; the change here is the last three commits (72d69b6, f3887b7, be1d869). Please review #120 first.

Summary

By default the generative model writes each quote and may lightly edit the participant's words in brackets. This adds an opt-in path where an extraction model marks the participant's own sentences instead, so every quote is verbatim text from the response.

The extraction model is pluggable behind one Extractor interface (src/models/extraction.py), with two backends:

• systemone: a System One host that answers extract questions. Of the hosts tested, only Inception Mercury Decide does; local Clef, pplx-decider, Jev and Clef on Workers AI accept only choice, noul and score.
• clairic: Cloudflare's Clairic extraction model on Workers AI (@cf/cloudflare/clairic). A field that allows several spans is asked as a list of text; a single-span field as text.

Both backends check every span against the request text before returning it and share the System One transport's bearer auth and retry policy. Quote extraction has its own settings, apart from DECISION_*, so judgments can stay on a local model while quotes come from either backend:

# Inception Mercury Decide
export QUOTE_ENDPOINT_TYPE=systemone
export QUOTE_MODEL=mercury-decide
export QUOTE_MODEL_HOST=https://api.inceptionlabs.ai/v1/decisions
export QUOTE_MODEL_API_KEY=<INCEPTION_API_KEY>

# or Cloudflare Clairic (host defaults to the Workers AI URL for CLOUDFLARE_ACCOUNT_ID)
export QUOTE_ENDPOINT_TYPE=clairic
export CLOUDFLARE_ACCOUNT_ID=<ACCOUNT_ID>
export QUOTE_MODEL_API_KEY=<CLOUDFLARE_API_TOKEN>

model_factory.get_extractor(prefix) reads {prefix}_ENDPOINT_TYPE, so other stages can take an extractor the same way.

How it works:

• SystemOneClient gains ExtractQuestion and ExtractAnswer. Every returned span, from either backend, is checked against the request text (offsets count Unicode code points) and a mismatch is a parse error. Clairic answers without offsets are dropped; a field missing from Clairic's response, or offsets that are not a [start, end] pair, are parse errors rather than an empty answer.
• For each topic, one extract question marks the sentences where the participant gives a view on it: what they want done, why, and any personal experience behind it.
• Marked spans are widened to whole sentences, and sentences that are not adjacent are joined with .... <response> segments are joined first, as --skip_quote_extraction does.
• A topic with nothing marked gets the whole response. Host errors fail the stage rather than silently degrading it.
• With QUOTE_ENDPOINT_TYPE unset the generative path is unchanged, and --skip_quote_extraction still takes precedence.

Evaluation

A 28-response transit survey (38 statement-topic pairs, including multi-part <response> answers and an emoji), comparing span quotes from Mercury with quotes from the existing generative path on local gemma4 (gemma4:26b-a4b-it):

	Span quotes (Mercury)	Generated quotes (gemma4)
Verbatim, allowing bracketed edits	38 of 38	33 of 37
Bracketed edits	0	3
Preferred by Jev (pairwise, both orders averaged)	25	5, with 7 ties
Preferred by pplx-decider:27b (same)	16	3, with 18 ties

The four generated quotes that are not verbatim changed capitalization or punctuation without brackets (for example "It is" became "it is"). One pair got no generated quote because the endpoint errored. The judges are decision models that produced neither set; both show position bias (about half of the pairs flip with order), which is why every comparison is asked in both orders and averaged.

The instruction wording was chosen on this set: an earlier wording that did not mention "what they want done" kept only 3 of 10 sentences in which participants asked for something (for example "Please build protected bike lanes"); the final wording keeps all 10. The set is small, so treat the numbers as a sanity check rather than a benchmark.

Running the quote stage on the same 38 pairs through each backend: Clairic and Mercury both produced 38 quotes, all verbatim, with no <response> fragments. Clairic marked nothing for 1 pair and Mercury for 3; those pairs got the whole response.

Test plan

• pytest (335 tests), including extractor interface tests for both backends (request shape, envelope unwrapping, dropped null spans, span checks, retry), the get_extractor factory, extract parsing and span checks, sentence widening, the QUOTE_* factory and the span path in the quote stage
• Live: span quotes for all 38 pairs are verbatim substrings of the cleaned response, with no <response> fragments and correct offsets after an emoji
• Live: the quote stage with QUOTE_ENDPOINT_TYPE=clairic and with =systemone (Mercury): 38 of 38 quotes verbatim on each
• Live: a host without extract support (local Clef) fails the stage with its 400 message, without retries or fallback
• Full categorization run with span quotes from Mercury, local Clef judgments and local gemma4 generation: exit 0, and every piece of all 39 quotes in the output CSV is an exact substring of the cleaned response. Quote extraction took 7.5 seconds, against about 9 minutes on the generative path with local gemma4.