Skip to content

Improve broad-query recall and source evidence - #357

Draft
forhappy wants to merge 38 commits into
mainfrom
codex/broad-query-quality
Draft

forhappy wants to merge 38 commits into
mainfrom
codex/broad-query-quality

Conversation

@forhappy

@forhappy forhappy commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Broad questions can miss implementations when they do not name exact symbols, or find a coordinator while omitting its delegated helpers. Add bounded native fielded retrieval, whole-question ranking, source-owned excerpts and an opt-in top-five shortlist shared by CLI and MCP. Eligible questions about one or more operations with only Low-relevance answers can also recover helper inspection leads through exact native call paths. Paths and previews remain explicitly unverified navigation, without transferring caller topics or relevance to a helper.

  • Treat ordinary Why/why/WHY as broad-question scaffolding. Previously a capitalized interrogative could become a missing named subject and suppress an otherwise relevant implementation. Preserve quoted/qualified subject guards and exact symbols named Why; refresh disposable responses under ranker29 without rebuilding graphs. This retrieves inspection evidence, not a causal explanation.
  • Honor explicit exact source targets. An inadmissible target remains unresolved instead of falling back to an unrelated same-spelled lexical, module or external declaration. This removes fabricated calls and references when a callable parameter shadows an outer function. Missing target IDs still fail ordinary evidence validation. AST cache34 refreshes current disposable graphs; historical realizations remain unchanged.
  • Publish immutable source-query/8-callable-comments evidence with up to 64 anchored comments per callable, retaining independent field/file/artifact bounds, ownership, omission counts and historical-profile capabilities. JSON/store retrieval shares the same source evidence and corpus statistics.
  • Rank using whole-question coverage, corpus frequencies and complete owned comment passages. Select exact source quotations using action coverage and known positive group frequencies, without borrowing sibling text or manufacturing matches from clipped prefixes. Bounded forward spelling hints retain literal topic guards, explicit mapping disclosure and Low relevance when dependent on hints.
  • Recognize ordinary rebuilt/rebuild and written/write spellings while retaining original query groups and explicit identity guards. Keep build/rebuild and write/rewrite distinct; sent/send already uses the existing canonical token profile. These spelling hints preserve action/subject classifiers.
  • Preserve discovery's typed exact-name/ID tier in ranked structural answers. Identity labels remain separate from relevance, ambiguity and graph coverage; an exact structural seed can still have Low relevance.
  • Continue raw discovery through bounded exact source-witnessed calls. Automatic deeper ordinary context starts only at High-relevance selected roots; explicit inspection retains bounded witnessed paths without promoting relevance. The separate new helper-candidate channel uses source-owned root topics, verified callable endpoints and exact AST call occurrences. Keep explicit-name, negative/temporal, scope, ambiguity and strong-answer guards. Default depth remains2; helper navigation honors the request up to4 hops.
  • Retain compact root/ordered-hop IDs, selected call sites and native attribution in callLink, with Low/Approximate helper labels. Only displayed helper paths retain context records after presentation caps, while work counts persist. Show a linked_callable_preview only when a displayed helper has no question-matching quote: direct immutable owner windows, at most16×512bytes/64 comment ranges, source-order ties, and no live-source reads or query-term/ranking changes. Strict consumers reject non-callable preview kinds, inconsistent paths and unsupported profiles.
  • Meter additional helper Calls lookup attempts before the backend, including empty/filtered/failed calls: a shared40-probe ceiling beside unchanged indexed recall291. Raw discovery2/work1 and ranked-view7 retain the two ledgers even when no helper survives. These are logical API work counts, not physical I/O or confidence. Bound linked metadata before copying and read store adjacency IDs before hydrating only the retained records; preserve typed limit/corruption errors.
  • Correct unconfigured Python module identities: bare roots use relative modules, verified package roots retain their namespace, and configured roots retain precedence/ambiguity. Preserve script/group source identities with dot/percent component escaping and reject unsupported package-entry layouts explicitly.
  • Preserve Rust literal/type references, scoped receiver/generic-bound identity, ambiguity and exact occurrences. Explicit deferred receivers cannot become a same-named local method solely because that method is unique.
  • Carry supported TypeScript generic arguments through definition-bound imported fields, aliases/substitutions and utility types. Keep export obligations, physical alternatives and original-syntax inventories explicit. Bound proof/load payloads before cloning and charge export walks/fixed-point closure to shared limits. Aggregate overflow retains eligible exact edges through explicit degraded partitioning; incomplete or truncated proofs cannot select a winner. These development controls do not establish compiler-wide package/type/runtime parity or language promotion.

Structural extraction and queries remain native and local, without Graphify, Python, credentials, embeddings, vector databases or grammar downloads at runtime.

Verification

All twelve current native/build/check groups pass on 837 byte-identical Rust/Cargo inputs in the authoritative and build worktrees. Compilation ran only in the build worktree's own external target; this is not a separate authoritative-worktree build:

  • Formatting and workspace library/binary Clippy with warnings denied. Both changed test targets (broad_retrieval and code_query_cli) also pass targeted Clippy with warnings denied.
  • 1,330 workspace library/binary tests, with two existing ignores across 36 executables.
  • 483 query tests across30 executables, including a retained RED/GREEN Why-question regression, case parity, JSON/store equality, reordered graph input and quoted/qualified/exact identity guards. A real-binary CLI regression covers both storage configurations, repeated requests and bounded source-cited text.
  • The unchanged exact-binding resolver slice previously passed 661 tests across40 executables at64b4c12c, including its three publication/validation controls and test-target Clippy; these prior results are not claimed as a fresh resolver-suite run for the query-only correction.
  • 65 CLI query, 18 MCP query and nine product tests, plus the product-boundary check in both worktrees.
  • The complete qualify_code_graph_v1.sh --fixtures-only gate, including lifecycle, scale, semantic/topology, Markdown and nested React checks.

A retained two-file authored fixture verifies the actual CLI transition: the old graph has 24 nodes/40 edges; ordinary update with the new binary removes four false calls and two false references to shadowed outer functions, preserving every node and legitimate edge byte-for-byte. The old binary reuses its cached sources; the new binary reextracts under cache34 without --force. Warm and fresh new builds have identical nodes and edges. Original source, old output, frozen binaries and all native inputs remain intact. This tests denial of unrelated targets; it does not add parameter references or identify their runtime implementations.

All 679 benchmark-harness tests, including 40 focused synthetic conditional-bound controls, pass on unchanged harness source. Scoped product-source reviews found no required correction to the exact-target denial or its cache/validation ownership; they do not certify execution, independent source meaning or comparative quality. Earlier helper/navigation controls remain attributed to their unchanged query source and retained verification. An additional source-only review of the eight Why-question product/test/doc files found no required correction; all frozen hashes match. That review saw no runtime or development-evaluation outcomes and does not certify empirical quality.

The published 64b4c12c CI run completed with 12 successes, including quality, full graph fixtures and all six native platforms, and two dependency failures. It predates this Why-question correction; new-head CI must run after publication. Rustls 0.23.46 clears its four findings. The remaining local policy errors are six ZIP vulnerability records (three advisories on each of ZIP6/7) and unmaintained unsafe-libyaml. Bans, licenses and sources pass; no waiver was added. ZIP7 is transitive through Candle, so upgrading only the direct ZIP dependency is insufficient.

The unchanged broader query-test lint failure at crates/compass-query/tests/explanation_members.rs:289 and historical strict language-promotion failure remain separate from the passing required native baseline. Development producers remain Qualifying. Neither failure is waived. Historical source-review aggregate/incidental disclosures remain recorded; no new real answers or comparison outcomes were supplied to the product-source reviewer. This PR remains draft, with no release or promotion claim.

Quality evidence and remaining work

The prospective developer graph-precision design2/diagnostics2 path adds exact integer hypergeometric tail inversion and simultaneous Bonferroni bounds over captured emitted-record populations. It uses population-size weights, rational difference endpoints, adversarial completion of unknown labels, full withholding after failed builds and bounded arithmetic. Legacy design1/result1 retain their shapes and null certified-interval fields. These calculations are conditional on ideal uniform sampling and truthful labels; the hash sampler does not establish those assumptions or independent review. Design acceptance, certified precision noninferiority, power and the final joint graph/query decision remain unfinished. This is developer tooling, with no Graphify runtime dependency. All679 harness tests, including40 focused synthetic controls, pass for this benchmark component.

Superiority over Graphify remains unproven: zero accepted independent qualification judgments.

A separately authorized evaluator froze six source-first Flask development questions (five supported, one unsupported control) before answers. The Why correction moves the relevant exception handler from an admission filter to rank1 on unchanged graph/source bytes. Required implementation owners in the first five improve 6/10 to 7/10; the other five questions are byte-identical across all15 default/ranked-text/ranked-JSON surfaces. The same frozen fact obligations still yield 0/5 complete behavioral answers: ranked JSON covers3/16 facts fully,6 partly and omits7. The unsupported control makes no invented retry claim but still lacks explicit premise rejection. Two admission failures, a rank10 lifecycle result, quotation omissions, partially omitted long-function windows and a first-line-only Python docstring anchor remain follow-up findings. The full docstring text and correct ownership edge are published; the cause of absent owned-document enrichment is unproven. This small exposed development batch is not held-out or comparative qualification. The owner-binding adapter delimiter correction and one explicitly disclosed evaluator executable-hash read outside the batch directory are retained in the evidence.

Eligible single-operation questions use a bounded lexical operation-head heuristic over the existing owning action classification and full query. Noun-like modifiers removed from navigation hints cannot become independent root-topic credit. Retain up to eight best exact target alternatives per root and display at most two distinct helpers; repeated paths retain one best path, with hint diversity, shortest length and stable occurrence IDs deciding order. More same-operation targets do not alone promote a root. This remains approximate/Low inspection navigation, not English parsing, intent resolution or proof of requested behavior. Current/best root maps can coexist; eight is not a global allocation bound.

The retained own-repository single-operation development replay at 9a79b45b includes validate_with_limits at rank2 with explicit depth4, its exact three-hop source witness and a215byte retained code preview. At default depth2 it reaches the validate wrapper. Adding context preserves the complete ranked response and work counters. These are exposed development observations, not independent answer judgments, general recall, semantic truth or performance comparisons. Native alternative controls separately require two distinct same-operation targets, shortest source paths, stable occurrence selection, backend semantic parity, truthful backend-specific indexed work, and coherent helper-work counters. The earlier detection-vocabulary/partial-filename and context-only experiments remain excluded; optional LSA remains an explicit native opt-in, not a runtime requirement.

Retain the original joint query/graph/cost qualification gates: at least +10 percentage points first-page essential-fact success with paired 95% lower bound above zero; claim-precision and exact/directed noninferiority margin −2 points; independent graph precision/recall, language/relation and occurrence audits, including at least 2000 accepted judgments, precision≥0.995 and Wilson 95% lower≥0.99; and original/lookup/union, output/budget/cost, build/query latency and RSS requirements. A paired graph endpoint, accepted independent source/answer/graph judgments, prospective power and an adequately powered untouched panel remain unfinished. Reverify and freeze the latest Graphify when starting final qualification. The latest development pin is v0.9.92, not a final comparison freeze.

692 observations remain unopened/unassigned and174 previously exposed registered tasks remain excluded. In addition to4 earlier external development answers, this new six-task source-first development batch and its two derived probes are excluded from future held-out claims:36 primary-view captures,36 stage traces and one external graph build. The earlier135 ordinary own-repository captures, one additional warmed JSON-startup diagnostic, and38 context attempts (37 successful/1 historical invalid flag) remain development inputs; the pre-query stale-manifest harness failure is retained separately and executed no query. Authored fixtures are separate from independent judgments. Grep matches, graph size, tests and source approval do not establish semantic truth, independence or comparative superiority. Developer graph helpers retain failures/unreviewed records and emit qualificationEstablished=false.

Compatibility

Source-query profile8 and AST cache34 retain documented rebuild/older-reader requirements; profiles1–7 and historical realizations keep their original meaning. The current broad-query slice uses ranker29, call-link/work1, raw discovery2 when metered, ranked-view7 when metered, and trace2 when metered; unmetered raw1/ranked4/trace1 remain. Experimental unpublished ranked5/6 are unsupported. Strict consumers must upgrade for the new raw/ranked majors and validate both work ledgers. The generic Agent View shape remains major1 with the actual raw identity and an informational work caveat; canonical counters remain in raw stats.

Ranker29, quotation epoch6-linked-preview and call-link epoch7-classified-operation-heads refresh disposable responses. Existing spelling/identity behavior remains. Query-only changes require no source rebuild by themselves; missing graph/source evidence still requires compass update <root>. Compatibility/reference docs, migration guidance, help/MCP descriptions and changelog specify the new profile/preview bounds. Operational stats remain outside semantic result digests; a raw major still distinguishes metered responses. No immutable historical graph is rewritten.

Python 2, TypeScript 5/JavaScript 4, Rust 4 and Go 2 development producers remain unpromoted. Evidence/2 includes the documented typed argument/load/inventory capabilities; strict older readers must upgrade. Fact-digest profile 4 refreshes current disposable state without rewriting history. Graph-v1 shape and provenance remain intact. Compatibility/reference docs, migration guidance and changelog describe the relevant changes. Local evidence, generated graphs, competitor artifacts, machine configuration and private source content are excluded.

Index bounded source-owned lexical evidence and rank implementation candidates without requiring exact symbols. Add opt-in CLI and MCP shortlists with source citations, relevance reasons, and whole-response budgets.

Preserve deterministic JSON/store contracts and exact identity precedence. Document cache/profile migrations, trust boundaries, and incomplete comparative qualification.
Add bounded capture, source-oracle, review, exposure-lineage, and prospective-power tooling for broad questions. Preserve frozen development registrations and mark their reused sources ineligible for confirmation.

Keep comparator execution external to the native product and tests. Independent query/graph qualification and a fresh latest-release comparison remain incomplete.
Long-form words elsewhere in the index suppressed abbreviated owners and caused coverage vetoes. Add bounded directional parameter, dictionary, second and nanosecond hints with separate literal context, Low relevance, explicit translations and identity guards.

Refresh response caches with ranker15. Native query/CLI regressions and the full workspace baseline pass. Unchanged exposed questions gain three first-five targets without losses; independent Graphify qualification remains incomplete.
Graph-only copies dropped source-body recall. Bind and retain each source-query sibling beside its graph, reject implicit inputs and recheck both original and retained artifacts.

Freeze exact native no-match exits prospectively. Keep old results immutable and retain all errors, budgets and raw outcomes.
Keep directional vocabulary hints tentative and disclosed. Preserve literal fields and source-stage dependencies so hints cannot inflate ranking or relevance.
Preserve historical lexical projections and keep code-dependent leads Low.
Separate identifier completeness from omitted code windows so existing
inspection preferences survive without hiding truncation.
Separate bindings from lexical mentions so defaults, annotations and
nested scopes cannot establish noun witnesses. Preserve early source
digests across conservative cache fallback; bound mismatch diagnostics.

Source profile6/AST26 require updated readers. Historical profiles keep
their original meaning; parameter roles do not change retrieval scores.

Refs #357
Decision words alone admitted unrelated owners. Use typed parameter
bindings and exact nominal declarations for ambiguous noun leads.
Keep tentative matches Low, preserve exact identities, and refresh
response caches with ranker21.

Refs #357
Publish source-query7/cache27 so raw literals can seed bounded native
code windows. Preserve historical profile meanings and ownership limits.

Refresh clipped incremental evidence through full cached/fresh extraction
to keep clean and updated artifacts equal. Native baseline and fixture
qualification pass; comparative query and graph quality remain unproven.
Retain every printed occurrence, typed source assertion and incomplete
precision state across bounded claim batches. Freeze the owned evaluator
source inventory so imported helper changes invalidate the actor tuple.

Keep public capture, assessment and panels on protocol1 until version,
lineage, full-matrix publication and statistical integration are complete.
Retain literal type references and receiver origins across lexical scopes.
Resolve complete generic bound inventories project-wide without borrowing
unproven targets or collapsing deferred callees by terminal name.
@forhappy forhappy changed the title Improve broad-query retrieval and ranked source evidence Improve broad-query retrieval and source graph evidence Oct 19, 2026
Carry definition-bound generic arguments through imported receivers so
unknown or shadowed arguments cannot inherit a same-spelled local type.
Require complete bounded export obligations and source inventories before
selecting a target; preserve unavailable branches and exact occurrences.

Bound proof/load retention before cloning. On aggregate overflow, retain
eligible exact edges with an explicit omission report. Unsupported type
semantics remain unresolved; development producers stay Qualifying.

Rust API migration: materialize returns a must-use omission report.
Strict evidence readers must accept the new typed and inventory fields.
AST cache31 and fact-digest profile4 refresh disposable current state;
historical realizations and release-promotion records remain unchanged.
Use small real files and private admission limits to avoid allocating
multi-GiB fixtures on Windows. Preserve production caps and verify
manifest accounting, rejection before copying, and complete publication.
Use portable separators for exact source keys and relative export
candidates. Keep Unix literal backslashes distinct from directories.
Retain unknown source facts and reject duplicate units. Reuse the
bounded repository/cluster estimator and expose micro recall separately.

These diagnostics do not establish graph qualification or joint
superiority; artifact bindings, precision and power gates remain open.
Verify source spans, tool profiles, graph records and bounded build
captures against frozen references before computing paired coverage.
Keep failed builds and unreviewed facts unknown; recheck all inputs.

Byte integrity does not establish semantic correctness or independent
qualification. Precision, power and the joint decision gate remain open.
Freeze edge quotas and a portable SHA-256/Floyd sampler before capture.
Retain unknown and unsampled records, source witnesses and raw record
hashes. Separate sample, weighted-point and census precision.

No population confidence interval or joint qualification is inferred.
Eight introductory comments could exhaust recall before the relevant
implementation explanation. Retain up to 64 with existing field, file
and artifact bounds; keep historical profiles at their original limits.

Rebuild current evidence with source-query profile 8 and AST cache 32.
Run compass update <root>; older readers must upgrade for profile 8.
Keep partial lexical roots at first-hop unless context inspection is explicit.
Select source windows using complete original tokens so clipped prefixes cannot
create quotation evidence. Preserve source anchors and refresh response caches.
Broad questions missed irregular operation forms, and ranked adapters
lost discovery's typed exact-name/ID tier. Preserve literal lookup,
operation distinctions, relevance and graph coverage.
Recover delegated inspection candidates through exact native calls.
Keep caller topics separate from helper matches and Low relevance.
Bound additional lookups and metadata, and retain source-owned previews.

BREAKING CHANGE: metered discovery emits discovery/2 and ranked-view/7.
Strict consumers must upgrade and validate the versioned work ledgers.
Unmetered discovery/1 and ranked-view/4 retain their existing meanings.
@forhappy forhappy changed the title Improve broad-query retrieval and source graph evidence Improve broad-query recall and source evidence Oct 19, 2026
Single-operation broad questions could miss source-witnessed helpers that
were recovered when the question asked for two operations. Select bounded
lexical operation heads without changing full-question matching or root
topic credit, and retain distinct native target alternatives with Low
relevance and deterministic shortest-path ties.

Cover repeated/native alternative paths, filtered-topic guards, selector
ties, and raw/ranked CLI and MCP projections. Refresh disposable ranker
responses and document the limits without claiming comparative quality.
Raise the workspace minimum and lock Rustls 0.23.46 for the four upstream
RustSec fixes. Preserve TLS features and every other locked package.

The native workspace baseline and database-client controls pass. ZIP and
unsafe-libyaml still fail dependency policy; add no waiver or claim that
all dependency findings are resolved.
Add opt-in prospective design2 with exact hypergeometric test inversion
and simultaneous population-count bounds. Preserve unknown labels and
failed builds, rational endpoints, bounded work and legacy output shape.
Bind the complete calculation source before capture and recheck it after.

These calculations remain conditional on ideal sampling and truthful
labels. Design acceptance, independence, NI and joint qualification are
not established; numeric qualification gates are unchanged.
Keep rejected exact targets unresolved instead of substituting same-named declarations. Refresh current graphs with AST cache34 while preserving historical realizations.
Capitalized Why was treated as a missing named subject.
Preserve literal identities and refresh disposable responses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant