Add an experimental automatic retrieval path

[jwils]

Why

ElasticGraph should choose efficient field retrieval without asking schema authors to select datastore APIs. Narrow scalar queries may benefit from avoiding source retrieval.

What

Add an opt-in automatic retrieval experiment, comparison harnesses, and a read-only compatibility preflight. Source retrieval remains the default, and returnable data stays stored in _source.

How

The GraphQL instance injects a source or automatic planner through the query builder. It plans against the final merged selection. Extensions can wrap it per instance, while document normalization keeps field resolvers independent of retrieval APIs.

Automatic mode uses doc values only when every requested payload field is an eligible direct scalar. Mixed selections, lists, objects, highlighting, unsupported mappings, and requests exceeding 100 fields retain source retrieval.

Before opting in, regenerate artifacts and run verify_indices.rb with the application's settings. It checks queryable physical indices, all matching rollover generations, and templates. Then validate historical data and compare complete GraphQL results using representative application queries. The benchmark and preflight checks now run explicitly in the supported-datastore CI jobs.

Risk

Shared query planning and response normalization change in both modes. The preflight checks current mappings and settings; it cannot prove historical data compatibility. Small-document or wide-selection queries may regress. The 100-field cap is a request limit, not a measured performance cutoff.

Testing

Before this rebase, the full suite passed: 5,143 examples, zero failures, and 100% line and branch coverage. All 14 retrieval acceptance, GraphQL harness, and live preflight checks passed on Elasticsearch 9.0.0/9.4.2 and OpenSearch 2.19.0/3.6.0. After rebasing, type checking, lint, spellcheck, and schema/configuration artifact verification passed.

The September 10 standalone benchmark ran twice on Elasticsearch 9.0.0. With 64 KiB source padding and 100 hits, source-free doc values reduced median client time by 12–22%. Mixed retrieval ranged from about 1% faster to 5% slower.

The GraphQL harness also ran twice against 2,000 large-source Widget documents, with 100 hits, 20 warmups, and 100 samples per mode. Scalar-query median execution time fell by 23–24%; complete results matched on every execution. These fixture-specific results do not establish a production policy.

Usage and commands · Results and limitations

Bigger picture

The rollout defines application qualification, canary stages, rollback thresholds, and graduation criteria. No consuming application has qualified for production yet. A default automatic policy requires maintained physical compatibility information and repeatable workload-level gains under concurrency. Keep storage reduction separate.

The design explains the roles of all four retrieval APIs and the planner's place in the resolver-built query pipeline.

Stack Info: Originally stacked on #1108 (merged); now targets main.

[myronmarston]

retrieved_from: :doc_values exposes a level of detail that, until now, we haven't exposed to ElasticGraph users. ElasticGraph does many things to query the data as efficiently as possible, without asking users to configure those things.

Can we do something similar here? For example, can we determine in which situations it's more efficient to use doc_values and automatically do so without requiring users to configure it at the field level?

Also, if we're going to change how fields are retrieved, I'd like to take a step back and review a full design that takes into account all the options Elasticsearch/OpenSearch offer us here. Besides _source and docvalue_fields there's also fields and stored_fields. I'd like to consider a unified design rather than just tacking this on as a one-off.