Skip to content

fix(python): describe the wire shape in JSON schemas of tagged and flattened enums - #210

Merged
hardbyte merged 4 commits into
mainfrom
antonzherdev/plat-1116-reflectapi-python-json-schema-of-externally-tagged-and
Oct 29, 2026
Merged

hardbyte merged 4 commits into
mainfrom
antonzherdev/plat-1116-reflectapi-python-json-schema-of-externally-tagged-and

Conversation

@antonzherdev

Copy link
Copy Markdown
Contributor

Problem

A FastAPI app that returns generated models gets an OpenAPI spec that looks typed but describes the wrong JSON for externally tagged enums, structs with a flattened enum, and open enums.

Pydantic builds a JSON schema without running a model's custom validators or serializers (the same mechanism as #208). With no return type declared on the serializer, it describes the class's own fields. Several generated classes reshape the wire in those hooks, so their schemas describe the Python fields instead of the wire, in both validation mode (what the model accepts) and serialization mode (what it emits):

Generated class Hooks Wire Schema before
Externally tagged enum (RootModel over variant classes) _parse_externally_tagged / _serialize_externally_tagged "Calm", {"Aggressive": [1.0, "x"]}, {"Other": {...}} the bare variant classes: "Calm", {field_0, field_1}, {description, notes}
Struct with #[serde(flatten)] enum _unflatten_enum / _flatten_enum the enum's keys next to the struct's: {"id": 7, "kind": "Dog", "breed": "corgi"} a required nested value property: {"id": 7, "value": {...}}
Open enum (#[reflectapi(open)]) _classify the known variants (and, forward-compatibly, unknown ones) the known variants or an object {tag: str, raw: any}, which is the client's in-memory placeholder, never a wire value

Internally and adjacently tagged enums (not flattened), untagged unions, field presence, and plain and partial structs were already right.

Example

enum Behavior { Calm, Aggressive(f64, String), Other { description: String, notes: Option<String> } }

Behavior.model_json_schema() (both modes):

before  anyOf: [{"const": "Calm"}, {"$ref": "AggressiveVariant"}, {"$ref": "OtherVariant"}]
        AggressiveVariant = {"type": "object", "properties": {"field_0": number, "field_1": string}}

after   oneOf: [{"const": "Calm", "type": "string"},
                {"type": "object", "required": ["Aggressive"],
                 "properties": {"Aggressive": {"type": "array", "prefixItems": [number, string], "minItems": 2, "maxItems": 2}}},
                {"type": "object", "required": ["Other"], "properties": {"Other": {"$ref": "OtherVariant"}}}]

A struct Envelope { id: u32, #[serde(flatten)] value: Animal } with an internally tagged Animal:

before  {"type": "object", "properties": {"id": integer, "value": {"$ref": "Animal"}}, "required": ["id", "value"]}
after   {"allOf": [{"$ref": "Animal"}, {"type": "object", "properties": {"id": integer}, "required": ["id"]}]}

An optional flattened enum (Option<Animal> with skip_serializing_if) becomes anyOf: [the enum, {"not": {"required": ["kind"]}}], that is, the enum or none of its keys. For an externally tagged enum the presence keys are the variant names, and a unit variant is {"Unit": null}, as serde writes it.

These are the shapes reflectapi's own OpenAPI generator already produces for these representations.

Fix

The generated classes describe their wire shape through __get_pydantic_json_schema__, which calls a runtime helper, in the same style as _parse_externally_tagged:

class Behavior(RootModel[BehaviorVariants]):
    ...
    @classmethod
    def __get_pydantic_json_schema__(cls, core_schema, handler):
        return _externally_tagged_json_schema(
            core_schema, handler, (("Calm", "unit"), ("Aggressive", "tuple"), ("Other", "struct"))
        )
  • externally_tagged_json_schema builds one alternative per variant from the (wire_name, shape) list the codegen emits, in the order of the root union. Struct variants reference the variant class's definition ($ref), tuple variants become prefixItems, newtype variants the inner value's schema.
  • flattened_enum_json_schema takes the struct's default schema, removes the enum's field, and combines the rest with the enum's schema in an allOf. Optional groups may also be absent.
  • known_variants_json_schema drops the unknown case from an open enum's union. UnknownEnumValue's own schema is a marker that this helper removes; Pydantic then garbage-collects its unused definition.

Hooks run for both modes, and none of them changes validation or serialization. Works on Pydantic 2.10.0 through 2.14 (the CI matrix floors and latest).

Open enums

An open enum is now described by its known variants only, like the server's schema describes it. Its model still accepts unknown variants (that is what open means), so the validation-mode schema is stricter than the model for those values; it is accurate for every known value. The alternative, an "anything" branch, would make every open enum an untyped value in generated OpenAPI clients. The Unknown value in the OpenUnit fixture's schema is a real Rust variant, not the placeholder.

Also fixed

A flattened unit variant of an externally tagged enum ({"id": 7, "Unit": null} on the wire) was parsed but failed to serialize (TypeError: 'str' object is not a mapping). _flatten_enum now writes it as "Unit": null.

Tests

  • New CI step "Check generated Python JSON schemas against serde samples" (tests/python/test_json_schema.py). The enum compatibility fixture now writes known_samples.json: serde's own output for every known variant of each of 23 fixtures (externally tagged with unit, newtype, tuple and struct variants; flattened internal, adjacent, external and fallback enums, required and optional; open and fallback enums; untagged; a nested envelope). Each value must round-trip through the generated output model and validate against both schemas. 48 of the 107 cases fail on main; all pass here.
  • test_wire.py also validates the codegen coverage client's serde samples against both schemas (untagged enums, which were fine; kept as a guard).
  • Runtime: tests/test_json_schema.py covers the three helpers on hand-written classes in the generated style. Full runtime suite: 483 passed.
  • Every Python step of the python-codegen-smoke job passes locally for all four matrix entries (3.12 and 3.14, latest and minimum dependencies), plus cargo fmt, clippy -D warnings, cargo test --workspace, and the Rust compatibility crates.
  • A pytest plugin that recorded every wire value the existing test suites validate or dump and checked it against both schemas found 8 mismatches for known variants on main (5 in the enum compatibility suite, 3 in the demo client tests) and none here. What remains is unknown-variant values of open and #[serde(other)] enums, by the choice above.

Not changed

  • The new test checks output models. On the input side, #[serde(other)] enums deliberately have no fallback variant (the existing request policy), and some flattened closed enums do not round-trip serde's wire today, independent of this change: FlattenAdjacentEnvelope and NestedFlattenEnvelope reject their flattened form ("Field required: value"), OptionalFlattenEnumEnvelope rejects {"id": 7}, and OptionalFlattenAdjacentEnvelope drops a unit variant ({"id": 7, "kind": "Unit"} becomes {"id": 7}). Worth a separate issue.
  • In validation mode, an internally tagged variant's tag is optional in the schema because the field has a default. That was the case before and is unchanged.
  • Snapshots and the committed demo client are regenerated; the diff is the new imports and hooks.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q7zEtTtfj5pVB2LpkShhWS

Anton Zherdev and others added 2 commits October 27, 2026 17:24
…attened enums

Pydantic builds a model's JSON schema from its fields and never runs its
custom validators or serializers. Generated classes that reshape the wire
in those hooks advertised the shape of their Python fields instead, in both
validation and serialization mode:

- Externally tagged enums showed each variant's class, such as
  `{field_0, field_1}` for a tuple variant, instead of `"Unit"`,
  `{"Tuple": [a, b]}`, `{"Newtype": value}` and `{"Struct": {...}}`.
- Structs with a flattened enum showed a required nested `value`
  property instead of the enum's keys next to the struct's own.
- Open enums listed their client-side unknown case as an object with
  `tag` and `raw`.

The generated classes now describe the wire through
`__get_pydantic_json_schema__`, which calls a runtime helper:
`externally_tagged_json_schema`, `flattened_enum_json_schema` and
`known_variants_json_schema`. An open enum is described by its known
variants, as the server's schema describes it.

A flattened unit variant of an externally tagged enum now serializes as
`"Unit": null`, as serde writes it, instead of failing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q7zEtTtfj5pVB2LpkShhWS
The enum compatibility fixture now also writes `known_samples.json`:
serde's output for every known variant of each fixture. A new CI step
round-trips each value through the generated output model and validates
it against both the validation-mode and serialization-mode JSON schema.
The serde samples of the codegen coverage client get the same check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q7zEtTtfj5pVB2LpkShhWS
@antonzherdev
antonzherdev requested a review from hardbyte October 27, 2026 17:25

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

📖 Documentation Preview: https://reflectapi-docs-preview-pr-210.partly.workers.dev

Updated automatically from commit 232b50f

@antonzherdev

Copy link
Copy Markdown
Contributor Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

A struct that flattens an externally tagged enum of unit variants only got a
JSON schema no value matches: the enum is a string enum, so its schema stayed
a string inside the struct's object. It is now one `{"Tag": null}` object per
variant, as serde writes it. Its generated model also rejected that wire value,
handing the string enum `{"Tag": None}`; it now takes the tag.

An unknown enum case's schema was a bare marker, which JSON Schema reads as
matching anything, so an open enum generated without the
`known_variants_json_schema` hook (a client generated before it) became
untyped. The marker now matches nothing, and such an enum describes its known
variants only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q7zEtTtfj5pVB2LpkShhWS

@hardbyte hardbyte left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wire-to-JSON-Schema rules now exist twice. They live once in openapi.rs (from reflectapi’s schema) and again in serde.py (from Pydantic’s schema plus hints from the generator). They agree today, and testing against serde’s samples is the right safeguard. Under the #129 direction, though, the variant shapes (unit/newtype/tuple/struct) should come from shared code next to #204’s Layout, so both generators read the same facts.

@hardbyte
hardbyte merged commit 53f50d1 into main Oct 29, 2026
11 checks passed
@antonzherdev
antonzherdev deleted the antonzherdev/plat-1116-reflectapi-python-json-schema-of-externally-tagged-and branch October 29, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants