Skip to content

fix(serializer): make list encoding linear instead of quadratic - #3

Merged
puzza007 merged 3 commits into
masterfrom
fix/linear-list-serialization
Sep 17, 2026
Merged

puzza007 merged 3 commits into
masterfrom
fix/linear-list-serialization

Conversation

@puzza007

@puzza007 puzza007 commented Sep 17, 2026

Copy link
Copy Markdown
Member

Problem

JSONAPI.Serializer.encode_data/5 for a list builds its result with acc ++ [encoded_data] inside Enum.map_reduce/3, copying the accumulator on every element. Serializing a collection is therefore O(n²) in the number of resources.

This is what is timing out GET /api/save/:id/relationships/assets in CoreGateway for large Edit style trainings (Sentry MAXWELL-7A7T, 3 users since 2026-08-31). For a 77k-asset style the serializer alone took ~25–30s on a laptop — more than the DB query, the Conduit transfer (73MB ETF, 0.3s) and Jason.encode! (1.2s) combined — and prod gateway hosts are slower, so Maxwell's 60s read timeout fires before the first byte is sent.

Fix

Map each element and Enum.unzip/1 the {to_include, encoded} pairs. Same output, same order.

Benchmark (remove_links: true, 5-field view, so this isolates the append):

resources before after
10,000 0.15s 0.01s
40,000 2.69s 0.07s
77,011 9.61s 0.17s

Tests

Example test — the existing serialize handles a list serialized three identical items so it could not catch an ordering regression. It now serializes 50 distinct posts and asserts the order of data and of included, plus dedup of a shared comment author.

Property tests (new stream_data test-only dep, test/jsonapi/serializer_property_test.exs) pin the list clause to the single-resource clause for arbitrary lists with nested includes:

  • data equals each element serialized on its own, in order
  • included equals flatten_included/1 over each element's included
  • splitting the list into chunks and concatenating gives the same result

The id space is deliberately small, and users/comments are drawn from fixed pools, so lists routinely contain repeated resources and structurally identical includes and the dedup path is actually exercised. Capped at 25 runs each (suite stays ~1s). The module is sync because the views read process-global Application env that other test modules mutate.

Mutation checks: reversing the list makes all three properties and the example test fail; dropping Enum.uniq/1 from flatten_included/1 fails three example tests.

mix test: 36 doctests, 3 properties, 110 tests, 0 failures. mix credo --strict clean.

Also in this PR

chore(deps): bumps plug and phoenix in the lockfile and removes the orphan earmark dev dep so mix hex.audit (the CI audit job) passes again. No runtime dep changes for consumers; plug stays on the 1.17 line because CI runs Elixir 1.14.

Follow-ups (not in this PR)

  • CoreGateway's mix.lock pins jsonapi 1.3.1; the timeout is not fixed there until that pin is bumped after this is released.
  • Per-resource cost is still ~55µs, almost all regex dasherize/camelize on every key of every resource. Memoizing the key transformation would take 77k resources from ~5s to well under 1s.
  • Two more ++ [x] sites at serializer.ex:122 and :133 are per-resource/per-relationship and not quadratic in the collection size; left alone.

🤖 Generated with Claude Code

@puzza007
puzza007 force-pushed the fix/linear-list-serialization branch from 35a7868 to 55be51e Compare September 17, 2026 22:21
puzza007 and others added 3 commits September 18, 2026 11:11
`encode_data/5` for a list built the result with `acc ++ [encoded_data]`
inside `Enum.map_reduce/3`, copying the accumulator on every element.
Serializing a collection was therefore O(n²) in the number of resources
and dominated response time for large index endpoints: 77k resources
took ~9.6s in the append alone (~25-30s end to end in CoreGateway with
links), which is what pushed
`GET /api/save/:id/relationships/assets` past Maxwell's 60s timeout for
large Edit style trainings (Sentry MAXWELL-7A7T).

Map each element and `Enum.unzip/1` the `{to_include, encoded}` pairs
instead. Same output, same order.

    resources   before    after
    10,000      0.15s     0.01s
    40,000      2.69s     0.07s
    77,011      9.61s     0.17s

The existing list test serialized three identical items and so could
not catch an ordering regression; it now serializes 50 distinct posts
and asserts the order of `data` and of `included`, plus dedup of a
shared comment author.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds stream_data (test only) and three properties pinning the list clause
of `encode_data/5` to the single-resource clause, for arbitrary lists of
resources with nested includes:

- `data` equals each element serialized on its own, in order
- `included` equals `flatten_included/1` over each element's `included`
- splitting the list into chunks and concatenating gives the same result

The id space is deliberately small, and users/comments are drawn from
fixed pools, so lists routinely contain repeated resources and
structurally identical includes and the dedup path is exercised.
Reversing the list in the fix makes all three fail within a few runs.

Capped at 25 runs each. The module is not async: the views read
process-global Application env that other test modules mutate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`mix hex.audit` (the CI `audit` job, which `is_releasable` depends on)
has not run against master since 2025-01 and now fails on advisories
published since. None of the flagged packages are runtime deps of a
consumer — plug is a library requirement (`~> 1.10`, unchanged) that the
consuming app pins itself, and phoenix/earmark are dev/test only — so this
only touches the lockfile:

- plug     1.16.1 -> 1.17.4  (EEF-CVE-2026-54892, -8468, -56814, -56813)
  Kept on the 1.17 line: plug 1.19+ needs Elixir >= 1.15 and CI runs 1.14
  (1.20.3 fails to compile there). Same line CoreGateway runs.
- phoenix  1.7.18 -> 1.7.24  (EEF-CVE-2026-32689, -56811, -56812)
- earmark  removed. It was an orphan `only: :dev` dep: nothing in lib/ or
  mix.exs references Earmark, and ex_doc uses earmark_parser. It is
  retired upstream and carries EEF-CVE-2026-48591.

`mix hex.audit`: "No retired or security advisory packages found", so no
allowlist is needed. Tests, credo and the formatter are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@puzza007
puzza007 force-pushed the fix/linear-list-serialization branch from 55be51e to dd5b0af Compare September 17, 2026 23:15
@puzza007
puzza007 merged commit 9ad82e3 into master Sep 17, 2026
10 checks passed
puzza007 added a commit that referenced this pull request Sep 18, 2026
Cuts the release for #3, whose commit body git_ops could not parse, so the release job on master did not bump the version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant