fix(serializer): make list encoding linear instead of quadratic - #3
Merged
Merged
Conversation
puzza007
force-pushed
the
fix/linear-list-serialization
branch
from
September 17, 2026 22:21
35a7868 to
55be51e
Compare
`encode_data/5` for a list built the result with `acc ++ [encoded_data]`
inside `Enum.map_reduce/3`, copying the accumulator on every element.
Serializing a collection was therefore O(n²) in the number of resources
and dominated response time for large index endpoints: 77k resources
took ~9.6s in the append alone (~25-30s end to end in CoreGateway with
links), which is what pushed
`GET /api/save/:id/relationships/assets` past Maxwell's 60s timeout for
large Edit style trainings (Sentry MAXWELL-7A7T).
Map each element and `Enum.unzip/1` the `{to_include, encoded}` pairs
instead. Same output, same order.
resources before after
10,000 0.15s 0.01s
40,000 2.69s 0.07s
77,011 9.61s 0.17s
The existing list test serialized three identical items and so could
not catch an ordering regression; it now serializes 50 distinct posts
and asserts the order of `data` and of `included`, plus dedup of a
shared comment author.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds stream_data (test only) and three properties pinning the list clause of `encode_data/5` to the single-resource clause, for arbitrary lists of resources with nested includes: - `data` equals each element serialized on its own, in order - `included` equals `flatten_included/1` over each element's `included` - splitting the list into chunks and concatenating gives the same result The id space is deliberately small, and users/comments are drawn from fixed pools, so lists routinely contain repeated resources and structurally identical includes and the dedup path is exercised. Reversing the list in the fix makes all three fail within a few runs. Capped at 25 runs each. The module is not async: the views read process-global Application env that other test modules mutate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`mix hex.audit` (the CI `audit` job, which `is_releasable` depends on) has not run against master since 2025-01 and now fails on advisories published since. None of the flagged packages are runtime deps of a consumer — plug is a library requirement (`~> 1.10`, unchanged) that the consuming app pins itself, and phoenix/earmark are dev/test only — so this only touches the lockfile: - plug 1.16.1 -> 1.17.4 (EEF-CVE-2026-54892, -8468, -56814, -56813) Kept on the 1.17 line: plug 1.19+ needs Elixir >= 1.15 and CI runs 1.14 (1.20.3 fails to compile there). Same line CoreGateway runs. - phoenix 1.7.18 -> 1.7.24 (EEF-CVE-2026-32689, -56811, -56812) - earmark removed. It was an orphan `only: :dev` dep: nothing in lib/ or mix.exs references Earmark, and ex_doc uses earmark_parser. It is retired upstream and carries EEF-CVE-2026-48591. `mix hex.audit`: "No retired or security advisory packages found", so no allowlist is needed. Tests, credo and the formatter are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
puzza007
force-pushed
the
fix/linear-list-serialization
branch
from
September 17, 2026 23:15
55be51e to
dd5b0af
Compare
puzza007
added a commit
that referenced
this pull request
Sep 18, 2026
Cuts the release for #3, whose commit body git_ops could not parse, so the release job on master did not bump the version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
JSONAPI.Serializer.encode_data/5for a list builds its result withacc ++ [encoded_data]insideEnum.map_reduce/3, copying the accumulator on every element. Serializing a collection is therefore O(n²) in the number of resources.This is what is timing out
GET /api/save/:id/relationships/assetsin CoreGateway for large Edit style trainings (Sentry MAXWELL-7A7T, 3 users since 2026-08-31). For a 77k-asset style the serializer alone took ~25–30s on a laptop — more than the DB query, the Conduit transfer (73MB ETF, 0.3s) andJason.encode!(1.2s) combined — and prod gateway hosts are slower, so Maxwell's 60s read timeout fires before the first byte is sent.Fix
Map each element and
Enum.unzip/1the{to_include, encoded}pairs. Same output, same order.Benchmark (
remove_links: true, 5-field view, so this isolates the append):Tests
Example test — the existing
serialize handles a listserialized three identical items so it could not catch an ordering regression. It now serializes 50 distinct posts and asserts the order ofdataand ofincluded, plus dedup of a shared comment author.Property tests (new
stream_datatest-only dep,test/jsonapi/serializer_property_test.exs) pin the list clause to the single-resource clause for arbitrary lists with nested includes:dataequals each element serialized on its own, in orderincludedequalsflatten_included/1over each element'sincludedThe id space is deliberately small, and users/comments are drawn from fixed pools, so lists routinely contain repeated resources and structurally identical includes and the dedup path is actually exercised. Capped at 25 runs each (suite stays ~1s). The module is sync because the views read process-global Application env that other test modules mutate.
Mutation checks: reversing the list makes all three properties and the example test fail; dropping
Enum.uniq/1fromflatten_included/1fails three example tests.mix test: 36 doctests, 3 properties, 110 tests, 0 failures.mix credo --strictclean.Also in this PR
chore(deps): bumpsplugandphoenixin the lockfile and removes the orphanearmarkdev dep somix hex.audit(the CIauditjob) passes again. No runtime dep changes for consumers; plug stays on the 1.17 line because CI runs Elixir 1.14.Follow-ups (not in this PR)
mix.lockpinsjsonapi 1.3.1; the timeout is not fixed there until that pin is bumped after this is released.dasherize/camelizeon every key of every resource. Memoizing the key transformation would take 77k resources from ~5s to well under 1s.++ [x]sites atserializer.ex:122and:133are per-resource/per-relationship and not quadratic in the collection size; left alone.🤖 Generated with Claude Code