Skip to content

perf(nixprofile): read manifest.json instead of running nix profile list - #3030

Merged
mikeland73 merged 2 commits into
mainfrom
mikeland73/perf-read-profile-manifest
Oct 29, 2026
Merged

mikeland73 merged 2 commits into
mainfrom
mikeland73/perf-read-profile-manifest

Conversation

@mikeland73

@mikeland73 mikeland73 commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Implements item 10 of specs/perf-improvements.md.

nixprofile.ProfileListItems ran nix profile list --json on every state
recompute, which took ~75ms. In nix >= 2.20, that command parses
<profile>/manifest.json and prints it back out (ProfileManifest::toJSON
in src/nix/profile.cc). The file nix writes uses the same toJSON, so for
a version 3 manifest the two are the same. ProfileListItems now reads the
file and only runs the command when it can't be sure.

  • Version 3 only. Versions 1 and 2 store elements in an array. nix >=
    2.20 derives a name for each one when it reads them (getNameFromURL,
    plus -1, -2 suffixes for duplicates), so the command returns named
    items while the file only has indexes. Rather than copy that logic, those
    versions go to the command. nix rewrites them as version 3 the next time
    it changes the profile.
  • Fallback to the command for unknown versions, files that don't parse
    into the expected types, elements missing active (nix fails on those
    today), and a read error other than "not found".
  • Matching what nix prints: store paths are sorted and deduplicated (nix
    keeps them in a sorted set), a missing storePaths becomes an empty list,
    and originalUrl/attrPath are ignored when url is empty (nix drops the
    source then).
  • Missing manifest: like nix, a profile with no manifest.json and no
    manifest.nix is empty. That covers a profile that doesn't exist yet and
    a default -> default-N-link that points at a garbage collected store
    path. A manifest.nix (nix-env profile) goes to the command. The profile
    symlinks are followed by the normal file read, the same way nix reads
    profile / "manifest.json".
  • The command path is unchanged apart from sharing the element struct and
    the element-to-item conversion.

The only production caller is syncNixProfileFromFlake, which only uses
the store paths. ProfileListNameOrIndex has no callers.

Testing

  • New tests with manifest fixtures: version 3 (store path element, flake
    element, two-output element with unsorted store paths, empty url,
    duplicate store paths), versions 1, 2, 4, missing version and malformed
    files (all fall back), the profile -> profile-1-link -> generation layout,
    missing / dangling / empty profiles (no command run), and fallback for v1,
    v2, v4, bad JSON, manifest.nix and an unreadable manifest (stubbed
    command).
  • TestManifestMatchesNixProfileList writes the fixtures to temp profiles
    and checks that reading the file gives the same items as
    profileListItemsFromCommand. It needs nix >= 2.20 and is skipped
    otherwise. nix only parses the store paths, so they don't need to exist.
  • Real-world comparison with nix 2.30.2, using a throwaway test (not
    committed) that ran both the file and the command paths and compared the
    items: 72 profiles on my machine. 70 matched: 10 devbox project and
    global profiles with 1–17 packages; a profile made with nix profile add
    of nixpkgs#hello, nixpkgs#jq^bin,man and a store path, so it has
    flake URLs and multiple outputs; 57 devbox profiles whose generation was
    garbage collected; ~/.nix-profile, which points at a missing
    directory; and a path that doesn't exist. The last three kinds give no
    items either way. The other 2 were hand-made v1 and v2 manifests, which
    fell back to the command as intended. I also diffed jq -S of manifest.json
    against nix profile list --json for all of the v3 profiles, and they
    were byte-for-byte the same.
  • End to end with a build of this branch: in a copy of this repo's
    devbox.json, removing a package and running devbox install removed it
    from the profile, adding it back installed it, and recomputes with no
    changes left the profile generation alone.
  • go test ./internal/nix/... ./internal/devbox/ passes. go tool golangci-lint run ./internal/nix/nixprofile/... ./internal/devbox/...
    reports 0 issues.
  • Timing, macOS arm64, profile with 4 packages:
main this PR
ProfileListItems (benchmark, 30 runs) 83.4ms 0.037ms
ProfileListItems function timer during devbox shellenv --recompute (10 runs) 71–122ms, median ~75ms under the timer's 1ms print threshold

Not handled

  • nix re-normalizes flake refs when it reads them (for example
    originalUrl: "nixpkgs" prints as flake:nixpkgs). Files nix writes
    already hold the normalized form, so this only matters for a manifest
    written by a nix whose normalization differs from the one installed now.
    Even then only unlockedReference/lockedReference could differ, and the
    only caller doesn't use them.
  • If someone downgrades nix below 2.20 while a version 3 manifest exists,
    the command fails today. Now the file is still read. Later nix profile
    calls would fail the same way as before.

🤖 Generated with Claude Code

mikeland73 and others added 2 commits October 29, 2026 14:09
Implements item 10 of specs/perf-improvements.md.

ProfileListItems ran `nix profile list --json` (~75ms) on every state
recompute. For a version 3 manifest (nix >= 2.20), that command reads
<profile>/manifest.json and prints it back out in the same format, so
ProfileListItems now reads the file instead.

- Only version 3 is read directly. Versions 1 and 2 keep elements in an
  array and newer nix derives names for them when reading, so those go to
  nix, as do unknown versions, unparseable files, and elements nix would
  reject (missing "active").
- Store paths are sorted and deduplicated, and originalUrl/attrPath are
  dropped when url is empty, matching what nix prints.
- A missing manifest (no profile yet, or a link to a garbage collected
  generation) returns no items, like nix. A manifest.nix (nix-env
  profile) goes to nix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Oct 29, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@mikeland73

Copy link
Copy Markdown
Collaborator Author

@claude ptal

@claude

claude Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Claude finished @mikeland73's task in 26s —— View job


Review of #3030

I found no blocking issues. I read profile.go and manifest_test.go in full. I didn't run the tests or check the nix profile list claims against nix itself. I relied on the PR description and the nix-gated TestManifestMatchesNixProfileList for that. The git history in this checkout is shallow, so I couldn't diff against origin/main.

What looks right

  • The fallback rules are conservative. Anything other than a clean v3 manifest goes to the command: other versions, parse or type errors, a missing active, and read errors other than not-found. A null element decodes to a zero struct with Active == nil, so it falls back too.
  • Using *bool for Active is a good way to detect a missing field. It also keeps the command path unchanged, because item() doesn't look at Active.
  • The sorted, deduplicated, never-nil storePaths handling and the "empty url drops originalUrl and attrPath" rule match the toJSON behavior you describe.
  • An absent manifest returns a non-nil empty slice, and manifest.nix still goes to nix. The tests cover both cases, including the dangling default -> default-N-link layout.
  • The tests are thorough. stubProfileListCommand(t, false) makes any accidental command run fail the test. The nix-gated comparison test checks the file path against the real command.

Nits (optional)

  1. profileListElement.Priority is parsed but never used. It does make a wrong-typed priority fall back (the "v3 wrong type" case), so keep it if that's intended. A comment saying so would help the next reader.
  2. In parseManifest, element.StorePaths is sorted in place. That's fine because the slice is freshly unmarshaled, but item() now relies on the caller having normalized it. A one-line note on item() would make that explicit.
  3. "elements": null decodes to an empty map and returns zero items. nix may reject that, though nix never writes it, so this is purely theoretical.
  4. In the "Not handled" section, the nix < 2.20 downgrade case is a behavior change. A v3 manifest that used to fail through the command now succeeds. That seems fine, and arguably better.

Overall this looks good to merge, assuming CI is green.

@mikeland73
mikeland73 merged commit 6968e53 into main Oct 29, 2026
28 checks passed
@mikeland73
mikeland73 deleted the mikeland73/perf-read-profile-manifest branch October 29, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant