Repository navigation
Speed up the versions endpoints of the Collection API - #1288
Open
clementbiron wants to merge 8 commits into
Open
clementbiron wants to merge 8 commits into
clementbiron wants to merge 8 commits into
Conversation
Version lists and contents weigh tens of kilobytes of JSON, and the reverse proxy of the deployment does not compress them. Compressing them in the API divides their size by about six. Partial responses, whose Content-Range counts uncompressed bytes, and responses whose media type is not compressible, such as the dataset archive, are left untouched, so that range requests keep working.
A repository opened for reading ran its Git processes one at a time, like the writer, so simultaneous API requests waited for each other. Read-only commands write neither the index nor the commit-graph, so a reader now runs up to four of them at once. The writer still runs its commands one at a time.
Readers relied on the core.quotePath setting written by the tracker. A repository that no writer configured, such as a plain clone, printed paths holding non-ASCII characters quoted, which altered service IDs such as "service·A". Readers now disable pathname quoting themselves.
Listing the versions of some terms spawned one git show per version to read its diff statistics, up to 500 per page. A single git show now reads the statistics of the whole page, through a new getDiffStatsBatch repository method. Repositories that cannot provide them answer with null values, as they do for getDiffStats.
Records of a service or of some terms were filtered with wildcard pathspecs such as "Service/Terms.*", which prevent Git from using the changed-path Bloom filters of the commit-graph: each request walked the whole history. These pathspecs also read "*", "?", "[" and a leading ":" in IDs as patterns, which could match other services. The history of a service is now walked once with a literal pathspec, its terms are picked in memory, and only the commits of the requested page are loaded. Records are selected from this history as Git selected them before: by a record prefix anywhere in the message for counts and lists, and among all the commits that changed the terms for navigation links and searches by date. Pages keep the order of the history, as before.
Counting all records listed and parsed every matching commit in memory, which took seconds on large histories. Git now counts them itself with rev-list, using the same filters as the listing.
Each request walked the history again, even when nothing had been recorded since, and the Explorer requests all the terms of a service at once. Readers now keep the history of each service and the count of all records for as long as HEAD points to the same commit. A commit SHA pins the whole history reachable from it, so results stay exact. The count is extended with the new commits when the history grows, and computed again after a rewrite. HEAD is read by the reader's own Git, within its limit of processes, and once for simultaneous requests. As a stored result outlives the request, a Git process that exits without an error message, such as one killed by the out-of-memory killer, now fails the read instead of returning its truncated output.
Member
|
Thank you for this suggestion. Do you have a use case that is currently limited by the performance of the Collection API? Thanks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The versions endpoints walked the whole history of the repository at each request with wildcard pathspecs, which prevent Git from using the changed-path Bloom filters of the commit-graph. On a synthetic history of 100,000 commits, listing the versions of some terms took about 10 s and finding a version by date about 11 s.
They now walk the history of a service once with a literal pathspec, pick its terms in memory, load only the commits of the requested page and read their diff statistics in a single Git process. Records are selected from this history as Git selected them before. Readers run up to four Git processes at once and keep their results for as long as HEAD does not move. Responses are also compressed. On the same history, listing the versions of some terms now takes about 1 s for the first request on a service, then about 0.1 s, and finding a version by date about 50 ms.
Literal pathspecs also stop service IDs and terms types holding
*,?or[from matching other services.