Repository navigation
collectors: do not cache a descriptor listing that failed part way - #572
Open
vignesh2027 wants to merge 2 commits into
Open
vignesh2027 wants to merge 2 commits into
vignesh2027 wants to merge 2 commits into
Conversation
When listing metric descriptors for a prefix failed on a later page, or the callback returned an error, the descriptors collected so far were still stored in the descriptor cache. With a descriptor cache TTL set, every scrape until the entry expired then only saw that partial list, and the missing metrics never came back after the API recovered. Only store the listing when it completed. Signed-off-by: vigneshwar <applemacbook6sep2004@gmail.com>
Signed-off-by: vigneshwar <applemacbook6sep2004@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
--monitoring.descriptor-cache-ttlis set, the collector lists the metric descriptors for each prefix and stores the result in the cache. The result was stored even when the listing had failed part way, for example when the second page of descriptors returned a 500, or when fetching the time series of a descriptor on an earlier page returned an error and aborted the listing.After that, every scrape until the entry expired used the partial list, so the descriptors that were missed stayed missing for the whole TTL even though the API had recovered.
The listing is now only stored when it completed without an error. The added test serves two descriptor pages from a fake monitoring API, fails the second page on the first scrape and then recovers. It fails on master because the descriptor from the second page is never queried.