Skip to content

collectors: do not cache a descriptor listing that failed part way - #572

Open
vignesh2027 wants to merge 2 commits into
prometheus-community:masterfrom
vignesh2027:fix-descriptor-cache-partial
Open

vignesh2027 wants to merge 2 commits into
prometheus-community:masterfrom
vignesh2027:fix-descriptor-cache-partial

Conversation

@vignesh2027

Copy link
Copy Markdown

When --monitoring.descriptor-cache-ttl is set, the collector lists the metric descriptors for each prefix and stores the result in the cache. The result was stored even when the listing had failed part way, for example when the second page of descriptors returned a 500, or when fetching the time series of a descriptor on an earlier page returned an error and aborted the listing.

After that, every scrape until the entry expired used the partial list, so the descriptors that were missed stayed missing for the whole TTL even though the API had recovered.

The listing is now only stored when it completed without an error. The added test serves two descriptor pages from a fake monitoring API, fails the second page on the first scrape and then recovers. It fails on master because the descriptor from the second page is never queried.

When listing metric descriptors for a prefix failed on a later page, or the callback returned an error, the descriptors collected so far were still stored in the descriptor cache. With a descriptor cache TTL set, every scrape until the entry expired then only saw that partial list, and the missing metrics never came back after the API recovered. Only store the listing when it completed.

Signed-off-by: vigneshwar <applemacbook6sep2004@gmail.com>
Signed-off-by: vigneshwar <applemacbook6sep2004@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant