Repository navigation
Use one help text per metric name across collectors - #571
Open
antoinedeschenes wants to merge 1 commit into
Open
antoinedeschenes wants to merge 1 commit into
antoinedeschenes wants to merge 1 commit into
Conversation
Each project collector builds its metric descriptors from its own project's metric descriptor. Projects can describe the same metric type differently, custom and auto-created metrics in particular, and the registry then rejects the gather with "has help X but should have Y", which fails the whole scrape. Add a MetricHelpStore that keeps the first help text seen for each metric name, for as long as that name keeps being reported. The Runtime creates one and passes it to every collector it builds through MonitoringCollectorOptions, so all collectors in an exposition serve the same help text. A collector built without a store uses one of its own. Names not reported for an hour are dropped, which bounds the store. Fixes prometheus-community#361 Signed-off-by: Antoine Deschênes <antoine.deschenes@linux.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #361
When two collectors report the same metric name with different help texts, the registry rejects the scrape with
has help X but should have Y. Under the defaultHTTPErrorOnErrorhandling, the whole/metricsrequest returns 500, so every metric from the exporter goes missing, not just the conflicting one.This happens in two ways:
Google rewrites a built-in description, as in Exporter fails when stackdriver updates their help strings #361. Projects can briefly report the old and the new description at the same time.
Projects define the same custom metric type with different descriptions. Custom and auto-created descriptors are per project, and their description is whatever the writer set. We hit this with a Spark metric published by Dataproc Serverless batches into two projects: one project's descriptor had an empty description, the other had
Auto-created custom metric.. Every scrape failed for the half hour the batches ran each day:This is not specific to one metric. Comparing four of our projects, every auto-created Dataproc descriptor (about 70 metric types under
custom.googleapis.com/spark/,agent/,cpu/,disk/,memory/andnetwork/) has an empty description in one project andAuto-created custom metric.in the other three, so the default text seems to depend on when the descriptor was created. Any exporter covering both kinds of project fails whenever both have live series.Change
A new
MetricHelpStorekeeps the first help text seen for each metric name and returns it for every later descriptor with that name.Runtimecreates one store and passes it to every collector it builds, through a newMetricHelpStorefield onMonitoringCollectorOptions. All collectors in one exposition therefore serve the same help text.NewMonitoringCollectoris unchanged and existing embedders keep working. Embedders that register several collectors in one registry should pass a shared store.The help text served for a name stays fixed for as long as that metric keeps being reported. If Google updates a description, the new text only shows after a restart, or after the metric has gone unreported for an hour. I think that's the right trade: help is metadata, and the alternative is a failed scrape.
Testing
TestCustomMetricWithDifferentDescriptionsPerProjectGathersreports one custom metric type from two projects with different descriptions and gathers them through a registry. Without a shared store it fails with the samehas help "Auto-created custom metric." but should have ""error we saw in production. With it, the gather returns one family with both series, under the first description seen.TestRuntimeCollectorsShareMetricHelpStorechecks that every collector aRuntimebuilds uses the runtime's store.I also ran both builds against the Cloud Monitoring API, with two projects whose descriptors for
custom.googleapis.com/cpu/utilizationdiffer only in description, and--monitoring.metrics-prefixes=custom.googleapis.com/cpu:/metricsmasterhas help "Auto-created custom metric." but should have ""On this branch both projects report
stackdriver_monitoring_last_scrape_error 0, and the family is served with one help text, the one from whichever collector reported it first.🤖 Generated with Claude Code