Repository navigation
Description
Background
My company has an internal monitoring/alerting solution that serves both our on-premise
and cloud needs; it's backed by Prometheus and Grafana. We get Cloud Monitoring (née
Stackdriver) metrics into our internal solution using this exporter.
I'm currently building some Grafana graphs for Dataflow jobs.
- Metric types under the
dataflow.googleapis.com
namespace can be filtered down to a specific Dataflow job using thejob_namelabel
since this label's value includes the Dataflow job name. (Technically, thejob_name
label is attached to thedataflow_job
monitored resource type). - Metric types under the
compute.googleapis.com
namespace can be filtered down to a specific Dataflow job using theinstance_name
label since this label's value includes the Dataflow job name. - Metric types under the
agent.googleapis.com
namespace CANNOT be filtered down to a specific Dataflow job. None of the labels
include the Dataflow job name. The closest label that gives me what I need is the
instance_idlabel that comes attached to thegce_instancemonitored resource type.
instance_idis numeric and there's no way to know what Dataflow job it belongs to.
Because I can't filter down to a specific Dataflow job with the agent.googleapis.com
metric types, I can't build a complete Grafana dashboard that visualizes all the metrics
I care about. Because of this, I still have to go back to using a combination of
Dataflow's Job Metrics UI and Cloud Monitoring's Metrics Explorer UI just to visualize
those metrics alongside others. I'm taking on the overhead of context-switching between
3 different monitoring tools. This slows down any analysis work that I want to do.
Request
As I understand it, in addition to the labels that are attached to metric types and
monitored resource types, Cloud Monitoring also attaches
resource metadata labels
to the monitored resource types. Specifically, the metadata.system_labels.name label
includes the GCE instance name which also includes the Dataflow job name. I would be
able to use this label to filter the agent.googleapis.com metric types down to a
specific Dataflow job.
The Stackdriver exporter doesn't seem to export these resource metadata labels though.
Is there a way we can start exporting these labels by default? Or, add a configuration
that allows users to enable collection of these extra labels?
References:
- LOC where this feature might need to be implemented
- These issues look related, but not exactly the same (let me know if I'm wrong though): How to get instance_name in metrics from agent.googleapis.com/agent/* #78, the metric for Agent of Stackdriver doesn't return 'instance name' correctly grafana/grafana#18750, Stackdriver: Add support for filtering, grouping and alias by on metadata labels grafana/grafana#18887
Happy to be corrected on anything stated here.
Activity
xbglowx commentedon Sep 3, 2021
It would be nice to have something similar to
aws_resource_infofrom the cloudwatch-exporter. That has allowed me to create custom tags on the aws resources themselves, i.e. owner, so that I can come up with a generic alerting rule and pass it along to alertmanager for proper routing.From what I have seen currently, in order to do something like that with stackdriver-exporter, one would need to create multiple alerting rules for each resource, that hardcodes the needed labels?
ibeauvais commentedon Nov 17, 2026
Proposed an alternative based on the Cloud Monitoring PromQL API, which avoids the aggregation configuration: #569