fix: prevent API v1 metric export loss

[anth-volk]

Fixes #3860

Summary

Prevent the metric export failures observed in the API v1 stack and add a maintained deployment workflow for the central OpenTelemetry Collector.

The reviewed production interval contained 46 metric export errors and 863 rejected points. Missing Google Monitoring locations accounted for 211 points. Shared metric resource identities accounted for the remaining 652 out-of-order, duplicate, or too-frequent points.

Changes

• Add location=us-central1 to the collector's metrics pipeline without changing trace regions.
• Give each API v1 Python process a stable unique service.instance.id.
• Add a manually dispatched collector workflow that validates configuration, builds a commit-tagged image, deploys the Cloud Run service, and checks readiness.
• Send nonempty authenticated trace and metric data, then poll Cloud Trace and Cloud Monitoring until both unique records are readable.
• Send a nonempty OTLP log record and require the collector to reject it.
• Restore the Cloud Run invoker IAM check, remove public invoker bindings, and verify the resulting access policy before reporting success.
• Validate every deployment variable before the workflow changes Cloud Run.
• Document the deployment inputs, IAM permissions, and production verification procedure.

Google Cloud configuration

• The production GitHub environment contains all seven collector deployment values.
• The deployment identity retains Cloud Run developer access in policyengine-observability.
• The custom collectorDeploymentIam role grants only run.services.getIamPolicy and run.services.setIamPolicy, and is bound only on the collector service.
• The custom collectorDeploymentVerifier project role grants only cloudtrace.traces.get, monitoring.timeSeries.list, and resourcemanager.projects.get.
• Artifact Registry writer access remains limited to the observability repository.
• Service-account user access remains limited to the collector runtime identity.
• The deployment identity can invoke the collector for authenticated verification.
• The collector runtime identity retains roles/telemetry.writer and roles/serviceusage.serviceUsageConsumer.
• The live collector has no public member and its invoker IAM check is enabled.

Testing

• 12 passed across collector configuration, ingestion-verifier, and API observability-runtime tests after rebasing onto Update API v1 to PolicyEngine.py 6.2.1 #3855.
• The verifier tests inspect nonempty OTLP trace, metric, and log payloads; partial rejections; Google Cloud read-back parameters; and paginated Monitoring responses.
• Ruff formatting passed for every changed Python module.
• Both shell scripts pass bash -n.
• The official collector 0.160.0 binary accepted the configuration.
• The immutable commit-tagged image deployed to policyengine-api-v1-otel-collector-staging as revision policyengine-api-v1-otel-collector-staging-00001-5dt; the production collector revision and image remained unchanged.
• Authenticated staging verification stored a synthetic trace and Prometheus-translated metric, and the collector rejected the synthetic OTLP log request.
• The staging service returns HTTP 403 without authentication, has no public IAM member, and grants the GitHub deployment identity only service-level policy-management and invocation roles.
• The staged run exposed and fixed the readiness parser, Cloud Monitoring metric type and resource query, first-descriptor timeout, and unnecessary API-project dependency resolution.

Dependency and rollout

PolicyEngine/policyengine-observability#34 published policyengine-observability 3.0.2. This branch now locks that release.

This PR is stacked on #3855 and inherits its policyengine[models]==6.2.1 dependency and lockfile. Merge #3855 first, then retarget this PR to master before merging it.

Merging this PR automatically runs the collector deployment because the merge changes collector deployment files. Later pushes to master deploy the collector only when its image, configuration, deployment scripts, verifier, or workflow changes. Manual dispatch remains available for an intentional redeployment.

After the API and simulation consumers deploy, run one society report and confirm that no new collector metric export errors match the four recorded rejection causes.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.68%. Comparing base (4ad5036) to head (f7356a5).

Additional details and impacted files

@@                     Coverage Diff                      @@
##           feat/policyengine-6-1-2-api    #3861   +/-   ##
============================================================
  Coverage                        87.68%   87.68%           
============================================================
  Files                              198      198           
  Lines                            12010    12010           
  Branches                          2106     2106           
============================================================
  Hits                             10531    10531           
  Misses                             901      901           
  Partials                           578      578           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.