Skip to content

fix: better handling for transient failures - #24

Draft
bosbaber wants to merge 1 commit into
mainfrom
stephan/20260917-error-handling
Draft

bosbaber wants to merge 1 commit into
mainfrom
stephan/20260917-error-handling

Conversation

@bosbaber

Copy link
Copy Markdown
Contributor

No description provided.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The retry loop currently logs full Helm output on every failed attempt, which can bloat CI logs and should be limited to the final/non-transient failure case.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR improves the robustness of the chartvalidator’s Helm rendering step by retrying helm template when failures look transient (network/HTTP issues), reducing flaky pipeline failures when rendering many non-OCI charts.

Changes:

  • Refactors Helm invocation into runHelm, adding bounded retries with jittered backoff on transient-looking failures.
  • Introduces retrySleepFn to make retry delays testable (no-op in tests, real sleep in production).
  • Adds unit tests covering retry success, retry exhaustion, and “do not retry” behavior for permanent errors.
File summaries
File Description
chartvalidator/checker/engine_chart_rendering.go Adds runHelm retry wrapper, sleep injection, and transient-error classification logic.
chartvalidator/checker/engine_chart_rendering_test.go Adds tests validating transient retry behavior and classification.
chartvalidator/checker/engine_app_checker.go Wires the renderer engine with a real sleep function for retries.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +190 to +196
msg := fmt.Sprintf("helm command failed: %s\nOutput: %s", err.Error(), string(output))
logEngineWarning(engine.name, workerId, msg)

if !isTransientHelmError(output) {
logEngineWarning(engine.name, workerId, fmt.Sprintf("not retrying %s: permanent failure detected", cmdStr))
break
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants