Skip to content

fix(queue): ISS-013 keep DLQ reconciliation retrying - #680

Open
sbalabanov wants to merge 1 commit into
mainfrom
sergeyb/fix-iss-013-unbounded-dlq-retry
Open

fix(queue): ISS-013 keep DLQ reconciliation retrying#680
sbalabanov wants to merge 1 commit into
mainfrom
sergeyb/fix-iss-013-unbounded-dlq-retry

Conversation

@sbalabanov

@sbalabanov sbalabanov commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Intent:

  • Keep final-DLQ reconciliation durable during dependency outages longer than any finite attempt budget.
  • Preserve finite retry budgets for primary subscriptions.

Changes:

  • Define MaxAttempts zero as unlimited in both direct Nack and visibility-expiry poll paths.
  • Configure the shared DLQ subscription for unlimited retries with second-level dead-lettering disabled.
  • Verify the orchestrator pipeline and Runway wiring inherit the shared behavior and update the operational docs.

Reproduction:

  • A signal or storage dependency remains unavailable for more than 1000 DLQ reconciliation attempts.
  • Previously the finite cap was exhausted; because the reconciliation subscription had its own DLQ disabled, MySQL acknowledged the row and advanced past it, losing the reconciliation message.
  • The row now remains retryable until reconciliation succeeds or an operator removes it.

Generated by the 🪄 pr-create skill in devexp-agent-marketplace

Test Plan

AI Verification

Validated at 68fa422 on Sep 4 22:22 UTC · 13 files analyzed · 1s

Validator Status Issues
arc-unit not_applicable 0
visual-android not_applicable 0
visual-web not_applicable 0
go-proto-lint not_applicable 0
java-lint not_applicable 0
ios-lint not_applicable 0
ios-test not_applicable 0
web-lint not_applicable 0
go-lint not_applicable 0
go-coverage not_applicable 0
go-gazelle not_applicable 0
web-typecheck not_applicable 0
fix-disclosure not_applicable 0
merge-conflict not_applicable 0
go-thrift-lint not_applicable 0
android-lint not_applicable 0
android-coverage not_applicable 0
uber-one not_applicable 0
web-unit not_applicable 0
web-repocheck not_applicable 0
arc-lint not_applicable 0
visual-ios not_applicable 0
java-coverage not_applicable 0
web-coverage not_applicable 0
diff-template not_applicable 0
ureview completed 0
custom not_applicable 0

0 issues detected

Skipped validators: claude · EngWiki

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

// A zero budget is not "dead-letter immediately" — it is unconfigured,
// and the poll loop still governs.
{name: "unset budget never dead-letters here", attempt: 9},
{name: "unlimited budget keeps retrying beyond former cap", attempt: 1001},

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for this regression test

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the unnecessary regression test in ee84d5f. The remaining table cases cover retry-budget behavior without testing the former implementation cap.

[addressed by agent]

Summary:
Intent:
- Keep final-DLQ reconciliation durable during dependency outages longer than any finite attempt budget.
- Preserve finite retry budgets for primary subscriptions.

Changes:
- Define MaxAttempts zero as unlimited in both direct Nack and visibility-expiry poll paths.
- Configure the shared DLQ subscription for unlimited retries with second-level dead-lettering disabled.
- Verify the orchestrator pipeline and Runway wiring inherit the shared behavior and update the operational docs.

Reproduction:
- A signal or storage dependency remains unavailable for more than 1000 DLQ reconciliation attempts.
- Previously the finite cap was exhausted; because the reconciliation subscription had its own DLQ disabled, MySQL acknowledged the row and advanced past it, losing the reconciliation message.
- The row now remains retryable until reconciliation succeeds or an operator removes it.

---

<sub>Generated by the 🪄 [pr-create](https://sg.uberinternal.com/code.uber.internal/uber-code/devexp-agent-marketplace/-/blob/claude-code/plugins/dev/uber-dev/skills/pr-create/SKILL.md) skill in devexp-agent-marketplace</sub>
@sbalabanov
sbalabanov force-pushed the sergeyb/fix-iss-013-unbounded-dlq-retry branch from 68fa422 to ee84d5f Compare September 8, 2026 22:56
@sbalabanov
sbalabanov marked this pull request as ready for review September 8, 2026 22:56
@sbalabanov
sbalabanov requested review from a team and behinddwalls as code owners September 8, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants