Skip to content

fix!: harden Security Hub ingestion boundaries - #24

Open
sodle-splunk wants to merge 10 commits into
mainfrom
sodle/espm-5228-awssecurityhub-validation-followup
Open

sodle-splunk wants to merge 10 commits into
mainfrom
sodle/espm-5228-awssecurityhub-validation-followup

Conversation

@sodle-splunk

@sodle-splunk sodle-splunk commented Jul 31, 2026 •

Copy link
Copy Markdown
Contributor

Written and implemented by Codex.

Summary

  • persist a stable first-run polling boundary and conservatively migrate legacy checkpoints to a UTC lookback
  • preserve an older valid checkpoint during migration so unprocessed findings remain eligible
  • request only the remaining local finding allowance and reject oversized provider pages before accumulation
  • limit omitted-result pagination to 1,000 findings and exact-ID validation to 100 retained findings
  • reuse the bounded exact-ID result for related-finding output instead of issuing a second raw request
  • persist successful SQS message occurrences with a body fingerprint so capped polls resume at the exact unprocessed tail
  • delete SQS messages only after every included finding has been durably ingested
  • rotate capped SQS retries past persistent failures and report deferred or invalid records
  • prioritize messages deferred by an earlier poll so repeated failures cannot starve later received messages
  • continue later direct findings after an individual failure while holding checkpoint progress before the earliest failure

These are separate commits with matching release-note entries so their release history remains independently reviewable.

Tracking

Verification

  • hosted pre-commit: passed at dee5a7b115775e83d73b03958aba6bfc8c2a7417
  • hosted compile: passed at dee5a7b115775e83d73b03958aba6bfc8c2a7417
  • hosted build and test coverage: passed at dee5a7b115775e83d73b03958aba6bfc8c2a7417
  • AWS CodeBuild: passed at dee5a7b115775e83d73b03958aba6bfc8c2a7417
  • local source compilation with Python 3.9: passed
  • local full pre-commit suite: passed, including app linter, Semgrep, documentation, dependency packaging, NOTICE, release-note, and static-test hooks

Integration and shared sanity testing are outside this workflow and do not block review.

Connector scope and residual platform work

Connector-local hardening in this PR is complete and core checks are green.

  • PSAAS-32278: connector-side work is complete.
  • PSAAS-31807: connector-side checkpoint hardening is complete. Exact invocation-mode classification remains a platform follow-up because BaseConnector owns that behavior.
  • PSAAS-32010: connector-side pagination and retention bounds are complete. Pre-materialization byte and item-size enforcement remains a shared AWS transport follow-up.

Those platform residuals do not block review or merge of the completed connector improvements.

Release impact

This intentionally produces a major release: omitted or oversized finding limits now return at most 1,000 findings per action, and exact-ID validation retains at most 100 findings. Scheduled polling may also conservatively revisit its configured lookback once while migrating legacy checkpoint state; stable source identifiers keep that overlap idempotent.

Merge strategy: merge commit only; do not squash.

Persist a fixed first-run boundary, migrate legacy checkpoints through a conservative UTC lookback, and keep scheduled checkpointing independent of poll-now classification.

Authored by Codex.
Request only the remaining local allowance and reject oversized provider pages before accumulating findings.

Authored by Codex.
Persist successful finding IDs per SQS message so capped polls resume at the unprocessed tail and delete only after full durable ingestion.

Authored by Codex.
@github-actions

github-actions Bot commented Jul 31, 2026 •

Copy link
Copy Markdown

Merging this PR will release 3.0.0 with the following release notes:

3.0.0 (2026-08-01)

Connector release changes

  • Stabilize scheduled polling windows and migrate invalid legacy checkpoints to a safe UTC lookback.
  • Preserve older valid checkpoints during migration so unprocessed findings remain in the polling window.
  • Reject provider pages that exceed the requested local finding allowance before accumulation.
  • Limit omitted-result pagination to 1,000 findings and exact-ID validation to 100 retained findings.
  • Reuse the bounded exact-ID result for related-finding output instead of issuing a second unbounded request.
  • Persist per-message SQS progress so capped polls resume at the unprocessed finding.
  • Track SQS ingestion by exact message occurrence and delete only fully ingested messages.
  • Rotate capped SQS retries past persistent failures and report deferred or invalid messages.
  • Prioritize messages deferred by an earlier poll so persistent failures cannot starve later messages.
  • Continue direct polling after individual ingestion failures without checkpointing past the earliest failure.

Authored by Codex.
Limit omitted-result pagination to 1,000 findings and exact-ID validation to one page.

BREAKING CHANGE: Omitted or oversized finding limits now return at most 1,000 findings per action.

Authored by Codex.
@sodle-splunk sodle-splunk changed the title fix: harden Security Hub ingestion boundaries fix!: harden Security Hub ingestion boundaries Aug 1, 2026
Describe the exact retained-item guarantees without implying a single provider request.

Authored by Codex.
Scope SQS ingestion progress to a message body and occurrence index so repeated finding identifiers cannot credit unrelated records.

Authored-by: Codex
Keep legacy checkpoints conservative, rotate capped message retries, continue later direct findings, and report deferred or invalid queue records.

Authored-by: Codex
Persist deferred message order so a repeatedly failing record cannot monopolize the poll cap and starve later received messages.

Authored-by: Codex
Avoid a second unbounded related-findings request after exact-ID validation by reusing the already validated bounded response.

Authored-by: Codex
@sodle-splunk

Copy link
Copy Markdown
Contributor Author

Codex scope update: connector-local hardening is complete and core gates are green. Invocation-mode classification and pre-materialization response caps are platform follow-ups and do not block this PR. Marking it ready for review.

@sodle-splunk
sodle-splunk marked this pull request as ready for review August 7, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant