CMR-11560: V2 Fix Large Collection Delete by jmaeng72 · Pull Request #2535 · nasa/Common-Metadata-Repository

jmaeng72 · 2026-11-04T16:03:19Z

Overview

What is the objective?

To let large collections be deleted properly without timeout or server errors

What are the changes?

NOTE: This was reverted a few days ago but is now back ready for PR.

Since the last PR, the only thing I changed was the default scroll-size, requests-per-sec and slices of the cascade collection delete

Created new index delete queue
Created new ingest delete exchange
Collection deletes (to delete its granules) routed to new exchange and queue (other concept deletes will remain going to original index queue and ingest exchange)
Collection cascade delete in DB is now async and runs in background
Race condition of users re-creating collection and grans while a delete in happening background is addressed by tracking timestamp of delete request and re-created coll and grans will not be deleted
Collection cascade delete in ES is now async and runs in background with longer SQS timeout and will only run one delete task at a time

What areas of the application does this impact?

Cascade collection delete - ingest, indexer, metadata-db

Required Checklist

New and existing unit and int tests pass locally and remotely
clj-kondo has been run locally and all errors in changed files are corrected
I have commented my code, particularly in hard-to-understand areas
I have made changes to the documentation (if necessary)
My changes generate no new warnings

Additional Checklist

I have removed unnecessary/dead code and imports in files I have changed
I have cleaned up integration tests by doing one or more of the following:
- migrated any are2 tests to are3 in files I have changed
- de-duped, consolidated, removed dead int tests
- transformed applicable int tests into unit tests
- reduced number of system state resets by updating fixtures. Ex) (use-fixtures :each (ingest/reset-fixture {})) to be :once instead of :each

…sitory into CMR-11560-Mini

… in delete by query

…sitory into CMR-11560-Mini

…riginal

…sitory into CMR-11560

…sitory into CMR-11560-Mini

…sitory into CMR-11560

…sitory into CMR-11560-Mini

…sitory into CMR-11560

…sitory into CMR-11560-Mini

…ata-Repository into CMR-11560

…sitory into CMR-11560

…sitory into CMR-11560-v2

jmaeng72 · 2026-11-04T16:04:07Z

  {:default 200
   :type Long})

+(defconfig force-delete-concept-by-params-db-batch-size


new change since last PR

jmaeng72 · 2026-11-04T16:04:25Z

+                                         "auto"
+                                         (Long/parseLong val)))
+                             :scroll-size (config/collection-delete-elastic-scroll-size)
+                             :requests-per-second (config/collection-delete-elastic-requests-per-second)


new change since last PR

jmaeng72 · 2026-11-04T16:05:47Z

  {:default false
   :type Boolean})
+
+(defconfig collection-delete-elastic-slices


new change since last PR

jmaeng72 · 2026-11-04T16:06:48Z

                 :revision-date :transaction-id)
         (dissoc deleted-coll :revision-date :transaction-id))))

+(defn- wait-until-granule-revisions-deleted


this is the updated delete test that @daniel-zamora wrote that was needed to be paired with the async coll delete changes in this PR

Common-Metadata-Repository/system-int-test/test/cmr/system_int_test/ingest/collection/collection_ingest_test.clj

Line 462 in 0d281f1

(is (not (mdb/concept-exists-in-mdb? (:concept-id gran1) (:revision-id gran1)))

Can you check here as well?

daniel-zamora · 2026-11-04T18:11:39Z

+              {:error error-code :error-message error-message :throwable e})))]
+
+    ;; Only run the post-commit-step if the above save was 100% successful aka returned nil
+    (when (nil? save-result)


Could the background delete start from try-to-save once the transaction-id check passes? If that check rolls the tombstone back with a 409, granules of a still-live collection may already be getting deleted.

daniel-zamora · 2026-11-04T18:12:40Z

+    ;; Only run the post-commit-step if the above save was 100% successful aka returned nil
+    (when (nil? save-result)
+      ;; start a separate thread to do the post-commit-step db actions
+      (future


Is it possible to run this too many times and use up the Oracle connection pool if we're deleting many large collections at once, for example through cleanup-expired-collections or a provider scripting deletes?

daniel-zamora · 2026-11-04T18:13:33Z

+          (post-commit-step db provider concept)
+
+          (catch Exception e
+            (error e (format "ERROR - Background Oracle background process save-concept failed in post-commit-step for concept %s" (:concept-id concept)))))))


If this throws or metadata-db restarts mid-loop, the remaining granules stay under the tombstone and nothing picks them up again. Do we care about that?

daniel-zamora · 2026-11-04T18:31:38Z

+                                         (Long/parseLong val)))
+                             :scroll-size (config/collection-delete-elastic-scroll-size)
+                             :requests-per-second (config/collection-delete-elastic-requests-per-second)
+                             :max-wait-ms (* (config/index-delete-queue-max-wait-secs) 1000)}


Are there any pathways to this function that wouldn't want this max-wait-ms?

daniel-zamora · 2026-11-04T18:32:22Z

                 :revision-date :transaction-id)
         (dissoc deleted-coll :revision-date :transaction-id))))

+(defn- wait-until-granule-revisions-deleted


Common-Metadata-Repository/system-int-test/test/cmr/system_int_test/ingest/collection/collection_ingest_test.clj

Line 462 in 0d281f1

(is (not (mdb/concept-exists-in-mdb? (:concept-id gran1) (:revision-id gran1)))

Can you check here as well?

daniel-zamora · 2026-11-04T18:39:35Z

+         (let [e (second result)
+               ex-data-map (ex-data e)]
+
+           (if (and (< attempt max-attempts) (= :scroll-context-error (:type ex-data-map)))


Is it ever possible to wait too long, and have SQS redeliver the collection delete message?

jmaeng72 added 30 commits October 1, 2026 18:39

added the delete ingest exchange and index queue

a023308

update indexer event handler to read from new exchange

9d97f6b

add logs

3af2a95

update logs

3e3c646

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

f367ee5

…sitory into CMR-11560-Mini

update cascade collection delete and remove unused mapping type param…

466efe8

… in delete by query

update db delete to be batched

ceeb16d

make db delete async

8e147a1

add more logs to see what is going on with db

6cf0c02

add clean up task for successful deletes to cmr

b6b29dc

add timestamp to delete request

0692770

update hardcoded vars to flex params

945c59f

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

c39a11c

…sitory into CMR-11560-Mini

update logs

1ab72b0

update time format, add default visibility timeout to index delete queue

9c3d371

clean up some logs and formatting

a1e8251

add log

69ff795

fix logs

bd31f07

update message queue lib to get indexer queue name directly

3b80114

update queue timeout func

210b782

update logs to debug and remove unnecessary logs

80b4d19

update sqs to use hardcoded queue name

52bfb51

update unit test mock to new func changes

fc6ca0d

update delete-by-query to use the mapping-type param again like the o…

77a3165

…riginal

add the new delete exchange to acl and all revisions queues

f5023b8

update delete timestamp boundary for db and es to track revision-date

465839c

fix missing import

9873a3c

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

f04d56e

…sitory into CMR-11560

fix delete-request-timestamp val to not be nil and use the right ts

65ba417

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

7ab0de0

…sitory into CMR-11560-Mini

jmaeng72 added 18 commits October 19, 2026 13:53

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

83530f1

…sitory into CMR-11560

Add logs for debugging

a357dbb

add one more log

31c6a00

update log

27ce152

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

32b4653

…sitory into CMR-11560-Mini

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

a9e4555

…sitory into CMR-11560

clean logs

b8cc67a

move visibility tiemout to indexer

0575cb3

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

a2e33d8

…sitory into CMR-11560

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

6e74a20

…sitory into CMR-11560-Mini

revert sqs.clj file changes

9ed7e89

Merge branch 'CMR-11560-Mini' of https://github.com/nasa/Common-Metad…

1e96939

…ata-Repository into CMR-11560

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

89a5265

…sitory into CMR-11560

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

3de1ab0

…sitory into CMR-11560

add delete test to consider async deletes

aa94724

add configs and reduce defaults to cascade coll deletes

c42de5e

update config type

4187ce1

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

0d281f1

…sitory into CMR-11560-v2

jmaeng72 commented Nov 4, 2026

View reviewed changes

jmaeng72 self-assigned this Nov 4, 2026

jmaeng72 requested review from daniel-zamora, eereiter, ibsenc, jaortega527, jceaser and zimzoom November 4, 2026 16:07

daniel-zamora reviewed Nov 4, 2026

View reviewed changes

Navigation Menu

Repository navigation

CMR-11560: V2 Fix Large Collection Delete - #2535

CMR-11560: V2 Fix Large Collection Delete#2535
jmaeng72 wants to merge 51 commits into
masterfrom
CMR-11560-v2

jmaeng72 commented Nov 4, 2026 •

edited

Loading

Uh oh!

jmaeng72 Nov 4, 2026

Uh oh!

jmaeng72 Nov 4, 2026

Uh oh!

jmaeng72 Nov 4, 2026

Uh oh!

jmaeng72 Nov 4, 2026

Uh oh!

daniel-zamora Nov 4, 2026

Uh oh!

daniel-zamora Nov 4, 2026

Uh oh!

daniel-zamora Nov 4, 2026

Uh oh!

daniel-zamora Nov 4, 2026

Uh oh!

daniel-zamora Nov 4, 2026

Uh oh!

daniel-zamora Nov 4, 2026

Uh oh!

daniel-zamora Nov 4, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jmaeng72 commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

What is the objective?

What are the changes?

What areas of the application does this impact?

Required Checklist

Additional Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jmaeng72 commented Nov 4, 2026 •

edited

Loading