Finish the exposure PSF run-dir rename (exp_psf fails on develop) - #888
Merged
Merged
Conversation
98bc085 renamed the exposure PSF stage's run dir from run_sp_exp_SxSePsfPi to run_sp_exp_SxSePsf, so that config_exp_psfex.ini and config_exp_mccd.ini write one path and nothing downstream of exp_psf has to branch on `psf:`. It updated the one INPUT_DIR that existed at the time and left every other copy of the name behind. The mask_query merge then reintroduced a stale one: the chain grew a module between sextractor and setools, and setools' INPUT_DIR came across from the healsparse branch still spelling the old name. smk-g7 lost all 127 exposures to it, in two different voices: ERROR: Invalid INPUT_DIR (.../run_sp_exp_SxSePsfPi/mask_query_runner/output) on the first attempt -- mask_query had in fact run and written all 40 sexcat_ext files, into run_sp_exp_SxSePsf, where setools was not looking -- and then, on every retry: ERROR: Directory .../run_sp_exp_SxSePsf already exists. which is the SAME bug wearing the STAGE_DIR hat. unit_pre clears the stage's run dir before each attempt precisely so a retry starts clean, and it takes the directory's name from completeness.STAGE_DIR. That entry was stale too, so the rm -rf had been clearing a directory nobody writes and the real one accumulated until ShapePipe refused it. So the retry cleanup needed no new mechanism; it needed the name it already had. Fixing STAGE_DIR fixes both the clearing and the counting. tests/unit/test_workflow_run_names.py holds the invariant that made this possible to get wrong: a run directory is named once in a config and then referred to by things that must find it, and nothing checked that they agreed. It asserts the agreement statically, and fails on the pre-fix tree with both of the errors above named in its messages. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar (cherry picked from commit e9e8564d7aa9b9034c49406bd96852df7ba44525)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hotfix: every
exp_psfjob fails on developTwo PRs that were developed in parallel disagree on the name of the exposure-PSF run directory. #847 (external healsparse masks) added a setools
INPUT_DIRthat readsrun_sp_exp_SxSePsfPi; #852 (the Snakemake workflow) renamed that directory torun_sp_exp_SxSePsf. Both were right on their own branch, git merged them without conflict, and no campaign ran the combined tree before #852 was merged — the last one (smk-g6) predates the cross-merge. The first run on develop (smk-g7) fails everyexp_psfjob withInvalid INPUT_DIR (…/run_sp_exp_SxSePsfPi/mask_query_runner/output), and retries fail withDirectory …/run_sp_exp_SxSePsf already exists, becausecompleteness.STAGE_DIR["exp_psf"]— which the retry uses to reset the unit — also still carried the old name.Fix. The four remaining consumers now agree on
run_sp_exp_SxSePsf: setools'INPUT_DIRinconfig_exp_psfex.ini,completeness.STAGE_DIR, and two docstrings.Test.
tests/unit/test_workflow_run_names.pyasserts that every run directory is named once per config and that every chainedINPUT_DIR, both PSF configs, andcompleteness.STAGE_DIRagree. It fails on the pre-fix tree with both production errors, so a future parallel rename cannot merge silently.Note:
completeness.pyis hashed into every rule's params, so merging this invalidates the rerun state of in-flight campaigns. Cheap — prepare usesRETRIEVE=symlink.— Fable on behalf of Cail
🤖 Generated with Claude Code