Repository navigation
feat(catalog-import): apply cited, checked corrections as the import's final pass - #672
Merged
Merged
Conversation
brickbots
marked this pull request as ready for review
October 23, 2026 16:09
…ssary An audit of the catalog against OpenNGC and Hipparcos found about 400 wrong values in pifinder_objects.db. Most come from loaders misreading their sources; a substantial minority come from the sources themselves, which a loader reads faithfully and must not "fix" by guessing. ADR 0046 settles where each kind of fix goes: the loader, a supplementary source joined on a stable key, a curated source edited directly, or, as the last resort, a correction. A correction is a cited, checked TOML entry keyed by designator, applied in one final pass after every other stage of the import. docs/ax/catalog.md gains section 9 on the import: the pipeline, where a fix goes, matching, the supplementary sources, corrections, name order, and when to commit a rebuilt database. The catalog CONTEXT.md gains an Import section defining source, curated source, loader, supplementary source, identification, not-found object, double star, common name, cross-designation, preferred name and correction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s final pass Implements the correction mechanism from ADR 0046. A correction fixes a value a published source has wrong for one object. It lives in astro_data/corrections/<source>.toml, one table per designator, and carries `why`, `sources` and, for each field it sets, `was.<field>`, the value it replaces. No corrections ship yet; they land with the one DB rebuild at the end of the catalog-audit stack. load_corrections() reads every file up front, so a malformed one fails the import before the loaders run. apply_corrections() runs after post-processing and before images are resolved, so a correction has the last word. It resolves each designator through catalog_objects, never objects.id, and applies all or nothing: - identifications first, splits (own_object) before joins (same_as), since they change what designators resolve to; - then every field and name correction is checked, and only then written. Redundant, stale, unresolvable and conflicting corrections fail, within 0.05 mag, 1 arcsec, and sizes compared parsed; - a split or join fails if its old sky object still carries the designator as another catalog's name, so a listing matched through the wrong identification can't be left behind. Values come in reference units: sexagesimal RA and Dec, sizes through parse_arcmin_size (now with a strict mode), and a magnitude as a number or a range. A position correction recomputes the constellation, as a loader would, unless a const correction overrules it. `hip` is parsed but not applied: hip_identifications() exposes it for the star cross-match, which needs it before it matches. astro_data/corrections/README.md is the format reference. tomli joins the dev requirements for Python before 3.11; the device never runs the import. ADR 0046 and catalog.md section 9.5 now say the left-behind check covers same_as as well as own_object, and link the README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…omputes Review follow-ups for the correction pass. In the real import, an M listing's "M nn" name comes with Steinicke's NGC record, so its origin is NGC; the test fixture gave it origin M. A split or join of an M listing therefore moves no name of its own, and the left-behind check fails on the NGC record's copy. That failure is wanted: the NGC record asserts the cross-identification the correction denies, so the author removes the name deliberately. The fixture now uses real origins, a test pins the failure and the names.remove that clears it, and the left-behind message names both remedies. The pass logs each constellation it recomputes from a corrected position (designator, old -> new), so the reviewer of the rebuild sees it in the import log. README: `sources` takes a full citation where no URL exists; catalog codes are the catalogs table's rather than a list that would go stale; a whole-object merge keeps the target's values; and the M case is spelled out under names and checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The correction pass decides two things by comparing a name to a designator: which names a moved listing takes with it, and whether a split or join leaves another catalog's listing behind. It compared names ignoring case and spaces, which matched neither the old normalize() fallback nor parse_designation(). It would have missed the akas that now join, such as Lyngå's "Collinder nnn" and Arp's zero-padded "NGC 0070". It would also have fired on names that join nothing, such as SAC's "H 24". resolves_to() now asks the question ObjectFinder.get_object_id asks: is the lowercased name the listing's exact key, or does parse_designation read it as the listing? The left-behind check uses it alone. A listing's own names also count in the "<code> <sequence>" form every loader writes, since the lookup can't resolve it for B, H or WDS, whose bare prefixes name no catalog. Tests pin "Cr 42" as Col 42, "NGC 0070" as NGC 70, "H 24" as not Herschel 400's H 24, and "M 2-9" as not M 29, both directly and through the left-behind check, plus a split that takes "B 33" and "Collinder 42" with their listings. Refs #670 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brickbots
force-pushed
the
feat/catalog-corrections
branch
from
October 24, 2026 01:05
f34b8d5 to
79fbab3
Compare
brickbots
changed the base branch from
fix/catalog-designation-allowlist
to
main
October 24, 2026 01:05
brickbots
added a commit
that referenced
this pull request
Nov 10, 2026
…d stricter correction checks (#694) * feat(catalog-corrections): the 84 hip identifications the star rules miss One table per designator, in the file of the source it lists: SAC's doubles and red stars, TLK, RASC's doubles and WDS. Each cites the Hipparcos star in SIMBAD and gives #685's evidence as its why. A dry run applies all 84: 2,261 sky objects merge into 1,795 stars. Refs #670 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(catalog-corrections): Steinicke's, NGC 2000.0's, Collinder's, Lyngå's and SAC's errors The corrections #670 and PRs 2-6 left for PR 7: Steinicke's types, magnitudes, sizes and Corwin's identifications; NGC 2000.0's misplaced names; Collinder's positions and sizes; Lyngå's, Caldwell's and Arp's identifications; SAC's misnamed rows and star types; La Superba's range. Each cites the audit's verdict or Corwin. A dry run applies all of them. Refs #670 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(catalog-corrections): a field correction can clear a magnitude or a size A split copies its old sky object's values, and a moved identification can leave a sky object holding another object's magnitude and size, with no source that gives its own: IC 5281, a double star, would keep NGC 7466's. `mag = ""` and `size = ""` clear them, with `was` required as for any field correction; clearing a field that is already empty is redundant and fails. Rich's decision. Refs #670 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(catalog-corrections): the judgement calls, decided NGC 4322 is Corwin's star, with the galaxy's magnitude and size cleared and V 13.9 moved to NGC 4323; IC 5281's and IC 998's copied values are cleared, and IC 997 takes the pair's from Steinicke's IC 998 row. NGC 6847 is C+N. Col 208 joins IC 2488 (Collinder's own identification), NGC 2029 and NGC 2030 take the Bica sizes at their positions, and STF 2576 is HIP 97222. The not-found objects Corwin identifies drop Steinicke's not-found lead from their descriptions. Refs #670 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(catalog-corrections): check a same_as against the match guard, and a size for finite, positive extents mrosseel's fault 7 on #670 (#672's correction checks): - A correction's size must parse to finite, positive extents. "nan", "inf", "0" and "-5" parsed as floats and passed strict parsing. The empty size = "" still clears a size. - A same_as must pass the loaders' match guard, from the listing's own type and position (those of the sky object it leaves) to the sky object it joins, as corrected in the same pass. Before, ["Lyn 919"] same_as = "NGC 224" passed at 100 degrees from the listing. - guard = false opts out a cited identification whose listing's position or type is wrong. It is valid only with a same_as, and on a same_as the guard accepts it is redundant and fails. - own_object gets no guard: its new sky object starts at the old position, and any field it moves carries its own was. Four shipped same_as fail the guard, each a cited identification with the source's position wrong, and now carry guard = false: Col 208 = IC 2488 (Collinder's Dec slip, 43'), Lyn 866 = NGC 6846 (Lynga's Dec slip, 2 degrees), Lyn 639 = Col 285 (one moving group with two centres, Rich's decision on #680), and NGC 6138 = NGC 6363 (Steinicke's row is on NGC 6141, 11 degrees away; Corwin). ADR 0046 decisions 3 and 6, catalog.md 9.5 and the corrections README say so. Refs #670 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(catalog-corrections): IC 2145 is NGC 2086, not a 0.2' size mrosseel's fault 8 on #670: #681 listed IC 2145 2' -> 0.2' citing Bica, but the vendored Bica row NGC2086,IC2145,LMC-N160C gives 0.85'x0.80'. The 0.2' is SIMBAD's 0.19' for the cluster, which the audit carried over from OpenNGC; no source this correction cites gives it. Corwin identifies IC 2145 with NGC 2086, and Bica names both in one row. PiFinder keeps them as two sky objects 0.97' apart, so the size correction becomes same_as = "NGC 2086", which already holds Bica's 0.85'x0.80'. Refs #670 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(catalog-corrections): refresh after #691 and #692 #692's star matching replaces the 85 hip corrections with its list of 64, judged on independent evidence. SaM 693 (STF 2576 = HIP 97222), this PR's own addition, is kept: no rule finds it. 20 fewer tables. Every was is refreshed against a build of main (4da2a0a) with the hip corrections. One value moved: La Superba's magnitude, now Hipparcos's V 5.42 (#691) instead of SAC's 4.8. #691's WGSN names make four name parts redundant: SaM 1771's preferred Jabbah, and the added Hatysa, Asellus Australis and Alrescha. The tables keep their removals of SAC's misspellings. Refs #670 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(catalog-corrections): four SAC red variables, Gumala, and SaM 1773's Laseth #692's V gate on rule 1 leaves five SAC red variables unmatched (#691's "Left for PR 7"). Four rows name an HD or BD number that SIMBAD gives as their Hipparcos star, so each gets a hip correction: - T Dra (SaR 139) = HIP 87820, with its GCVS V range [7.2, 13.5]; - BE CMa (SaR 96) = HIP 35865, with its GCVS V range [8.6, 9.4]; - Z Lup (SaR 187) = HIP 71386 and NY Gem (SaR 155) = HIP 31829, which take Hipparcos's V through #691 (GCVS gives photographic ranges only). BE CMi (SaR 98) is left: SIMBAD has HIP 37022 as another star, 46" away, and SAC's 8.7 lies inside its GCVS V range. WDS 99456 (LLO 18) is HIP 94645, Gumala: 1.4" from it, and WDS's note K marks its 4.94 as a K-band magnitude, so the V gate refused it. #692 split SaM 1773 (nu Sco's CD row) off the system, so SaM 1771's removal of the misspelled 'Laseth' no longer reached it. SaM 1773 now has its own removal. Refs #670 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
15 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #670 (PR 1 in its plan). Builds on #671 (merged): the left-behind check uses its
parse_designation.This PR adds the catalog import's correction mechanism and documents the import. It adds no corrections and doesn't change the committed database. On today's import the pass is a no-op, because
astro_data/corrections/holds only its README.Why
The audit in #670 found about 400 wrong catalog values. Most come from loaders misreading their sources, and those get loader fixes in later PRs. The rest come from the sources themselves, such as Steinicke's 120′ for IC 424 or NGC 2000.0's "Bode's nebulae" on M 82, and a loader must not guess its way around them. PiFinder had nowhere to put a value that contradicts its source. ADR 0046 sets out where each kind of error is fixed, and why source errors become cited, checked corrections applied last.
What's in it
docs/ax/catalog.md§9, on the import: the pipeline, precedence, matching, supplementary sources, corrections, name order, and when to commit the database.CONTEXT.md, defining source, curated source, loader, supplementary source, correction, identification, preferred name and related terms.catalog_imports/corrections.py:load_corrections()reads and validatesastro_data/corrections/*.toml. It runs at the start of the import, so a malformed file fails before any loader runs.apply_corrections()is the final pass. It runs after post-processing and before image resolution.hip_identifications()is the hook for the star cross-match coming in PR 4.astro_data/corrections/README.md: the format reference. One table per designator; fields named like the database columns, in reference units (sexagesimal RA/Dec, sizes throughparse_arcmin_size, magnitudes as a number or a range);names.add,names.removeandnames.preferred; and the identificationssame_as,own_objectandhip.parse_arcmin_size(strict=True), so a hand-typed size must parse whole.tomliadded torequirements_dev.txtfor Python before 3.11. Nothing on the device imports the catalog import, so it needs no new dependency.What the pass enforces
Any of these fails the import, and the pass rolls back as one transaction:
whyorsources;was.<field>;wasnor the corrected value, within 0.05 mag, 1″, or 1″ per parsed size extent;Calls made beyond the ADR
Each is documented in the README.
wasapply to the sky object its designator ends up on.same_ason a sky object's only listing merges the whole object into the target, and the target's values stand.own_object) starts the new sky object as a copy of the old one.constis corrected too (NGC 1404). Each recompute is logged.names.removeon the NGC designator. A test pins this.resolves_to: the exact key, orparse_designation). So "Cr 42" counts as Col 42 and "NGC 0070" as NGC 70, while SAC's "H 24" doesn't count as Herschel 400's H 24. A listing also keeps its own "" name when it moves, including for B, H and WDS, whose bare prefixes the lookup doesn't read.Testing
Python 3.9:
nox -s lintandnox -s type_hintspass: no issues in 154 source files.nox -s smoke_tests: 5 passed.nox -s unit_tests: 1691 passed, on top of #671.tests/test_catalog_corrections.py, on a fixture database built in the test, never the committed one. They cover every operation, every failure, and the tolerances. Every failure test also checks that the database is unchanged.ADR 0036–0045 are claimed by #379, #534 and #362, so this one is 0046. No open PR adds another 0046.
🤖 Generated with Claude Code