Skip to content

feat(catalog-import): apply cited, checked corrections as the import's final pass - #672

Merged
brickbots merged 4 commits into
mainfrom
feat/catalog-corrections
Oct 24, 2026
Merged

brickbots merged 4 commits into
mainfrom
feat/catalog-corrections

Conversation

@brickbots

@brickbots brickbots commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Part of #670 (PR 1 in its plan). Builds on #671 (merged): the left-behind check uses its parse_designation.

This PR adds the catalog import's correction mechanism and documents the import. It adds no corrections and doesn't change the committed database. On today's import the pass is a no-op, because astro_data/corrections/ holds only its README.

Why

The audit in #670 found about 400 wrong catalog values. Most come from loaders misreading their sources, and those get loader fixes in later PRs. The rest come from the sources themselves, such as Steinicke's 120′ for IC 424 or NGC 2000.0's "Bode's nebulae" on M 82, and a loader must not guess its way around them. PiFinder had nowhere to put a value that contradicts its source. ADR 0046 sets out where each kind of error is fixed, and why source errors become cited, checked corrections applied last.

What's in it

  • Docs (first commit):
    • ADR 0046.
    • docs/ax/catalog.md §9, on the import: the pipeline, precedence, matching, supplementary sources, corrections, name order, and when to commit the database.
    • An Import section in the catalog CONTEXT.md, defining source, curated source, loader, supplementary source, correction, identification, preferred name and related terms.
  • catalog_imports/corrections.py:
    • load_corrections() reads and validates astro_data/corrections/*.toml. It runs at the start of the import, so a malformed file fails before any loader runs.
    • apply_corrections() is the final pass. It runs after post-processing and before image resolution.
    • hip_identifications() is the hook for the star cross-match coming in PR 4.
  • astro_data/corrections/README.md: the format reference. One table per designator; fields named like the database columns, in reference units (sexagesimal RA/Dec, sizes through parse_arcmin_size, magnitudes as a number or a range); names.add, names.remove and names.preferred; and the identifications same_as, own_object and hip.
  • parse_arcmin_size(strict=True), so a hand-typed size must parse whole.
  • tomli added to requirements_dev.txt for Python before 3.11. Nothing on the device imports the catalog import, so it needs no new dependency.

What the pass enforces

Any of these fails the import, and the pass rolls back as one transaction:

  • a missing why or sources;
  • a field correction without was.<field>;
  • a redundant correction, where the field already holds the corrected value;
  • a stale correction, where the field holds neither was nor the corrected value, within 0.05 mag, 1″, or 1″ per parsed size extent;
  • a designator that resolves to nothing;
  • two corrections reaching the same field or name of one sky object;
  • a name operation or identification that is already true or impossible;
  • a split or join that leaves another catalog's listing behind on the old sky object.

Calls made beyond the ADR

Each is documented in the README.

  • Order: splits run before joins, and both run before field and name corrections. So a table's other fields and its was apply to the sky object its designator ends up on.
  • Merging and splitting:
    • A same_as on a sky object's only listing merges the whole object into the target, and the target's values stand.
    • A split (own_object) starts the new sky object as a copy of the old one.
    • A listing moves with the names that are its designator.
  • Constellation: a corrected position recomputes the constellation, as a loader does at insert, unless const is corrected too (NGC 1404). Each recompute is logged.
  • Size: a size correction keeps the stored position angle.
  • Messier names: Steinicke's NGC records carry the "M nn" names, so moving an M listing also needs a names.remove on the NGC designator. A test pins this.
  • Matching names to a designator: the "left behind" check asks the same question as the loader lookup from fix(catalogs): join akas by allowlisted designation, and four import fixes #671 (resolves_to: the exact key, or parse_designation). So "Cr 42" counts as Col 42 and "NGC 0070" as NGC 70, while SAC's "H 24" doesn't count as Herschel 400's H 24. A listing also keeps its own " " name when it moves, including for B, H and WDS, whose bare prefixes the lookup doesn't read.

Testing

Python 3.9:

  • nox -s lint and nox -s type_hints pass: no issues in 154 source files.
  • nox -s smoke_tests: 5 passed.
  • nox -s unit_tests: 1691 passed, on top of #671.
  • 126 new tests in tests/test_catalog_corrections.py, on a fixture database built in the test, never the committed one. They cover every operation, every failure, and the tolerances. Every failure test also checks that the database is unchanged.
  • I applied three sample corrections (IC 424's size, NGC 1404's constellation, M 82's names) to a scratch copy of the committed database. They came out right, and re-applying them failed as redundant.

ADR 0036–0045 are claimed by #379, #534 and #362, so this one is 0046. No open PR adds another 0046.

🤖 Generated with Claude Code

brickbots and others added 4 commits October 23, 2026 18:05
…ssary

An audit of the catalog against OpenNGC and Hipparcos found about 400
wrong values in pifinder_objects.db. Most come from loaders misreading
their sources; a substantial minority come from the sources themselves,
which a loader reads faithfully and must not "fix" by guessing.

ADR 0046 settles where each kind of fix goes: the loader, a
supplementary source joined on a stable key, a curated source edited
directly, or, as the last resort, a correction. A correction is a cited,
checked TOML entry keyed by designator, applied in one final pass after
every other stage of the import.

docs/ax/catalog.md gains section 9 on the import: the pipeline, where a
fix goes, matching, the supplementary sources, corrections, name order,
and when to commit a rebuilt database. The catalog CONTEXT.md gains an
Import section defining source, curated source, loader, supplementary
source, identification, not-found object, double star, common name,
cross-designation, preferred name and correction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s final pass

Implements the correction mechanism from ADR 0046. A correction fixes a
value a published source has wrong for one object. It lives in
astro_data/corrections/<source>.toml, one table per designator, and
carries `why`, `sources` and, for each field it sets, `was.<field>`, the
value it replaces. No corrections ship yet; they land with the one DB
rebuild at the end of the catalog-audit stack.

load_corrections() reads every file up front, so a malformed one fails
the import before the loaders run. apply_corrections() runs after
post-processing and before images are resolved, so a correction has the
last word. It resolves each designator through catalog_objects, never
objects.id, and applies all or nothing:

- identifications first, splits (own_object) before joins (same_as),
  since they change what designators resolve to;
- then every field and name correction is checked, and only then
  written. Redundant, stale, unresolvable and conflicting corrections
  fail, within 0.05 mag, 1 arcsec, and sizes compared parsed;
- a split or join fails if its old sky object still carries the
  designator as another catalog's name, so a listing matched through the
  wrong identification can't be left behind.

Values come in reference units: sexagesimal RA and Dec, sizes through
parse_arcmin_size (now with a strict mode), and a magnitude as a number
or a range. A position correction recomputes the constellation, as a
loader would, unless a const correction overrules it.

`hip` is parsed but not applied: hip_identifications() exposes it for
the star cross-match, which needs it before it matches.

astro_data/corrections/README.md is the format reference. tomli joins
the dev requirements for Python before 3.11; the device never runs the
import. ADR 0046 and catalog.md section 9.5 now say the left-behind
check covers same_as as well as own_object, and link the README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…omputes

Review follow-ups for the correction pass.

In the real import, an M listing's "M nn" name comes with Steinicke's
NGC record, so its origin is NGC; the test fixture gave it origin M. A
split or join of an M listing therefore moves no name of its own, and
the left-behind check fails on the NGC record's copy. That failure is
wanted: the NGC record asserts the cross-identification the correction
denies, so the author removes the name deliberately. The fixture now
uses real origins, a test pins the failure and the names.remove that
clears it, and the left-behind message names both remedies.

The pass logs each constellation it recomputes from a corrected position
(designator, old -> new), so the reviewer of the rebuild sees it in the
import log.

README: `sources` takes a full citation where no URL exists; catalog
codes are the catalogs table's rather than a list that would go stale;
a whole-object merge keeps the target's values; and the M case is
spelled out under names and checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The correction pass decides two things by comparing a name to a
designator: which names a moved listing takes with it, and whether a
split or join leaves another catalog's listing behind. It compared names
ignoring case and spaces, which matched neither the old normalize()
fallback nor parse_designation(). It would have missed the akas that now
join, such as Lyngå's "Collinder nnn" and Arp's zero-padded "NGC 0070".
It would also have fired on names that join nothing, such as SAC's
"H 24".

resolves_to() now asks the question ObjectFinder.get_object_id asks: is
the lowercased name the listing's exact key, or does parse_designation
read it as the listing? The left-behind check uses it alone. A listing's
own names also count in the "<code> <sequence>" form every loader
writes, since the lookup can't resolve it for B, H or WDS, whose bare
prefixes name no catalog.

Tests pin "Cr 42" as Col 42, "NGC 0070" as NGC 70, "H 24" as not
Herschel 400's H 24, and "M 2-9" as not M 29, both directly and through
the left-behind check, plus a split that takes "B 33" and "Collinder 42"
with their listings.

Refs #670

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brickbots
brickbots force-pushed the feat/catalog-corrections branch from f34b8d5 to 79fbab3 Compare October 24, 2026 01:05
@brickbots
brickbots changed the base branch from fix/catalog-designation-allowlist to main October 24, 2026 01:05
@brickbots
brickbots merged commit bd03070 into main Oct 24, 2026
4 checks passed
@brickbots
brickbots deleted the feat/catalog-corrections branch October 24, 2026 01:06
brickbots added a commit that referenced this pull request Nov 10, 2026
…d stricter correction checks (#694)

* feat(catalog-corrections): the 84 hip identifications the star rules miss

One table per designator, in the file of the source it lists: SAC's
doubles and red stars, TLK, RASC's doubles and WDS. Each cites the
Hipparcos star in SIMBAD and gives #685's evidence as its why. A dry
run applies all 84: 2,261 sky objects merge into 1,795 stars.

Refs #670

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(catalog-corrections): Steinicke's, NGC 2000.0's, Collinder's, Lyngå's and SAC's errors

The corrections #670 and PRs 2-6 left for PR 7: Steinicke's types,
magnitudes, sizes and Corwin's identifications; NGC 2000.0's misplaced
names; Collinder's positions and sizes; Lyngå's, Caldwell's and Arp's
identifications; SAC's misnamed rows and star types; La Superba's
range. Each cites the audit's verdict or Corwin. A dry run applies all
of them.

Refs #670

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(catalog-corrections): a field correction can clear a magnitude or a size

A split copies its old sky object's values, and a moved identification
can leave a sky object holding another object's magnitude and size,
with no source that gives its own: IC 5281, a double star, would keep
NGC 7466's. `mag = ""` and `size = ""` clear them, with `was`
required as for any field correction; clearing a field that is already
empty is redundant and fails. Rich's decision.

Refs #670

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(catalog-corrections): the judgement calls, decided

NGC 4322 is Corwin's star, with the galaxy's magnitude and size cleared
and V 13.9 moved to NGC 4323; IC 5281's and IC 998's copied values are
cleared, and IC 997 takes the pair's from Steinicke's IC 998 row.
NGC 6847 is C+N. Col 208 joins IC 2488 (Collinder's own
identification), NGC 2029 and NGC 2030 take the Bica sizes at their
positions, and STF 2576 is HIP 97222. The not-found objects Corwin
identifies drop Steinicke's not-found lead from their descriptions.

Refs #670

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(catalog-corrections): check a same_as against the match guard, and a size for finite, positive extents

mrosseel's fault 7 on #670 (#672's correction checks):

- A correction's size must parse to finite, positive extents. "nan",
  "inf", "0" and "-5" parsed as floats and passed strict parsing. The
  empty size = "" still clears a size.
- A same_as must pass the loaders' match guard, from the listing's own
  type and position (those of the sky object it leaves) to the sky object
  it joins, as corrected in the same pass. Before, ["Lyn 919"] same_as =
  "NGC 224" passed at 100 degrees from the listing.
- guard = false opts out a cited identification whose listing's position
  or type is wrong. It is valid only with a same_as, and on a same_as the
  guard accepts it is redundant and fails.
- own_object gets no guard: its new sky object starts at the old
  position, and any field it moves carries its own was.

Four shipped same_as fail the guard, each a cited identification with
the source's position wrong, and now carry guard = false: Col 208 =
IC 2488 (Collinder's Dec slip, 43'), Lyn 866 = NGC 6846 (Lynga's Dec
slip, 2 degrees), Lyn 639 = Col 285 (one moving group with two centres,
Rich's decision on #680), and NGC 6138 = NGC 6363 (Steinicke's row is on
NGC 6141, 11 degrees away; Corwin).

ADR 0046 decisions 3 and 6, catalog.md 9.5 and the corrections README
say so.

Refs #670

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(catalog-corrections): IC 2145 is NGC 2086, not a 0.2' size

mrosseel's fault 8 on #670: #681 listed IC 2145 2' -> 0.2' citing Bica,
but the vendored Bica row NGC2086,IC2145,LMC-N160C gives 0.85'x0.80'.
The 0.2' is SIMBAD's 0.19' for the cluster, which the audit carried
over from OpenNGC; no source this correction cites gives it.

Corwin identifies IC 2145 with NGC 2086, and Bica names both in one row.
PiFinder keeps them as two sky objects 0.97' apart, so the size
correction becomes same_as = "NGC 2086", which already holds Bica's
0.85'x0.80'.

Refs #670

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(catalog-corrections): refresh after #691 and #692

#692's star matching replaces the 85 hip corrections with its list
of 64, judged on independent evidence. SaM 693 (STF 2576 = HIP 97222),
this PR's own addition, is kept: no rule finds it. 20 fewer tables.

Every was is refreshed against a build of main (4da2a0a) with the
hip corrections. One value moved: La Superba's magnitude, now
Hipparcos's V 5.42 (#691) instead of SAC's 4.8.

#691's WGSN names make four name parts redundant: SaM 1771's preferred
Jabbah, and the added Hatysa, Asellus Australis and Alrescha. The
tables keep their removals of SAC's misspellings.

Refs #670

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(catalog-corrections): four SAC red variables, Gumala, and SaM 1773's Laseth

#692's V gate on rule 1 leaves five SAC red variables unmatched (#691's
"Left for PR 7"). Four rows name an HD or BD number that SIMBAD gives
as their Hipparcos star, so each gets a hip correction:

- T Dra (SaR 139) = HIP 87820, with its GCVS V range [7.2, 13.5];
- BE CMa (SaR 96) = HIP 35865, with its GCVS V range [8.6, 9.4];
- Z Lup (SaR 187) = HIP 71386 and NY Gem (SaR 155) = HIP 31829, which
  take Hipparcos's V through #691 (GCVS gives photographic ranges only).

BE CMi (SaR 98) is left: SIMBAD has HIP 37022 as another star, 46"
away, and SAC's 8.7 lies inside its GCVS V range.

WDS 99456 (LLO 18) is HIP 94645, Gumala: 1.4" from it, and WDS's note
K marks its 4.94 as a K-band magnitude, so the V gate refused it.

#692 split SaM 1773 (nu Sco's CD row) off the system, so SaM 1771's
removal of the misspelled 'Laseth' no longer reached it. SaM 1773 now
has its own removal.

Refs #670

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant