Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 172 additions & 0 deletions astro_data/corrections/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
# Catalog corrections

A **correction** is a cited value that replaces what a source gives, because
the source is wrong for one object. The catalog import applies every
correction in this directory as its final pass, so a correction always has
the last word. It fails the import rather than apply one that no longer fits
the data.

Corrections are the fix of last resort. Before writing one, check that the
fix belongs here and not in one of the other three places
([`docs/ax/catalog.md`](../../docs/ax/catalog.md) §9.2, and
[ADR 0046](../../docs/adr/0046-catalog-corrections-cited-checked-applied-last.md)):

| the cause | the fix |
|---|---|
| a loader misreads its source | fix the loader |
| one better reference covers more than a handful of objects by a stable key | a supplementary source |
| a curated source (EGC, `bright_stars.csv`, `messier_names.dat`) is wrong | edit that file |
| a published source is wrong for one object | **a correction, here** |

If you are about to write more than a handful of corrections citing the same
reference, it is a supplementary source.

## Files

One TOML file per published source, named after it: `steinicke.toml`,
`sac_multistars.toml`, `ngc2000_names.toml`. Each file is also the list of
errors to report to that source's publisher.

**A correction goes in the file of the source whose error it fixes, even when
its key is another catalog's.** NGC 2000.0's names file puts "Horsehead
nebula" on IC 434. Both halves of the fix go in `ngc2000_names.toml`: the
removal keyed `["IC 434"]`, and the name keyed `["B 33"]`.

Files are read in name order, and tables in file order. Other files here,
such as this README, are ignored.

## Tables

Each correction is one table, headed by a **designator**: a catalog code, one
space, and the number, quoted. The code is the one in the database's
`catalogs` table, such as `NGC`, `IC`, `M`, `Sh2`, `SaM` or `WDS`.

```toml
["IC 424"]
why = "Steinicke gives 120'; the nebula is 2.5' across."
sources = ["https://…"]
size = "2.5'"
was.size = "120'"
```

The table's correction applies to the **sky object** that the designator
resolves to, which every catalog listing on it shares, with one exception:
a `description` belongs to that one catalog listing. `["M 42"]` and
`["NGC 1976"]` reach the same sky object, so both may not correct its
magnitude.

Every table needs:

- **`why`**: what is wrong, and what the sources say.
- **`sources`**: a list of citations: a URL where one exists, otherwise a
full citation (a book and page, or a bibcode).

## Fields

A table can set any of these fields, named like the database columns. Each
field needs **`was.<field>`**, the value it replaces, written the same way.
If the field is empty in the database, write `was.<field> = ""`.

| field | written as | example |
|---|---|---|
| `obj_type` | a type code from `PiFinder/obj_types.py` | `"C+N"` |
| `ra` | sexagesimal hours | `"05 35 17.3"`, `"05:35:17.3"`, `"5h35m17.3s"` |
| `dec` | sexagesimal degrees, signed | `"-05 23 28"`, `"+41°16′09″"`, `"-00 30 00"` |
| `const` | an IAU abbreviation | `"Eri"` |
| `size` | arcminutes, or with a unit: `'`, `"`, `°` | `"2.5'"`, `"60'x30'"`, `"36\""` |
| `mag` | a number, or a list for a variable's range | `11.4`, `[4.86, 7.32]` |
| `description` | text; a multi-line string is fine | `"""…"""` |

- `ra` and `dec` take two or three parts, and only the last may have
decimals. A bare number is refused, because it could be hours or degrees.
- `size` is read by the loaders' own `parse_arcmin_size`, and must parse
whole. It corrects the extents. The position angle stays as it was.
- Correcting `ra` or `dec` recomputes the constellation from the new
position, as a loader would. To overrule that, for a boundary case like
NGC 1404, correct `const` too.

## Names

```toml
["M 82"]
why = "NGC 2000.0 gives the M 81/M 82 pair's name to M 82 alone."
sources = ["https://…"]
names.remove = ["Bode's nebulae"]
names.preferred = "Cigar Galaxy"
```

- **`names.add`**: a list of names to append.
- **`names.remove`**: a list of names to delete. Every copy goes.
- **`names.preferred`**: one name, placed first among the sky object's names
so the device and the atlas show it. It is added if the sky object lacks
it.

Names are compared exactly, including case.

## Identifications

These correct which sky object a designator belongs to.

- **`same_as = "IC 2480"`**: this designator is the same sky object as that
one. Its listing moves there. If it was the sky object's only listing, the
whole sky object merges in, names and all, and is deleted. The target's
values stand and the merged object's are dropped, so correct anything
worth keeping in the same table.
- **`own_object = true`**: this designator is a sky object of its own. Its
listing moves to a new sky object that starts as a copy of the old one.
Correct its fields in the same table, with `was` stating the copied
values.
- **`hip = 102989`**: this star listing is that Hipparcos star. Use it where
the source's position is too far off for the star cross-match to find it.

A listing moves with the names from its own catalog that are its
designator: its own "IC 5281", or any form the loaders' lookup resolves to
it, such as "Collinder 42" from Col. Its other names stay where they are.
If it has no such name, as with an M listing, whose "M nn" name comes
with Steinicke's NGC record, the designator is added as a name where it
lands.

All splits (`own_object`) run before all joins (`same_as`), and both run
before any other correction. So the other fields in an identification's
table, and `was`, apply to the sky object the designator ends up on.

`hip` is the one correction the final pass does not apply. The star
cross-match reads it before it matches, and uses it in place of the
position.

## Checks

The import fails, naming the file and the designator, when:

- a file isn't valid TOML, or a table has an unknown key or a malformed
value;
- `why` or `sources` is missing;
- a field has no `was.<field>`, or `was` equals the correction;
- a field already holds the corrected value (**redundant**: delete the
correction);
- a field holds neither `was` nor the correction (**stale**: the data
underneath has changed, so check the correction against its source);
- a designator resolves to nothing;
- two corrections reach the same field, or the same name, of one sky object;
- two corrections identify the same listing, or both give it a `hip`;
- `names.remove` names a name the sky object doesn't have, `names.add` one it
already has, or `names.preferred` the name that is already first;
- a `same_as` is already true, or an `own_object` designator is already its
own sky object;
- a `same_as` points at a designator that has its own `same_as` (point it at
the final sky object instead);
- a split or join leaves its old sky object still carrying another
catalog's name that the loaders' lookup resolves to the designator: its
exact key ("NGC70"), or a designation `parse_designation` reads as it
("Cr 42" for Col 42, "NGC 0070" for NGC 70, but not "H 24", since a bare
"H" names no catalog). That catalog's listing was matched through the
wrong identification and would be left behind, so correct it too. If the
name is only a stray, a `names.remove` on the old sky object clears it.
That is the usual case for an M listing: Steinicke's NGC record carries
its "M nn" name, so splitting or moving one needs a `names.remove` of that
name keyed by the NGC designator, which records that the NGC record's
cross-identification is wrong.

A check compares values within tolerances: 0.05 for magnitudes, 1″ for RA
and Dec, each taken separately, and 1″ for each size extent once parsed. So
`"120'"` and `"2°"` are the same size.
46 changes: 46 additions & 0 deletions docs/adr/0046-catalog-corrections-cited-checked-applied-last.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Catalog errors are fixed in one of three places; source errors become cited, checked corrections applied last

**Status: accepted** (2026-10-22). Vocabulary: [`docs/ax/catalog/CONTEXT.md`](../ax/catalog/CONTEXT.md), *Import*. Architecture: [`docs/ax/catalog.md`](../ax/catalog.md) §9.

An audit of `objects.db` against OpenNGC and Hipparcos (paperChart_dev, 2026-10) found about 400 wrong values: types, names, positions, magnitudes, sizes. Most come from **loaders** misreading their **sources**. A substantial minority come from the sources themselves — Steinicke's 120′ for IC 424, SAC's "Beid" for Keid, NGC 2000.0's "Bode's nebulae" on M 82 — which a loader reads faithfully and must not "fix" by guessing. PiFinder had no place for a value that contradicts its source: the only precedent was `extra_names.dat`, three lines no loader reads.

## Decision

1. **Every wrong value is fixed in exactly one of three places.**
- The loader misreads its source → **fix the loader**.
- A published source is wrong, and one better reference covers more than a handful of objects by a stable key → **a supplementary source**: a vendored reference joined on that key (Hipparcos and IAU WGSN by HIP, Bica et al. 2008 by designation), which fills or replaces one field by rule and never adds listings.
- A published source is wrong for one object, and no rule or reference can produce the value → **a correction**.

A **curated source** — a list PiFinder compiles itself, such as EGC or the bright stars — is none of these: it is edited directly.

2. **A correction is keyed by a designator** (`"NGC 604"`) and applies to the sky object that designator resolves to; a description correction applies to that one catalog listing. `objects.id` is an autoincrement and changes on every rebuild, so it cannot be the key. Two corrections that reach the same field of one sky object through different designators (M 42, NGC 1976) fail the build, as does a designator that resolves to nothing.

3. **A correction does one of three things:** set fields (`obj_type`, `ra`, `dec`, `const`, `size`, `mag`, `description`); edit names (`names.add`, `names.remove`, and `names.preferred`, which puts a name first); or correct an **identification** — in either direction (`same_as = "IC 2480"`, or `own_object = true` to detach a designator into a sky object of its own), or, for a star listing, by naming its Hipparcos number (`hip = 102989`). It never creates a sky object from nothing or deletes a listing: every orphan and duplicate the audit found is a loader bug.

4. **Corrections are applied in one final pass**, after every loader, the star cross-match, the supplementary sources and type normalisation, and before image resolution and the database is finalised. A correction therefore always has the last word, and its type is checked against the normalised type codes. A split (`own_object`) or a join (`same_as`) fails the build if the sky object it leaves still carries another catalog's name equal to the moved designator, since that catalog's listing was matched through the wrong identification and the move would leave it behind.

**The one exception is `hip`.** Stars are identified by cross-matching each listing's position to Hipparcos, and a source position whole degrees off (SAC's α Mic and ξ Lup) finds no match — so a position corrected in the final pass would come too late to merge the star or to let Hipparcos supply its values. The star-identity stage therefore reads `hip` corrections before it cross-matches, and uses them in place of position. An identification feeds matching; every other correction overwrites a value, and those stay last. A `hip` correction is redundant if the cross-match alone finds that number.

5. **The format is TOML**: one table per designator, one file per corrected source, under `astro_data/corrections/` (`steinicke.toml`, `sac_multistars.toml`, `ngc2000_names.toml`…). Field names are the database's columns; values are in the units references publish — sexagesimal RA and Dec, sizes through the loaders' own `parse_arcmin_size`, a magnitude as a number or a variable's range as a list. `why` and `sources` (URLs) are required.

6. **Every field correction states the value it replaces** (`was.size = "120'"`), and the pass checks it, within tolerances (0.05 mag, 1″, sizes compared parsed). If the field already holds the corrected value, the correction is redundant; if it holds anything other than `was`, the correction is stale. Both fail the build. Name and identification operations carry the same check implicitly: a `names.remove` of an absent name, or a `same_as` on designators already one sky object, fails.

## Considered options

- **Edit the published source files.** Rejected: a vendored source would no longer be what its publisher released, an updated release could not be dropped in, and nothing would record what was changed or why. Curated sources are the exception because PiFinder is their publisher.
- **Special-case objects in loader code.** Rejected: values without citations, scattered across thirteen loaders, invisible to anyone reviewing the data. `add_missing_messier_objects` is the existing example of the pattern, and is left as it is rather than extended.
- **Corrections for the systematic errors too** — copy Bica's 49 Magellanic sizes and WGSN's names in by hand. Rejected: hundreds of transcribed rows that drift from their reference and cannot be regenerated. That is what a supplementary source is for.
- **Apply corrections at insert time, as each loader writes.** Rejected: a later loader's fill or a supplementary source's replacement would land on top, and a correction would no longer have the last word.
- **Build the whole catalog in memory, correct it, write once.** The cleanest reading of "correct before it reaches the DB", but it rewrites every loader. The database being built is a scratch file until it is finalised, so a final pass gives the same guarantee — the shipped file never holds an uncorrected value — for a fraction of the change.
- **Key by `object_id` or by position.** `object_id` is unstable across rebuilds; positions are ambiguous, since the audit found 582 groups of NGC/IC objects at identical coordinates.
- **YAML, TSV, JSON Lines.** YAML 1.1 reads `13:42:47` as a base-60 integer — a trap for RA. TSV cannot express name edits or identifications. JSON has no comments, and hand-editing it is error-prone. TOML has comments and multi-line strings, diffs cleanly, and matches paperChart's `labels.toml`.
- **Warn on stale or redundant corrections instead of failing.** Rejected: the import runs on a developer's machine, where a failure costs a minute, and a warning in a long import log is a warning nobody reads.

## Consequences

- **Corrections are written last, after the loader fixes.** Those fixes change what some designators resolve to and what values sit underneath — attaching Steinicke's duplicates to their twins moves `NGC 2239` onto `NGC 2244`'s sky object — and the `was` check reports exactly which corrections they disturbed.
- **Re-vendoring a source fails the import until its corrections are reviewed.** That is the point: a correction Steinicke has since fixed should be deleted, not silently re-applied over the fix.
- **Each corrections file is the upstream report for its publisher.** `sac_multistars.toml` is the list to send SAC.
- **Unresolved identifications stay as the sources give them.** A correction needs a decided value and a citation; NGC 6002, NGC 3355 and the other rows the audit could not settle have neither.
- **Name order carries meaning**, because `names.preferred` and the WGSN names work by placing a name first. The import must therefore keep names in a reproducible order; a `set()` of names follows Python's per-process hash seed and does not.
- **The import gains a development dependency**: `tomli` on Python 3.9, `tomllib` from 3.11. The device never runs the import.
Loading
Loading