Repository navigation
Implement APRS decoding and encoding across the vector corpus - #5
Conversation
afourney
left a comment
There was a problem hiding this comment.
Review of #5 at 977a523
This is substantial, careful work. On the core profiles the public decoder is close to the original corpus. I checked it independently, and the problems I found are mostly outside what the corpus covers: everyday Mic-E traffic, re-encoding, and a few send-time rules. Two findings are P1. The test harness also hides two divergences from the source vectors.
All reproductions use from hamkit import aprs as a on Python 3.12.
How I checked
- Provenance. Every row's
packet,source_expected, title, profile, source, outcome, note,scope_note,extension_tagsandoptionsis byte-identical to the base (b97beb7). Onlyexpected(105 rows),unmapped, gaps and sequence state changed. I reviewed all 105expectedchanges; apart from the items below they are schema mappings consistent with their sources. - Public decoder vs. original assertions. I wrote a checker that does not use
_compatibility.view()or the stored projections. It parses each non-FAP row witha.parse_packet(RF for strict-envelope/ax25, otherwiseAPRS_IS) and compares the result directly withsource_expected. About 11,200 comparisons run, covering header, position, ambiguity cells, symbols, comments, timestamps, motion, PHG/RNG/DFS/DF bearing, altitude, compression csT, Mic-E message/fix, objects/items, status grid/beam, messages, telemetry, weather (in wire units), storms and areas. Everything matches except OTH-033/035 (finding 3). - Re-encoding. Of 890 accepted non-FAP vectors re-serialized under a permissive policy, 811 reproduce the original bytes exactly and 6 are refused (the deliberately invalid BAD-033…040 extension text). I reviewed all 73 differences. Most are legitimate normalizations: placeholders, ignored-csT padding,
/A=placement, latitude-imposed ambiguity, beta DTIs. The exceptions are findings 4, 10 and 11. - I spot-checked base91 comment telemetry against the specification's
|ss11|example; it is correct.
P1
1. Mic-E manufacturer prefixes make ordinary Yaesu and Byonics packets Malformed.
_mic_e.read treats a leading ` or ' in the status text as APRS101's original hex telemetry (_mic_e.py:111). APRS 1.2 deprecates that format ("Original Mic-E Telemetry Format is depricated. Overtaken by MFR TYPE codes"). Today those characters mainly mark the manufacturer: Yaesu sends `…_" or `…_% and Byonics sends '…|3. As a result:
base = b"N0CALL>S32UVT:`(_f\x1e\x1eOj/"
a.parse_packet(base + b'`"4T}Hi_%') # Yaesu with altitude -> Malformed: invalid Mic-E hex telemetry
a.parse_packet(base + b"'Hello|3") # Byonics TinyTrack3 -> Malformed
a.parse_packet(base + b"'1020 x") # 4 hex after ' (FAP-348) -> Malformed- When the comment happens to start with hex digits, it is silently read as telemetry instead.
- FAP-430 (a real Byonics packet) is
Malformedfrom the public parser and only passes through the compatibility adapter. - The encoder rejects any device marker other than
]/>(_mic_e.py:208), so these packets can't be composed either. - The corpus has no Yaesu or Byonics core vectors, so nothing caught this.
Suggestion: recognize the APRS 1.2 prefix/suffix type codes, interpret hex telemetry only when the complete legacy form is present, and never let an unrecognized status prefix make the packet Malformed; fall back to comment text. Please add core vectors for Yaesu, Byonics and other common markers.
2. A corrupted NMEA sentence is re-sent with a valid checksum.
A sentence received with a bad checksum decodes as checksum_valid=False, position_usable=False with a "checksum mismatch" diagnostic. Yet check_packet returns (), and serialize_packet recomputes the checksum:
p = a.parse_packet(b"N0CALL>APRS:$GPRMC,180000,A,3350.000,S,15112.000,E,0.0,0.0,151026,,*00")
a.check_packet(p) # ()
a.serialize_packet(p) # b'...$GPRMC,180000,A,3350,S,15112,E,0,0,151026,,*0B'Downstream receivers will trust data this library knew was corrupt. The PR description mentions this ("three intentionally bad NMEA checksums are repaired during encoding"), but it should not be the default. Suggestion: make checksum_valid is False a conformance finding, so sending needs an explicit repair, for example clearing the flag.
P2
3. The core profile searches for an embedded ! (X1J), and the harness hides it.
OTH-033/035 (core-1.1, outcome opaque, kind non-aprs-beacon) decode as LegacyPosition from the public parser. The corpus records that the APRS 1.1 addendum deprecated this search in 2012; only the historical profile should apply it. _compatibility.view() then swaps the payload for a Beacon whenever the profile isn't historical (_compatibility.py, the LegacyPosition branch), so the test passes while the public parser disagrees with the vector. Suggestion: don't search in the public parser (or make it an explicit option), and remove the rewrite so the test compares real output.
4. Mic-E longitude 100–109° is encoded with control characters.
MIC-116…125 are the corpus's ordinary vectors for these degrees, using l…u (bytes 108–117) as in APRS101's longitude table. The encoder emits bytes 28–37 instead, the FS…US control characters followed by !…%:
MIC-116 b'`l_f\x1e\x1eOj/' -> b'`\x1c_f\x1e\x1eOj/'
This decodes arithmetically, but it isn't the specified form and puts control bytes on the air unnecessarily. Please emit l–u for 100–109°.
5. Standard weather reports are refused for exceeding the 43-byte comment budget.
Positioned weather is now a structured extension, so _conformance._position counts it against "extensions and comment exceed 43 bytes". An ordinary report is refused:
p = a.parse_packet(b"N0CALL>APRS:@092345z4903.50N/07201.75W_220/004g005t077r000p000P000h50b09900L123wRSW")
a.check_packet(p) # ['extensions and comment exceed 43 bytes'] -> cannot be sentAPRS101's weather chapter doesn't apply the 43-character limit to weather data, and the APRS 1.1 fields (L/l, s) push ordinary reports past it. Suggestion: exclude weather data from the comment budget, or cite the rule if the strictness is intended.
6. check_packet doesn't predict encode-time errors, and Mic-E needs a hand-computed destination.
check_packet promises "every conformance finding serialize_packet would raise". Several encoders raise FieldError only during serialization: Mic-E destination mismatch, unsupported Mic-E device markers, and Mic-E altitude reference. Composing a Mic-E packet also requires the caller to supply a destination that already encodes latitude, hemisphere, ambiguity and message, and no public helper derives it:
m = a.MicE(33.4, -112.1, a.Symbol("/", ">"), message="en_route",
extensions=a.Extensions(motion=a.Motion(90, 10)))
p = a.Packet(a.Header("N0CALL", "S32UVT"), m)
a.check_packet(p) # ()
a.serialize_packet(p) # FieldError: Mic-E destination does not match latitude, ...Suggestion: derive the destination in serialize_packet (or expose a helper), and report encode-time problems through check_packet.
7. Ordinary messages that start like a query response become Malformed.
QRY-015…017 are classified by their sources as ordinary messages (kind: message). The PR now recognizes response grammars in message text. That is reasonable as an addition, but when the grammar doesn't fit the parser fails instead of falling back:
a.parse_packet(b"N0CALL>APRS::W1AW :Directs= are fine, see you at 7")
# Malformed: requires a station token of 1..9 charactersThe same response also decodes as Response without a message ID but as Message with one, so consumers must handle both types. Suggestion: fall back to Message whenever the response grammar doesn't match exactly.
8. APRS_IS accepts SSID -0.
Both cited pages say so: "SSID -0 is dropped as conflicts with not present SSID" (ServerDesign) and "never explicitly define -0 as a SSID" (Connecting). _APRS_IS_SOURCE/_APRS_IS_ADDRESS accept N0CALL-0. Please exclude -0 from the APRS-IS patterns. The RF/AX.25 policy is unaffected.
9. The Mic-E grid is stored inside device_code.
MIC-265's source asserts type_marker: ">", grid_locator: IO91SX and grid_symbol: /G as separate facts. The public output is device_code=">IO91SX/G", and the test passes only because device_code isn't in that row's projection. A device marker field shouldn't carry a locator and symbol. Suggestion: add a Mic-E grid field (with its symbol), or leave the grid in the comment.
P3
10. The NMEA encoder rewrites numeric fields. It strips decimal points from coordinates (3350.000 becomes 3350, 01131.000 becomes 01131), drops leading zeros (satellites 08 becomes 8) and precision (47.0 becomes 47). The checksum is recomputed, so the sentence is valid, but it differs from what the GPS produced, and many NMEA consumers expect ddmm.mm with a decimal point. Consider keeping the received field text (fields_raw exists) or fixed NMEA formats.
11. Other canonical-form changes worth confirming against the spec:
- PARM/UNIT are padded to 13 fields (
PARM.Temp,,,,,,,,,,,,), which wastes message bytes and can push a valid definition past 67 bytes. - Query footprint radius
0200becomes200. ?APRSH N0CALLloses the callsign padding.
12. 000000z turns the whole packet Malformed. TIM-011 allows this ("retaining other fields with a warning is allowed"). But FAP-454…456 show it is common in real traffic, the compatibility view rescues those packets for FAP, and the position is otherwise intact. Consider returning a Packet without the timestamp, plus a diagnostic.
13. Question: telemetry decimals. T#005,1.5,… is decoded but marked nonconforming ("requires an integer in 0..255"), and FAP-432/433 accept decimals. If the intended baseline is APRS101 only, that's fine; if APRS 1.2 telemetry is in scope, please confirm.
Test architecture
- Most rows don't compare
parse_packetdirectly. Only PHG rows and the 19SCHEMA_NATIVE_IDSdo; every other row goes through_compatibility.view(). For core profiles that is nearly a passthrough: it selects RF for strict-envelope rows, plus the rewrite in finding 3. For FAP it is effectively a second decoder: it strips comments, uppercases calls, re-parses weather tails, rounds course, maps 0° to 360°, decodes relaxed telemetry and Mic-E, and replaces the rejection decision with_receiver_rejects. The description qualifies this, but "all 1,561 vectors pass" shouldn't be read as coverage of the public parser for FAP rows. - Test-only code ships in the package.
_compatibility.py(719 lines) is only used by tests but lives insrc/and ships in the wheel. Consider moving it undertests/. - Hard-coded IDs.
SCHEMA_NATIVE_IDSis a hand-maintained list. Selecting rows by a property of the row would be more robust.
Minor
- The
Positiondocstring in_models.pystill says recognized-but-undecodable extensions make the payloadOpaque, but DAO and base91 telemetry are now decoded.
Verified correct (no action needed)
Compressed positions and csT, ambiguity cells including latitude-imposed ambiguity, Mic-E message bits including custom/unknown, standard and alternate speed/course, area offsets and corridors, storm fields, status beam (^B7 gives 110° and 490 W), base91 comment telemetry, reply-ack, and the 000/000 disputed-zero policy (WX-060).
Removing
|
| Reason | Rows |
|---|---|
| Comment whitespace kept as sent (FAP trims it) | 22 |
Uncompressed wind unit kept as disputed_knots_or_mph (FAP assumes mph) |
4 |
NMEA default symbol from the source SSID, APRS101 (-11 is the balloon, /O) |
3 |
000000z timestamp is Malformed, with fields kept in Malformed.recovered (TIM-011 allows either) |
3 |
Mic-E type codes, APRS 1.2: ]= (TM-D710) suffix; ' with incomplete hex is a type byte; Byonics '|3 |
4 |
| Callsign case kept as sent | 2 |
| Compressed course byte 0 kept as 0° (FAP rewrites it to 360) | 2 |
| Text with no position in the first 40 bytes is a non-APRS beacon (FAP rejects it) | 2 |
O is not a weather field in any spec, so the rest stays in the comment |
2 |
FAP rejects, the spec allows: {{ user-defined data, unknown $ sentence (kept as opaque), $GPVTG |
3 |
Corrupted Mic-E packet is Malformed (FAP's accept_broken_mice guesses a repair) |
1 |
| Mic-E hex telemetry has no sequence number (FAP reports 0) | 1 |
| NMEA course keeps its precision, 27.9 (FAP rounds to 28) | 1 |
GLL with no time or fix status is Malformed; nothing is invented |
1 |
Telemetry: comment kept / short report keeps the channels sent / non-numeric channel is Malformed |
3 |
Lowercase parm. is an ordinary message (APRS101 keywords are upper case) |
1 |
| Weather symbol with no weather fields is a position with a comment | 1 |
Some rows have more than one reason, so the column adds up to more than 52.
No longer asserted: FAP's own extra fields (display resolution, raw timestamps, warnings, indoor sensors) for about 430 rows. They describe FAP's output, not our schema, and stay in source_expected as provenance. The 15 FAP PHG rows still check theirs from public models.
Parser and API changes
1. The embedded ! search is now opt-in (new parameter)
- APRS101 p.18 (X1J) allowed a
!position within the first 40 bytes of other text, and the APRS 1.1 addendum deprecated that search. Until now the parser always did it, and the adapter hid that from the core vectors OTH-033/035. - New:
parse_packet,parse_payloadandparse_tnc2_linetakelegacy_positions: bool = False. By default such text is aBeacon;Truerestores theLegacyPositionsearch. A non-bool value raisesTypeError. - Corpus: historical-1.0 and FAP rows parse with
legacy_positions=True, since those receivers do this search.
2. Mic-E type codes follow APRS 1.2 (fixes review finding #1)
- The rules come from aprs.org/aprs12/mic-e-types.txt. In 2008 it redefined a leading
`or'as a device type byte, replacing APRS101's hex telemetry. A declared suffix at the end of the status text names the model. device_codenow holds the type byte plus any declared suffix, for example`_%(Yaesu FTM-400DR),'|3(Byonics TinyTrack3),]=(Kenwood TM-D710) or>^(TH-D74). The comment is the text between them, and the encoder writes the suffix back at the end.- Legacy hex telemetry is read only in its complete APRS101 form (4 hex digits after
`, 10 after') and only when no declared suffix ends the text. The core vectors MIC-261/262 are unchanged. - No more
Malformedfrom markers: ordinary Yaesu and Byonics packets used to beMalformed("invalid Mic-E hex telemetry"). No type byte or suffix can do that now. - Only declared suffixes are recognized. The "awaiting confirmation" entries are left out, so ordinary text such as "text-3" isn't stripped. A leading space, also listed as a type byte, stays comment text.
- Sending:
check_packetnow reports what used to fail only inside the encoder:- an undeclared type code;
- a type code combined with hex or binary telemetry, since both use the first status byte;
- a comment that would be read back as a different type code, for example
"Hi_%"or a comment starting with]. This uses a new encode-and-read-back check of the status text.
- Encoder byte order: hex or binary telemetry is now written immediately after the symbol, then altitude, then the grid, matching the decoder.
3. Short telemetry is a nonconforming packet, not Malformed
- Missing channels:
T#001,42andT#1,1,,3,,5decode with only the channels sent. They are marked nonconforming against APRS101, so sending is refused. This follows the parser's own rule: aPacketwhenever every field decodes, with departures from APRS rules reported as nonconforming. - Digital bits: when channels are missing, eight binary digits at the end are read as the digital bits. They can never be a valid APRS101 analog value (0..255). Before, they were silently read as a fifth analog value of 0 (BAD-102).
- Stricter numbers: channel values must be plain decimals. The decoder used
float(), which acceptedinf,nan,1e5and-1.; those are nowMalformed("invalid telemetry channel"). - Encoder: it writes only the parts present, so a short report round-trips through
serialize_payloadinternals without gaining a trailing comma.
4. APRS 1.2 weather fields (schema addition)
- These come from aprs.org/aprs12/weather-new.txt.
- New
Weatherfields:radiation, in the new unitnSv/h.Xnnnis two digits times a power of ten:X123is 12 × 10³ nSv/h.battery_voltage, in the new unitvolts.Vnnnis tenths of a volt:V128is 12.8 V....means "not available" for both.
- Bug fixed:
X123andV128used to be read as weather software/unit codes (softwareX, unit123). The adapter hid this by clearing the codes. Real codes after these fields, as inX123V128XRSW, still decode. - Water level, two spellings told apart by syntax:
- A decimal value (
F20.1,f-1.2) is the APRS 1.1 note's feet or meters, unchanged (core vectors WX-063 to 066). - Four bytes without a decimal point (
F0123,F-042,F....) are the 1.2 proposal's tenths of a foot.F0123used to decode as 123 ft; it is now 12.3 ft. - On re-send, a value is written in the decimal form (
F12.3). An unavailable feet reading is written asF.....
- A decimal value (
- Sending:
check_packetreports radiation that can't be written as two digits times a power of ten, and battery voltage too large for its three-digit field. For positionless reports, it also encodes and reads back the report, and reports a comment that would be read back as weather fields (for example"X123 hot"). - FAP rows: the generator now maps FAP's
Radiation,BatteryVoltageandWaterLevelinto these fields. FAP reports water level in meters, which is converted to feet.
5. Malformed.recovered now covers objects
- An object whose timestamp is invalid (for example
;TestObj *000000z…) now recovers its name, alive flag, timestamp fields, position, symbol and comment. - Recovery only runs when bytes 10–16 look like a timestamp, so an object with no timestamp (BAD-085) isn't misread.
6. Removed: the "compatibility" telemetry encoding
- The adapter was the only thing that produced this value, and no vector expected it. It is removed from
TelemetryEncoding, the message encoder, the conformance rules and the docs.
Tests
- Corpus test (
tests/packets/test_corpus.py): one code path, parse then compare. It uses RF for strict-envelope and AX.25 rows, APRS-IS otherwise, andlegacy_positionsfor historical and FAP rows.SCHEMA_NATIVE_IDSand the temporary pending list are gone. - Round-trip test: parses with the same
legacy_positionssetting. Its exceptions were updated with the exact current reasons. FAP-348 and FAP-430 now round-trip and were removed. - Schema-migration tests: these asserted FAP's values (mph, no NMEA symbol, course 28). They now accept a recorded divergence, and assert the newly mapped radiation, battery and water-level values.
- Contract tests: they patch the public
parse_packetinstead of the adapter. - New regression tests:
- Mic-E type codes (9 cases) and the checks before sending them;
- the
legacy_positionsoption, including the boundary at byte 40; - the APRS 1.2 weather fields (10 cases) and the checks before sending them;
- short telemetry, and plain-decimal channel values;
- object recovery after an invalid timestamp.
Counts
| Before | Now | |
|---|---|---|
| Rows compared through the adapter | ~1,500 | 0 |
| Successful round trips (permissive envelope policy) | 1,303 | 1,305 |
| Refusals with recorded reasons (malformed / conformance) | 261 (181 / 80) | 259 (176 / 83) |
Valid under the default APRS_IS policy |
1,288 | 1,290 |
3,543 tests pass on Python 3.11, 3.12 and 3.15. ruff, mypy, black and prettier are clean, and generate_vectors.py reproduces the JSONL exactly.
Public API changes in this round
parse_packet,parse_payloadandparse_tnc2_linetakelegacy_positions=False. Embedded!positions are no longer found by default.- Mic-E
device_codeis now the APRS 1.2 type code (type byte plus declared suffix).`and'markers no longer produceMalformed. Weather.radiationandWeather.battery_voltageare new, with the new unitsnSv/handvolts.TelemetryEncodingno longer includes"compatibility".- Classic telemetry with missing channels is a nonconforming
Packet. Non-decimal channel values (inf,nan,1e5,-1.) areMalformed. Fwater level with four bytes and no decimal point is read in tenths of a foot.
Still open
- Mic-E grid (review finding #9): the grid still lives in
device_code, for example>IO91SX/G. - Encode-time checks (finding Move APRS tests to JSONL case vectors, compared against FAP and live traffic #6): composing Mic-E still requires a destination computed by hand.
- Corpus vectors: there are no core vectors for Yaesu, Byonics or Kenwood-suffix Mic-E, or for the APRS 1.2 weather fields. They could be added as amended rows, as with WX-067 to 069.
- PR description: it still cites the old counts.
Stacked on #4: targets
scaffold/aprs-vectorsatb97beb7970c6b6fec2c0b6e30200f348bb87d96c.Implement APRS decoding and encoding across the vector corpus, including compressed positions, Mic-E, objects/items, weather, telemetry, messages, queries, status, and NMEA. Preserve packet models and their JSON schema. Replace the envelope profile strings with the agreed configurable policy API.
parse_packetand compared with its stored expectation; there is no receiver-emulation layer. Receiver state remains in the test harness.Envelope policies
All six envelope entry points now accept
policy=instead ofprofile=: parsing and serialization for packets and TNC2 lines,check_packet, andvalidate_packet. Public exports include the immutableEnvelopePolicy,RF, andAPRS_IS;EnvelopeProfileis removed.Nonemeans unlimited; zero is a real limit.RFuses AX.25 address syntax, eight path entries, and 256 information bytes.APRS_ISis the default. It enforces internet address formats, including that SSID-0is never sent, and a 512-byte line limit.The new default is stricter than the former monitor profile. An explicit permissive policy,
EnvelopePolicy(".*", ".*", ".*"), preserves representational round trips for deliberately invalid addresses.APRS_IS.checksum_valid. Sending never re-signs a sentence known to be corrupt.Parsing and composing
Embedded
!positions are opt-in.!position within the first 40 bytes of other text, and the APRS 1.1 addendum deprecates that search.parse_packet,parse_payloadandparse_tnc2_linetakelegacy_positions: bool = False. By default such text is aBeacon;Truereturns aLegacyPosition.legacy_positions=True.Mic-E
device_codeholds the type byte plus any declared suffix:`_%(Yaesu FTM-400DR),'|3(Byonics TinyTrack3),]=(TM-D710),>^(TH-D74).Malformed.check_packetreports an undeclared type code, a type code combined with telemetry, and a comment that would be read back as a different type code.l–u, following APRS101's table, never as control bytes.Weather
Weather.radiation(Xnnn, nSv/h) andWeather.battery_voltage(Vnnn, volts), with...meaning not available. These are no longer misread as software/unit codes.F20.1keeps the APRS 1.1 meaning (feet). Four bytes without a decimal point, such asF0123, are the 1.2 proposal's tenths of a foot. Unencodable radiation or battery values are reported before sending.Telemetry
Packetholding only the channels sent, so sending refuses it. Eight binary digits at the end are read as the digital bits.inf,nan,1e5and-1.areMalformed."compatibility"telemetry encoding is gone.Other
Directs= are fine…) stays aMessageinstead of becomingMalformed.Malformed.recoverednow covers objects with an invalid timestamp, as it already did for positions and status reports.PHG/PHGR
PHGabcdr/rate alphabet (0–9,A–Z) and consume its required separator. A digit or letter without the slash remains comment text. Encode the rate and separator before other extensions.Stored vector schema and FAP divergences
The finalized public schema is stored in the JSONL expectations themselves. The offline migration rewrites expectations for equivalent weather units, compressed wind nesting, legacy-position and vendor-weather wrappers, typed NMEA observations, and Mic-E device codes. It also maps FAP's radiation, battery and water-level readings into the new weather fields.
tests/fap_divergences.jsonrecords the 52 FAP rows where FAP's interpretation differs from the specification-following public decoder, each with its reasons. Examples: comment whitespace kept as sent, disputed wind units, NMEA default symbols and precision, Mic-E type codes, invalid timestamps and incomplete GLL or telemetry that FAP accepts, and packets FAP refuses but the spec allows. The generator applies them and records the reasons in each row'sdivergencefield;outcomeandsource_expectedstay as FAP captured them.source_expected, without assertions. PHG rows still check theirs from public models.Validation
On Python 3.11, 3.12 and 3.15:
See the envelope policy documentation for migration and customization, and the compatibility-adapter removal comment for details of the latest round.
🤖 Generated with Claude Code