diff --git a/CHANGELOG.md b/CHANGELOG.md index 7af887015..ddde9f1c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,7 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` to `1.5.0b3` (all 2026-09-19) and - `register` refuses what `get` refuses. `TCP.register(70000, 'x')` and `Hardware.register(70000, 'x')` minted a member that `get` then returned, because no registry overrode the `_validate_value` hook `register` calls. Each generated registry, `AppType` included, now overrides it with the check its `_missing_` makes, raising `EnumValueError`. `get(value, default)` on a value the hook refuses still returns `default` and logs nothing ([#1792](https://github.com/JarryShaw/PyPCAPKit/issues/1792), [#1818](https://github.com/JarryShaw/PyPCAPKit/pull/1818), [#1816](https://github.com/JarryShaw/PyPCAPKit/issues/1816), [#1823](https://github.com/JarryShaw/PyPCAPKit/pull/1823); the quiet miss is [#1845](https://github.com/JarryShaw/PyPCAPKit/issues/1845), [#1873](https://github.com/JarryShaw/PyPCAPKit/pull/1873), under `pcapkit.corekit`). `AppType.register()` and `OptionType.register()` keep that rejection loud inside a quiet `get`, as `EnumRegistry.register()` does ([#1916](https://github.com/JarryShaw/PyPCAPKit/issues/1916), [#1927](https://github.com/JarryShaw/PyPCAPKit/pull/1927)). - `Command.get()` refuses a key that is not a `str` with `EnumValueError`, as `Method.get()` does. `get(b'\x01')` minted a member keyed by `bytes`, and `get(-1)` raised `AttributeError` whatever its `default` ([#1824](https://github.com/JarryShaw/PyPCAPKit/issues/1824), [#1826](https://github.com/JarryShaw/PyPCAPKit/pull/1826)). - IANA's port spans that name no transport answer `AppType.get()` with the asking registry's transport rather than `TransportProtocol.undefined`, as its `unknown` fallback for a port in no span already did: `TCP.get(8012)` is `unassigned [tcp]`. A port IANA lists on other transports only stays `unknown`: 8011 is `reserved [tcp]` and `bier-echo [udp]`, and `unknown [dccp]` and `unknown [sctp]`. Lookups still mint nothing ([#860](https://github.com/JarryShaw/PyPCAPKit/issues/860)) and an out-of-range port still raises ([#2048](https://github.com/JarryShaw/PyPCAPKit/issues/2048), [#2052](https://github.com/JarryShaw/PyPCAPKit/pull/2052)). +- `Method._missing_` resolves an HTTP method whose registered name differs from its own value, so `Method('baseline-control')` and `Method('version-control')` give `BASELINE_CONTROL` and `VERSION_CONTROL` rather than minting an unregistered member: the lookup falls back to the value map on the upper-cased value before it mints. `Method.get()` stays case-sensitive ([#2284](https://github.com/JarryShaw/PyPCAPKit/issues/2284), [#2351](https://github.com/JarryShaw/PyPCAPKit/pull/2351)). ### pcapkit.corekit @@ -106,6 +107,7 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` to `1.5.0b3` (all 2026-09-19) and - defects off CPython on Linux. On GraalPy, `SeekableReader`'s reads aliased a buffer a later slide rewrote, so they copy out of it, and a `MultiDict` nested each value list, so it is built from a `dict` ([#1963](https://github.com/JarryShaw/PyPCAPKit/issues/1963), [#1974](https://github.com/JarryShaw/PyPCAPKit/pull/1974)); and a sentinel type refused the cache writes `_py_abc` makes, so `isinstance` against it raised `TypeError` ([#1960](https://github.com/JarryShaw/PyPCAPKit/issues/1960), [#1966](https://github.com/JarryShaw/PyPCAPKit/pull/1966)). `Field.pack` hands `struct` a `bytearray` as `bytes` for the `s`, `p` and `c` formats, which PyPy refuses for `s` and `p` and GraalPy accepts for `c` where CPython does not, so it packs alike on every interpreter ([#2004](https://github.com/JarryShaw/PyPCAPKit/issues/2004), [#2007](https://github.com/JarryShaw/PyPCAPKit/pull/2007), [#2015](https://github.com/JarryShaw/PyPCAPKit/issues/2015), [#2018](https://github.com/JarryShaw/PyPCAPKit/pull/2018)). On Windows, `SeekableReader(buffer_save=True)` with no `buffer_path` could not reopen its temporary file; it creates one it can, and deletes it on `close()` or collection ([#2098](https://github.com/JarryShaw/PyPCAPKit/issues/2098), [#2105](https://github.com/JarryShaw/PyPCAPKit/pull/2105)). - `PortEnumField` and `SwitchField` are exported from `pcapkit.corekit`, `pcapkit.corekit.fields` and `pcapkit.all`, and `SchemaField`, `nested_packet_context` and `BytesField` from the modules defining them, so `import *` binds them ([#2174](https://github.com/JarryShaw/PyPCAPKit/issues/2174), [#2191](https://github.com/JarryShaw/PyPCAPKit/pull/2191)). - a `ListField` callback is handed the per-item copy it configures, not the shared field, and a `SchemaField` with a callable length sizes its template from the resolved length rather than `1024s`. No parse in the library reached either ([#2173](https://github.com/JarryShaw/PyPCAPKit/issues/2173), [#2190](https://github.com/JarryShaw/PyPCAPKit/pull/2190)). +- `PortEnumField` range-checks the port it is handed, so a directly built TCP, UDP or SCTP schema raises `FieldValueError` for a port outside the field's width, where `srcport=70000` was masked to `0x1170` and `-1` to `0xffff`. `NumberField` keeps masking, which is its documented contract; the `NumberField` and `Schema` docstrings now say that direct construction range-checks nothing, and name where the checks live ([#2314](https://github.com/JarryShaw/PyPCAPKit/issues/2314), [#2335](https://github.com/JarryShaw/PyPCAPKit/pull/2335)). ### pcapkit.dumpkit @@ -128,6 +130,7 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` to `1.5.0b3` (all 2026-09-19) and - the `engine='pypcap'` and `engine='pcap_ct'` extraction engines, independent distributions of the same `libpcap` interface, and `engine='pypcapfile'` ([#386](https://github.com/JarryShaw/PyPCAPKit/pull/386), [#405](https://github.com/JarryShaw/PyPCAPKit/pull/405)). They buy speed by doing less -- neither `pypcap` nor `pcap_ct` dissects at all, so they offer no reassembly or flow tracing, and `pypcapfile` has no IPv6 decoder. Install only **one** of `pypcap` and `pcap-ct`: both own the top-level `pcap` module, and with both present `pcap-ct` wins the import and the other becomes unselectable. The missing interface constants `PyPCAP`, `PCAP_CT` and `PyPCAPFile` are now exported alongside `DPKT`, `Scapy`, `PyShark` and `PCAPKit` ([#412](https://github.com/JarryShaw/PyPCAPKit/pull/412)). Python 3.11 is the last interpreter on which every built-in engine can run, and even there `pypcap` and `pcap-ct` cannot coexist. - `EngineBase.unsupported_reason()`, a preflight every engine answers and `Extractor.run()` consults before anything is imported. An engine that cannot run in the current environment now gives one warning naming the real cause (a Python version, a missing `tshark` or `libpcap`, the wrong `pcap` distribution) and a clean fall back to `pcapkit`'s own parser, rather than an error from inside the third-party package ([#396](https://github.com/JarryShaw/PyPCAPKit/pull/396), [#405](https://github.com/JarryShaw/PyPCAPKit/pull/405)). The same holds for an engine whose package is not installed at all ([#1045](https://github.com/JarryShaw/PyPCAPKit/issues/1045)). - `conflict` on the reassembly data models: absolute, inclusive ranges where two fragments claimed the same span with different bytes, previously lost silently on both the IP ([#482](https://github.com/JarryShaw/PyPCAPKit/pull/482)) and TCP ([#443](https://github.com/JarryShaw/PyPCAPKit/issues/443), [#478](https://github.com/JarryShaw/PyPCAPKit/pull/478)) paths. +- the declarations that pair a `__proto__` dispatch registry with the enumeration its codes come from, replacing the module-level tables that held the pairing apart from the `__proto__` they describe. `__proto_key__`, on each class owning such a registry (`Link`, `Internet`, `Frame`, `PCAPNG`, `TCP`, `UDP` and `SCTP`), names that enumeration and whether a bare `int` is also valid ([#2392](https://github.com/JarryShaw/PyPCAPKit/pull/2392)); `__proto_dest__`, on the `LinkType`, `EtherType`, `TransType` and `PayloadProtocolIdentifier` registries, names the destination each enumeration keys, which is how `code=` infers one without an explicit destination. Both are rendered attributes, and the vendor crawlers generate the latter from a new `PROTO_DEST` ([#2403](https://github.com/JarryShaw/PyPCAPKit/pull/2403)). #### Changed @@ -171,6 +174,10 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` to `1.5.0b3` (all 2026-09-19) and - `engine='pypcap'` warns and falls back to the default engine when the `pcap` module is installed but cannot load, as `engine='pcap_ct'` already did, where it raised the loader's `OSError` ([#2132](https://github.com/JarryShaw/PyPCAPKit/issues/2132), [#2133](https://github.com/JarryShaw/PyPCAPKit/pull/2133)). - TCP reassembly bounded its holes with `sys.maxsize`, which on GraalPy and 32-bit builds is `2**31 - 1`, inside TCP's sequence space, so a stream past it never completed. The bound is `HOLE_OPEN_BOUND`, `2**63 - 1`, the value 64-bit builds already used ([#1959](https://github.com/JarryShaw/PyPCAPKit/issues/1959), [#1965](https://github.com/JarryShaw/PyPCAPKit/pull/1965)). - the method-name registrars, such as `register_ipv4_option`, and `register_linktype`, `register_pcap`, `register_pcapng`, `register_ethertype` and `register_transtype` refuse a `code` that is not an enum member with `RegistryError` before touching a registry, where they registered it and then raised `AttributeError`, leaving the entry behind ([#2186](https://github.com/JarryShaw/PyPCAPKit/issues/2186), [#2193](https://github.com/JarryShaw/PyPCAPKit/pull/2193)). +- `pcapkit.foundation.registry.protocols.register_protocol_code()`, and the `code=` class keyword, check every destination's key before registering any of them. A key belonging to another enum, such as `{Frame: TransType.ICMP}`, is refused with `RegistryError` where it replaced the shipped `LinkType.ETHERNET` entry, and a bad target late in the list no longer leaves the earlier ones registered. A destination keeping a registry of its own, such as `Loopback` and `IPX`, is not checked against another enum ([#2268](https://github.com/JarryShaw/PyPCAPKit/issues/2268), [#2343](https://github.com/JarryShaw/PyPCAPKit/pull/2343)). +- the number registrars `register_tcp`, `register_udp`, `register_sctp` and `register_apptype` refuse a `bool` code with `RegistryError`, where `register_tcp(True, ...)` registered port 1, since `bool` is an `int` subclass ([#2269](https://github.com/JarryShaw/PyPCAPKit/issues/2269), [#2344](https://github.com/JarryShaw/PyPCAPKit/pull/2344)). +- IP reassembly reports a datagram that is not itself fragmented but whose BUFID is already buffered: it submits that buffer as incomplete and then reassembles the datagram in a buffer of its own, per [RFC 791 Section 3.2](https://datatracker.ietf.org/doc/html/rfc791#section-3.2), where the datagram was never reported at all. On IPv6 an atomic fragment is reassembled in a buffer of its own and submitted without flushing the datagram already being reassembled, so that one's final fragment still completes it, per [RFC 6946 Section 4](https://datatracker.ietf.org/doc/html/rfc6946#section-4) ([#2270](https://github.com/JarryShaw/PyPCAPKit/issues/2270), [#2345](https://github.com/JarryShaw/PyPCAPKit/pull/2345)). +- the PyPCAP, pcap-ct and pypcapfile engines warn with `AttributeWarning`, naming the engine, that they ignore a parsing context supplied through `Extractor`'s `context=`, as the dpkt, scapy and pyshark engines already did, where they dropped it silently ([#2313](https://github.com/JarryShaw/PyPCAPKit/issues/2313), [#2332](https://github.com/JarryShaw/PyPCAPKit/pull/2332)). ### pcapkit.interface @@ -178,6 +185,10 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` to `1.5.0b3` (all 2026-09-19) and - `pcapkit.interface.registry`, making public the registry-entry helpers `same_entry`, `restore_shipped` and `restore_shipped_dumper` ([#1519](https://github.com/JarryShaw/PyPCAPKit/issues/1519), [#1568](https://github.com/JarryShaw/PyPCAPKit/pull/1568)). +#### Fixed + +- the messages the interface functions and the helpers behind them raise and warn read in the house `: : ... {value!r}` form. `reassemble` and `trace` lowercase their unsupported-protocol error and quote the protocol; `follow_tcp_stream` warns `using default engine instead`, the wording `Extractor` already used, and quotes an engine that has no reassembly adapter; `TraceFlowBase`'s output-format warning is lowercased; and `Extractor.make_name` quotes the format it does not know, where an empty one ended the message at the colon with nothing after it ([#2303](https://github.com/JarryShaw/PyPCAPKit/issues/2303), [#2374](https://github.com/JarryShaw/PyPCAPKit/pull/2374)). + ### pcapkit.protocols #### Added @@ -226,6 +237,7 @@ Layer-limited extraction: `layer='link'` no longer sets the termination flag on - **a breaking change to** `NTPTimestamp`: it and `PMIPv6Timestamp` are `Info` classes in `pcapkit.protocols.data.internet.mh` rather than tuples, and `NTPTimestamp` keeps no alias at its 1.4.x home in `pcapkit.protocols.internet.mh`. As Mappings they no longer index, unpack, compare or hash as a tuple does; `.seconds` and `.fraction` still work, and the dumpers write both as keys ([#1886](https://github.com/JarryShaw/PyPCAPKit/issues/1886), [#1887](https://github.com/JarryShaw/PyPCAPKit/pull/1887)). - **a breaking change to** rebuilding from `to_dict()`: the payload is dispatched only from the `__next_name__` and `__next_type__` keys, and the IPv6 extension-header chain only from `__exthdr__`, never from key order, which on PyPy follows an earlier instance's attribute layout and so picked the wrong layer. `to_dict()`, iteration and `len()` carry the dunder keys a new `Info.__included__` names, and `Info.__excluded__` only keeps a key out of `repr`, `str` and the dumps. A hand-written dict needs those keys, `__short_read__` among them, to rebuild, and a custom `Info` lists its own dunder keys in `__included__` ([#2003](https://github.com/JarryShaw/PyPCAPKit/issues/2003), [#2014](https://github.com/JarryShaw/PyPCAPKit/pull/2014)). - **a breaking change to** `PCAPNG.context`: the block's section `Context`, which it returned from 1.0.0 through 1.4.x, is `PCAPNG.section_context`, with no alias, and `context` is the caller-supplied `ContextRegistry` every protocol inherits from `ProtocolBase`, on a Section Header Block too. `PCAPNG.section_context` raises `UnsupportedCall` where `context` did: on a block that holds no section, such as a Section Header Block the `PCAPNG` engine has not assigned the section it opens. Code reading `block.context` for the section reads `block.section_context` ([#2172](https://github.com/JarryShaw/PyPCAPKit/issues/2172), [#2205](https://github.com/JarryShaw/PyPCAPKit/pull/2205)). +- **a breaking change to** `FTP_DATA`: it derives from `Application` as well as `Raw`, so its `__layer__` names the application layer where it was `None` -- a `layer='application'` extraction stops at it, and `extract(layer=FTP_DATA)` resolves to that layer rather than to `'none'`. Raw still does the parsing and the rebuild, and `__index__` is the application layer's, raising `IntError` where it raised `UnsupportedCall`. The class moves to `pcapkit.protocols.application.ftp_data` with nothing left behind on the old path in `pcapkit.protocols.application.ftp`, where it shipped from 1.0.0 through 1.4.1 ([#2285](https://github.com/JarryShaw/PyPCAPKit/issues/2285), [#2357](https://github.com/JarryShaw/PyPCAPKit/pull/2357)). - **a breaking change to** HIP's `REG_FAILED` parameter: its first octet is the Failure Type, and it carries no lifetime ([RFC 8003 Section 4.5](https://datatracker.ietf.org/doc/html/rfc8003#section-4.5)), so `pcapkit.protocols.data.internet.hip.RegFailedParameter` replaces `lifetime` with `failure_type`, a `RegistrationFailure`, with no alias, and its `reg_type` holds `Registration` values rather than failure types. `HIP._make_param_reg_failed()` takes `failure_type` in place of `lifetime`, and `reg_failed` takes registration types. The octets are unchanged ([#2309](https://github.com/JarryShaw/PyPCAPKit/issues/2309)). - **a breaking change to** the HTTP/2 `RST_STREAM` and `GOAWAY` builders: a frame constructed without `error` carries `NO_ERROR` (`0x0`), the code [RFC 9113 Section 7](https://datatracker.ietf.org/doc/html/rfc9113#section-7) gives a condition that is not the result of an error, rather than `HTTP_1_1_REQUIRED` (`0xd`), which asks the peer to use HTTP/1.1 instead. A `make` that omits `error` emits different octets; one that passes it is unchanged ([#2311](https://github.com/JarryShaw/PyPCAPKit/issues/2311)). @@ -351,6 +363,33 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r - IPX dissects its payload from a registry of its own, `IPX.__proto__`, keyed by IPX Packet Type, where it looked the Packet Type up as an IP protocol number, so a type 17 (NCP) payload parsed as `IPX:UDP` and type 4 as `IPX:IPv4`. No IPX-native protocol ships, so the payload is `Raw` unless one is registered ([#2169](https://github.com/JarryShaw/PyPCAPKit/issues/2169), [#2187](https://github.com/JarryShaw/PyPCAPKit/pull/2187)). - HIP's `ECHO_RESPONSE_UNSIGNED` maker builds an `EchoResponseUnsignedParameter` schema, where it built the request's; the octets were already right ([#2170](https://github.com/JarryShaw/PyPCAPKit/issues/2170), [#2188](https://github.com/JarryShaw/PyPCAPKit/pull/2188)). - **a breaking change to** PCAP-NG's Interface Description Block maker: `snaplen` defaults to 0, which the PCAP-NG specification reads as no limit, where its default of `2**64 - 1` was truncated to `0xFFFFFFFF` by the 32-bit SnapLen field and the made `info` disagreed with the octets. A block made without `snaplen` writes 0 there, and a `snaplen` outside `0` to `0xFFFFFFFF` raises `ProtocolError` where it was truncated silently ([#2278](https://github.com/JarryShaw/PyPCAPKit/issues/2278)). +- HIP parameters read and rebuilt against the wrong definition. `TRANSACTION_ID` and `OVERLAY_ID` keep the identifier's wire width on rebuild, through a new `id_len` field, where the length was sized from the integer's own `bit_length` and the leading zero octets were lost ([#2271](https://github.com/JarryShaw/PyPCAPKit/issues/2271)); the `HIP_TRANSFORM` maker raises `ProtocolError` for the HIPv2 parameter its own reader already rejects ([#2275](https://github.com/JarryShaw/PyPCAPKit/issues/2275)); `LOCATOR_SET`'s maker derives the locator `type` from whether `spi` was given, where an `spi=` locator was left at type 0, a combination the locator selector itself rejects, and an explicit `type` that disagrees raises naming both values ([#2276](https://github.com/JarryShaw/PyPCAPKit/issues/2276)); `REG_INFO`, `REG_REQUEST` and `REG_RESPONSE` lifetimes decode as the `2**((v-64)/8)` seconds [RFC 8003 Section 4.1](https://datatracker.ietf.org/doc/html/rfc8003#section-4.1) defines, with a zero octet its own case, where the octet was read as raw seconds, and their makers take either a duration or the raw exponent ([#2281](https://github.com/JarryShaw/PyPCAPKit/issues/2281)); `OVERLAY_TTL`'s `ttl` is the hop count [RFC 6079](https://datatracker.ietf.org/doc/html/rfc6079) defines rather than a `timedelta` ([#2282](https://github.com/JarryShaw/PyPCAPKit/issues/2282)); and `CERT`'s `cert_group` is a plain `int`, since [RFC 8002 Section 2](https://datatracker.ietf.org/doc/html/rfc8002#section-2) defines no registry for it, where it was decoded through the DH Group registry ([#2283](https://github.com/JarryShaw/PyPCAPKit/issues/2283)). `HOST_ID`'s maker prefixes its errors in the house `HIPv{version}: [ParamNo {code}]` form ([#2299](https://github.com/JarryShaw/PyPCAPKit/issues/2299)). The data-model and signature changes are breaking ([#2400](https://github.com/JarryShaw/PyPCAPKit/pull/2400)). +- TCP's Multipath TCP options: an unknown subtype sizes its declared `length` from what the schema writes, so one built with `data=b''` declares a Length of 3, the header octets it always packs, rather than 2 ([#2272](https://github.com/JarryShaw/PyPCAPKit/issues/2272)); and a parsed subtype schema no longer carries an `option` attribute it does not declare and no maker mirrors -- the declared `kind` already holds `Multipath_TCP`, and the octets were identical either way ([#2293](https://github.com/JarryShaw/PyPCAPKit/issues/2293), [#2396](https://github.com/JarryShaw/PyPCAPKit/pull/2396)). +- PCAP-NG's Decryption Secrets Block maker builds from a `bytes` `secrets_data`, its `b''` default included, by reading those octets through the same secrets schema the parse side selects, where every secrets type raised because the readers expect a parsed schema ([#2273](https://github.com/JarryShaw/PyPCAPKit/issues/2273), [#2346](https://github.com/JarryShaw/PyPCAPKit/pull/2346)). +- HTTP/2's `make` defaults `frame` to the frame type's own constructor rather than to a `b''` packed verbatim with Length 0, so `PRIORITY`, `RST_STREAM`, `PUSH_PROMISE`, `PING`, `GOAWAY` and `WINDOW_UPDATE` build a frame that reads back, where `HTTP(type=..., sid=...)` raised; an explicit `b''` still packs raw, and the default is byte-identical to `frame={}`. `settings=None` builds the parameterless SETTINGS frame [RFC 9113 Section 6.5](https://datatracker.ietf.org/doc/html/rfc9113#section-6.5) permits rather than raising ([#2274](https://github.com/JarryShaw/PyPCAPKit/issues/2274), [#2395](https://github.com/JarryShaw/PyPCAPKit/pull/2395)). +- IPv4's Timestamp option builds from its own default: `timestamp=None` gives a Timestamp-only option with `counts` empty slots, where the default raised `ProtocolError` because the maker took only a list or a dict, and any other value is reported with its `repr` ([#2277](https://github.com/JarryShaw/PyPCAPKit/issues/2277), [#2347](https://github.com/JarryShaw/PyPCAPKit/pull/2347)). +- a short read raises `StructError` for every size, not only for the sizes with a `struct` format code: the `int.from_bytes` path checks the length it got, where it returned a smaller integer. `quiet` still returns the partial value ([#2279](https://github.com/JarryShaw/PyPCAPKit/issues/2279), [#2349](https://github.com/JarryShaw/PyPCAPKit/pull/2349)). +- the Mobility Header's LMAA option carries the code its address family requires: the code defaults to that family's own and an explicit one that disagrees raises `ProtocolError`, where `address='10.0.0.1'` packed the IPv6 code anyway ([RFC 5949 Section 6.2.2](https://datatracker.ietf.org/doc/html/rfc5949#section-6.2.2)). A captured mismatch still rebuilds byte for byte ([#2280](https://github.com/JarryShaw/PyPCAPKit/issues/2280), [#2350](https://github.com/JarryShaw/PyPCAPKit/pull/2350)). +- a base class's `make` keywords stop reaching a subclass whose own `make` ignores them. A new `ProtocolBase.__own_make_keywords__` marker, read per class like the `__keywords__` opt-out, keeps a base's parameters out of the allowlist once a subclass overrides `make`, and `IPv6_Ext` sets it, so `IPv6_Route(len=99)` is refused where it built the same octets as `IPv6_Route()` and a misspelling was steered to the dead keyword ([#2286](https://github.com/JarryShaw/PyPCAPKit/issues/2286), [#2358](https://github.com/JarryShaw/PyPCAPKit/pull/2358)). +- TCP's, UDP's and SCTP's `make` raise `ProtocolError` for a `bool` port, where `UDP(srcport=True)` packed port 1 and `TCP(dstport=False)` port 0 ([#2287](https://github.com/JarryShaw/PyPCAPKit/issues/2287), [#2359](https://github.com/JarryShaw/PyPCAPKit/pull/2359)). +- SCTP's Supported Address Types and Missing Mandatory Parameter makers raise `ProtocolError` for a parameter type that is neither a `Parameter` member nor an `int`, a `bool` included, where `True` became parameter type 1, `False` raised `EnumValueError`, and a `str` or a `float` was stored unchecked ([#2288](https://github.com/JarryShaw/PyPCAPKit/issues/2288), [#2363](https://github.com/JarryShaw/PyPCAPKit/pull/2363)). +- `Loopback.make` raises `ProtocolError` for an address family that is not an `int`, from the same check as its range test, where `True` packed family 1, `'2'` and `None` raised `TypeError` and `2.0` raised `AttributeError` ([#2289](https://github.com/JarryShaw/PyPCAPKit/issues/2289), [#2364](https://github.com/JarryShaw/PyPCAPKit/pull/2364)). +- FTP's `make` raises `ProtocolError` for a command that is none of `str`, `bytes` and `bytearray`, a `bool` included, where its fallback read `cmmd.value`, so an `int`, a `bool` or a `float` raised a bare `AttributeError` while any object that happened to carry a `.value` was accepted unchecked. A `bytearray` builds as `bytes` ([#2290](https://github.com/JarryShaw/PyPCAPKit/issues/2290), [#2365](https://github.com/JarryShaw/PyPCAPKit/pull/2365)). +- `_make_pack`, and `_make_index` packing through it, raise `StructError` for a size with no `struct` format code, as the `struct` path does, where a value out of range, a non-integer or a bad size leaked `OverflowError`, `AttributeError` or `ValueError` ([#2291](https://github.com/JarryShaw/PyPCAPKit/issues/2291), [#2367](https://github.com/JarryShaw/PyPCAPKit/pull/2367)). +- a binary read at the end of the stream raises `StructError`, as an unpack does, where an exhausted or short stream raised `TypeError` out of `ord()` ([#2292](https://github.com/JarryShaw/PyPCAPKit/issues/2292), [#2368](https://github.com/JarryShaw/PyPCAPKit/pull/2368)). +- IPv4's SEC option warns about the bit it observed: the field termination indicator [RFC 1108](https://datatracker.ietf.org/doc/html/rfc1108) puts in the final protection authority octet is reported as set, with its value, where the message said the indicator was not set ([#2295](https://github.com/JarryShaw/PyPCAPKit/issues/2295), [#2372](https://github.com/JarryShaw/PyPCAPKit/pull/2372)). +- a maker that refuses a caller's argument names the argument and quotes the value, rather than reusing the reader's `invalid format` text, which says nothing about which argument was wrong: SCTP's chunk-flag, empty-data, hostname-termination and reserved-length guards, the Mobility Header's data-type, option-length, empty-data and Pad1 guards, TCP's `MP_CAPABLE` optional-field guard, and HIP's and PCAP-NG's fallback branches ([#2296](https://github.com/JarryShaw/PyPCAPKit/issues/2296), [#2405](https://github.com/JarryShaw/PyPCAPKit/pull/2405)), then HOPOPT's and IPv6-Opts' Pad1 warning and HTTP/2's and PCAP-NG's `make` dispatchers ([#2406](https://github.com/JarryShaw/PyPCAPKit/issues/2406), [#2407](https://github.com/JarryShaw/PyPCAPKit/pull/2407)). The readers still say `invalid format`, which is what a parse found. +- ESP's errors read in the house `ESP: : ... {value!r}` form: the messages raised by the cipher and integrity lookups, by `esp.SecurityAssociation`'s SPI, destination, AEAD, ICV, key-splitting, decryption and encryption checks, by the context registrar and by `make` all carry the prefix and quote the offending value, where none carried an `ESP:` prefix and most printed the value bare ([#2297](https://github.com/JarryShaw/PyPCAPKit/issues/2297), [#2366](https://github.com/JarryShaw/PyPCAPKit/pull/2366)). +- HTTP's version, data, start-line and settings errors carry the `HTTP:` prefix and the rejected value, as in `HTTP: invalid version: 3` and `HTTP/2: [Type 4] invalid settings: 42`, where the prefixes were missing or malformed, some were capitalised with a full stop, and some left the value out ([#2298](https://github.com/JarryShaw/PyPCAPKit/issues/2298), [#2373](https://github.com/JarryShaw/PyPCAPKit/pull/2373)). +- the capture guards read in that form too. The guards that reject a file's magic number or byte order carry the `PCAP:` and `PCAP-NG:` prefixes, and PCAP's magic-number guard says what it read ([#2300](https://github.com/JarryShaw/PyPCAPKit/issues/2300)); and PCAP-NG's option guards drop their trailing full stop, with the block-placement guard naming the block type through the `BlockType` registry rather than printing the raw number `_type` holds while a block is being made ([#2301](https://github.com/JarryShaw/PyPCAPKit/issues/2301), [#2398](https://github.com/JarryShaw/PyPCAPKit/pull/2398)). +- `IPv6_Ext`'s version gate reads `invalid version: must be 6, not {version!r}`, in the house form and quoting the rejected value, where its bespoke tail read `only valid for IPv6, got version=4` -- naming the version it requires in prose rather than as the number it compares against ([#2302](https://github.com/JarryShaw/PyPCAPKit/issues/2302), [#2397](https://github.com/JarryShaw/PyPCAPKit/pull/2397)). +- a trial parse kept as `Raw` because it raised no longer replays its warnings: the holding context honours its `discard` on an exception exit too, and the next-layer reader discards a raising trial's warnings under `beholder`, so an Ethernet frame cut inside its IPv4 header stops warning `packet length < 0` for a layer it does not have. Outside `beholder` the error reaches the caller and the warnings are still reported, and the Raw's error names the cut ([#2308](https://github.com/JarryShaw/PyPCAPKit/issues/2308), [#2339](https://github.com/JarryShaw/PyPCAPKit/pull/2339)). +- the declarations that described a value the code never produces now match it, so a type checker and the rendered API agree with what is stored: CALIPSO's `cmpt_bitmap` is the tuple of ints its reader stores, the SMF hash-based DPD option declares its own schema rather than its sibling's, an `RR` or `SSR` `route` that is always built is not `Optional`, `NoPayload` narrows `protochain` to `None`, ARP's address maker admits the `bytearray` its body handles, and the PCAP-NG flags stub and an undeclared `entry_raw` follow. HIP's `OverlayTTLParameter` and `ViaRVSParameter` join their modules' `__all__` and the package re-exports those lists require. Parsing and rebuilding are unchanged ([#2317](https://github.com/JarryShaw/PyPCAPKit/issues/2317), [#2377](https://github.com/JarryShaw/PyPCAPKit/pull/2377)). +- HIP's `parameters=[(code, data)]` form rebuilds a parsed parameter through its maker positionally, so the parameters whose maker keyword is spelled differently from the data model's own field name -- `suite_id` against `suites`, `group_id` against `group`, `reg_type` against `reg_request` and its siblings, and the nested locator keys -- keep their values, where they fell back to the maker's default or raised `AttributeError`: `HIP_TRANSFORM` rebuilt as `0241000000000000`, losing its `suite_id`. The dict form, as in `parameters=[(code, {'suites': [...]})]`, is unchanged ([#2352](https://github.com/JarryShaw/PyPCAPKit/issues/2352), [#2383](https://github.com/JarryShaw/PyPCAPKit/pull/2383)). +- PCAP-NG settles the byte order before a block maker runs, the way `pack` does -- the section's, else the one the packet data declares, else the host's -- so an option or a record handed over as raw `bytes`, whose type code is read back out of those octets, builds in any block rather than raising `AttributeError: 'PCAPNG' object has no attribute '_byte'`. A Section Header Block still overrides it from its own Byte-Order Magic, and `__packet__=None` degrades to no packet data rather than raising `TypeError`, as `Schema.pack()` already treated it ([#2353](https://github.com/JarryShaw/PyPCAPKit/issues/2353), [#2369](https://github.com/JarryShaw/PyPCAPKit/pull/2369)). +- a PCAP-NG block made from raw `bytes` is read back off the octets `pack` has just written, as a reader of the file would, rather than off the schema that carries them, so a Decryption Secrets Block built that way parses and rebuilds byte for byte, where the read looked for the block's fields on the schema holding the octets and raised `AttributeError`. Octets that are no block raise `StreamEOFError` or `ProtocolError` ([#2354](https://github.com/JarryShaw/PyPCAPKit/issues/2354), [#2370](https://github.com/JarryShaw/PyPCAPKit/pull/2370)). +- a `str` enumeration name resolves where the namespace could not be inferred from the argument's own default. HOPOPT's and IPv6-Opts' `seed_type` and `tid_type` and HTTP/1's `status` default to `None`, so a name such as `'SEEDID_16_BIT_UNSIGNED_INTEGER'` raised `ProtocolNotImplemented` while the member and `int` forms of the same call worked ([#2356](https://github.com/JarryShaw/PyPCAPKit/issues/2356), [#2393](https://github.com/JarryShaw/PyPCAPKit/pull/2393)); and HIP's list-argument makers, where the name is an item of a list parameter rather than a bare argument, resolve too, with the enumerations they need imported at runtime rather than only under `TYPE_CHECKING` ([#2394](https://github.com/JarryShaw/PyPCAPKit/issues/2394), [#2401](https://github.com/JarryShaw/PyPCAPKit/pull/2401)). +- a PCAP-NG block made without an explicit timestamp reads its interface's resolution from the same lookup the maker already makes for the scale, so the Enhanced Packet Block, the obsolete Packet Block and the Interface Statistics Block default their timestamp where they raised `AttributeError: '_info'`, having no parsed data model to resolve the interface through yet ([#2362](https://github.com/JarryShaw/PyPCAPKit/issues/2362), [#2382](https://github.com/JarryShaw/PyPCAPKit/pull/2382)). ### pcapkit.toolkit @@ -370,6 +409,8 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r - the TCP adapters: pyshark accepts tshark's `True`/`False` TCP flags ([#1514](https://github.com/JarryShaw/PyPCAPKit/issues/1514), [#1527](https://github.com/JarryShaw/PyPCAPKit/pull/1527)); a TCP segment whose header the snapshot cut is traced and reassembled ([#1518](https://github.com/JarryShaw/PyPCAPKit/issues/1518), [#1556](https://github.com/JarryShaw/PyPCAPKit/pull/1556)); dpkt reads SYN, FIN and RST whatever the NS bit ([#1618](https://github.com/JarryShaw/PyPCAPKit/issues/1618), [#1621](https://github.com/JarryShaw/PyPCAPKit/pull/1621)); and the scapy adapters stop at the IP datagram's declared length ([#1584](https://github.com/JarryShaw/PyPCAPKit/issues/1584), [#1589](https://github.com/JarryShaw/PyPCAPKit/pull/1589)). - timestamps: dpkt and pypcapfile hand over the correctly rounded timestamp ([#1668](https://github.com/JarryShaw/PyPCAPKit/issues/1668), [#1690](https://github.com/JarryShaw/PyPCAPKit/pull/1690)), and a copied `DecimalTimestamp` keeps `orig_len` under `_pydecimal` ([#1647](https://github.com/JarryShaw/PyPCAPKit/issues/1647), [#1653](https://github.com/JarryShaw/PyPCAPKit/pull/1653)). - the third-party engines are best effort: the code each adapter carried only to match the default engine is reverted ([#1718](https://github.com/JarryShaw/PyPCAPKit/pull/1718), [#1721](https://github.com/JarryShaw/PyPCAPKit/pull/1721), [#1724](https://github.com/JarryShaw/PyPCAPKit/pull/1724), [#1727](https://github.com/JarryShaw/PyPCAPKit/pull/1727), closing [#1708](https://github.com/JarryShaw/PyPCAPKit/issues/1708) through [#1743](https://github.com/JarryShaw/PyPCAPKit/pull/1743)). What each engine leaves out, such as TCP behind a tunnel or an IPv6 extension header, is documented rather than warned about. The pypcapfile engine keeps its warning, once per capture, that it leaves TCP over IPv6 out ([#1513](https://github.com/JarryShaw/PyPCAPKit/issues/1513), [#1535](https://github.com/JarryShaw/PyPCAPKit/pull/1535)). +- the pcap-ct, PyPCAP, pypcapfile, pyshark and scapy adapters lead the messages below with their engine's name, read off the engine class's own `__engine_name__` rather than hand-typed, so a rename under `pcapkit/foundation/engines/` cannot leave an adapter spelling the old one: the pcap-ct and PyPCAP reassembly refusals, pypcapfile's address errors, its IPv6 `UnsupportedCall` and its TCP-over-IPv6 warning, pyshark's boolean-field, timestamp and missing-layer messages, and scapy's import-time warning, which quotes the installable name, all read in the house `: : ... {value!r}` form. The dpkt adapter and the adapters' `ModuleNotFound` raises keep the messages they had ([#2304](https://github.com/JarryShaw/PyPCAPKit/issues/2304), [#2399](https://github.com/JarryShaw/PyPCAPKit/pull/2399)). +- the IPv4 and IPv6 reassembly adapters take the first layer of that IP version, from the outside in, that is actually a fragment -- IPv4 with MF set or a non-zero offset, IPv6 with a Fragment header -- so a fragmented inner datagram in an IP-in-IP tunnel is reassembled on its own header, where it was reassembled whole under the tunnel's header, or dropped when the tunnel set DF or carried no Fragment header of its own. With no fragmented layer they take the outermost, as before. Applied to the default engine's PCAP and PCAP-NG adapters and to the scapy and dpkt ones, which picked the outermost too ([#2307](https://github.com/JarryShaw/PyPCAPKit/issues/2307), [#2338](https://github.com/JarryShaw/PyPCAPKit/pull/2338)). ### pcapkit.utilities @@ -414,6 +455,7 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r - `pcapkit.vendor.ipx` names its `sys.path` temporary `_path` rather than `path`, which read as a public name of the package, and the public-API test covers `pcapkit.vendor`, which it used to skip ([#1772](https://github.com/JarryShaw/PyPCAPKit/issues/1772), [#1779](https://github.com/JarryShaw/PyPCAPKit/pull/1779)). - the generated const comments render as written: a URL reference followed by another bracket no longer autolinks into it, a leading enumerator such as `(S)` is escaped, and a hyphenated word is no longer split across lines ([#1846](https://github.com/JarryShaw/PyPCAPKit/issues/1846), [#1847](https://github.com/JarryShaw/PyPCAPKit/issues/1847), [#1871](https://github.com/JarryShaw/PyPCAPKit/pull/1871)). - without the `vendor` extra, `pcapkit-vendor` and `python -m pcapkit.vendor` exit 1 with one line naming the missing packages and `pypcapkit[vendor]`, where they died with a traceback, and `--help` and `--version` still work. Importing a crawler then raises `ModuleNotFoundError` ([#1850](https://github.com/JarryShaw/PyPCAPKit/issues/1850), [#1875](https://github.com/JarryShaw/PyPCAPKit/pull/1875)). +- the ESP cipher and integrity crawlers strip the brackets off IANA's `[UNSPECIFIED]` "no document" cell before testing it against the unknown placeholder, so `pcapkit.const.esp.integrity` no longer reads `#: AUTH_DES_MAC [UNSPECIFIED] (DEPRECATED ...)` for `AUTH_DES_MAC` and `AUTH_KPDK_MD5` ([#2294](https://github.com/JarryShaw/PyPCAPKit/issues/2294), [#2371](https://github.com/JarryShaw/PyPCAPKit/pull/2371)). ### Project infrastructure @@ -475,6 +517,7 @@ This resolves [#548](https://github.com/JarryShaw/PyPCAPKit/issues/548), which r - `pcapkit-cli`'s help strings say what each option does: `-o` describes the output rather than the input, an extension is appended only under `--auto-extension`, and each format flag names the flags that take precedence over it ([#1796](https://github.com/JarryShaw/PyPCAPKit/issues/1796), [#1810](https://github.com/JarryShaw/PyPCAPKit/pull/1810)). - the Conda package: its vendored dependencies are the package's `dependencies` and its `all` extra, with their requirements, shipped with their data files rather than as `.py` files only ([#1814](https://github.com/JarryShaw/PyPCAPKit/issues/1814), [#1821](https://github.com/JarryShaw/PyPCAPKit/pull/1821)); the `osx-64` package is built on an Intel runner, where converting the arm64 build shipped arm64 binaries ([#1822](https://github.com/JarryShaw/PyPCAPKit/issues/1822), [#1825](https://github.com/JarryShaw/PyPCAPKit/pull/1825)); and on Intel macOS it pins `cryptography` 48.0.1, which has a wheel for that platform where the newer pin has none ([#1827](https://github.com/JarryShaw/PyPCAPKit/issues/1827), [#1828](https://github.com/JarryShaw/PyPCAPKit/pull/1828)). `util/conda_vendor.py --check` reports a package pinned differently under one marker ([#1852](https://github.com/JarryShaw/PyPCAPKit/issues/1852), [#1865](https://github.com/JarryShaw/PyPCAPKit/pull/1865)); the platform pins move to `conda/platform-pins.toml`, each with a review-by date, and Conda Update moves them up ([#1851](https://github.com/JarryShaw/PyPCAPKit/issues/1851), [#1885](https://github.com/JarryShaw/PyPCAPKit/pull/1885)); and the release's conda leg installs none of pcapkit's extras and smoke-tests the package before uploading it ([#1854](https://github.com/JarryShaw/PyPCAPKit/issues/1854), [#1855](https://github.com/JarryShaw/PyPCAPKit/issues/1855), [#1868](https://github.com/JarryShaw/PyPCAPKit/pull/1868)). - a wheel built with the oldest setuptools `[build-system] requires` allowed, without build isolation, left out `pcapkit/py.typed`, so type checkers read the installed package as untyped. Nothing declares the file, and setuptools ships it unasked only from 69.0.0, so that is the floor now, in `pyproject.toml` and the Conda recipe alike ([#2062](https://github.com/JarryShaw/PyPCAPKit/issues/2062), [#2064](https://github.com/JarryShaw/PyPCAPKit/pull/2064)). +- `examples/legacy_smoke`'s `test_perf` benchmarks every engine its support table lists, writing each result to `../captures/engines/.txt` or skipping the engine with its reason, where it looked each benchmark up in a table that did not cover pypcap, pcap-ct or pypcapfile, so the first of those to pass preflight raised `KeyError` and ended the run ([#2305](https://github.com/JarryShaw/PyPCAPKit/issues/2305), [#2375](https://github.com/JarryShaw/PyPCAPKit/pull/2375)). --- diff --git a/docs/source/changelog/1.5.0.rst b/docs/source/changelog/1.5.0.rst index 797938953..d00fb9e86 100644 --- a/docs/source/changelog/1.5.0.rst +++ b/docs/source/changelog/1.5.0.rst @@ -384,6 +384,12 @@ Fixed ``bier-echo [udp]``, and ``unknown [dccp]`` and ``unknown [sctp]``. Lookups still mint nothing (:issue:`860`) and an out-of-range port still raises (:issue:`2048`, :pr:`2052`). +* ``Method._missing_`` resolves an HTTP method whose registered name differs from + its own value, so ``Method('baseline-control')`` and ``Method('version-control')`` + give ``BASELINE_CONTROL`` and ``VERSION_CONTROL`` rather than minting an + unregistered member: the lookup falls back to the value map on the upper-cased + value before it mints. :meth:`.Method.get` stays case-sensitive (:issue:`2284`, + :pr:`2351`). pcapkit.corekit --------------- @@ -797,6 +803,13 @@ Fixed shared field, and a :class:`.SchemaField` with a callable length sizes its template from the resolved length rather than ``1024s``. No parse in the library reached either (:issue:`2173`, :pr:`2190`). +* :class:`.PortEnumField` range-checks the port it is handed, so a directly built + TCP, UDP or SCTP schema raises :exc:`.FieldValueError` for a port outside the + field's width, where ``srcport=70000`` was masked to ``0x1170`` and ``-1`` to + ``0xffff``. :class:`.NumberField` keeps masking, which is its documented contract; + the :class:`.NumberField` and :class:`.Schema` docstrings now say that direct + construction range-checks nothing, and name where the checks live (:issue:`2314`, + :pr:`2335`). pcapkit.dumpkit --------------- @@ -901,6 +914,20 @@ Added * ``conflict`` on the reassembly data models: absolute, inclusive ranges where two fragments claimed the same span with different bytes, previously lost silently on both the IP (:pr:`482`) and TCP (:issue:`443`, :pr:`478`) paths. +* the declarations that pair a ``__proto__`` dispatch registry with the + enumeration its codes come from, replacing the module-level tables that held the + pairing apart from the ``__proto__`` they describe. ``__proto_key__``, on each + class owning such a registry (``Link``, ``Internet``, ``Frame``, ``PCAPNG``, + ``TCP``, ``UDP`` and ``SCTP``), names that enumeration and whether a bare ``int`` + is also valid (:pr:`2392`); ``__proto_dest__``, on the + :class:`~pcapkit.const.reg.linktype.LinkType`, + :class:`~pcapkit.const.reg.ethertype.EtherType`, + :class:`~pcapkit.const.reg.transtype.TransType` and + :class:`~pcapkit.const.sctp.payload_protocol_identifier.PayloadProtocolIdentifier` + registries, names the destination each enumeration keys, which is how + ``code=`` infers one without an explicit destination. Both are rendered + attributes, and the vendor crawlers generate the latter from a new + :attr:`~pcapkit.vendor.default.Vendor.PROTO_DEST` (:pr:`2403`). Changed ~~~~~~~ @@ -1212,6 +1239,29 @@ Fixed and ``register_transtype`` refuse a ``code`` that is not an enum member with :exc:`.RegistryError` before touching a registry, where they registered it and then raised ``AttributeError``, leaving the entry behind (:issue:`2186`, :pr:`2193`). +* :func:`pcapkit.foundation.registry.protocols.register_protocol_code`, and the + ``code=`` class keyword, check every destination's key before registering any of + them. A key belonging to another enum, such as ``{Frame: TransType.ICMP}``, is + refused with :exc:`.RegistryError` where it replaced the shipped + ``LinkType.ETHERNET`` entry, and a bad target late in the list no longer leaves + the earlier ones registered. A destination keeping a registry of its own, such as + ``Loopback`` and ``IPX``, is not checked against another enum (:issue:`2268`, + :pr:`2343`). +* the number registrars ``register_tcp``, ``register_udp``, ``register_sctp`` and + ``register_apptype`` refuse a ``bool`` code with :exc:`.RegistryError`, where + ``register_tcp(True, ...)`` registered port 1, since ``bool`` is an ``int`` + subclass (:issue:`2269`, :pr:`2344`). +* IP reassembly reports a datagram that is not itself fragmented but whose BUFID is + already buffered: it submits that buffer as incomplete and then reassembles the + datagram in a buffer of its own, per :rfc:`791#section-3.2`, where the datagram + was never reported at all. On IPv6 an atomic fragment is reassembled in a buffer + of its own and submitted without flushing the datagram already being reassembled, + so that one's final fragment still completes it, per :rfc:`6946#section-4` + (:issue:`2270`, :pr:`2345`). +* the PyPCAP, pcap-ct and pypcapfile engines warn with :exc:`.AttributeWarning`, + naming the engine, that they ignore a parsing context supplied through + :class:`.Extractor`'s ``context=``, as the dpkt, scapy and pyshark engines already + did, where they dropped it silently (:issue:`2313`, :pr:`2332`). pcapkit.interface ----------------- @@ -1223,6 +1273,18 @@ Added ``same_entry``, ``restore_shipped`` and ``restore_shipped_dumper`` (:issue:`1519`, :pr:`1568`). +Fixed +~~~~~ + +* the messages the interface functions and the helpers behind them raise and warn + read in the house ``: : ... {value!r}`` form. + ``reassemble`` and ``trace`` lowercase their unsupported-protocol error and quote + the protocol; ``follow_tcp_stream`` warns ``using default engine instead``, the + wording :class:`.Extractor` already used, and quotes an engine that has no + reassembly adapter; :class:`.TraceFlowBase`'s output-format warning is lowercased; + and ``Extractor.make_name`` quotes the format it does not know, where an empty one + ended the message at the colon with nothing after it (:issue:`2303`, :pr:`2374`). + pcapkit.protocols ----------------- @@ -1497,6 +1559,16 @@ Changed :class:`~pcapkit.foundation.engines.pcapng.PCAPNG` engine has not assigned the section it opens. Code reading ``block.context`` for the section reads ``block.section_context`` (:issue:`2172`, :pr:`2205`). +* **a breaking change to** ``FTP_DATA``: it derives from ``Application`` as well as + :class:`~pcapkit.protocols.misc.raw.Raw`, so its ``__layer__`` names the + application layer where it was ``None`` -- a ``layer='application'`` extraction + stops at it, and ``extract(layer=FTP_DATA)`` resolves to that layer rather than to + ``'none'``. Raw still does the parsing and the rebuild, and ``__index__`` is the + application layer's, raising ``IntError`` where it raised + :exc:`.UnsupportedCall`. The class moves to + :mod:`pcapkit.protocols.application.ftp_data` with nothing left behind on the old + path in :mod:`pcapkit.protocols.application.ftp`, where it shipped from 1.0.0 + through 1.4.1 (:issue:`2285`, :pr:`2357`). * **a breaking change to** HIP's ``REG_FAILED`` parameter: its first octet is the Failure Type, and it carries no lifetime (:rfc:`8003#section-4.5`), so :class:`pcapkit.protocols.data.internet.hip.RegFailedParameter` replaces ``lifetime`` @@ -2762,6 +2834,178 @@ Fixed ``info`` disagreed with the octets. A block made without ``snaplen`` writes 0 there, and a ``snaplen`` outside ``0`` to ``0xFFFFFFFF`` raises :exc:`.ProtocolError` where it was truncated silently (:issue:`2278`). +* HIP parameters read and rebuilt against the wrong definition. ``TRANSACTION_ID`` + and ``OVERLAY_ID`` keep the identifier's wire width on rebuild, through a new + ``id_len`` field, where the length was sized from the integer's own ``bit_length`` + and the leading zero octets were lost (:issue:`2271`); the ``HIP_TRANSFORM`` maker + raises :exc:`.ProtocolError` for the HIPv2 parameter its own reader already + rejects (:issue:`2275`); ``LOCATOR_SET``'s maker derives the locator ``type`` from + whether ``spi`` was given, where an ``spi=`` locator was left at type 0, a + combination the locator selector itself rejects, and an explicit ``type`` that + disagrees raises naming both values (:issue:`2276`); ``REG_INFO``, + ``REG_REQUEST`` and ``REG_RESPONSE`` lifetimes decode as the + ``2**((v-64)/8)`` seconds :rfc:`8003#section-4.1` defines, with a zero octet its + own case, where the octet was read as raw seconds, and their makers take either a + duration or the raw exponent (:issue:`2281`); ``OVERLAY_TTL``'s ``ttl`` is the hop + count :rfc:`6079` defines rather than a ``timedelta`` (:issue:`2282`); and + ``CERT``'s ``cert_group`` is a plain ``int``, since :rfc:`8002#section-2` defines + no registry for it, where it was decoded through the DH Group registry + (:issue:`2283`). ``HOST_ID``'s maker prefixes its errors in the house + ``HIPv{version}: [ParamNo {code}]`` form (:issue:`2299`). The data-model and + signature changes are breaking (:pr:`2400`). +* TCP's Multipath TCP options: an unknown subtype sizes its declared ``length`` from + what the schema writes, so one built with ``data=b''`` declares a Length of 3, the + header octets it always packs, rather than 2 (:issue:`2272`); and a parsed subtype + schema no longer carries an ``option`` attribute it does not declare and no maker + mirrors -- the declared ``kind`` already holds ``Multipath_TCP``, and the octets + were identical either way (:issue:`2293`, :pr:`2396`). +* PCAP-NG's Decryption Secrets Block maker builds from a ``bytes`` ``secrets_data``, + its ``b''`` default included, by reading those octets through the same secrets + schema the parse side selects, where every secrets type raised because the readers + expect a parsed schema (:issue:`2273`, :pr:`2346`). +* HTTP/2's ``make`` defaults ``frame`` to the frame type's own constructor rather + than to a ``b''`` packed verbatim with Length 0, so ``PRIORITY``, ``RST_STREAM``, + ``PUSH_PROMISE``, ``PING``, ``GOAWAY`` and ``WINDOW_UPDATE`` build a frame that + reads back, where ``HTTP(type=..., sid=...)`` raised; an explicit ``b''`` still + packs raw, and the default is byte-identical to ``frame={}``. ``settings=None`` + builds the parameterless SETTINGS frame :rfc:`9113#section-6.5` permits rather + than raising (:issue:`2274`, :pr:`2395`). +* IPv4's Timestamp option builds from its own default: ``timestamp=None`` gives a + Timestamp-only option with ``counts`` empty slots, where the default raised + :exc:`.ProtocolError` because the maker took only a list or a dict, and any other + value is reported with its ``repr`` (:issue:`2277`, :pr:`2347`). +* a short read raises :exc:`.StructError` for every size, not only for the sizes + with a ``struct`` format code: the ``int.from_bytes`` path checks the length it + got, where it returned a smaller integer. ``quiet`` still returns the partial + value (:issue:`2279`, :pr:`2349`). +* the Mobility Header's LMAA option carries the code its address family requires: + the code defaults to that family's own and an explicit one that disagrees raises + :exc:`.ProtocolError`, where ``address='10.0.0.1'`` packed the IPv6 code anyway + (:rfc:`5949#section-6.2.2`). A captured mismatch still rebuilds byte for byte + (:issue:`2280`, :pr:`2350`). +* a base class's ``make`` keywords stop reaching a subclass whose own ``make`` + ignores them. A new ``ProtocolBase.__own_make_keywords__`` marker, read per class + like the ``__keywords__`` opt-out, keeps a base's parameters out of the allowlist + once a subclass overrides ``make``, and + :class:`~pcapkit.protocols.internet.ipv6_ext.IPv6_Ext` sets it, so + ``IPv6_Route(len=99)`` is refused where it built the same octets as + ``IPv6_Route()`` and a misspelling was steered to the dead keyword + (:issue:`2286`, :pr:`2358`). +* TCP's, UDP's and SCTP's ``make`` raise :exc:`.ProtocolError` for a ``bool`` port, + where ``UDP(srcport=True)`` packed port 1 and ``TCP(dstport=False)`` port 0 + (:issue:`2287`, :pr:`2359`). +* SCTP's Supported Address Types and Missing Mandatory Parameter makers raise + :exc:`.ProtocolError` for a parameter type that is neither a ``Parameter`` member + nor an ``int``, a ``bool`` included, where ``True`` became parameter type 1, + ``False`` raised :exc:`.EnumValueError`, and a ``str`` or a ``float`` was stored + unchecked (:issue:`2288`, :pr:`2363`). +* ``Loopback.make`` raises :exc:`.ProtocolError` for an address family that is not + an ``int``, from the same check as its range test, where ``True`` packed family 1, + ``'2'`` and ``None`` raised ``TypeError`` and ``2.0`` raised ``AttributeError`` + (:issue:`2289`, :pr:`2364`). +* FTP's ``make`` raises :exc:`.ProtocolError` for a command that is none of ``str``, + ``bytes`` and ``bytearray``, a ``bool`` included, where its fallback read + ``cmmd.value``, so an ``int``, a ``bool`` or a ``float`` raised a bare + ``AttributeError`` while any object that happened to carry a ``.value`` was + accepted unchecked. A ``bytearray`` builds as ``bytes`` (:issue:`2290`, + :pr:`2365`). +* ``_make_pack``, and ``_make_index`` packing through it, raise :exc:`.StructError` + for a size with no ``struct`` format code, as the ``struct`` path does, where a + value out of range, a non-integer or a bad size leaked ``OverflowError``, + ``AttributeError`` or ``ValueError`` (:issue:`2291`, :pr:`2367`). +* a binary read at the end of the stream raises :exc:`.StructError`, as an unpack + does, where an exhausted or short stream raised ``TypeError`` out of ``ord()`` + (:issue:`2292`, :pr:`2368`). +* IPv4's SEC option warns about the bit it observed: the field termination indicator + :rfc:`1108` puts in the final protection authority octet is reported as set, with + its value, where the message said the indicator was not set (:issue:`2295`, + :pr:`2372`). +* a maker that refuses a caller's argument names the argument and quotes the value, + rather than reusing the reader's ``invalid format`` text, which says nothing about + which argument was wrong: SCTP's chunk-flag, empty-data, hostname-termination and + reserved-length guards, the Mobility Header's data-type, option-length, + empty-data and Pad1 guards, TCP's ``MP_CAPABLE`` optional-field guard, and HIP's + and PCAP-NG's fallback branches (:issue:`2296`, :pr:`2405`), then HOPOPT's and + IPv6-Opts' Pad1 warning and HTTP/2's and PCAP-NG's ``make`` dispatchers + (:issue:`2406`, :pr:`2407`). The readers still say ``invalid format``, which is + what a parse found. +* ESP's errors read in the house ``ESP: : ... {value!r}`` form: the + messages raised by the cipher and integrity lookups, by + :class:`.esp.SecurityAssociation`'s SPI, destination, AEAD, ICV, key-splitting, + decryption and encryption checks, by the context registrar and by ``make`` all + carry the prefix and quote the offending value, where none carried an ``ESP:`` + prefix and most printed the value bare (:issue:`2297`, :pr:`2366`). +* HTTP's version, data, start-line and settings errors carry the ``HTTP:`` prefix + and the rejected value, as in ``HTTP: invalid version: 3`` and + ``HTTP/2: [Type 4] invalid settings: 42``, where the prefixes were missing or + malformed, some were capitalised with a full stop, and some left the value out + (:issue:`2298`, :pr:`2373`). +* the capture guards read in that form too. The guards that reject a file's magic + number or byte order carry the ``PCAP:`` and ``PCAP-NG:`` prefixes, and PCAP's + magic-number guard says what it read (:issue:`2300`); and PCAP-NG's option guards + drop their trailing full stop, with the block-placement guard naming the block + type through the ``BlockType`` registry rather than printing the raw number + ``_type`` holds while a block is being made (:issue:`2301`, :pr:`2398`). +* :class:`~pcapkit.protocols.internet.ipv6_ext.IPv6_Ext`'s version gate reads + ``invalid version: must be 6, not {version!r}``, in the house form and quoting the + rejected value, where its bespoke tail read ``only valid for IPv6, got version=4`` + -- naming the version it requires in prose rather than as the number it compares + against (:issue:`2302`, :pr:`2397`). +* a trial parse kept as :class:`~pcapkit.protocols.misc.raw.Raw` because it raised + no longer replays its warnings: the holding context honours its ``discard`` on an + exception exit too, and the next-layer reader discards a raising trial's warnings + under ``beholder``, so an Ethernet frame cut inside its IPv4 header stops warning + ``packet length < 0`` for a layer it does not have. Outside ``beholder`` the error + reaches the caller and the warnings are still reported, and the Raw's error names + the cut (:issue:`2308`, :pr:`2339`). +* the declarations that described a value the code never produces now match it, so + a type checker and the rendered API agree with what is stored: CALIPSO's + ``cmpt_bitmap`` is the tuple of ints its reader stores, the SMF hash-based DPD + option declares its own schema rather than its sibling's, an ``RR`` or ``SSR`` + ``route`` that is always built is not ``Optional``, ``NoPayload`` narrows + ``protochain`` to ``None``, ARP's address maker admits the ``bytearray`` its body + handles, and the PCAP-NG flags stub and an undeclared ``entry_raw`` follow. HIP's + ``OverlayTTLParameter`` and ``ViaRVSParameter`` join their modules' ``__all__`` + and the package re-exports those lists require. Parsing and rebuilding are + unchanged (:issue:`2317`, :pr:`2377`). +* HIP's ``parameters=[(code, data)]`` form rebuilds a parsed parameter through its + maker positionally, so the parameters whose maker keyword is spelled differently + from the data model's own field name -- ``suite_id`` against ``suites``, + ``group_id`` against ``group``, ``reg_type`` against ``reg_request`` and its + siblings, and the nested locator keys -- keep their values, where they fell back + to the maker's default or raised ``AttributeError``: ``HIP_TRANSFORM`` rebuilt as + ``0241000000000000``, losing its ``suite_id``. The dict form, as in + ``parameters=[(code, {'suites': [...]})]``, is unchanged (:issue:`2352`, + :pr:`2383`). +* PCAP-NG settles the byte order before a block maker runs, the way ``pack`` does -- + the section's, else the one the packet data declares, else the host's -- so an + option or a record handed over as raw ``bytes``, whose type code is read back out + of those octets, builds in any block rather than raising + ``AttributeError: 'PCAPNG' object has no attribute '_byte'``. A Section Header + Block still overrides it from its own Byte-Order Magic, and ``__packet__=None`` + degrades to no packet data rather than raising ``TypeError``, as + :meth:`.Schema.pack` already treated it (:issue:`2353`, :pr:`2369`). +* a PCAP-NG block made from raw ``bytes`` is read back off the octets ``pack`` has + just written, as a reader of the file would, rather than off the schema that + carries them, so a Decryption Secrets Block built that way parses and rebuilds + byte for byte, where the read looked for the block's fields on the schema + holding the octets and raised ``AttributeError``. Octets that are no block + raise :exc:`.StreamEOFError` or :exc:`.ProtocolError` (:issue:`2354`, + :pr:`2370`). +* a ``str`` enumeration name resolves where the namespace could not be inferred from + the argument's own default. HOPOPT's and IPv6-Opts' ``seed_type`` and ``tid_type`` + and HTTP/1's ``status`` default to ``None``, so a name such as + ``'SEEDID_16_BIT_UNSIGNED_INTEGER'`` raised ``ProtocolNotImplemented`` while the + member and ``int`` forms of the same call worked (:issue:`2356`, :pr:`2393`); and + HIP's list-argument makers, where the name is an item of a list parameter rather + than a bare argument, resolve too, with the enumerations they need imported at + runtime rather than only under ``TYPE_CHECKING`` (:issue:`2394`, :pr:`2401`). +* a PCAP-NG block made without an explicit timestamp reads its interface's + resolution from the same lookup the maker already makes for the scale, so the + Enhanced Packet Block, the obsolete Packet Block and the Interface Statistics + Block default their timestamp where they raised ``AttributeError: '_info'``, + having no parsed data model to resolve the interface through yet (:issue:`2362`, + :pr:`2382`). pcapkit.toolkit --------------- @@ -2865,6 +3109,25 @@ Fixed such as TCP behind a tunnel or an IPv6 extension header, is documented rather than warned about. The pypcapfile engine keeps its warning, once per capture, that it leaves TCP over IPv6 out (:issue:`1513`, :pr:`1535`). +* the pcap-ct, PyPCAP, pypcapfile, pyshark and scapy adapters lead the messages + below with their engine's name, read off the engine class's own + ``__engine_name__`` rather than hand-typed, so a rename under + ``pcapkit/foundation/engines/`` cannot leave an adapter spelling the old one: + the pcap-ct and PyPCAP reassembly refusals, pypcapfile's address errors, its IPv6 + :exc:`.UnsupportedCall` and its TCP-over-IPv6 warning, pyshark's boolean-field, + timestamp and missing-layer messages, and scapy's import-time warning, which + quotes the installable name, all read in the house + ``: : ... {value!r}`` form. The dpkt adapter and + the adapters' :exc:`.ModuleNotFound` raises keep the messages they had + (:issue:`2304`, :pr:`2399`). +* the IPv4 and IPv6 reassembly adapters take the first layer of that IP version, + from the outside in, that is actually a fragment -- IPv4 with MF set or a non-zero + offset, IPv6 with a Fragment header -- so a fragmented inner datagram in an + IP-in-IP tunnel is reassembled on its own header, where it was reassembled whole + under the tunnel's header, or dropped when the tunnel set DF or carried no + Fragment header of its own. With no fragmented layer they take the outermost, as + before. Applied to the default engine's PCAP and PCAP-NG adapters and to the + scapy and dpkt ones, which picked the outermost too (:issue:`2307`, :pr:`2338`). pcapkit.utilities ----------------- @@ -3071,6 +3334,11 @@ Fixed exit 1 with one line naming the missing packages and ``pypcapkit[vendor]``, where they died with a traceback, and ``--help`` and ``--version`` still work. Importing a crawler then raises ``ModuleNotFoundError`` (:issue:`1850`, :pr:`1875`). +* the ESP cipher and integrity crawlers strip the brackets off IANA's + ``[UNSPECIFIED]`` "no document" cell before testing it against the unknown + placeholder, so :mod:`pcapkit.const.esp.integrity` no longer reads + ``#: AUTH_DES_MAC [UNSPECIFIED] (DEPRECATED ...)`` for ``AUTH_DES_MAC`` and + ``AUTH_KPDK_MD5`` (:issue:`2294`, :pr:`2371`). Project infrastructure ---------------------- @@ -3613,3 +3881,8 @@ Fixed installed package as untyped. Nothing declares the file, and setuptools ships it unasked only from 69.0.0, so that is the floor now, in ``pyproject.toml`` and the Conda recipe alike (:issue:`2062`, :pr:`2064`). +* ``examples/legacy_smoke``'s ``test_perf`` benchmarks every engine its support + table lists, writing each result to ``../captures/engines/.txt`` or + skipping the engine with its reason, where it looked each benchmark up in a table + that did not cover pypcap, pcap-ct or pypcapfile, so the first of those to pass + preflight raised ``KeyError`` and ended the run (:issue:`2305`, :pr:`2375`).