From fa61a1cb8d53632bdc181ca9d9268985b5ddd4f8 Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 19:45:43 -0700 Subject: [PATCH 01/11] fix: Diagrams act as summary --- docs/native-resources-management.md | 867 +++++++++------------------- src/c2pa/c2pa.py | 129 ++--- tests/perf/README.md | 2 - tests/perf/thread_scenarios.py | 80 +-- tests/test_unit_tests.py | 4 - tests/test_unit_tests_threaded.py | 69 +-- 6 files changed, 325 insertions(+), 826 deletions(-) diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index 5240c146..a1964054 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -1,14 +1,32 @@ # Native resource management -`ManagedResource` is the internal base class the C2PA Python SDK uses to wrap native (Rust/FFI) pointers. A new wrapper subclasses it and follows these lifecycle rules. +`ManagedResource` is the internal base class the C2PA Python SDK uses to wrap native (Rust/FFI) pointers. `Reader`, `Builder`, `Signer`, `Context`, and `Settings` all subclass it. -## Ways to clean up +A `Reader`, for example, holds a pointer to memory the native library allocated. Python's garbage collector tracks the `Reader` object itself, but has no visibility into that native memory, so it can never free it on its own. `ManagedResource` closes that gap: it frees the native pointer exactly once, however the object stops being used. -Three ways to clean up: `with`, explicit `close()`, or the destructor fallback. +## Vocabulary -### Using a "with" statement +A **native pointer** is an address that says where a piece of memory lives. The C2PA SDK wraps a Rust library, the "native" library, which allocates memory Python cannot see. -Use a `with` statement: +A **handle** is the one native pointer a `ManagedResource` object holds at a time, stored in its `_handle` attribute. + +**Ownership** answers one question: who must free a given piece of native memory, exactly once. Freeing it zero times leaks memory. Freeing it twice corrupts the allocator and can crash the process. + +A pointer is **consumed** when a native call takes ownership of it, often returning a replacement pointer in its place. Once consumed, Python must never free the original. + +## Garbage collection + +Python's garbage collector mainly works by reference counting: each object counts how many references point to it, and reaching zero frees it. This works for pure Python objects, but a `Reader`'s native pointer sits outside that system entirely. The collector sees the `Reader` wrapper and tracks references to it, but has no idea the `_handle` attribute points at memory of its own, and never calls the native free function. Collect the wrapper without freeing that memory first, and it leaks. + +### `__del__` is not enough + +`__del__`, Python's finalizer hook, could in principle free the native pointer whenever an object is collected, and `ManagedResource` does use it as a fallback. But its timing is unpredictable: garbage collection itself is non-deterministic, an object caught in a reference cycle waits for a separate cycle collector to run, and during interpreter shutdown Python may collect objects in any order, so a `__del__` that reads global state can find it already torn down. On implementations that don't use reference counting, PyPy and GraalPy, `__del__` may not run until long after the last reference is gone, or not before the process exits. Every class that holds a native pointer should inherit from `ManagedResource` rather than relying on `__del__` alone. + +## Releasing + +`ManagedResource` gives every object three ways to release its native pointer: a `with` statement, an explicit `close()`, or, as a fallback only, the destructor. + +### `with` ```python with Reader("image.jpg") as reader: @@ -28,15 +46,15 @@ finally: reader.close() ``` -Calling `close()` directly is equivalent to exiting a `with` block. It is idempotent: later calls do nothing. +Calling `close()` directly is equivalent to exiting a `with` block. It is idempotent: a second call does nothing. -### Destructor fallback +### Destructor -Without `with` or `.close()`, `__del__` attempts the free when Python garbage-collects the object. Per [Why `__del__` is not reliable enough](#why-__del__-is-not-reliable-enough), its timing is unpredictable, and it may not run at all. Treat it as a safety net, not the primary mechanism. +Without `with` or `.close()`, `__del__` attempts the free when Python garbage-collects the object, for the reasons in [`__del__` is not enough](#__del__-is-not-enough). Treat it as a safety net, not the primary mechanism. -## Nesting resources +### Nesting -Multiple native resources can share one `with` statement or nest in separate blocks. Either way, Python cleans them up in reverse order: right to left, or inner to outer. +Multiple resources can share one `with` statement or nest in separate blocks. Either way, Python cleans them up in reverse order: right to left, or inner to outer. ```python with open("photo.jpg", "rb") as file, Reader("image/jpeg", file) as reader: @@ -44,461 +62,184 @@ with open("photo.jpg", "rb") as file, Reader("image/jpeg", file) as reader: # reader is closed first, then file ``` -Equivalent, if more readable nested: +The order matters because the `Reader`'s native pointer reads the file's data through a [`Stream`](#streams) wrapper: the native library calls back into that stream to read bytes. Close the file first, and those callbacks are still reachable from native code but read from a closed file, which can read freed memory. Closing the Reader first frees the native pointer while the file is still open, and only then closes the file. `with` guarantees this order: whatever is listed later, or nested deeper, is torn down first. -```python -with open("photo.jpg", "rb") as file: - with Reader("image/jpeg", file) as reader: - manifest = reader.json() -``` - -The order matters because resources depend on each other. Here the `Reader`'s native pointer reaches the file's data through a [`Stream`](#streams) wrapper, and the native library reads the file by calling back into that stream's callbacks. Close the file first, and those callbacks stay reachable from native code but reading a closed file, which can read freed memory or segfault on any later access, cleanup included. Closing the Reader first frees the native pointer while the file is still open. `with` guarantees this order: whatever is listed later, or nested deeper, is torn down first. - -## Reader lifecycle - -A `Reader` wraps a stream, or opens a file, passes it to the native library, and holds the returned pointer. Every method using the pointer, `.json()`, `.detailed_json()`, `.resource_to_stream()`, and the rest, checks state via `_ensure_valid_state()` before the FFI call. - -```mermaid -stateDiagram-v2 - direction LR - [*] --> UNINITIALIZED : __init__() - UNINITIALIZED --> ACTIVE : Reader("image.jpg") - ACTIVE --> CLOSED : close() / exit with block - CLOSED --> [*] -``` - -`ACTIVE` methods repeat freely without changing state. `.close()` on an already-closed Reader is a no-op, and any other method on a closed Reader raises `C2paError`. Closing runs `_release()` first, dropping file handles and stream wrappers, then frees the native pointer via `c2pa_free`. - -## Builder lifecycle +## Lifecycle states -A `Builder` follows the Reader pattern with one difference: **signing closes the builder**, so a `Builder` is single-use. +Every `ManagedResource` tracks one of three states: ```mermaid stateDiagram-v2 direction LR [*] --> UNINITIALIZED : __init__() - UNINITIALIZED --> ACTIVE : Builder.from_json(manifest) - ACTIVE --> CLOSED : .sign() or close() + UNINITIALIZED --> ACTIVE : _activate(handle) + UNINITIALIZED --> CLOSED : close() before activation + ACTIVE --> CLOSED : close() / __exit__ / __del__ CLOSED --> [*] - - note left of CLOSED - .sign() closes the builder - to enforce single use - end note ``` -`ACTIVE` methods like `.add_ingredient()` and `.add_action()` repeat freely. `.sign()` closes the Builder on return, whether signing succeeded or failed. An argument rejected before signing starts, a first argument that is neither a `Signer` nor a format string, raises without closing, since nothing was signed. Closing without signing frees the pointer the same way. - -The native sign call borrows the builder's pointer rather than taking it, so `Builder` frees it normally through `c2pa_free`: the close enforces single use, not memory management. - -The sign call runs inside `self._native_call()`, nesting `signer._native_call()` when the caller passes an explicit `Signer`, and `close()` runs after that block exits (see [Locking and in-flight tracking](#locking-and-in-flight-tracking) for why the order matters and what it protects against). - -## Why ManagedResource? - -`ManagedResource` manages native pointers owned by the C2PA Python SDK. It guarantees: - -- Native memory is freed exactly once: no double-free. -- Cleanup is deterministic, via context managers or explicit `close()`. -- Ownership transfers (signer to context, for instance) never free the same pointer twice. -- Cleanup never raises. Errors during it are logged instead. - -## Definitions - -A **native pointer** is an address that says where a piece of memory lives. The C2PA SDK wraps a Rust library, the "native" library, which allocates memory Python cannot see. A `Reader`, `Builder`, `Signer`, `Context`, or `Settings` object holds a native pointer to memory the native library allocated and manages. Python's garbage collector tracks the Python object, but knows nothing about the native memory behind it, so it cannot free that. - -The **native side** is the Rust library, reached through its C FFI. - -A **handle** is the single native pointer a `ManagedResource` object holds, stored in its `_handle` attribute. Each object owns one handle at a time, and the lifecycle machinery mostly tracks it: creating, swapping, freeing. - -**Ownership** answers one question: who must free a piece of native memory, exactly once. Nobody freeing it leaks memory. Two owners each freeing it corrupts the allocator and can crash the process. So exactly one side owns each pointer at any moment, and that side frees it. - -**Taking** or **transferring ownership** means that responsibility moves from one holder to another. When Python hands a pointer to the native side and it takes ownership, Python must stop trying to free it. [Ownership transfer](#ownership-transfer) covers how the SDK handles this. - -A pointer is **consumed** when a native call takes ownership of it, often returning a replacement. Once consumed, Python must not free it again. - -## Why is native resources management needed? - -### How Python's garbage collector works - -Python manages its own objects' memory through garbage collection. CPython, the standard interpreter, mainly uses reference counting: each object counts how many references point to it, and reaching zero deallocates it. A secondary cycle-detecting collector handles objects that reference each other in a loop, since their counts never reach zero on their own. - -### Why garbage collection is not enough for native memory - -This works well for pure Python objects, but native memory sits outside it entirely. The garbage collector sees the Python wrapper, a `Reader` instance say, and tracks references to it, but has no visibility into the native memory its `_handle` attribute points to. It does not know that allocation's size, cannot tell when it is no longer needed, and never calls `c2pa_free`. Collect the wrapper without calling `c2pa_free` first, and the native memory leaks. - -### Why __del__ is not reliable enough - -Python offers `__del__`, a finalizer hook that runs when an object is collected, and `ManagedResource` uses it as a fallback. But `__del__` cannot be the primary mechanism. Its timing is unpredictable, since garbage collection is itself non-deterministic, and it may not run at all during interpreter shutdown. Other implementations, like PyPy and GraalPy, do not use reference counting, making it even less predictable there. - -In CPython, `__del__` runs synchronously when the last reference disappears, predictable in simple cases like a local variable going out of scope. But an object in a reference cycle never reaches a zero count on its own. The cycle collector must find and break the cycle first, and runs periodically, not immediately, so the object can sit in memory an arbitrary time before `__del__` fires. It also gives no order when finalizing a group in a cycle, so one `__del__` may find another already partly torn down. Interpreter shutdown is worse: CPython clears module globals and may collect objects in any order, so a `__del__` reading global state, the `_lib` handle say, can fail silently because that global is already `None`. PyPy and GraalPy use tracing collectors instead, periodically walking the object graph for unreachable objects rather than counting references, so `__del__` there does not run the instant the last reference disappears. It runs whenever the collector next traces that part of the heap, seconds or minutes later, or never if the process exits first. - -Every class holding a native pointer should inherit from `ManagedResource`, which handles its lifecycle and cleanup. - -## Streams - -Bytes reach the native library through a `Stream`. - -`Stream` wraps a Python stream-like object, a file or a memory stream, so the native library can read and write it via callbacks. It does not inherit from `ManagedResource` and uses `c2pa_release_stream()` instead of `c2pa_free()` for cleanup. - -### Why is `Stream` not a `ManagedResource`? - -A `Reader` or `Builder` holds a native resource that Python code calls methods on. A `Stream` holds a native handle the native library calls back into instead, for read, seek, write, and flush. Because of those callbacks, ownership means something different here, and the native library needs its own release function to tear down that machinery. - -`Stream` tracks its own state with two flags, `_closed` and `_initialized`, rather than the lifecycle-state machinery covered later in [Lifecycle states](#lifecycle-states), but still supports the same three cleanup paths: context manager, explicit `.close()`, `__del__` fallback. - -### Callbacks re-enter Python - -A `Stream` registers four ctypes callbacks, `_read_cb`, `_seek_cb`, `_write_cb`, `_flush_cb`. Reading from a stream calls one of them, running caller-supplied Python. - -A native call driving a stream callback cannot hold a lock across the call: the callback may re-enter this API on the same thread and deadlock against a lock already taken. [Locking and in-flight tracking](#locking-and-in-flight-tracking) covers this in full. - -The callback objects must also outlive the native side's use of them, covered in [Preventing garbage collection of live references](#preventing-garbage-collection-of-live-references). - -Each callback checks `_initialized` and `_closed` before touching the underlying Python stream, returning `-1` if either is unfavorable, so a callback arriving after teardown reports an error instead of reading through dropped state. - -### `Stream` cleanup - -`Stream` holds `_close_lock`, an `RLock`, serializing the three cleanup paths: `close()`, `__del__`, and a `close()` on another thread. Without it, two of them could reach the same stream and call `c2pa_release_stream` twice on one handle. `Stream` needs its own lock: it is not a `ManagedResource` subclass, so it has no lock to inherit. [Locking and in-flight tracking](#locking-and-in-flight-tracking) covers the equivalent machinery those classes share. - -The lock is reentrant for the same reason the one in [Locking and in-flight tracking](#locking-and-in-flight-tracking) is. `close()` sets the four callback attributes to `None` inside the locked region, which can drop the last reference to an object whose finalizer runs at that same bytecode boundary. `__del__` takes the same lock, and a plain `Lock` would deadlock against itself when that finalizer belongs to the stream being closed. - -Cleanup runs in the direction of dependency: whatever can still invoke or reach the other is torn down first. Callbacks run the opposite way for a `Stream`, so its close order reverses `ManagedResource`'s: - -| | Order | Why | -| --- | --- | --- | -| `ManagedResource` | `_release()` first (dropping streams and callbacks), then `c2pa_free` | The native pointer depends on the Python-side resources, so those are torn down while the pointer is still valid. | -| `Stream` | `c2pa_release_stream` first, then drop the callbacks | The native stream invokes the callbacks. Releasing it first guarantees none can fire, and the callback objects are dropped after that. | - -`close()` performs both steps: it calls `c2pa_release_stream`, then sets `_read_cb`, `_seek_cb`, `_write_cb`, and `_flush_cb` to `None`. - -`__del__` performs only the first, calling `c2pa_release_stream` and leaving the four attributes pointing at their callbacks. Nothing leaks: `__del__` runs when the `Stream` itself is being collected, freeing those attributes with it. Explicit cleanup drops them itself instead of waiting for that. - -`Stream` does not own the Python object it wraps and never closes it: the caller that opened a file owns that file. A `Reader` that opened the file itself tracks it as `_backing_file` and closes it during its own `_release()`. - -Both `close()` and `__del__` take the foreign-process branch, marking the stream closed without calling into the native library. [Fork safety](#fork-safety) covers why. Both also check `is_foreign_process()` before acquiring `_close_lock`, the same order `_teardown()` and `_lock()` use: a forked child inherits the lock in whatever state it had at `fork()`, and the thread holding it does not exist there to release it, so a child that acquired first would wait on it forever. - -### Reference cycles in the callbacks - -Each ctypes callback closes over the `Stream` it belongs to. Captured directly, that forms a cycle: the `Stream` holds the callback, and the callback's closure holds the `Stream`. Nothing in that loop reaches a zero refcount, so cleanup would fall to the cycle collector, whose timing [Why `__del__` is not reliable enough](#why-__del__-is-not-reliable-enough) covers. The closures capture a `weakref` to the `Stream` instead, resolving it on each call, so the reference count can reach zero and cleanup stays on the deterministic path. - -## Threads, the GIL, and locks - -### How a native memory bug reports itself - -A Python bug raises an exception with a traceback. A native memory bug terminates the process, and the operating system reports it as a signal. - -SIGSEGV, segmentation violation, comes from the hardware: every process has a map of which address ranges it may touch and how, and the CPU traps on a read or write to an unmapped address, or a write to one mapped read-only. The kernel then sends SIGSEGV, that event, a segmentation fault or segfault. It fires whenever a dereferenced pointer does not point at accessible memory: a null pointer, a returned stack frame, or memory that was freed and unmapped. Freeing memory does not always unmap it. Allocators often keep the pages and reuse them, so reading through a freed pointer often succeeds anyway, returning another object's bytes and corrupting state far from the code responsible. SIGSEGV is the other outcome, when the pages happen to be gone. - -SIGABRT, abort, has a different origin: no hardware raises it. The program raises it against itself, calling `abort()` on detecting its own broken invariant. Allocators do this. A `free()` handed a pointer it never issued, or the same pointer twice, or a heap whose bookkeeping a stray write damaged, stops the process rather than continuing on a corrupt heap. - -Which signal arrives depends on the allocator. glibc prints a diagnostic and raises SIGABRT. The macOS allocator traps instead, giving SIGTRAP. A use-after-free that reaches unmapped pages gives SIGSEGV on both. - -Both terminate the process from inside native code. No exception is raised. No `finally` block runs. No traceback is printed. - -### A close arriving mid-call - -A race here is two threads, one object, one native pointer, and operations that take several steps: - -1. Thread A calls `reader.json()` (for instance), which validates the handle and enters the native call. -2. While that call is running, thread B calls `reader.close()` (for instance, close on same object), which frees the native pointer. -3. Thread A's native code, still running, reads through the pointer it was given. - -Step 3 reads freed memory: SIGSEGV if those pages are gone, or another object's bytes if the allocator kept and reissued that address. Both happen inside native code. - -Both threads reach that same `Reader` because a Python object has no owning thread: it belongs to whoever holds a reference to it, and nothing about `reader = Reader(...)`, or passing `reader` into a closure, hands ownership from one thread to another. The GIL stops the two threads from running Python bytecode at the same instant, but not from touching the same object, and it is handed away entirely for the duration of a native call, exactly the window thread B's `close()` lands in. So the race is not a rare coincidence: any two threads sharing a reference to the same `Reader`, `Builder`, `Signer`, or `Context` can reach it. - -Without a guard, that race plays out exactly as the three steps above: thread A's validity check passes, thread B frees the handle, and nothing re-checks it before thread A reads through the now-freed pointer. With the guard this doc describes, thread B's `close()` still runs immediately from its own point of view, but only marks the object `CLOSED` and records that a free is owed. The actual `c2pa_free` is deferred until thread A's call returns and `_inflight` drops back to zero. The object becomes unusable to any new caller the moment `close()` is called, but the memory it points to outlives the call still reading it. - -### What the GIL can guarantee - -CPython compiles source into bytecode, small instructions the interpreter executes one at a time. A single line of Python becomes several: `self._ensure_valid_state()` compiles to four, load `self`, load the attribute, call, discard the result. - -CPython has a Global Interpreter Lock, a single interpreter-wide lock: only one thread executes bytecode at a time, so one instruction cannot interleave with another, and built-in containers do not corrupt structurally under concurrent access. - -Anything longer than one instruction gets no such guarantee. The interpreter can switch threads between any two instructions, so a single line of Python can be interrupted partway through. - -### Why the GIL is not enough here - -ctypes releases the GIL for a foreign function call's duration, so other Python threads run in parallel with the native code. Every native call in this layer opens that window, and a `close()` can arrive inside it. [A close arriving mid-call](#a-close-arriving-mid-call) walks through that race. - -The check-then-use sequences here span many instructions: `_ensure_valid_state()` validates the handle, then a later instruction loads `self._handle` and passes it to native. The interpreter can switch threads in between, so a passing check does not promise the handle is still live at the point of use. `_op_lock` makes those pairs one unit. - -`_native_call()` closes the window by making the free wait for the call to finish: - -```mermaid -flowchart TD - subgraph guarded [Guarded by _native_call] - direction TB - C1["Thread A: enters guard
_inflight becomes 1"] --> C2["Thread A: native call runs"] - D1["Thread B: close()"] --> D2{"_inflight nonzero?"} - D2 -->|yes| D3["Record _pending_teardown,
mark CLOSED, free nothing"] - C2 --> C3["Thread A: leaves guard
_inflight becomes 0"] - D3 -.->|"free deferred to
whoever leaves last"| C3 - C3 --> C4["Deferred free runs
pointer no longer in use"] - end -``` - -### Why races are observed - -A race on a native pointer produces a use-after-free or a double-free: a crash or silent memory corruption that can surface arbitrarily far from its cause. [Double-free risk mitigations](#double-free-risk-mitigations) covers the three specific hazards and the mechanism for each. - -> [!NOTE] -> Free-threaded CPython builds remove the GIL entirely, so code that was accidentally relying on it for atomicity loses that protection anyway. - -## Locking and in-flight tracking - -Each `ManagedResource` holds a reentrant lock, `_op_lock`, and a counter, `_inflight`, together serializing teardown against concurrent use from other threads. - -A plain lock, Python's `threading.Lock`, can be acquired once: a second `acquire()` from the same thread blocks forever, waiting on a lock that thread itself holds. A reentrant lock, `threading.RLock`, tracks which thread holds it and how many times. The owning thread can acquire it again without blocking, releasing only once it has released it the same number of times it acquired it. A different thread still blocks until the owner releases fully. - -`_op_lock` is an `RLock`, not a plain `Lock`, for two reasons. First, a finalizer can run at any bytecode boundary, including inside a method that already holds the lock on that same thread, so `__del__` calling back into locked code must not deadlock against itself. Second, a consuming call tears the handle down from inside the locked region it already holds, so `_teardown()` needs to acquire the same lock again rather than block on itself. `_lock()` returns it, except in a forked child, where it raises `C2paError` immediately instead of blocking: the thread that might hold the lock at fork time does not exist in the child to release it, so waiting on it would hang forever (see [Fork safety](#fork-safety)). - -The lock is never held across a native call that drives a stream callback. Construction, `resource_to_stream`, the Builder stream methods, and signing all release the GIL and re-enter caller-supplied Python, which may itself call into this API on another thread (see [Callbacks re-enter Python](#callbacks-re-enter-python)), and holding `_op_lock` there would deadlock against that reentry. Those calls go through `_native_call()` instead: a context manager that increments `_inflight` under the lock, yields to run the native call unlocked, then decrements `_inflight` on the way out. A `_teardown()` running while a call is in flight records the requested `free_handle` in `_pending_teardown` and marks the resource `CLOSED` right away, so no other caller can start using it, but defers the actual free. The last `_native_call()` to exit picks up `_pending_teardown` and runs `_teardown()` for real. - -A recorded `_pending_teardown` only ever moves from `True` to `False`, never back: if one caller records a teardown that frees and another does not, the one that does not wins. A pointer the native side already took is never freed on the way out. - -Cleanup idempotency keys on a `_released` flag, not the `CLOSED` state, since the deferred path marks the resource `CLOSED` when it records the teardown while still owing the release that runs later. - -`Context.__init__` does not wrap the signer hand-off in `signer._native_call()`: the consuming call marks the signer `CLOSED` under its own lock before calling native, which stops a `signer.close()` on another thread from freeing the handle mid-transfer (see [Borrowing versus consuming](#borrowing-versus-consuming)). The Builder's `close()` after signing runs outside its own `_native_call()` block, so a teardown deferred during the call still executes once it returns. - -### Lock ordering - -Two threads acquiring the same pair of locks in opposite orders deadlock: each holds what the other waits for. A single order of acquisition avoids that. - -An operation holds guards by role, from outermost to innermost: - -1. A method-specific lock, where a method serializes itself against other calls to itself. -2. The guard of the object the method is called on. -3. The guard of any object it borrows for the duration of the call. - -Rules that follow from the order: - -- A borrowed object's guard is always taken inside the operating object's guard, never the reverse, so two concurrent operations sharing a borrowed object acquire in one direction. -- A lock held across a native call must be one no callback path acquires. -- `Stream._close_lock` is a leaf. Nothing is acquired while holding it, so it cannot join a cycle. - -No code path holds two resources' `_op_lock` at once. `_native_call()` counts a call as in flight instead of holding the lock across it, which keeps that true, and `test_no_nested_op_locks` checks it over the Reader read path by intercepting `_lock()` and recording any acquisition made while another is held. - -### Borrowing versus consuming - -Deferring a teardown protects a consuming call against a racing `close()`, but not a borrowing call against a racing consume: that is a different risk with a different mitigation. +- `UNINITIALIZED`: the Python object exists but has no native pointer yet. This is transient, lasting only for the duration of construction. +- `ACTIVE`: the native pointer is valid, and the object can be used. +- `CLOSED`: the native pointer has been freed, or ownership of it has moved elsewhere. Any further use raises `C2paError`. -A borrowing call passes the handle to native and gets it back unchanged. A consuming call hands ownership over, and the native side frees the pointer during the call. A borrowing call validates the pointer once on entry, then holds it for the whole operation without consulting the pointer registry again, so a consume starting midway through a borrow frees memory the borrowing call is still reading, and the usual `-1` rejection never happens since validation already succeeded. +This is one-way: once `CLOSED`, an object never becomes `ACTIVE` again. A construction that fails before activation can also close straight from `UNINITIALIZED`, since there is nothing to free, only a state to record. -`ManagedResource` therefore refuses the consume rather than letting it start: `_begin_consume()` runs `_ensure_not_borrowed()`, rejecting the call when `_inflight` is nonzero, then marks the resource `CLOSED` before releasing `_op_lock`. +## Close during a call -The check catches a borrow already in flight, and the `CLOSED` mark catches one arriving afterward: `_native_call()` calls `_ensure_valid_state()` under the same lock, so a borrow starting later is refused instead of reaching a pointer about to be freed. The lock cannot simply be held across the native call, because those calls run caller-supplied stream callbacks that re-enter this API. +A native call takes several steps in sequence: check the object is usable, hand the pointer to native code, let that code run. Two threads sharing one object can interleave those steps: -The mark is provisional: `_abort_consume()` restores the previous state when the native call turns out not to have taken the handle, so the retained branch of the [ownership-taken triage](#why-an-ownership-taken-failure-does-not-free) still hands back a usable object. +1. Thread A calls `reader.json()`. It checks the Reader is usable, then enters the native call. +2. While that call is still running, thread B calls `reader.close()`, which frees the native pointer. +3. Thread A's native code, still running, reads through the pointer it was given, now freed. -`_raise_consume_failure()` performs that restore, on the pre-consume branch only, holding the reservation until the branch is known. `_read_native_error()` is itself a native call and releases the GIL, so a resource restored to `ACTIVE` before the error is classified is visible as usable to another thread, even while the native side may already own its handle. +Step 3 reads freed memory. Depending on what the allocator has done with that memory since, this either crashes the process or returns another object's bytes, corrupting state far from the code responsible (see [Crashes](#crashes)). -`_consume_and_swap()` is excluded: it requires the resource to stay `ACTIVE`, remaining usable with its replacement pointer, so there is no `CLOSED` mark to make and no check. Its callers, `Reader.with_fragment` and `Builder.with_archive`, pass streams whose callbacks re-enter this API, so they hold their own `_native_call()`. `Reader.with_fragment()` also serializes itself with a lock of its own, described in [`Reader.with_fragment()`](#readerwith_fragment). +A Python object has no owning thread: it belongs to whoever holds a reference to it, and nothing about creating a `Reader` or passing it into a closure hands ownership from one thread to another. Any two threads sharing a reference to the same `Reader`, `Builder`, `Signer`, or `Context` can hit this race, so [Lifecycle states](#lifecycle-states) alone is not the whole model. A resource also needs a way to say "a call is using me right now, don't free me out from under it," which is what in-flight tracking adds next. -### Context lifetime during a context-sign +## In-flight calls -A `Context` built with a signer keeps that signer alive after consuming it: `Context.__init__` copies the signer's ctypes callback into `_signer_callback_cb`, since the consuming call closes the `Signer` object while the native side still needs the trampoline to call. `Context._release()` drops that reference. +Alongside its lifecycle state, every `ManagedResource` counts calls currently running against it. This is separate from the `ACTIVE`/`CLOSED` state: a resource can be `ACTIVE` and idle, or `ACTIVE` with one or more calls in flight, and those are different things a caller needs to know. -`c2pa_builder_sign_context` calls back into that trampoline for the sign's duration, so the Context must stay alive across it, just as a borrowed `Signer` does. `Builder._sign_internal` wraps the call in `self._context._native_call()`, and a `close()` arriving on another thread then takes the deferred branch: it records the teardown, marks the Context `CLOSED`, and leaves `_release()` for whichever caller leaves the guard last. The callback stays pinned and the handle stays valid until the sign finishes. +A call in flight is one of two kinds: -A sign already in flight is never cut short: closing a Context mid-sign takes effect once the call returns. Without the guard, `_release()` would free the ctypes trampoline while native is calling through it, and the process takes SIGSEGV rather than raising. +- **Shared**: several threads can run this kind of call on the same object at once. Reading a manifest with `.json()` is shared: nothing stops two threads from reading the same data at the same time. +- **Mutating**: only one thread may run this kind of call at a time, and no shared call may start while it runs. Signing with `.sign()` is mutating: it changes what the object holds, so a concurrent read could see a half-updated result or a pointer being replaced out from under it. -A sign cannot start once the Context is closed: it raises `C2paError` instead. A released Context has already dropped the callback, and the native side would sign without ever invoking it, so an error is the only way that would be visible to the caller. +A `close()` arriving while any call, shared or mutating, is in flight does not free the pointer immediately. It marks the resource so no new caller can start using it, and defers the actual free until every in-flight call has returned. This is what closes the race from the previous section: thread B's `close()` still takes effect right away from its own point of view, but the memory thread A is still reading stays valid until thread A's call returns. -That guard is what makes the error visible at all. A ctypes callback is an ordinary Python object with ordinary reference counting, and nothing on the native side holds a reference to it, so keeping it alive as long as native might still call it is Python's job, here falling to the `Context`. Without the check that raises `C2paError`, a callback already dropped by the time a sign started would leave native signing without ever calling back into it, reporting success anyway. The output file would come out looking signed, with a valid manifest structure, while the signer that was supposed to produce the signature never ran: no exception, no error message, just a signed-looking file the supplied callback never actually signed. +## The full picture -## Class hierarchy +Lifecycle state and in-flight calls happen at the same time. A resource is always in one lifecycle state, `UNINITIALIZED`, `ACTIVE`, or `CLOSED`. Independently, while it is `ACTIVE`, zero or more calls may be in flight on it right now, and if any of them is mutating, no other call may start. Put together, this is the full picture every guard in `ManagedResource` checks before letting a call through: ```mermaid -classDiagram - class ManagedResource { - <> - } - - class ContextProvider { - <> +stateDiagram-v2 + [*] --> UNINITIALIZED + UNINITIALIZED --> ACTIVE: _activate(handle) + UNINITIALIZED --> CLOSED: close() before activation + + state ACTIVE { + [*] --> Idle + Idle --> SharedBorrow: a shared call starts + SharedBorrow --> Idle: it returns + Idle --> Mutating: a mutating call starts + Mutating --> Idle: it returns + Mutating --> [*]: a consume succeeds } - ManagedResource <|-- Settings - ManagedResource <|-- Context - ManagedResource <|-- Reader - ManagedResource <|-- Builder - ManagedResource <|-- Signer - - ContextProvider <|-- Context + ACTIVE --> CLOSED: close() / __del__ + CLOSED --> [*] ``` -Notes: - -- `Context` inherits from both `ManagedResource` and `ContextProvider`, which Python's multiple inheritance allows. -- `Settings` inherits from `ManagedResource` only. -- `ContextProvider` is an ABC requiring two properties, `is_valid` and `execution_context`. `is_valid` lives on `ManagedResource`, so `Context` satisfies that part of the contract without duplicating it. - -> [!NOTE] -> **How `is_valid` resolves across both parents for Context** -> -> Python's MRO, Method Resolution Order, is the order Python searches parent classes for a method or property. For `Context(ManagedResource, ContextProvider)`, that order is `Context`, `ManagedResource`, `ContextProvider`, `ABC`, `object`. Accessing `context.is_valid` walks it left to right and finds `ManagedResource.is_valid` first. `ContextProvider.is_valid` is abstract, declaring the requirement with no implementation, so the concrete version on `ManagedResource` provides the behavior and satisfies the ABC contract both. -> -> The MRO is computed by C3 linearization, which enforces two rules: children appear before their parents, and the class definition's left-to-right order is preserved. For `class Context(ManagedResource, ContextProvider)`: -> -> 1. `Context`: the class itself always comes first. -> 2. `ManagedResource`: first listed parent, nothing else requires it to appear later. -> 3. `ContextProvider`: second listed parent, must come after `ManagedResource` to preserve declaration order. -> 4. `ABC`: parent of `ContextProvider`, must come after its child. -> 5. `object`: root of everything (all objects), always last. -> -> Putting `ManagedResource` first in the declaration matters: lookup finds the concrete `is_valid` immediately, rather than hitting the abstract declaration on `ContextProvider` first. +The `Mutating --> [*]` exit inside `ACTIVE` is a consuming call: one that hands the pointer to native and gets a replacement or a closed resource back, covered in [Consuming](#consuming). -## Python frees only what Python owns +| State | `is_valid` | What a caller sees | +| --- | --- | --- | +| `UNINITIALIZED` | False | `C2paError`: "not properly initialized" | +| `ACTIVE`, idle | True | normal operation | +| `ACTIVE`, shared call(s) in flight | True | normal operation; more shared calls may join | +| `ACTIVE`, a mutating call in flight | **False** | `C2paError`: "running a mutating operation" | +| `CLOSED` | False | `C2paError`: "is closed" | -The C FFI is consistent about one thing that shapes this layer: some calls consume the pointer passed to them and hand back a replacement, since the native side may free and reallocate it. A pointer that went into a consuming call must never be freed by Python afterward, since its address may already belong to a different object, address space being finite and reused. +`is_valid` answers one question: would a call be accepted right now? It is `ACTIVE`, holding a handle, and no mutating call in flight. It is a lock-free snapshot, so a passing check does not keep the handle alive. Only an actual guarded call does that. -Python owns and frees two kinds of things: the **single current native handle** for each object, and **Python-side resources it created itself**, like stream wrappers, pinned callbacks, and caches. It swaps that one tracked handle to whatever a consuming call returns, and on the success path never frees the value taken. It also carries bookkeeping it never frees, like lifecycle state, the owning process ID, and a borrowed reference to a caller-supplied `Context`. It does not manage native reallocation itself: it swaps handles, and on ambiguous failure paths reads the native error tags to decide who still owns the pointer rather than assuming. +A `Reader`'s read methods, `.json()`, `.detailed_json()`, `.resource_to_stream()`, are shared. A `Builder`'s `.sign()` is mutating, and ends by consuming the Builder: signing closes it, so a `Builder` is single-use. `Reader.with_fragment()` is also mutating, and also consumes: it replaces the Reader's handle with a new one rather than closing the object (see [Consuming](#consuming)). -Therefore, the managed resources have the following principles: +## Crashes -- Each `ManagedResource` holds exactly one `_handle`. `_consume_and_swap()` replaces it with the pointer a consuming call returned and does not free the old value, since the native side took it (see [Consume-and-swap](#consume-and-swap)). -- `_teardown(free_handle=False)`, `_consume_no_replacement()`, and `_consume_into()` all close or advance the object without calling `c2pa_free`, because ownership moved to the native side. -- Only a few sites free a live handle, and most free a pointer this layer still provably owns: normal teardown (`_teardown(free_handle=True)`), the create-then-validate path that frees a freshly created pointer if activation fails, and the constructors that free a raw pointer when wrapping it raises since no instance took ownership (`Signer.from_info`, `Signer.from_callback`, `Builder.from_archive`). The exception is `_release_handle()`, a *guarded* free used only when ownership is unknown: a consuming call failed without setting an error, or a Python exception was raised before the native side reported anything. If the native side already took the pointer, its address is no longer in the registry, and `c2pa_free` is a `-1` no-op that touches no memory. No path frees a pointer known to have been consumed and reallocated (see [Why an ownership-taken failure does not free](#why-an-ownership-taken-failure-does-not-free)). -- `_release()` drops stream wrappers, callbacks, and caches before the native pointer is freed (see [Subclass-specific cleanup with `_release()`](#subclass-specific-cleanup)). +A Python bug raises an exception with a traceback. A native memory bug terminates the process, and the operating system reports it as a signal, since the failure happens outside anything Python's exception machinery watches. -### Double-free risk mitigations +SIGSEGV, segmentation fault, comes from the hardware: the CPU traps on a read or write to memory the process is not allowed to touch, and the kernel delivers SIGSEGV. This fires when a pointer no longer points at accessible memory, for instance freed memory the allocator has unmapped. Freeing memory does not always unmap it, though. Allocators often keep the pages and reuse them for a later allocation, so reading through a freed pointer frequently succeeds anyway, silently returning another object's bytes and corrupting state far from the code responsible. SIGSEGV is what happens on the unlucky path, when the pages are actually gone. -Freeing the same native pointer twice corrupts the allocator's bookkeeping. An allocator that detects it stops the process (see [How a native memory bug reports itself](#how-a-native-memory-bug-reports-itself)). One that does not lets the damage surface later, somewhere unrelated to the code responsible. Three situations lead here: a single flow misreading who owns a pointer, a forked child freeing its parent's memory, and two threads racing. +SIGABRT, abort, comes from software: a program calls `abort()` on itself after detecting a broken invariant. Allocators do this when `free()` is handed a pointer it never issued, the same pointer twice, or a heap whose bookkeeping a stray write has damaged. Which signal a given bug produces depends on the allocator: glibc raises SIGABRT with a diagnostic, the macOS allocator gives SIGTRAP, and a use-after-free that reaches unmapped pages gives SIGSEGV on both. In every case, the process terminates from inside native code. No exception, no `finally` block, no traceback. -Each risk and its mechanism: +## Locking -| Hazard | Covered by | How | -| --- | --- | --- | -| Freeing a pointer a consuming call already took (single flow) | `_consume_and_swap` / `_teardown(free_handle=False)` triage | The consumed pointer is abandoned, never freed. The retained-vs-consumed decision reads the native error tag (`UntrackedPointer:` / `WrongPointerType:` / `NullParameter:` / `InvalidBufferSize:` mean not taken). | -| A forked child freeing a pointer its parent owns | PID stamp (`record_owner_pid` / `is_foreign_process`) | Cleanup in a process that did not allocate the pointer nulls the handle and marks `CLOSED` without freeing (see [Fork safety](#fork-safety)). | -| Two **threads** racing a `close()` against an in-flight native call on the same object, where the allocator recycles a just-freed address | `_op_lock` / `_native_call()` / `_pending_teardown` | A close arriving while a native call is in flight is recorded rather than applied. The last caller to leave `_native_call()` performs the deferred free (see [Locking and in-flight tracking](#locking-and-in-flight-tracking)). | +Each `ManagedResource` holds a reentrant lock, `_op_lock`, and the in-flight counters from [The full picture](#the-full-picture). Together they enforce that rule: a mutating call excludes every other call, and a `close()` arriving mid-call is deferred rather than applied immediately. -The PID stamp is fork-only: it compares process IDs, and two threads in the same process always match on that PID. Sharing one `ManagedResource` instance across threads still needs locks, since nothing here protects two threads racing on genuinely distinct objects that share an allocator. +`_op_lock` is a `threading.RLock`, reentrant, rather than a plain `Lock`, for two reasons. A finalizer can run at any point, including inside a method that already holds the lock on that thread, so `__del__` calling back into locked code must not deadlock against itself. And a consuming call tears the handle down from inside a region it already holds the lock in, so it needs to reacquire rather than block. -## Guarantees provided by ManagedResource +The lock is never held across a native call that drives a stream callback (construction, `resource_to_stream`, signing, and the rest), since that callback can call back into this API on the same thread, and holding the lock there would deadlock against that reentry. Those calls increment an in-flight counter under the lock, release the lock, run the native call, then decrement the counter, which is the mechanism the [state diagram](#the-full-picture) describes as "a call in flight." -`ManagedResource` provides these guarantees. A subclass must maintain them: +This is why the interpreter's Global Interpreter Lock, the GIL, does not make this safe on its own. CPython executes one bytecode instruction at a time under the GIL, so simple operations cannot corrupt a built-in container. But a foreign function call through ctypes releases the GIL for its duration, so another thread runs freely while native code is running, and a `close()` can land inside that exact window. `_op_lock` and the in-flight counters are what close it, not the GIL. -| Guarantee | Description | -| --- | --- | -| **Pointer freed exactly once** | Each native pointer is passed to `c2pa_free` at most once. No leak (zero frees) and no double-free. | -| **Cleanup is idempotent** | Calling `close()`, or exiting a `with` block, multiple times does nothing after the first successful cleanup. | -| **Cleanup never raises (ordinary errors)** | The cleanup path catches and logs `Exception`, never re-raising it. `_release()` runs inside `_safe_release()`, which logs and swallows. The `c2pa_free` call has its own handler, and `_cleanup_resources()` wraps both. The original exception from the `with` block, if any, is never masked. **Asynchronous interrupts are the exception.** The handlers catch `Exception`, not the `BaseException` signals the interpreter raises to unwind a process, a cancellation or an exit in progress. Those propagate through cleanup untouched, and the remaining free may not run. Such a signal means the process is being torn down, address space and native allocations included, so catching it would only delay a shutdown the caller asked for to finish a free that is about to become irrelevant. | -| **State transitions are one-way** | Lifecycle moves only from UNINITIALIZED to ACTIVE to CLOSED. A closed resource cannot be reactivated. | -| **Transitions go through helper methods** | Subclasses call `_activate()`, `_consume_and_swap()` or `_teardown()` and never assign `_handle` or `_lifecycle_state` directly. `_activate()` and `_consume_and_swap()` validate before mutating, so an object cannot end up active with a null handle. | -| **Ownership transfer is safe** | When a pointer is transferred elsewhere (e.g. via `_teardown(free_handle=False)`), the object stops managing it and does not call `c2pa_free` on it. | -| **Public methods validate lifecycle state** | Every public method that uses the handle calls `_ensure_valid_state()` first. Closed or invalid state yields `C2paError` instead of undefined behavior or a crash. The exceptions touch no handle: `is_valid` reports state rather than requiring it, and the `get_supported_mime_types` classmethods query the library itself. | - -## Preventing garbage collection of live references +### Lock ordering -When a Python object passes a callback or pointer to the native library, that reference must stay alive as long as the native side might use it. Python's garbage collector has no way to know that native code still holds one. +Two threads acquiring the same pair of locks in opposite orders can deadlock, each waiting on what the other holds. The SDK avoids this with one fixed order, from outermost to innermost: a method-specific lock (where a method serializes itself against other calls to itself), then the lock of the object a method is called on, then the lock of any object it borrows for the call. A borrowed object's lock is always taken inside the operating object's lock, never the reverse, and a lock held across a native call must be one no callback path acquires. -The SDK stores these references as instance attributes on the owning object instead. `Stream`, for example, stores its four callback objects, `_read_cb`, `_seek_cb`, `_write_cb`, `_flush_cb`, as instance attributes, so as long as the `Stream` is alive its callbacks have a nonzero reference count and stay uncollected (see [Streams](#streams) for how those callbacks avoid a reference cycle with the `Stream` itself). Similarly, a `Signer` consumed by a `Context` has its `_callback_cb` copied to the Context's `_signer_callback_cb` attribute, so the callback survives even after the Signer object closes. +### Borrowing vs consuming -During cleanup, `_release()` sets these attributes to `None`, dropping the reference count so they can be collected. It runs first in the cleanup sequence, before `c2pa_free` frees the native pointer, so subclass-specific resources like open file handles and stream wrappers are torn down before the pointer they depend on is freed. +A shared call, a **borrow**, passes the handle to native and gets it back unchanged. A mutating call that ends by consuming the handle hands ownership to native, which frees the original pointer during the call. A borrow validates the pointer once on entry and then holds it for the whole call without checking again, so a consume starting midway through a borrow would free memory the borrow is still reading. -This ordering applies to `ManagedResource`. `Stream` releases in the opposite order, and [`Stream` cleanup](#stream-cleanup) explains why each side is correct for the direction its callbacks run. +`ManagedResource` prevents this by refusing to start a consume while a borrow is in flight, and by reserving the handle as a mutating call for the whole duration of the consume, the same reservation any other mutating call makes. Every entry point checks the in-flight counters under `_op_lock` before it starts, so a call that would otherwise race a consume is refused instead with "running a mutating operation," and the resource's lifecycle state never changes until the consume is fully classified as a success or a failure. `Reader.with_fragment()` also serializes itself against other calls to itself with a lock of its own, covered in [`Reader.with_fragment()`](#readerwith_fragment). -## How native memory is freed +## Double frees -The native Rust library exposes a single C FFI function, `c2pa_free`, that deallocates memory it previously allocated. `ManagedResource` wraps it in a static method: +Freeing the same native pointer twice corrupts the allocator's bookkeeping. A detecting allocator stops the process. An allocator that misses it lets the damage surface later, somewhere unrelated to the code responsible. Three distinct hazards lead here, each with its own guard: -```python -@staticmethod -def _free_native_ptr(ptr): - return _lib.c2pa_free(ptr) -``` +| Hazard | Guarded by | +| --- | --- | +| Freeing a pointer a consuming call already took | The consumed pointer is abandoned rather than freed; the native error message says whether ownership actually moved (see [Consuming](#consuming)). | +| A forked child process freeing a pointer its parent owns | A process-ID stamp on every object; cleanup in a process that did not allocate the pointer marks it closed without freeing (see [Fork safety](#fork-safety)). | +| Two threads racing a `close()` against an in-flight call on the same object | `_op_lock` and the in-flight counters (see [Locking](#locking)): a `close()` mid-call is deferred, not applied. | -All native pointers are freed through this single path, regardless of which constructor created them: `c2pa_reader_from_stream`, `c2pa_builder_from_json`, `c2pa_signer_from_info`, and the rest. No explicit `ctypes.cast` is needed, since `c2pa_free`'s declared argtype is `c_void_p`, and ctypes converts any pointer instance on the way in. Casting explicitly with `ctypes.cast(ptr, c_void_p)` does the same conversion, but leaves a reference cycle behind on every call, adding load to the garbage collector. +Sharing one `ManagedResource` instance across threads still needs these guards. Nothing here protects two threads racing on distinct objects that happen to share an allocator, since that is the allocator's own concurrency guarantee, not this layer's. -It returns `c2pa_free`'s status code: `0` when the pointer was really freed, `-1` when the native registry rejected an already-consumed or untracked address. That `-1` is expected on the guarded-free paths, and the native library handles it gracefully too. +## Consuming -`ManagedResource` guarantees `c2pa_free` is called exactly once per pointer: not zero times, a leak, and not twice, a double-free. +Consuming a handle is a mutating call that hands the pointer to native, which takes ownership and either returns a replacement pointer or frees the original. Python must never free a pointer it handed to a consuming call, whichever way the call ends, since the address may already belong to a different object by the time it returns. -## Lifecycle states +There are two shapes a consuming call takes. -Each `ManagedResource` tracks its state with a `LifecycleState` enum: +**Consume-and-swap** replaces the object's internal state without discarding the Python-side wrapper. `Reader.with_fragment()` does this, feeding a new BMFF fragment into an existing Reader so the native library can rebuild its internal representation from prior fragments plus the new one, since a fresh `Reader` would lose that accumulated state. `Builder.with_archive()` does the same, loading an archive into an existing Builder while keeping its context and settings. ```mermaid stateDiagram-v2 - direction LR - [*] --> UNINITIALIZED : __init__() - UNINITIALIZED --> ACTIVE : _activate(handle) - UNINITIALIZED --> CLOSED : close() before activation - ACTIVE --> ACTIVE : _consume_and_swap(new_handle) - ACTIVE --> CLOSED : close() / __exit__ / __del__ / _teardown() -``` - -- `UNINITIALIZED`: the Python object exists but has no native pointer yet, a transient state during construction. -- `ACTIVE`: the native pointer is valid, and the object can be used. -- `CLOSED`: the native pointer has been freed or ownership transferred. Any further use raises `C2paError`. - -`CLOSED` is a one-way state: once closed, an object cannot be reactivated. It is normally reached from `ACTIVE`, but a construction failing before `_activate()` closes straight from `UNINITIALIZED` on `close()` or `__del__`, since there is nothing to free, only a mark to make. - -Each transition has one method that performs it, and subclasses must go through them rather than assigning `_handle` or `_lifecycle_state` directly: - -| Method | Transition | What it enforces | -| --- | --- | --- | -| `_activate(handle)` | UNINITIALIZED to ACTIVE | Rejects a null handle and refuses to run on an already-activated resource. A rejected activation leaves the object exactly as it was. | -| `_consume_and_swap(new_handle)` | ACTIVE to ACTIVE | Requires the resource to already be active and the replacement to be non-null. Used when an FFI call consumed the old handle and returned a new one. | -| `_teardown(free_handle=False)` | ACTIVE to CLOSED | Drops the handle without freeing it, for when ownership passed to the native side (e.g. `Signer` into `Context`). Runs `_release()` first, so subclass cleanup still happens. Unlike the other two it enforces no precondition on the current state: it closes whatever it is given. | -| `_release_handle()` | ACTIVE to CLOSED | Frees the handle, guarded, via `_teardown(free_handle=True)`, and closes the object, the same post-state as the consumed teardown. A resource already non-ACTIVE takes the other branch instead, clearing the handle without freeing it. That is why the reserved consume paths call `_teardown()` directly. | - -Because activation is the only way in, no code path can leave an object ACTIVE while holding a null handle. - -Two terms recur throughout this document. An **owned free** calls `c2pa_free` on a pointer this layer still provably holds, as the normal `close()` / `__del__` path and the create-then-validate failure path both do. A **guarded free** is the same call made when ownership is uncertain, what `_release_handle()` does: the native pointer registry tolerates being asked to free an address it no longer tracks, returning `-1` instead of crashing, so it never double-frees a pointer already taken. That tolerance makes a guarded free safe to *issue*, but not free of consequence under concurrency: on a branch where the value is already known to be consumed, the layer skips the free rather than relying on the `-1`, since a stale free can race a recycled address (see [Why an ownership-taken failure does not free](#why-an-ownership-taken-failure-does-not-free)). - -`_teardown(free_handle)` performs the ACTIVE to CLOSED transition, and the boolean decides the only thing varying between its two exit paths: whether the native pointer is freed. Both run `_release()`, set `CLOSED`, and null the handle. - -| `free_handle` | When | What it does with the pointer | -| --- | --- | --- | -| `True` | Either the pointer is still provably ours, normal `close()` or `__del__`, an owned free, or ownership is unknown after a failure (`_release_handle()`), a guarded free. | Calls `c2pa_free`. On the owned paths the pointer is really freed. On the unknown-ownership path the registry returns `-1` without touching memory if the native side already took it. | -| `False` | The native side already took ownership: a consuming FFI call swallowed the pointer, or it passed to another object. | Frees nothing. The new owner does. A `c2pa_free` here would double-free, or hit the guarded `-1` no-op that overwrites the error and risks racing a recycled address. | - -Every public method that uses the handle calls `_ensure_valid_state()` before doing any work. It raises `C2paError` unless the resource is ACTIVE with a non-null handle. - -## Error handling during cleanup + state "ACTIVE (ptr A)" as A + state "ACTIVE (ptr B)" as B -Cleanup must not raise an *ordinary* exception: a failure during it, the native library crashing on free say, must not mask the original exception that caused the `with` block to exit. `ManagedResource` enforces this: + A --> B : consuming call swaps ptr A for ptr B + note right of B + Same Python object, + new native pointer + end note +``` -- `close()` delegates to `_cleanup_resources()`, which wraps the whole sequence in a try/except catching and silencing `Exception`. It does no teardown itself: after the foreign-process and already-closed checks it calls `_teardown(free_handle=True)` to do the rest. -- `_release()` is never called directly during cleanup. It runs inside `_safe_release()`, which logs any `Exception` with a traceback and returns normally, so a subclass whose `_release()` raises an ordinary error cannot stop the native pointer from being freed afterward. -- A failed free of the native pointer is logged through `logging`, not re-raised. -- `_teardown()` sets `CLOSED` before running `_release()` or freeing anything, so a cleanup failing halfway still leaves the object marked closed, preventing a second attempt from doing further damage. It then moves the pointer into a local and nulls `_handle` before calling `c2pa_free`, so no other caller can read a handle about to be freed. -- Cleanup is idempotent: `close()` on an already-closed object returns immediately. +On success the object stays `ACTIVE`: the lifecycle state never changes, only the pointer underneath it, and callers keep using the same object. -These handlers catch `Exception`, not `BaseException`. The interpreter's own unwinding signals, a cancellation or an exit in progress, are `BaseException`, so they pass through cleanup untouched and the remaining free may not run. That is intentional: the signal means the process is going away, address space and native allocations reclaimed on exit, so holding cleanup open to finish a free about to become irrelevant would only delay the shutdown the caller asked for. +**Consume-and-close** takes the pointer and leaves the Python object nothing to wrap. Signing a `Builder`, or handing a `Signer` to a `Context`, both end this way: the object goes `CLOSED`, but without freeing the pointer, since native still owns it. -All three cleanup entry points converge on the same method. Exception handling sits at three different levels inside it: +A consuming FFI call can fail two different ways that return the identical value to Python, a null pointer or a non-zero status: it can reject the pointer before taking ownership, or take ownership and then drop the value on a later failure. Which one happened decides whether Python still owns the pointer. ```mermaid flowchart TD - E["close() / __exit__ / __del__"] --> CR["_cleanup_resources()"] - CR --> FP{"foreign process?"} - FP -->|yes| N["null the handle, set CLOSED,
do not free"] --> DONE([return]) - FP -->|no| ST{"already CLOSED?"} - ST -->|yes| DONE - ST -->|no| TD["_teardown(free_handle=True)"] - TD --> SET["mark _released, set CLOSED"] - SET --> REL["_safe_release()
logs and swallows"] - REL --> NULL["take the pointer into a local,
set _handle = None"] - NULL --> H{"pointer was set?"} - H -->|no| DONE - H -->|yes| FREE["_free_native_ptr()
logs on failure"] --> DONE + CALL["FFI call(handle)"] --> V{"validate arguments,
then the handle"} + V -->|invalid| R["reject: handle NOT taken"] --> F1["returns a failure value
(null, or non-zero status)"] + V -->|valid| TAKE["take ownership of handle"] + TAKE --> WORK{"execute function logic"} + WORK -->|fails| DROP["native drops the value itself"] --> F2["returns a failure value
(null, or non-zero status)"] + WORK -->|succeeds| OK["returns replacement / 0 / new pointer"] + + F1 -.same value.- AMB(["Python must read the native error
to tell these apart"]) + F2 -.same value.- AMB ``` -The `foreign process` branch is explained under [Fork safety](#fork-safety). +The native error message tells the two apart. A specific set of rejection tags means the handle was never taken, so Python keeps and later frees it normally. Any other error means native took it and dropped it, so Python's cleanup runs without freeing anything. If no error was set at all, ownership is unknown, and Python falls back to a **guarded free**: the native pointer registry safely rejects a free of an address it no longer tracks, returning `-1` rather than crashing, so this is harmless whether or not native still holds it. + +The one case a guarded free is not used defensively is once ownership is already known to have moved: there, the free is skipped rather than issued and left for the registry to reject. A freed address can eventually be reused by an unrelated allocation, and an unnecessary free landing after that reuse would destroy the wrong object. Skipping a free that provides no value avoids that window entirely. -## Ownership transfer +Three helpers share this triage, differing only in what a successful call returns: -Some operations transfer a native pointer from one object to another. When this happens, the original object must stop managing it, so it is not freed twice. +| Helper | On success | Result | +| --- | --- | --- | +| `_consume_and_swap()` | a replacement pointer | installs the replacement; resource stays `ACTIVE` | +| `_consume_no_replacement()` | a status code | `CLOSED`, pointer not freed | +| `_consume_into()` | a *different* object's pointer | `CLOSED`; the new pointer is handed to the caller to own | -`_teardown(free_handle=False)` handles this: it runs `_release()`, then sets `_handle = None` and `_lifecycle_state = CLOSED` without freeing the pointer. +`_consume_no_replacement()` is how a `Signer` is fed to a `Context`, and `_consume_into()` is how that same `Context` returns its newly built native pointer. -In the SDK this happens in one place: passing a `Signer` to a `Context`. The Context runs a short-lived native context builder, feeds the signer into it, builds the context, and activates the result. The builder itself is wrapped in `_NativeBuilder`, a small `ManagedResource`, so every failure inside the `with` block frees it through `close()` unless a consuming call already took it, with no raw pointer held across the calls and no bespoke error handler. +### Signer to context -The signer transfer goes through `_consume_no_replacement()`, which routes any failure to the shared triage (see [Why an ownership-taken failure does not free](#why-an-ownership-taken-failure-does-not-free)), which decides per error whether the signer was actually consumed: `set_signer` does not take ownership unconditionally. +The transfer of a `Signer` into a `Context` shows the whole protocol in one place: the reservation that protects the handle during the call, and the triage that decides ownership afterward. ```mermaid sequenceDiagram @@ -510,266 +251,209 @@ sequenceDiagram C->>X: Context(settings, signer) X->>B: with _NativeBuilder() (owns the builder, close() frees it on any failure) - X->>S: _ensure_valid_state() - X->>X: copy signer._callback_cb to _signer_callback_cb - Note right of X: Pin the callback first:
the Signer is about to be consumed + X->>X: copy signer's callback to the Context + Note right of X: Copy it before the transfer:
a successful consume drops the Signer's own reference X->>S: _consume_no_replacement(set_signer) - S->>S: _begin_consume(): under _op_lock,
refuse if borrowed, then mark CLOSED - Note right of S: The CLOSED mark is the protection:
a close() on another thread finds it
already closed and frees nothing + S->>S: reserve the handle as a mutating call + Note right of S: The reservation is the protection:
a close() on another thread is deferred,
and other calls are refused S->>N: c2pa_context_builder_set_signer(builder_ptr, handle) alt status 0 (success) - S->>S: _teardown(free_handle=False) - Note right of S: Consumed: native took the signer + S->>S: close, pointer not freed (native took it) else non-zero status - S->>S: _raise_consume_failure() reads the native error - Note right of S: The error is read before any free,
so a free's own error cannot overwrite it.
The reservation is held until the branch is known - alt error carries a pre-consume tag - S->>S: _abort_consume(): restore the previous state - Note right of S: Rejected before ownership moved:
Signer stays ACTIVE, typed error raised - else any other error - S->>S: _teardown(free_handle=False) - Note right of S: Native took it, then failed and
dropped the value itself: free nothing - else error slot empty - S->>S: _teardown(free_handle=True) guarded free - Note right of S: Ownership unknown: a real free if still
ours, a -1 no-op if native took it + S->>S: read the native error + alt handle was never taken + S->>S: raise, handle retained and still usable + else native took it, then failed + S->>S: close, pointer not freed + else no error set + S->>S: guarded free (real free if still ours,
no-op if native took it) end end - X->>B: _consume_into(build) + X->>B: build the context from the consumed builder B->>N: c2pa_context_builder_build(builder_ptr) - N-->>X: context_ptr (builder consumed) - X->>X: _activate(context_ptr) (outside the with) + N-->>X: context_ptr + X->>X: activate the new Context ``` -Details in that sequence that are easy to get wrong: - -- The callback is copied to the Context *before* the transfer, since a successful consume runs `_release()`, dropping the Signer's reference to it. A Context that copied it afterward would point at a callback nothing keeps alive. -- The transfer is not wrapped in `signer._native_call()`. It is protected by the `CLOSED` mark `_begin_consume()` makes under `_op_lock` before the native call starts: a racing `signer.close()` finds the resource already closed and frees nothing, and a later borrow is refused for the same reason. That check also refuses the consume outright when another thread is already borrowing the handle to sign with (see [Borrowing versus consuming](#borrowing-versus-consuming)). -- `set_signer` does not always take the pointer. A pre-consume rejection, one of `_PRE_CONSUME_ERROR_TAGS`, leaves the Signer `ACTIVE` and retained, so the triage must read the native error before deciding to close it. Treating every failure as "consumed" would close a signer the native side never took. -- A `ctypes.ArgumentError` from `set_signer` is re-raised untouched by `_invoke_consume`. Marshalling failed, the native function never ran, and the Signer still owns its handle. Only calls that reached native go through the consumed/retained triage. -- The builder is never held as a raw local across the signer and build calls. `_NativeBuilder`'s `with` block owns it. A settings error, a retained-signer error, a build rejection, or an async interrupt all free it through `close()`, and a successful build consumes it, so `close()` is then a no-op. - -### Adopting a handle the SDK already owns +A few details in that sequence matter beyond what the diagram shows. The transfer is not wrapped in a shared-call reservation; it uses the mutating reservation described in [Borrowing vs consuming](#borrowing-vs-consuming), which is what defers a racing `signer.close()` until the transfer is classified. `set_signer` does not always take the pointer, so the triage has to read the native error before deciding whether the Signer closed. And the temporary native builder used to construct the Context is itself a small `ManagedResource`, held only inside a `with` block, so any failure along the way frees it through the same `close()` path rather than a bespoke handler. -Ownership can also arrive from the other direction: a native call returns a pointer that needs a Python wrapper around it. `_wrap_native_handle()`, the classmethod for that, builds an instance with `object.__new__`, runs `ManagedResource.__init__` on it to set the lifecycle fields and stamp the owning process ID, runs `_init_attrs()` for subclass attribute defaults, and calls `_activate()` with the handle. +### Adopting a handle -It skips `__init__` on purpose, since `__init__` would try to create a *new* native resource. That is why attribute defaults belong in `_init_attrs()`, the only initialization step this path runs. +Ownership can also arrive from the other direction: a native call returns a pointer that needs a Python wrapper around it, with no `__init__` call, since `__init__` would try to create a new native resource rather than wrap an existing one. `_wrap_native_handle()` handles this: it builds a bare instance, sets its lifecycle bookkeeping, runs `_init_attrs()` for subclass defaults, and activates the handle. Ownership transfers only once that call returns; if it raises, no wrapper exists yet, and the caller still owns the pointer and must free it itself. -Ownership transfers only if the call returns. If `_wrap_native_handle()` raises, no wrapper exists to free the pointer, and the caller still owns it and must free it. +`Reader._init_from_context` and `Builder._init_from_context` both do something that looks backward: they create a native object and activate it immediately, before making the consuming call that will actually feed it data. This is deliberate. A consuming call needs an already-active resource to read the handle from and swap the result into, and activating first puts the intermediate pointer under normal cleanup right away: whichever way the consuming call goes, `close()` and `__del__` free it correctly. Holding the raw pointer in a local variable instead would leave every failure path to decide for itself whether to free it. -## Consume-and-swap +## Guarantees -`_teardown(free_handle=False)` closes an object permanently. A different pattern is needed when the native library must replace an object's internal state without discarding the Python-side object. Fragmented media needs this: `Reader.with_fragment()` feeds a new BMFF fragment, used in DASH/HLS streaming, into an existing Reader, and the native library rebuilds its internal representation for the new data by consuming the old pointer and returning a new one, since a fresh `Reader` would not have the accumulated state from prior fragments. +`is_valid`, defined in [The full picture](#the-full-picture), is a lock-free snapshot: it does not itself keep the handle alive, only a guarded call does that. `Context` implements the abstract `ContextProvider.is_valid` by inheriting the concrete one from `ManagedResource`, which Python's method resolution order finds first as long as `ManagedResource` is listed before `ContextProvider` in the class definition (`class Context(ManagedResource, ContextProvider)`). Listing them the other way around would leave the abstract declaration in front and raise `TypeError` at class definition time. -`Builder.with_archive()` follows the same pattern, loading an archive into an existing Builder while preserving its context and settings. +Every subclass gets these guarantees from `ManagedResource`, and must not break them: -In both cases the FFI call consumes the current pointer and returns a replacement: - -```mermaid -stateDiagram-v2 - state "ACTIVE (ptr A)" as A - state "ACTIVE (ptr B)" as B - - A --> B : C FFI call consumes ptr A, returns ptr B - note right of B - Same Python object, - new native pointer - end note -``` - -On success the object stays `ACTIVE`, since it is still valid: a live native pointer, working public methods, and callers may keep using it, reading the updated manifest or feeding in another fragment. The lifecycle state does not change, since nothing has closed from `ManagedResource`'s perspective. Only the underlying native pointer has been swapped, unlike a consumed teardown (`_teardown(free_handle=False)`), where the object transitions to `CLOSED` and becomes unusable. `ManagedResource` must not free the old pointer here, since the native library already consumed it as part of the FFI call. The failure path differs, and the triage in [`_consume_and_swap()`](#_consume_and_swap) covers it. - -### `Reader.with_fragment()` - -One `with_fragment()` call does two things: - -1. The FFI call consumes the Reader's current handle and returns a replacement, which `_consume_and_swap()` stores. -2. The Reader updates its own Python-side fields, the `Stream` wrappers it owns and the manifest caches. Both still describe the consumed handle. - -`_fragment_streams` holds the `Stream` wrapper for the current fragment. Each call replaces that list rather than appending to it, closing the previous wrapper immediately, since the native reader never reads a superseded fragment back, and each open wrapper pins a native stream, its callbacks, and the caller's buffer. - -#### Serializing `with_fragment()` against itself - -Those two steps have to run as one unit: two threads interleaving them can close a `Stream` the native reader is still reading through, or leave the Reader holding wrappers and caches for a handle another thread has already replaced. - -`Reader._fragment_lock` covers both steps. It is an `RLock`, and `with_fragment()` is the only method that takes it. - -The guard spans the native call, which drives caller-supplied stream callbacks, so `with_fragment()` takes it with `acquire(blocking=False)` and releases it in a `finally`. A second thread finding it held is refused with `C2paError` rather than parked behind a native call waiting on a callback to return: a callback that starts a thread of its own and waits for it would otherwise deadlock, the new thread waiting for the guard while the call holding the guard waits for the callback. - -A refusal leaves the Reader untouched: no stream is built, no handle consumed, so the call succeeds once the other thread returns. - -Reentrancy applies to the owning thread only. A callback that calls `with_fragment()` synchronously passes the guard and reaches the native call, which rejects the handle it already consumed. +| Guarantee | What it means | +| --- | --- | +| Freed exactly once | Every native pointer reaches `c2pa_free` at most once: no leak, no double-free. | +| Cleanup is idempotent | Calling `close()`, or exiting a `with` block, more than once does nothing after the first time. | +| Cleanup never raises | An ordinary exception during cleanup is caught and logged, never re-raised, so it cannot mask an exception the `with` block itself raised. An interpreter shutdown signal is the one exception to this: it is allowed to interrupt cleanup, since the process is already going away. | +| State transitions are one-way | Lifecycle only ever moves `UNINITIALIZED` to `ACTIVE` to `CLOSED`. Nothing reactivates a closed resource. | +| Public methods check state first | Every method that uses the handle validates it before the native call, raising `C2paError` on anything but a usable `ACTIVE` resource rather than risking undefined behavior. | -The two locks nest in a [fixed order](#lock-ordering): `_fragment_lock` outside, then `_native_call()` and `_op_lock` inside it. +## Keeping references alive -The fork check runs before `_fragment_lock` is taken, for the same reason `_teardown()` checks first: a forked child cannot wait on a lock no surviving thread will release, so it reports the same error `_lock()` reports rather than hanging (see [Fork safety](#fork-safety)). +When a Python object passes a callback or a pointer to the native library, that reference must stay alive for as long as native code might still use it, and the garbage collector has no way to know that on its own: it only sees Python references. -#### Cache invalidation +The SDK keeps these references as plain instance attributes on the owning object. A `Stream` stores its four callback objects this way, so they stay referenced for as long as the `Stream` itself is alive (see [Reference cycles](#reference-cycles) for how those callbacks avoid keeping the `Stream` alive in return). A `Signer` consumed by a `Context` has its callback copied to an attribute on the `Context`, so the callback survives the `Signer` object closing. -A `Reader` caches the manifest data. The caches describe the handle they were read from, so a successful swap invalidates them, and both that invalidation and the cache reads in `json()` run under `_op_lock`, so a reader never observes a half-updated cache. +`_release()` sets these attributes to `None` during cleanup, letting them be collected, and it runs before the native pointer is freed, so anything the pointer might still depend on, an open file, a stream wrapper, is torn down first. This is `ManagedResource`'s ordering; `Stream` releases in the reverse order for reasons covered in [`Stream` cleanup](#stream-cleanup). -Replacing the handle and clearing the caches are two separate steps, and `_op_lock` is not held between them: +## Freeing memory -1. `_consume_and_swap()` replaces the handle. This runs inside `_native_call()`, which counts the call as in flight rather than holding `_op_lock`. -2. `with_fragment()` acquires `_op_lock` and clears the caches. +The native library exposes one C function, `c2pa_free`, that deallocates memory it previously allocated. Every native pointer, whatever kind of object created it, is freed through this one path: -Between the two, the Reader has the new handle and the old manifest. A `json()` on another thread acquiring `_op_lock` in that gap finds the cache populated and returns it without consulting the handle at all: the manifest it serves belongs to the fragment just replaced. +```python +@staticmethod +def _free_native_ptr(ptr): + return _lib.c2pa_free(ptr) +``` -`_fragment_lock` does not prevent this. It serializes `with_fragment()` against other calls to `with_fragment()`, and readers never take it, so callers sharing a `Reader` across threads must serialize `with_fragment()` against their own reads themselves. +It returns `0` when the pointer was really freed, and `-1` when the registry rejected an already-consumed or untracked address, the same `-1` a guarded free relies on. `ManagedResource` guarantees this is called exactly once per pointer. -### `_consume_and_swap()` +## Cleanup errors -Every call of this shape goes through one helper, which takes the FFI call as a callable and handles the outcomes: +Cleanup must not let an ordinary exception mask the exception that caused a `with` block to exit in the first place. `ManagedResource` enforces this at every level: -```python -# Reader.with_fragment() internally does: -self._consume_and_swap( - lambda handle: _lib.c2pa_reader_with_fragment( - handle, format_arg, main_obj._stream, frag_obj._stream, - ), - Reader._ERROR_MESSAGES['fragment_error']) -``` +- `close()` delegates to `_cleanup_resources()`, which wraps the whole sequence in a try/except that catches and logs `Exception` rather than re-raising it. +- `_release()` runs inside a wrapper that logs any exception it raises and continues, so a subclass's `_release()` failing cannot stop the native pointer from being freed afterward. +- A failed free of the native pointer is logged, not re-raised. +- The object is marked `CLOSED` before `_release()` runs or anything is freed, so a cleanup that fails partway still leaves the object closed, and a second attempt does not repeat the damage. +- `close()` on an already-closed object returns immediately. -The call is passed as a lambda, since the helper supplies the handle and, on success, replaces it via `_consume_and_swap()`. +These handlers catch `Exception`, not `BaseException`. The interpreter's own unwinding signals, a cancellation or a shutdown in progress, are `BaseException`, so they pass through untouched and the remaining free may not run. This is deliberate: such a signal means the process is already going away, address space and native allocations included, so holding cleanup open to finish a free that is about to become irrelevant would only delay the shutdown the caller asked for. -The helper exists because a failed return can be ambiguous. The native functions run in phases: validate the **borrowed pointer**, passed in without transferring ownership unless explicitly taken over, then take ownership, then do the work. A failure in the first phase and one after the second come back to Python as the same value, a null pointer or a non-zero status, but leave ownership in opposite places. +All three cleanup entry points converge on one method: ```mermaid flowchart TD - CALL["FFI call(handle)"] --> V{"validate arguments,
then the borrowed handle"} - V -->|invalid| R["reject: handle NOT taken
sets one of _PRE_CONSUME_ERROR_TAGS"] --> F1["returns a failure value
(null, or non-zero status)"] - V -->|valid| TAKE["take ownership of handle"] - TAKE --> WORK{"execute function logic"} - WORK -->|fails| DROP["native drops the value itself
sets some other error"] --> F2["returns a failure value
(null, or non-zero status)"] - WORK -->|succeeds| OK["returns replacement / 0 / new pointer"] - - F1 -.failure value returned to Python.- AMB(["needs to consult error to know failure mode from Python"]) - F2 -.failure value returned to Python.- AMB + E["close() / __exit__ / __del__"] --> CR["_cleanup_resources()"] + CR --> FP{"foreign process?"} + FP -->|yes| N["null the handle, mark CLOSED,
do not free"] --> DONE([return]) + FP -->|no| ST{"already CLOSED?"} + ST -->|yes| DONE + ST -->|no| TD["run teardown"] + TD --> SET["mark released, CLOSED"] + SET --> REL["release subclass resources
(logs, never raises)"] + REL --> NULL["take the pointer into a local,
null the handle"] + NULL --> H{"pointer was set?"} + H -->|no| DONE + H -->|yes| FREE["free the native pointer
(logs on failure)"] --> DONE ``` -The two failure paths are indistinguishable from the return value alone. Only the native error message set alongside them tells the phases apart: - -| Native error | Who owns the handle | What the helper does | -| --- | --- | --- | -| One of `_PRE_CONSUME_ERROR_TAGS` | Still ours: rejected before ownership moved | Handle kept, resource stays `ACTIVE`, typed error raised. Normal cleanup frees it later. | -| Any other error | Taken, then the operation failed | `_teardown(free_handle=False)`: the native side already dropped the value, so nothing is freed here. Resource goes `CLOSED`, error typed from the native message. | -| No error at all | Unknown | Guarded free, the caller's message is raised with `"Unknown error"` filled in. A reserved consume frees through `_teardown(free_handle=True)`, because `_release_handle()` treats a reserved resource as one it does not own. | +The "foreign process" branch is explained in [Fork safety](#fork-safety), next. -This triage relies on the native error still being readable, and being the last one encountered, after the call returns: reading an error copies the message out and frees the copy, but leaves the native slot set until the next error overwrites it. - -Three consume helpers share this triage. They differ only in what the FFI call returns on success: +## Fork safety -| Helper | Success return | Success action | -| --- | --- | --- | -| `_consume_and_swap()` | a replacement pointer | installs the replacement, resource stays `ACTIVE` | -| `_consume_no_replacement()` | a status code (`0` = ok) | `_teardown(free_handle=False)`, resource `CLOSED` | -| `_consume_into()` | a *different* object's pointer | `_teardown(free_handle=False)`, the pointer returned for the caller to own | +`fork()` copies the calling process, including every Python object holding a native pointer, but the underlying native allocation is not duplicated: there is still only one, and the parent owns it. -`_consume_no_replacement()` is how a `Signer` is fed to a `Context`, since `set_signer` returns a status code, and `_consume_into()` is how that same `Context` build returns the new context pointer. A failure in any of the three goes through the same native-error triage, so a pre-consume rejection retains the handle rather than assuming it was taken. +If a forked child cleaned up its copy of that object normally, two things would go wrong. It would free a pointer the parent is still using, a double-free. And more subtly, `fork()` only carries over the thread that called it, so if any other thread held a native lock at fork time, that lock stays held forever in the child, and calling into native code to free anything can then block on it forever. -#### Why an ownership-taken failure does not free +So the SDK never frees native memory in a process that did not allocate it. Every object is stamped with its creating process's ID at construction, and cleanup compares that stamp against the current process before doing anything: -A consuming FFI call can fail: it may reject the borrowed pointer before taking it, or take ownership first and drop the value itself on a later failure. +```mermaid +sequenceDiagram + participant P as Parent process + participant O as Reader object + participant F as Forked child -The native error message indicates which happened. A rejection carrying one of the `_PRE_CONSUME_ERROR_TAGS` means the handle was never taken and is retained. Any other error message means the native side may have taken ownership and already dropped the value. Preparing the call's own arguments can also fail in Python before the native function runs, encoding a bad value or a ctypes marshalling error other than `ArgumentError`, and that outcome is handled separately. + P->>O: __init__ stamps the owning process ID + P->>F: fork() + Note over O,F: Child inherits a copy of the object.
One native allocation, two Python copies. -The two settled branches each take the exact action their ownership implies. A pre-consume rejection, one of the `_PRE_CONSUME_ERROR_TAGS`, means the handle is still the caller's, retained and freed later by normal cleanup. Any other native error means the value is already gone, so `_teardown(free_handle=False)` runs the Python-side cleanup without freeing anything. + F->>F: child's copy is cleaned up + F->>F: process ID does not match + Note right of F: Do not free: the parent owns it,
and a native lock may be held
by a thread that did not survive the fork + F->>F: null the handle, mark CLOSED -A stale free looks like a harmless `-1` no-op, so always issuing the guarded free even where the value is known to be gone seems appealing. It is only harmless while the freed address stays unclaimed. The native registry rejects an address it no longer tracks, but once another thread allocates a fresh tracked object at that recycled address, the registry tracks it again. A stale free aimed at the old value would then find a live entry and destroy a different thread's object. That needs a second thread inside its own FFI call, an allocator handing back the exact address just freed, and that reuse landing in the narrow window between the native drop and this free. Unlikely, but not unreachable under concurrent use, and the failure would be a silent cross-thread corruption rather than a clean error. The free is not needed on this branch in the first place, so where the value is known to be consumed, it is skipped rather than issued and left for the registry to reject. The native error slot also holds whatever it last held until the next error overwrites it. An unneeded free would set an untracked-pointer error there that a later caller could mistake for the failure it actually asked about, so skipping the free keeps the slot free for the next real error. + P->>O: close() + P->>P: frees the native pointer normally +``` -`_release_handle()`, a guarded free, is reserved for the two branches where ownership is uncertain: a Python exception raised before native reports anything, and a failure leaving the error slot empty, which no defined native failure is expected to produce. In both, it is a real free when the handle is still ours, and a `-1` no-op when the native side already took it. +The child's copy is not just skipped, it is nulled and marked `CLOSED`, so nothing in the child can go on to use or free it later. This never touches the parent's copy, which stays valid. -The retained-vs-consumed decision comes entirely from the native pointer registry and its thread-local error slot, not a Python-side lock. `_op_lock` guards concurrent teardown against a call still in flight (see [Locking and in-flight tracking](#locking-and-in-flight-tracking)), but plays no part in reading which rejection prefix the native side set: a different hazard from [Fork safety](#fork-safety), which concerns a forked child process, not a thread within one. +The memory a child skips freeing is not lost for good: a child that calls `exec()` replaces its whole address space, and a child that exits has its memory reclaimed by the operating system. Even a long-lived forked worker retains at most the objects it inherited at fork time, a bounded amount rather than a growing leak, since anything it allocates itself carries its own process ID and is freed normally. -A consuming C FFI function first removes the pointer from its registry, then reconstructs the owned value from it: `untrack_or_return!` runs ahead of `Box::from_raw` in `c2pa_c_ffi`. If the address is unknown or the wrong type, the untrack step fails before ownership is taken and sets an error whose prefix, one of the `_PRE_CONSUME_ERROR_TAGS`, identifies it as a pre-consume rejection. Once the value has been reconstructed, a later failure simply drops it, the same as any owned value going out of scope. The Python side stays defensive rather than assuming exact behavior: it retains the handle when it recognizes one of those rejection prefixes, and falls back to the guarded free where the outcome is unclear. A native side behaving differently would degrade in one of two bounded ways. A pointer still held natively but treated as consumed by Python would leak, since nothing would free it. A pointer already released natively but freed anyway by Python would just get `-1` back from the registry, untouched. +## Class hierarchy -The error slot has a failure mode needing no second thread at all. There is a stretch of time on whichever thread just made a native call, between that call returning and its error message being read out of the slot, and CPython can run a finalizer at any bytecode boundary, including inside that stretch. Say a different, unrelated object becomes unreachable at exactly that moment and its `__del__` fires, freeing its own native pointer. If that free lands while the first call's message is still in the slot, it overwrites the slot with its own complaint, something like `UntrackedPointer: 0x...`, and by the time the first call reads the slot, the real message is gone and the wrong one decides the error raised. The fix marks that stretch as a section: a free that would otherwise run inside it is queued instead, running only once the section closes and the message has already been read. No two threads need to race for this: one finalizer firing at the wrong bytecode boundary, on the same thread, is enough. +```mermaid +classDiagram + class ManagedResource { + <> + } -### Adopting the handle before giving it away + class ContextProvider { + <> + } -`Reader._init_from_context` and `Builder._init_from_context` both create a native object, immediately activate it, and only then make the consuming call. `_create_and_activate()` handles the create-then-activate half: it calls the FFI constructor, validates the result with `_check_ffi_operation_result`, and `_activate()`s it, freeing the pointer if either step fails so a rejected creation leaks nothing. Reduced to its shape: + ManagedResource <|-- Settings + ManagedResource <|-- Context + ManagedResource <|-- Reader + ManagedResource <|-- Builder + ManagedResource <|-- Signer -```python -with context._native_call(): - self._create_and_activate( - lambda: _lib.c2pa_reader_from_context(context.execution_context), - Reader._ERROR_MESSAGES['reader_error']) - -self._consume_and_swap( - lambda handle: _lib.c2pa_reader_with_stream( - handle, format_arg, self._own_stream._stream, - ), - Reader._ERROR_MESSAGES['reader_error']) + ContextProvider <|-- Context ``` -Activating a handle about to be handed to the native library looks backward, for two reasons. `_consume_and_swap` needs an active resource to read the handle from and swap the result into, and it puts the intermediate pointer under normal cleanup before anything can go wrong: whichever way the consuming call goes, `close()` and `__del__` free the pointer if the native side did not take it. Holding the pointer in a local variable instead would leave every failure path to decide for itself whether to free it. +`Context` inherits from both `ManagedResource` and `ContextProvider`. Python's multiple inheritance allows this. `ContextProvider` is an abstract base class requiring `is_valid` and `execution_context`; `Context` satisfies `is_valid` by inheriting the concrete version from `ManagedResource`, as long as `ManagedResource` is listed first in the class definition, `class Context(ManagedResource, ContextProvider)`. Listed the other way, Python's method resolution order would find the abstract declaration first and refuse to let `Context` be instantiated at all. -## Subclass-specific cleanup +## Streams -Each subclass can override `_release()` to clean up its own resources before the native pointer is freed. The base implementation does nothing. +Bytes reach the native library through a `Stream`, which wraps a Python file or memory stream so the native library can read and write it via callbacks. It does not inherit from `ManagedResource`, and uses a separate release function instead of `_free_native_ptr`. -Examples from the codebase: +### Not a `ManagedResource` -| Class | What `_release()` cleans up | -| --- | --- | -| Reader | Drops the manifest caches, closes owned file handles and stream wrappers, and drops the reference to the Context | -| Builder | Drops the reference to the Context | -| Context | Drops the reference to the signer callback. `has_signer` is left as it was, since it records how the Context was configured and stays readable after close. | -| Signer | Drops the reference to the signing callback | -| Settings | (no override, nothing extra to clean up) | +A `Reader` or `Builder` holds a resource that Python code calls methods on. A `Stream` holds a resource the native library calls back into instead, for read, seek, write, and flush. Ownership means something different for a resource that receives calls rather than makes them, so `Stream` gets its own release path instead of the shared one. -The cleanup order matters: `_release()` runs first, closing streams and dropping callbacks, then `c2pa_free` frees the native pointer, so the native library never accesses a Python object that no longer exists. +`Stream` tracks its state with two flags, `_closed` and `_initialized`, rather than the `LifecycleState` machinery from [Lifecycle states](#lifecycle-states), but still supports the same three cleanup paths: context manager, explicit `.close()`, `__del__` fallback. -### Dropping a Context reference +### Reentrant callbacks -`Reader` and `Builder` both keep a `_context` attribute that is written once and never read. It is not dead code: it keeps the Context alive while the native handle depends on it. Without it, `Reader("image/jpeg", stream, context=Context())` would let the Context become collectable as soon as the constructor returned, even though the reader is still using it. +A `Stream` registers four ctypes callbacks. Reading from the stream runs one of them, running caller-supplied Python, and the same reentrancy hazard applies as anywhere else in this doc: a native call driving one of these callbacks cannot hold a lock across the call, since the callback may call back into this API on the same thread. -Clearing it in `_release()` is the other half of that. A closed Reader has no further use for the Context, and holding the reference would keep alive an object nothing can reach through the Reader's public API. +Each callback checks the stream's own state before touching the underlying Python object, and reports an error rather than reading through it if the stream has already been torn down. -Dropping the reference before the native pointer is freed does not depend on the Python object staying alive, either. When a reader is created from a shared context, the native side takes its own reference-counted handle on that context, an `Arc` clone in the Rust library, so it keeps the context alive independently of whether Python still points at it. That native behavior is not visible from this repo, which ships a prebuilt binary, but is the contract this code is written against. +### `Stream` cleanup -## Fork safety +`Stream` holds its own reentrant lock, `_close_lock`, serializing `close()`, `__del__`, and a `close()` from another thread against each other, for the same reason `_op_lock` is reentrant: a callback attribute set to `None` inside the locked region can drop the last reference to an object whose own finalizer then needs the same lock. -`fork()` copies the calling process, including every Python object holding a native pointer. The child gets its own copy of the wrapper object, but there is still only one native allocation, and the parent owns it. +Cleanup runs in the direction of dependency: whatever can still invoke or reach the other side is torn down first. This reverses the order `ManagedResource` uses, because a `Stream`'s callbacks are invoked by native code rather than the other way around: `Stream` releases the native handle first, guaranteeing no callback can fire again, and only then drops the callback objects. `ManagedResource` instead runs subclass cleanup first, since its native pointer depends on those resources staying valid until then. -If the child's copy were cleaned up normally, two things would go wrong: a double-free, the child freeing a pointer the parent is still using, and more subtly, a deadlock. `fork()` only carries over the calling thread, so a native mutex held by any other thread at fork time stays locked forever in the child, and calling into the native library to free anything can then block on it forever. +`close()` performs both steps. `__del__` performs only the release, leaving the callback attributes in place, but this leaks nothing: `__del__` runs when the `Stream` itself is being collected, taking those attributes down with it. -So the SDK never frees native memory in a process that did not allocate it. `ManagedResource.__init__` stamps the creating process ID onto the object (`record_owner_pid`), and `is_foreign_process()` compares it against the current PID during cleanup: +`Stream` never closes the Python object it wraps. The caller that opened a file owns that file. A `Reader` that opened the file itself tracks it separately and closes it during its own cleanup. -```mermaid -sequenceDiagram - participant P as Parent process - participant O as Reader object - participant F as Forked child +### Reference cycles - P->>O: __init__ stamps _owner_pid - P->>F: fork() - Note over O,F: Child inherits a copy of the object.
One native allocation, two Python copies. +Each ctypes callback closes over the `Stream` it belongs to. Captured directly, that would be a reference cycle, the `Stream` holding the callback and the callback holding the `Stream`, so nothing in the loop would reach a zero reference count on its own, leaving cleanup to the slower, less predictable cycle collector. The callbacks capture a weak reference instead, resolved fresh on each call, so the `Stream`'s reference count can still reach zero and its cleanup stays on the deterministic path. - F->>F: child's copy is cleaned up - F->>F: is_foreign_process() is true - Note right of F: Do not free: the parent owns it,
and a native mutex may be locked
by a thread that did not survive the fork - F->>F: null the handle, mark CLOSED +### `Reader.with_fragment()` - P->>O: close() - P->>P: frees the native pointer normally -``` +A `with_fragment()` call does two things: the FFI call consumes the Reader's current handle and returns a replacement, and the Reader updates its own Python-side fields, the `Stream` wrappers it owns and its manifest caches, to describe that new handle. Those two steps have to run as one unit, since two threads interleaving them could close a stream the native reader is still reading through, or leave stale wrappers describing a handle another thread already replaced. + +`Reader._fragment_lock` covers both steps, and `with_fragment()` is the only method that takes it. The guard spans the native call, so it cannot block waiting for the lock, since that call drives caller-supplied callbacks that could re-enter this API and deadlock. A second thread finding the lock held is refused immediately instead, leaving the Reader untouched: no stream built, no handle consumed, so the call succeeds once the other thread returns. This means one thread feeds fragments to a Reader at a time, and the caller must serialize those calls itself. The refusal messages are stable enough for callers to match on: -Both `_cleanup_resources()` and the consumed teardown take this branch, and neither simply skips the work: they null the handle and mark the object `CLOSED`, so the child cannot go on to use or free it later. Mutating the child's copy has no effect on the parent's, which stays untouched and valid. +- `Reader is already processing a fragment on another thread`: another thread is already inside `with_fragment()` on this Reader. +- `Reader is in use by another operation and cannot be consumed`: some other native call is in flight on this Reader. +- `Reader is running a mutating operation`: what a read method on another thread sees while `with_fragment()` runs, per [The full picture](#the-full-picture). -`_teardown()` checks `is_foreign_process()` before taking `_op_lock`, not after, so the foreign-process branch never tries to acquire a lock in the child. The lock itself would raise there anyway (see [Locking and in-flight tracking](#locking-and-in-flight-tracking)), but `_teardown()` needs to finish cleanup rather than raise, so it settles the fork case first and only reaches for the lock once it knows this process owns the pointer. +`resource_to_stream()`, by contrast, is a shared call: the underlying SDK method only reads, so other read methods on the same Reader run concurrently with it, a `close()` arriving meanwhile is deferred until it returns, and `with_fragment()` is refused until it returns, the same as any other mutating call. -The memory the child skips is not lost for good: a child calling `exec()` replaces its address space, and a child that exits has its memory reclaimed by the OS. Even a long-lived child, a `multiprocessing` worker using the fork start method, retains at most the objects it inherited at fork time, a bounded, one-off amount rather than a growing leak. Anything it allocates itself carries its own PID and is freed normally. +#### Cache invalidation -> [!NOTE] -> `is_foreign_process()` returns `False` when no owner PID was ever recorded, so an object that missed the stamp is cleaned up as before rather than leaking silently. +A `Reader` caches its manifest data, keyed to the handle it was read from. A successful `with_fragment()` swap invalidates that cache, and both the invalidation and every cache read in `json()` happen under the resource's lock, so a reader never observes a half-updated cache. The handle swap itself runs as a shared-style in-flight call rather than under that lock (native calls never hold it, per [Locking](#locking)), so there is a brief window between the new handle landing and the old cache being cleared. A `json()` on another thread that acquires the lock inside that window finds the stale cache still populated and returns it without ever consulting the handle, serving a manifest for the fragment just replaced. `_fragment_lock` does not close this window, since it only serializes `with_fragment()` against itself; a caller sharing one Reader across threads for both fragment-feeding and reading needs to serialize those two together itself. -## Which method to use when? +## Which method -`_create_and_activate`, `_consume_and_swap`, `_consume_no_replacement`, -`_consume_into`, `_release_handle`, and `_wrap_native_handle` each apply to a -different situation when writing a new subclass: +Writing a new `ManagedResource` subclass, each situation maps to one call: | Situation | Call this | | --- | --- | @@ -777,14 +461,11 @@ different situation when writing a new subclass: | An FFI call consumes the current handle and returns a replacement for the same object | `_consume_and_swap(ffi_call, error_message)` | | An FFI call consumes the current handle to configure or feed another object, returning only a status code | `_consume_no_replacement(ffi_call, error_message)` | | An FFI call consumes the current handle and returns a *different* object's pointer, for that object to own | `_consume_into(ffi_call, error_message)` | -| A call fails and it is unclear whether native took the handle first (a Python exception before native reported anything, or an empty error slot) | `_release_handle()` | +| A call fails and it is unclear whether native took the handle first | `_release_handle()` | | A Python instance needs to wrap a handle a native call already returned, without creating a new one | `_wrap_native_handle(handle)` (classmethod) | -| Ordinary teardown (`close()`, `__del__`) | Neither. These already route through `_cleanup_resources()` and `_teardown()`. Nothing outside `ManagedResource` itself calls `_teardown()` directly. | - -`_activate()` and `_consume_and_swap()` are two low-level primitives this -situation table builds on. +| Ordinary teardown (`close()`, `__del__`) | Neither. These already route through the shared cleanup path. Nothing outside `ManagedResource` itself tears an object down directly. | -## Implementing a subclass of `ManagedResource` +## Subclassing To wrap a new native resource, inherit from `ManagedResource` and follow these rules: @@ -844,18 +525,10 @@ class NativeResource(ManagedResource): ### Troubleshooting -- An attribute set only in `__init__` is missing on an instance built by `_wrap_native_handle()`, since that path never runs `__init__`. The failure shows up later as an `AttributeError` from whichever method reads the attribute, often `_release()` during cleanup. Attributes belong in `_init_attrs()`, which each subclass `__init__` calls and `_wrap_native_handle()` calls in its place. `ManagedResource.__init__` does not call it, so a subclass omitting the call gets neither path. - +- An attribute set only in `__init__` is missing on an instance built by `_wrap_native_handle()`, since that path never runs `__init__`. The failure shows up later as an `AttributeError` from whichever method reads the attribute, often `_release()` during cleanup. Attributes belong in `_init_attrs()`, which each subclass `__init__` calls and `_wrap_native_handle()` calls in its place. - Calling `_init_attrs()` after an FFI call that can raise leaves `_release()` reading attributes that do not exist yet when that call fails, crashing with `AttributeError`. It belongs right after `super().__init__()`, before anything that can fail. - -- Assigning `self._handle` or `self._lifecycle_state` directly bypasses the checks that make the lifecycle safe. `_activate()` refuses a null handle and refuses to run on an already-active object, and `_consume_and_swap()` requires the resource active and the replacement non-null. Direct assignment gives up both, and the resulting bugs, an ACTIVE object with a null handle, or a silently discarded pointer, surface far from their cause. - -- A `_release()` that raises has its exception silently swallowed by `_cleanup_resources()`, visible only in the logs. A small lifecycle for managed resources would let `_release()` check whether they need releasing, the release call itself wrapped in try/except as a fallback for unexpected failures. - +- Assigning `self._handle` or `self._lifecycle_state` directly bypasses the checks that make the lifecycle safe: `_activate()` refuses a null handle and an already-active object, and `_consume_and_swap()` requires an active resource and a non-null replacement. Direct assignment gives up both, and the resulting bugs, an `ACTIVE` object with a null handle or a silently discarded pointer, surface far from their cause. +- A `_release()` that raises has its exception silently swallowed, visible only in the logs. Guard it so it can check whether there is anything left to release, with a try/except as the fallback for unexpected failures. - `_release()` can be called more than once, via `close()` then `__del__`, or multiple `close()` calls, so it must handle running on an already-cleaned-up object. The standard pattern sets attributes to `None` after closing them. - -- Call `c2pa_free` through `ManagedResource`, not directly. A redundant free of an already-released pointer is not a crash: the native pointer registry rejects an untracked address without touching memory and returns `-1`, and `ManagedResource` relies on that guard so unknown-ownership failure paths can issue a guarded free without risking a double-free. A manual free still bypasses the lifecycle's state checks. - -- When a subclass inherits from both `ManagedResource` and an ABC like `ContextProvider`, and both define a property with the same name, `is_valid` say, Python resolves it using the MRO. The parent listed first in the class definition wins. List the ABC first, and Python finds the abstract property before the concrete one, raising `TypeError: Can't instantiate abstract class`. The class with the concrete implementation goes first: `class Context(ManagedResource, ContextProvider)`, not the reverse. - -- When two parent classes define the same method or property with different concrete implementations, the MRO silently picks the first one, which can cause subtle bugs from using the wrong one. `ClassName.__mro__` or `ClassName.mro()` confirms the expected resolution order for shared property names across multiple inheritance. +- Call `c2pa_free` through `ManagedResource`, not directly, so the lifecycle's state checks stay in effect. A redundant free is not itself a crash, the registry rejects an untracked address safely, but a manual free bypasses everything this doc describes. +- Multiple inheritance ordering matters for shared property names, covered in [Class hierarchy](#class-hierarchy). `ClassName.__mro__` confirms the resolution order when in doubt. diff --git a/src/c2pa/c2pa.py b/src/c2pa/c2pa.py index 02388b9e..289d980a 100644 --- a/src/c2pa/c2pa.py +++ b/src/c2pa/c2pa.py @@ -381,6 +381,7 @@ def _native_call(self): """ with self._live_op_lock(): self._ensure_valid_state() + self._ensure_no_mutating_call() self._inflight = getattr(self, '_inflight', 0) + 1 try: with _native_section(): @@ -722,13 +723,12 @@ def _invoke_consume(self, ffi_call, error_message, *, reserved=False): raise except Exception as e: if reserved: - # Resource left close (handle set). self._teardown(free_handle=True) else: self._release_handle() raise C2paError(error_message.format(e)) from e - def _raise_consume_failure(self, error_message, previous_state=None): + def _raise_consume_failure(self, error_message, *, reserved: bool): """Raise the error from an FFI handler consuming call. The native error is read before any free so a free's own @@ -740,18 +740,13 @@ def _raise_consume_failure(self, error_message, previous_state=None): as no error rather than as a stale one left by an earlier call. A caller that reserved the handle with _begin_consume() passes - previous_state and stays reserved until this classification finishes. - _read_native_error() is a native call and releases the GIL, so a - resource restored to ACTIVE before the tags are examined is visible as - usable to another thread while native may already own its handle. Only - the pre-consume branch hands the resource back. + reserved=True (in-flight mark). Args: error_message: Format string with one placeholder, used when the native layer offers no error of its own. - previous_state: Lifecycle state to restore if the handle turns out - to have been rejected before native took ownership. None when - the caller holds no reservation. + reserved: True when the caller reserved this handle with + _begin_consume(). Raises: C2paError: Always; typed by the native error when there is one. @@ -764,8 +759,6 @@ def _raise_consume_failure(self, error_message, previous_state=None): "ownership (%s); handle retained", type(self).__name__, error) - if previous_state is not None: - self._abort_consume(previous_state) _raise_typed_c2pa_error(error) # A non-tag error means the native side took ownership then failed, @@ -777,9 +770,7 @@ def _raise_consume_failure(self, error_message, previous_state=None): # No error of its own: ownership is unknown, so free defensively. # c2pa_free returns -1 for an address native already reclaimed. - # A reservation leaves the resource CLOSED with the handle set, - # which _release_handle() nulls without freeing. - if previous_state is not None: + if reserved: self._teardown(free_handle=True) else: self._release_handle() @@ -787,15 +778,11 @@ def _raise_consume_failure(self, error_message, previous_state=None): def _begin_consume(self): """Reserve this handle for a consuming call, or raise. - This is the initiation of an exclusive borrow. - - Marks the resource as closed, stopping other borrows. - After this, the call is considered in-flight. - The caller owns the matching decrement. - - Returns: - The lifecycle state to restore if the call turns out not to have - consumed the handle. + This is the initiation of an exclusive borrow, and "counts" + as an in-progress call that mutates something. + This reservation is exclusive. + The resource stays ACTIVE while being consumed. + A teardown (deferred while the consume is in flight) closes it. Raises: C2paError: Unusable resource or native call in progress. @@ -805,26 +792,16 @@ def _begin_consume(self): # without this the call would pass a null pointer to native. self._ensure_valid_state() self._ensure_not_borrowed() - previous = self._lifecycle_state - self._lifecycle_state = LifecycleState.CLOSED + self._mut_inflight = getattr(self, '_mut_inflight', 0) + 1 self._inflight = getattr(self, '_inflight', 0) + 1 - return previous - - def _abort_consume(self, previous_state): - """Undo _begin_consume() after a call that did not take the handle. - - A pre-consume tag usually means the handle is still ours, so the - resource becomes usable again. The tag can also name another tracked - argument, which this does not distinguish. - A deferred free still happens when the section drains, so a resource - with a queued teardown stays closed. - """ + def _end_consume(self): + """Release a _begin_consume() reservation, then run any teardown + that arrived while it was held.""" with self._live_op_lock(): - if self._pending_teardown is not None: - return - if self._lifecycle_state == LifecycleState.CLOSED and self._handle: - self._lifecycle_state = previous_state + self._mut_inflight -= 1 + self._inflight -= 1 + self._maybe_flush_pending() def _consume_and_swap(self, ffi_call, error_message): """Run an FFI call consuming the handle, reserving it. @@ -832,7 +809,7 @@ def _consume_and_swap(self, ffi_call, error_message): (a returned null value is a failure). """ - previous_state = self._begin_consume() + self._begin_consume() try: with _native_section(): new_ptr = self._invoke_consume( @@ -840,18 +817,10 @@ def _consume_and_swap(self, ffi_call, error_message): if new_ptr: with self._live_op_lock(): self._handle = new_ptr - if self._pending_teardown is None: - self._lifecycle_state = previous_state return - self._raise_consume_failure(error_message, previous_state) - except BaseException: - self._abort_consume(previous_state) - raise + self._raise_consume_failure(error_message, reserved=True) finally: - # Decrement to handle parallel potential in-flight consumers. - with self._live_op_lock(): - self._inflight -= 1 - self._maybe_flush_pending() + self._end_consume() def _consume_reserved(self, ffi_call, error_message, *, succeeded): """Run a reserved consuming call and mark the handle consumed on @@ -865,7 +834,7 @@ def _consume_reserved(self, ffi_call, error_message, *, succeeded): Returns: The call's raw result, for callers that hand it on. """ - previous_state = self._begin_consume() + self._begin_consume() try: with _native_section(): result = self._invoke_consume( @@ -873,16 +842,9 @@ def _consume_reserved(self, ffi_call, error_message, *, succeeded): if succeeded(result): self._teardown(free_handle=False) return result - self._raise_consume_failure(error_message, previous_state) - except BaseException: - self._abort_consume(previous_state) - raise + self._raise_consume_failure(error_message, reserved=True) finally: - # Same order as _consume_and_swap: drop _inflight under the - # lock, then flush. - with self._live_op_lock(): - self._inflight -= 1 - self._maybe_flush_pending() + self._end_consume() def _consume_no_replacement(self, ffi_call, error_message): """Run an FFI call that consumes this handle on success, when the native @@ -952,10 +914,14 @@ def _cleanup_resources(self): @property def is_valid(self) -> bool: - """Check if the resource is in a valid (active) state.""" + """Is the resource usable now? + ACTIVE, holding a handle, or a shared borrow, + and no mutating (exclusive) or consuming native call in progress now. + """ return ( self._lifecycle_state == LifecycleState.ACTIVE and self._handle is not None + and getattr(self, '_mut_inflight', 0) == 0 ) def close(self) -> None: @@ -2054,15 +2020,16 @@ class ContextProvider(ABC): @property @abstractmethod def is_valid(self) -> bool: - """Whether this provider is in a usable state. + """Whether a call using this provider would be accepted right now. - Return True when the underlying native context is active - and its handle has not been freed or consumed. Return - False after the provider has been closed or invalidated. + Return True when the underlying native context is active, holds a + handle, and has no mutating or consuming native call in flight. + Return False after the provider has been closed or invalidated. + The answer is a snapshot and does not keep the handle alive. The ManagedResource base class provides a standard - implementation that checks lifecycle state and handle - presence. + implementation that checks lifecycle state, handle presence and + the mutating in-flight count. """ ... @@ -2283,13 +2250,6 @@ def __init__( check=lambda r: r != 0) if signer is not None: - # No in-flight guard around the hand-off: the consume - # marks the signer CLOSED under its lock before calling - # native, which is what stops a signer.close() on another - # thread from freeing the handle mid-transfer. That mark - # also makes the consume refuse to start while another - # thread is borrowing the handle to sign with. - # # Retain a rejected signer for later teardown. self._signer_callback_cb = signer._callback_cb _check_handle_arg('builder', nb._handle) @@ -3377,8 +3337,17 @@ def with_fragment(self, format: Optional[str], stream, On failure the native call may already have consumed the underlying object, in which case this Reader is closed and cannot be retried: create a new one. - C2paError: If another thread is inside this method on the same - Reader, or another native call is in flight on it. + C2paError: "Reader is already processing a fragment on another + thread" when another thread is inside this method on the same + Reader. The Reader is untouched, and the call can be retried + after the other one returns. One thread feeds fragments to a + Reader, and the caller serializes those calls. + C2paError: "Reader is in use by another operation and cannot be + consumed" when another native call is in flight on it. + + While this call runs, read methods on other threads raise + C2paError("Reader is running a mutating operation") and is_valid is + False. The Reader is usable again when the call returns successfully. """ format_arg = _format_ffi_arg(_encode_format(format, "Reader")) @@ -3626,7 +3595,7 @@ def get_validation_results(self) -> Optional[dict]: return self._get_manifest_field(lambda d: d.get("validation_results")) def resource_to_stream(self, uri: str, stream: Any) -> int: - """Write a resource to a stream. + """Write a resource to a stream (shared borrow). Args: uri: The URI of the resource to write @@ -3640,7 +3609,7 @@ def resource_to_stream(self, uri: str, stream: Any) -> int: """ _check_cstr_arg("uri", uri) uri_str = uri.encode('utf-8') - with self._exclusive_native_call(), Stream(stream) as stream_obj: + with self._native_call(), Stream(stream) as stream_obj: result = _lib.c2pa_reader_resource_to_stream( self._handle, uri_str, stream_obj._stream) diff --git a/tests/perf/README.md b/tests/perf/README.md index 26ac8736..a9a87a93 100644 --- a/tests/perf/README.md +++ b/tests/perf/README.md @@ -338,8 +338,6 @@ it works. | --- | --- | --- | --- | | `trampoline_held_during_sign` | the signer trampoline outlives a `Context` closed mid-sign | `HELD` | `DROPPED` | | `no_free_during_parked_call` | no `c2pa_free` while a call still holds the handle | `freed=0` | `freed=1` | -| `read_refused_during_mutation` | a read during a mutating call is refused | `REFUSED` | `ALLOWED` | -| `second_mutation_refused` | a second mutating call is refused | `REFUSED` | `ALLOWED` | `no_free_during_parked_call` is the most direct: it instruments `ManagedResource._free_native_ptr`, the single funnel every free passes through, and diff --git a/tests/perf/thread_scenarios.py b/tests/perf/thread_scenarios.py index 63c7679c..5bfa530e 100644 --- a/tests/perf/thread_scenarios.py +++ b/tests/perf/thread_scenarios.py @@ -33,7 +33,6 @@ from c2pa import ( Builder, - C2paError, C2paSigningAlg, Context, Reader, @@ -156,8 +155,8 @@ def _callback_signer(inside, release) -> Signer: def _first_resource_uri(reader: Reader): """A resource identifier from the reader's manifest, or None. - resource_to_stream needs one, and it is the mutating call these scenarios - park inside. + resource_to_stream needs one, and it is the shared borrow that + no_free_during_parked_call parks inside. """ manifest = json.loads(reader.json()) for entry in manifest.get("manifests", {}).values(): @@ -393,77 +392,6 @@ def worker(): return _tally(rounds, one_round) -def scenario_read_refused_during_mutation(rounds: int = 20) -> dict: - """A read must be refused while a mutating call is in flight. - - Serving a read from a resource whose handle is mid-swap can return another - object's bytes. The guard refuses instead, and must refuse rather than block: - blocking here would deadlock against the callback that holds the call open. - - REFUSED: C2paError, as designed. - ALLOWED: the read was served during the mutation. - """ - signed = SIGNED_JPEG.read_bytes() - - def one_round(): - reader = Reader("image/jpeg", io.BytesIO(signed)) - reader.json() - uri = _first_resource_uri(reader) - if uri is None: - _close_quietly(reader) - return "NO_RESOURCE_URI" - - try: - with _ParkedResourceCall(reader, uri) as parked: - if not parked.parked: - return NOT_PARKED - try: - reader.detailed_json() - outcome = "ALLOWED" - except C2paError: - outcome = "REFUSED" - return outcome - finally: - _close_quietly(reader) - - return _tally(rounds, one_round) - - -def scenario_second_mutation_refused(rounds: int = 20) -> dict: - """A second mutating call must be refused while one is in flight. - - Two concurrent mutating calls can both drive the handle swap, which loses - track of which pointer native owns. - - REFUSED: C2paError, as designed. - ALLOWED: both mutations proceeded. - """ - signed = SIGNED_JPEG.read_bytes() - - def one_round(): - reader = Reader("image/jpeg", io.BytesIO(signed)) - reader.json() - uri = _first_resource_uri(reader) - if uri is None: - _close_quietly(reader) - return "NO_RESOURCE_URI" - - try: - with _ParkedResourceCall(reader, uri) as parked: - if not parked.parked: - return NOT_PARKED - try: - reader.resource_to_stream(uri, io.BytesIO()) - outcome = "ALLOWED" - except C2paError: - outcome = "REFUSED" - return outcome - finally: - _close_quietly(reader) - - return _tally(rounds, one_round) - - # Scenario name -> (function, expected outcome on hardened code). # The expected value is what the driver asserts; anything else fails the run. THREAD_SCENARIOS = { @@ -473,10 +401,6 @@ def one_round(): scenario_no_free_during_parked_call, "freed=0"), "builder_no_free_during_parked_sign": ( scenario_builder_no_free_during_parked_sign, "freed=0"), - "read_refused_during_mutation": ( - scenario_read_refused_during_mutation, "REFUSED"), - "second_mutation_refused": ( - scenario_second_mutation_refused, "REFUSED"), } # Derived so the name list cannot drift from the registry. diff --git a/tests/test_unit_tests.py b/tests/test_unit_tests.py index ba5e9ccd..a45d8f01 100644 --- a/tests/test_unit_tests.py +++ b/tests/test_unit_tests.py @@ -10368,10 +10368,6 @@ def tearDown(self): def test_generic_exception_frees_the_reserved_handle(self): """A reserved consume that raises must free, not drop, the handle. - - _begin_consume() leaves the resource CLOSED with the handle still set, - which _release_handle() reads as "not ours" and nulls without freeing, - while _abort_consume() can no longer restore it. """ def boom(handle): raise RuntimeError("callback failed after the reservation") diff --git a/tests/test_unit_tests_threaded.py b/tests/test_unit_tests_threaded.py index 457c2a55..eb0d6bb3 100644 --- a/tests/test_unit_tests_threaded.py +++ b/tests/test_unit_tests_threaded.py @@ -3644,9 +3644,10 @@ def hold_then_reenter(): class TestConsumeReservationWindow(unittest.TestCase): """The consume reservation must outlast ownership classification. - _read_native_error() is a native call that releases the GIL, so a resource - restored to ACTIVE before the error is classified is visible as usable to - another thread while native may already own its handle. + The reservation is a mutating in-flight mark. The lifecycle state stays + ACTIVE. _read_native_error() is a native call that releases the GIL, so a + reservation released before the error is classified would let another + thread use a handle native may already own. """ def test_no_thread_sees_a_consumed_handle_as_valid(self): @@ -5409,25 +5410,6 @@ def test_deferred_teardown_survives_a_flush_inside_a_section(self): finally: ManagedResource._free_native_ptr = real_free - def test_abort_consume_leaves_a_queued_teardown_closed(self): - """A resource whose free is already queued must not become usable. - - The deferred free still runs when the section drains, so restoring - ACTIVE would hand the caller a resource that closes underneath it. - """ - context = Context() - with _native_section(): - context.close() - self.assertIsNotNone(context._pending_teardown) - - context._abort_consume(LifecycleState.ACTIVE) - self.assertEqual( - context._lifecycle_state, LifecycleState.CLOSED, - "a resource with a queued teardown was revived") - self.assertFalse( - context.is_valid, - "a resource with a queued teardown reported itself usable") - def test_section_drain_error_does_not_mask_the_body_error(self): """The body's exception is what the caller asked for, so it wins.""" @@ -5854,49 +5836,6 @@ def seek(self, *args): builder.add_action('{"action": "c2pa.color_adjustments"}') builder.close() - def test_read_during_mutation_is_rejected(self): - with open(os.path.join(FIXTURES_FOLDER, "C.jpg"), "rb") as f: - image = f.read() - reader = Reader("image/jpeg", io.BytesIO(image)) - manifest = reader.get_active_manifest() - uri = (manifest or {}).get("thumbnail", {}).get("identifier") - self.assertTrue(uri, "fixture must carry a thumbnail resource") - - inside = threading.Event() - release = threading.Event() - - class BlockingSink(io.BytesIO): - def write(self, data): - inside.set() - release.wait(10) - return super().write(data) - - def seek(self, *args): - inside.set() - release.wait(10) - return super().seek(*args) - - worker = threading.Thread( - target=lambda: reader.resource_to_stream(uri, BlockingSink()), - daemon=True) - worker.start() - try: - self.assertTrue( - inside.wait(10), - "resource_to_stream never reached its callback") - - with self.assertRaises(Error) as raised: - reader.detailed_json() - self.assertIn("mutating operation", str(raised.exception)) - finally: - release.set() - worker.join(10) - - self.assertFalse(worker.is_alive(), "resource_to_stream hung") - # Works again once the mutating call has returned. - self.assertTrue(reader.detailed_json()) - reader.close() - if __name__ == '__main__': unittest.main() From 0e32cb57f47889749ce0894613d2a7370b8eb2e3 Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 20:03:41 -0700 Subject: [PATCH 02/11] fix: Shorten doc --- docs/native-resources-management.md | 80 ++++++++++++++++------------- 1 file changed, 44 insertions(+), 36 deletions(-) diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index a1964054..a47a39a9 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -2,7 +2,7 @@ `ManagedResource` is the internal base class the C2PA Python SDK uses to wrap native (Rust/FFI) pointers. `Reader`, `Builder`, `Signer`, `Context`, and `Settings` all subclass it. -A `Reader`, for example, holds a pointer to memory the native library allocated. Python's garbage collector tracks the `Reader` object itself, but has no visibility into that native memory, so it can never free it on its own. `ManagedResource` closes that gap: it frees the native pointer exactly once, however the object stops being used. +A `Reader`, for example, holds a pointer to memory the native library allocated. Python's garbage collector tracks the `Reader` object, but has no visibility into that native memory, so it can never free it. `ManagedResource` closes that gap: it frees the native pointer once, however the object stops being used. ## Vocabulary @@ -16,11 +16,11 @@ A pointer is **consumed** when a native call takes ownership of it, often return ## Garbage collection -Python's garbage collector mainly works by reference counting: each object counts how many references point to it, and reaching zero frees it. This works for pure Python objects, but a `Reader`'s native pointer sits outside that system entirely. The collector sees the `Reader` wrapper and tracks references to it, but has no idea the `_handle` attribute points at memory of its own, and never calls the native free function. Collect the wrapper without freeing that memory first, and it leaks. +Python's garbage collector works by reference counting: each object counts how many references point to it, and reaching zero frees it. This works for pure Python objects, but a `Reader`'s native pointer sits outside that system. The collector sees the `Reader` wrapper and tracks references to it, but has no idea the `_handle` attribute points at memory of its own, and never calls the native free function. Collect the wrapper without freeing that memory first, and it leaks. -### `__del__` is not enough +### About the finalizer hook `__del__` -`__del__`, Python's finalizer hook, could in principle free the native pointer whenever an object is collected, and `ManagedResource` does use it as a fallback. But its timing is unpredictable: garbage collection itself is non-deterministic, an object caught in a reference cycle waits for a separate cycle collector to run, and during interpreter shutdown Python may collect objects in any order, so a `__del__` that reads global state can find it already torn down. On implementations that don't use reference counting, PyPy and GraalPy, `__del__` may not run until long after the last reference is gone, or not before the process exits. Every class that holds a native pointer should inherit from `ManagedResource` rather than relying on `__del__` alone. +`__del__`, Python's finalizer hook, could free the native pointer whenever an object is collected, and `ManagedResource` uses it as a fallback. But its timing is unpredictable: garbage collection is non-deterministic, an object caught in a reference cycle waits for a separate cycle collector to run, and during interpreter shutdown Python may collect objects in any order, so a `__del__` that reads global state can find it torn down. On implementations that don't use reference counting, PyPy and GraalPy, `__del__` may not run until long after the last reference is gone, or not before the process exits. Every class that holds a native pointer should inherit from `ManagedResource` rather than rely on `__del__` alone. ## Releasing @@ -50,7 +50,7 @@ Calling `close()` directly is equivalent to exiting a `with` block. It is idempo ### Destructor -Without `with` or `.close()`, `__del__` attempts the free when Python garbage-collects the object, for the reasons in [`__del__` is not enough](#__del__-is-not-enough). Treat it as a safety net, not the primary mechanism. +Without `with` or `.close()`, `__del__` attempts the free when Python garbage-collects the object, for the reasons in [About the finalizer hook `__del__`](#about-the-finalizer-hook-__del__). Treat it as a safety net, not the primary mechanism. ### Nesting @@ -62,7 +62,7 @@ with open("photo.jpg", "rb") as file, Reader("image/jpeg", file) as reader: # reader is closed first, then file ``` -The order matters because the `Reader`'s native pointer reads the file's data through a [`Stream`](#streams) wrapper: the native library calls back into that stream to read bytes. Close the file first, and those callbacks are still reachable from native code but read from a closed file, which can read freed memory. Closing the Reader first frees the native pointer while the file is still open, and only then closes the file. `with` guarantees this order: whatever is listed later, or nested deeper, is torn down first. +The order matters because the `Reader`'s native pointer reads the file's data through a [`Stream`](#streams) wrapper: the native library calls back into that stream to read bytes. Close the file first, and those callbacks stay reachable from native code but read from a closed file, which can read freed memory. Closing the Reader first frees the native pointer while the file is open, then closes the file. `with` guarantees this order: whatever is listed later, or nested deeper, is torn down first. ## Lifecycle states @@ -92,7 +92,7 @@ A native call takes several steps in sequence: check the object is usable, hand 2. While that call is still running, thread B calls `reader.close()`, which frees the native pointer. 3. Thread A's native code, still running, reads through the pointer it was given, now freed. -Step 3 reads freed memory. Depending on what the allocator has done with that memory since, this either crashes the process or returns another object's bytes, corrupting state far from the code responsible (see [Crashes](#crashes)). +Step 3 reads freed memory. Depending on what the allocator has done with that memory, this crashes the process or returns another object's bytes, corrupting state far from the code responsible (see [Crashes](#crashes)). A Python object has no owning thread: it belongs to whoever holds a reference to it, and nothing about creating a `Reader` or passing it into a closure hands ownership from one thread to another. Any two threads sharing a reference to the same `Reader`, `Builder`, `Signer`, or `Context` can hit this race, so [Lifecycle states](#lifecycle-states) alone is not the whole model. A resource also needs a way to say "a call is using me right now, don't free me out from under it," which is what in-flight tracking adds next. @@ -105,7 +105,7 @@ A call in flight is one of two kinds: - **Shared**: several threads can run this kind of call on the same object at once. Reading a manifest with `.json()` is shared: nothing stops two threads from reading the same data at the same time. - **Mutating**: only one thread may run this kind of call at a time, and no shared call may start while it runs. Signing with `.sign()` is mutating: it changes what the object holds, so a concurrent read could see a half-updated result or a pointer being replaced out from under it. -A `close()` arriving while any call, shared or mutating, is in flight does not free the pointer immediately. It marks the resource so no new caller can start using it, and defers the actual free until every in-flight call has returned. This is what closes the race from the previous section: thread B's `close()` still takes effect right away from its own point of view, but the memory thread A is still reading stays valid until thread A's call returns. +A `close()` arriving while any call, shared or mutating, is in flight does not free the pointer right away. It marks the resource so no new caller can start using it, and defers the free until every in-flight call returns. This closes the race from the previous section: thread B's `close()` takes effect at once from its own point of view, but the memory thread A is reading stays valid until thread A's call returns. ## The full picture @@ -140,7 +140,7 @@ The `Mutating --> [*]` exit inside `ACTIVE` is a consuming call: one that hands | `ACTIVE`, a mutating call in flight | **False** | `C2paError`: "running a mutating operation" | | `CLOSED` | False | `C2paError`: "is closed" | -`is_valid` answers one question: would a call be accepted right now? It is `ACTIVE`, holding a handle, and no mutating call in flight. It is a lock-free snapshot, so a passing check does not keep the handle alive. Only an actual guarded call does that. +`is_valid` answers one question: would a call be accepted right now? It is `ACTIVE`, holding a handle, and no mutating call in flight. It is a lock-free snapshot, so a passing check does not keep the handle alive. Only a guarded call does that. A `Reader`'s read methods, `.json()`, `.detailed_json()`, `.resource_to_stream()`, are shared. A `Builder`'s `.sign()` is mutating, and ends by consuming the Builder: signing closes it, so a `Builder` is single-use. `Reader.with_fragment()` is also mutating, and also consumes: it replaces the Reader's handle with a new one rather than closing the object (see [Consuming](#consuming)). @@ -148,9 +148,9 @@ A `Reader`'s read methods, `.json()`, `.detailed_json()`, `.resource_to_stream() A Python bug raises an exception with a traceback. A native memory bug terminates the process, and the operating system reports it as a signal, since the failure happens outside anything Python's exception machinery watches. -SIGSEGV, segmentation fault, comes from the hardware: the CPU traps on a read or write to memory the process is not allowed to touch, and the kernel delivers SIGSEGV. This fires when a pointer no longer points at accessible memory, for instance freed memory the allocator has unmapped. Freeing memory does not always unmap it, though. Allocators often keep the pages and reuse them for a later allocation, so reading through a freed pointer frequently succeeds anyway, silently returning another object's bytes and corrupting state far from the code responsible. SIGSEGV is what happens on the unlucky path, when the pages are actually gone. +SIGSEGV, segmentation fault, comes from the hardware: the CPU traps on a read or write to memory the process is not allowed to touch, and the kernel delivers SIGSEGV. This fires when a pointer no longer points at accessible memory, for instance freed memory the allocator has unmapped. Freeing memory does not always unmap it. Allocators often keep the pages and reuse them for a later allocation, so reading through a freed pointer often succeeds anyway, returning another object's bytes and corrupting state far from the code responsible. SIGSEGV happens on the unlucky path, when the pages are gone. -SIGABRT, abort, comes from software: a program calls `abort()` on itself after detecting a broken invariant. Allocators do this when `free()` is handed a pointer it never issued, the same pointer twice, or a heap whose bookkeeping a stray write has damaged. Which signal a given bug produces depends on the allocator: glibc raises SIGABRT with a diagnostic, the macOS allocator gives SIGTRAP, and a use-after-free that reaches unmapped pages gives SIGSEGV on both. In every case, the process terminates from inside native code. No exception, no `finally` block, no traceback. +SIGABRT, abort, comes from software: a program calls `abort()` on itself after detecting a broken invariant. Allocators do this when `free()` is handed a pointer it never issued, the same pointer twice, or a heap whose bookkeeping a stray write has damaged. Which signal a bug produces depends on the allocator: glibc raises SIGABRT with a diagnostic, the macOS allocator gives SIGTRAP, and a use-after-free that reaches unmapped pages gives SIGSEGV on both. In every case, the process terminates from inside native code. No exception, no `finally` block, no traceback. ## Locking @@ -160,7 +160,15 @@ Each `ManagedResource` holds a reentrant lock, `_op_lock`, and the in-flight cou The lock is never held across a native call that drives a stream callback (construction, `resource_to_stream`, signing, and the rest), since that callback can call back into this API on the same thread, and holding the lock there would deadlock against that reentry. Those calls increment an in-flight counter under the lock, release the lock, run the native call, then decrement the counter, which is the mechanism the [state diagram](#the-full-picture) describes as "a call in flight." -This is why the interpreter's Global Interpreter Lock, the GIL, does not make this safe on its own. CPython executes one bytecode instruction at a time under the GIL, so simple operations cannot corrupt a built-in container. But a foreign function call through ctypes releases the GIL for its duration, so another thread runs freely while native code is running, and a `close()` can land inside that exact window. `_op_lock` and the in-flight counters are what close it, not the GIL. +This is why the interpreter's Global Interpreter Lock, the GIL, does not make this safe on its own. CPython executes one bytecode instruction at a time under the GIL, so simple operations cannot corrupt a built-in container. But a foreign function call through ctypes releases the GIL for its duration, so another thread runs while native code runs, and a `close()` can land inside that window. `_op_lock` and the in-flight counters close it, not the GIL. + +### Why the native side cannot guard this alone + +The native library keeps its own bookkeeping for the pointers it hands out. That bookkeeping guards the pointer itself. It can't guard what Python does with the pointer. + +A callback is Python code the native library calls partway through a call, to read a stream or sign a message. It belongs to Python, so the native library keeps no bookkeeping for it. If Python frees that callback's object while the call runs, the next invocation reaches memory Python gave up. `_op_lock` and the in-flight counters keep that memory alive for the call. + +Using a handle takes two steps: check it, then act on it. Native bookkeeping guards only the second step. A second thread can free the same address between the first thread's check and its use, a gap of machine instructions. The address can even be reissued to another object before that use runs, so a passed check now points at memory belonging to something else. Holding `_op_lock` across both steps closes that gap. ### Lock ordering @@ -168,13 +176,13 @@ Two threads acquiring the same pair of locks in opposite orders can deadlock, ea ### Borrowing vs consuming -A shared call, a **borrow**, passes the handle to native and gets it back unchanged. A mutating call that ends by consuming the handle hands ownership to native, which frees the original pointer during the call. A borrow validates the pointer once on entry and then holds it for the whole call without checking again, so a consume starting midway through a borrow would free memory the borrow is still reading. +A shared call, a **borrow**, passes the handle to native and gets it back unchanged. A mutating call that ends by consuming the handle hands ownership to native, which frees the original pointer during the call. A borrow validates the pointer once on entry, then holds it for the whole call without checking again, so a consume starting midway through a borrow would free memory the borrow is reading. `ManagedResource` prevents this by refusing to start a consume while a borrow is in flight, and by reserving the handle as a mutating call for the whole duration of the consume, the same reservation any other mutating call makes. Every entry point checks the in-flight counters under `_op_lock` before it starts, so a call that would otherwise race a consume is refused instead with "running a mutating operation," and the resource's lifecycle state never changes until the consume is fully classified as a success or a failure. `Reader.with_fragment()` also serializes itself against other calls to itself with a lock of its own, covered in [`Reader.with_fragment()`](#readerwith_fragment). ## Double frees -Freeing the same native pointer twice corrupts the allocator's bookkeeping. A detecting allocator stops the process. An allocator that misses it lets the damage surface later, somewhere unrelated to the code responsible. Three distinct hazards lead here, each with its own guard: +Freeing the same native pointer twice corrupts the allocator's bookkeeping. A detecting allocator stops the process. An allocator that misses it lets the damage surface later, somewhere unrelated to the code responsible. Three hazards lead here, each with its own guard: | Hazard | Guarded by | | --- | --- | @@ -182,11 +190,11 @@ Freeing the same native pointer twice corrupts the allocator's bookkeeping. A de | A forked child process freeing a pointer its parent owns | A process-ID stamp on every object; cleanup in a process that did not allocate the pointer marks it closed without freeing (see [Fork safety](#fork-safety)). | | Two threads racing a `close()` against an in-flight call on the same object | `_op_lock` and the in-flight counters (see [Locking](#locking)): a `close()` mid-call is deferred, not applied. | -Sharing one `ManagedResource` instance across threads still needs these guards. Nothing here protects two threads racing on distinct objects that happen to share an allocator, since that is the allocator's own concurrency guarantee, not this layer's. +Sharing one `ManagedResource` instance across threads needs these guards. Nothing here protects two threads racing on distinct objects that happen to share an allocator, since that is the allocator's own concurrency guarantee, not this layer's. ## Consuming -Consuming a handle is a mutating call that hands the pointer to native, which takes ownership and either returns a replacement pointer or frees the original. Python must never free a pointer it handed to a consuming call, whichever way the call ends, since the address may already belong to a different object by the time it returns. +Consuming a handle is a mutating call that hands the pointer to native, which takes ownership and either returns a replacement pointer or frees the original. Python must never free a pointer it handed to a consuming call, whichever way the call ends, since the address may belong to a different object by the time it returns. There are two shapes a consuming call takes. @@ -223,9 +231,9 @@ flowchart TD F2 -.same value.- AMB ``` -The native error message tells the two apart. A specific set of rejection tags means the handle was never taken, so Python keeps and later frees it normally. Any other error means native took it and dropped it, so Python's cleanup runs without freeing anything. If no error was set at all, ownership is unknown, and Python falls back to a **guarded free**: the native pointer registry safely rejects a free of an address it no longer tracks, returning `-1` rather than crashing, so this is harmless whether or not native still holds it. +The native error message tells the two apart. A set of rejection tags means the handle was never taken, so Python keeps and later frees it normally. Any other error means native took it and dropped it, so Python's cleanup runs without freeing anything. If no error was set, ownership is unknown, and Python falls back to a **guarded free**: the native pointer registry rejects a free of an address it no longer tracks, returning `-1` rather than crashing, so this is harmless whether or not native still holds it. -The one case a guarded free is not used defensively is once ownership is already known to have moved: there, the free is skipped rather than issued and left for the registry to reject. A freed address can eventually be reused by an unrelated allocation, and an unnecessary free landing after that reuse would destroy the wrong object. Skipping a free that provides no value avoids that window entirely. +The one case a guarded free is not used is once ownership is known to have moved: there, the free is skipped rather than issued and left for the registry to reject. A freed address can be reused by an unrelated allocation, and an unnecessary free landing after that reuse would destroy the wrong object. Skipping a free that provides no value avoids that window. Three helpers share this triage, differing only in what a successful call returns: @@ -277,13 +285,13 @@ sequenceDiagram X->>X: activate the new Context ``` -A few details in that sequence matter beyond what the diagram shows. The transfer is not wrapped in a shared-call reservation; it uses the mutating reservation described in [Borrowing vs consuming](#borrowing-vs-consuming), which is what defers a racing `signer.close()` until the transfer is classified. `set_signer` does not always take the pointer, so the triage has to read the native error before deciding whether the Signer closed. And the temporary native builder used to construct the Context is itself a small `ManagedResource`, held only inside a `with` block, so any failure along the way frees it through the same `close()` path rather than a bespoke handler. +A few details in that sequence matter beyond what the diagram shows. The transfer is not wrapped in a shared-call reservation; it uses the mutating reservation described in [Borrowing vs consuming](#borrowing-vs-consuming), which defers a racing `signer.close()` until the transfer is classified. `set_signer` does not always take the pointer, so the triage reads the native error before deciding whether the Signer closed. The temporary native builder used to construct the Context is itself a small `ManagedResource`, held inside a `with` block, so any failure along the way frees it through the same `close()` path rather than a bespoke handler. ### Adopting a handle -Ownership can also arrive from the other direction: a native call returns a pointer that needs a Python wrapper around it, with no `__init__` call, since `__init__` would try to create a new native resource rather than wrap an existing one. `_wrap_native_handle()` handles this: it builds a bare instance, sets its lifecycle bookkeeping, runs `_init_attrs()` for subclass defaults, and activates the handle. Ownership transfers only once that call returns; if it raises, no wrapper exists yet, and the caller still owns the pointer and must free it itself. +Ownership can also arrive from the other direction: a native call returns a pointer that needs a Python wrapper around it, with no `__init__` call, since `__init__` would try to create a new native resource rather than wrap an existing one. `_wrap_native_handle()` handles this: it builds a bare instance, sets its lifecycle bookkeeping, runs `_init_attrs()` for subclass defaults, and activates the handle. Ownership transfers once that call returns; if it raises, no wrapper exists, and the caller still owns the pointer and must free it itself. -`Reader._init_from_context` and `Builder._init_from_context` both do something that looks backward: they create a native object and activate it immediately, before making the consuming call that will actually feed it data. This is deliberate. A consuming call needs an already-active resource to read the handle from and swap the result into, and activating first puts the intermediate pointer under normal cleanup right away: whichever way the consuming call goes, `close()` and `__del__` free it correctly. Holding the raw pointer in a local variable instead would leave every failure path to decide for itself whether to free it. +`Reader._init_from_context` and `Builder._init_from_context` both do something that looks backward: they create a native object and activate it before making the consuming call that will feed it data. This is deliberate. A consuming call needs an active resource to read the handle from and swap the result into, and activating first puts the intermediate pointer under normal cleanup right away: whichever way the consuming call goes, `close()` and `__del__` free it correctly. Holding the raw pointer in a local variable instead would leave every failure path to decide for itself whether to free it. ## Guarantees @@ -301,11 +309,11 @@ Every subclass gets these guarantees from `ManagedResource`, and must not break ## Keeping references alive -When a Python object passes a callback or a pointer to the native library, that reference must stay alive for as long as native code might still use it, and the garbage collector has no way to know that on its own: it only sees Python references. +When a Python object passes a callback or a pointer to the native library, that reference must stay alive as long as native code might use it, and the garbage collector has no way to know that: it only sees Python references. -The SDK keeps these references as plain instance attributes on the owning object. A `Stream` stores its four callback objects this way, so they stay referenced for as long as the `Stream` itself is alive (see [Reference cycles](#reference-cycles) for how those callbacks avoid keeping the `Stream` alive in return). A `Signer` consumed by a `Context` has its callback copied to an attribute on the `Context`, so the callback survives the `Signer` object closing. +The SDK keeps these references as plain instance attributes on the owning object. A `Stream` stores its four callback objects this way, so they stay referenced as long as the `Stream` is alive (see [Reference cycles](#reference-cycles) for how those callbacks avoid keeping the `Stream` alive in return). A `Signer` consumed by a `Context` has its callback copied to an attribute on the `Context`, so the callback survives the `Signer` object closing. -`_release()` sets these attributes to `None` during cleanup, letting them be collected, and it runs before the native pointer is freed, so anything the pointer might still depend on, an open file, a stream wrapper, is torn down first. This is `ManagedResource`'s ordering; `Stream` releases in the reverse order for reasons covered in [`Stream` cleanup](#stream-cleanup). +`_release()` sets these attributes to `None` during cleanup, letting them be collected, and it runs before the native pointer is freed, so anything the pointer depends on, an open file, a stream wrapper, is torn down first. This is `ManagedResource`'s ordering; `Stream` releases in the reverse order for reasons covered in [`Stream` cleanup](#stream-cleanup). ## Freeing memory @@ -317,7 +325,7 @@ def _free_native_ptr(ptr): return _lib.c2pa_free(ptr) ``` -It returns `0` when the pointer was really freed, and `-1` when the registry rejected an already-consumed or untracked address, the same `-1` a guarded free relies on. `ManagedResource` guarantees this is called exactly once per pointer. +It returns `0` when the pointer was freed, and `-1` when the registry rejected an already-consumed or untracked address, the same `-1` a guarded free relies on. `ManagedResource` guarantees this is called once per pointer. ## Cleanup errors @@ -329,7 +337,7 @@ Cleanup must not let an ordinary exception mask the exception that caused a `wit - The object is marked `CLOSED` before `_release()` runs or anything is freed, so a cleanup that fails partway still leaves the object closed, and a second attempt does not repeat the damage. - `close()` on an already-closed object returns immediately. -These handlers catch `Exception`, not `BaseException`. The interpreter's own unwinding signals, a cancellation or a shutdown in progress, are `BaseException`, so they pass through untouched and the remaining free may not run. This is deliberate: such a signal means the process is already going away, address space and native allocations included, so holding cleanup open to finish a free that is about to become irrelevant would only delay the shutdown the caller asked for. +These handlers catch `Exception`, not `BaseException`. The interpreter's own unwinding signals, a cancellation or a shutdown in progress, are `BaseException`, so they pass through untouched and the remaining free may not run. This is deliberate: such a signal means the process is going away, address space and native allocations included, so holding cleanup open to finish a free that is about to become irrelevant would only delay the shutdown the caller asked for. All three cleanup entry points converge on one method: @@ -355,7 +363,7 @@ The "foreign process" branch is explained in [Fork safety](#fork-safety), next. `fork()` copies the calling process, including every Python object holding a native pointer, but the underlying native allocation is not duplicated: there is still only one, and the parent owns it. -If a forked child cleaned up its copy of that object normally, two things would go wrong. It would free a pointer the parent is still using, a double-free. And more subtly, `fork()` only carries over the thread that called it, so if any other thread held a native lock at fork time, that lock stays held forever in the child, and calling into native code to free anything can then block on it forever. +If a forked child cleaned up its copy of that object normally, two things would go wrong. It would free a pointer the parent is using, a double-free. And more subtly, `fork()` only carries over the thread that called it, so if another thread held a native lock at fork time, that lock stays held forever in the child, and calling into native code to free anything can then block on it forever. So the SDK never frees native memory in a process that did not allocate it. Every object is stamped with its creating process's ID at construction, and cleanup compares that stamp against the current process before doing anything: @@ -378,9 +386,9 @@ sequenceDiagram P->>P: frees the native pointer normally ``` -The child's copy is not just skipped, it is nulled and marked `CLOSED`, so nothing in the child can go on to use or free it later. This never touches the parent's copy, which stays valid. +The child's copy is not just skipped, it is nulled and marked `CLOSED`, so nothing in the child can go on to use or free it. This never touches the parent's copy, which stays valid. -The memory a child skips freeing is not lost for good: a child that calls `exec()` replaces its whole address space, and a child that exits has its memory reclaimed by the operating system. Even a long-lived forked worker retains at most the objects it inherited at fork time, a bounded amount rather than a growing leak, since anything it allocates itself carries its own process ID and is freed normally. +The memory a child skips freeing is not lost for good: a child that calls `exec()` replaces its address space, and a child that exits has its memory reclaimed by the operating system. Even a long-lived forked worker retains at most the objects it inherited at fork time, a bounded amount rather than a growing leak, since anything it allocates carries its own process ID and is freed normally. ## Class hierarchy @@ -413,33 +421,33 @@ Bytes reach the native library through a `Stream`, which wraps a Python file or A `Reader` or `Builder` holds a resource that Python code calls methods on. A `Stream` holds a resource the native library calls back into instead, for read, seek, write, and flush. Ownership means something different for a resource that receives calls rather than makes them, so `Stream` gets its own release path instead of the shared one. -`Stream` tracks its state with two flags, `_closed` and `_initialized`, rather than the `LifecycleState` machinery from [Lifecycle states](#lifecycle-states), but still supports the same three cleanup paths: context manager, explicit `.close()`, `__del__` fallback. +`Stream` tracks its state with two flags, `_closed` and `_initialized`, rather than the `LifecycleState` machinery from [Lifecycle states](#lifecycle-states), but supports the same three cleanup paths: context manager, explicit `.close()`, `__del__` fallback. ### Reentrant callbacks A `Stream` registers four ctypes callbacks. Reading from the stream runs one of them, running caller-supplied Python, and the same reentrancy hazard applies as anywhere else in this doc: a native call driving one of these callbacks cannot hold a lock across the call, since the callback may call back into this API on the same thread. -Each callback checks the stream's own state before touching the underlying Python object, and reports an error rather than reading through it if the stream has already been torn down. +Each callback checks the stream's own state before touching the underlying Python object, and reports an error rather than reading through it if the stream has been torn down. ### `Stream` cleanup `Stream` holds its own reentrant lock, `_close_lock`, serializing `close()`, `__del__`, and a `close()` from another thread against each other, for the same reason `_op_lock` is reentrant: a callback attribute set to `None` inside the locked region can drop the last reference to an object whose own finalizer then needs the same lock. -Cleanup runs in the direction of dependency: whatever can still invoke or reach the other side is torn down first. This reverses the order `ManagedResource` uses, because a `Stream`'s callbacks are invoked by native code rather than the other way around: `Stream` releases the native handle first, guaranteeing no callback can fire again, and only then drops the callback objects. `ManagedResource` instead runs subclass cleanup first, since its native pointer depends on those resources staying valid until then. +Cleanup runs in the direction of dependency: whatever can invoke or reach the other side is torn down first. This reverses the order `ManagedResource` uses, because a `Stream`'s callbacks are invoked by native code rather than the other way around: `Stream` releases the native handle first, guaranteeing no callback can fire again, then drops the callback objects. `ManagedResource` instead runs subclass cleanup first, since its native pointer depends on those resources staying valid until then. -`close()` performs both steps. `__del__` performs only the release, leaving the callback attributes in place, but this leaks nothing: `__del__` runs when the `Stream` itself is being collected, taking those attributes down with it. +`close()` performs both steps. `__del__` performs the release, leaving the callback attributes in place, but this leaks nothing: `__del__` runs when the `Stream` is being collected, taking those attributes down with it. `Stream` never closes the Python object it wraps. The caller that opened a file owns that file. A `Reader` that opened the file itself tracks it separately and closes it during its own cleanup. ### Reference cycles -Each ctypes callback closes over the `Stream` it belongs to. Captured directly, that would be a reference cycle, the `Stream` holding the callback and the callback holding the `Stream`, so nothing in the loop would reach a zero reference count on its own, leaving cleanup to the slower, less predictable cycle collector. The callbacks capture a weak reference instead, resolved fresh on each call, so the `Stream`'s reference count can still reach zero and its cleanup stays on the deterministic path. +Each ctypes callback closes over the `Stream` it belongs to. Captured directly, that would be a reference cycle, the `Stream` holding the callback and the callback holding the `Stream`, so nothing in the loop would reach a zero reference count on its own, leaving cleanup to the slower cycle collector. The callbacks capture a weak reference instead, resolved fresh on each call, so the `Stream`'s reference count can reach zero and its cleanup stays on the deterministic path. ### `Reader.with_fragment()` A `with_fragment()` call does two things: the FFI call consumes the Reader's current handle and returns a replacement, and the Reader updates its own Python-side fields, the `Stream` wrappers it owns and its manifest caches, to describe that new handle. Those two steps have to run as one unit, since two threads interleaving them could close a stream the native reader is still reading through, or leave stale wrappers describing a handle another thread already replaced. -`Reader._fragment_lock` covers both steps, and `with_fragment()` is the only method that takes it. The guard spans the native call, so it cannot block waiting for the lock, since that call drives caller-supplied callbacks that could re-enter this API and deadlock. A second thread finding the lock held is refused immediately instead, leaving the Reader untouched: no stream built, no handle consumed, so the call succeeds once the other thread returns. This means one thread feeds fragments to a Reader at a time, and the caller must serialize those calls itself. The refusal messages are stable enough for callers to match on: +`Reader._fragment_lock` covers both steps, and `with_fragment()` is the only method that takes it. The guard spans the native call, so it cannot block waiting for the lock, since that call drives caller-supplied callbacks that could re-enter this API and deadlock. A second thread finding the lock held is refused at once instead, leaving the Reader untouched: no stream built, no handle consumed, so the call succeeds once the other thread returns. This means one thread feeds fragments to a Reader at a time, and the caller must serialize those calls itself. The refusal messages are stable enough for callers to match on: - `Reader is already processing a fragment on another thread`: another thread is already inside `with_fragment()` on this Reader. - `Reader is in use by another operation and cannot be consumed`: some other native call is in flight on this Reader. @@ -449,7 +457,7 @@ A `with_fragment()` call does two things: the FFI call consumes the Reader's cur #### Cache invalidation -A `Reader` caches its manifest data, keyed to the handle it was read from. A successful `with_fragment()` swap invalidates that cache, and both the invalidation and every cache read in `json()` happen under the resource's lock, so a reader never observes a half-updated cache. The handle swap itself runs as a shared-style in-flight call rather than under that lock (native calls never hold it, per [Locking](#locking)), so there is a brief window between the new handle landing and the old cache being cleared. A `json()` on another thread that acquires the lock inside that window finds the stale cache still populated and returns it without ever consulting the handle, serving a manifest for the fragment just replaced. `_fragment_lock` does not close this window, since it only serializes `with_fragment()` against itself; a caller sharing one Reader across threads for both fragment-feeding and reading needs to serialize those two together itself. +A `Reader` caches its manifest data, keyed to the handle it was read from. A successful `with_fragment()` swap invalidates that cache, and both the invalidation and every cache read in `json()` happen under the resource's lock, so a reader never observes a half-updated cache. The handle swap runs as a shared-style in-flight call rather than under that lock (native calls never hold it, per [Locking](#locking)), so there is a window between the new handle landing and the old cache being cleared. A `json()` on another thread that acquires the lock inside that window finds the stale cache populated and returns it without consulting the handle, serving a manifest for the fragment just replaced. `_fragment_lock` does not close this window, since it only serializes `with_fragment()` against itself; a caller sharing one Reader across threads for both fragment-feeding and reading needs to serialize those two together itself. ## Which method From 67c4834aa47928d1122d95097aadb94b6c09e636 Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 20:13:30 -0700 Subject: [PATCH 03/11] fix: Shorten doc --- docs/native-resources-management.md | 38 ++++++++++++++--------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index a47a39a9..f1e68614 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -1,37 +1,37 @@ # Native resource management -`ManagedResource` is the internal base class the C2PA Python SDK uses to wrap native (Rust/FFI) pointers. `Reader`, `Builder`, `Signer`, `Context`, and `Settings` all subclass it. +`ManagedResource` is the internal base class the C2PA Python SDK uses to wrap native (Rust/C FFI) pointers. `Reader`, `Builder`, `Signer`, `Context`, and `Settings` all subclass it. A `Reader`, for example, holds a pointer to memory the native library allocated. Python's garbage collector tracks the `Reader` object, but has no visibility into that native memory, so it can never free it. `ManagedResource` closes that gap: it frees the native pointer once, however the object stops being used. ## Vocabulary -A **native pointer** is an address that says where a piece of memory lives. The C2PA SDK wraps a Rust library, the "native" library, which allocates memory Python cannot see. +A **native pointer** is an address that says where a piece of memory lives. The C2PA SDK wraps a Rust library, the "native" library, which allocates (native) memory Python cannot see. -A **handle** is the one native pointer a `ManagedResource` object holds at a time, stored in its `_handle` attribute. +A **handle** is a native pointer a `ManagedResource` object holds at a time, stored in its `_handle` attribute. -**Ownership** answers one question: who must free a given piece of native memory, exactly once. Freeing it zero times leaks memory. Freeing it twice corrupts the allocator and can crash the process. +**Ownership** answers one question: who must free native memory (the handle), exactly one time. Freeing it zero times leaks memory. Freeing it twice corrupts the allocator and can crash the process. -A pointer is **consumed** when a native call takes ownership of it, often returning a replacement pointer in its place. Once consumed, Python must never free the original. +A pointer is **consumed** when a native call takes ownership of it, often returning a replacement pointer in its place (updating the handle). Once consumed, Python must never free the original. ## Garbage collection -Python's garbage collector works by reference counting: each object counts how many references point to it, and reaching zero frees it. This works for pure Python objects, but a `Reader`'s native pointer sits outside that system. The collector sees the `Reader` wrapper and tracks references to it, but has no idea the `_handle` attribute points at memory of its own, and never calls the native free function. Collect the wrapper without freeing that memory first, and it leaks. +Python's garbage collector works by reference counting: each object counts how many references point to it, and reaching zero frees the object. This works for pure Python objects, but a `Reader`'s native pointer sits outside that system. The collector sees the `Reader` wrapper and tracks references to it, but does not know the `_handle` attribute points at memory of its own (allocated by the native library), and the garbage collector never calls the native free function. ### About the finalizer hook `__del__` -`__del__`, Python's finalizer hook, could free the native pointer whenever an object is collected, and `ManagedResource` uses it as a fallback. But its timing is unpredictable: garbage collection is non-deterministic, an object caught in a reference cycle waits for a separate cycle collector to run, and during interpreter shutdown Python may collect objects in any order, so a `__del__` that reads global state can find it torn down. On implementations that don't use reference counting, PyPy and GraalPy, `__del__` may not run until long after the last reference is gone, or not before the process exits. Every class that holds a native pointer should inherit from `ManagedResource` rather than rely on `__del__` alone. +`__del__`, Python's finalizer hook, could free the native pointer whenever an object is collected, and `ManagedResource` uses it too. But the timing is unpredictable: garbage collection is non-deterministic, an object caught in a reference cycle waits for a separate cycle collector to run, and during interpreter shutdown Python may collect objects in any order, so a `__del__` that reads global state can find it already gone. Every class that holds a native pointer should inherit from `ManagedResource` rather than rely on `__del__` alone. -## Releasing +## Releasing memory -`ManagedResource` gives every object three ways to release its native pointer: a `with` statement, an explicit `close()`, or, as a fallback only, the destructor. +`ManagedResource` gives every object three ways to release its native pointer: a `with` statement, an explicit `close()`, or, as a fallback, the destructor. -### `with` +### `with` statement ```python with Reader("image.jpg") as reader: print(reader.json()) -# reader is automatically closed here, even if an exception occurs +# reader is automatically closed here. ``` On exit, `__exit__` calls `close()`, freeing the native pointer even if the block raised. @@ -46,15 +46,15 @@ finally: reader.close() ``` -Calling `close()` directly is equivalent to exiting a `with` block. It is idempotent: a second call does nothing. +Calling `close()` directly is equivalent to exiting a `with` block. `close()` is idempotent: a second call does nothing. ### Destructor -Without `with` or `.close()`, `__del__` attempts the free when Python garbage-collects the object, for the reasons in [About the finalizer hook `__del__`](#about-the-finalizer-hook-__del__). Treat it as a safety net, not the primary mechanism. +Without `with` or `.close()`, `__del__` attempts the free when Python garbage-collects the object (and it can't e known in advance when the garbage collector will run, and when it will release those resources). ### Nesting -Multiple resources can share one `with` statement or nest in separate blocks. Either way, Python cleans them up in reverse order: right to left, or inner to outer. +Multiple resources can share one `with` statement or nest in separate `with` blocks. They are cleaned up in reverse order: right to left (when sharing one statement), or inner to outer (when statements are nested). ```python with open("photo.jpg", "rb") as file, Reader("image/jpeg", file) as reader: @@ -62,11 +62,11 @@ with open("photo.jpg", "rb") as file, Reader("image/jpeg", file) as reader: # reader is closed first, then file ``` -The order matters because the `Reader`'s native pointer reads the file's data through a [`Stream`](#streams) wrapper: the native library calls back into that stream to read bytes. Close the file first, and those callbacks stay reachable from native code but read from a closed file, which can read freed memory. Closing the Reader first frees the native pointer while the file is open, then closes the file. `with` guarantees this order: whatever is listed later, or nested deeper, is torn down first. +`with` guarantees a release order: whatever is listed later, or nested deeper, is torn down first. ## Lifecycle states -Every `ManagedResource` tracks one of three states: +Every `ManagedResource` has 3 states: ```mermaid stateDiagram-v2 @@ -78,15 +78,15 @@ stateDiagram-v2 CLOSED --> [*] ``` -- `UNINITIALIZED`: the Python object exists but has no native pointer yet. This is transient, lasting only for the duration of construction. +- `UNINITIALIZED`: the (Python) object exists but has no native pointer (handle) yet. This is transient, lasting only for the duration of construction. - `ACTIVE`: the native pointer is valid, and the object can be used. - `CLOSED`: the native pointer has been freed, or ownership of it has moved elsewhere. Any further use raises `C2paError`. -This is one-way: once `CLOSED`, an object never becomes `ACTIVE` again. A construction that fails before activation can also close straight from `UNINITIALIZED`, since there is nothing to free, only a state to record. +Once `CLOSED`, an object never becomes `ACTIVE` again. A construction that fails before activation can also close directly from `UNINITIALIZED`, since there is nothing to free, only a state to record. ## Close during a call -A native call takes several steps in sequence: check the object is usable, hand the pointer to native code, let that code run. Two threads sharing one object can interleave those steps: +A native call takes several steps in sequence: check the object is usable, hand the pointer to native code, let the code run. Two threads sharing one object can interleave those steps: 1. Thread A calls `reader.json()`. It checks the Reader is usable, then enters the native call. 2. While that call is still running, thread B calls `reader.close()`, which frees the native pointer. From 62c57f3b24e03625ef8d62c585b152ececda9f78 Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 20:21:48 -0700 Subject: [PATCH 04/11] fix: Shorten doc --- docs/native-resources-management.md | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index f1e68614..09393f8f 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -90,26 +90,30 @@ A native call takes several steps in sequence: check the object is usable, hand 1. Thread A calls `reader.json()`. It checks the Reader is usable, then enters the native call. 2. While that call is still running, thread B calls `reader.close()`, which frees the native pointer. -3. Thread A's native code, still running, reads through the pointer it was given, now freed. +3. Thread A's native code, still running, reads through the pointer it was given, now freed (which crashes). -Step 3 reads freed memory. Depending on what the allocator has done with that memory, this crashes the process or returns another object's bytes, corrupting state far from the code responsible (see [Crashes](#crashes)). +A Python object has no owning thread: it belongs to whoever holds a reference to it, and nothing about creating an object or passing it to another thread, in a closure or an argument, hands exclusive access to that thread. Both threads above hold a plain reference to the same object instance. Python lets either one call any method on it at any time. -A Python object has no owning thread: it belongs to whoever holds a reference to it, and nothing about creating a `Reader` or passing it into a closure hands ownership from one thread to another. Any two threads sharing a reference to the same `Reader`, `Builder`, `Signer`, or `Context` can hit this race, so [Lifecycle states](#lifecycle-states) alone is not the whole model. A resource also needs a way to say "a call is using me right now, don't free me out from under it," which is what in-flight tracking adds next. +The interleaving in step 2 could be possible because the CPython interpreter switches between threads between bytecode instructions, and a native call spans many of them. Calling a method by itself does not stop thread B from calling `close()` on the same object while that call runs, so thread B's `close()` can land at any point during thread A's call, including partway through. The `ManagedResource` class has functionalities to avoid that. + +Depending on what the allocator has done with that freed memory, this crashes the process or returns another object's bytes, corrupting state far from the code responsible. + +Any two threads sharing a reference to the same `Reader`, `Builder`, `Signer`, or `Context` can hit this race, so [Lifecycle states](#lifecycle-states) alone is not the whole model. A resource also needs a way to say "a call is using me right now, don't free me out from under it," which is what in-flight tracking adds next. ## In-flight calls -Alongside its lifecycle state, every `ManagedResource` counts calls currently running against it. This is separate from the `ACTIVE`/`CLOSED` state: a resource can be `ACTIVE` and idle, or `ACTIVE` with one or more calls in flight, and those are different things a caller needs to know. +Alongside its lifecycle state, every `ManagedResource` counts calls currently running against it. This is separate from the `ACTIVE`/`CLOSED` state: a resource can be `ACTIVE` and idle, or `ACTIVE` with one or more calls in flight. -A call in flight is one of two kinds: +A call in flight can be either: - **Shared**: several threads can run this kind of call on the same object at once. Reading a manifest with `.json()` is shared: nothing stops two threads from reading the same data at the same time. -- **Mutating**: only one thread may run this kind of call at a time, and no shared call may start while it runs. Signing with `.sign()` is mutating: it changes what the object holds, so a concurrent read could see a half-updated result or a pointer being replaced out from under it. +- **Mutating**: only one thread may run this kind of call at a time, and no shared call may start while it runs, because a concurrent read could see a half-updated result or a pointer being replaced out from under it. -A `close()` arriving while any call, shared or mutating, is in flight does not free the pointer right away. It marks the resource so no new caller can start using it, and defers the free until every in-flight call returns. This closes the race from the previous section: thread B's `close()` takes effect at once from its own point of view, but the memory thread A is reading stays valid until thread A's call returns. +A `close()` arriving while any call, shared or mutating, is in flight does not free the pointer immediately. It marks the resource so no new caller can start using it, and defers the free until all in-flight calls have returned. For instance, thread B's `close()` takes effect at once from its own point of view, but the memory thread A is reading stays valid until thread A's call returns. -## The full picture +## Lifecycle overview -Lifecycle state and in-flight calls happen at the same time. A resource is always in one lifecycle state, `UNINITIALIZED`, `ACTIVE`, or `CLOSED`. Independently, while it is `ACTIVE`, zero or more calls may be in flight on it right now, and if any of them is mutating, no other call may start. Put together, this is the full picture every guard in `ManagedResource` checks before letting a call through: +Lifecycle state and in-flight calls work together to manage a native resource. A resource is always in one lifecycle state, `UNINITIALIZED`, `ACTIVE`, or `CLOSED`. Independently, while it is `ACTIVE`, zero or more calls may be in flight on it right now, and if any of them is mutating, no other call may start. Together, this is the full picture every guard in `ManagedResource` checks before letting a call through: ```mermaid stateDiagram-v2 @@ -140,9 +144,9 @@ The `Mutating --> [*]` exit inside `ACTIVE` is a consuming call: one that hands | `ACTIVE`, a mutating call in flight | **False** | `C2paError`: "running a mutating operation" | | `CLOSED` | False | `C2paError`: "is closed" | -`is_valid` answers one question: would a call be accepted right now? It is `ACTIVE`, holding a handle, and no mutating call in flight. It is a lock-free snapshot, so a passing check does not keep the handle alive. Only a guarded call does that. +`is_valid` defines if a call would be accepted right now. It is `ACTIVE`, holding a handle, and no mutating call in flight. It is a lock-free snapshot, so a passing check does not keep the handle alive. Only a guarded call does that. -A `Reader`'s read methods, `.json()`, `.detailed_json()`, `.resource_to_stream()`, are shared. A `Builder`'s `.sign()` is mutating, and ends by consuming the Builder: signing closes it, so a `Builder` is single-use. `Reader.with_fragment()` is also mutating, and also consumes: it replaces the Reader's handle with a new one rather than closing the object (see [Consuming](#consuming)). +A `Reader`'s read methods, `.json()`, `.detailed_json()`, `.resource_to_stream()`, are shared. A `Builder`'s `.sign()` is mutating, and ends by consuming the Builder: signing closes it, so a `Builder` is single-use (see [Consuming](#consuming)). ## Crashes From b4c3e9b718daa1f37fbb8a9903f7b45ec4efa1aa Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 20:27:00 -0700 Subject: [PATCH 05/11] fix: Shorten doc --- docs/native-resources-management.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index 09393f8f..a5c0d86a 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -150,23 +150,23 @@ A `Reader`'s read methods, `.json()`, `.detailed_json()`, `.resource_to_stream() ## Crashes -A Python bug raises an exception with a traceback. A native memory bug terminates the process, and the operating system reports it as a signal, since the failure happens outside anything Python's exception machinery watches. +Usually, native memory bug terminates the process, and the operating system reports it as a signal, since the failure happens outside anything Python's exception machinery watches. -SIGSEGV, segmentation fault, comes from the hardware: the CPU traps on a read or write to memory the process is not allowed to touch, and the kernel delivers SIGSEGV. This fires when a pointer no longer points at accessible memory, for instance freed memory the allocator has unmapped. Freeing memory does not always unmap it. Allocators often keep the pages and reuse them for a later allocation, so reading through a freed pointer often succeeds anyway, returning another object's bytes and corrupting state far from the code responsible. SIGSEGV happens on the unlucky path, when the pages are gone. +SIGSEGV, segmentation fault, comes from the hardware: the CPU traps on a read or write to memory the process is not allowed to touch, and the kernel delivers SIGSEGV. This fires when a pointer no longer points at accessible memory, for instance freed memory the allocator has unmapped. Freeing memory does not always unmap it. Allocators often keep the pages and reuse them for a later allocation, so reading through a freed pointer can succeed, returning another object's bytes and corrupting state far from the code responsible. SIGSEGV happens when the pages are gone. -SIGABRT, abort, comes from software: a program calls `abort()` on itself after detecting a broken invariant. Allocators do this when `free()` is handed a pointer it never issued, the same pointer twice, or a heap whose bookkeeping a stray write has damaged. Which signal a bug produces depends on the allocator: glibc raises SIGABRT with a diagnostic, the macOS allocator gives SIGTRAP, and a use-after-free that reaches unmapped pages gives SIGSEGV on both. In every case, the process terminates from inside native code. No exception, no `finally` block, no traceback. +SIGABRT, abort, comes from software: a program calls `abort()` on itself after detecting a broken invariant. Allocators do this when `free()` is handed a pointer it never issued, the same pointer twice, or a heap whose bookkeeping a stray write has damaged. + +In every case, the process terminates from inside native code. No exception, no `finally` block, no traceback can be run by the Python code. ## Locking Each `ManagedResource` holds a reentrant lock, `_op_lock`, and the in-flight counters from [The full picture](#the-full-picture). Together they enforce that rule: a mutating call excludes every other call, and a `close()` arriving mid-call is deferred rather than applied immediately. -`_op_lock` is a `threading.RLock`, reentrant, rather than a plain `Lock`, for two reasons. A finalizer can run at any point, including inside a method that already holds the lock on that thread, so `__del__` calling back into locked code must not deadlock against itself. And a consuming call tears the handle down from inside a region it already holds the lock in, so it needs to reacquire rather than block. - -The lock is never held across a native call that drives a stream callback (construction, `resource_to_stream`, signing, and the rest), since that callback can call back into this API on the same thread, and holding the lock there would deadlock against that reentry. Those calls increment an in-flight counter under the lock, release the lock, run the native call, then decrement the counter, which is the mechanism the [state diagram](#the-full-picture) describes as "a call in flight." +`_op_lock` is a `threading.RLock`, reentrant, rather than a plain `Lock`. A finalizer can run at any point, including inside a method that already holds the lock on that thread, so `__del__` calling back into locked code must not deadlock against itself. And a consuming call tears the handle down from inside a region it already holds the lock in, so it needs to reacquire rather than block. -This is why the interpreter's Global Interpreter Lock, the GIL, does not make this safe on its own. CPython executes one bytecode instruction at a time under the GIL, so simple operations cannot corrupt a built-in container. But a foreign function call through ctypes releases the GIL for its duration, so another thread runs while native code runs, and a `close()` can land inside that window. `_op_lock` and the in-flight counters close it, not the GIL. +The lock is never held across a native call that drives a stream callback, since that callback can call back into this API on the same thread, and holding the lock there would deadlock against that reentry. Those calls increment an in-flight counter under the lock, release the lock, run the native call, then decrement the counter, which is the mechanism the [state diagram](#the-full-picture) describes as "a call in flight." -### Why the native side cannot guard this alone +This is why the interpreter's Global Interpreter Lock, the GIL, does not make this safe on its own. CPython executes one bytecode instruction at a time under the GIL, so simple operations cannot corrupt a built-in container. But a foreign function call through ctypes releases the GIL for its duration, so another thread runs while native code runs, and a `close()` can land inside that window. `_op_lock` and the in-flight counters avoids this case. The native library keeps its own bookkeeping for the pointers it hands out. That bookkeeping guards the pointer itself. It can't guard what Python does with the pointer. @@ -182,13 +182,13 @@ Two threads acquiring the same pair of locks in opposite orders can deadlock, ea A shared call, a **borrow**, passes the handle to native and gets it back unchanged. A mutating call that ends by consuming the handle hands ownership to native, which frees the original pointer during the call. A borrow validates the pointer once on entry, then holds it for the whole call without checking again, so a consume starting midway through a borrow would free memory the borrow is reading. -`ManagedResource` prevents this by refusing to start a consume while a borrow is in flight, and by reserving the handle as a mutating call for the whole duration of the consume, the same reservation any other mutating call makes. Every entry point checks the in-flight counters under `_op_lock` before it starts, so a call that would otherwise race a consume is refused instead with "running a mutating operation," and the resource's lifecycle state never changes until the consume is fully classified as a success or a failure. `Reader.with_fragment()` also serializes itself against other calls to itself with a lock of its own, covered in [`Reader.with_fragment()`](#readerwith_fragment). +`ManagedResource` prevents this by refusing to start a consume while a borrow is in flight, and by reserving the handle as a mutating call for the whole duration of the consume, the same reservation any other mutating call makes. Every entry point checks the in-flight counters under `_op_lock` before it starts, so a call that would otherwise race a consume is refused instead with "running a mutating operation," and the resource's lifecycle state never changes until the consume is fully classified as a success or a failure. `Reader.with_fragment()` also serializes itself against other calls to itself with a lock of its own. ## Double frees -Freeing the same native pointer twice corrupts the allocator's bookkeeping. A detecting allocator stops the process. An allocator that misses it lets the damage surface later, somewhere unrelated to the code responsible. Three hazards lead here, each with its own guard: +Freeing the same native pointer twice corrupts the allocator's bookkeeping. A detecting allocator stops the process. An allocator that misses it lets the damage surface later, somewhere unrelated to the code responsible. -| Hazard | Guarded by | +| Risk | Mitigated by | | --- | --- | | Freeing a pointer a consuming call already took | The consumed pointer is abandoned rather than freed; the native error message says whether ownership actually moved (see [Consuming](#consuming)). | | A forked child process freeing a pointer its parent owns | A process-ID stamp on every object; cleanup in a process that did not allocate the pointer marks it closed without freeing (see [Fork safety](#fork-safety)). | From b53cbd2d83eac6e41db674a5a3779d41cbffa48b Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 20:33:59 -0700 Subject: [PATCH 06/11] fix: Shorten doc --- demo/10-native-section.html | 223 ------------------ demo/20-close-during-call.html | 157 ------------ demo/30-context-sign-callback.html | 158 ------------- demo/index.html | 43 ---- demo/style.css | 354 ---------------------------- docs/native-resources-management.md | 69 +----- 6 files changed, 12 insertions(+), 992 deletions(-) delete mode 100644 demo/10-native-section.html delete mode 100644 demo/20-close-during-call.html delete mode 100644 demo/30-context-sign-callback.html delete mode 100644 demo/index.html delete mode 100644 demo/style.css diff --git a/demo/10-native-section.html b/demo/10-native-section.html deleted file mode 100644 index 0be5b7b4..00000000 --- a/demo/10-native-section.html +++ /dev/null @@ -1,223 +0,0 @@ - - - - - -The native section - - - -
- -
Demo walkthrough  /  10  ·  concept
- -

The native section

-

A marked stretch of time on one thread, between a call returning and its error being read. Any free inside it destroys the message.

- -
- -
-
backgroundshared objects, per-thread windows
-
- - - - - - a Python object has no owning thread: it belongs to whoever holds a reference - - - one Resource - one native handle inside it - - thread A - thread B - - - same reference - same reference - - - may open a native section - - may open its own, separately - - nothing copies it, nothing hands it over: a closure, an attribute or an argument is enough - - - the GIL stops both threads running Python at the same instant, but it does not stop them - touching the same object, and it is given away entirely during a native call - -
The section is thread-local because the error slot it protects is. A window open on thread A must not gate thread B's frees, or the two would block each other.
-
- -
-

A Python object lives on the process heap: one region of memory shared by every thread. A thread is not a container that holds objects; it is a separate execution position, with its own call stack, walking through that same shared memory.

- -

So nothing switches threads and nothing is handed over. In the probe behind this figure, a Reader created on MainThread and used from worker-1 stayed at the same address the whole time, with the same id(). Two threads simply looked at the same place.

- -

What is shared is the name. An ordinary closure is enough:

- -
r = Reader("image/jpeg", io.BytesIO(img))
-
-def worker():
-    return r.json()          # closure captures r
-
-threading.Thread(target=worker).start()
- -

No serialisation, no copy, no transfer step. Compare multiprocessing, where a separate heap per process forces objects to be pickled across: there id(r) would differ and mutations would not be visible. Threads have no such boundary between them.

-
-
- -

The glossary sentence that follows is about executing bytecode. The GIL keeps MainThread and worker-1 from running Python instructions in the same instant. Both can still hold a reference to one Reader at once, and one can call close() while the other is mid-call.

- -
-
backgroundwhat the GIL is, and why it does not save you here
-
- - - - - - the global interpreter lock: a token inside CPython. Only its holder runs Python bytecode. - - - "assure that only one thread executes Python bytecode at a time," Python glossary - - thread A holds it - - thread B holds it - - thread A holds it - taking turns, never simultaneously - - - - 1  it is handed away for the whole of a native call - - "the GIL is always released when doing I/O," Python glossary - - thread A: inside the native call - - thread B: running Python - at once - - - - 2  and it never made a statement indivisible - - "Python does not guarantee that high-level statements are atomic," Python glossary - self._ensure_valid_state() ← the handle is checked - _lib.c2pa_reader_json(self._handle) ← it is used - - another thread - runs in this gap - -
Two reasons the GIL allows these bugs: it is given up entirely during a native call, and a check followed by its use was never one atomic step.
-
-
- -
-
beforemain: a finalizer's free lands in the window
-
- - - - - one thread, no second thread involved - - - - the window - - - call returns - message now in slot - - - error read - ownership decided from it - - - unrelated object collected → __del__ → c2pa_free - - - slot overwritten: "Other: UntrackedPointer: 0x..." - the real message is gone, and the wrong one steers the decision - -
A free of an untracked pointer writes its own complaint into the same slot. CPython runs finalizers at any bytecode boundary, so this needs no threads at all.
-
-
- -
-
afterthe window is marked; frees inside it are queued
-
- - - - - one thread - - - - _native_section: depth > 0 - - - call returns - - error read - message intact - - - same __del__ → teardown sees depth > 0 - - - queued on pending_resources: no free yet - - - depth 0: - queue drains - - the free still happens, just after the message has been read - nested sections raise the depth; only the outermost close drains - -
The free is postponed. Without the pending list it would never happen, because nothing else would touch that object again.
-
-
- -
- -
-

Notes

-

Objects do not belong to threads. If they did, a close() on thread B could not reach thread A's handle, and most of this branch would be unnecessary.

-

Why the window exists. A C interface cannot raise an exception, so failure arrives in two pieces: a return value, and a message fetched by a separate c2pa_error() call. Between the two, arbitrary Python runs.

- -

What the GIL promises. Picture one token. Whichever thread holds it runs Python code; every other thread waits its turn. That is the entire guarantee: which thread's Python statements run right now, nothing about which objects exist or when they get freed.

- -

Why the token changes hands. A native call such as c2pa_reader_json() runs compiled library code, not Python bytecode. The interpreter hands the token to another thread for the whole call, and takes it back when the call returns. That thread can now run any Python it likes, including code that frees objects, while the first thread's native call is still going.

- -

What freeing means here. A Reader or a Context wraps one native handle: a raw pointer the C library allocated. Freeing calls c2pa_free() on it, exactly once. A second free, or a read after one, is undefined behavior on the native side: a memory fault, or silent corruption, with no Python exception.

- -

The cross-wire. Thread A calls a method, hands the token away, and sits inside the native call holding the pointer. Thread B holds the same Python reference, because objects do not belong to threads. With the token free, thread B calls close() or drops the last reference, and either path ends in c2pa_free() on the pointer thread A is still using. The GIL only ever promised A and B would not run Python bytecode at the same instant. A was inside native code at that instant, token already gone.

- -

The native section. A critical section is a lock: whoever wants in waits for the holder to leave, and every other thread is excluded. The native section runs thread B's close() immediately, every time. _in_native_section() checks thread A's own per-thread depth; if it is above zero, close() records the resource on thread A's pending list. Thread A frees it when its own section closes.

- -

Thread-local because the native slot is: one thread's section must not gate another's frees. Depth-counted because native calls nest, and _read_native_error is itself one. A boolean would be cleared by the innermost exit while an outer classification was still reading.

-

The drain surfaces your exception unchanged. It keeps the first cleanup failure and logs it; the bare raise re-raises whatever the body threw. A cleanup problem is logged, and the error you were reporting is what you get back.

-
-c2pa.py:1040-1101  _native_section, _in_native_section, _register_for_section_flush
-c2pa.py:473-474  registration  ·  c2pa.py:517-521  re-registration
-c2pa.py:1071-1081  _drain: swaps the list, isolates each failure
-main  none of these symbols exist
-tests  test_native_section_defers_unrelated_finalizer_free, test_section_drain_error_does_not_mask_the_body_error -
-
- -
- - 20  Close during a call → -
- -
- - diff --git a/demo/20-close-during-call.html b/demo/20-close-during-call.html deleted file mode 100644 index 58fa95fa..00000000 --- a/demo/20-close-during-call.html +++ /dev/null @@ -1,157 +0,0 @@ - - - - - -Closing something another thread is using - - - -
- -
Demo walkthrough  /  20
- -

Closing something another thread is using

-

A close() frees a handle another thread has already passed into a native call.

- -
- -
-
backgroundhow thread B gets hold of thread A's reader
-
- - - - - - a Python object has no owning thread: it belongs to whoever holds a reference - - - one Reader - one native handle inside it - - thread A - thread B - - - same reference - same reference - - - reader.json() - - reader.close() - - nothing copies it, nothing hands it over: a closure, an attribute or an argument is enough - - - the GIL stops both threads running Python at the same instant, but it does not stop them - touching the same object, and it is given away entirely during a native call - -
No handover happens. Both threads simply hold the same reference, so both may call methods on it at any time.
-
-
- -
-
beforemain: close frees immediately
-
- - - - - - thread A - native - thread B - - - - - reader.json() - handle checked: valid - - hands the GIL away - - - running, holding - the pointer - - close() - - frees now - - - memory released - native still reading it - - - crash, or garbage - -
The validity check passed before the free. Nothing re-checks it, and the fault happens inside native code with no Python traceback.
-
-
- -
-
aftercalls are counted; the close is recorded and deferred
-
- - - - - - thread A - native - thread B - - - - - reader.json() - _inflight = 1 - - - - running, holding - the pointer - - close() - - marks CLOSED now, - records the free - frees nothing yet - - - call returns intact - - _inflight = 0 - - recorded free runs here, once - -
The object is unusable from the moment close is called, but the memory outlives the call that is using it.
-
-
- -
- -
-

Notes

-

Both threads reach the same object because both hold a reference to it. The GIL keeps them from running Python at the same instant but is handed away entirely during a native call: the GIL figure on page 10 shows what it does and does not cover.

-

Locking across the call deadlocks. These native calls run caller-supplied stream callbacks, which can call back into this API, possibly from a new thread. A lock held across the call would deadlock against that re-entry, so the lock here is held only long enough to change a counter.

-

Two independent reasons defer a teardown: this object's own call being in flight, and the native section. The recorded flag merges with and, so a "close without freeing" can never be upgraded to a free by a later caller who does not know the pointer already moved.

-
-c2pa.py:265-273  the new per-resource state
-c2pa.py:346-363  _native_call: counts, does not lock across the call
-c2pa.py:464-476  _teardown: the deferring branch
-main:264-267  __init__ was three assignments; main:330-337  _teardown freed at once
-tests  test_close_inside_callback_defers_free, test_deferred_consume_is_not_upgraded_to_free -
-
- -
- ← 10  The native section - 30  The signer callback freed mid-signature → -
- -
- - diff --git a/demo/30-context-sign-callback.html b/demo/30-context-sign-callback.html deleted file mode 100644 index 8704439e..00000000 --- a/demo/30-context-sign-callback.html +++ /dev/null @@ -1,158 +0,0 @@ - - - - - -The signer callback freed mid-signature - - - -
- -
Demo walkthrough  /  30
- -

The signer callback freed mid-signature

-

What gets freed is a function pointer Python created, which the library calls through while signing.

- -
- -
-
backgroundthe context is shared, and so is its callback
-
- - - - - - a Python object has no owning thread: it belongs to whoever holds a reference - - - one Context - one native handle inside it - - thread A - thread B - - - same reference - same reference - - - signs through its callback - - closes it, dropping that reference - - nothing copies it, nothing hands it over: a closure, an attribute or an argument is enough - - - the GIL stops both threads running Python at the same instant, but it does not stop them - touching the same object, and it is given away entirely during a native call - -
The trampoline is kept alive by one attribute on the shared Context. Either thread can drop the last reference to it.
-
-
- -
-
beforemain: close drops the callback, unguarded
-
- - - - - - thread A - native - thread B - - - - - builder.sign() - - - - signing - - - calls back through - the trampoline - - context.close() - _signer_callback_cb = None - - last reference gone - - - trampoline collected - - next callback enters freed memory - -
Native holds the trampoline's address but no reference to it, so ordinary Python reference counting can free it mid-call.
-
-
- -
-
afterthe context is held in flight for the duration of the sign
-
- - - - - - thread A - native - thread B - - - - - builder.sign() - _context_guard(context) - - - - signing - - - trampoline pinned - for the whole call - - context.close() - - marks CLOSED, records - the release - - - sign completes - - - callback released here, by the last to leave - -
The same guard also refuses a sign that starts on an already-closed context. A sign without a signer never runs.
-
-
- -
- -
-

Notes

-

Both threads reach the same object because both hold a reference to it. The GIL keeps them from running Python at the same instant but is handed away entirely during a native call: the GIL figure on page 10 shows what it does and does not cover.

-

What a trampoline is. To let native code call a Python function, ctypes builds a small object native can call like a C function. It is an ordinary Python object with ordinary reference counting, and nothing on the native side holds a reference to it. Keeping it alive as long as native might call it is the caller's job; here the Context holds that reference.

-

The failure with no error. If the callback is already gone when the sign begins, native signs without calling it and reports success. The output file looks signed; the signer never ran. The guard's validity check turns that into an exception.

-

The guard is duck-typed, so a caller-supplied context implementing only the published contract still works, at the cost of no in-flight protection. A test exists to ensure the built-in Context never falls into that unprotected branch.

-
-c2pa.py:4362-4372  the guarded context-sign
-c2pa.py:1945-1957  _context_guard: duck-typed on _native_call
-main:1722-1724  _release dropped the callback unconditionally
-tests  test_context_sign_after_close_raises_rather_than_skipping_signer, test_built_in_context_still_gets_in_flight_protection -
-
- -
- ← 20  Closing something another thread is using - -
- -
- - diff --git a/demo/index.html b/demo/index.html deleted file mode 100644 index b2f6c931..00000000 --- a/demo/index.html +++ /dev/null @@ -1,43 +0,0 @@ - - - - - -What this branch fixes - - - -
- -

Notes

-

Helper pages for review.

- -

Why? Fixes native memory corruption (which did not raise exception, but SIGSEV/SIGABORT). The visible symptom is a crash somewhere unrelated, a hang, or wrong output.

- -
- - -
10
-
The native section concept
-

A critical section protects data from other threads. This protects a stretch of time on one thread: the gap between a call returning and its error being read. A finalizer for an unrelated object, running in that gap, destroys the message.

- - - -
20
-
Closing something another thread is using memory
-

A close() on one thread frees a handle another thread has already passed into a native call. Holding a lock across the call deadlocks, because those calls run your own code.

- - - -
30
-
The signer callback freed mid-signature memory
-

What gets freed is a function pointer Python created, which native calls through while signing. On main, the file gets signed without the signer ever running, and the call reports success.

- - -
- -

Code references are to src/c2pa/c2pa.py; the “before” quotes are from git show main:src/c2pa/c2pa.py.

- -
- - diff --git a/demo/style.css b/demo/style.css deleted file mode 100644 index 1f9727af..00000000 --- a/demo/style.css +++ /dev/null @@ -1,354 +0,0 @@ -:root { - color-scheme: light dark; - --bg: #fbfaf8; - --fg: #1c1a17; - --muted: #5d5750; - --rule: #ddd7cf; - --card: #ffffff; - --code-bg: #f4f1ec; - --accent: #b3261e; - --accent-soft: rgba(179, 38, 30, 0.12); - --ok: #1c6b4a; - --ok-soft: rgba(28, 107, 74, 0.12); -} - -@media (prefers-color-scheme: dark) { - :root:not([data-theme="light"]) { - --bg: #16151a; - --fg: #eae7e2; - --muted: #a49e97; - --rule: #35323a; - --card: #1e1d23; - --code-bg: #232228; - --accent: #ff8f84; - --accent-soft: rgba(255, 143, 132, 0.16); - --ok: #6cc79b; - --ok-soft: rgba(108, 199, 155, 0.16); - } -} - -:root[data-theme="dark"] { - --bg: #16151a; - --fg: #eae7e2; - --muted: #a49e97; - --rule: #35323a; - --card: #1e1d23; - --code-bg: #232228; - --accent: #ff8f84; - --accent-soft: rgba(255, 143, 132, 0.16); - --ok: #6cc79b; - --ok-soft: rgba(108, 199, 155, 0.16); -} - -* { box-sizing: border-box; } - -body { - margin: 0; - background: var(--bg); - color: var(--fg); - font: 16px/1.65 -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif; - -webkit-font-smoothing: antialiased; -} - -.wrap { - max-width: 46rem; - margin: 0 auto; - padding: 3rem 1.25rem 6rem; -} - -.crumb { - font-size: 0.8rem; - color: var(--muted); - margin-bottom: 2rem; - letter-spacing: 0.02em; -} - -.crumb a { color: var(--muted); } - -h1 { - font-size: 1.95rem; - line-height: 1.2; - margin: 0 0 0.4rem; - letter-spacing: -0.02em; -} - -.standfirst { - font-size: 1.08rem; - color: var(--muted); - margin: 0 0 2.6rem; - line-height: 1.55; -} - -h2 { - font-size: 1.18rem; - margin: 3rem 0 0.9rem; - padding-top: 1.4rem; - border-top: 1px solid var(--rule); - letter-spacing: -0.01em; -} - -h3 { - font-size: 1rem; - margin: 2rem 0 0.6rem; -} - -p { margin: 0 0 1rem; } - -a { color: inherit; text-decoration-color: var(--rule); text-underline-offset: 2px; } -a:hover { text-decoration-color: currentColor; } - -code { - font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; - font-size: 0.87em; - background: var(--code-bg); - padding: 0.1em 0.34em; - border-radius: 3px; -} - -pre { - background: var(--code-bg); - border: 1px solid var(--rule); - border-radius: 6px; - padding: 0.9rem 1rem; - overflow-x: auto; - margin: 0 0 1rem; -} - -pre code { background: none; padding: 0; font-size: 0.8rem; line-height: 1.55; } - -.filename { - font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; - font-size: 0.74rem; - color: var(--muted); - margin-bottom: 0.35rem; - letter-spacing: 0.01em; -} - -figure { margin: 2rem 0; } - -figure svg { - display: block; - width: 100%; - max-width: 100%; - height: auto; - color: var(--fg); -} - -figcaption { - font-size: 0.85rem; - color: var(--muted); - margin-top: 0.85rem; - line-height: 1.5; -} - -ol, ul { margin: 0 0 1rem; padding-left: 1.4rem; } -li { margin-bottom: 0.5rem; } - -.steps { counter-reset: step; list-style: none; padding-left: 0; } - -.steps li { - counter-increment: step; - position: relative; - padding-left: 2.1rem; - margin-bottom: 0.8rem; -} - -.steps li::before { - content: counter(step); - position: absolute; - left: 0; - top: 0.08rem; - width: 1.45rem; - height: 1.45rem; - border-radius: 50%; - background: var(--code-bg); - border: 1px solid var(--rule); - color: var(--muted); - font-size: 0.76rem; - font-weight: 600; - display: flex; - align-items: center; - justify-content: center; -} - -.steps li.bad::before { - background: var(--accent-soft); - border-color: var(--accent); - color: var(--accent); -} - -.note { - border-left: 3px solid var(--rule); - padding: 0.15rem 0 0.15rem 1rem; - margin: 1.5rem 0; - color: var(--muted); - font-size: 0.94rem; -} - -.note.warn { border-left-color: var(--accent); } -.note strong { color: var(--fg); } - -.tests { list-style: none; padding-left: 0; } - -.tests li { - padding: 0.6rem 0; - border-bottom: 1px solid var(--rule); - font-size: 0.93rem; -} - -.tests li:first-child { border-top: 1px solid var(--rule); } - -.tests .tname { - font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; - font-size: 0.8rem; - display: block; - margin-bottom: 0.15rem; -} - -.tests .twhat { color: var(--muted); font-size: 0.88rem; } - -table { border-collapse: collapse; width: 100%; font-size: 0.9rem; margin: 0 0 1rem; } -th, td { text-align: left; padding: 0.55rem 0.7rem 0.55rem 0; border-bottom: 1px solid var(--rule); vertical-align: top; } -th { font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.06em; color: var(--muted); font-weight: 600; } - -.scroll { overflow-x: auto; } - -.pagenav { - display: flex; - justify-content: space-between; - gap: 1rem; - margin-top: 4rem; - padding-top: 1.4rem; - border-top: 1px solid var(--rule); - font-size: 0.9rem; -} - -.pagenav a { color: var(--muted); } -.pagenav a:hover { color: var(--fg); } - -/* index */ -.cards { display: grid; gap: 0; margin-top: 2rem; } - -.card { - display: block; - padding: 1.3rem 0; - border-top: 1px solid var(--rule); - text-decoration: none; - color: inherit; -} - -.card:last-child { border-bottom: 1px solid var(--rule); } -.card:hover .card-title { text-decoration: underline; text-underline-offset: 3px; } - -.card-num { - font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; - font-size: 0.74rem; - color: var(--muted); -} - -.card-title { font-size: 1.05rem; font-weight: 600; margin: 0.2rem 0 0.35rem; } -.card-desc { font-size: 0.92rem; color: var(--muted); margin: 0; line-height: 1.55; } - -.tag { - display: inline-block; - font-size: 0.7rem; - letter-spacing: 0.04em; - text-transform: uppercase; - padding: 0.15rem 0.45rem; - border-radius: 3px; - border: 1px solid var(--rule); - color: var(--muted); - margin-left: 0.5rem; - vertical-align: 0.1rem; -} - -.tag.crash { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); } - -/* diagram-first page format */ -.diagrams { margin: 2.5rem 0 0; } - -.panel { margin: 0 0 2.6rem; } - -.panel-label { - display: flex; - align-items: baseline; - gap: 0.6rem; - margin-bottom: 0.7rem; -} - -.panel-tag { - font-size: 0.7rem; - letter-spacing: 0.08em; - text-transform: uppercase; - font-weight: 700; - padding: 0.18rem 0.5rem; - border-radius: 3px; -} - -.panel-tag.before { color: var(--accent); background: var(--accent-soft); } -.panel-tag.after { color: var(--ok); background: var(--ok-soft); } - -.panel-claim { font-size: 0.95rem; color: var(--muted); } - -.panel figure { margin: 0; } -.panel figcaption { margin-top: 0.6rem; } - -.footnote { - margin-top: 3rem; - padding-top: 1.4rem; - border-top: 1px solid var(--rule); - font-size: 0.92rem; - color: var(--muted); -} - -.footnote p { margin: 0 0 0.7rem; } -.footnote strong { color: var(--fg); } -.footnote code { font-size: 0.85em; } - -.refs { - margin-top: 1.2rem; - font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; - font-size: 0.74rem; - color: var(--muted); - line-height: 1.9; -} - -.panel-tag { color: var(--muted); background: var(--code-bg); } - -.panel-note { - margin-top: 1.1rem; - font-size: 0.92rem; - color: var(--muted); - line-height: 1.6; -} - -.panel-note p { margin: 0 0 0.8rem; } -.panel-note p:last-child { margin-bottom: 0; } -.panel-note strong { color: var(--fg); } - -.panel-note pre { - margin: 0.9rem 0; - background: var(--code-bg); -} - -.bridge { - margin: 0 0 2.6rem; - padding-left: 1rem; - border-left: 3px solid var(--rule); - font-size: 0.94rem; - color: var(--muted); - line-height: 1.6; -} - -.panel-tag.also { color: var(--muted); background: var(--code-bg); } - -.footnote h2 { - font-size: 0.78rem; - text-transform: uppercase; - letter-spacing: 0.08em; - color: var(--muted); - margin: 0 0 0.9rem; - padding: 0; - border: 0; - font-weight: 600; -} diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index a5c0d86a..92d2cc21 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -160,11 +160,13 @@ In every case, the process terminates from inside native code. No exception, no ## Locking -Each `ManagedResource` holds a reentrant lock, `_op_lock`, and the in-flight counters from [The full picture](#the-full-picture). Together they enforce that rule: a mutating call excludes every other call, and a `close()` arriving mid-call is deferred rather than applied immediately. +Freeing the same native pointer twice corrupts the allocator's bookkeeping. A detecting allocator stops the process. An allocator that misses it lets the damage surface later, somewhere unrelated to the code responsible. Two threads racing a `close()` against an in-flight call on the same object is one way this happens: without a guard, thread B's `close()` could free the pointer while thread A's call is still reading through it, and if thread A's own cleanup runs afterward, that same pointer gets freed a second time. + +Each `ManagedResource` holds a reentrant lock, `_op_lock`, and the in-flight counters from [Lifecycle overview](#lifecycle-overview). Together they guard against that: a mutating call excludes every other call, and a `close()` arriving mid-call is deferred rather than applied immediately. `_op_lock` is a `threading.RLock`, reentrant, rather than a plain `Lock`. A finalizer can run at any point, including inside a method that already holds the lock on that thread, so `__del__` calling back into locked code must not deadlock against itself. And a consuming call tears the handle down from inside a region it already holds the lock in, so it needs to reacquire rather than block. -The lock is never held across a native call that drives a stream callback, since that callback can call back into this API on the same thread, and holding the lock there would deadlock against that reentry. Those calls increment an in-flight counter under the lock, release the lock, run the native call, then decrement the counter, which is the mechanism the [state diagram](#the-full-picture) describes as "a call in flight." +The lock is never held across a native call that drives a stream callback, since that callback can call back into this API on the same thread, and holding the lock there would deadlock against that reentry. Those calls increment an in-flight counter under the lock, release the lock, run the native call, then decrement the counter, which is the mechanism the [state diagram](#lifecycle-overview) describes as "a call in flight." This is why the interpreter's Global Interpreter Lock, the GIL, does not make this safe on its own. CPython executes one bytecode instruction at a time under the GIL, so simple operations cannot corrupt a built-in container. But a foreign function call through ctypes releases the GIL for its duration, so another thread runs while native code runs, and a `close()` can land inside that window. `_op_lock` and the in-flight counters avoids this case. @@ -184,21 +186,9 @@ A shared call, a **borrow**, passes the handle to native and gets it back unchan `ManagedResource` prevents this by refusing to start a consume while a borrow is in flight, and by reserving the handle as a mutating call for the whole duration of the consume, the same reservation any other mutating call makes. Every entry point checks the in-flight counters under `_op_lock` before it starts, so a call that would otherwise race a consume is refused instead with "running a mutating operation," and the resource's lifecycle state never changes until the consume is fully classified as a success or a failure. `Reader.with_fragment()` also serializes itself against other calls to itself with a lock of its own. -## Double frees - -Freeing the same native pointer twice corrupts the allocator's bookkeeping. A detecting allocator stops the process. An allocator that misses it lets the damage surface later, somewhere unrelated to the code responsible. - -| Risk | Mitigated by | -| --- | --- | -| Freeing a pointer a consuming call already took | The consumed pointer is abandoned rather than freed; the native error message says whether ownership actually moved (see [Consuming](#consuming)). | -| A forked child process freeing a pointer its parent owns | A process-ID stamp on every object; cleanup in a process that did not allocate the pointer marks it closed without freeing (see [Fork safety](#fork-safety)). | -| Two threads racing a `close()` against an in-flight call on the same object | `_op_lock` and the in-flight counters (see [Locking](#locking)): a `close()` mid-call is deferred, not applied. | - -Sharing one `ManagedResource` instance across threads needs these guards. Nothing here protects two threads racing on distinct objects that happen to share an allocator, since that is the allocator's own concurrency guarantee, not this layer's. - ## Consuming -Consuming a handle is a mutating call that hands the pointer to native, which takes ownership and either returns a replacement pointer or frees the original. Python must never free a pointer it handed to a consuming call, whichever way the call ends, since the address may belong to a different object by the time it returns. +Consuming a handle is a mutating call that hands the pointer to native, which takes ownership and either returns a replacement pointer or frees the original. Python must never free a pointer it handed to a consuming call, whichever way the call ends, since the address may belong to a different object by the time it returns. Freeing a pointer a consuming call already took would double-free it, so the consumed pointer is abandoned rather than freed. There are two shapes a consuming call takes. @@ -220,36 +210,7 @@ On success the object stays `ACTIVE`: the lifecycle state never changes, only th **Consume-and-close** takes the pointer and leaves the Python object nothing to wrap. Signing a `Builder`, or handing a `Signer` to a `Context`, both end this way: the object goes `CLOSED`, but without freeing the pointer, since native still owns it. -A consuming FFI call can fail two different ways that return the identical value to Python, a null pointer or a non-zero status: it can reject the pointer before taking ownership, or take ownership and then drop the value on a later failure. Which one happened decides whether Python still owns the pointer. - -```mermaid -flowchart TD - CALL["FFI call(handle)"] --> V{"validate arguments,
then the handle"} - V -->|invalid| R["reject: handle NOT taken"] --> F1["returns a failure value
(null, or non-zero status)"] - V -->|valid| TAKE["take ownership of handle"] - TAKE --> WORK{"execute function logic"} - WORK -->|fails| DROP["native drops the value itself"] --> F2["returns a failure value
(null, or non-zero status)"] - WORK -->|succeeds| OK["returns replacement / 0 / new pointer"] - - F1 -.same value.- AMB(["Python must read the native error
to tell these apart"]) - F2 -.same value.- AMB -``` - -The native error message tells the two apart. A set of rejection tags means the handle was never taken, so Python keeps and later frees it normally. Any other error means native took it and dropped it, so Python's cleanup runs without freeing anything. If no error was set, ownership is unknown, and Python falls back to a **guarded free**: the native pointer registry rejects a free of an address it no longer tracks, returning `-1` rather than crashing, so this is harmless whether or not native still holds it. - -The one case a guarded free is not used is once ownership is known to have moved: there, the free is skipped rather than issued and left for the registry to reject. A freed address can be reused by an unrelated allocation, and an unnecessary free landing after that reuse would destroy the wrong object. Skipping a free that provides no value avoids that window. - -Three helpers share this triage, differing only in what a successful call returns: - -| Helper | On success | Result | -| --- | --- | --- | -| `_consume_and_swap()` | a replacement pointer | installs the replacement; resource stays `ACTIVE` | -| `_consume_no_replacement()` | a status code | `CLOSED`, pointer not freed | -| `_consume_into()` | a *different* object's pointer | `CLOSED`; the new pointer is handed to the caller to own | - -`_consume_no_replacement()` is how a `Signer` is fed to a `Context`, and `_consume_into()` is how that same `Context` returns its newly built native pointer. - -### Signer to context +### Example: Transferring a Signer to a Context The transfer of a `Signer` into a `Context` shows the whole protocol in one place: the reservation that protects the handle during the call, and the triage that decides ownership afterward. @@ -273,14 +234,8 @@ sequenceDiagram alt status 0 (success) S->>S: close, pointer not freed (native took it) else non-zero status - S->>S: read the native error - alt handle was never taken - S->>S: raise, handle retained and still usable - else native took it, then failed - S->>S: close, pointer not freed - else no error set - S->>S: guarded free (real free if still ours,
no-op if native took it) - end + S->>S: guarded free (real free if still ours,
no-op if native took it) + S->>S: raise end X->>B: build the context from the consumed builder @@ -289,7 +244,7 @@ sequenceDiagram X->>X: activate the new Context ``` -A few details in that sequence matter beyond what the diagram shows. The transfer is not wrapped in a shared-call reservation; it uses the mutating reservation described in [Borrowing vs consuming](#borrowing-vs-consuming), which defers a racing `signer.close()` until the transfer is classified. `set_signer` does not always take the pointer, so the triage reads the native error before deciding whether the Signer closed. The temporary native builder used to construct the Context is itself a small `ManagedResource`, held inside a `with` block, so any failure along the way frees it through the same `close()` path rather than a bespoke handler. +The transfer is not wrapped in a shared-call reservation. It uses the mutating reservation described in [Borrowing vs consuming](#borrowing-vs-consuming), which defers a racing `signer.close()` until the transfer is classified.The temporary native builder used to construct the Context is itself a small `ManagedResource`, held inside a `with` block, so any failure along the way frees it through the same `close()` path rather than a bespoke handler. ### Adopting a handle @@ -299,7 +254,7 @@ Ownership can also arrive from the other direction: a native call returns a poin ## Guarantees -`is_valid`, defined in [The full picture](#the-full-picture), is a lock-free snapshot: it does not itself keep the handle alive, only a guarded call does that. `Context` implements the abstract `ContextProvider.is_valid` by inheriting the concrete one from `ManagedResource`, which Python's method resolution order finds first as long as `ManagedResource` is listed before `ContextProvider` in the class definition (`class Context(ManagedResource, ContextProvider)`). Listing them the other way around would leave the abstract declaration in front and raise `TypeError` at class definition time. +`is_valid`, defined in [Lifecycle overview](#lifecycle-overview), is a lock-free snapshot: it does not itself keep the handle alive, only a guarded call does that. `Context` implements the abstract `ContextProvider.is_valid` by inheriting the concrete one from `ManagedResource`, which Python's method resolution order finds first as long as `ManagedResource` is listed before `ContextProvider` in the class definition (`class Context(ManagedResource, ContextProvider)`). Listing them the other way around would leave the abstract declaration in front and raise `TypeError` at class definition time. Every subclass gets these guarantees from `ManagedResource`, and must not break them: @@ -369,7 +324,7 @@ The "foreign process" branch is explained in [Fork safety](#fork-safety), next. If a forked child cleaned up its copy of that object normally, two things would go wrong. It would free a pointer the parent is using, a double-free. And more subtly, `fork()` only carries over the thread that called it, so if another thread held a native lock at fork time, that lock stays held forever in the child, and calling into native code to free anything can then block on it forever. -So the SDK never frees native memory in a process that did not allocate it. Every object is stamped with its creating process's ID at construction, and cleanup compares that stamp against the current process before doing anything: +So the SDK never frees native memory in a process that did not allocate it. A process-ID stamp on every object guards this: cleanup in a process that did not allocate the pointer marks it closed without freeing. Every object is stamped with its creating process's ID at construction, and cleanup compares that stamp against the current process before doing anything: ```mermaid sequenceDiagram @@ -455,7 +410,7 @@ A `with_fragment()` call does two things: the FFI call consumes the Reader's cur - `Reader is already processing a fragment on another thread`: another thread is already inside `with_fragment()` on this Reader. - `Reader is in use by another operation and cannot be consumed`: some other native call is in flight on this Reader. -- `Reader is running a mutating operation`: what a read method on another thread sees while `with_fragment()` runs, per [The full picture](#the-full-picture). +- `Reader is running a mutating operation`: what a read method on another thread sees while `with_fragment()` runs, per [Lifecycle overview](#lifecycle-overview). `resource_to_stream()`, by contrast, is a shared call: the underlying SDK method only reads, so other read methods on the same Reader run concurrently with it, a `close()` arriving meanwhile is deferred until it returns, and `with_fragment()` is refused until it returns, the same as any other mutating call. From f3cf3b783575c62f31d4c2315cb4b326a6466680 Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 20:41:23 -0700 Subject: [PATCH 07/11] fix: Shorten doc --- docs/native-resources-management.md | 40 ++++++++--------------------- 1 file changed, 11 insertions(+), 29 deletions(-) diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index 92d2cc21..980d35d6 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -84,7 +84,7 @@ stateDiagram-v2 Once `CLOSED`, an object never becomes `ACTIVE` again. A construction that fails before activation can also close directly from `UNINITIALIZED`, since there is nothing to free, only a state to record. -## Close during a call +## Closing during a (native) call A native call takes several steps in sequence: check the object is usable, hand the pointer to native code, let the code run. Two threads sharing one object can interleave those steps: @@ -178,7 +178,7 @@ Using a handle takes two steps: check it, then act on it. Native bookkeeping gua ### Lock ordering -Two threads acquiring the same pair of locks in opposite orders can deadlock, each waiting on what the other holds. The SDK avoids this with one fixed order, from outermost to innermost: a method-specific lock (where a method serializes itself against other calls to itself), then the lock of the object a method is called on, then the lock of any object it borrows for the call. A borrowed object's lock is always taken inside the operating object's lock, never the reverse, and a lock held across a native call must be one no callback path acquires. +Two threads acquiring the same pair of locks in opposite orders can deadlock, each waiting on what the other holds. This Python SDK avoids this with one fixed order, from outermost to innermost: a method-specific lock (where a method serializes itself against other calls to itself), then the lock of the object a method is called on, then the lock of any object it borrows for the call. A borrowed object's lock is always taken inside the operating object's lock, never the reverse, and a lock held across a native call must be one no callback path acquires. ### Borrowing vs consuming @@ -248,11 +248,11 @@ The transfer is not wrapped in a shared-call reservation. It uses the mutating r ### Adopting a handle -Ownership can also arrive from the other direction: a native call returns a pointer that needs a Python wrapper around it, with no `__init__` call, since `__init__` would try to create a new native resource rather than wrap an existing one. `_wrap_native_handle()` handles this: it builds a bare instance, sets its lifecycle bookkeeping, runs `_init_attrs()` for subclass defaults, and activates the handle. Ownership transfers once that call returns; if it raises, no wrapper exists, and the caller still owns the pointer and must free it itself. +A native call can return a pointer that needs a Python wrapper around it, with no `__init__` call, since `__init__` would try to create a new native resource rather than wrap an existing one. `_wrap_native_handle()` handles this: it builds a bare instance, sets its lifecycle bookkeeping, runs `_init_attrs()` for subclass defaults, and activates the handle. Ownership transfers once that call returns; if it raises, no wrapper exists, and the caller still owns the pointer and must free it itself. -`Reader._init_from_context` and `Builder._init_from_context` both do something that looks backward: they create a native object and activate it before making the consuming call that will feed it data. This is deliberate. A consuming call needs an active resource to read the handle from and swap the result into, and activating first puts the intermediate pointer under normal cleanup right away: whichever way the consuming call goes, `close()` and `__del__` free it correctly. Holding the raw pointer in a local variable instead would leave every failure path to decide for itself whether to free it. +`Reader._init_from_context` and `Builder._init_from_context` both do something that looks backward: they create a native object and activate it before making the consuming call that will feed it data. A consuming call needs an active resource to read the handle from and swap the result into, and activating first puts the intermediate pointer under normal cleanup right away: whichever way the consuming call goes, `close()` and `__del__` free it correctly. Holding the raw pointer in a local variable instead would leave failure paths to decide whether to free it. -## Guarantees +## Object usability checks `is_valid`, defined in [Lifecycle overview](#lifecycle-overview), is a lock-free snapshot: it does not itself keep the handle alive, only a guarded call does that. `Context` implements the abstract `ContextProvider.is_valid` by inheriting the concrete one from `ManagedResource`, which Python's method resolution order finds first as long as `ManagedResource` is listed before `ContextProvider` in the class definition (`class Context(ManagedResource, ContextProvider)`). Listing them the other way around would leave the abstract declaration in front and raise `TypeError` at class definition time. @@ -268,9 +268,9 @@ Every subclass gets these guarantees from `ManagedResource`, and must not break ## Keeping references alive -When a Python object passes a callback or a pointer to the native library, that reference must stay alive as long as native code might use it, and the garbage collector has no way to know that: it only sees Python references. +When a Python object passes a callback or a pointer to the native library, that reference must stay alive as long as native code might use it. But the garbage collector has no way to know that: it only sees Python references. -The SDK keeps these references as plain instance attributes on the owning object. A `Stream` stores its four callback objects this way, so they stay referenced as long as the `Stream` is alive (see [Reference cycles](#reference-cycles) for how those callbacks avoid keeping the `Stream` alive in return). A `Signer` consumed by a `Context` has its callback copied to an attribute on the `Context`, so the callback survives the `Signer` object closing. +This Python SDK keeps these references as plain instance attributes on the owning object. A `Stream` stores its four callback objects this way, so they stay referenced as long as the `Stream` is alive (see [Reference cycles](#reference-cycles) for how those callbacks avoid keeping the `Stream` alive in return). A `Signer` consumed by a `Context` has its callback copied to an attribute on the `Context`, so the callback survives the `Signer` object closing. `_release()` sets these attributes to `None` during cleanup, letting them be collected, and it runs before the native pointer is freed, so anything the pointer depends on, an open file, a stream wrapper, is torn down first. This is `ManagedResource`'s ordering; `Stream` releases in the reverse order for reasons covered in [`Stream` cleanup](#stream-cleanup). @@ -284,7 +284,7 @@ def _free_native_ptr(ptr): return _lib.c2pa_free(ptr) ``` -It returns `0` when the pointer was freed, and `-1` when the registry rejected an already-consumed or untracked address, the same `-1` a guarded free relies on. `ManagedResource` guarantees this is called once per pointer. +It returns `0` when the pointer was freed, and `-1` when the registry rejected an already-consumed or untracked address. `ManagedResource` guarantees this is called once per pointer. ## Cleanup errors @@ -316,15 +316,13 @@ flowchart TD H -->|yes| FREE["free the native pointer
(logs on failure)"] --> DONE ``` -The "foreign process" branch is explained in [Fork safety](#fork-safety), next. - ## Fork safety `fork()` copies the calling process, including every Python object holding a native pointer, but the underlying native allocation is not duplicated: there is still only one, and the parent owns it. -If a forked child cleaned up its copy of that object normally, two things would go wrong. It would free a pointer the parent is using, a double-free. And more subtly, `fork()` only carries over the thread that called it, so if another thread held a native lock at fork time, that lock stays held forever in the child, and calling into native code to free anything can then block on it forever. +If a forked child cleaned up its copy of that object normally, two things would go wrong. It would free a pointer the parent is using, a double-free. And `fork()` copies only the calling thread, not every thread the parent was running. Any lock another thread held at fork time comes over still locked, with no thread left in the child able to release it. If that lock happens to be one the native library uses internally, calling into native code to free anything in the child can then block forever. -So the SDK never frees native memory in a process that did not allocate it. A process-ID stamp on every object guards this: cleanup in a process that did not allocate the pointer marks it closed without freeing. Every object is stamped with its creating process's ID at construction, and cleanup compares that stamp against the current process before doing anything: +So this Python SDK never frees native memory in a process that did not allocate it. A process-ID stamp on every object guards this: cleanup in a process that did not allocate the pointer marks it closed without freeing. Every object is stamped with its creating process's ID at construction, and cleanup compares that stamp against the current process before doing anything: ```mermaid sequenceDiagram @@ -402,23 +400,7 @@ Cleanup runs in the direction of dependency: whatever can invoke or reach the ot Each ctypes callback closes over the `Stream` it belongs to. Captured directly, that would be a reference cycle, the `Stream` holding the callback and the callback holding the `Stream`, so nothing in the loop would reach a zero reference count on its own, leaving cleanup to the slower cycle collector. The callbacks capture a weak reference instead, resolved fresh on each call, so the `Stream`'s reference count can reach zero and its cleanup stays on the deterministic path. -### `Reader.with_fragment()` - -A `with_fragment()` call does two things: the FFI call consumes the Reader's current handle and returns a replacement, and the Reader updates its own Python-side fields, the `Stream` wrappers it owns and its manifest caches, to describe that new handle. Those two steps have to run as one unit, since two threads interleaving them could close a stream the native reader is still reading through, or leave stale wrappers describing a handle another thread already replaced. - -`Reader._fragment_lock` covers both steps, and `with_fragment()` is the only method that takes it. The guard spans the native call, so it cannot block waiting for the lock, since that call drives caller-supplied callbacks that could re-enter this API and deadlock. A second thread finding the lock held is refused at once instead, leaving the Reader untouched: no stream built, no handle consumed, so the call succeeds once the other thread returns. This means one thread feeds fragments to a Reader at a time, and the caller must serialize those calls itself. The refusal messages are stable enough for callers to match on: - -- `Reader is already processing a fragment on another thread`: another thread is already inside `with_fragment()` on this Reader. -- `Reader is in use by another operation and cannot be consumed`: some other native call is in flight on this Reader. -- `Reader is running a mutating operation`: what a read method on another thread sees while `with_fragment()` runs, per [Lifecycle overview](#lifecycle-overview). - -`resource_to_stream()`, by contrast, is a shared call: the underlying SDK method only reads, so other read methods on the same Reader run concurrently with it, a `close()` arriving meanwhile is deferred until it returns, and `with_fragment()` is refused until it returns, the same as any other mutating call. - -#### Cache invalidation - -A `Reader` caches its manifest data, keyed to the handle it was read from. A successful `with_fragment()` swap invalidates that cache, and both the invalidation and every cache read in `json()` happen under the resource's lock, so a reader never observes a half-updated cache. The handle swap runs as a shared-style in-flight call rather than under that lock (native calls never hold it, per [Locking](#locking)), so there is a window between the new handle landing and the old cache being cleared. A `json()` on another thread that acquires the lock inside that window finds the stale cache populated and returns it without consulting the handle, serving a manifest for the fragment just replaced. `_fragment_lock` does not close this window, since it only serializes `with_fragment()` against itself; a caller sharing one Reader across threads for both fragment-feeding and reading needs to serialize those two together itself. - -## Which method +## Methods to use with a `ManagedResource` Writing a new `ManagedResource` subclass, each situation maps to one call: From b7207b27f7d6a6911b5e1d8c0883401763ef7e27 Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 20:45:56 -0700 Subject: [PATCH 08/11] fix: Shorten doc --- docs/native-resources-management.md | 54 ++--------------------------- 1 file changed, 3 insertions(+), 51 deletions(-) diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index 980d35d6..e1e2f366 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -126,15 +126,15 @@ stateDiagram-v2 Idle --> SharedBorrow: a shared call starts SharedBorrow --> Idle: it returns Idle --> Mutating: a mutating call starts - Mutating --> Idle: it returns - Mutating --> [*]: a consume succeeds + Mutating --> Idle: it returns, or a consume-and-swap succeeds + Mutating --> [*]: a consume-and-close succeeds } ACTIVE --> CLOSED: close() / __del__ CLOSED --> [*] ``` -The `Mutating --> [*]` exit inside `ACTIVE` is a consuming call: one that hands the pointer to native and gets a replacement or a closed resource back, covered in [Consuming](#consuming). +`Mutating` is entered by any mutating call, but the two ways out differ by what kind of call it was. An ordinary mutating call, or a consume-and-swap, returns to `Idle`: the resource stays `ACTIVE`, either unchanged or holding a new pointer. Only a consume-and-close exits `ACTIVE` into `CLOSED`, since that is the one kind of consuming call that leaves nothing to wrap. Both consuming shapes are covered in [Consuming](#consuming). | State | `is_valid` | What a caller sees | | --- | --- | --- | @@ -194,58 +194,10 @@ There are two shapes a consuming call takes. **Consume-and-swap** replaces the object's internal state without discarding the Python-side wrapper. `Reader.with_fragment()` does this, feeding a new BMFF fragment into an existing Reader so the native library can rebuild its internal representation from prior fragments plus the new one, since a fresh `Reader` would lose that accumulated state. `Builder.with_archive()` does the same, loading an archive into an existing Builder while keeping its context and settings. -```mermaid -stateDiagram-v2 - state "ACTIVE (ptr A)" as A - state "ACTIVE (ptr B)" as B - - A --> B : consuming call swaps ptr A for ptr B - note right of B - Same Python object, - new native pointer - end note -``` - On success the object stays `ACTIVE`: the lifecycle state never changes, only the pointer underneath it, and callers keep using the same object. **Consume-and-close** takes the pointer and leaves the Python object nothing to wrap. Signing a `Builder`, or handing a `Signer` to a `Context`, both end this way: the object goes `CLOSED`, but without freeing the pointer, since native still owns it. -### Example: Transferring a Signer to a Context - -The transfer of a `Signer` into a `Context` shows the whole protocol in one place: the reservation that protects the handle during the call, and the triage that decides ownership afterward. - -```mermaid -sequenceDiagram - participant C as Caller - participant S as Signer - participant X as Context - participant B as _NativeBuilder - participant N as Native lib - - C->>X: Context(settings, signer) - X->>B: with _NativeBuilder() (owns the builder, close() frees it on any failure) - X->>X: copy signer's callback to the Context - Note right of X: Copy it before the transfer:
a successful consume drops the Signer's own reference - X->>S: _consume_no_replacement(set_signer) - S->>S: reserve the handle as a mutating call - Note right of S: The reservation is the protection:
a close() on another thread is deferred,
and other calls are refused - S->>N: c2pa_context_builder_set_signer(builder_ptr, handle) - - alt status 0 (success) - S->>S: close, pointer not freed (native took it) - else non-zero status - S->>S: guarded free (real free if still ours,
no-op if native took it) - S->>S: raise - end - - X->>B: build the context from the consumed builder - B->>N: c2pa_context_builder_build(builder_ptr) - N-->>X: context_ptr - X->>X: activate the new Context -``` - -The transfer is not wrapped in a shared-call reservation. It uses the mutating reservation described in [Borrowing vs consuming](#borrowing-vs-consuming), which defers a racing `signer.close()` until the transfer is classified.The temporary native builder used to construct the Context is itself a small `ManagedResource`, held inside a `with` block, so any failure along the way frees it through the same `close()` path rather than a bespoke handler. - ### Adopting a handle A native call can return a pointer that needs a Python wrapper around it, with no `__init__` call, since `__init__` would try to create a new native resource rather than wrap an existing one. `_wrap_native_handle()` handles this: it builds a bare instance, sets its lifecycle bookkeeping, runs `_init_attrs()` for subclass defaults, and activates the handle. Ownership transfers once that call returns; if it raises, no wrapper exists, and the caller still owns the pointer and must free it itself. From fda1fdd1809cb8aa8b5458e4fa3835aeca2f1420 Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 20:49:29 -0700 Subject: [PATCH 09/11] fix: Shorten doc --- docs/native-resources-management.md | 42 +++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index e1e2f366..e3f274f1 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -198,6 +198,48 @@ On success the object stays `ACTIVE`: the lifecycle state never changes, only th **Consume-and-close** takes the pointer and leaves the Python object nothing to wrap. Signing a `Builder`, or handing a `Signer` to a `Context`, both end this way: the object goes `CLOSED`, but without freeing the pointer, since native still owns it. +### Example: signing a Builder + +`Builder.sign()` is a consume-and-close call: + +```python +builder = Builder(manifest_json) +builder.sign(signer, "image/jpeg", source, dest) +# builder is now CLOSED: sign() consumed its handle +``` + +`sign()` hands the Builder's handle to `c2pa_builder_sign`, which takes ownership and writes the signed asset. There is no replacement pointer to install, so `ManagedResource` marks the Builder `CLOSED` without freeing anything: native already owns the pointer at that point. This is also why a `Builder` is single-use. Calling `sign()` again raises `C2paError`, since `is_valid` is now False. + +Each call below is a real `ManagedResource` method, in the order `sign()` calls them: + +```mermaid +sequenceDiagram + participant C as Caller + participant B as Builder + + C->>B: sign(signer, format, source, dest) + B->>B: _ensure_valid_state() + Note right of B: Raises C2paError if not ACTIVE.
Builder is still ACTIVE here. + + B->>B: _exclusive_native_call() + Note right of B: Reserves the handle as a mutating call.
Builder is ACTIVE, mutating. + + B->>B: c2pa_builder_sign(handle, ...) + Note right of B: Native call runs.
On return, native owns the handle
whether this succeeded or raised. + + alt native call raised + B->>B: close() + Note right of B: Builder is now CLOSED. + B-->>C: re-raises as C2paError + else native call returned + B->>B: close() + Note right of B: Builder is now CLOSED,
unconditionally, in a finally block. + B-->>C: returns manifest bytes + end +``` + +Both branches end the same way: `close()` runs either way, so the Builder is always `CLOSED` once `sign()` returns or raises. Nothing about the outcome changes whether `close()` frees the pointer, since native already took it in `_exclusive_native_call()`'s reservation. + ### Adopting a handle A native call can return a pointer that needs a Python wrapper around it, with no `__init__` call, since `__init__` would try to create a new native resource rather than wrap an existing one. `_wrap_native_handle()` handles this: it builds a bare instance, sets its lifecycle bookkeeping, runs `_init_attrs()` for subclass defaults, and activates the handle. Ownership transfers once that call returns; if it raises, no wrapper exists, and the caller still owns the pointer and must free it itself. From 343e0add1a2bb45e08a74ec273bc2fccb38b18d2 Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 21:15:22 -0700 Subject: [PATCH 10/11] fix: Add example in docs --- docs/native-resources-management.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index e3f274f1..c1830979 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -92,6 +92,21 @@ A native call takes several steps in sequence: check the object is usable, hand 2. While that call is still running, thread B calls `reader.close()`, which frees the native pointer. 3. Thread A's native code, still running, reads through the pointer it was given, now freed (which crashes). +Any code that hands work to a thread pool and waits on it with a timeout can hit it: + +```python +with Reader("image.jpg") as reader: + future = thread_pool.submit(reader.json) + try: + future.result(timeout=2.0) + except TimeoutError: + pass +# the `with` block exits here, calling reader.close(), +# whether or not the pool thread's reader.json() finished +``` + +`future.result(timeout=...)` gives up waiting after the timeout. It does not stop the pool thread already running `reader.json()`, so that thread can still be mid-call when the `with` block exits on the timeout path. `close()` runs regardless, on the calling thread, while `reader.json()` may still be running on the pool thread. Same race as thread A and thread B above, reached through a wait-with-timeout instead of a hand-rolled thread. + A Python object has no owning thread: it belongs to whoever holds a reference to it, and nothing about creating an object or passing it to another thread, in a closure or an argument, hands exclusive access to that thread. Both threads above hold a plain reference to the same object instance. Python lets either one call any method on it at any time. The interleaving in step 2 could be possible because the CPython interpreter switches between threads between bytecode instructions, and a native call spans many of them. Calling a method by itself does not stop thread B from calling `close()` on the same object while that call runs, so thread B's `close()` can land at any point during thread A's call, including partway through. The `ManagedResource` class has functionalities to avoid that. From 81e7f506acf1953ae3263cb12d68a44c30412b90 Mon Sep 17 00:00:00 2001 From: tmathern <60901087+tmathern@users.noreply.github.com> Date: Thu, 24 Sep 2026 21:24:01 -0700 Subject: [PATCH 11/11] fix: Add example in docs 2 --- docs/native-resources-management.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/native-resources-management.md b/docs/native-resources-management.md index c1830979..6a4706fc 100644 --- a/docs/native-resources-management.md +++ b/docs/native-resources-management.md @@ -185,6 +185,8 @@ The lock is never held across a native call that drives a stream callback, since This is why the interpreter's Global Interpreter Lock, the GIL, does not make this safe on its own. CPython executes one bytecode instruction at a time under the GIL, so simple operations cannot corrupt a built-in container. But a foreign function call through ctypes releases the GIL for its duration, so another thread runs while native code runs, and a `close()` can land inside that window. `_op_lock` and the in-flight counters avoids this case. +Free-threaded Python builds (no GIL at all) do not change this. `ctypes` is not a compiled C extension, so it is not subject to the opt-in check that silently re-enables the GIL for unmarked extensions: a native library loaded through `ctypes` runs with no GIL protection whether or not the GIL exists elsewhere in the process. A native call already ran with the GIL released, so removing the GIL entirely makes that the normal case instead of a temporary window. + The native library keeps its own bookkeeping for the pointers it hands out. That bookkeeping guards the pointer itself. It can't guard what Python does with the pointer. A callback is Python code the native library calls partway through a call, to read a stream or sign a message. It belongs to Python, so the native library keeps no bookkeeping for it. If Python frees that callback's object while the call runs, the next invocation reaches memory Python gave up. `_op_lock` and the in-flight counters keep that memory alive for the call.