Repository navigation
feat(shadow): local, opt-in detection of repeated work - #156
Merged
Merged
Conversation
Add openadapt_capture.shadow, off by default (ShadowConfig(enabled=True) in code is the only switch). Events become value-free symbols: typed text and data labels become keyed HMAC digests, secure fields are dropped, URLs are reduced to a coarse pattern, and window titles, the clipboard, and the screen are never read. Symbols live in a ring buffer capped by size and age, in memory or in an owner-only SQLite file with secure_delete. A suffix-array miner finds repeats that span at least two apps, occur N times in the window, and pass a Bonferroni-corrected Poisson chance test; near-repeats join by structural similarity. A fixed-template prompt (You did this N times today.) offers Review draft or Ignore; Ignore suppresses the motif and close variants. Review draft returns an openadapt.shadow.episodes/v1 export for Flow's induce. Nothing runs and nothing touches the network. The tray hook is an interface with a collecting stub; Desktop wiring and a live input-observer connection are not included. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
Describe what shadow mining observes and keeps, the ring buffer, the detection thresholds, the prompt, the path from draft to admitted program, and the workplace-monitoring obligations (consent, DPIA, works-council co-determination). Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
Recorder(..., live_event_tap=tap) and record(..., live_event_tap=tap) copy each retained action, window, and browser event to a LiveEventTap after it is queued for writing. Screen frames and audio are never published. The tap sees events after the authentication-handoff gate, so an interval Capture keeps out of a recording stays out of the tap too. Delivery runs on a daemon thread behind a bounded queue. publish never blocks and never raises; a full queue drops and counts the event, and a failing subscriber is logged and counted. close() returns even when the subscriber is stuck. Without a tap (the default) nothing is published. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…scope, notice Key and buffer. One random key per epoch of at most 24 hours lives in the OS credential store (keyring), with an in-memory fallback. It derives the HMAC key for typed-value and label digests and the AES-256-GCM key that seals every row of the on-disk buffer. When the epoch ends, the key is deleted and the buffer purged with it, so no digest of a short identifier (a 7 to 10 digit MRN, a 9 digit SSN) outlives the key that could test a guess. A key dated in the future is distrusted. The buffer's time cap is now at most 24 hours, and an unencrypted disk buffer is refused. Export. openadapt.shadow.episodes/v2 carries no keyed digest: each becomes a reference local to the export (label-a, value-c). Ignores are kept on coarse tokens so they still hold after rotation. Scrubbing before storage. A URL's first path segment and subdomain labels survive only when every word is a known URL word; anything else, including digit and hex runs, UUIDs, and name-like tokens, becomes <id>. A short button or menu label survives only when every word is a common UI word; anything else becomes <dynamic_label>. Work scope. ShadowConfig.work_domains is a local allowlist, empty by default. Browser activity elsewhere becomes one non-work-browser marker with no host, path, label, or value. Secret fields. Native secure roles and subroles, UIA IsPassword, input type, autocomplete hints, and names such as password, PIN, one-time code, SSN, or CVV drop the value. Typing with no accessibility evidence is treated as secret. Notice. Turning shadow mode on needs an Acknowledgement of the current notice (what is captured, the 24-hour encrypted local buffer, nothing leaves the machine, no administrator can see or enable it, one-click delete and turn off). ConsentLedger records acknowledgement and revocation times locally. purge_and_disable deletes the buffer file, the key, and the ignore list. Live hook. RecorderShadowBridge maps the recorder's live tap events (clicks, key presses, window focus, browser events) onto ShadowEvents. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…live tap Update SHADOW_MINING.md for the acknowledged notice and consent ledger, purge and disable, the 24-hour key in the OS credential store and its destruction with the buffer, the encrypted buffer, URL and label scrubbing with examples, the work-domain allowlist, secret-field detection, the v2 export without digests, and the recorder's live tap. Update COMPLIANCE.md and the README to match, and list what is still not done. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…pect it The notice claimed that no administrator could see the buffer. That isn't true of someone with administrator access to the computer, or of software running under the person's account, while shadow mode is on. The notice now says only that OpenAdapt, an employer, and other apps can't turn it on or read it through OpenAdapt, and states the remaining exposure. NOTICE_VERSION is now 2026-10-17.2, so an acknowledgement of the old text no longer counts and the person has to accept the new notice. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
When the epoch ended and the credential store refused the delete, the observer created a new key and moved the active pointer to it. Nothing named the old key again, so it stayed in the keychain forever, next to a buffer it could still decrypt until the rows expired. destroy_keys now records any key id it couldn't delete under shadow-key|retired and retries it on every later destroy, and the observer passes the id it holds in case the pointer no longer names it. purge_and_disable returns False when the key wasn't deleted, so a caller can say so instead of reporting success. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
A review found several ways a typed character, or the shape of a password, reached the buffer and a draft: - Option and AltGr chords type characters (@, é), but the live bridge counted Alt as a command modifier, so the character was kept in plain text as a shortcut key, in a password box too. Alt alone is now typing, and a shortcut keeps its key only when it's a named key or one ASCII letter or digit pressed with Ctrl, Cmd, or Meta, and never in a secret field. - A modifier whose release was missed turned every later key press into a plain-text shortcut. A modifier now counts as held for at most five seconds, and a window change clears it. - Typing with an accessibility observation but no element role was digested. It now counts as secret, like typing with no observation. - Each key press in a secret field became its own symbol, so the buffer counted the password's characters. Secret typing now merges into one symbol, and its text is dropped on arrival. - An exception inside the tokenizer could carry typed text into a log or crash report through its message, its chained cause, or the frame locals that loguru's diagnose mode prints. The tokenizer now raises ShadowTokenizeError with no event content and nothing chained, and drops the pending text. Also: switching apps forgets that the browser was on a work site, so a native click in a browser waits for a browser event again, and a damaged ignore list no longer stops shadow mode from starting. Tests: the live-fed secret test checked ciphertext bytes, which can't show a plaintext leak; it now checks the sealed symbols. The typed-value test searched ciphertext for "ssn" and failed when random base64 contained it. A new test checks the file for markers base64 can't hold. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
The notice and the docs overclaimed. They said each entry is deleted after 24 hours, but expiry runs only while OpenAdapt runs, so a buffer and key left by a closed app wait until the next start. A saved review draft is kept, unencrypted, until shadow mode is turned off, and Ignore keeps a fingerprint of the task. The key may be in memory only. And the list of what it records left out keyboard shortcuts and timing. The notice now says all of that, and "Delete everything and turn off" names the drafts and the ignore list. NOTICE_VERSION is 2026-10-17.3, so the person accepts the new text. The docs describe the review fixes and list closed-app expiry and in-memory key zeroing as not done. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
…on exit - storage.app_data_dir returns Application Support on macOS, LocalAppData on Windows, and XDG_DATA_HOME on Linux. ShadowObserver refuses a store_path that resolves, through symlinks, inside Documents, Desktop, iCloud Drive, ~/Library/CloudStorage, a OneDrive folder from the environment, a Dropbox folder from its info.json, or a folder with a known sync client's name. It raises ShadowStorageRefused before making a key or a file. - pause() drops every event until unpause(), and drops typing in progress, so a person can stop it before viewing sensitive material. - shutdown() clears the buffer, deletes its file and key, and stops, keeping only the ignore list. It is for a clean exit or a plain turn-off. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
Counsel reviewed the notice. "Nothing leaves this computer" isn't true when OS backup, sync, or endpoint security tools copy local files, so the notice now says only "OpenAdapt never sends shadow mode data off this computer." It has five sections: - What it captures: steps with values stripped, keyed fingerprints of typed text, scrubbed labels and URLs, work sites only, and that the record may include personal, confidential, or health information. - Why: to suggest repeated tasks as draft automations. - How long it is kept: an encrypted buffer in the unsynced app-data folder, cleared with its key at least every 24 hours, on turn-off, and on quit. - How to erase it: turn it off, or Delete everything and turn off. Pause before viewing sensitive material. - Who can access it: as before, plus IT administrators and endpoint security or backup tools on a work-managed computer. NOTICE_VERSION is 2026-10-17.4, so it has to be accepted again. A test fails if the struck claim returns to the notice, the README, the docs, or the shadow source. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
abrichr
marked this pull request as ready for review
October 17, 2026 21:36
Windows reports 0666 for files under local app data because it uses ACLs, so the 0600 assertions failed on the Windows test job. Assert the modes on macOS and Linux only, and say in the docs what Windows does instead. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
abrichr
enabled auto-merge (squash)
October 17, 2026 22:06
Relocked uv.lock on top of the dependency bumps from main. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
Member
Author
|
Review after merge. Auto-merge, armed earlier, landed this as 3dcfdf8 once the rebased head (ae767c8) went green. I read through
The merge of main relocked End-to-end with Desktop #205: a debug Tauri bundle drove the real engine. Quit, erase, pause, and crash recovery all behaved as documented. Details are on openadapt-desktop#205. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds shadow mining to openadapt-capture. It's an opt-in watcher that keeps its data on the computer. It notices a task done many times across apps and offers to make it into a draft ("You did this N times today." with Review draft or Ignore). It never runs anything. To turn a draft into an automation, a person still records the task and reviews and admits it.
What changed
openadapt_capture/shadow/): turns events into value-free symbols, mines repeated sequences with a suffix array, and builds generic prompts. Exportsopenadapt.shadow.episodes/v2for openadapt-flow.keyring) or only in memory. The key is destroyed, along with the buffer, when it rotates. A key the keychain refused to delete is retried on every later destroy. Keyed digests never appear in an export: v2 swaps them for refs that only mean something inside that export (label-a,value-b).secure_deleteon, file mode 0600, and at most 24 hours kept. The buffer won't run on disk without a cipher.shadow/storage.py):app_data_dir()returns Application Support on macOS and LocalAppData on Windows.ShadowObserverrefuses astore_paththat resolves, through symlinks, inside Documents, Desktop, iCloud Drive,~/Library/CloudStorage, a OneDrive or Dropbox folder, or a folder named after a sync client (ShadowStorageRefused), before it makes a key or a file.pause()drops every event, and typing in progress, untilunpause().shutdown()clears the buffer and destroys its key for a clean exit or turn-off./patient-987654becomes/<id>). Short labels keep only common words, soDoe, John (12345)becomes<dynamic_label>.non-work-browsermarker.shadow/consent.py): version2026-10-17.4, rewritten after counsel's review into five sections: what it captures (and that it may include personal, confidential, or health information), why, how long it is kept, how to erase it (with advice to pause before viewing sensitive material), and who can access it. It says "OpenAdapt never sends shadow mode data off this computer" and names IT administrators and endpoint security or backup tools on a managed computer.ShadowObserver(enabled=True)needs anAcknowledgementof the current notice.purge_and_disable()deletes everything in one call.IsPassword, automation IDs, browsertypeandautocomplete, and label words. Typing with no structural evidence counts as secure. Consecutive secure key presses merge into one symbol, so length doesn't leak. Option and AltGr chords count as typing, shortcuts keep only a command letter, a stuck modifier expires after five seconds, and a tokenizer error carries no typed text.Recorder(live_event_tap=...)passes kept events to a bounded, non-blocking queue.RecorderShadowBridgemaps them to shadow events.shadowextra:cryptography>=42,keyring>=24.Not in this PR
Tests
pytest tests/test_shadow_mining.py tests/test_shadow_privacy.py tests/test_shadow_storage.py tests/test_live_tap.py: 164 passed. A test fails if "Nothing leaves this computer" returns to the notice, the README, the docs, or the shadow source. Ruff check and format are clean on the shadow files.Decisions needed
erp.example.combecomes<id>.example.comunless that subdomain is on the allowlist), and labels limited to a fixed vocabulary. Are these too aggressive?Dropbox,OneDrive*,Nextcloud). That may refuse an unusual but unsynced folder.Companion PR: OpenAdaptAI/openadapt-desktop#205 (UI). The openadapt-flow consumer is committed locally and not pushed while Flow is frozen.