fix(SESF-79): bound the memory sync loop and report a stalled one

AGENTS.md

@@ -44,7 +44,7 @@ Semantic search over agent session transcripts. Independent project, originally
 - **Secret redaction guard (SESF-41)** — ingestion-time redaction hooked once in `add_turns` (covers the embedding, Milvus `document`, and FTS `content` sinks plus `add_turns_async`, and runs before `_extract_issue_ids`). Pure engine in `secret_redaction.py`: `redact(text, *, mode, allowlist) -> (redacted_text, hits)`. Config: `SESSIONFLOW_REDACT` (on/off, default **on**); `SESSIONFLOW_REDACT_MODE` (`enforce`|`report`, default **report** when unset — detect + count, store raw so operators can size the false-positive rate before enforcing); `SESSIONFLOW_REDACT_ALLOWLIST` (path to an operator regex allowlist, one pattern per line). Per-rule detection counts surface via `get_stats` under the `redaction` key. Tier-3 entropy is a length-gated Shannon scanner (detect-secrets' entropy plugins are unused — `scan_line` ignores their limit); GitHub/GitLab use custom regexes (detect-secrets reports a truncated prefix).
 - **EmbeddingGemma padding mask (SESF-57)** — mlx-embeddings 0.0.5 hands the 0/1 tokenizer mask to SDPA as an *additive* float mask, so padding leaked into shorter docs in a batch. `rag_engine._embed_gemma` runs the gemma forward itself with a boolean key-padding mask (`_key_padding_mask`). Batched now matches single to cos ≥ 0.99999, and a batch of one (every query) is unchanged. `tests/test_embed_padding.py` pins this, so don't revert to calling `model(input_ids, attention_mask=...)`. The 512-token sliding window is still not applied on local layers (SESF-62, measure first). Stored vectors were rewritten in place by `reembed.py` (`cleanup.py reembed {plan|run|drive|status|verify}`; chunked under the MLX crash limit; old vectors backed up to `~/.sessionflow/audit/reembed-<run>.f16`).
 - **Memory store is Postgres (DEVS-145)** — memories commit to Postgres first; the Milvus `memories` collection and `~/.sessionflow/memories_fts.db` are derived indexes. The DSN comes from `SESSIONFLOW_MEMORY_DSN` only: SessionFlow calls no secrets tool and reads no DSN file, so a launchd install sets it in the LaunchAgent environment. Memory starts after transcript startup on its own thread; a missing DSN or a down database leaves `/health` `memory.ready` false with the reason and transcript search untouched. Blocking store calls go through `memory_store.run_db` (the `memory-db` executor), never the event loop or `mlx-embed`.
-- **Never write a memory index directly (DEVS-145)** — every write goes through `MemoryService` to Postgres, which resets that memory's rows in `memory_index_acks` in the same transaction. The sync loop (`SESSIONFLOW_MEMORY_SYNC_INTERVAL`, plus a full reconcile at start and every 6 hours) applies what each consumer (`milvus`, `fts:<host>`) lacks. A Milvus write re-reads the row under `pg_advisory_xact_lock` and upserts only if the embedded hash still matches; the FTS file keeps a per-record watermark so an older write cannot revive a deleted record. `change_seq` orders work but does not follow commit order, so pending work is decided by acknowledgements, not by a cursor.
+- **Never write a memory index directly (DEVS-145)** — every write goes through `MemoryService` to Postgres, which resets that memory's rows in `memory_index_acks` in the same transaction. The sync loop (`SESSIONFLOW_MEMORY_SYNC_INTERVAL`, plus a full reconcile at start and every 6 hours) applies what each consumer (`milvus`, `fts:<host>`) lacks, this host's FTS first, and skips a Milvus record another server already acknowledged. Each tick and reconcile is bounded by `SESSIONFLOW_MEMORY_TICK_TIMEOUT` (600 s): a timeout logs every asyncio task's await chain, cancels the work and lands in `/health` `memory.sync_error`, and `memory.index.stale` turns true when `last_tick` falls behind. A failed reconcile retries after 300 s, and a pass is skipped while three of the four memory-db workers are busy (`memory_store.db_calls_in_flight`), because a cancelled `run_db` await leaves its thread running. `kill -USR1 <pid>` logs the same task dump on demand; thread profilers such as py-spy cannot see suspended coroutines. A Milvus write re-reads the row under `pg_advisory_xact_lock` and upserts only if the embedded hash still matches; the FTS file keeps a per-record watermark so an older write cannot revive a deleted record. `change_seq` orders work but does not follow commit order, so pending work is decided by acknowledgements, not by a cursor.
 - **Memory embeds share the MLX thread (DEVS-145)** — `memory_index.embed_for_index` waits on `embedding_control.EmbedThrottle` (200 ms floor) and runs on `mlx-embed`, where it waits again and stamps the throttle; it never uses the transcript budget, which drops batches. `EmbeddingBudget.before_batch` honors the same throttle, so neither consumer starts inside the other's cooldown. `memory_cli` never embeds: `rebuild` and `reindex` only reset acknowledgements, and `restore-check` asks the running server's loopback `/memory/restore-check` route to index into a temporary `memories_restore` collection.
 - **Memory embedding generation (DEVS-145)** — `memory_index_meta` names the memory model and generation. `cleanup.py reembed` starts a new generation on a fresh run and resets the Milvus acknowledgements; the server re-embeds from Postgres text. A server whose local model differs refuses memory vector writes and vector searches (`MemoryModelMismatch`; search answers from FTS) and reports it in `/health` (`memory.index.milvus_blocked`).
 - **Memory secret check (DEVS-145)** — `add_memory` rejects text with any `secret_redaction.scan_spans` hit and names the rule. Paths and code identifiers can trip it; the fix is a narrow pattern in the operator allowlist (`SESSIONFLOW_REDACT_ALLOWLIST`), never a looser rule.

CLAUDE.md

@@ -45,7 +45,7 @@ Semantic search over Claude Code session transcripts. Independent project, origi
 - **Retroactive sanitizer (SESF-42)** — removes secrets *already* indexed (the cleanup half of SESF-35; SESF-41 is the forward guard). Orchestrator in `sanitize.py` (`scan` dry-run / `apply` redact|drop), driven by `cleanup.py sanitize` (CLI) and the `sanitize_index` MCP tool. **Dry-run is the default and writes nothing**; `--apply`/`apply` **refuse without `--yes`/`confirm`**. Reuses the SESF-41 detector via `secret_redaction.scan_spans` (per-occurrence, value-free audit spans — snippets mask every detected span in-window, including advisory Tier-3 whose forcing keyword sits outside the ±24 window). New Milvus primitives `rag_engine.upsert_document` (upsert-by-PK + **metadata-preserving** FTS delete-then-insert; `new_vector=None` keeps the stored vector on FTS-only re-converge) and `delete_by_doc_id` (`DeleteResult(deleted, fts_ok)`). **FTS rewrite/delete failure keeps a turn unfinished/retryable** — FTS healing only hydrates *missing* doc_ids, so a stale row left in place would never be re-redacted. Per-run audit JSONL at `~/.sessionflow/audit/redaction-<runid>.jsonl` + checkpoint `~/.sessionflow/sanitize_state.json` (both `0700` dir / `0600` file, value-free). Sanitization is irreversible — **redaction ≠ rotation; rotate any indexed key**.
 - **EmbeddingGemma padding mask (SESF-57)** — mlx-embeddings 0.0.5 hands the 0/1 tokenizer mask to SDPA as an *additive* float mask, so padding leaked into shorter docs in a batch. `rag_engine._embed_gemma` runs the gemma forward itself with a boolean key-padding mask (`_key_padding_mask`). Batched now matches single to cos ≥ 0.99999, and a batch of one (every query) is unchanged. `tests/test_embed_padding.py` pins this, so don't revert to calling `model(input_ids, attention_mask=...)`. The 512-token sliding window is still not applied on local layers (SESF-62, measure first). Stored vectors were rewritten in place by `reembed.py` (`cleanup.py reembed {plan|run|drive|status|verify}`; chunked under the MLX crash limit; old vectors backed up to `~/.sessionflow/audit/reembed-<run>.f16`).
 - **Memory store is Postgres (DEVS-145)** — memories commit to Postgres first; the Milvus `memories` collection and `~/.sessionflow/memories_fts.db` are derived indexes. The DSN comes from `SESSIONFLOW_MEMORY_DSN` only: SessionFlow calls no secrets tool and reads no DSN file, so a launchd install sets it in the LaunchAgent environment. Memory starts after transcript startup on its own thread; a missing DSN or a down database leaves `/health` `memory.ready` false with the reason and transcript search untouched. Blocking store calls go through `memory_store.run_db` (the `memory-db` executor), never the event loop or `mlx-embed`.
-- **Never write a memory index directly (DEVS-145)** — every write goes through `MemoryService` to Postgres, which resets that memory's rows in `memory_index_acks` in the same transaction. The sync loop (`SESSIONFLOW_MEMORY_SYNC_INTERVAL`, plus a full reconcile at start and every 6 hours) applies what each consumer (`milvus`, `fts:<host>`) lacks. A Milvus write re-reads the row under `pg_advisory_xact_lock` and upserts only if the embedded hash still matches; the FTS file keeps a per-record watermark so an older write cannot revive a deleted record. `change_seq` orders work but does not follow commit order, so pending work is decided by acknowledgements, not by a cursor.
+- **Never write a memory index directly (DEVS-145)** — every write goes through `MemoryService` to Postgres, which resets that memory's rows in `memory_index_acks` in the same transaction. The sync loop (`SESSIONFLOW_MEMORY_SYNC_INTERVAL`, plus a full reconcile at start and every 6 hours) applies what each consumer (`milvus`, `fts:<host>`) lacks, this host's FTS first, and skips a Milvus record another server already acknowledged. Each tick and reconcile is bounded by `SESSIONFLOW_MEMORY_TICK_TIMEOUT` (600 s): a timeout logs every asyncio task's await chain, cancels the work and lands in `/health` `memory.sync_error`, and `memory.index.stale` turns true when `last_tick` falls behind. A failed reconcile retries after 300 s, and a pass is skipped while three of the four memory-db workers are busy (`memory_store.db_calls_in_flight`), because a cancelled `run_db` await leaves its thread running. `kill -USR1 <pid>` logs the same task dump on demand; thread profilers such as py-spy cannot see suspended coroutines. A Milvus write re-reads the row under `pg_advisory_xact_lock` and upserts only if the embedded hash still matches; the FTS file keeps a per-record watermark so an older write cannot revive a deleted record. `change_seq` orders work but does not follow commit order, so pending work is decided by acknowledgements, not by a cursor.
 - **Memory embeds share the MLX thread (DEVS-145)** — `memory_index.embed_for_index` waits on `embedding_control.EmbedThrottle` (200 ms floor) and runs on `mlx-embed`, where it waits again and stamps the throttle; it never uses the transcript budget, which drops batches. `EmbeddingBudget.before_batch` honors the same throttle, so neither consumer starts inside the other's cooldown. `memory_cli` never embeds: `rebuild` and `reindex` only reset acknowledgements, and `restore-check` asks the running server's loopback `/memory/restore-check` route to index into a temporary `memories_restore` collection.
 - **Memory embedding generation (DEVS-145)** — `memory_index_meta` names the memory model and generation. `cleanup.py reembed` starts a new generation on a fresh run and resets the Milvus acknowledgements; the server re-embeds from Postgres text. A server whose local model differs refuses memory vector writes and vector searches (`MemoryModelMismatch`; search answers from FTS) and reports it in `/health` (`memory.index.milvus_blocked`).
 - **Memory secret check (DEVS-145)** — `add_memory` rejects text with any `secret_redaction.scan_spans` hit and names the rule. Paths and code identifiers can trip it; the fix is a narrow pattern in the operator allowlist (`SESSIONFLOW_REDACT_ALLOWLIST`), never a looser rule.

README.md

@@ -105,6 +105,7 @@ SessionFlow also stores memories: notes an agent writes on purpose, such as sess
 |----------|---------|-------------|
 | `SESSIONFLOW_MEMORY_DSN` | unset | Postgres connection string. SessionFlow reads it from the environment only. |
 | `SESSIONFLOW_MEMORY_SYNC_INTERVAL` | `5` | Seconds between index sync passes. |
+| `SESSIONFLOW_MEMORY_TICK_TIMEOUT` | `600` | Seconds one sync pass may run. A pass that runs longer is cancelled, its waiting tasks are logged, and `/health` reports it under `memory.sync_error`. `memory.index.stale` is true when the last pass is older than this plus three intervals. |
 | `SESSIONFLOW_MEMORY_SIMILAR_THRESHOLD` | `0.88` | Similarity at which `add_memory` lists a memory as similar. |
 | `SESSIONFLOW_MEMORY_AUDIT_AGE_DAYS` | `21` | Age at which the audit lists an open item or blocker. |
 | `SESSIONFLOW_HOST_NAME` | short host name | Names this machine's keyword index in the acknowledgements. |