diff --git a/README.md b/README.md index d98837f..1f41dfe 100644 --- a/README.md +++ b/README.md @@ -1,241 +1,89 @@ # 2dph — deductionphile [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -[![Go](https://img.shields.io/badge/Go-1.24+-00ADD8.svg)](https://go.dev) +[![Go](https://img.shields.io/badge/Go-1.26+-00ADD8.svg)](https://go.dev) [![Tests](https://github.com/eSlider/2dph/actions/workflows/ci.yml/badge.svg)](https://github.com/eSlider/2dph/actions/workflows/ci.yml) [![Latest Release](https://img.shields.io/github/v/tag/eSlider/2dph?sort=semver&label=release)](https://github.com/eSlider/2dph/releases) [![GitHub Stars](https://img.shields.io/github/stars/eSlider/2dph?style=social)](https://github.com/eSlider/2dph/stargazers) -An evidence-first brain. **Facts need two independent sources, or they are +An evidence-first brain. **A fact needs two independent sources, or it is `(not confirmed)`.** -`2dph` is a single embedded knowledge graph (LadybugDB) with native **HNSW -vector** + **BM25 full-text** indexes. Search is *deduction*: confirmed facts -first, supporting info second, `web-search` as the independent second source -when the local graph cannot confirm. - -## What's in 2dph today - -- **Single embedded store** — one file `var/kb.lbug` (LadybugDB). -- **Native property graph** + Cypher. -- **Native HNSW** (vectors, 256-dim model2vec) + **BM25 FTS**. -- **Hybrid search** — `facts → info → web`. -- **Graph-hop** (`--hop N`: File → Commit → Person). -- **ACID transactions** — facts + info in one transaction. -- **Incremental write** + bulk rebuild. -- **DuckDB** as auxiliary (D22 / OQ3): quantiles, JSONL stats via duckdb-go - in-process — a helper tool, **not** the primary store. - -Run it: [docs/runbook.md](docs/runbook.md). Design: [docs/design.md](docs/design.md). -Docs index: [docs/README.md](docs/README.md). - -## Tool layout (D14) - -Every command lives at `bin/{subject}/{method}.go` — one method per file, shared -logic in `internal/`. The filename *is* the invocation, and the subject is the -domain area it acts on: - -| Subject | Method | Does | -|---------|--------|------| -| `bin/brain` | `search.go` | deduction search (facts → info → web) | -| `bin/brain` | `index.go` / `add.go` | bulk rebuild / incremental write | -| `bin/brain` | `serve.go` | HTTP API + OpenAPI/MCP | -| `bin/facts` | `extract.go` / `audit.go` / `crm.go` | 2-source pairing, confidence, CRM proof | -| `bin/rules` | `book.go` | rule-source extraction (PDF → gitignored chapter text) | -| `bin/mail` | `sync.go` / `import.go` / `ocr.go` | mail ETL (Gmail/OO/M365) | -| `bin/web` | `search.go` | SearXNG second source | -| `bin/git` | `import.go` | commit history leafs | -| `bin/chat` | `sync.go` / `import.go` / `apply.go` | conversations | -| `scripts/stack` | `start` / `status` / `stop` | compose dispatcher | - -Go methods are executable (`go run` shebang); a few are thin bash launchers -(`bin/chat`, `scripts/db/psql-yq`). Shell completions for all tools (D23) come from -`bin/shell/complete.go` — see the runbook. Keep it one-command-one-file so the -surface stays deductive: you read the path, you know the tool. - -**`bin/cgo`** is the CGO toolchain, **not** CI/CD: `zig` (the pinned Zig -compiler), `zcc` / `zc++` (wrappers). Ladybug and tokenizer C libraries are -compiled with `zig cc` (D21), so brain read/write Go binaries link CGO without -a system gcc. CI/CD lives separately in `.github/workflows/ci.yml`. +2dph is a single embedded knowledge graph (LadybugDB) with BM25 full-text and +vector search. One query runs a deduction pass: confirmed facts first, +supporting notes second, web search as the independent second source when the +local graph cannot confirm. + +## How it works + +- **Two roots, one transaction** — `facts` (confirmed) and `info` (context) + live in one ACID store, so the audit always sees a consistent snapshot. +- **Deduction read path** — `facts → info → web`; every hit carries evidence, + confidence and freshness. +- **Evidence gate** — two-source pairing, contradiction and staleness checks. +- **Hybrid retrieval** — BM25 FTS + 256-dim model2vec embeddings, merged in + the ranker. +- **Graph hops** — `--hop N` walks File → Commit → Person. ## Architecture ```mermaid graph LR - subgraph src["Sources"] - direction TB - DOC["documents"] - MAIL["mail"] - CHAT["chats"] - CONTACT["contacts"] - GIT["git history"] - end - - subgraph etl["Adapters → leafs"] - direction TB - SPLIT["markdown/split-leaf"] - MAILI["mail/sync · mail/import"] - CHATI["chat/sync · chat/import"] - CONV["contact/list"] - GITI["brain/import-git"] - end - - subgraph store["Embedded store — Ladybug (one kb.lbug)"] - ROOTS["roots: facts | info
one ACID transaction"] - IDXN["HNSW vectors + BM25 FTS
(model2vec embeddings)"] - GRAPH["property graph · Cypher
File→Commit→Person hops"] - end - - subgraph read["Deduction read path"] - SRCH["brain/search
facts → info → web"] - SERVE["brain/serve
HTTP · OpenAPI · MCP"] - end - - subgraph gate["Evidence gate"] - EXTR["facts/extract
2-source pairing"] - AUD["facts/audit-db
confidence + staleness"] - end - - WEB["web-search
independent 2nd source"] - AGENT["agents · operators"] - - DOC --> SPLIT - MAIL --> MAILI - CHAT --> CHATI - CONTACT --> CONV - GIT --> GITI - - SPLIT --> ROOTS - MAILI --> ROOTS - CHATI --> ROOTS - CONV --> ROOTS - GITI --> GRAPH - EXTR --> ROOTS - IDXN --- ROOTS - GRAPH --- ROOTS - - ROOTS --> SRCH - SRCH <-.-> WEB - SRCH --> SERVE - SERVE --> AGENT - AUD --> ROOTS -``` - -Reads are deduction: confirmed `facts` first, supporting `info` second, -`web-search` as the independent second source when the local graph cannot -confirm. Writes never bypass the store's single transaction. - -## The method - -Every assertion is `Who / What / How / Where / When + evidence + confidence`, -mirroring the detective method: **≥2 independent sources confirm a -fact; conflicting sources or a single source → `hypothesis` → `(not confirmed)`.** - -| root | meaning | used for answers | -|------|---------|------------------| -| `facts` | assertions backed by ≥2 sources (`confirmed`) | yes, with evidence links | -| `info` | descriptive/narrative leafs (how-tos, notes) | context only, marked `(not confirmed)` | - -## Deduction search - -```bash -bin/brain/search.go "Matrix federation over HTTPS" # facts → info → web -bin/brain/search.go "onlyoffice postgres" --root facts -bin/brain/search.go "where is cs-lexicon" --json | yq '.' -bin/brain/search.go "upstream flag" --no-web # local graph only -bin/brain/get.go --body # full chunk on demand -bin/brain/stats.go # index health -bin/brain/eval.go # recall@5 gate + SRC["sources
docs · mail · chat · git · contacts"] + --> ADAPT["adapters → leafs"] + ADAPT --> STORE["Ladybug — one kb.lbug
graph · facts/info · BM25"] + ANN["IVF ANN · 256-dim"] --- STORE + STORE --> READ["deduction search
facts → info → web"] + READ --> API["HTTP · OpenAPI · MCP"] + GATE["evidence gate
2-source pairing"] --> STORE ``` -`--hop N` walks File/Commit/Person from each hit (max 3). Search is `bin/brain/search.go`. - -Git history is read with [go-git](https://github.com/go-git/go-git) (no git binary): - -```bash -bin/brain/import-git.go --json --limit 100 # commit leafs for this repo -bin/brain/import-git.go --root "$PROJECTS_ROOT" --json # one pass per .git under root -``` +## Why IVF, not HNSW -Conversion only. Graph write (`File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`) stays with `bin/brain/index.go`. +Ladybug's own HNSW index crashes once the graph grows (~1.3k nodes, #192), so +vector search lives **outside** liblbug. `coder/hnsw` was tried and rejected: +on 313k real embeddings only 34% of nodes stayed reachable and recall@5 fell +to 0.16. -Web search (second independent source) goes through SearXNG. Empty results mean **throttled**, not “nothing exists”: +The replacement is a small pure-Go **IVF** index (`internal/brain/ann`): +k-means cells plus exact cosine inside the probed cells, persisted as +`var/state/vector.ann` with an append-only WAL. It is updated incrementally, +rebuilt only when missing or more than 10% behind, and falls back to a linear +scan when absent or damaged — search never breaks. -```bash -bin/web/search.go "LadybugDB vector index" --json -# Optional local instance (skip if BRAIN_SEARCH_URL already points at one): -# SEARXNG_SECRET=$(openssl rand -hex 32) docker compose --profile searxng up -d -``` +Measured on 313,689 vectors: recall@5 vs exact scan = **1.0**, p50 ≈ **470 ms** +(#204/#206). -Mail is a first-class corpus (retrievable through the same search): +## Quick start ```bash -bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/corpus/mail # raw sync (Go) -bin/mail/sync.go --source m365 --env ~/.config/brain/mail.env # Microsoft 365 Graph -scripts/stack/start-mail-sync # compose ETL (300s; no auto-rebuild) -bin/mail/import.go --from-raw var/corpus/mail # JSON → markdown -bin/brain/add.go --text T --root facts --source "a.md x b.md" -bin/brain/index.go --rebuild --with-facts --with-chats # facts extract + chats md -bin/brain/index.go --rebuild # rebuild brain (incl. mail) -bin/brain/search.go "invoice from last week" # same search over mail leafs +git clone https://github.com/eSlider/2dph && cd 2dph +bin/brain/index.go --rebuild # build the brain +bin/brain/search.go "Matrix federation over HTTPS" # ask a question +bin/brain/serve.go # HTTP + OpenAPI + MCP :8630 ``` -## Storage - -- **LadybugDB** — single `var/kb.lbug`, Cypher + HNSW + BM25, embedded. - Read tools (`get` / `stats` / `eval`) are Go + Zig CGO (`bin/cgo/zcc`). - Incremental write is `bin/brain/add.go`; bulk rebuild is - Compose profile `index` (`bin/brain/index.go --rebuild`). -- **potion-multilingual-128M** — 256-dim embeddings (Go/Ladybug, CPU, no Ollama) - runtime dependency. -- facts and info split by `root` but written in the same transaction. - -Ladybug 0.19 DROP INDEX warning: [docs/runbook.md](docs/runbook.md). - -## Tooling conventions - -`bin/{subject}/{method}.go` — self-describing: shebang on line 1, usage comment -from line 2. Shared code in `internal/`. YAML default output, `--json` for -machines. Tests gate every commit. HTTP: `bin/brain/serve.go` calls -`internal/brain` in-process (`/health` `/search` `/get` `/stats` `/audit` `/ingest` `/openapi.json` `/mcp`). - -## Development +Incremental write is `bin/brain/add.go`; background indexing runs in the +compose `index-sync` cycle. Full setup: [docs/runbook.md](docs/runbook.md). -See the portable runbook: [docs/runbook.md](docs/runbook.md). +## What's inside -```bash -bin/facts/audit.go self -go test ./... -``` +| Path | Role | +|------|------| +| `bin/brain` | search · index · serve | +| `bin/facts` | extract · audit — the evidence gate | +| `bin/mail`, `bin/chat`, `bin/git`, `bin/web` | corpus adapters | +| `internal/brain` | store, rank, ANN, FTS | +| `scripts/stack` | compose dispatcher | -Docker (optional, cached model + var volumes): +## Docs -```bash -scripts/stack/start # brain HTTP/MCP :8630 -scripts/stack/start-assistant # + qwen3.5:9b + PicoClaw agent -scripts/stack/status -scripts/stack/stop -docker compose up -d brain # API (Zig CGO serve :8630) -docker compose --profile index run --rm index # Go Ladybug rebuild (zig cgo) -docker compose --profile picoclaw up brain-mcp # MCP on 127.0.0.1:8630 -docker compose --profile reasoner up -d reasoner # CPU Ollama 127.0.0.1:11435 -docker compose up brain-watch # auto re-index on change -``` +[runbook](docs/runbook.md) · [design](docs/design.md) · +[docs index](docs/README.md) · [decisions](PLAN.md). ## Related -eSlider DevOps engineer practice: ops, OnlyOffice, and mail feed the facts -root through `bin/facts/extract` (two-source pairing). - - [go-second-brain](https://github.com/eSlider/go-second-brain) — the earlier Neo4j + Qdrant + Matrix RAG brain -- [agent-skills](https://github.com/eSlider/agent-skills) — upstream - skills (`web-search`, `postgres`, …) that 2dph integrates -- detective method — the two-source method - -Work board (issues): [epic #16](https://git.produktor.io/eSlider/2dph/issues/16) -on [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues). -PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph). - -See [PLAN.md](PLAN.md) for decisions, [docs/roadmap.md](docs/roadmap.md) for -the gap to v1, and v2 open questions. +- [agent-skills](https://github.com/eSlider/agent-skills) — upstream skills + (`web-search`, `postgres`, …) that 2dph integrates diff --git a/docs/roadmap.md b/docs/roadmap.md index 9e7efda..25aaa1e 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -53,7 +53,7 @@ None for epic #16 (closed). Remaining v2: OQ4 (deferred). ``` question │ - ├─ FTS + HNSW ← in + ├─ FTS + IVF ANN ← in ├─ facts / info roots ← in ├─ web (D17) ← in ├─ brain/add ACID ← in diff --git a/internal/brain/annbridge.go b/internal/brain/annbridge.go index d260e37..e67d64f 100644 --- a/internal/brain/annbridge.go +++ b/internal/brain/annbridge.go @@ -14,7 +14,7 @@ import ( "github.com/eSlider/2dph/internal/config" ) -// ANN integration (issue #204): an incremental HNSW index outside liblbug +// ANN integration (issue #204): an incremental IVF index outside liblbug // (whose own HNSW SIGSEGVs on grown graphs, #192). The index lives at // var/state/vector.ann (+ .wal), is built/upserted by bin/brain/ann.go and // read here for the query vector path. Missing/corrupt/disabled index falls diff --git a/internal/brain/doc.go b/internal/brain/doc.go index d5670bd..5162ea7 100644 --- a/internal/brain/doc.go +++ b/internal/brain/doc.go @@ -1,3 +1,3 @@ -// Package brain is deduction search over Ladybug (FTS + HNSW). +// Package brain is deduction search over Ladybug (FTS + external IVF ANN). // Query/embed code that needs cgo lives behind the system_ladybug tag. package brain diff --git a/internal/brain/ingest_soak_test.go b/internal/brain/ingest_soak_test.go index 0ae9e50..82d7f13 100644 --- a/internal/brain/ingest_soak_test.go +++ b/internal/brain/ingest_soak_test.go @@ -55,7 +55,7 @@ func stubEmbeddings(t *testing.T) { t.Cleanup(func() { embedIngestLeafs = prev }) } -// soakSeedDB builds a pre-built corpus (schema + leafs + FTS/HNSW indexes) +// soakSeedDB builds a pre-built corpus (schema + leafs + FTS index) // the way production serves over an existing kb.lbug, returning two seed ids. func soakSeedDB(t *testing.T) (idA, idB string) { t.Helper() diff --git a/internal/brain/search.go b/internal/brain/search.go index 19552da..a82a375 100644 --- a/internal/brain/search.go +++ b/internal/brain/search.go @@ -194,7 +194,7 @@ func queryFTS(text string, limit int) ([]Hit, error) { } func queryVector(emb []float64, limit int) ([]Hit, error) { - // ANN path (issue #204): top-k from the incremental HNSW index, then + // ANN path (issue #204): top-k from the incremental IVF index, then // metadata via point lookups. Falls back to the brute-force scan when // the index is absent/corrupt/disabled — search never fails because of // ANN. diff --git a/skills/duckdb/SKILL.md b/skills/duckdb/SKILL.md index bd3758d..74c0cb9 100644 --- a/skills/duckdb/SKILL.md +++ b/skills/duckdb/SKILL.md @@ -24,7 +24,7 @@ CC=gcc CXX=g++ CGO_CFLAGS= CGO_LDFLAGS= go test ./pkg/duckdb | Store | Job | |-------|-----| -| Ladybug | graph + FTS + HNSW (facts/info) | +| Ladybug | graph + FTS + IVF ANN (facts/info) | | modernc sqlite | web-search KV cache + throttle | | duckdb-go | OLAP: quantiles, counts, scans of many rows/files | | mikefarah/yq | small YAML/JSON/XML/CSV/TOML/HCL slice, not bulk |