Gateway Module — `vault-index/`

1. Purpose

The vault-document indexer (D361 / Fuda AK-287). It walks the Sage Library vault, FCF-classifies each markdown file, heading-chunks it (1,500-token cap), embeds and FTS-indexes every chunk, and exposes the result as a SearchAdapter so the context assembly fanout can retrieve vault knowledge as a first-class source (vault_index store). It runs a differential scan (mtime-gated, hash-checked, deletion-reconciled, path-traversal-guarded) and maintains a per-FCF-type effectiveness table fed by the RAG feedback loop — measuring which document types (ENT, Docu, Intel, …) actually get used after being assembled.

2. File Inventory

3. Public API Surface

REST Endpoints

Mounted at /api/vault-index. Auth per AK-415: GETs → Bearer (write OR read-only); writes → Bearer (write only). /api/vault-index was explicitly dropped from the former GET-exemption set (D913 Fix 2) — now Bearer-gated.

MCP Tools

None. This module has no direct MCP tool — its retrieval reaches agents indirectly via assemble_context (the vault_index source store in the context fanout). Scans are triggered by launchd/cron or manual REST call, not MCP.

4. Internal API

• createVaultIndexLayer(db, embeddingProvider): {repo, router, scan(vaultRoot)}.
• VaultIndexRepository — full CRUD + createSearchAdapter() (consumed by context/index.ts fanout as vaultIndexSearch) + effectiveness eval. document + chunk CRUD (path lookup, list, upsert, chunk insertion incl. vector, deletion, and count helpers).
• fcf-filter exports (extractFcfType, extractMissionNamespace, classifyFile, extractFcfTypeFromFrontmatter) — reusable FCF parsing, currently scanner-internal.
• chunkMarkdown + Chunk — reusable heading chunker.

5. Background Services

• Differential scan (scanVault) — not self-scheduled; invoked by POST /api/vault-index/scan (launchd/cron cadence) or the module scan() method. Guarded by an in-memory scanInProgress mutex (overlapping triggers → 409 / rejected result). Per-file: mtime-gate skip, size-cap skip (512KB), FCF-classify skip, chunk+embed, deletion reconciliation at end.
• Type-effectiveness eval — POST /type-effectiveness/eval; scans a 168-hour context_assembly_log window, tallies assembled-vs-used per FCF type (parsed from the [FcfType] content prefix), upserts vault_type_effectiveness. Feeds source-type signal-to-noise back to the operator.

6. Data Contracts

• FCF_TYPE_WHITELIST = ['ENT','Docu','Seed','MissionBrief','Intel','TechSpec','IDEA','IN'] (8 indexable types; other types are skipped).
• ScanConfigSchema = {vault_root, max_file_size_bytes=512000, max_chunk_tokens=1500, exclude_patterns=[artefacts/**, .obsidian/**, .claude/**, .trash/**, templates/**, temporal/**, media/**, node_modules/**]}.
• ScanResultSchema = {total_files_scanned, files_indexed, files_skipped, files_deleted, chunks_created, duration_ms, errors[]}.
• VaultDocumentSchema / VaultChunkSchema — row shapes.
• TypeEffectiveness = {fcf_type, assembled_count, used_count, effectiveness, last_eval_at}.

Owned tables (declared in memory/db.ts): vault_documents, vault_chunks (+index, FK cascade), vault_fts (fts5), vault_vec (vec0), vault_type_effectiveness, and sidecar triggers vault_chunks_ai/ad_sidecar (FTS trigger-maintained).

7. Dependencies

• Gateway modules consumed: memory/types (EmbeddingProvider), context/types (SearchAdapter, SearchResult — the adapter output contract), shared/search-utils (estimateTokens, buildFts5Query, reciprocalRankFusion, normaliseRrfScores). Reads context_assembly_log (context module's table) for effectiveness eval.
• External libraries: better-sqlite3, sqlite-vec, express, zod, Node crypto + fs/promises + path.
• Environment variables: VAULT_ROOT (scan root; falls back to . in config default and a 6-levels-up resolve in the /scan route).

8. Test Coverage

72 targeted tests across the module's three behavioural surfaces.

9. Known Limitations

• Scan is manual/cron-triggered, not event-driven — the index drifts stale between scans; a freshly-written vault note is invisible to assemble_context until the next scan. mtime gating means the freshness ceiling = scan cadence.
• In-memory mutex only — scanInProgress is per-process; a multi-process deployment (not current) could run concurrent scans.
• Glob matching is prefix-only — matchesExcludePattern supports dir/** prefix + exact match, not full glob (no *.ext, no mid-path wildcards).
• 8-type whitelist — Matter, Data, Code, Log, Meeting, CMD, TMPL, etc. are not indexed; vault knowledge in those types is invisible to RAG.
• Effectiveness eval parses a content prefix — relies on [FcfType] literal at the start of the assembled vault_index content; a formatting change in the adapter's prefix would silently break the eval tally.

10. Change History