Existing foundation
SourceCorpus already provides game-scoped FTS5 with URL/hash versioning. Extend it rather than replacing it. The live LM Studio endpoint exposes text-embedding-nomic-embed-text-v1.5, making local semantic search feasible.
Recommended retrieval stack
query + game_id + spoiler policy→FTS5 BM25 candidates+vector candidates→reciprocal-rank fusion→metadata/spoiler filter→optional local reranker→bounded evidence pack
Build-vs-buy
| Decision | Recommendation | Why |
|---|---|---|
| Metadata/full text | Keep SQLite FTS5 | Already present, reliable, inspectable and easy to back up. |
| Vectors | Start with SQLite table storing float32 blobs + brute-force per-game cosine | YAGNI for early corpus sizes; avoids native extension packaging in UWP bridge. |
| Scale-up | Evaluate sqlite-vec only after benchmarks | Add dependency only if measured latency/corpus size requires it. |
| Embeddings | LM Studio OpenAI-compatible endpoint | Local and already exposes Nomic model; cache by chunk hash + model. |
| Reranking | Optional small local generative judge after fusion | Background/quality mode only; lexical+vector must work without it. |
Chunking
- Structure-aware: headings, PDF pages, FAQ entries, walkthrough sections.
- Target 500–900 tokens with 10–15% overlap only across natural section boundaries.
- Store heading path, page, source version, edition/game version, spoiler tags and trust.
- Embed normalized chunks; raw document remains canonical.
Retrieval contract
retrieve(game_id, query, spoiler_level, source_types?, as_of?, limit=8)
→ [{chunk_id, exact_text, source_version_id, title, uri, page,
lexical_score, semantic_score, fused_score, trust, spoiler_tags}]
Evaluation set
Create Torment-specific questions across exact names, paraphrases, quest state, mechanics and spoiler constraints. Measure Recall@k, source correctness, forbidden-spoiler leakage, latency and context tokens. Compare FTS-only against hybrid before enabling by default.
Acceptance
- Retrieval cannot cross
game_id. - Every injected passage is quotable and points to exact source version/page.
- Embedding model/version changes trigger lazy re-embedding, not destructive overwrite.
- System degrades cleanly to FTS when LM Studio is offline.
- Benchmarks justify any future vector extension or reranker.