Live-Use Retrieval & RAG Evaluation

The primary benchmark is derived from the user's live Torment session—not generic trivia. The durable game state contains 104 user-role turns, of which 33 are genuine manual/chat interactions after excluding automatic Recall-review prompts. Preserve original spelling and ambiguity in the test input; evaluation should measure whether retrieval and reconciliation work in real play.

Benchmark layers

Torment live-use evaluation set v1

Evaluation protocol

1. Freeze a corpus snapshot and campaign/Recall fixture for each case.
2. Run FTS-only baseline, hybrid retrieval, then optional reranked mode with identical prompt/state.
3. Record retrieved chunk IDs, source versions, scores, latency, context tokens and answer.
4. Score retrieval before judging prose, so a fluent unsupported answer fails.
5. Replay at spoiler levels none, hint and full where relevant.
6. For mutation cases, compare exact applied action sets and resulting records.

Scoring

Fixture sources

• Actual 33 manual/chat turns and their associated assistant outputs.
• Timestamp-aligned Recall entries/screenshots and raw VLM outputs after migration.
• Campaign records before and after each validated action.
• Verified Torment Extras manuals/strategy guide/novellas with page anchors.
• Versioned web artifacts actually consulted by GTC.

LM Studio generative-model benchmark

The live LM Studio endpoint advertises 21 generative models plus one Nomic embedding model. Do not benchmark every quantization as if it were a distinct capability. Build a representative shortlist by parameter class, architecture and quantization, then run identical Torment-derived tasks with fixed prompts and deterministic settings where supported.

Representative live candidates

• Small/fast: Holo 3.1 4B/9B and Qwopus/Qwable 9B variants for explanation, OCR cleanup and classification.
• Mid-size dense: Qwen 3.6 27B variants and Darwin 28B for reranking and evidence synthesis.
• MoE: Qwen/Qwable/Ornith 35B-A3B variants for higher-quality background explanation and action proposal.
• Embedding baseline: text-embedding-nomic-embed-text-v1.5.

Harness contract

benchmark_case = {
  case_id, task, game_id, prompt_version,
  input_artifact_hashes, expected_labels_or_actions,
  spoiler_level, max_context, output_schema
}
result = {
  model_id, quantization, params, latency_ms, tokens,
  peak_vram_if_available, output, parse_valid,
  automatic_scores, human_review
}

Model names exposed by LM Studio are a dynamic inventory, not proof that each model is currently loaded or supports the same context/structured-output features. Probe capabilities, record the exact model ID/quantization, and benchmark one loaded model at a time to avoid VRAM/concurrency contamination.

Selection policy

1. Reject models that fail schema adherence, leak spoilers, or invent raw fields regardless of speed.
2. Select the smallest model within 2–3% of the best quality score for routine background derivations.
3. Permit a larger optional “deep explanation” model only when its measured gain justifies latency/VRAM.
4. Never let a benchmarked local model directly mutate campaign data; it proposes, the bridge validates and records provenance.

Game expansion cohorts

Recommended progression

Do not require all four games before shipping the first retrieval implementation. Torment is the quality gate; each later game adds a distinct failure mode and expands the benchmark suite.