Start narrow, then grow
Build an explicit onboarding wizard rather than an uncontrolled crawler. It proposes sources, previews scope/size/license hints, and asks the player to approve ingestion categories.
Verified Torment Extras inventory
The live example directory exists. It contains Traveler’s Guide/manual PDFs (including a duplicate), Guide/Torment Strategy Guide.pdf, five novella PDFs, concept-art PDF/CBZ, a world-map image, Extras.rar, and many soundtrack/wallpaper/avatar/ringtone assets.
Initial policy: prioritize manuals, strategy guides, novellas, and text-bearing PDFs. Keep audio and decorative images metadata-only unless multimodal indexing is explicitly enabled.
Initialization stages
- Detect identity: normalized
game_id, executable, install roots, Steam app ID where available. - Scan local extras: approved install subdirectories only. Example candidate:
E:/SteamLibrary/steamapps/common/Torment Tides of Numenera/Extras/. - Inventory: PDFs, HTML, TXT, Markdown, images, manuals and guides; show sizes/hashes before extraction.
- Official discovery: official site/support/manual/store/community hub links.
- Community source proposals: wikis, GameFAQs, Steam guides, Reddit threads—ranked by authority, completeness and spoiler level.
- User selection: full ingest, metadata-only, exclude, or defer.
- Extract/index: preserve document structure and page/heading provenance.
- Generate guide: a concise corpus manifest, spoiler policy, available references and gaps—not a fabricated walkthrough.
Source adapters
| Adapter | Technique | Guardrails |
|---|---|---|
| Local PDF/manual | PyMuPDF text + page boundaries; OCR fallback only when needed | User-approved roots; hash/version; retain page references |
| Static wiki | GTC map/crawl with domain/path limits | Budget, robots/terms, canonical URLs, incremental refresh |
| Steam guide | GTC scrape selected guide URLs | Author attribution; avoid account/private content |
| GameFAQs/Reddit | Selected URLs or bounded search results | Community trust label; timestamps; do not treat comments as canonical |
| Official media | Manuals, patch notes, developer posts, subtitles/transcripts when legally accessible | Type/edition/version tagging |
Spoiler-aware metadata
Classify documents/chunks as mechanics, early-game, quest-specific, full-walkthrough, lore, patch/version, and unknown. Retrieval applies the player’s current spoiler policy before content enters the prompt.
PowerShell-first tooling
scripts/scan-game-content.ps1— read-only inventory producing JSON.scripts/import-game-content.ps1— explicit approved manifest submission.- Bridge owns parsing/indexing and rejects paths outside approved roots.
- Tests use temporary fixture directories; never scan live Steam libraries by default.
Acceptance
- Torment Extras can be inventoried without reading outside the approved game root.
- PDF chunks retain page numbers and source file hash.
- Wizard previews estimated storage and source count before ingestion.
- Stopping/restarting resumes idempotently.
- Full walkthrough content is filtered under none/hint spoiler levels.