GroktoCrawl x Hermes Agent

1. Project Overview & Architecture

GroktoCrawl is a self-hosted, MIT-licensed alternative to Firecrawl. It implements the entire Firecrawl v2 API surface as a set of Python FastAPI microservices running in Docker. It provides web scraping, crawling, search, browser automation, document parsing, structured data extraction, and an autonomous AI research agent - all behind a single REST API on port 8080.

Architecture Diagram

2. GroktoCrawl Component Inventory

The repo at C:\Users\Gaem4090\groktocrawl contains these services, all orchestrated by a single docker-compose.yml:

Pre-built Docker Images (GHCR)

• ghcr.io/groktopus/groktocrawl-agent
• ghcr.io/groktopus/groktocrawl-scraper
• ghcr.io/groktopus/groktocrawl-browser
• ghcr.io/groktopus/groktocrawl-semantic
• ghcr.io/groktopus/groktocrawl-portal

3. Full API Surface (27 Endpoints)

Extracted from openapi.json in the repo. All endpoints are on http://localhost:8080 (the agent-svc container).

4. Installation: Docker Stack Deployment

Step 1: Create the .env file

# From the groktocrawl directory
cd C:\Users\Gaem4090\groktocrawl
cp .env.sample .env

# Edit .env - set these required values:
# LLM provider (use Featherless - you already have this)
LLM_API_KEY=your-featherless-key
LLM_BASE_URL=https://api.featherless.ai/v1
LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct

# Search backend (Brave API key - get free at https://brave.com/search/api/)
BRAVE_API_KEY=your-brave-search-api-key

# Optional: protect your API with a key
# API_KEY=generate-a-random-secret-here

Step 2: Start the Docker stack

cd C:\Users\Gaem4090\groktocrawl

# Pull pre-built images and start all services
docker compose up -d

# Watch the logs to confirm everything starts
docker compose logs -f agent-svc

# Verify the API is healthy (in another terminal)
curl http://localhost:8080/health

Step 3: Verify each service

# Scrape a page
curl -X POST http://localhost:8080/v2/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

# Search the web
curl -X POST http://localhost:8080/v2/search \
  -H "Content-Type: application/json" \
  -d '{"query":"rust programming","limit":3}'

# Map a site
curl -X POST http://localhost:8080/v2/map \
  -H "Content-Type: application/json" \
  -d '{"url":"https://docs.python.org","limit":10}'

# Portal dashboard
open http://localhost:8082

5. Configuration & Environment

GroktoCrawl .env (key variables)

Hermes-side environment

The Hermes plugin reads one env var to know where GroktoCrawl is:

# Set in Hermes config or ~/.hermes/.env
GROKTOCRAWL_API_URL=http://localhost:8080

# If you set API_KEY in GroktoCrawl's .env, also set:
GROKTOCRAWL_API_KEY=your-api-key-here

6. Hermes Plugin Architecture

Hermes has a first-class plugin system. Plugins live in ~/.hermes/plugins/ (user plugins) or plugins/ (bundled). Each plugin is a Python package with an __init__.py that exports a register(ctx) function. The PluginContext object provides ctx.register_tool() which registers tools in the global tool registry.

How plugin loading works (verified from source)

1. model_tools.py calls discover_plugins() at startup
2. PluginManager.discover_and_load() scans ~/.hermes/plugins/ for directories with __init__.py
3. Each plugin's register(ctx) is called with a PluginContext
4. ctx.register_tool(name, toolset, schema, handler, ...) adds tools to the global registry
5. Tools appear in the model's tool schema automatically

PluginContext.register_tool signature (from hermes_cli/plugins.py:319)

def register_tool(
    self,
    name: str,           # Tool name (e.g. "gtc_scrape")
    toolset: str,        # Toolset group (e.g. "groktocrawl")
    schema: dict,        # JSON schema for the tool
    handler: Callable,   # Async or sync function called when tool is invoked
    check_fn: Callable | None = None,    # Availability check (return False to hide)
    requires_env: list | None = None,    # Env vars that must be set
    is_async: bool = False,              # Whether handler is async
    description: str = "",
    emoji: str = "",
    override: bool = False,              # True to replace existing tool
) -> None

Plugin directory structure

~/.hermes/plugins/
  web/
    groktocrawl/           # <-- Our new plugin
      __init__.py          # register(ctx) entry point
      provider.py          # HTTP client + tool handlers
      manifest.yaml        # Plugin metadata (optional)

7. Native Tool Definitions (12 Tools)

These 12 tools will be registered as native Hermes tools. The agent (me) will be able to call them directly, just like web_search or terminal. Each tool maps to one or more GroktoCrawl API endpoints.

Tool 1: gtc_scrape

{
  "name": "gtc_scrape",
  "description": "Scrape a single URL and return clean markdown. Handles JS-rendered pages, PDFs, and site-specific adapters (GitHub, YouTube, etc.).",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {"type": "string", "description": "The URL to scrape"},
      "formats": {"type": "array", "items": {"type": "string"}, "default": ["markdown"], "description": "Output formats: markdown, html, links, rawHtml"},
      "only_main_content": {"type": "boolean", "default": true, "description": "Strip nav/header/footer boilerplate"},
      "timeout": {"type": "integer", "default": 30000, "description": "Per-page timeout in ms"}
    },
    "required": ["url"]
  }
}

Tool 2: gtc_crawl

{
  "name": "gtc_crawl",
  "description": "Crawl a website starting from a URL. Returns a job_id - poll with action='status'. Configurable depth, page limit, and path patterns.",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {"type": "string", "description": "Starting URL"},
      "action": {"type": "string", "enum": ["start", "status", "cancel", "errors", "active"], "default": "start"},
      "job_id": {"type": "string", "description": "Job ID for status/cancel/errors actions"},
      "max_depth": {"type": "integer", "default": 2},
      "limit": {"type": "integer", "default": 20, "description": "Max pages to crawl"},
      "includes": {"type": "array", "items": {"type": "string"}, "description": "Glob patterns to include"},
      "excludes": {"type": "array", "items": {"type": "string"}, "description": "Glob patterns to exclude"}
    },
    "required": ["url", "action"]
  }
}

Tool 3: gtc_search

{
  "name": "gtc_search",
  "description": "Search the web. Fast mode returns raw results (<1s). Rich mode scrapes top results and synthesizes with LLM (1-3s).",
  "parameters": {
    "type": "object",
    "properties": {
      "query": {"type": "string"},
      "limit": {"type": "integer", "default": 5},
      "search_type": {"type": "string", "enum": ["fast", "rich"], "default": "fast"},
      "sources": {"type": "array", "items": {"type": "string"}, "description": "Search verticals"},
      "categories": {"type": "array", "items": {"type": "string"}, "description": "SearXNG categories"}
    },
    "required": ["query"]
  }
}

Tool 4: gtc_agent

{
  "name": "gtc_agent",
  "description": "Start an autonomous research agent. It searches the web, scrapes sources, and synthesizes a cited answer. Use for complex research questions.",
  "parameters": {
    "type": "object",
    "properties": {
      "query": {"type": "string", "description": "Research question"},
      "action": {"type": "string", "enum": ["start", "status", "cancel"], "default": "start"},
      "job_id": {"type": "string", "description": "For status/cancel"},
      "max_searches": {"type": "integer", "default": 5, "description": "Max search rounds"},
      "model": {"type": "string", "description": "Override LLM model for this job"}
    },
    "required": ["query", "action"]
  }
}

Tool 5: gtc_answer

{
  "name": "gtc_answer",
  "description": "Answer a question with grounded citations. Searches, scrapes top results, and synthesizes with [N] citation markers. 1-3s latency.",
  "parameters": {
    "type": "object",
    "properties": {
      "query": {"type": "string"},
      "num_sources": {"type": "integer", "default": 5, "minimum": 1, "maximum": 20},
      "model": {"type": "string", "description": "Override LLM model"}
    },
    "required": ["query"]
  }
}

Tool 6: gtc_extract

{
  "name": "gtc_extract",
  "description": "Extract structured data from URLs using a JSON schema. LLM-powered extraction.",
  "parameters": {
    "type": "object",
    "properties": {
      "urls": {"type": "array", "items": {"type": "string"}},
      "action": {"type": "string", "enum": ["start", "status"], "default": "start"},
      "job_id": {"type": "string"},
      "output_schema": {"type": "object", "description": "JSON schema for extracted data"},
      "system_prompt": {"type": "string", "description": "Guide extraction behavior"}
    },
    "required": ["urls", "action"]
  }
}

Tool 7: gtc_map

{
  "name": "gtc_map",
  "description": "Discover all URLs on a website. Uses sitemap.xml, robots.txt, and HTML link extraction.",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {"type": "string"},
      "limit": {"type": "integer", "default": 50}
    },
    "required": ["url"]
  }
}

Tool 8: gtc_browser

{
  "name": "gtc_browser",
  "description": "Manage remote Playwright browser sessions. Create, execute actions (click/type/scroll/screenshot), list, and destroy.",
  "parameters": {
    "type": "object",
    "properties": {
      "action": {"type": "string", "enum": ["create", "list", "destroy", "execute"]},
      "session_id": {"type": "string", "description": "For destroy/execute"},
      "actions": {"type": "array", "items": {"type": "object"}, "description": "Browser actions for execute"},
      "ttl": {"type": "integer", "default": 300, "description": "Session TTL in seconds"}
    },
    "required": ["action"]
  }
}

Tool 9: gtc_monitor

{
  "name": "gtc_monitor",
  "description": "Manage change monitors. Create monitors that periodically check a URL and alert on content changes.",
  "parameters": {
    "type": "object",
    "properties": {
      "action": {"type": "string", "enum": ["create", "list", "get", "update", "delete"]},
      "monitor_id": {"type": "string"},
      "url": {"type": "string", "description": "URL to monitor (for create)"},
      "interval": {"type": "string", "default": "1h", "description": "Check interval"},
      "selector": {"type": "string", "description": "CSS selector to watch"}
    },
    "required": ["action"]
  }
}

Tool 10: gtc_parse

{
  "name": "gtc_parse",
  "description": "Parse a document file (PDF, EPUB, DOCX, PPTX, XLSX) to markdown.",
  "parameters": {
    "type": "object",
    "properties": {
      "file_path": {"type": "string", "description": "Local file path to parse"}
    },
    "required": ["file_path"]
  }
}

Tool 11: gtc_llmstxt

{
  "name": "gtc_llmstxt",
  "description": "Generate an llms.txt file for a website. Crawls and produces a structured summary for LLMs.",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {"type": "string"},
      "action": {"type": "string", "enum": ["start", "status"], "default": "start"},
      "job_id": {"type": "string"},
      "max_urls": {"type": "integer", "default": 10}
    },
    "required": ["url", "action"]
  }
}

Tool 12: gtc_batch_scrape

{
  "name": "gtc_batch_scrape",
  "description": "Scrape multiple URLs in batch. Returns a job_id for polling.",
  "parameters": {
    "type": "object",
    "properties": {
      "urls": {"type": "array", "items": {"type": "string"}},
      "action": {"type": "string", "enum": ["start", "status", "cancel", "errors"], "default": "start"},
      "job_id": {"type": "string"}
    },
    "required": ["urls", "action"]
  }
}

8. Wiring: How Hermes Calls GroktoCrawl

The plugin is a thin HTTP client. Each tool handler makes an HTTP request to the GroktoCrawl API and returns the response. Here is the complete plugin code:

~/.hermes/plugins/web/groktocrawl/__init__.py

"""GroktoCrawl web scraping plugin - self-hosted Firecrawl alternative."""

from __future__ import annotations
from plugins.web.groktocrawl.provider import GroktoCrawlProvider


def register(ctx) -> None:
    """Register all GroktoCrawl tools with the plugin context."""
    provider = GroktoCrawlProvider()
    provider.register_tools(ctx)

~/.hermes/plugins/web/groktocrawl/provider.py

"""GroktoCrawl provider - HTTP client + tool handlers."""

from __future__ import annotations
import os
import httpx
from typing import Any


def _get_api_url() -> str:
    return os.getenv("GROKTOCRAWL_API_URL", "http://localhost:8080").rstrip("/")


def _get_api_key() -> str | None:
    return os.getenv("GROKTOCRAWL_API_KEY")


def _headers() -> dict:
    h = {"Content-Type": "application/json"}
    key = _get_api_key()
    if key:
        h["Authorization"] = f"Bearer {key}"
    return h


def _is_available() -> bool:
    """Check if GroktoCrawl API is reachable."""
    try:
        r = httpx.get(f"{_get_api_url()}/health", timeout=3)
        return r.status_code == 200
    except Exception:
        return False


class GroktoCrawlProvider:
    """Registers 12 tools that proxy to the GroktoCrawl API."""

    def register_tools(self, ctx) -> None:
        ctx.register_tool(
            name="gtc_scrape",
            toolset="groktocrawl",
            schema={
                "type": "function",
                "function": {
                    "name": "gtc_scrape",
                    "description": "Scrape a URL to clean markdown.",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "url": {"type": "string"},
                            "formats": {"type": "array", "items": {"type": "string"}, "default": ["markdown"]},
                            "only_main_content": {"type": "boolean", "default": True},
                            "timeout": {"type": "integer", "default": 30000},
                        },
                        "required": ["url"],
                    },
                },
            },
            handler=self._scrape,
            check_fn=lambda: _is_available(),
            requires_env=["GROKTOCRAWL_API_URL"],
            is_async=True,
            description="Scrape a URL to markdown",
            emoji="spider",
        )
        # ... 11 more tools registered the same way
        # (gtc_crawl, gtc_search, gtc_agent, gtc_answer, gtc_extract,
        #  gtc_map, gtc_browser, gtc_monitor, gtc_parse, gtc_llmstxt,
        #  gtc_batch_scrape)

    async def _scrape(self, url, formats=None, only_main_content=True, timeout=30000):
        body = {"url": url, "formats": formats or ["markdown"],
                "onlyMainContent": only_main_content, "timeout": timeout}
        async with httpx.AsyncClient(timeout=60) as client:
            r = await client.post(f"{_get_api_url()}/v2/scrape",
                                   headers=_headers(), json=body)
            r.raise_for_status()
            return r.json()

    async def _crawl(self, url, action="start", job_id=None,
                     max_depth=2, limit=20, includes=None, excludes=None):
        base = _get_api_url()
        if action == "start":
            body = {"url": url, "maxDepth": max_depth, "limit": limit}
            if includes: body["includes"] = includes
            if excludes: body["excludes"] = excludes
            async with httpx.AsyncClient(timeout=30) as c:
                r = await c.post(f"{base}/v2/crawl", headers=_headers(), json=body)
                r.raise_for_status()
                return r.json()
        elif action == "status" and job_id:
            async with httpx.AsyncClient(timeout=30) as c:
                r = await c.get(f"{base}/v2/crawl/{job_id}", headers=_headers())
                r.raise_for_status()
                return r.json()
        elif action == "cancel" and job_id:
            async with httpx.AsyncClient(timeout=30) as c:
                r = await c.delete(f"{base}/v2/crawl/{job_id}", headers=_headers())
                r.raise_for_status()
                return r.json()
        elif action == "active":
            async with httpx.AsyncClient(timeout=30) as c:
                r = await c.get(f"{base}/v2/crawl/active", headers=_headers())
                r.raise_for_status()
                return r.json()

    async def _search(self, query, limit=5, search_type="fast",
                      sources=None, categories=None):
        body = {"query": query, "limit": limit, "searchType": search_type}
        if sources: body["sources"] = sources
        if categories: body["categories"] = categories
        async with httpx.AsyncClient(timeout=30) as c:
            r = await c.post(f"{_get_api_url()}/v2/search", headers=_headers(), json=body)
            r.raise_for_status()
            return r.json()

    async def _agent(self, query, action="start", job_id=None,
                     max_searches=5, model=None):
        base = _get_api_url()
        if action == "start":
            body = {"query": query, "maxSearches": max_searches}
            if model: body["model"] = model
            async with httpx.AsyncClient(timeout=30) as c:
                r = await c.post(f"{base}/v2/agent", headers=_headers(), json=body)
                r.raise_for_status()
                return r.json()
        elif action == "status" and job_id:
            async with httpx.AsyncClient(timeout=30) as c:
                r = await c.get(f"{base}/v2/agent/{job_id}", headers=_headers())
                r.raise_for_status()
                return r.json()

    async def _answer(self, query, num_sources=5, model=None):
        body = {"query": query, "numSources": num_sources}
        if model: body["model"] = model
        async with httpx.AsyncClient(timeout=60) as c:
            r = await c.post(f"{_get_api_url()}/v2/answer", headers=_headers(), json=body)
            r.raise_for_status()
            return r.json()

    async def _map(self, url, limit=50):
        async with httpx.AsyncClient(timeout=30) as c:
            r = await c.post(f"{_get_api_url()}/v2/map", headers=_headers(),
                              json={"url": url, "limit": limit})
            r.raise_for_status()
            return r.json()

    async def _browser(self, action, session_id=None, actions=None, ttl=300):
        base = _get_api_url()
        if action == "create":
            async with httpx.AsyncClient(timeout=30) as c:
                r = await c.post(f"{base}/v2/browser", headers=_headers(), json={"ttl": ttl})
                r.raise_for_status()
                return r.json()
        elif action == "list":
            async with httpx.AsyncClient(timeout=30) as c:
                r = await c.get(f"{base}/v2/browser", headers=_headers())
                r.raise_for_status()
                return r.json()
        elif action == "destroy" and session_id:
            async with httpx.AsyncClient(timeout=30) as c:
                r = await c.delete(f"{base}/v2/browser/{session_id}", headers=_headers())
                r.raise_for_status()
                return r.json()
        elif action == "execute" and session_id:
            async with httpx.AsyncClient(timeout=60) as c:
                r = await c.post(f"{base}/v2/browser/{session_id}/execute",
                                  headers=_headers(), json={"actions": actions or []})
                r.raise_for_status()
                return r.json()

    async def _parse(self, file_path):
        import pathlib
        p = pathlib.Path(file_path)
        if not p.exists():
            return {"error": f"File not found: {file_path}"}
        async with httpx.AsyncClient(timeout=120) as c:
            with open(p, "rb") as f:
                r = await c.post(f"{_get_api_url()}/v2/parse",
                                  headers={k: v for k, v in _headers().items()
                                           if k != "Content-Type"},
                                  files={"file": (p.name, f)})
                r.raise_for_status()
                return r.json()

9. Verification & Testing

Step 1: Verify GroktoCrawl is running

# Health check
curl http://localhost:8080/health
# Expected: {"status":"healthy"}

# Test scrape
curl -X POST http://localhost:8080/v2/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

Step 2: Verify Hermes plugin loads

# Check Hermes sees the plugin
hermes plugins list

# Should show: groktocrawl (web) - 12 tool(s)

# Check tools are registered
hermes tools list | grep gtc_

# Should show all 12 tools:
# gtc_scrape, gtc_crawl, gtc_search, gtc_agent, gtc_answer,
# gtc_extract, gtc_map, gtc_browser, gtc_monitor, gtc_parse,
# gtc_llmstxt, gtc_batch_scrape

Step 3: End-to-end test from Hermes

Ask the agent (me) to use the tools:

# In Hermes chat:
"Use gtc_scrape to scrape https://example.com and tell me what it says"
"Use gtc_search to search for 'rust async programming' with 3 results"
"Use gtc_map to find all URLs on https://docs.python.org"

Step 4: Check the portal dashboard

Open http://localhost:8082 in your browser to see job history, active crawls, and monitor status.

10. Operations: Startup, Logs, Updates

Daily operations

# Start the stack
cd C:\Users\Gaem4090\groktocrawl
docker compose up -d

# Stop the stack
docker compose down

# Check logs
docker compose logs -f agent-svc     # API logs
docker compose logs -f scraper-svc   # Scraper logs
docker compose logs -f slopsearx     # Search logs

# Check container status
docker compose ps

# Restart a single service
docker compose restart agent-svc

Updates

cd C:\Users\Gaem4090\groktocrawl
git pull
docker compose pull    # Pull new images
docker compose up -d   # Recreate changed containers

Resource usage

Troubleshooting