PartialUpdate × Featherless AI Integration Plan

Fork philholden/partialupdate locally, remove Cloudflare AI Gateway dependency, add a generic OpenAI-compatible provider pointing at Featherless AI (GLM-5.2, 256K context) with Valkey cross-process concurrency coordination.

1. Context & Current State

2. Concurrency Problem Analysis

Featherless plans reserve concurrent inference capacity rather than billing per token. Requests have model-size-derived concurrency costs, and a request returns HTTP 429 if it would push used_cost above limit. Without coordination, three apps (Hermes, GTC, partialupdate) hitting Featherless simultaneously will collide.

3. Chosen Architecture

Request Flow

1. User submits prompt → Durable Object enqueues in llmQueue
2. drainLlmQueue() picks up next item (serial, 1 at a time per chat)
3. Valkey preflight: EVAL Lua reserve script with cost=4 (GLM-5.2), prefix pu:{doId}:{uuid}
4. If Valkey says no (capacity full) → wait + retry with backoff (max 30s)
5. If Valkey says yes → fetch() Featherless /v1/chat/completions with stream: true
6. Stream SSE chunks through existing parseSseTextStream() → UpdateStreamParser → WebSocket broadcast
7. Valkey release: EVAL Lua release script with reservation ID (in finally block)
8. If Valkey is unreachable → fail open (proceed without coordination, serial queue is backstop)

4. Implementation Plan

Phase 1: Add openai-direct Provider TypeScript

// Add to MODEL_PROVIDER union:
| "openai-direct"

// Add new fields to AppEnv:
OPENAI_API_KEY?: string;
OPENAI_BASE_URL?: string;      // default: https://api.featherless.ai/v1
OPENAI_MODEL?: string;          // default: zai-org/GLM-5.2
OPENAI_MAX_TOKENS?: number | string;
OPENAI_TEMPERATURE?: number | string;
OPENAI_MODEL_SETTINGS?: Record<unknown> | string;

async function* streamOpenAIDirectResponse(
  env: AppEnv,
  messages: LlmMessage[],
): AsyncIterable<string> {
  const baseUrl = env.OPENAI_BASE_URL || "https://api.featherless.ai/v1";
  const model = env.OPENAI_MODEL || "zai-org/GLM-5.2";
  const maxTokens = numberEnv(env.OPENAI_MAX_TOKENS, 8192);
  const temperature = numberEnv(env.OPENAI_TEMPERATURE, 0.7);
  const modelSettings = parseGatewayModelSettings(env.OPENAI_MODEL_SETTINGS);

  const response = await fetch(`${baseUrl}/chat/completions`, {
    body: JSON.stringify({
      ...modelSettings,
      messages: gatewayChatMessagesFromMessages(messages),
      model,
      stream: true,
      max_tokens: maxTokens,
      temperature,
    }),
    headers: {
      Authorization: `Bearer ${env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    method: "POST",
  });

  if (!response.ok || !response.body) {
    console.warn("OpenAI direct did not return a stream", {
      model, status: response.status, statusText: response.statusText,
      body: await response.text().catch(() => ""),
    });
    yield fallbackUpdate(lastUserMessage(messages), "");
    return;
  }

  yield* parseSseTextStream(response.body);
}

if (provider === "openai-direct" && env.OPENAI_API_KEY) {
  yield* streamOpenAIDirectResponse(env, messages);
  return;
}

5. Valkey Coordination Design

Valkey Connection

// Cloudflare Workers don't have a native Redis client, but we can
// use fetch() to talk to Valkey via its HTTP interface, or use a
// TCP socket via connect() from "cloudflare:sockets".
//
// Option A: Valkey RESP over TCP (cloudflare:sockets)
//   - Workers support outbound TCP via cloudflare:sockets
//   - Implement minimal RESP protocol (INLINE commands)
//   - EVAL script as a single RESP array
//
// Option B: Redis HTTP proxy (if available)
//   - Some Redis setups expose an HTTP interface
//   - Not standard for Valkey, would need a sidecar
//
// Recommended: Option A (TCP via cloudflare:sockets)

Reservation ID Format

Reserve Lua Script (same as Hermes/GTC)

-- KEYS[1] = featherless:concurrency:held
-- KEYS[2] = featherless:concurrency:slots
-- KEYS[3] = featherless:concurrency:slot_costs
-- ARGV[1] = cost (4 for GLM-5.2)
-- ARGV[2] = reservation_id
-- ARGV[3] = now (unix ms)
-- ARGV[4] = stale_seconds (1800)
-- ARGV[5] = remote_limit (from Featherless /account/concurrency)
-- ARGV[6] = remote_used (from Featherless /account/concurrency)

-- 1. Garbage-collect stale reservations
local stale_cutoff = tonumber(ARGV[3]) - (tonumber(ARGV[4]) * 1000)
local stale = redis.call('ZRANGEBYSCORE', KEYS[2], '-inf', stale_cutoff)
for _, sid in ipairs(stale) do
  local scost = redis.call('HGET', KEYS[3], sid)
  if scost then
    redis.call('DECRBY', KEYS[1], tonumber(scost))
    redis.call('HDEL', KEYS[3], sid)
  end
end
redis.call('ZREMRANGEBYSCORE', KEYS[2], '-inf', stale_cutoff)

-- 2. Check capacity
local held = tonumber(redis.call('GET', KEYS[1]) or '0')
local limit = tonumber(ARGV[5])
local used = tonumber(ARGV[6])
local available = limit - used - held

if available < tonumber(ARGV[1]) then
  return 0  -- denied
end

-- 3. Reserve
redis.call('INCRBY', KEYS[1], tonumber(ARGV[1]))
redis.call('ZADD', KEYS[2], tonumber(ARGV[3]), ARGV[2])
redis.call('HSET', KEYS[3], ARGV[2], tonumber(ARGV[1]))
return 1  -- granted

Release Lua Script (same as Hermes/GTC)

-- KEYS[1] = featherless:concurrency:held
-- KEYS[2] = featherless:concurrency:slots
-- KEYS[3] = featherless:concurrency:slot_costs
-- ARGV[1] = reservation_id

local cost = redis.call('HGET', KEYS[3], ARGV[1])
if not cost then return 0 end  -- already released or unknown

redis.call('DECRBY', KEYS[1], tonumber(cost))
redis.call('ZREM', KEYS[2], ARGV[1])
redis.call('HDEL', KEYS[3], ARGV[1])
return 1

Fail-Open Behavior

Remote Limit/Used Refresh

Before each reserve, partialupdate fetches GET https://api.featherless.ai/account/concurrency to get limit and used_cost. This is a lightweight call (JSON snapshot). The result is cached for 5 seconds to avoid excessive polling.

6. Configuration

wrangler.jsonc vars

"vars": {
  "MODEL_PROVIDER": "openai-direct",
  "OPENAI_BASE_URL": "https://api.featherless.ai/v1",
  "OPENAI_MODEL": "zai-org/GLM-5.2",
  "OPENAI_MAX_TOKENS": 8192,
  "OPENAI_TEMPERATURE": 0.7,
  // Valkey coordination
  "VALKEY_URL": "localhost:6390",
  "VALKEY_CONCURRENCY_STALE_SECONDS": 1800,
  // ... existing vars
}

.dev.vars (secrets — not committed)

OPENAI_API_KEY=rc_8fc960939d6331abb6a20e6b6d09178f344b1ddf942037bb0f1629d078539e3b

.env.example additions

# openai-direct provider (Featherless, OpenRouter, local Ollama, etc.)
MODEL_PROVIDER=openai-direct
OPENAI_API_KEY=
OPENAI_BASE_URL=https://api.featherless.ai/v1
OPENAI_MODEL=zai-org/GLM-5.2
OPENAI_MAX_TOKENS=8192
OPENAI_TEMPERATURE=0.7

# Valkey cross-process concurrency coordination (optional but recommended)
# If unset, partialupdate fails open (no coordination with Hermes/GTC)
VALKEY_URL=localhost:6390
VALKEY_CONCURRENCY_STALE_SECONDS=1800

7. Testing & Verification

Phase 1: Provider Works (no Valkey)

1. npm install in partialupdate repo
2. Set .dev.vars with Featherless API key
3. npm run dev — Worker starts on http://localhost:8787
4. Open browser, send a prompt, verify LLM response streams as HTML
5. Check debug page (/c/chat/debug) — verify LLM messages contain GLM-5.2 responses
6. Verify SSE streaming works (progressive HTML rendering)

Phase 2: Valkey Coordination

1. Verify Valkey is running: docker exec groktocrawl-valkey-1 redis-cli ping → PONG
2. Check current ledger: redis-cli -p 6390 GET featherless:concurrency:held
3. Send prompt from partialupdate → verify held increments by 4 during call
4. After response completes → verify held returns to previous value
5. Simulate capacity full: manually SET featherless:concurrency:held 8 (if limit is 8)
6. Send prompt → verify partialupdate waits/retries (not 429)
7. Clear: SET featherless:concurrency:held 0
8. Kill Valkey: docker stop groktocrawl-valkey-1
9. Send prompt → verify partialupdate still works (fail-open)
10. Restart Valkey: docker start groktocrawl-valkey-1

Phase 3: Cross-App Coordination

1. Start a long Hermes chat (triggers Featherless call)
2. While Hermes is generating, send prompt from partialupdate
3. Verify partialupdate waits for Hermes' slot to release before proceeding
4. Check Valkey: redis-cli -p 6390 ZRANGE featherless:concurrency:slots 0 -1 WITHSCORES
5. Verify both hermes:... and pu:... reservation IDs appear

8. Risks & Mitigations

Key Technical Uncertainty: cloudflare:sockets

Implementation Priority