Can latent-space communication improve context transfer between Hermes, Codex, and Fable?
Promising concept Not practical today Adjacent research is directly useful
Neuralese — the idea that models communicate via raw latent vectors instead of human language — is a real and growing research area. The Coconut (Chain of Continuous Thought) paper (arXiv:2412.06769) proves that reasoning in latent space is not only possible but outperforms chain-of-thought on certain tasks while using fewer tokens. The foundational Translating Neuralese paper (arXiv:1704.06960) shows we can build translation layers between agent-internal vector messages and human language.
However, directly using Neuralese for Hermes⇄Codex⇄Fable handovers today is not practical for three reasons:
But the adjacent research IS directly applicable: The Handoff Debt paper (arXiv:2606.02875) shows that structured context handoffs reduce agent takeover costs by 42–63% in tokens. And KV-cache sharing research (PolyKV, QKVShare) demonstrates the infrastructure pattern for when you do control the models.
The term "Neuralese" was popularized by the 2017 paper "Translating Neuralese" (Andreas et al., arXiv:1704.06960) and refers to the dense vector representations that neural models use internally — the continuous, high-dimensional embeddings that carry meaning before it's decoded into discrete tokens.
Diego Dotta's Medium article frames it evocatively: Neuralese is the "most spoken language you'll never speak" — the private internal chatter of models, never designed for human consumption. He distinguishes three categories:
| Category | Description | Example |
|---|---|---|
| 1. Human languages | Messy, culture-soaked, built for wet brains | English, Portuguese, Python |
| 2. Machine protocols | Rule-bound, discrete, zero ambiguity | JSON over HTTPS, HTTP headers |
| 3. Latent representations | Continuous vector soup inside a model, never transmitted | Hidden states, embeddings |
The key insight from Dotta's article: when machines talk to each other today (Alexa → Roomba), they use category 2 (protocols). But as agents become more autonomous and learn on the fly, they could skip human language entirely and trade vectors — category 3. That's Neuralese.
Authors: Shibo Hao et al. (Meta/UCSD) | Published: Dec 2024, revised Nov 2025 | v3
Standard LLMs reason in "language space" — they generate a chain-of-thought (CoT) as word tokens. But most tokens in a CoT exist for grammatical fluency, not reasoning. The paper introduces Coconut (Chain of Continuous Thought): instead of decoding the last hidden state into a word token, feed it directly back as the next input embedding. The model reasons in continuous latent space, never passing through the language bottleneck.
Standard CoT:
Question → [token] → [token] → [token] → Answer
↑ decode ↑ decode ↑ decode
to word to word to word
Coconut:
Question → [hidden state] → [hidden state] → [hidden state] → Answer
↑ NO decode ↑ NO decode ↑ NO decode
(feed raw vector back as next input embedding)
The model switches between language mode (standard autoregressive token generation) and latent mode (feeding hidden states directly as next embeddings). Special tokens <bot> and <eot> mark the boundaries. Training uses a multi-stage curriculum: start with full CoT, then progressively replace reasoning steps with continuous thoughts (k×c latent thoughts replace each language step).
The paper cites Zhu et al. (2025a, 2025b), who developed a theoretical framework showing continuous CoT is more efficient than discrete CoT on certain tasks by encoding multiple reasoning paths in superposition states — and analyzed the training dynamics that explain how this superposition emerges.
Authors: Jacob Andreas et al. (UC Berkeley) | Published: Apr 2017, revised Dec 2018 | v5
This is the foundational paper for the term "Neuralese." It addresses a specific problem: when multi-agent RL systems learn to communicate through differentiable channels, the messages are opaque vectors. How do you interpret them?
The key insight: agent messages and natural language strings mean the same thing if they induce the same belief about the world in a listener. Instead of needing parallel data (which doesn't exist), they train a translation model that preserves the functional effect of messages — semantics and pragmatics — ensuring players communicating through a translation layer don't suffer substantial reward loss.
This paper is the closest academic precedent to the question you're asking. It establishes that:
But: This was done in a narrow RL setting (referential games) with small custom-trained agents. Scaling to frontier LLMs from different families is an open problem.
Authors: Dipesh KC, Anjila Budathoki | Published: June 2026
This paper studies exactly your use case: what happens when one coding agent takes over an interrupted task from another. It introduces the concept of "handoff debt" — the rediscovery cost imposed when a predecessor's work is opaque or incomplete.
The protocol interrupts a coding agent at deterministic handoff points, freezes the repository, and evaluates successor agents under four handoff views:
| Handoff View | What the successor sees |
|---|---|
| Repository state only | Just the code — no context about what was done or why |
| Raw trace | The full predecessor agent trace (all tool calls, outputs) |
| Summary notes | A natural-language summary of what was done |
| Structured notes | Structured notes with specific fields (goal, progress, blockers, next steps) |
Implication for Hermes⇄Codex⇄Fable: The biggest win isn't from Neuralese — it's from structured natural-language handoff notes. The paper shows that well-structured text summaries capture most of the benefit at a fraction of the cost.
Authors: Pratik Honavar, Tejpratap GVSL | Published: May 2026
This is the closest thing to "Neuralese for agent handoffs" in practice. QKVShare studies how to hand off latent context (KV caches) between agents efficiently.
Critical limitation for your use case: QKVShare works between agents running the same model (Llama-3.1-8B → Llama-3.1-8B). The KV cache is model-specific — you cannot inject a GLM KV cache into a GPT model. This is the fundamental barrier to Neuralese-based handoffs between different model families.
Authors: Ishan Patel, Ishan Joshi | Published: April 2026
PolyKV takes the KV-sharing concept further: multiple concurrent agents share a single compressed KV cache pool. Keys quantized at int8, Values at 3-bit via Walsh-Hadamard Transform + Lloyd-Max quantization.
Same limitation: same-model only. But it demonstrates the infrastructure pattern for when you do control the models.
Your three agents run on different model families:
| Agent | Model Family | Hidden Dim | Tokenizer | API Access |
|---|---|---|---|---|
| Hermes | GLM-5.2 (via Featherless) | GLM-specific | GLM BPE | Token-level only |
| Codex | GPT-5.6-sol (OpenAI) | GPT-specific | GPT BPE | Token-level only |
| Fable | Claude Fable 5 (Anthropic) | Claude-specific | Claude BPE | Token-level only |
Neuralese requires a shared latent space. These three models have:
A hidden state vector from GLM-5.2 is meaningless if injected into GPT-5.6. It's like trying to play a Blu-ray disc in a DVD player — the physical format is incompatible.
To make Neuralese work across model families, you'd need:
This is a multi-million-dollar research program, not an engineering task.
The Handoff Debt paper (2606.02875) directly answers your question with data. The biggest efficiency gains come not from latent-space transfer, but from well-structured natural-language handoff documents:
The paper tested four handoff views and found that structured notes (with fields like goal, progress, blockers, next steps) and summary notes both dramatically outperform raw traces and repo-only handoffs. The raw trace — which is the closest analog to "passing everything" — is actually less efficient than a good summary because the successor has to parse through noise.
Here's where each approach falls on the efficiency/feasibility trade-off for your specific use case:
| Approach | Efficiency Gain | Cross-Model? | Feasible Today? | Investment |
|---|---|---|---|---|
| Repo-only handoff (successor reads code, no context) |
Baseline (worst) | Yes | Yes | None |
| Raw trace handoff (full predecessor log) |
Marginal | Yes | Yes | Low |
| Summary notes (NL summary of work done) |
42–63% token reduction | Yes | Yes | Low |
| Structured notes (goal/progress/blockers/next) |
42–63% token reduction | Yes | Yes | Low |
| Prompt compression (LLMLingua, PCToolkit) |
2–20× compression | Yes | Yes | Medium |
| KV-cache sharing (QKVShare, PolyKV) |
2.6× TTFT, 97% memory | Same model only | If self-hosted | High |
| Coconut (latent reasoning) | Fewer tokens, BFS | Single model | Requires training | Very High |
| Cross-model Neuralese (latent vector handoff) |
Theoretical max | Requires alignment layer | No | Research program |
Adopt structured handoff notes as the standard format for Hermes⇄Codex⇄Fable transitions.
Based on the Handoff Debt paper's findings, create a standard handoff document with these fields:
## Handoff Document
- Task: [original goal]
- Status: [completed | in-progress | blocked]
- Progress: [what was accomplished, with file/line refs]
- Blockers: [what failed and why]
- Next Steps: [ordered list of immediate actions]
- Key Decisions: [architectural/design choices made]
- Files Modified: [list with brief change descriptions]
- Tests: [what passes, what fails, what's missing]
- Context: [non-obvious things the successor needs to know]This alone should give you 40–60% token reduction on agent takeovers, based on the paper's data.
Add prompt compression for the raw trace. Tools like LLMLingua-2 (arXiv:2403.12968) can compress prompts 2–20× while preserving semantic content. When you do need to pass a full predecessor trace, compress it first. This is model-agnostic and works through APIs.
If you ever move to self-hosted models (e.g., Qwen3.5-9B on the RTX 4090, which you're already evaluating for Nagato), then KV-cache sharing becomes viable. QKVShare and PolyKV show that you can transfer compressed KV caches between agents running the same model with minimal quality loss. This would eliminate re-prefill costs entirely for same-model handoffs.
But this only works when both agents run the same model — not across GLM/GPT/Claude.
Cross-model latent alignment is an active research area. The "Translating Neuralese" approach (belief-preserving translation) is the theoretical foundation, but nobody has demonstrated it at frontier-model scale. If a paper emerges showing cross-family KV-cache transfer with acceptable quality loss, that would change the calculus. Until then, structured text remains the best cross-model handoff format.
One finding from Dotta's Medium article deserves special attention for your use case. He ran a simple experiment: asking non-reasoning models the "Sally has 3 brothers" question under different constraints:
| Test | Approach | Tokens | Latency | Correct? |
|---|---|---|---|---|
| 1. Direct | Just ask | 8 | 0.87s | No |
| 2. JSON schema | Force step-by-step in JSON | 164 | 2.56s | Yes |
| 3. 4-letter words | Constrain vocab to ≤4 letters | 64 | 1.63s | Yes |
Test 3 used 60% fewer tokens than test 2 and was faster, while still getting the right answer. The "language" the model reasoned in was a stripped-down, minimalist English — "bros" and "sis" instead of "brothers" and "sisters." No fluff.
This is a practical bridge between full Neuralese and full natural language: a compressed, efficiency-optimized reasoning dialect. It's not vectors, but it captures the spirit — less language overhead, more reasoning signal per token. This could be directly applied to handoff notes: a compressed format that prioritizes information density over readability.
Is Neuralese useful for Hermes⇄Codex⇄Fable handovers?
Not directly, not today. The models are from different families with incompatible latent spaces, and none expose hidden states through their APIs. Cross-model Neuralese transfer is an unsolved research problem.
Is it efficient?
Theoretically yes, practically no. Coconut proves latent reasoning is more token-efficient than CoT. But the efficiency gains are locked behind custom training and same-model constraints.
What should you do instead?
Structured handoff notes. The Handoff Debt paper provides hard evidence that well-structured natural-language handoff documents reduce successor agent costs by 42–63% in tokens and 20–59% in events. This works across any model family, requires zero infrastructure, and captures most of the practical benefit today.
When to revisit: If you move to self-hosted same-model agents (Qwen on the 4090), KV-cache sharing becomes viable. If cross-model latent alignment research breaks through at frontier scale, Neuralese handoffs become possible. Until then, the best "Neuralese" is a really good handoff document.