Anatomy of a sovereign chatbot
A two-brain design — a small router model for intent and tools, a RAG brain for knowledge — running entirely on hardware we own.
- Chatbot
- LLMs
- RAG
- Sovereignty
A chatbot that has to be trusted with real data can’t phone a foreign API for every turn. Ours run on the client’s own GPUs, and split the work across two “brains”: a small, fast model that decides what kind of turn this is, and a knowledge brain that actually answers.
The two-brain flow
-
Message
user turn + history
-
Router
small model: intent + tools
-
Branch
tool call · knowledge · chit-chat
-
RAG brain / tools
grounded answer or action
-
Guardrail
ground-check, PII, scope
-
Reply
streamed to the user
Why split it
Routing is a cheap, latency-sensitive classification problem; generation is an expensive, quality-sensitive one. Forcing one large model to do both wastes GPU and slows every reply. A small instruct model (2–3B) handles intent detection and tool selection in milliseconds; only genuine knowledge questions reach the heavier generation path — which is the citation-enforced RAG engine described separately.
The models
| Job | What we run | Why |
|---|---|---|
| Routing / tools | Small instruct model (2–3B) | Fast, cheap, good enough to classify intent and emit tool calls |
| Generation | Mid-size open model, grounded mode | Quality answers, constrained to retrieved sources |
| Embeddings | Multilingual embedding model | Indian-language corpora retrieve as well as English |
| Serving | Ollama / vLLM behind LiteLLM | One gateway, local-first, hosted only if justified |
How it was built
- No egress. Every model is served locally; the LiteLLM gateway routes to hosted providers only for non-sensitive, opt-in tasks, with cost tracked per call.
- Streaming + timeouts. Replies stream token-by-token; each stage has a deadline and a graceful fallback, so a slow tool never hangs the conversation.
- Guardrails on the way out. Before a reply leaves, it passes a scope and grounding check — the bot stays inside its remit and cites when it should.
The same pattern scales down to a 2B router on a modest box and up to a multi-GPU generation tier — but the contract never changes: the data stays home.
Building something like this?
We'd rather talk specifics than slogans. Tell us what you're trying to build.
Talk to us