Nexus AI

The Problem With Black-Box Assistants

Most chat assistants give you an answer and nothing else. You can't see whether it came from a web page, one of your own files or the model's memory, and you can't tell how confident it is. For anyone checking a policy document or making a real decision, that is the part that matters most.

The first version of Nexus, which I built in March 2026, had the same problem, and a more urgent one. It defaulted to llama-3.1-8b-instant on Groq, and when Groq shut down its Llama models for free-tier keys on 16 August 2026, the app could no longer answer anything. I used the rebuild in September to fix both: an assistant that shows its route, its sources and its confidence, on models that are actually available.

Constraints I Set

Cost came first. It had to run on free tiers: Groq for inference, the Hugging Face Inference API for embeddings and DuckDuckGo for search, with no paid vector database. Groq's free plan allows roughly 8,000 tokens per minute per model, so every extra LLM call had to justify itself.

Second, it had to work as an open trial with no sign-up. That meant isolating strangers from each other without accounts, and capping usage per IP address: 25 messages a day, 15 per chat, 5 uploads and 30 voice recordings. Third, the backend had to sleep when idle, so nothing could depend on an always-on process.

Architecture: A Six-Node LangGraph Pipeline

Every message runs through a LangGraph state graph with six nodes: classify, retrieve, web_search, deliberate, generate and decide. Conditional edges route each question to documents, the web or straight to the writer, and in Think mode the decide node runs in parallel with generate. The FastAPI backend turns LangGraph's event stream into Server-Sent Events: node starts and ends with timings and token counts, sources, thinking notes, answer tokens, and a final done or error event.

• Next.js 16 frontend: chat UI, live pipeline view, decision board, and a local cache of chats and usage
• FastAPI API: SSE streaming, per-IP quotas with refunds, uploads, and voice transcription via Whisper
• LangGraph graph: routing, retrieval, deliberation, writing and decision scoring as separate nodes
• Groq models: GPT-OSS 20B as the default writer and for extraction, GPT-OSS 120B for Think mode, Qwen 3.8 27B as a preview option
• JEV by TypeSafe: question routing and option scoring through typed Choice and Score questions
• JSON store on a persistent volume: chats, documents with embeddings, memory and encrypted API keys, per workspace

Why JSON Files on a Volume Over a Database

The original build used Supabase with pgvector. For a trial with no accounts, a database was mostly overhead: another service to provision, secure and pay for once the free tier ran out. I replaced it with JSON files on a single persistent volume, one folder per workspace, where a workspace ID is a uuid5 of a random browser ID plus the visitor's IP. Writes go through a temporary file and an atomic os.replace, guarded by a per-workspace asyncio lock.

Document search moved in-process: a line-aware chunker splits each page into windows of up to 900 characters with a two-line overlap, vectors are normalised at upload, and retrieval is a cosine scan returning the top 4 passages (6 in documents mode) above a 0.2 similarity threshold. The trade-off is explicit: this only works with one backend instance, which is why the deployment is pinned to a single replica. A linear scan is fine for 20 documents per workspace, but it is the first thing I would replace at scale.

Why a Decision Model Scores the Options, Not the LLM

For Think mode I wanted a decision board: options down one side, criteria across the top, every cell scored. My first plan treated JEV as another chat model behind an OpenAI-compatible endpoint. Reading TypeSafe's API reference showed it is a decision model: you POST state and typed questions to /v1/systemone and get back probabilities and a confidence for each answer, not generated text.

So I split the job. GPT-OSS 20B, in JSON mode, extracts the options and criteria from the reasoning notes, retrying on GPT-OSS 120B if the output can't be parsed. JEV then scores every option against every criterion on a five-level scale and picks a winner, all in one batched request. I also use JEV for routing, accepting its answer only above 0.55 confidence and otherwise falling back to the Groq router, so the scores come with calibrated confidence instead of an LLM's guess and routing no longer spends Groq tokens, at the cost of a second external dependency.

The Hardest Problem: A Reply That Silently Vanished

A test message came back as an empty reply: the Nexus avatar and no text, no error, and a trial message deducted. The backend log showed the request finishing, so I ran the streaming generator directly in a script with the same placeholder Groq key, and the traceback appeared at once. Inside the generator, a local variable named usage held token counts, which made Python treat usage as local throughout the function. When the model call failed, the error handler called usage.refund on the trial-quota module and raised UnboundLocalError instead.

That crash inside the error handler meant no error event was ever sent, the stream simply closed, and the refund never ran. Worse, the linter's auto-fix had removed the module import it now saw as unused, so the bug survived a clean lint. I fixed it at three levels: the module is now imported under an alias that can't collide, error cleanup runs inside its own guarded block, and the frontend treats any stream that ends without done or error as a failure with a Try Again button. Generation also got a 180-second timeout, so a hung provider can never leave the UI waiting forever.

Results

In local testing with real keys, all seven Groq model options answered correctly, and JEV scored a full decision board of 18 option-criterion cells plus its pick in one request of about 0.4 seconds. Loading the OpenAI and Anthropic SDKs only when a bring-your-own-key model is used cut backend import time from 1.2 to 1.6 seconds down to about 0.7, which matters because every cold start after the backend sleeps pays it. A page visit now makes zero backend requests until someone sends a message, which I verified by recording network traffic in Playwright.

The codebase is about 3,500 lines of Python and 9,800 lines of TypeScript across 14 commits.

What I Would Do Differently

I would write tests for the failure paths before the happy paths. The vanished-reply bug and a later one, where GPT-OSS occasionally tries to call a built-in tool and Groq rejects the response, were both found by running the app by hand. A test that forces a provider error and checks that the stream ends with an error event would have caught the first one on day one.

I would also verify external constraints before designing around them. The Llama shutdown broke the first version, Groq's 8,000 tokens-per-minute limit shaped how many calls Think mode can afford, and I initially planned JEV as a chat model. Ten minutes with each provider's current docs would have saved a round of rework on all three.