The chat demo at the bottom of this page, dissected

We advise businesses on AI integrations, and until this week the most sophisticated AI feature on our own website was a paragraph describing one. That gap bugged us. A consultancy that sells working software should be able to point at a working thing. So this page now ends with a real chat: it streams tokens from a live model, it is rate-limited and cost-capped, and if nobody has configured a key, it says so quietly and gets out of your way. This essay is the build notes.

What is actually running

The chat is a small React island hydrated on scroll. Under it sits one server route that speaks Server-Sent Events. The glue is TanStack AI — @tanstack/ai on the server, @tanstack/ai-react in the island — talking to OpenRouter, which fronts a cheap open-weight model. The model id is a constant on one line of the route. Changing it is a one-line diff, which is the point: nothing about the model is load-bearing architecture.

The server route is deliberately boring. It checks for an API key and returns a 503 “sleeping” status without one. It rate-limits per IP: ten requests per five minutes, an old-fashioned counter, no cleverness. It rejects bodies over 16 KB. It forwards at most the last twelve messages of the conversation, caps the reply at 300 tokens, and scopes the system prompt to this essay’s topics — ask it about anything else and it declines politely. Every bound in that list exists because the endpoint is public and we pay per token.

Why a sleeping demo is a feature

The key is read at request time, so the same build serves the waking and sleeping states. Deploys without the secret work unchanged; the island shows a single mono-font line instead of a broken widget. We like this shape generally: a public demo should degrade into an explanation, not an error. The sleeping state is also the honest state — this feature costs money per message, and the site should not pretend otherwise.

The pre-1.0 tax, itemised

TanStack AI is young — the versions we pinned are pre-1.0 — and we want to be specific about what that cost, because “it’s early days” is the kind of sentence that hides real work:

• The model id parameter is typed against a list of OpenAI model names. An OpenRouter id fails typecheck even though the wire format is identical, so the route carries one cast and a comment explaining it.
• The output-token cap is named max_tokens on the chat-completions wire but typed after OpenAI’s newer Responses-API spelling, which cost another one-line cast.
• The request body must satisfy a strict agent-input shape — thread id, run id, empty tool and context arrays — with no schema library to explain why. The validation errors point at a migration doc, not at the field.

None of this is a complaint. We pinned exact versions, read the types when the docs drifted, and got a streaming chat with a smaller surface than any framework we have replaced. That is the trade a pre-1.0 library offers: rough edges in exchange for leverage, and the types are good enough to find the edges before runtime. We would make the same call again for a demo this size. For anything longer-lived we would wrap the library behind our own interface, so the swap cost when 1.0 lands is one file.

The ownership angle

Everything on this page is code in our repository: the island, the route, the tests that assert the failure paths call the model zero times, the contrast-checked styling. No third-party embed script, no rented widget with someone else’s analytics, no data leaving through a vendor we cannot inspect. If we part ways with TanStack AI or OpenRouter tomorrow, the wire format is plain SSE and the replacement is an afternoon.

That is the same test we apply to client work. A demo you cannot own is a commercial for someone else’s platform. A demo you own is evidence.

Our fine-tuning service is the grown-up version of this demo: models trained on your data, deployed where your systems run, with the weights and evals in your hands.