cbebc32ce7
"Local" was conflated with "on this host". The premise of this router is that a local model classifies the task before a cloud model answers it, and that does not require the GPU to be in the machine you are typing on -- usually it isn't. Most developers already have WireGuard or a VPN back to a home lab, so the normal shape is router and editor on the laptop, Ollama on the workstation. classifier.base_url already accepted any OpenAI-compatible endpoint, but two things stopped it working. The API key was hardcoded to the literal "ollama". Fine for Ollama, which ignores it, and a 401 for anything that checks. classifier.api_key_env now names the env var to read, and is left unset for the Ollama case. A key_env that is set but missing from the environment raises at client construction rather than being passed through: classify() catches everything and degrades to a fallback tier, so an auth failure would otherwise be indistinguishable from a slow model, forever. The bigger one: local verification derived its URL by stripping /v1 off classifier.base_url. It speaks Ollama's NATIVE /api/chat, which is the only way to set think=False, so it cannot follow the classifier to a cloud provider -- and moving the classifier anywhere at all silently redirected it. It now has its own verification.base_url and verification.model. verification.model may stay null only while both run on one host, which is the common case including across a VPN, since both point at the same Ollama. Once the hostnames differ, config load REFUSES the null. That guard exists because the failure it prevents is silent, and was observed rather than imagined: with the classifier pointed at NeuralWatt, the verifier POSTed deepseek-v4-flash to localhost:11434, 404d, caught it, logged "local verification unavailable" and recorded no sample. Verification would have looked enabled while producing nothing -- the same shape as the harness bugs this project keeps finding. Verified end to end over a genuine non-loopback address, both paths: /route classified with source=classifier, and the verifier POSTed to the remote host with the right model. Ollama binds 127.0.0.1 by default, so deploy/ollama-over-vpn.conf carries the drop-in for the serving host. It binds the VPN address rather than 0.0.0.0 deliberately: Ollama has no authentication of any kind, and 0.0.0.0 publishes it on whatever wifi the laptop is sitting on. Same reasoning as the dispatcher's loopback bind. Recorded while measuring this, because it undercuts an assumption the design rests on: a cloud endpoint satisfies the classifier interface too, and did it better. Five prompts, same system prompt, temperature 0 -- NeuralWatt's deepseek-v4-flash averaged 1.02s against qwen3.5's 11.58s on an RTX 6000, agreed with the label 5/5 against 2/4, and cost $0.093 per thousand calls. qwen3.5's one hard failure was the documented runaway-thinking-trace mode: 15.76s, no JSON, silent fallback tier. Local inference is not free, it is unbilled. The shipped default stays local Ollama, because switching spends quota and that is a deployment choice rather than a code one. Tests 256 -> 264. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ
2.1 KiB
2.1 KiB