cbebc32ce7
"Local" was conflated with "on this host". The premise of this router is that a local model classifies the task before a cloud model answers it, and that does not require the GPU to be in the machine you are typing on -- usually it isn't. Most developers already have WireGuard or a VPN back to a home lab, so the normal shape is router and editor on the laptop, Ollama on the workstation. classifier.base_url already accepted any OpenAI-compatible endpoint, but two things stopped it working. The API key was hardcoded to the literal "ollama". Fine for Ollama, which ignores it, and a 401 for anything that checks. classifier.api_key_env now names the env var to read, and is left unset for the Ollama case. A key_env that is set but missing from the environment raises at client construction rather than being passed through: classify() catches everything and degrades to a fallback tier, so an auth failure would otherwise be indistinguishable from a slow model, forever. The bigger one: local verification derived its URL by stripping /v1 off classifier.base_url. It speaks Ollama's NATIVE /api/chat, which is the only way to set think=False, so it cannot follow the classifier to a cloud provider -- and moving the classifier anywhere at all silently redirected it. It now has its own verification.base_url and verification.model. verification.model may stay null only while both run on one host, which is the common case including across a VPN, since both point at the same Ollama. Once the hostnames differ, config load REFUSES the null. That guard exists because the failure it prevents is silent, and was observed rather than imagined: with the classifier pointed at NeuralWatt, the verifier POSTed deepseek-v4-flash to localhost:11434, 404d, caught it, logged "local verification unavailable" and recorded no sample. Verification would have looked enabled while producing nothing -- the same shape as the harness bugs this project keeps finding. Verified end to end over a genuine non-loopback address, both paths: /route classified with source=classifier, and the verifier POSTed to the remote host with the right model. Ollama binds 127.0.0.1 by default, so deploy/ollama-over-vpn.conf carries the drop-in for the serving host. It binds the VPN address rather than 0.0.0.0 deliberately: Ollama has no authentication of any kind, and 0.0.0.0 publishes it on whatever wifi the laptop is sitting on. Same reasoning as the dispatcher's loopback bind. Recorded while measuring this, because it undercuts an assumption the design rests on: a cloud endpoint satisfies the classifier interface too, and did it better. Five prompts, same system prompt, temperature 0 -- NeuralWatt's deepseek-v4-flash averaged 1.02s against qwen3.5's 11.58s on an RTX 6000, agreed with the label 5/5 against 2/4, and cost $0.093 per thousand calls. qwen3.5's one hard failure was the documented runaway-thinking-trace mode: 15.76s, no JSON, silent fallback tier. Local inference is not free, it is unbilled. The shipped default stays local Ollama, because switching spends quota and that is a deployment choice rather than a code one. Tests 256 -> 264. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ
42 lines
2.1 KiB
Plaintext
42 lines
2.1 KiB
Plaintext
# Serve this host's Ollama to the other machines you work from.
|
|
#
|
|
# The premise of this router is that a local LLM classifies your task before a
|
|
# cloud model answers it. "Local" means your hardware, not necessarily the
|
|
# machine you are typing on — the box with the GPU usually is not the laptop.
|
|
# Most developers already have WireGuard or a VPN back to a home lab, so the
|
|
# normal shape is: router and editor on the laptop, Ollama on the workstation,
|
|
# classifier.base_url pointing across the tunnel.
|
|
#
|
|
# Ollama listens on 127.0.0.1:11434 by default, so that fails with
|
|
# connection-refused until this drop-in is applied on the SERVING host.
|
|
#
|
|
# Install (needs root — ollama.service is a system unit):
|
|
# sudo mkdir -p /etc/systemd/system/ollama.service.d
|
|
# sudo cp deploy/ollama-over-vpn.conf \
|
|
# /etc/systemd/system/ollama.service.d/override.conf
|
|
# sudo nano /etc/systemd/system/ollama.service.d/override.conf # set the address
|
|
# sudo systemctl daemon-reload && sudo systemctl restart ollama
|
|
#
|
|
# Then on the client machine, in config.yaml:
|
|
# classifier.base_url: http://<vpn-ip-of-this-host>:11434/v1
|
|
# verification.base_url: http://<vpn-ip-of-this-host>:11434
|
|
#
|
|
# Bind to the VPN address, NOT 0.0.0.0. Ollama has no authentication of any
|
|
# kind: anything that can reach the port can run inference, enumerate your
|
|
# models and pull new ones. 0.0.0.0 also publishes it on whatever cafe or
|
|
# hotel wifi the laptop is sitting on. Same reasoning that keeps the
|
|
# dispatcher itself on loopback.
|
|
#
|
|
# Replace with this host's own VPN address — 10.x.x.x for WireGuard,
|
|
# 100.x.x.x for Tailscale. `ip -4 -o addr show` will tell you.
|
|
|
|
[Service]
|
|
Environment="OLLAMA_HOST=10.0.0.1:11434"
|
|
|
|
# Keep the model resident between requests. Without this, the first
|
|
# classification after an idle gap pays a cold load — measured at 43s, which
|
|
# exceeds even the 120s ceiling once the SDK's retries are counted. Serving a
|
|
# laptop makes this matter more, not less: requests arrive in bursts separated
|
|
# by long gaps, which is exactly the pattern that keeps evicting the model.
|
|
Environment="OLLAMA_KEEP_ALIVE=24h"
|