alee/6krrt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(classifier): record why the classifier declined, and bound the degraded-warn knobs #112

Merged
alee merged 2 commits from feat/classifier-observability into main 2026-10-05 03:08:46 +00:00
Conversation 0 Commits 2 Files Changed 16 +1068 -55
alee commented 2026-10-05 02:45:42 +00:00
Owner
Copy Link

Summary

local_decision (qwen3.5:4b) was measured degrading about 30% of live turns on 2026-10-04, and the database could not say how close each declined answer was to the threshold: route_decisions.confidence is exactly 1.0 on 43,804 of 43,856 rows because the chat path re-routes through the override branch, which hard-codes it. This records the classifier's own attempt so confidence_min can be tuned from data instead of a guess.

  • New ClassifierAttempt {confidence, coverage, reject_reason} on Classification, persisted as route_decisions.classifier_confidence, classifier_coverage, classifier_reject. Old rows stay NULL.
  • Reason codes: below_confidence_min, below_coverage_min, no_logprobs, timeout, transport_error, parse_error, primary_failed, skipped_<why>. They live in src/classifier_rejection.py (stdlib only) and are raised from local_decision.parse_logprobs as ClassifierRejected, a RuntimeError subclass, so the existing cascade handlers are unchanged.
  • The /metrics degraded-share warning now says why the classifier declined.
  • Decisions page: a tooltip with the attempt, and an amber badge for session_history / session_stale so replayed labels are visible.
  • classifier.degraded_warn_threshold must be in (0, 1] and degraded_warn_min at least 1, refused at config load. The bounds are named constants in config.py so a runtime admin write, which skips Pydantic, can import them instead of restating them.

No cascade behaviour changes. This does not touch confidence_min, session_history age, or the static fallback; those wait on the data this collects.

Operational notes

  • The three columns are added by ALTER at startup, so the live DB gains them on the first restart after merge. The first rows to carry values are the next real traffic.
  • config/config.yaml is unchanged. The live overlay has degraded_warn_threshold: 0.2 (0.5 never fired at a 30% degraded day); that is machine-local.

Test plan

  • scripts/verify_commit.py --repo <wt> --base origin/main --full HEAD exits 0: 2644 tests, no new lint.
  • tests/test_classifier_attempt.py: reason mapping for each raise site, chat-path capture, persistence on a migrated and an unmigrated DB, config validators, and the bound constants pinned to the validators' edges.
  • Golden route_decision_no_header.json updated for the three new keys.
  • After merge and restart: read classifier_confidence split by classifier_reject on real Atlas/Prometheus traffic before touching confidence_min.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KkCGRantZsSwmcFpet6FTa

## Summary `local_decision` (qwen3.5:4b) was measured degrading about 30% of live turns on 2026-10-04, and the database could not say how close each declined answer was to the threshold: `route_decisions.confidence` is exactly 1.0 on 43,804 of 43,856 rows because the chat path re-routes through the override branch, which hard-codes it. This records the classifier's own attempt so `confidence_min` can be tuned from data instead of a guess. - New `ClassifierAttempt {confidence, coverage, reject_reason}` on `Classification`, persisted as `route_decisions.classifier_confidence`, `classifier_coverage`, `classifier_reject`. Old rows stay NULL. - Reason codes: `below_confidence_min`, `below_coverage_min`, `no_logprobs`, `timeout`, `transport_error`, `parse_error`, `primary_failed`, `skipped_<why>`. They live in `src/classifier_rejection.py` (stdlib only) and are raised from `local_decision.parse_logprobs` as `ClassifierRejected`, a `RuntimeError` subclass, so the existing cascade handlers are unchanged. - The `/metrics` degraded-share warning now says why the classifier declined. - Decisions page: a tooltip with the attempt, and an amber badge for `session_history` / `session_stale` so replayed labels are visible. - `classifier.degraded_warn_threshold` must be in (0, 1] and `degraded_warn_min` at least 1, refused at config load. The bounds are named constants in `config.py` so a runtime admin write, which skips Pydantic, can import them instead of restating them. No cascade behaviour changes. This does not touch `confidence_min`, `session_history` age, or the static fallback; those wait on the data this collects. ## Operational notes - The three columns are added by `ALTER` at startup, so the live DB gains them on the first restart after merge. The first rows to carry values are the next real traffic. - `config/config.yaml` is unchanged. The live overlay has `degraded_warn_threshold: 0.2` (0.5 never fired at a 30% degraded day); that is machine-local. ## Test plan - [x] `scripts/verify_commit.py --repo <wt> --base origin/main --full HEAD` exits 0: 2644 tests, no new lint. - [x] `tests/test_classifier_attempt.py`: reason mapping for each raise site, chat-path capture, persistence on a migrated and an unmigrated DB, config validators, and the bound constants pinned to the validators' edges. - [x] Golden `route_decision_no_header.json` updated for the three new keys. - [ ] After merge and restart: read `classifier_confidence` split by `classifier_reject` on real Atlas/Prometheus traffic before touching `confidence_min`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KkCGRantZsSwmcFpet6FTa
alee added 2 commits 2026-10-05 02:45:43 +00:00
feat(classifier): record the classifier's own attempt on route_decisions 92031dbd0b 
A declined answer left one number behind and it was in a log line, so
confidence_min could not be tuned from data. route_decisions.confidence cannot
say it either: the chat path re-routes through the override branch, which
hard-codes 1.0, and 43,804 of 43,856 live rows hold exactly that.

Measured on 2026-10-04 under local_decision (qwen3.5:4b): 769 of 2,557 turns
(30%) had no fresh classification, against 0% for local_llm and local_encoder.
The cause was recorded in only 4 of them.

- classifier_confidence, classifier_coverage and classifier_reject on
  route_decisions, filled from a ClassifierAttempt carried on Classification.
  Both accepted and declined answers carry one, so the two distributions can be
  compared around the floor. Reason codes are listed in docs/data-model.md.
- ClassifierRejected (a RuntimeError subclass, messages unchanged) replaces the
  plain RuntimeErrors at the six floor-miss sites and the two local_decision
  refusals, so the number and reason travel out of the raise site.
- The chat path captures the classifier's verdict before the re-route and passes
  it to persist_route_decision (attempt_of), like it already does for source.
- Admin decisions page: the source badge's tooltip shows the attempt, and
  session_history / session_stale get an amber badge instead of neutral grey.
  TUI detail popup and the live event carry the same three keys.
- The /metrics degraded-share warning lists the recorded reasons instead of
  claiming the classifier "has been failing", which was wrong for a classifier
  that answers and is declined.
- degraded_warn_threshold must be in (0, 1] and degraded_warn_min at least 1,
  refused at load: a value above 1 could never fire.

The three columns arrive by ALTER and are NULL on every earlier row; metrics
selects them only when present, so the live DB reads NULL until its restart.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkCGRantZsSwmcFpet6FTa
refactor(config): name the degraded-warn bounds so a runtime write can import them 4c00f2fee5 
A runtime admin write goes straight past Pydantic, so the registry's bounds
are the only check between a request body and a field the warning reads. The
validators inlined `0 < v <= 1` and `v >= 1`, so a registry entry would have
had to restate them and could drift: it would accept 0.0 where load refuses.

Name the three edges (exclusive low, high, min floor), use them in the
validators, and pin them to the validators' actual boundary behaviour.
No behaviour change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkCGRantZsSwmcFpet6FTa
alee merged commit 0125c445aa into main 2026-10-05 03:08:46 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
alee
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: alee/6krrt#112
Delete Branch "feat/classifier-observability"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?