This demo sleeps when nobody is using it. The first page load after a quiet spell can take a few seconds to wake up.

bakeoff

2252 training and 748 held-out synthetic support tickets over 14 intents, seed 20260831. Every arm gets the same split. Sort by macro-F1, then read the cost and latency columns beside it: only the two prompted arms carry a bill, and only they are measured in seconds rather than milliseconds. The local arms retrain for real on every re-run. The prompted arms and the fine-tuned encoder replay predictions from caches shipped with the repo and are badged cached, so nothing on this page calls an API or loads a 250 MB model.

Arms
Sort
arm macro-F1accuracyECE p50 msp95 mscost / 1k fit s
finetuned-encoder 0.653 0.684 0.083 7.1 21.9 free 2887.80 cached, trained offline
tfidf-linearsvc-raw 0.624 0.668 0.408 0.7 0.9 free 0.15
tfidf-logreg 0.619 0.654 0.059 0.8 0.9 free 0.35
tfidf-linearsvc 0.614 0.662 0.081 5.1 5.9 free 0.28
llm-fewshot 0.611 0.658 0.251 1511.7 2726.6 $0.0630 0.03 cached, no training
char-nb 0.602 0.635 0.351 0.6 0.6 free 0.23
embed-logreg 0.584 0.610 0.140 0.2 0.3 free 0.25
embed-knn 0.584 0.635 0.088 4.6 5.1 free 0.01
llm-zeroshot 0.527 0.560 0.289 1612.6 5731.4 $0.0313 0.00 cached, no training
keyword-rules 0.331 0.316 0.177 0.0 0.0 free 0.00

All 10 arms came from this session's cache. Press Re-run to fit them again.

Pick one for me

The best arm by macro-F1 that fits inside both ceilings. This is the decision the leaderboard exists to support.

Escalate what it is not sure about

Answer above a confidence threshold and send the rest to a person. The first pair of columns is what you get by trusting the arm's stated confidence at 0.95. The next is the threshold actually needed to hold the target. Compare tfidf-linearsvc with tfidf-linearsvc-raw: same model, one calibrated, and only one of them gives a threshold you can act on.

Pick an arm from the leaderboard to see its calibration and its worst slices.

Source · Linux x86_64, Python 3.12.14 · run 2026-10-31T00:49:31+00:00