bakeoff — one leaderboard for accuracy, cost, latency and calibration

2252 training and 748 held-out synthetic support tickets over 14 intents, seed 20260831. Every arm gets the same split. Sort by macro-F1, then read the cost and latency columns beside it: only the two prompted arms carry a bill, and only they are measured in seconds rather than milliseconds. The local arms retrain for real on every re-run. The prompted arms and the fine-tuned encoder replay predictions from caches shipped with the repo and are badged cached, so nothing on this page calls an API or loads a 250 MB model.

arm	macro-F1	accuracy	ECE	p50 ms	p95 ms	cost / 1k	fit s
finetuned-encoder	0.653	0.684	0.083	7.1	21.9	free	2887.80	cached, trained offline
tfidf-linearsvc-raw	0.624	0.668	0.408	0.8	1.2	free	0.16
tfidf-logreg	0.619	0.654	0.059	1.0	1.1	free	0.42
tfidf-linearsvc	0.614	0.662	0.081	5.9	7.6	free	0.34
llm-fewshot	0.611	0.658	0.251	1511.7	2726.6	$0.0630	0.04	cached, no training
char-nb	0.602	0.635	0.351	0.7	0.9	free	0.29
embed-logreg	0.584	0.610	0.140	0.3	0.4	free	0.32
embed-knn	0.584	0.635	0.088	6.4	7.8	free	0.01
llm-zeroshot	0.527	0.560	0.289	1612.6	5731.4	$0.0313	0.00	cached, no training
keyword-rules	0.331	0.316	0.177	0.0	0.0	free	0.00

Trained and scored keyword-rules, tfidf-logreg, char-nb, tfidf-linearsvc, tfidf-linearsvc-raw, finetuned-encoder, embed-logreg, embed-knn, llm-zeroshot, llm-fewshot in 6.87 s.