acp ask answers plain-English questions about my Claude Code usage. It matches the question to a catalog of tested SQL queries with a small local embedding model, all-MiniLM-L6-v2.
I chose that model at first because it was lightweight and fast, but without running any benchmarks against other models.
Then the stress suite showed me questions like “tokens per hour”. The catalog has only “tokens per day”. The model said: close enough. ask ran the daily query, and the answer looked correct.
My first idea: a better embedding model could fix this, but a wider range of models for comparison was needed. So I tested seven.
A fair race is harder than it looks
The first problem was the threshold. ask accepts a match at a score of 0.5 — a value tuned for MiniLM. Other models use a different scale. At 0.5, three of them accepted every question that was not in the catalog. They looked broken, but they were only on a different scale.
So each model got its own operating point: the lowest threshold at which at most 10% of out-of-catalog questions are accepted. I also measured how well each model separates good matches from bad ones without any threshold (AUC), and how many near misses it accepts — questions that look like a catalog row, but no row answers.
The results
These are retrieval-only numbers on a small comparison set, so they differ from the 162-question held-out eval in the first post. Same seed, 28 catalog rows, 11 out-of-catalog questions, 13 near misses (runs on 2026-10-18):
| Model | recall@1 | AUC | near misses accepted |
|---|---|---|---|
| all-MiniLM-L6-v2 (default) | 92.3% | 0.79 | 100% |
| bge-small-en-v1.5 | 93.6% | 0.82 | 62% |
| all-mpnet-base-v2 | 89.7% | 0.84 | 92% |
| all-MiniLM-L12-v2 | 89.7% | 0.81 | 77% |
| multi-qa-MiniLM-L6-cos-v1 | 92.3% | 0.83 | 92% |
| gte-small | 92.3% | 0.83 | 77% |
| bge-base-en-v1.5 | 91.0% | 0.75 | 54% |
No model clearly wins. With 11 and 13 questions, one question moves a rate by 8–9 points, so these gaps are noise. bge-base accepts the fewest near misses only because it accepts too little of everything. And the most important column: every model accepts most near misses.
The model was not the problem
To a sentence-embedding model, “shortest session” and “longest session” are almost the same sentence. So are “least” and “most”, “read” and “write”. That is how embeddings work. A different model does not change it.
An end-to-end eval on 162 held-out questions confirmed it. The best other model, mpnet, raised accuracy from 69.1% to 73.5% — but it is almost five times larger on disk. Two changes that were not about the model did much more:
- Better retrieval: paraphrases, a keyword score mixed with the embedding score, typo correction, a gap check between the top two matches, and a guard for opposite words. Accuracy 69.1% → 79.0%, wrong answers 22.2% → 9.9%.
- Eleven new catalog queries for the near misses that people really ask, like tokens per hour or the shortest session. Accuracy 82.7%, wrong answers 5.6%.
Many near misses were not retrieval failures. They were missing queries.
So I kept MiniLM and kept the threshold.
What I learned
- Compare models at their own threshold. A shared threshold measures the score scale, not the quality.
- Small test sets cannot rank close models. Count what one question is worth before you trust a gap.
- Find the cause before you change the model. Here the cause was embeddings in general and a catalog with gaps.
- Sometimes the fix is more data, not a better model. Better retrieval and eleven new SQL queries did more than any of the seven models.
More information
Deep dive with the full table and metrics: Seven embedding models, no winner.
How acp ask works: Asking my Claude usage data questions in plain English.
Project page: Claude Code Session Analytics.
Code: GitHub.
