This is the full embedding model comparison behind acp ask feature: seven models, the metrics that make the comparison fair, and why I kept the default.
acp ask answers plain-English questions about my Claude Code usage. It is an optional feature of Claude Code Session Analytics, and Asking my Claude usage data questions in plain English explains how it works.
Why I compared models
acp ask matches a plain-English question to a curated catalog of SQL queries. A local sentence-embedding model, all-MiniLM-L6-v2, embeds the question and the catalog, and the closest row runs if its score is at least 0.5. I chose MiniLM because it is small and fast, not because I had compared it with anything.
The stress suite showed a specific failure: near misses. A near miss is a question that looks like a catalog row, but no row answers it — “tokens per hour” when the catalog has only “tokens per day”. If the retriever accepts it, ask runs a query that answers a different question, and the result looks correct.
The obvious hypothesis: a better embedding model separates these questions better. So I built a comparison to test it.
Building a fair comparison
scripts/ask_stress.pytakes a repeatable--model/-mflag;--compareruns MiniLM, bge-small and mpnet.make ask-stress-compareandmake ask-stress-prefetchwrap it.- Each model gets the same battery: recall on catalog paraphrases, false accepts on out-of-catalog questions, wrong accepts on a new near-miss set, encode latency, load time and disk size.
- The report has one comparison table (one row per model), a near-miss table per model, and a CSV with a
modelcolumn. - Models load one at a time and are released before the next one.
One practical trap: after the first cached model loads, the Hugging Face backend sets the process offline. A second model that is not in the cache then fails in the middle of the run. So each model is prefetched in its own process, and a multi-model run checks the cache first and stops with exit code 2 if a model is missing. The first version of that check treated HF_HUB_OFFLINE=0 as “offline is set” and skipped itself; now only 1, true, yes or on count.
Metrics that do not favour one model
A fixed threshold of 0.5 is not fair across models. It was tuned for MiniLM. Other models put their scores in a different range: at 0.5, bge-small, gte-small and bge-base accepted every out-of-catalog question (false-accept rate 1.0). Compared at 0.5, they look broken; they are only on a different scale.
So the comparison uses three threshold-fair metrics:
- Operating point t* — the lowest threshold at which the out-of-catalog false-accept rate is 10% or less. It is what you would set as that model’s threshold. Accept rate and near-miss wrong-accept are reported at t*.
- Separation AUC — the probability that a correct in-catalog match scores above an out-of-catalog or near-miss question. It needs no threshold.
- Near-miss wrong-accept — the share of near-miss questions whose top score clears the threshold. Nothing in the catalog answers them, so every accept is wrong. Near-miss questions never tune t*.
The near-miss set started with 15 questions. I removed two (“how many tokens per session”, “how many api calls per session”) because the catalog has per-session rows that arguably answer them. Borderline negatives would count correct answers as errors. The final set has 13.
Seven models
Three models ran with --compare, four more by hand: same seed, 28 catalog rows, 11 out-of-catalog questions, 13 near misses. These are retrieval-only numbers, so they differ from the 162-question held-out eval below.
| Model | recall@1 | t* | accept@t* | AUC | near-miss WA@t* | encode p50 ms |
|---|---|---|---|---|---|---|
| all-MiniLM-L6-v2 (default) | 0.923 | 0.435 | 0.923 | 0.791 | 1.000 | 5.4 |
| BAAI/bge-small-en-v1.5 | 0.936 | 0.843 | 0.846 | 0.822 | 0.615 | 7.6 |
| all-mpnet-base-v2 | 0.897 | 0.522 | 0.872 | 0.835 | 0.923 | 11.8 |
| all-MiniLM-L12-v2 | 0.897 | 0.598 | 0.885 | 0.807 | 0.769 | 7.0 |
| multi-qa-MiniLM-L6-cos-v1 | 0.923 | 0.508 | 0.923 | 0.833 | 0.923 | 7.8 |
| thenlper/gte-small | 0.923 | 0.905 | 0.859 | 0.830 | 0.769 | 7.7 |
| BAAI/bge-base-en-v1.5 | 0.910 | 0.843 | 0.615 | 0.749 | 0.538 | 12.3 |
Disk size for the three --compare models: MiniLM 87 MB, bge-small 128 MB, mpnet 418 MB. Two p95 encode times spiked (mpnet 308 ms, gte-small 317 ms) on 30 samples; they look like one-off outliers and I did not investigate them.
What the numbers say
- No model clearly beats the default. AUC spreads from 0.75 to 0.84 and recall@1 from 0.90 to 0.94. With 11 out-of-catalog and 13 near-miss questions, one question moves a rate by 8–9 points. These gaps are inside the noise.
- The best new candidate is
multi-qa-MiniLM-L6-cos-v1: the same recall and accept rate as the default and AUC 0.833. That is not enough to justify a model swap and a new threshold. - The lowest near-miss rate is not a win. bge-base wrongly accepts 54% of near misses, but only because its accept rate collapses to 62%. Its AUC is the worst of the seven.
- Every model accepts most near misses. The best is 54%; the default accepts all 13. At 0.5, all of them accept almost all.
The last point is the result. Embedding models put “read” and “write”, “shortest” and “longest”, “least” and “most” close together, because the sentences are almost the same. This is how sentence embeddings work, not a defect of one model. A different embedder does not fix it.
Decision: keep the comparison tooling, keep MiniLM, and keep the threshold.
What fixed it instead
I also built an end-to-end eval: 162 held-out questions that score the full ask decision, not only retrieval. It confirmed the result. On that test split, MiniLM had accuracy 69.1% and wrong-accept 22.2%; mpnet, the best at 0.5, had 73.5% and 17.3%. A few points, from a model almost five times larger on disk (418 MB vs 87 MB).
Two changes that were not about the model did more:
- Retrieval changes (paraphrases per row, a 0.7 cosine + 0.3 BM25 hybrid score, a spell check, a gap check between the top two rows, and a contrast guard for opposite terms): accuracy 69.1% → 79.0%, wrong-accept 22.2% → 9.9%. Near-miss wrong-accept moved only from 60% to 55%.
- Eleven new catalog rows for the near misses that real users would ask: tokens and cost per hour, tokens per week, output tokens per project, shortest and longest session. Ten test questions that had to be refused became answerable, and I changed their labels. Accuracy 82.7%, wrong-accept 5.6%, near-miss wrong-accept 15%.
The second change is honest about what it is: many near misses were not retrieval failures. They were missing queries. The fix for a question the catalog cannot answer is often to make it answerable.
Lessons
- Compare models at each one’s own operating point. Scores are not comparable across models; a shared threshold measures the scale, not the quality.
- Small eval sets cannot rank close models. Grow the out-of-catalog and near-miss sets before you trust a gap of a few points.
- Find the failure’s real cause before you change the model. The near misses were a property of embeddings and of the catalog, not of MiniLM.
- Keep fast tests on a fake backend. The unit tests use a hash-based fake embedder; real models run by hand, with
make ask-stress-compare.
How to run it
make ask-stress-prefetch downloads the models once (approximately 90 MB, 130 MB and 420 MB for MiniLM, bge-small and mpnet). make ask-stress-compare runs the three-model comparison. Add models with ASK_MODELS="…" or -m on scripts/ask_stress.py. The reports go to data/reports/, which is not tracked. Details are in the repository’s scripts guide on GitHub.