Dashboards are great for a general visual overview, but what if we just want answers to specific questions as “How many tokens were spent per model?”
The obvious answer today is to let a model write the SQL. I wanted to try a different local-first approach. So I created the acp ask feature, by default, never writes SQL — and in that default mode it runs fully offline.
Choose SQL, do not write it
ask has a catalog of tested queries, each with example questions and paraphrases. A small local embedding model, mixed with a keyword score, compares my question with the examples and runs the best match, read-only. If no match is confident enough, or two matches are too close, it does not guess: it shows the nearest questions. It also fixes typos against the catalog vocabulary, and a contrast guard checks for opposite words like “least” and “most”.
Refuse instead of guessing
“Which model is used least last month” is very similar to “which model is used least”. But the second query has no date filter. Running it would give an answer that looks right and is wrong.
So each catalog query declares the filters it can bind: session, date, project, model, top N. ask parses the same filters from the question. If the match can not bind one of them, it refuses.
Read-only method
ask must only read. For security reasons, and to prevent unintended alterations or deletions of data, the tool must
- uses the PostgreSQL parser itself (
pglast): the statement must parse to read-only statements only. - connects as a SELECT-only database role, so the database says no as well.
How good is it?
I measure the whole decision — match, filters, refusal — on a held-out set of 162 questions that I never tune on:
- accuracy 82.7%,
- wrong answers (ran a query that does not answer the question): 5.6%,
- refused a question it could answer: 11.7%,
- correct query ranked first: 93.3%; in the top 3: 99.2%.
The weak spots are clear: it refuses too many paraphrases (26.5% of 68), it accepts 15% of the near misses — questions that only look like a catalog question — and opposites are still the hardest case: 5 of 12 got the wrong query, even with the contrast guard.
And when I do want a model to write SQL
There is an opt-in LLM mode, acp ask --llm. It gives Claude the closest catalog examples and the schema, and Claude writes one SELECT. The SQL is always printed and goes through the same guard and the same read-only role. This mode is not offline: it sends the question, the examples and the schema to the Anthropic API, and with --summarize also up to 50 result rows. It needs an ANTHROPIC_API_KEY and plain acp ask never calls the API.
What I learned
- Retrieval beats generation for a small domain. A tested catalog can not invent a column.
- Refusing is a feature. A wrong answer that looks right is worse than no answer.
- Stress-test the guard, not only the model. The worst defect was in the safety check.
- Keep a held-out set. Thresholds tuned on the test questions look better than they are.
More information
acp ask needs the optional nlp dependency group (uv sync --group nlp) and runs on the host; the app image does not include it.
Deep dive: Asking my Claude usage data questions in plain English. Project page: Claude Code Session Analytics. Code: GitHub.
