This page describes acp ask, an optional feature of the Claude Code Session Analytics project.
The idea: two modes
I wanted to ask the database questions in plain English, for example “how many tokens were spent per model?”. acp ask has two modes:
- Catalog mode (default). It does not write SQL. It chooses SQL from a curated catalog of tested queries and runs it read-only. It is fully offline: no network access and no API key.
- LLM mode (opt-in,
--llm). It retrieves catalog examples and asks Claude to write the SQL. This is retrieval-augmented generation, and it sends data to the Anthropic API (see below).
Most of this page is about catalog mode.
How catalog mode matches a question
- Each catalog row has a question, paraphrases, a description, the SQL, and the filters (slots) that the SQL can bind.
- A spell check against the catalog vocabulary fixes typos first (
sesions→sessions). - Each row gets the best score over its texts: 0.7 × cosine similarity from a local sentence-embedding model (
all-MiniLM-L6-v2) plus 0.3 × BM25 over character 3–5-grams. - A match is accepted only if the top score is at least 0.5 and leads the next row by at least 0.03. Otherwise it does not guess: it prints the nearest questions.
- A contrast guard demotes a row whose question says the opposite of the question asked: least/most, read/write, input/output, cost/tokens.
- With
--sqlit shows the matched query; with--no-executeit only previews it.
I also tried a cross-encoder re-ranker. On the tuning set it scored worse than the fused scores: it is trained to match a query to a passage, not a question to a question. It stays off by default (--rerank).
Slots, and refusing instead of guessing
The question is parsed for filters: a session ID (or a bare UUID), since 2026-01, last N days, last month (the previous calendar month, as a closed date range), a known project or model, top N.
If the best match can not bind a filter that the question has, ask refuses. “Which model is used least last month” matches the all-time “which model is used least” query — running it would give a wrong answer that looks right. Values bind as query parameters, never as SQL text.
The read-only guard
ask runs only read-only SQL, the stress suite found a critical bypass: a dollar-quoted string could hide ; DROP TABLE from the check.
The applied fix replaced the regex with a parse-tree check (pglast, the PostgreSQL parser): the statement must parse to read-only statements only. The same change fixed three false rejections.
A second layer is in the database: ask connects as ask_reader, a role with SELECT only, read-only sessions and a 15-second statement timeout.
Measuring the whole decision
An end-to-end eval scores the full ask decision — slots, retrieval and refusal — against the real catalog. The cases are in two splits: a tune split, the only one on which a threshold may be chosen, and a frozen, held-out test split. Each case has a kind: paraphrase, near miss (similar, but no catalog query answers it), opposite, out of domain, typo, slot, unbindable.
Baseline on the test split (162 questions, all-MiniLM-L6-v2):
- accuracy 82.7%,
- wrong-accept 5.6% — it ran a query that does not answer the question,
- over-refusal 11.7% — it refused a question the catalog can answer,
- recall@1 93.3%, recall@3 99.2%.
Typos, out-of-domain and unbindable questions scored 100%. The weak spots:
- paraphrases are refused too often (over-refusal 26.5%, n=68)
- near misses are accepted 15% of the time (n=20)
- opposite questions 41.7% (n=12).
A pytest check compares each run with this baseline: it fails if accuracy drops, or wrong-accept rises, by more than 0.02.
A separate stress suite checks robustness to perturbations (typos, casing, truncation, unicode noise), latency at 100, 1,000 and 10,000 catalog rows, the read-only guard against an adversarial corpus, and compares embedding models (MiniLM, bge-small, mpnet by default). Both are opt-in, because they load a real model.
LLM mode, and what leaves the machine
acp ask --llm is a non-local mode and uses the Anthropic API
It retrieves the top 5 catalog examples and the schema rows they mention, and asks Claude to write one SELECT. The SQL is always printed, goes through the same pglast guard, gets LIMIT 200, and runs as ask_reader. It may only read claude.* objects that the model was shown.
This mode sends the question, the catalog examples and the schema to the Anthropic API. --summarize also sends up to 50 result rows. --no-execute stops after printing the SQL.
It needs the llm and nlp dependency groups (uv sync --group llm --group nlp), ANTHROPIC_API_KEY, and an explicit ASK_DATABASE_URL (no fallback to the owner role). The model is set with ACP_ASK_LLM_MODEL. A missing key or package is an error, never a silent fallback to catalog mode. Plain acp ask never calls the API.
Future improvements
- Paraphrases are refused too often, and near-miss and opposite questions are sometimes accepted.
- The catalog is small and hand-written.
Links
Project page: Claude Code Session Analytics
Code: GitHub