Before you ship a live-web feature, test it on questions your users will ask. Choose 20 from one workflow and define what each answer must include. Then check whether the answer covers the question and whether its sources support the claims.
The Python evaluation kit handles the run records and review files. It includes 20 candidate questions for an API-documentation assistant and one live pilot run. That pilot shows why coverage and source support need separate checks.
Define what a useful answer includes
For a documentation assistant, select questions that test five common problems:
| Category | What to check |
|---|---|
| Source discovery | Finding the relevant official documentation |
| Exact contract | Returning the right fields, parameters, and limits |
| Synthesis | Combining pages without losing qualifications |
| Freshness and version scope | Using the requested date, version, or plan |
| Unresolved evidence | Recognizing facts the sources cannot establish |
We use four questions per category. Adjust the mix to your workflow, including straightforward lookups as well as harder research tasks.
Write the required answer elements before running each question. For example:
What fields does Ollama’s web-search API return for each result, and does its
contentfield contain the full page?
A complete answer must name title, url, and content, explain that content contains a relevant snippet, and cite the official response definition. The Ollama documentation provides the reference:
content (string): relevant content snippet from the web page
The kit stores each question in JSON Lines. Here is Q05, expanded for readability:
{
"id": "Q05",
"category": "exact_contract",
"question": "What fields does Ollama's web-search API return for each result, and does its content field contain the full page?",
"required_elements": [
{
"id": "E1",
"criterion": "Name title, url, and content as the result fields."
},
{
"id": "E2",
"criterion": "Describe content as a relevant snippet, not the full page."
},
{
"id": "E3",
"criterion": "Cite the official response definition."
}
],
"reference_sources": [
{
"url": "https://docs.ollama.com/capabilities/web-search",
"passage": "content (string): relevant content snippet from the web page",
"retrieved_at_utc": "2026-10-06T17:37:54Z",
"date_or_version": "Documentation as retrieved; no version identified in this passage."
}
],
"evidence_status": "reviewed",
"critical_failure_conditions": [
"States that web_search returns full page content."
]
}The critical failure condition identifies an error your application should reject even when the rest of the answer is correct. Set these rules yourself and keep them consistent across implementations.
Review the evidence, then freeze the set
Open each reference and save the relevant passage, URL, and retrieval time. Record the source’s version or date when available. For time-sensitive questions, specify the date in the question rather than asking only for “current” information.
Only Q05 has reviewed evidence in the candidate set. The other 19 questions remain pending, so the kit refuses to run the full set. Review their references before including them in scored results.
Freezing records the dataset version and hash. The kit rejects a changed dataset or evidence older than the freeze record allows, which defaults to 30 days. Once the committed Q05 evidence is more than 30 days old, refresh it and freeze a new version before you run the pilot.
If a source changes, update the reference under a new dataset version. Do not rewrite the expected answer to fit a result you have already seen.
Run a one-question pilot
The kit extends the traced Python Research example, which saves the report, ordered sources, and request timeline. Start with one question to check the output and review process before expanding the run.
The recorded pilot used macOS arm64, Python 3.12.13, uv 0.11.6, and the tabstack Python SDK 2.8.5. Create a key in the Tabstack console, then run:
git clone https://github.com/Mozilla-Ocho/tabstack-cited-research-python.git
cd tabstack-cited-research-python
git checkout 460b72e174187796eae5707b7a028aa13b8af09c
uv sync --frozen
export TABSTACK_API_KEY=... # Set your key locally; do not commit it.
cp acceptance-evals/pilot/questions.pilot-q05.jsonl acceptance-evals/pilot/questions.my-pilot.jsonl
uv run cited-research-accept freeze \
--dataset acceptance-evals/pilot/questions.my-pilot.jsonl \
--version my-pilot-v1 \
--allow-subset
uv run cited-research-accept run-one \
--dataset acceptance-evals/pilot/questions.my-pilot.jsonl \
--question Q05 \
--run acceptance-evals/runs/my-pilot-q05 \
--mode fast \
--nocache \
--silence-timeout 120 \
--deadline 300 \
--pilotrun-one disables SDK and application retries, so each attempt represents one request. It sends the question and a neutral output instruction to Research; the expected answer elements stay in reviewer-side files. The --pilot flag keeps this run out of scored totals.
The committed pilot’s terminal output includes:
complete report -> acceptance-evals/runs/20261006-pilot-q05/answers/Q05-a1/report.md
sources (3) -> acceptance-evals/runs/20261006-pilot-q05/answers/Q05-a1/sources.json
1. Web search https://docs.ollama.com/capabilities/web-search
2. Web search https://docs.ollama.com/capabilities/web-search.md
3. Testing Ollama Web Search 🔍 and a Thinking Model 💭 https://dev.to/aairom/testing-ollama-web-search-and-a-thinking-model-1dh7
review sheet -> acceptance-evals/runs/20261006-pilot-q05/answers/Q05-a1/review-sheet.csv (unreviewed)
attempt Q05-a1: complete (exit 0), 3 sources -> acceptance-evals/runs/20261006-pilot-q05/attempts.jsonlUse a new run directory. The kit refuses to overwrite an existing attempt; --another-attempt creates a separate record such as Q05-a2.
Check coverage and source support
Prepare the review files:
uv run cited-research-accept prepare-review \
--run acceptance-evals/runs/my-pilot-q05In reviews/coverage.csv, score each required answer element. In reviews/claims.csv, check individual claims against the cited passages. Both use the same scale:
| Score | Coverage | Source support |
|---|---|---|
2 |
Covers the element fully | Supports the claim at its stated scope |
1 |
Covers part or misses a qualifier | Supports part or requires a qualifier |
0 |
Omits or contradicts the element | Does not support or contradicts the claim |
U |
Unresolved after review | Uninspected or inaccessible evidence |
Leave a cell blank if you have not reviewed it yet.
Record the answer passage and your reason for each score. The citation review guide explains the source checks in more detail.
The Q05 pilot answered the main question correctly:
Crucially, the
contentfield within each web search result contains a relevant content snippet from the web page, not the full page content [1][2][3].
The first review scored all three required elements 2. But the answer also added a sentence saying that web_fetch retrieves the full content of a page. The documentation says main content, so the claim review scored that statement 1. The review split that sentence into two claims.
That extra claim did not affect coverage because the question did not ask about web_fetch. It still needs correction before a developer relies on it.
A second, blind review that could not see the first scores caught one more. The answer says the search API “returns an array” of results, but the documentation says it returns an object containing a results array. That claim also scored 1, so three of the five claims scored 2.
Sources [1] and [2] are HTML and Markdown versions of the same official page, not independent evidence. Source [3] was not opened, and the review records that gap. The CLI preserves source order so each citation number continues to point to the correct page.
An AI agent did both reviews; the second could not see the first one’s scores. The final decision remains blank because the pilot defined only a critical failure condition, not full release criteria.
Read the summary without hiding failures
After completing the reviews, run:
uv run cited-research-accept summarize \
--run acceptance-evals/runs/my-pilot-q05The summary separates completed requests from accepted answers. It includes failed requests in the attempt total and shows unfinished reviews separately. For coverage and support scores, it uses inspected rows and leaves U and blank cells unscored.
For example, the synthetic offline test has six attempts: three completed and three failed. One answer was accepted, giving 1/6 accepted attempts. Coverage was 2/4 because two of the four inspected elements were fully covered. The remaining U and blank cells stayed visible but unscored. These are test-fixture results, not live API performance.
Cost remains unavailable until you attach authoritative usage records in reviews/usage.csv. Research streams model token counts but no per-request credit record. Include billed failures and retries; do not treat missing usage as zero.
Handle the stream’s error event explicitly. A client timeout stops the wait but does not establish server-side cancellation. The attempt log preserves these outcomes so a failed request cannot disappear from the evaluation.
Expand after the pilot
The test receipt reports 255 passing tests, with Ruff and Pyright passing. The tests cover dataset validation, terminal states, incomplete reviews, source order, and summary calculations.
Only one question has run live. All failure tests use offline fixtures. Balanced mode, --fetch-timeout, and --another-attempt remain untested against the API, as do Linux, Windows, and reproduction by a second engineer. Cost calculations use synthetic values only. Twenty questions can reveal specific defects, but they cannot establish production reliability or provider superiority.
Finish reviewing the references for your remaining questions, then use the results to choose the next engineering change. Wrong version scope calls for better version selection. Unsupported claims call for a better evidence path or narrower wording. Slow responses call for a separately evaluated configuration or a different interaction.
Keep private questions and credentials out of the public dataset. Review the Research guide, Trust page, and Privacy Notice before using hosted processing for a sensitive workflow.
Run the kit on one question and complete both reviews before expanding to the full set.
