Before you ship a feature that answers questions from the live web, define which answers it can accept and what happens when a request fails. Review the evidence for answer quality alongside timing, cost, data handling, and the team’s ability to disable the feature.
A documentation assistant makes the distinction clear. It can cite a real release page and still recommend a runtime that no longer receives support. The request completed, but the answer does not meet the application’s requirements.
Use the seven gates below to review one workflow. Each gate needs an owner, dated evidence, and a decision. Set thresholds for your application rather than borrowing a universal quality score or latency target.
1. Define the answers your feature can accept
Specify the projects, question types, and source requirements the feature supports. “Answer questions from the web” gives the reviewer no useful target.
For a documentation assistant, an acceptable answer might need to identify a supported version, explain its relevant limits, and cite the official documentation. An undocumented parameter should produce an evidence gap rather than an invented definition.
- Define the supported questions and the response to requests outside that scope.
- Write the required answer elements and critical failure conditions before testing.
- Separate accepted, review-needed, unanswered, and failed results in the application.
- Name the owner who approves the acceptance rules.
Evaluate the full path from question to displayed answer, including any work your own model performs after receiving a web result.
2. Check coverage and source support
Coverage asks whether the answer addresses the question. Source support asks whether the cited passages establish its claims. An answer can name the right runtime but omit its support window. It can also include every requested field while attaching a date from the wrong release.
The citation review guide explains how to inspect a claim’s passage, qualifiers, and version. The traced Python example preserves the report and source mapping for that review.
- Review each required answer element and each decision-driving claim.
- Record the source’s relevant date or version, including when it is unknown.
- Keep inaccessible sources, unreviewed claims, and duplicate references visible.
- Apply critical failure rules before considering average scores.
Freeze the question set, expected elements, scoring rules, and implementation configuration before running the evaluation. Keep every attempt, including failures, and separate live results from pilots and synthetic tests. A small sample helps identify defects; stronger reliability claims require a broader evaluation. The 20-question evaluation set walkthrough builds this with a Python kit that hashes the frozen set, keeps every attempt, and labels pilot and synthetic runs.
Automatic checks can flag missing citations or malformed URLs. Use human review or a separately validated verifier to decide whether a passage supports a claim. In that kit’s pilot, a second blind review caught a claim the first review had scored as fully supported.
3. Test the wait your product allows
Set expectations for first feedback and the final answer. Progress tells the user that work continues; it does not mean the answer is ready.
Tabstack Research streams progress and returns the report on complete. It has no overall server-side timeout. The fetch_timeout parameter limits one page fetch, not the whole request.
Your client owns the product’s wait policy. A silence timer catches a gap between events. A total deadline limits how long the user waits even when progress continues. The evaluation kit sets both (--silence-timeout and --deadline) and records each as its own failure state.
- Define first-feedback and final-result expectations for the interface.
- Measure elapsed time from client dispatch to terminal success or failure.
- Test both a stalled stream and an active request that exceeds the deadline.
- Show a clear timeout state without claiming server-side cancellation.
Stopping the client does not establish that the server stopped processing. Account for that uncertainty before retrying or interpreting usage.
4. Contain failures before they reach the user
Test the interface as well as the exception handler. Exercise HTTP rejection, streamed task errors, interrupted connections, malformed output, missing sources, and a stream that closes without a terminal event.
For Research, an HTTP success does not establish task completion. Handle both complete and error; record an incomplete result if the stream closes without either.
- Give each failure a defined application state and user-visible response.
- Apply the evidence rule when a response has no source list.
- Record SDK and application retries separately, including possibly accepted requests. The evaluation kit sets SDK retries to zero, so one recorded attempt is one request.
- Test fallbacks so they cannot silently present old or unchecked answers as current.
A cached answer is a valid fallback only when its retrieval date remains visible and its age fits the question. A runtime-support feature should not relabel an old recommendation as current.
5. Measure cost from the same requests you reviewed
Set a workflow budget and decide what the application does at its limit. Include retries, failed requests, and model work your application performs after receiving the answer.
Research can use a variable number of actions per task, as the pricing page explains. A per-action rate alone does not establish the cost of a complete request. The Research stream reports token counts per model but no per-request credit record, so attach usage from your account records.
- Attach authoritative usage records to the requests they cover.
- Include billed failures and retries; leave missing usage unavailable rather than zero.
- Set usage limits or alerts appropriate to the rollout.
- Review plan limits, expected volume, and support requirements.
Calculate cost per accepted answer only for a fully reviewed batch with complete usage records. Divide the cost of all attempts in that batch by its accepted answers. Do not combine full-batch spend with a selected subset of successful responses.
6. Review where data goes and what it can do
Map which services receive the question, retrieved content, and instructions. Review your application’s logs and access controls as well as the provider’s handling. Running your own model does not make a hosted web call local.
The Privacy Notice is the canonical statement of how Tabstack handles your data. It says Tabstack processes your Inputs to run them through third-party models and generate Outputs, and that it stores history, including your content and Outputs, for 90 days or until you delete it. It also asks you to be careful about sharing confidential or sensitive information in your Inputs.
- Check the processing path against the workflow’s privacy requirements.
- Keep credentials and private customer data out of public test artifacts.
- Allowlist logged fields and define who can inspect them.
- Treat retrieved pages as untrusted input, validate source URLs, and restrict downstream actions.
A retrieved page cannot grant an agent permission to run commands or change an account. Set tool permissions and approval rules in the application.
If the requirement is that data never leave your controlled environment, hosted Research does not fit. Pilo provides self-hosted automation, not a self-hosted Research service.
7. Assign an owner and test the rollback
Name who responds when a source changes, a dependency fails, quality regresses, or usage exceeds its budget. A managed research service removes work from the application, but your team still owns the feature’s behavior.
- Give every release gate an owner, evidence, and a recorded decision.
- Limit the initial rollout and define when to expand it.
- Test the path that disables the feature or selects a labeled fallback.
- Document escalation contacts and repeatable regression checks.
Keep configuration and dataset versions with the runbook so another engineer can reproduce the result and identify what changed.
Record the release decision
Use a compact record that links each decision to its evidence:
| Gate | Evidence to attach | Decision |
|---|---|---|
| Scope and acceptance | Supported workflow and critical failure rules | What the feature handles or refuses |
| Answer quality | Coverage and claim-support reviews | Which answers the application accepts |
| Time budget | Client timings and deadline tests | Whether the interaction meets its wait policy |
| Failure handling | Attempt log, failure fixtures, and fallback tests | How the application contains failures |
| Cost | Matched usage records and budget controls | Whether the workload fits the budget |
| Data and security | Processing map and reviewed controls | Whether the architecture meets requirements |
| Ownership and rollback | Runbook and disable-path test | Who acts when behavior changes |
The evaluation kit includes a blank release-review template with these seven gates. Mark each gate pass, fail, or unknown. Resolve critical unknowns before expanding the rollout, and retain the evidence behind any decision to accept a remaining risk.
Start with one public question and one failure case. Review the answer’s source support, then test what the user sees when the request cannot finish. Those two records give the team concrete work to resolve before release.
Read the Research guide to check the endpoint contract. When you need to discuss fit, bring your workflow and constraints to the Tabstack team.
