The 22-line demo and everything it was missing

Have you ever wanted to ask a question about a framework or package, and just get a simple answer? I have. Something like “what has changed in the latest Laravel release that affects queue workers?”. Important questions that you could answer from the changelog, if you have the time and patience (which I don’t).

So, 22 lines of Python later (thanks Claude), and 14.7 seconds after asking, it told me “The latest Laravel release, 12.x, …”. Oops. Packagist was listing v13.34.0 as the latest stable release when I ran it.

Nothing crashed, to be fair. It did exactly what I asked: one research call, print whatever comes back. The problem is everything around that call. It had no idea when not to make it, no idea how it ended, kept no record of what it said, and couldn’t tell a good answer from a confidently wrong one (see above).

So that’s what this tutorial is about. We’re going to take that demo to something I’d actually put in production, one step at a time. It’s all in JustSteveKing/what-changed, and every step below is in the commit history, so you can follow along with the diffs if you like.

Fair warning before we start: all the results in here come from twenty questions, run on a single day in October. That’s a small sample. It tells you what happened that day with those questions, and I’m not going to pretend it says anything about latency, reliability or correctness in general.

What we’re building

A “what changed?” answerer for PHP and Laravel packages. You ask it a question, and you get back an answer plus the sources it came from, in order, so you can check it.

It knows about Laravel, Livewire, Pest, Filament and Inertia’s Laravel adapter. The code is Python, but if you spend your days in PHP the domain will feel familiar, and none of the patterns care which language you use. You don’t need to know anything about Tabstack.

The stack is small on purpose: Python with uv, pydantic for the data contracts, pytest and mypy, the official Tabstack SDK, and httpx for talking to Packagist and GitHub. No frameworks.

If you want to follow along, clone the repo and start at the demo tag:

git clone https://github.com/JustSteveKing/what-changed.git
cd what-changed
make install
git checkout demo

You’ll need a TABSTACK_API_KEY for anything that makes a research call (those spend credits). A GITHUB_TOKEN is optional, but it helps once we get to the GitHub lookups. To see every step after the demo, git log --reverse demo.. lists them in order.

Step 1: the demo

Here’s the whole thing:

# The demo: one Research call, print the answer, done.
# Deliberately naive. See the git history for what changes after this.
import sys

from tabstack import Tabstack

question = " ".join(sys.argv[1:])
if not question:
    print('Usage: uv run src/demo.py "your question"', file=sys.stderr)
    sys.exit(1)

client = Tabstack()
stream = client.agent.research(query=question, mode="fast")

for event in stream:
    if event.event == "complete":
        print(event.data.report)
        print("\nSources:")
        for page in event.data.metadata.cited_pages or []:
            print(f"- {page.title or '(untitled)'}: {page.url}")
    if event.event == "error":
        raise RuntimeError(event.data.error.message)

If you’ve looked at the Tabstack Python SDK, that’s very nearly the official example. Run it with a question and it works:

uv run src/demo.py "What changed in the latest Laravel release that affects queue workers?"

And it does work, in the sense that it runs and prints something. That’s about where it stops, though. Every question costs a research call, including the ones a quick metadata lookup could answer and the ones that shouldn’t be answered at all. There’s no deadline. If the stream closes without a complete or error event, it prints nothing and exits 0, so whatever called it thinks it succeeded. Nothing gets recorded, the SDK’s default retries are still on, and there’s no way to tell a good answer from a bad one. Which is how we ended up with “12.x”.

The rest of this tutorial closes those gaps, one at a time.

Step 2: write the questions before you fix anything

The obvious next move is to start fixing code. I didn’t, and here’s why: if you change things before you’ve decided what a correct answer looks like, you’ve no way of knowing whether a change helped. You end up judging answers on how convincing they sound (and “12.x” sounded very convincing).

So the first real change is a set of twenty test questions. Some should be answered with a quick lookup, most need actual research, and a few should be declined. For each one I wrote down where it should go, what a correct answer has to include, where the truth lives (changelogs, release notes, upgrade guides), and when the answer is likely to go stale.

Here’s the queue-worker question, trimmed down a bit:

{
  "id": "q06",
  "question": "What changed in the latest Laravel release that affects queue workers?",
  "expectedRoute": "research",
  "expectedOutcome": "answered",
  "requiredElements": [
    { "id": "release", "check": "Identifies which release is the latest (v13.34.0 when written).", "source": "https://github.com/laravel/framework/releases/tag/v13.34.0" },
    { "id": "max-exceptions", "check": "Mentions that jobs can count worker crashes towards maxExceptions (PR #61737).", "source": "https://github.com/laravel/framework/releases/tag/v13.34.0" }
  ],
  "freshness": "Written against v13.34.0 (published 2026-09-29). If a newer laravel/framework release exists at run time, the 'latest' is different and these elements no longer apply; the reviewer must score against that release's notes or mark the question stale."
}

I read the ground truth from those official sources rather than from memory. If an answer and the changelog disagree, the changelog wins.

A few of the questions are there to catch one specific failure:

• “What changed in Laravel 14?” should be refused, because there is no Laravel 14.
• “What changed in the latest Livewire release?” is a trap of a different kind. The official release notes for that version are just a compare link, so an honest answer is thin, and only a person reading it can tell.
• One question says “Inertia” and means the Laravel adapter. That ambiguity is deliberate.

Anything pinned to “the latest release” goes stale the day the next one ships, which is what the freshness note is for.

Step 3: route before you call

The biggest change, and the one that does most of the work, is a router that decides what to do with a question before anything expensive happens.

Every question goes one of three ways. A direct question gets answered by one lookup against Packagist or GitHub, with no model involved. A research question needs several sources read and pieced together, so it gets one Tabstack Research call. Anything else gets a decline: opinions, questions about the future or unreleased versions, and packages it doesn’t know about.

Declining is a proper route here, with its own reason and its own exit code. Most of the demo’s worst answers were to questions it should have just said no to.

The router is plain rules, no model. I wanted something I could unit test, explain in an article, run for free and get an answer from instantly, and a handful of regexes and if statements does all of that. The rules run in order and the first match wins:

1. Empty, or far too long: decline.
2. No supported package found: decline.
3. Opinion or comparison (“should I”, “better than”, “vs”): decline.
4. Future or unreleased (“upcoming”, “when will”, “next major”): decline.
5. More than one package: research.
6. One package, asking only for the latest version, date or release notes: direct.
7. Change words, or a named version: research.
8. Anything else: decline.

Here’s the package list and the patterns behind rules 3 and 4:

# The GitHub repo is listed explicitly because it does not always match the Packagist
# name: filament/filament's Packagist source is the filamentphp/panels subsplit, but
# its GitHub releases are published on filamentphp/filament.
KNOWN_PACKAGES = (
    KnownPackage("laravel/framework", "laravel/framework", ("laravel framework", "laravel")),
    KnownPackage("livewire/livewire", "livewire/livewire", ("livewire",)),
    KnownPackage("pestphp/pest", "pestphp/pest", ("pestphp", "pest")),
    KnownPackage("filament/filament", "filamentphp/filament", ("filament",)),
    KnownPackage("inertiajs/inertia-laravel", "inertiajs/inertia-laravel", ("inertia",)),
)

OPINION = re.compile(r"\b(should i|should we|which is better|better than|recommend\w*|vs\.?|versus|compare\w*|comparison)\b")
FUTURE = re.compile(r"\b(upcoming|roadmap|unreleased|planned|when will|be released|next (major|minor|release|version))\b")

That comment about Filament is one of those things you only find out by looking. Its Packagist source points at a subsplit repo, but the releases are published on the main one, so the mapping has to be written down.

The future pattern is narrower than you might expect. “will” on its own doesn’t count, so “Will my workers break when upgrading?” still goes to research. The tests also make sure “Laravel Livewire” counts as one package, and that “PHP 8.5” never gets read as a package version.

The top of the router looks like this:

def route(raw_question: str) -> RouteDecision:
    question = raw_question.strip()
    if not 1 <= len(question) <= 2000:
        return DeclineDecision(route="decline", reason="The question is empty or longer than 2000 characters.")

    text = normalise(question)
    mentions = find_packages(text)

    if not mentions:
        return DeclineDecision(route="decline", reason="No supported PHP package was identified in the question.")
    if OPINION.search(text):
        return DeclineDecision(route="decline", reason="The question asks for an opinion or comparison, not what changed.")
    if FUTURE.search(text):
        return DeclineDecision(route="decline", reason="The question asks about a future or unreleased version.")

Every decline carries a reason, so when the router says no, you can see why.

Step 4: check the version exists

There’s one more guard before research, and it’s tiny. If a question names a version, ask Packagist whether that version exists before going any further:

    # A research call about a version that has never been tagged would answer
    # from speculation, so check Packagist first.
    if decision.version_mentioned is not None:
        package, version = decision.version_mentioned.package, decision.version_mentioned.version
        versions = await fetch_versions(package, config, cancel)
        if not version_exists(version, [v.version_normalized for v in versions]):
            return declined(f"Packagist lists no tagged release of {package} matching {version}."), False
    return await research(question, config, cancel), False

So “What changed in Laravel 14?” now gets declined with “Packagist lists no tagged release of laravel/framework matching 14.” and never reaches the research API. Before this, the same question came back with a confident description of a release that doesn’t exist. One Packagist lookup, and that research call never happens.

Step 5: one answer shape for every route

Whatever route a question takes, it comes back in the same shape:

class RouteResult(Model):
    terminal_state: TerminalState
    # True when the call completed but the answer cannot be relied on as a full answer.
    incomplete: bool
    incomplete_reasons: list[str]
    text: str
    sources: list[SourceRecord]
    error: AnswerError | None
    # Only ever a value the API itself reported. Never estimated.
    cost: None
    cost_note: str


class ApplicationAnswer(RouteResult):
    question: str
    decision: RouteDecision
    # True when a direct lookup was served from the cache. Defaults to False so
    # records written before caching existed still parse.
    cached: bool = False
    started_at: AwareDatetime
    ended_at: AwareDatetime

The Tabstack event objects never leave the code that talks to Tabstack, so the CLI, the stored records and the scorer don’t have to know what a streamed event is (and they’re much simpler for it).

And yes, cost is typed as None. The API doesn’t report cost, so I made it impossible to put anything else there. No estimates, no guesses.

Step 6: read what the APIs actually return

The direct routes look like the easy part. Then you read the actual responses.

Packagist’s metadata is minified. Each version entry only carries the fields that changed from the one before it, and "__unset" removes a field. You have to expand it before you can use it, so the code does what Composer itself does:

# The p2 format is minified: each entry only carries the fields that changed from the
# previous one, and "__unset" removes a field. This mirrors Composer's own
# MetadataMinifier::expand (github.com/composer/metadata-minifier).
def expand_minified(versions: list[dict[str, Any]]) -> list[dict[str, Any]]:
    expanded: list[dict[str, Any]] = []
    previous: dict[str, Any] = {}
    for entry in versions:
        current = dict(previous)
        for key, value in entry.items():
            if value == "__unset":
                current.pop(key, None)
            else:
                current[key] = value
        expanded.append(current)
        previous = current
    return expanded

The “stable” metadata also still has pre-release tags in it (233 of them for Filament alone when I checked), so the code only treats plain numeric versions as stable. The order isn’t documented either, so it sorts versions itself.

GitHub held fewer surprises. The “latest release” endpoint skips drafts and pre-releases, and the unauthenticated rate limit is low, which is why the token is worth setting.

What I checked, and when, is all written down in docs/VERIFIED.md.

Step 7: measure the demo

Before improving anything, you need a baseline. So I ran all twenty questions through the demo’s behaviour, every one straight to a research call, and saved the results in the same shape as everything else.

I ran it twice, once in fast mode and once in balanced mode. Fast mode got through the lot in about four minutes. Balanced mode took nearly three quarters of an hour, with a typical call taking around two minutes and the slowest over four. In both runs, only 13 of the 20 questions ended up where they should have.

The summary looks fine. Open the stored answers and it looks a lot less fine:

• The Laravel 14 question got a description of Laravel 14’s changes in both runs, though no such release existed.
• “When will Pest 6 be released?” opened, in balanced mode, by talking about PEST 6.05, an environmental modelling package from 2003.
• An opinion question got a full comparison instead of a refusal, both times.
• The Inertia question answered about the JavaScript client instead of the Laravel adapter, both times.

Every one of those runs completed, and most of them passed most of the automatic checks. Hang on to that, we’ll come back to it.

Step 8: put a deadline on the whole call

The SDK does give research calls a ten minute timeout, but it took reading the source to see what it covers. It applies to each read, so a stream that keeps sending events never hits it.

The fix is one helper that wraps every outbound call with a deadline on the whole thing:

# Runs `awaitable` with a limit on the whole call, and stops it early if `cancel` is
# set (Ctrl+C in the CLI). Either way the work in progress is cancelled, which closes
# any open stream, and the exception says which of the two happened.
async def with_deadline(awaitable: Awaitable[T], seconds: float, what: str, cancel: asyncio.Event | None = None) -> T:
    work = asyncio.ensure_future(awaitable)
    stop = asyncio.ensure_future(cancel.wait()) if cancel is not None else None
    waiting: set[asyncio.Future[Any]] = {work, stop} if stop is not None else {work}
    try:
        done, _ = await asyncio.wait(waiting, timeout=seconds, return_when=asyncio.FIRST_COMPLETED)
    finally:
        if stop is not None:
            stop.cancel()

    if work in done:
        return work.result()

    work.cancel()
    with contextlib.suppress(asyncio.CancelledError):
        await work
    if cancel is not None and cancel.is_set():
        raise Cancelled(f"{what} was cancelled")
    raise TimedOut(f"{what} timed out after {round(seconds * 1000)} ms")

The important line is work.cancel(). Cancelling the task is what closes the stream, so hitting the deadline actually frees the connection rather than leaving it running in the background. The limits are configurable, with 10 seconds for lookups and five minutes for research by default.

The test is a local server that sends one event and then hangs, and the call has to give up at the limit.

Step 9: give every ending a name

A request can end in more ways than “worked” or “threw”. Name each one, and give each its own exit code so scripts can tell them apart:

TerminalState = Literal["completed", "failed", "stream_closed", "timed_out", "cancelled", "declined"]

# Each terminal state has its own CLI exit code so scripts can tell them apart.
# 130 follows the shell convention for a process stopped by Ctrl+C (SIGINT).
EXIT_CODES: dict[TerminalState, int] = {
    "completed": 0,
    "failed": 1,
    "declined": 2,
    "stream_closed": 3,
    "timed_out": 4,
    "cancelled": 130,
}

The one the demo got silently wrong was a stream that just stops. Now it gets a result of its own:

    # The stream ended without a complete or error event.
    return RouteResult(
        terminal_state="stream_closed",
        incomplete=True,
        incomplete_reasons=["The stream ended without a terminal event."],
        text="",
        sources=[],
        error=None,
        cost=None,
        cost_note=COST_NOTE,
    )

In the demo that case printed nothing and exited 0. Now it says what happened.

Step 10: handle Ctrl+C properly

You might have noticed the deadline helper already waits on a cancel event alongside the work. That’s the hook. The CLI sets it on the first Ctrl+C:

async def ask(question: str) -> ApplicationAnswer:
    # First Ctrl+C cancels the in-flight call and lets it report a cancelled state.
    # A second Ctrl+C exits immediately.
    cancel = asyncio.Event()

    def on_interrupt() -> None:
        if cancel.is_set():
            os._exit(EXIT_CODES["cancelled"])
        print("Cancelling. Press Ctrl+C again to exit immediately.", file=sys.stderr)
        cancel.set()

    asyncio.get_running_loop().add_signal_handler(signal.SIGINT, on_interrupt)
    return await answer(question, cancel=cancel)

The first press lets the call wind down and record a cancelled answer. The second gets you out immediately. The test runs the real CLI, sends it a Ctrl+C once the request is in flight, and checks it exits with the cancelled code.

Step 11: store what you answered

Every answer is saved exactly as it came back, with its sources in order. It’s a small change, and it’s what made the rest of this possible: everything I’ve said about the baseline came out of these files.

Step 12: log every request

One JSON line per request, with the route, how long it took, how it ended, whether it came from the cache, how many sources, and any error. The question text is left out on purpose, because logs end up in far more places than stored answers do.

Step 13: retry the cheap things only

Lookups against Packagist and GitHub are cheap and safe to repeat, so they get a few attempts with backoff. Not every failure is worth another go, though:

# Network errors, per-attempt timeouts, 429 and 5xx are worth another try.
# Other 4xx responses will not change on retry.
def is_retryable(error: Exception) -> bool:
    if isinstance(error, Cancelled):
        return False
    if isinstance(error, HttpError):
        if error.rate_limit_exhausted:
            return False
        return error.status == 429 or error.status >= 500
    return True

It also gives up straight away if GitHub’s rate limit is used up, or if the server asks it to wait more than a few seconds.

Research is never retried. Each call is billed, it takes minutes, and asking again can get you a different answer built from different sources, which is closer to a second opinion than a retry. And this is the bit you’d only find by reading the SDK: it retries twice by default. Leave the client alone and you’re retrying research calls without knowing it, so switch it off:

    # No automatic retries for research. A Research call is billed per call, takes
    # seconds to minutes, and a second attempt can return a different answer from
    # different sources. Retrying quietly would hide a failure and could double the
    # spend, so a failed call is reported and the caller decides whether to ask again.
    # The SDK retries twice by default, so this has to be switched off explicitly.
    client = client or AsyncTabstack(max_retries=0)

    async def call() -> RouteResult:
        stream = await client.agent.research(query=question, mode=config.research_mode)
        async with stream:
            return await collect_research(stream)

    # The SDK's timeout is per read, so a stream that keeps sending events is never
    # cut off by it. This deadline covers the whole call.
    # Setting `cancel` (Ctrl+C in the CLI) stops it early and closes the stream.
    return await with_deadline(call(), config.research_timeout_s, "Research", cancel)

There’s a test that fails if the default ever sneaks back in. With it left on, the local test server saw three requests for a single call.

Step 14: cache the lookups, not the research

    if decision.route == "direct":
        key = normalise(question)
        hit = cache.get(key)
        if hit is not None:
            return hit, True
        result = await direct(decision.lookup, config, cancel)
        # Only successful lookups are cached; a failure should be retried next time.
        if result.terminal_state == "completed":
            cache.set(key, result)
        return result, False

Lookups are cached for a minute, and only when they succeed. Research isn’t cached at all, because the whole point of asking is a current answer. Tabstack does keep its own server-side cache, though, and every run here used the default setting for it.

Step 15: test the stream without the network

You don’t want your tests making billed calls, but you also don’t want to mock away the bit most likely to break. So the tests point the real SDK at a small local server that replays recorded events. The SDK’s own parsing, streaming and cancelling code all gets exercised, with no API key and no network.

There are three replay files, and they didn’t all come from the same place. The successful one was recorded from a real call. The “stream just stops” one is that recording with the final event taken off. The error one is real events followed by an error I built by hand, because I never actually saw a real streamed error. Each file says which of those it is, so nobody mistakes the made-up one for a recording later.

There’s also a test that makes sure neither API key ever ends up in a saved answer or a replay file, which you want in place before you record anything live.

Back on main, make check runs the whole suite plus mypy, without touching the network or spending any credits:

git checkout main
make check

To ask the finished version a question yourself:

export TABSTACK_API_KEY=...
make ask Q="What is the latest stable version of livewire/livewire?"

That one’s a direct Packagist lookup, so it costs nothing. Only questions that get routed to research spend credits.

Step 16: run it again and compare

With everything in place, the same twenty questions went through again in balanced mode, this time through the router.

Every question went where it was supposed to. Five turned into lookups that came back in well under a second, three were turned away without any call at all, and only twelve needed research instead of twenty. The whole run took about 24 minutes instead of 46.

None of the safety nets fired, though. No failures, timeouts, dropped streams or cancellations in any live run, so everything I’ve said about deadlines and cancellation is backed by the tests and nothing else.

The research questions went through exactly the same call both times, so any difference in those answers comes from Tabstack, not from anything I changed. They took about as long as before too, a little under two minutes for a typical one.

A couple of the automatic checks look worse on paper, but only because a declined answer has no sources by design. Leave the declines out and every answer has sources.

One thing got worse for real, which you’ll see in the next step: the research answers scored slightly lower than before, with the same call on the same questions. The router didn’t cause that and couldn’t have prevented it.

The saved answers turned up one more oddity, too. The Livewire question dated Livewire 4’s first release to August 2025. Packagist has v4.0.0 in January 2026.

Step 17: score the answers separately

Remember those baseline runs that completed and passed most of the checks? Those checks look at things like whether the route was right, whether there are sources, and whether the URLs are valid. All plumbing. The Livewire question passes every one of them in every run without ever saying what changed in that release.

So I scored the answers on their own, against the things each correct answer had to include, marking each one as fully there, partly there or missing.

Full disclosure on how that was done: a model (Claude Opus 5.5) drafted the scores, and for each one it had to quote the exact bit of the answer it was judging. A validator rejects any quote that isn’t really in the answer. I went through every judgement call myself (21 of them, every “partly” plus the borderline ones) and agreed with all of them. The rest I accepted as drafted, and the scoring sheets say which is which.

Here’s how many of the 44 required points each run got fully right:

The improvement comes from the parts the router controls. Direct lookups went from 7 of 9 to all 9, and declines from 1 of 3 to all 3. The research answers went from 25 of 32 to 23, and that’s Tabstack on the day.

The validator stops the model inventing quotes. It can’t stop it misreading a quote that’s really there, which is why I went through the judgement calls myself, and why I’m telling you which ones I didn’t.

Step 18: the production checklist

The last piece is a checklist covering quality, latency, cost, failure handling, privacy and ownership. Each item is a pass, a fail, or “can’t verify from outside”, because some of this you just can’t check from your side of the API.

Cost is the clearest case. Tabstack’s pricing page says plans, spend and credit balance are in the console, “not in any API response”. So the app can’t record cost per request at all, and there’s no cost figure anywhere in this tutorial.

Retention comes from Tabstack’s Privacy Notice, which is the source of truth for it. Research questions are run through third-party models, and history, including your content and outputs, is stored “for 90 days or unless you delete it.” So that’s the number the checklist works from: anything you put in a question is kept for up to 90 days unless you delete it, which is worth knowing before you send anything sensitive.

The data handling page adds that SOC 2 and ISO 27001 aren’t currently held. Read the Privacy Notice itself rather than relying on my summary.

The app has fails of its own. Nothing spots a stalled stream before the overall deadline, there’s no spending limit, and Tabstack’s rate-limit headers aren’t checked up front. Saved answers never expire. Nobody is named as the owner, and it’s nobody’s job to refresh the questions as they go stale.

And on latency, the slowest research call took just over four minutes against a five minute deadline, which doesn’t leave much margin.

What I’d take from this

The demo works. Production is everything around the call, and for this project most of that turned out to be deciding when not to make it.

If I was starting again, the bits I’d do on day one are: write the questions and the right answers down first, route before you call, check a version exists before asking about it, and give every way a request can end its own name. Then read the SDK source (the default retries and the timeout behaviour weren’t obvious anywhere else), and never retry anything that costs money or can come back with a different answer.

And be upfront about what you don’t know. Twenty questions on one day is a small sample, a model drafted most of the scores, and some things, like cost, you can’t verify from outside at all.

Where it goes next

The checklist fails are basically my to-do list: something to catch a stalled stream well before the overall deadline, a spending limit, actually reading the rate-limit headers instead of finding out the hard way, an expiry for saved answers, a named owner, and a schedule for refreshing stale questions. I also want to try turning off Tabstack’s cache and see what a guaranteed fresh fetch does to the answers.

If you want to check any of the numbers, docs/EVIDENCE.md has the lot.

Grab the demo tag, ask it about a package you actually use, and write down the right answer before you read what it says. Then go and have a look at your own demos (mine clearly needed it).