What does the model you run know about today?

I recently asked Llama 3.1 8B, running locally in Ollama, what the current Node.js Active LTS release is.

It gave me a version that was several releases behind. It also told me its information was a couple of years old and that I should check the Node.js website. For a model that can’t see past its training data, that’s a fair and honest answer. It’s also wrong.

Then I asked the same model the same question, but gave it a research report and a list of sources first. This time it named the current release and cited nodejs.org.

Nothing changed except what went into the prompt. I wanted that difference to be easy to see on anyone’s setup, so I built a small CLI called freshness-check. You point it at a model you run, it asks a set of questions with and without web research, and it writes an HTML report with the two answers side by side.

I should say how I built it. My Python is rusty, so I worked through Claude Code. I wrote the brief, approved the plan before any code existed, and made the calls on what to change when reviews turned things up. Claude Code wrote most of the code. When I say “I built” in this post, that’s what I mean.

The problem with running your own model

If you run your own model you already know the trade. You decide where your data goes and what it costs, and in return you lose anything the model didn’t see during training.

For plenty of work that doesn’t matter. Refactoring a function or explaining a regular expression doesn’t depend on last month’s news. Ask about the latest release of anything, though, and the model can either admit it doesn’t know or guess from old data. Usually you get a polite mix of both.

The way round it without switching to a hosted chat model is to do the web work first and pass the results in as context. That’s what the second path in freshness-check does, using Tabstack Research. If you want the wider picture of what that web work involves for a model you run, I broke it down in the seven stages of web research for a model you run.

Two paths, one model

Every question runs twice against the same model:

1. Model alone. The question goes straight to the model with no web input.
2. Model with Research. The question goes to Tabstack Research first. The report and cited URLs go into a second prompt, and the model answers from that.

Both prompts include today’s date. Without it the model alone can’t even tell how stale its knowledge is, and the comparison stops being fair. So both open the same way:

Today's date is {date}. Answer the question from your own knowledge.
If you are not sure your information is current, say so.

Question: {question}

The Research prompt swaps both instructions for “answer using only the research below” and “cite the source URLs you relied on”, then adds the report and a numbered list of sources:

Today's date is {date}. Answer the question using only the research below.
Cite the source URLs you relied on. If the research does not answer the question, say so.

Research:
{report}

Sources:
{sources}

Question: {question}

Both templates live in prompts.py, so you can change the wording and rerun without touching anything else.

On the model side the tool uses the openai Python package pointed at whatever base URL you give it. Ollama, LM Studio, llama.cpp and vLLM all expose an OpenAI-compatible chat completions endpoint, so one client covers the lot.

Handling a streaming research call

Tabstack Research always streams. Instead of one response you get events as it works: planning, each search round, writing the report, and then either complete with the report or error with a reason.

A fast-mode call takes anywhere from a few seconds to about a minute, so how long should you wait before deciding something has gone wrong?

A total timeout is the obvious answer and it’s the wrong one. Set it to 30 seconds and you kill calls that were fine. Set it to five minutes and a stuck call holds up your run for five minutes. The Research docs recommend a silence timeout instead: reset a timer on every event, and only fail when nothing has arrived for a while.

Here’s the helper Claude Code wrote for it:

def iter_with_silence_timeout(iterable, seconds, on_timeout=None):
    items = queue.Queue()

    def pump():
        try:
            for item in iterable:
                items.put(("item", item))
            items.put(("done", None))
        except BaseException as error:
            items.put(("error", error))

    threading.Thread(target=pump, daemon=True).start()

    while True:
        try:
            kind, value = items.get(timeout=seconds)
        except queue.Empty:
            if on_timeout is not None:
                on_timeout()
            raise SilenceTimeout(seconds) from None
        if kind == "done":
            return
        if kind == "error":
            raise value
        yield value

A background thread reads the stream into a queue, and the main loop waits on that queue with a timeout, so every event resets the clock. Because it doesn’t care what it’s iterating over, I could test it with plain Python generators that sleep halfway through.

The timeout can’t catch a stream that closes cleanly without ever sending complete or error, so the tool checks for that separately. It counts as a failed Research call and the report says so on that question’s card.

Things I ran into

Showing a minimum rather than a guess. The tool prints the least a run can cost and waits for you to confirm before it sends anything:

10 questions in fast mode. Research bills per action (250 credits per action in fast mode)
and may run more than one action per question, so this is a minimum of 2,500 credits.

The report carries the same figure and labels it a minimum, because that’s the only number I can stand behind.

Ollama can cut your prompt without telling you. When a prompt is longer than the model’s context window, Ollama trims it to fit and carries on without an error. The research report and sources make the second prompt a lot longer than the first, so this is the path where it would bite.

The OpenAI-compatible API won’t tell you a model’s context window, but it does return usage.prompt_tokens. The tool compares that with a rough estimate from the prompt length:

possibly_truncated = not estimated and input_tokens < estimate * TRUNCATION_RATIO

If the runtime counted fewer than half the tokens the prompt should have, the report notes that the prompt was probably cut. When the runtime reports no usage at all, the tool falls back to the estimate and marks the figure “est.” wherever it appears.

What the first run showed

I ran all ten built-in questions against that same Llama model, and every one finished on both paths. Three of them:

The model alone was honest. It hedged on most answers and pointed at official sources, which is what the prompt asked for. Its information was just old.

Adding the research cost very little context. The model-alone prompts were around 50 tokens and the Research prompts roughly 200 to 300. Each fast-mode Research call took somewhere between six and thirteen seconds.

The first model call of the run took just over half a minute, and every model-alone call after it came in under 20 seconds. That first one is most likely Ollama loading the model into memory, so ignore it when you read the timings in your own report.

The report doesn’t score any of this. No grades, no winner banner. Each question gets a card with both answers, the sources Research cited, how many pages it analysed, and the research report the model actually received, so you can check one against the other. If the model alone gets something right, the card shows that just as plainly.

Try it on your own setup

You need uv, a model behind an OpenAI-compatible endpoint, and a Tabstack API key. With Ollama it looks like this:

ollama pull llama3.1:8b
export TABSTACK_API_KEY="your-tabstack-key"

uvx --from git+https://github.com/juststeveking/freshness-check freshness-check \
  --model llama3.1:8b

Add --limit 2 to try a couple of questions before running all ten, or --questions to bring your own as a .txt or .jsonl file. For other runtimes, pass --base-url: LM Studio uses http://localhost:1234/v1, llama.cpp http://localhost:8080/v1, and vLLM http://localhost:8000/v1.

The tool sends your questions to two places: your model endpoint and Tabstack Research.

Two limits to keep in mind. The results depend on the day you run it, because the questions are about things that change. And the Research path only works if the model follows the instruction to answer from the research. When it answers from memory anyway, the report shows that answer, and the research report sits on the same card so you can compare.

If you’d rather have your model call Research itself from a chat, the Tabstack tool for Open WebUI does that.

If you run it, I’d like to hear what your model said. The code is on GitHub.

If you want to see what the Research step returns on its own, make a research call or read the docs.