The most private way to run open models without your own hardware is inference inside a hardware enclave you can verify, and for a popular open model it costs about the same as an ordinary hosted API. If you can’t use that, pick a host whose zero data retention (ZDR) promise sits in a contract, turn it on, and close the leaks around the model too.
A developer building an app on bank statements and client records asked this on X recently. They can’t run models locally, and they wanted to know whether a hosted open-model service that advertises zero data retention was a good pick, or whether something was better. The replies were full of real setups. This post sorts them by what actually protects your data, and what it costs.
Local and cloud are two different jobs
Open models come in sizes, and the size decides where they run. The models that fit on a laptop are the ones you build and test with. The larger models that handle harder tasks, or serve real traffic, run in the cloud. Many teams use both: one model family, a local size while building and a cloud size in production. gpt-oss, for example, ships as gpt-oss-20b and gpt-oss-120b.
| Local | Cloud | |
|---|---|---|
| Example models | qwen3:8b, qwen3:32b, gpt-oss-20b | gpt-oss-120b, qwen3:235b, GLM-5.3, Kimi K3 |
| Good for | Building and testing, offline work, single-user tools | Production traffic, many users, long context, harder tasks |
| What you need | A machine with 16 to 64GB of memory | An account and an API key |
| How you pay | Hardware up front | Per token or per GPU hour |
| Who handles your prompts | Only your machine | The host, under its terms |
If you can’t run models on your own hardware, like the developer who asked, you’re picking from the cloud column, so that’s what most of this post is about. I cover local near the end.
Open weights don’t make hosting private
One reply put it plainly: hosted open models are the same as using any other hosted API. The original poster agreed the hosted option only feels more private.
Once a model runs on someone else’s servers, the weights being open changes nothing about who handles your prompts. The host’s terms decide that: what it stores, for how long, who can read it, and where it runs. As another reply said, you’re moving trust from one company to another. So the job is to check what each company has actually committed to.
How strong a privacy promise is depends on where it’s written
The same promise means different things depending on where it lives:
- Marketing page. A homepage headline. Useful, not binding.
- Privacy policy or docs. More specific, and it usually carries the exceptions.
- Contract or data processing agreement (DPA). Binding, and what an auditor asks for.
- Hardware attestation. You verify it yourself on every connection, so you don’t have to take anyone’s word.
Read past the headline. The policy and the contract are where you’ll find the details, and the exceptions, that apply to your data.
Defaults decide more than features
Most hosts offer a private mode. Fewer turn it on for you. Here’s what happens if you change nothing:
| Provider | Default, from its own docs |
|---|---|
| Ollama Cloud | Doesn’t log or train on prompts; requires zero data retention from the partners that host its models |
| Together AI | Stores prompts and responses; ZDR is a setting you turn on, and it applies only to requests after you enable it |
| OpenRouter | data_collection defaults to allow, which includes providers that store and may train; set zdr: true or the account setting to restrict routing |
| Groq | No retention by default, except debug and abuse logs kept up to 30 days unless you enable ZDR |
| Mistral | Keeps input and output 30 rolling days for abuse monitoring unless ZDR is approved on request |
| Fireworks | Doesn’t log or store prompts for open models unless you opt in; its Responses API keeps up to 30 days unless store=False |
| DeepInfra | Doesn’t store submitted data on disk or log request content |
| AWS Bedrock | Set data_retention_mode to none for zero retention; the default follows each model’s policy |
Terms can also differ model by model on one platform. On Bedrock, the open-weight gpt-oss models are excluded from the 30-day abuse retention that applies to OpenAI’s other models, while Anthropic’s Claude Fable models have all traffic retained up to 30 days for abuse detection. Check the terms for the exact model you call.
The options, from easiest to most private
Hosted open-model APIs with ZDR on
The cheapest way in, per token, with an OpenAI-compatible endpoint and no infrastructure. Privacy rests on the host’s policy, so read it and turn on its private mode. If you use a router, your request passes through two companies, the router and the host it picks.
Big cloud: Bedrock, Vertex, Azure
Pricier to set up, but you get region pinning, enterprise contracts, and compliance coverage. Several replies picked this path for exactly that reason; one runs Bedrock because it’s covered under their company’s HIPAA business associate agreement (BAA). Bedrock’s docs say model providers have no access to your prompts, and its none retention mode writes nothing to durable storage.
Vertex and Azure don’t train on your data, with caveats to know. On Vertex, customers on the standard self-serve Google Cloud terms are in scope for prompt logging when abuse classifiers flag a request, and Grounding with Google Search keeps derived queries for up to three days with no way to turn that off. On Azure, flagged content can go to human review unless you’re approved for modified abuse monitoring.
If AWS setup is the blocker, one reply had a practical fix: an agent can write the Terraform or drive the console for you.
EU-hosted
Three replies pointed to EU providers for data residency. Lyceum runs serverless open models in Paris and Finland, says it doesn’t keep prompts or outputs, and attaches a DPA to its terms. Mistral offers EU hosting with the 30-day default above. Scaleway runs inference in Paris.
Rented GPUs you run yourself
RunPod, Modal, Lambda, and Vast.ai rent you a GPU by the hour or second, and you run vLLM or Ollama on it. You control the logs and the endpoint. The host operator still has physical access to the machine, so this is privacy by isolation and policy, not cryptography. Use vetted data centers for sensitive data (RunPod’s Secure Cloud, Vast.ai’s Secure Cloud filter), and budget for it: an H100 running all month costs roughly $2,100 to $2,900.
Hardware enclaves you can verify
Confidential-computing GPUs run the model inside a hardware enclave, and the client checks a signed attestation before sending anything. Tinfoil’s SDKs verify the enclave and its published code on every connection, so you don’t have to trust Tinfoil’s operators. For gpt-oss-120b, Tinfoil lists the same per-token price as the hosted APIs above. Some enclave services don’t run every model in an enclave, so check per model.
Local
Nothing leaves your machine for inference. The model has to fit in GPU memory or Mac unified memory with room for context:
| Memory | What fits comfortably |
|---|---|
| 16GB | 4B to 8B models; gpt-oss-20b (14GB download) is tight |
| 24 to 32GB | 14B to 32B models at 4-bit; gpt-oss-20b |
| 64GB | 30B-class models with long context |
| 96GB and up, or an 80GB data-center GPU | gpt-oss-120b (65GB download) |
Local still leaves the machine itself to protect: disk encryption, backups, chat history, and never exposing the model’s port to the network without auth. Some local apps also ship cloud features, such as hosted models or web search, so check which calls stay local.
Price and privacy, side by side
Per 1M tokens, input / output, as listed on each provider’s pricing page at publication.
| Option | Where it runs | Default retention | Example price |
|---|---|---|---|
| Ollama Cloud | Mainly US, may route to Europe | Not logged or trained on | gpt-oss-120b $0.15 / $0.60 |
| Together AI | Optional US-only endpoint | Stores until you enable ZDR | gpt-oss-120b $0.15 / $0.60 |
| Fireworks | Not stated | No logging for open models | gpt-oss-120b $0.15 / $0.60 |
| AWS Bedrock | Your chosen region | Follows each model’s policy; set none for zero retention |
Qwen3 Next 80B $0.15 / $1.20 |
| Lyceum | Paris, Finland | Not stored; DPA in terms | GLM-5.3 $1.40 / $4.40 |
| Tinfoil (enclave) | Confidential-computing GPUs | Enforced by hardware, verifiable | gpt-oss-120b $0.15 / $0.60 |
| Rented H100 | Provider data center | Whatever you configure | $2.89 to $3.95 per hour |
| Local | Your machine | None leaves it | Hardware up front |
The model is one leak point of several
Swapping the host doesn’t protect data that leaves through the rest of the app:
- Tools the agent calls. Web search and fetch tools are separate services with their own terms. OpenRouter notes its ZDR covers inference routing, not plugins like web search. A search query built from a bank statement can carry the sensitive part.
- Tracing and observability. Traces hold full inputs and outputs. LangSmith, for example, keeps them 14 or 180 days and lets you hide content with
LANGSMITH_HIDE_INPUTSandLANGSMITH_HIDE_OUTPUTS. - Embeddings and vector stores. A hosted embeddings API sees the full document. OWASP warns that embeddings can be inverted to recover source text, so keep the store on infrastructure you control, with access controls.
- Test fixtures and logs. One reply found real account data in their test fixtures, and their secret scanner passed clean. Scan fixtures, seed data, eval sets, and request logs for real records.
- What you send at all. Four replies suggested redacting or pseudonymizing personal details before they reach any model. Swap names and account numbers for placeholders on your side, and put them back after the answer comes back.
What I’d do for bank statements on a budget
- Start with an enclave you can verify, or a big-cloud model with retention set to
none. - Redact personal details before anything leaves your app.
- Keep embeddings and the vector store on infrastructure you control.
- Turn off content capture in tracing, and scan fixtures and logs.
- Build a small eval set on redacted samples and pick the model by results, as one reply recommended.
Your private model still needs the web
Whichever column you pick, an open model only knows what it was trained on. When your app needs a current rate, a bank’s wire rules, or what changed on a page this week, the model needs the web. Wiring that in usually means a search tool, a fetcher, and a loop for the model to plan, read, and cite on its own, with your queries going to whoever runs the search.
Tabstack gives your model the web without giving up your privacy. One call returns a cited answer, clean text, matching JSON, or a completed browser task, and Tabstack runs the search, the browsers, and the web loop outside your model. It works with the setups in this post, local or cloud, through the Python and TypeScript SDKs or the CLI.
FAQ
Is a hosted open model more private than a closed model’s API? Not by itself. Once it’s hosted, the host’s retention, abuse-monitoring, and region terms decide privacy. Closed-model APIs also offer ZDR, often by agreement rather than a self-serve setting.
Is turning on ZDR enough? It covers the model call. Check where the promise is written, what abuse-monitoring exceptions apply, and what else in your app sends data out.
What’s the cheapest private option? For gpt-oss-120b, an attested enclave lists the same per-token price as ordinary hosted APIs. Local costs more up front and nothing per token.
Do I need an EU provider? If your users or contracts require EU data residency, yes. Pin the region and check where the provider’s GPU hosts are.
