Is a local LLM good enough to run an AI agent system?

It depends entirely on which job you give it. In my own system, local open-weight models on an 8GB GPU were fine for embeddings and hopeless for judgement: on a replay of 36 real financial emails, a hosted Kimi K2.5 extracted 32 correctly, while a local 4B model managed 13. Speed was worse than the accuracy gap — the local model averaged 23 seconds per answer against 2 seconds hosted. If your GPU is large and your workload is steady, local wins on cost. If your GPU is small and the work arrives in bursts, it loses on both speed and quality.

How much does it cost to run a personal AI system on a hosted model?

Mine costs about five cents a day. Over the last seven measured days my whole agent fleet — inbox triage, financial extraction, alert investigation, research summaries, daily briefings — spent $0.38 in total across roughly 290 model calls. That is the number that ended the argument for me: I had been protecting a cost that turned out to be less than a coffee a month.

Can you switch an LLM app between local and hosted models without rewriting it?

Yes, if you put a proxy in the middle from the start. I route every model call through LiteLLM and refer to models by tier — fast, balanced, smart — rather than by name. Which actual model a tier resolves to is one line of config. Moving the whole system from local Ollama to Bedrock was a config edit, not a refactor, and moving it back would be the same edit in reverse.

Should embeddings and chat use the same model?

No, and they usually should not. They are different jobs with different constraints. Embeddings are cheap, fast, run fine on modest hardware, and are effectively locked in once you have a corpus — the stored vectors and the query vector must come from the same model, and the vector dimension is baked into your database schema. Changing your chat model is a config edit; changing your embedding model is a migration plus a full re-index.

Why AEGIS moved off my local GPU — and still runs on yours

Yesterday I asked my own knowledge base a question through its web UI and watched the spinner for over a minute. Nothing was broken. The request completed, 200 OK, after 76.6 seconds.

The same question, on the model the rest of the system already used, would have taken about two.

That was the last push I needed to finish a migration I had been half-doing for months: AEGIS, my personal AI orchestration layer, now does its thinking on Amazon Bedrock instead of on the GPU under my desk.

I want to be careful about what that does and does not mean, because "local LLM guy moves to cloud" is a story people are keen to over-read. So: the models were not the problem. My GPU was. And the system still runs on a local GPU if that is what you have — that part is one line of config, and I will show you the line.

The honest reason

My homelab box has an 8GB GPU. It is not a rig. It also runs Postgres, and a handful of other things that were there first.

AEGIS asks a model to make a judgement maybe a few hundred times a day, in bursts: a batch of email arrives and wants classifying, an alert fires and wants investigating, a receipt lands and wants parsing into a ledger entry. Bursty, latency-visible, and each call matters — a wrong classification quietly puts a bill in the wrong place.

That is close to the worst possible shape for a small local GPU. The hardware is idle most of the day, then asked for six things at once, on a card that is also paging against a database.

This is not a new insight — I wrote the general version of this argument in local model or paid API: the honest math back in June. What follows is what happened when I actually ran the experiment on myself for three months.

What the production numbers said

Every model call AEGIS makes is recorded — model, purpose, latency, status, cost. Here is the last thirty days, unedited:

model	calls	avg latency	p95 latency
`qwen3.5:9b` (local)	2	41,341 ms	54,665 ms
`gemma4:e2b` (local)	41	23,364 ms	43,717 ms
`bedrock-kimi-k2.5`	368	2,010 ms	4,986 ms
`bedrock-glm-4.7-flash`	9	850 ms	1,390 ms

Ten to twenty-five times slower, on the same prompts, for the same jobs.

You will notice the local call counts are small. That is the migration in progress — those 41 calls are the knowledge-base answers that were still routed to the local model, which is the leftover that produced yesterday's 76-second spinner. The bulk moved months ago.

It was not only speed

I had assumed the trade was "local is slower but adequate". I tested that instead of assuming it, by replaying 36 real financial emails — ones I already knew the correct answer for — through each candidate model and scoring the extraction.

model	correct extractions (of 36)
Kimi K2.5 (hosted)	32
GLM 4.7 Flash (hosted)	24
gemma4:e4b (local)	13
qwen3.5:9b (local)	did not finish in budget

A model that gets 13 of 36 financial extractions right is not a cheaper version of one that gets 32. It is a different tool. For a system whose entire purpose is to interrupt me less, a model that is wrong two-thirds of the time doesn't save me work — it adds a review step to everything it touches.

The two failures that actually cost me

Benchmarks are tidy. The reasons I actually gave up on running judgement at home were messier, and both were silent.

A reasoning model ate its own output budget. Models that reason before answering bill that hidden reasoning against the same token limit as the visible reply. Give one a tight cap and it spends the whole budget thinking and returns an empty string. My morning briefing ran that way for six days — 100% of calls returning nothing — and produced no error, because an empty string is a perfectly valid response. The briefing simply fell back to its non-LLM version and looked fine.

A local model wedged the server for ninety minutes. One reasoning model, given a task it found hard, burned its budget, triggered a retry at a larger context, and locked up the inference server for an hour and a half. Restarting the container cleared it. Nothing else could get a model call through in the meantime — including the infrastructure agent whose job is to notice things like that.

Neither of these is an argument that local models are bad. They are arguments that a small, shared GPU is a single point of failure in a system that is supposed to be watching for single points of failure.

What it actually costs

This is the part that embarrassed me.

day	model calls	cost
2026-09-14	45	$0.099
2026-09-13	35	$0.059
2026-09-12	41	$0.073
2026-09-11	52	$0.052
2026-09-10	38	$0.042
2026-09-09	27	$0.028
2026-09-08	51	$0.026

About five cents a day. Roughly $1.50 a month to run the entire fleet — inbox triage, financial extraction, alert investigation, research summaries, daily briefings.

I had spent months optimising around a cost that turned out to be smaller than the electricity the GPU was drawing to be slower. That is the whole argument, and I did not have it until I measured it.

One honesty note: those figures are what my proxy reports the calls cost, not a reconciled provider invoice. Providers price caching and round in their own ways. For billed truth, read the bill.

It still runs on a local GPU

Here is the part I care most about, because it is the design decision that made the migration cheap and would make the reverse migration equally cheap.

AEGIS never names a model at a call site. Every call asks for a tier — fast, balanced, smart — and every call goes through LiteLLM, a proxy that speaks one OpenAI-shaped API to whatever is behind it. Which model a tier resolves to is config:

tiers:
  fast:     "bedrock-glm-4.7-flash"
  balanced: "bedrock-kimi-k2.5"
  smart:    "bedrock-kimi-k2.5"

Point those at Ollama aliases instead and the system runs entirely on your own hardware. No code changes. Nothing in the application knows or cares where inference happens.

There is a second layer above it, which is the one I used for yesterday's fix. Every call also declares a purpose — what job it is doing — and purposes map to categories, and categories map to models:

routes:
  categories:
    extract:  {model: "bedrock-kimi-k2.5"}   # structured JSON from documents
    classify: {model: "bedrock-kimi-k2.5"}   # short labels and verdicts
    write:    {model: "bedrock-kimi-k2.5"}   # prose a person reads
  purposes:
    money_event_extraction: extract
    gmail_classification: classify
    knowledge_ask: write

That last line is the 76-second bug, fixed. The useful unit of change is the category, not the call site: "every structured extraction moves to the local model" is one edit here, and the same change spread over twenty call sites is twenty chances to miss one.

If you have a 24GB card and a steady workload, set those to local aliases and you get a system with no per-token cost and no third party in the loop. The architecture does not have an opinion. My hardware did.

What stayed local, and why that is the interesting part

AEGIS still runs a local model. Every day, for everything it reads.

Embeddings never left. The knowledge store holds 507,066 chunks across 22,896 documents, all embedded with nomic-embed-text running locally. It is staying there, and not for sentimental reasons:

Embedding is a cheap, steady, non-judgement job. It is exactly the shape a small GPU is good at — no reasoning, no burst, no correctness cliff.
The stored vectors and the query vector must come from the same model, or cosine similarity is comparing two different coordinate systems. You cannot migrate one side.
The vector dimension is in the database schema — vector(768) with an index built on it. Changing the embedding model is a migration plus re-embedding half a million chunks, not a config edit.

So the same system, on the same day, sends a question's meaning to a local model and the question's answer to a hosted one. That is not a compromise between two positions. It is what happens when you stop asking "local or cloud?" and start asking it per job.

The rule I'd give someone else

Not "use Bedrock". The useful version is three lines:

Match the model to the job, not to a philosophy. Judgement, embeddings and bulk backfill have different constraints. A single answer for all three is a wrong answer for at least two.
Measure before you optimise. I defended a five-cent-a-day cost for months. One query against my own call log ended the debate in about a minute — and the only reason I could run it is that I had been recording every call from the start.
Keep the seam. A proxy and a tier name cost you almost nothing on day one and turn "we're moving providers" from a refactor into a config edit. Mine paid for itself twice: once moving out, and once when I wanted to move one single job back.

The thing I was actually afraid of — being locked into someone else's inference — turned out to be a problem I had already solved by accident, on the first day, by refusing to write a model name in application code.

AEGIS is open source, MIT-licensed: github.com/hikmahtech/aegis. The model tier map, the routing table and the call ledger described here are all in the repo — config/models.yaml and the llm_calls table.