How to Run an LLM Locally: Complete Guide…

How to Run an LLM Locally: The Complete Guide

What if I told you the biggest reason people give up on local AI models has nothing to do with their hardware?

How to Run an LLM Locally: Complete Guide — featured image Photo by Pavel Danilyuk on Pexels

Here's the deal. If your GPU has 8GB of VRAM, you can run a genuinely useful 7–8 billion parameter model on your own machine today, offline, for the cost of electricity. If it has 24GB, you're in 30B territory. And if you're stuck on a laptop CPU with 16GB of RAM, 3–4B models still run at readable speed — just don't expect miracles.

That's the whole answer compressed into three sentences. The rest of this guide explains the math behind those numbers so you can make the call for your own hardware instead of guessing and hoping.

The thing that surprised me most when I started doing this: installation isn't the hard part. Installation takes four minutes. The hard part is that most people pick a model that's way too big, watch it crawl along at 2 tokens per second, and conclude local LLMs "don't work." They work fine. The sizing was wrong. That's it. That's the entire failure mode, and I'd bet money it accounts for 80% of the people who tried this once and never came back.

Who this guide is for:

Privacy-constrained professionals — lawyers, clinicians, finance teams handling data that legally can't leave the building
Developers who want an API-compatible endpoint without watching a per-token meter spin
Anyone curious about what open-weight models actually cost to run

What you'll learn:

The VRAM formula that tells you which models fit your machine before you download 40GB of nothing
A step-by-step setup using free, open-source tools — no account, no credit card, no email capture
The seven mistakes that make people quit, and how to sidestep each one

Why Bother Running an LLM Locally?

Let's be honest about the tradeoff first. Cloud models are cheap, fast, and always the newest shiny thing. So what's the argument for taking the harder path?

Privacy is the strongest case, full stop

When you send a prompt to a hosted API, that text leaves your network. For most people, honestly, that's completely fine. For some it's a compliance problem with real teeth.

Under HIPAA, protected health information sent to a third-party processor generally requires a Business Associate Agreement — see the U.S. Department of Health and Human Services guidance on business associates. Attorney work product raises similar questions. A local model sidesteps the entire category: no data leaves the machine, so there's nothing to govern.

The NIST AI Risk Management Framework treats data governance as a core function of trustworthy AI systems. Local deployment is one of the bluntest, dumbest, most effective ways to satisfy it.

The cost math flips once you hit volume

One-off questions? Cloud wins, easily, and it isn't close. But automated pipelines chewing through thousands of documents nightly? Different story.

A used 24GB GPU runs roughly $700–1,400 in 2026. At typical API rates, heavy batch workloads can burn through that in three or four months. Electricity for a 300W card running eight hours a day comes to maybe $8–15/month depending on what your utility charges.

I'll be honest about the flip side, because a lot of guides won't be: your time isn't free either. If setup and maintenance eats ten hours, price that in. At $100/hour of your time, that's a $1,000 hidden line item that nobody puts on the spreadsheet.

Two misconceptions worth killing right now

"You need a $2,000 GPU." Nope. Modern 3–4B parameter models running on quantized weights are genuinely capable at summarization, extraction, and classification. Those run on integrated graphics and Apple Silicon laptops that people already own.

"Local models are years behind." This one was true in 2023 and people just kept repeating it. The gap narrowed sharply. Open-weight releases from Meta, Mistral, Alibaba, and Google now land within months of frontier capability on plenty of benchmarks — though frontier reasoning models still lead clearly on hard math and long-horizon agentic tasks. Just know what you're trading away before you trade it.

Core Concepts: The Vocabulary That Actually Matters Photo by Pavel Danilyuk on Pexels

Core Concepts: The Vocabulary That Actually Matters

Before we get to the setup steps, you need six terms. Skip this section and the error messages later will read like hieroglyphics. If you re-read one part of this guide, make it this one.

Parameters, weights, and why size predicts everything

A model's parameters are the learned numbers inside it. "8B" means eight billion of them. More parameters generally means more capability, and strictly means more memory. That second part isn't a tendency, it's arithmetic.

Weights are those parameter values as stored on disk. Open-weight models publish them for download. That's a different thing from "open source" — most open-weight models ship under licenses with usage restrictions, so actually read the license if you're deploying commercially. I know nobody reads licenses. Read this one.

Quantization is the single most important concept here

Models train at 16-bit precision. Each parameter takes 2 bytes. So an 8B model at full precision needs about 16GB just for the weights, before anything else.

Quantization compresses those numbers down to fewer bits. That same 8B model at 4-bit drops to roughly 4.5GB. You lose some accuracy. Usually a lot less than you'd expect — this was the thing that genuinely shocked me the first time I A/B tested it.

Quantization	Bits per parameter	8B model size	Quality impact
FP16 / BF16	16	~16 GB	Baseline (none)
Q8_0	8	~8.5 GB	Essentially imperceptible
Q6_K	6	~6.6 GB	Very minor
Q5_K_M	5	~5.7 GB	Minor, hard to notice
Q4_K_M	4	~4.9 GB	Small but measurable
Q3_K_M	3	~4.0 GB	Noticeable degradation
Q2_K	2	~3.2 GB	Often unusable

Q4_K_M is the default answer for roughly nine out of ten people. It's the quality-per-gigabyte sweet spot, and it's what I reach for unless I have a specific, articulable reason not to.

Honest caveat: quantization damage isn't uniform across tasks. Code generation and multi-step arithmetic degrade noticeably faster than summarization does. If your use case is code, spend the extra couple of gigabytes on Q6 or Q8 and don't think about it again.

Context window, KV cache, and the memory nobody budgets for

Context window is how much text the model can hold in its head at once, measured in tokens (roughly 0.75 words each). A 128K context window handles about 96,000 words — call it a short novel.

Here's the trap that gets everybody. Context isn't free. The KV cache stores intermediate state for every single token in context, and it lives in the exact same memory as your weights. A long context can eat several gigabytes entirely on its own.

That's why a model that loaded perfectly fine crashes twenty messages into a conversation. Nothing broke. You just ran out of room, quietly, one message at a time.

Tokens per second, and what actually counts as usable

Speed (tok/s)	Feel	Good for
Under 5	Painful	Batch jobs only
5–15	Slow but workable	Background processing
15–40	Comfortable	Interactive chat
40+	Snappy	Coding assistance, agents

Human reading speed sits at roughly 5–8 tokens/second. Anything above 15 feels perfectly fine in conversation — the model is outpacing your eyes and that's all you need.

GPU offloading: the hybrid middle ground

If a model doesn't fully fit in VRAM, inference engines can split it up — some layers on the GPU, the rest over on CPU RAM. It works. It's also slow, because CPU RAM bandwidth is typically 5–10× lower than GPU VRAM bandwidth.

Rule of thumb: even 20% of layers stranded on the CPU can cut throughput by half or worse. Fitting entirely in VRAM is worth real effort, and it's worth downgrading a model size to get there.

Apple Silicon is a genuine special case

Macs with M-series chips use unified memory — CPU and GPU share one pool instead of shuffling data between two. A 32GB MacBook can allocate roughly 24GB to a model with no discrete GPU anywhere in the picture. Memory bandwidth on the Pro and Max chips is high enough that this works genuinely well, not "well for a laptop" well.

Hot take: Apple Silicon is the most underrated local LLM platform, and it isn't particularly close. A used M1 Max with 32GB is a serious inference machine that also happens to be silent, run on battery, and fit in a bag. Meanwhile people are building open-air GPU rigs in their spare bedroom and heating the room to 30°C. (Related tangent: the used Mac market got weird after the M4 launch — a lot of people dumped M1 Max machines for reasons that have nothing to do with inference performance, which stayed excellent. Their impatience, your bargain.)

Setting It Up, Step by Step

Now the actual steps. This section assumes zero prior setup — no Python environment, no CUDA background, nothing.

Step 1 — Measure your actual VRAM

Don't guess. Don't go off the spec sheet you half-remember. Check.

Windows: Task Manager → Performance → GPU → "Dedicated GPU memory"
Linux: run nvidia-smi (NVIDIA) or rocm-smi (AMD)
macOS: Apple menu → About This Mac → Memory (this is unified memory)

Write that number down somewhere. Every single decision after this depends on it.

Step 2 — Run the sizing formula

Estimated memory needed:

(Parameters in billions × bits per parameter ÷ 8) × 1.2 = GB required

That 1.2 multiplier covers KV cache and general overhead. For contexts above 16K, bump it to 1.4.

Worked example — 8B model at Q4:

(8 × 4 ÷ 8) × 1.2 = 4.8 GB

Fits comfortably in 8GB VRAM with room to spare. Second example — 70B at Q4:

(70 × 4 ÷ 8) × 1.2 = 42 GB

That one needs dual GPUs or a 64GB+ Mac. Definitely not a consumer card, no matter how many forum posts claim otherwise.

Your VRAM / unified RAM	Largest comfortable model (Q4_K_M)	Realistic expectation
6 GB	3–4B	Fast, good at simple tasks
8 GB	7–8B	The sweet spot for most people
12 GB	12–14B	Noticeably better reasoning
16 GB	14–22B	Strong general capability
24 GB	27–32B	Near-frontier on many tasks
32–48 GB	32–70B (tight)	Serious work
64 GB+	70B+ comfortable	Diminishing returns per dollar

Step 3 — Pick an inference engine

There are three real options in 2026, and everything else is a wrapper around one of them.

Ollama — command line, one-line installs, built-in model library, OpenAI-compatible API sitting on port 11434. This is what I'd tell most people to start with. Free, MIT-licensed.

LM Studio — graphical app, model browser with VRAM-fit indicators built right in, chat UI included. Best if you want to avoid the terminal entirely, and there is zero shame in that. Free for personal use.

llama.cpp — the C++ engine humming underneath both of the above. Maximum control, maximum fiddling, maximum time spent reading GitHub issues. Choose this if you're embedding inference in your own software.

Engine	Interface	Setup time	Best for
Ollama	CLI + API	~5 min	Developers, automation
LM Studio	GUI	~10 min	Non-technical users
llama.cpp	CLI (build required)	20–60 min	Custom integration

Step 4 — Install and pull your first model

With Ollama, here's the entire process:

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run an 8B model
ollama run llama3.1:8b

First run downloads roughly 4.7GB. Then you're chatting. That's genuinely it — two commands and you have a language model running on your own hardware with no account anywhere.

Windows users grab an installer from the official site, then run the same commands afterward in PowerShell.

Step 5 — Verify it's actually using your GPU

This is the step everyone skips, and it's precisely why they walk away thinking local models are slow.

Run a prompt, then check GPU utilization (nvidia-smi on Linux/Windows, Activity Monitor → GPU on macOS). If utilization is sitting near zero while the model generates text, congratulations, you're running on CPU. Something's wrong — usually missing drivers or a CUDA/ROCm toolkit version mismatch.

Ollama reports layer placement in its logs. Look for a line showing how many layers got offloaded to GPU. You want all of them, not most of them.

Step 6 — Tune context length to what you actually need

Default context is often somewhere between 2K and 8K tokens. Raising it costs memory, immediately and linearly.

# Ollama: set context at runtime
/set parameter num_ctx 8192

Don't set 128K "just in case." Look, I get the impulse — bigger number, better model, right? But you'll allocate gigabytes of KV cache you never touch, and you'll push actual model layers off the GPU to make room for empty space. Match context to your longest realistic input and nothing more.

Step 7 — Wire it into your existing tools

Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1. Most libraries that talk to OpenAI will happily talk to it by changing the base URL and passing literally any placeholder string as the API key.

Which means existing scripts often work with a two-line change. Fun fact: this compatibility layer is arguably why Ollama won the adoption race — not because the inference was better, but because nobody had to rewrite anything. For a deeper walkthrough of connecting local models to applications, see our related guide.

Seven Mistakes That Make People Quit

Ranked by how often I've watched them derail somebody.

1. Downloading a model that doesn't fit

The most common failure by an enormous margin. Someone reads that a 70B model is good, downloads 40GB over two hours, and gets 1.5 tokens/second because 80% of it landed in CPU RAM. Then they uninstall everything and tweet about how local AI isn't ready.

Fix: run the sizing formula before you download. Every time. It takes ten seconds.

2. Forgetting the KV cache exists

Weights fit in 7.5GB, you have 8GB, seems fine — right up until context fills and it OOMs mid-conversation, usually on the one long document you actually cared about.

Fix: leave 20–30% headroom. That's exactly what the 1.2 multiplier is there for.

3. Over-quantizing down to Q2 or Q3

Q2 models load fast and produce confident nonsense, which is the worst possible combination. The degradation is real and it hits hardest exactly where you'd notice it least — subtle factual errors rather than obvious gibberish. Gibberish you catch. A plausible wrong number you ship.

Fix: treat Q4_K_M as the floor. Below that, use a smaller model at higher precision instead. A 4B at Q6 beats an 8B at Q2 for basically all real work.

4. Not verifying GPU offload

Covered above, but honestly it bears repeating, because the symptom (it's slow) looks completely identical to the wrong diagnosis (my model is too big). People downgrade to a smaller model, get the same 2 tok/s, and give up entirely.

Fix: check utilization on your first run, and again every time you change hardware or update drivers.

5. Grabbing a base model instead of an instruct model

Base models complete text. Instruct/chat models follow instructions. Pull a base model expecting conversation and you'll get rambling continuations of your own question, which is a genuinely disorienting experience the first time.

Fix: look for -instruct, -chat, or -it in the model name. That's the whole fix.

6. Skipping the license

Open weights aren't public domain, and the vibes-based assumption that they are will eventually bite somebody expensively. Some licenses restrict commercial use above a user threshold; some prohibit specific application categories outright. The Open Source Initiative's definition of open source AI is worth ten minutes if you're deploying professionally — plenty of popular "open" models don't actually meet it.

Fix: read the model card's license section before you build anything on top of it.

7. Expecting frontier-tier reasoning

Local models are strong at extraction, summarization, classification, drafting, and routine code. They're weaker at multi-step reasoning, long-horizon planning, and obscure factual recall. That's not a bug, it's a size constraint.

Fix: match the tool to the task. Hybrid setups — local for volume, cloud for the genuinely hard problems — are where most serious users end up after a few months. See our related guide on splitting workloads.

Three People Who Actually Did This Photo by Sinan KRIYA on Pexels

Three People Who Actually Did This

Concrete cases, real numbers.

Case 1 — Solo attorney, privacy-first document review

Setup: M2 Pro Mac Mini, 32GB unified memory, about $1,200 used. LM Studio running a 14B instruct model at Q5_K_M (~10GB).

Workload: summarizing discovery documents and drafting routine correspondence. Roughly 40 documents a week.

Result: ~28 tokens/second. Plenty fast for interactive use. Nothing touches the internet, which resolves the confidentiality question completely rather than partially.

Honest limitation: the 14B model occasionally misses nuance in dense contractual language. It's a first-pass tool, not a substitute for reading the document yourself.

Case 2 — Small dev team, CI code summarization

Setup: an existing Linux workstation with an RTX 4070 (12GB). Ollama serving a 14B code-tuned model at Q4_K_M, called via the local API from CI hooks.

Workload: roughly 200 pull-request summaries per month, plus ad-hoc queries throughout the day.

Result: ~45 tokens/second. The hardware was already sitting there, so marginal cost is basically electricity. The team estimated they'd been spending $60–90/month on API calls for equivalent volume — not enormous, but it was money leaving for something they already owned the hardware to do.

Honest limitation: they still route architecture-level review to a frontier model. The local model handles the routine 80% and knows its lane.

Case 3 — Researcher on a laptop, no GPU whatsoever

Setup: a ThinkPad with 16GB RAM and integrated graphics. Ollama with a 3B instruct model at Q4_K_M (~2GB).

Workload: classifying survey free-text responses into categories, offline, at 35,000 feet.

Result: ~9 tokens/second on pure CPU. Too slow for chat, perfectly fine for a batch script grinding through 800 responses over lunch.

Honest limitation: 3B models need tightly-specified prompts or they wander. Vague instructions produce inconsistent output. Few-shot examples in the prompt make an enormous difference at this size — bigger than at any other scale, in my experience.

Tools and Official Resources

Everything below is free. No affiliate links anywhere in this guide — these are just the sources I'd actually check myself.

Software

Tool	What it does	Cost
Ollama	CLI runner + local API server	Free, MIT
LM Studio	GUI model browser and chat	Free (personal)
llama.cpp	Core inference engine	Free, MIT
Hugging Face Hub	Model weights repository	Free

Standards, policy, and safety references

NIST AI Risk Management Framework — the U.S. federal reference for evaluating AI system risk. Directly useful if you're documenting a deployment for compliance.
NIST Generative AI Profile (AI 600-1) — companion guidance specific to generative models.
FTC guidance on AI claims and consumer protection — matters the moment your local model's output reaches a customer.
EU AI Act official text — relevant for anyone deploying in or serving the EU market.
Stanford HAI AI Index Report — annual, data-heavy, and the best free source for open-vs-closed capability trends.

Benchmarks worth trusting

Public leaderboards drift and get gamed, and I've grown pretty cynical about them. Treat any single benchmark number with suspicion and test on your own data instead — a model ranked 15th overall might be flat-out first for your specific task, and you'd never know from the leaderboard. The Stanford HAI report above is the most methodologically careful public source I've come across.

For a broader look at evaluating model quality, see our related guide.

Frequently Asked Questions

Is running an LLM locally actually free?

The software is free. The hardware isn't, and neither is electricity. If you already own a capable machine, marginal cost is a few dollars a month in power — genuinely trivial. If you're buying a GPU specifically for this, budget $400–1,400 and do the break-even math against API pricing honestly, including your setup time.

What's the minimum hardware that actually works?

16GB of system RAM and any reasonably modern CPU will run a 3–4B model at around 8–10 tokens/second. That's genuinely useful for batch tasks. For comfortable interactive chat, you want 8GB+ of VRAM or 16GB+ of Apple unified memory.

Can local models match GPT-class cloud models?

On everyday tasks — summarizing, extracting, classifying, drafting — a good 30B local model gets close enough that you'll stop noticing the difference. On hard reasoning, long-context work, and agentic multi-step tasks, frontier cloud models still lead clearly. Don't let anyone tell you the gap is zero. Don't let anyone tell you it's insurmountable either. Both camps are selling something.

Do I need to know how to code?

Nope. LM Studio is a normal desktop app — download, click, chat. Ollama needs exactly one terminal command. Coding only enters the picture if you want to wire the model into your own software.

Is my data really private?

Yes, with one caveat worth understanding. Inference runs entirely on your machine and generates zero network traffic. But the application wrapped around it might phone home for telemetry or update checks. Check your tool's privacy settings, and if it genuinely matters for your work, verify with a network monitor rather than taking anyone's word for it. Pulling a model does require internet once — after that, you can pull the ethernet cable and it'll keep working.

How much disk space should I plan for?

Each quantized model runs 2–45GB. Most people accumulate five or six while experimenting, because trying models is fun. Budget 100GB of free space, and use an SSD — load times on spinning disks are genuinely miserable.

What about fine-tuning on my own data?

Completely different problem, and much heavier. Fine-tuning needs substantially more memory than inference — often 3–5× the model size even with efficient methods like LoRA. Get inference working first. And honestly? For most use cases, a well-built prompt with retrieval over your documents beats fine-tuning outright, costs a fraction as much, and takes an afternoon instead of a week. Fine-tuning is overrated for the average use case and I'll stand by that.

Which model should I actually download first?

An 8B instruct model at Q4_K_M, assuming you've got 8GB+ of VRAM. Best capability-per-gigabyte point available, downloads in minutes, gives you a real baseline before you sink more time in.

The Verdict

Local LLMs crossed the line from "interesting demo" to "practical tool" sometime in the last eighteen months. If you have modern hardware, there's no longer a good reason not to have one sitting installed on your machine.

The takeaways:

Size before you download. The formula — (params × bits ÷ 8) × 1.2 — takes ten seconds and prevents the single most common failure by a mile.
Q4_K_M on an 8B instruct model is the default answer. Deviate only with a specific reason you can say out loud.
Verify GPU offload on your first run. Most "local models are slow" complaints are actually "my model is quietly running on CPU."

My hot take, for whatever it's worth: the people getting real value from local models aren't the ones chasing the biggest model their hardware can just barely hold. They're the ones running a comfortably-sized model fast, wired into an actual workflow, handling volume that would be annoying to pay for. It's boring. It works. Boring and working beats impressive and stalled every single time, and the local AI community could stand to internalize that a bit more.

Next step: check your VRAM right now — it takes fifteen seconds — run the formula once, and pull whatever model it points you to. You'll know within twenty minutes whether this fits how you work. And if it doesn't, you're out one download and nothing else.