Running LLMs locally — on your own hardware, without cloud APIs — has moved from novelty to practical capability. In 2026, a mid-range gaming PC can run capable models. A workstation can run very capable ones. Small edge devices run specialised models. The reasons to run locally range from privacy and cost to latency and experimentation. The ecosystem has matured dramatically — Ollama, LM Studio, llama.cpp, and vLLM serve millions of users. This guide covers why you might run LLMs locally, what hardware you need for different model sizes, which tools to use, what quality to expect, and how to decide whether local or cloud makes sense for your specific use case.
Why run LLMs locally
Legitimate reasons.
Privacy. Data never leaves your hardware. Important for sensitive information, regulated industries, or just principle.
Cost. After hardware investment, inference is "free" compared to API costs. Economically relevant at high usage.
Latency. No network round trip. Faster for some applications.
Offline. Works without internet. Matters for specific deployment environments.
Customisation. Fine-tuning, specialised quantisation, custom sampling — full control.
Learning. Understanding what is actually happening in AI by running it yourself.
Experimentation. Rapid iteration without rate limits or API costs.
These reasons matter variably. For most casual use, cloud APIs win. For specific needs, local is compelling.
When cloud wins anyway
Honest about limits.
Peak capability. Best models (Claude Opus, GPT-o3, Gemini Ultra) only available through API.
Ease of use. APIs are always available. Local setups require maintenance.
Scaling. Cloud scales effortlessly. Local is capped by your hardware.
Multimodal and specialised. Some capabilities only available in specific API products.
Cost at low volume. APIs cheaper than dedicated hardware if usage is modest.
The honest answer. For most needs, API is right. Local is for specific scenarios where its advantages matter.
Hardware requirements by model size
Practical tiers.
1-3B parameter models. Run on any modern laptop. CPU fast enough. 8GB RAM sufficient. Models like Phi-3, Llama 3.2 3B, Gemma 2B. Useful for simple tasks.
7-8B parameter models. Gaming GPU (RTX 4060 or better) with 8GB+ VRAM. Or Apple Silicon Mac (M2+ with 16GB+). Decent laptops with GPU. Models like Llama 3.1 8B, Mistral 7B. Capable for many tasks.
13-14B models. Better GPU (4070 or better) with 12GB+ VRAM. Apple Silicon with 24GB+. Not trivial but accessible.
30-34B models. Workstation territory. 24GB+ VRAM (RTX 4090). Or 48GB Apple Silicon. Significant hardware investment.
70B models. Multiple GPUs or high-end Apple Silicon (64GB+). Serious setup.
200B+ models. Data center gear. Not practical for individuals.
Most home users run 7B-14B models. Serious enthusiasts run 30-70B.
Quantisation
How bigger models fit on smaller hardware.
The technique. Reduce precision of model weights. Smaller memory footprint. Slight quality impact.
Formats. Q8 (near-lossless), Q6 (very minor loss), Q5 (small loss), Q4 (moderate loss), Q3/Q2 (significant loss).
Practical recommendation. Q4 or Q5 for most purposes. Q8 when quality critical and memory allows.
GGUF format. Most common for llama.cpp-based tools. Various quantisation levels available.
GPTQ, AWQ. Other quantisation formats. Different tradeoffs.
Effect on hardware requirements. Q4 quantisation reduces memory need by roughly 4x versus full precision. Makes bigger models accessible.
Tool ecosystem
What people actually use.
Ollama. Most popular. Simple to install. Large model library. CLI and API. Good defaults.
LM Studio. GUI-based. Model discovery. Local API server. Good for less technical users.
llama.cpp. Underlying library. C++ implementation. Many frontends.
vLLM. Higher performance inference. More complex setup. Production-oriented.
Text Generation WebUI. Feature-rich web interface. Less maintained recently.
Jan. Privacy-focused GUI alternative.
Msty. Another GUI option with good UX.
Most home users should start with Ollama or LM Studio. Move to more specialised tools for specific needs.
Model selection
What to actually run.
Llama 3.1 8B Instruct. Good all-around. Strong performance for size. Permissive license.
Mistral 7B Instruct v0.3. Strong alternative to Llama. Very efficient.
Qwen 2.5 7B Instruct. Strong multilingual. Competitive performance.
Phi-3 Medium. Microsoft model. Very efficient. Small hardware footprint.
Gemma 2 9B. Google model. Competitive.
Coding-specific. Qwen Coder, DeepSeek Coder, CodeLlama. For coding tasks, specialised models outperform general.
For serious users, 30B+ tier. Mixtral 8x7B (MoE architecture). Llama 3 70B. Qwen 2.5 72B. Capable but hardware-intensive.
Installation walkthrough (Ollama)
Concrete setup.
Install. Download Ollama from ollama.ai. Run installer. On Linux, curl | sh. On Mac, install app.
First model. ollama run llama3.1. Downloads model, starts chat. About 5GB download.
Other models. ollama run mistral, ollama run qwen2.5:7b. Easy to try various models.
API. Ollama runs API server at localhost:11434. Compatible with OpenAI API format for many uses.
Integration. Many tools support Ollama. VSCode plugins, chat interfaces, applications.
Resource usage. Check task manager while model runs. Understand what your hardware handles.
Most users have Ollama working in 15 minutes.
Performance expectations
What speeds to expect.
7B model on RTX 4070. 50-100 tokens/second typical. Fast for most uses.
13B on same hardware. 30-50 tokens/second.
7B on M2 Mac with 16GB. 30-50 tokens/second.
70B on 2x RTX 4090. 15-25 tokens/second. Usable but slower.
7B CPU-only (good CPU). 5-15 tokens/second. Usable for non-interactive tasks.
Comparison to APIs. Cloud APIs typically 50-200 tokens/second. Local can be competitive on consumer GPUs.
Quality expectations
How local models compare.
7-8B local vs frontier API. Significant quality gap. Local models weaker on reasoning, complex tasks, nuanced responses.
70B local vs frontier API. Closer. Competitive for many tasks.
Specialised local models vs API. Can match or exceed on specific tasks (coding models, domain-specific models).
The honest assessment. Local models are capable but not at frontier. Gap closing but persistent.
For tasks where frontier quality matters, use API. For tasks where "good enough" suffices, local works.
Privacy and local inference
The biggest value proposition for many.
Your data stays local. Nothing sent to external providers. Important for sensitive work.
No logging by third parties. Model provider cannot log your queries.
Compliance benefit. Easier to satisfy regulations requiring data residency.
Practical caveat. Your system still logs in your OS, any apps you use. Privacy requires attention at multiple layers.
For people and organisations with privacy-sensitive uses, this benefit often justifies local setup on its own.
Cost analysis
Economics of local versus cloud.
Hardware investment. Gaming PC with capable GPU: $1,500-3,000. Mac with serious silicon: $2,500-5,000. High-end workstation: $5,000-15,000+.
Electricity. GPUs use significant power. $10-30/month in electricity for moderate use.
API costs for comparison. Moderate use: $20-200/month. Heavy use: $500-5,000/month.
Break-even. Local wins at moderate-to-heavy sustained use over 12-24 months. Casual use, cloud APIs cheaper.
Organisational scale. For teams with many users, local hosting can be substantially cheaper.
Depreciation. Hardware loses value but still serves. Cloud costs keep coming.
Use cases where local shines
Specific good fits.
Coding assistance for privacy-sensitive work. Code never leaves your machine.
Document Q&A over personal documents. Private by design.
Data analysis on sensitive datasets. Medical, financial, legal.
Experimentation. Rapid iteration without API costs.
Offline work. Travel, remote locations, unreliable connectivity.
Regulated industries. Healthcare, financial, government contexts where data residency matters.
Custom fine-tuning workflows. Full control over training and inference.
Use cases where cloud remains better
Honest acknowledgment.
Cutting-edge reasoning. Research, complex analysis benefiting from frontier models.
High variability of use. Bursty usage patterns poorly served by fixed hardware.
Convenience. No maintenance, always available, works from any device.
Multimodal. Image, audio, video capabilities often better via API.
Integration. Cloud services integrate easily with cloud AI.
Not all uses benefit from local inference.
Running local models in production applications
Beyond personal use.
Integration via API. Ollama, LM Studio, vLLM all expose HTTP APIs compatible with OpenAI format.
Your application calls local API. Same pattern as cloud API but pointed at localhost.
Production considerations. Monitoring, error handling, scaling within hardware.
Team deployment. Self-hosted server for team access. Single inference box serves many users.
Fallback. Cloud API as fallback when local unavailable or insufficient.
Many small-to-medium businesses deploy local LLMs for internal tools. Pattern is proven.
Apple Silicon for LLMs
Specific note on this ecosystem.
M2, M3, M4 series strong for LLM inference. Unified memory architecture is advantage.
Memory matters more than anything. 16GB minimum for serious use. 32-64GB for large models.
Native tools. MLX framework for Apple-optimised inference. llama.cpp has Metal acceleration.
Power efficiency. Macs draw less power than GPU PCs. Relevant for always-on uses.
Cost. Apple premium real. Dollar-for-dollar GPU PCs cheaper for pure inference speed.
For developers wanting quiet, efficient, capable machine. Macs with lots of RAM are great.
Edge device LLMs
Beyond desktop/workstation.
Smartphone inference. Apple Intelligence, Gemini Nano, proprietary models on Samsung. 3-4B parameter models running on phones.
Raspberry Pi and similar. 1-3B models run (slowly). Good for learning, specific use cases.
NVIDIA Jetson. ARM SBCs with GPUs. Good for robotics, IoT.
Pattern. Edge AI growing. Smaller models specifically designed for edge.
Not all AI tasks need massive models. Right-sized models on appropriate hardware is often the answer.
Advanced: building custom serving stacks
For serious deployments.
vLLM. Python-based. Higher performance than defaults. Good for production.
TensorRT-LLM. NVIDIA-specific. Best performance on NVIDIA hardware.
Triton. NVIDIA inference server. Production-grade.
Custom setups. Model quantisation, speculative decoding, prefix caching — advanced optimisations.
These require specialised knowledge but meaningfully improve performance and capability.
Ongoing maintenance
What local LLM setup requires.
Updates. New models regularly. Keep software current.
Hardware maintenance. GPUs get dusty. Temperatures matter. Occasional attention needed.
Monitoring. Understand system behavior. Catch problems.
Backups. Configurations, custom models, fine-tunes. Backup strategy.
Security. Local systems exposed to network need security attention.
Not substantial but not zero either.
Worked example: a writer's local setup
Concrete scenario.
User. Professional writer with privacy concerns about manuscripts.
Hardware. M3 MacBook Pro with 32GB RAM.
Software. Ollama with Llama 3.1 8B and Qwen 2.5 7B. LM Studio for GUI.
Integration. Custom tool for manuscript analysis. Ollama API backend.
Workflow. Submit chapter, get structural analysis, suggestions, continuity checks. All local.
Cost. $3,000 one-time hardware. $0 ongoing inference cost.
Quality. Sufficient for tasks user cares about. Not frontier but adequate.
Privacy. Full — no content ever leaves the machine.
Worked example: a small business local LLM
Another scenario.
Organisation. Law firm with 20 lawyers. Sensitive client data. Strong privacy preferences.
Infrastructure. Workstation with 2x RTX 4090 in office. Hosts Qwen 2.5 72B quantised. Serves firm via internal API.
Use. Document analysis, drafting assistance, research. All local.
Cost. $10,000 hardware investment. Compared to equivalent API usage ($3,000-10,000/month for 20 users), pays back in months.
Maintenance. IT person spends a few hours per week on upkeep.
Compliance. Satisfies client privacy expectations better than any API option would.
Pattern. Increasingly common for privacy-sensitive businesses.
Benchmarking your setup
Understanding what you have.
Speed tests. Tokens per second across different models and prompts. Establishes baseline.
Quality tests. Your specific tasks evaluated. Catch regressions when changing models.
Memory usage. How much headroom at different model sizes.
Comparison. Your setup versus commonly reported numbers.
Having numbers lets you make informed decisions about upgrades or changes.
Learning and experimentation
A genuine reason for local.
Understanding prompting. Rapid iteration without API costs teaches you what models actually do.
Model comparison. Try many models side-by-side. Develop intuition for strengths and weaknesses.
Sampling experimentation. Temperature, top-p, other parameters. Direct control.
Fine-tuning experimentation. Small-scale fine-tuning on local hardware. Learn the mechanics.
For people genuinely wanting to understand AI, nothing beats running it yourself.
Troubleshooting common issues
What goes wrong and how to fix.
Out of memory. Model too large for available VRAM. Use more aggressive quantisation or smaller model. Close other applications using GPU.
Slow inference. CPU fallback when GPU expected — check configuration. Thermal throttling — improve cooling. Background processes — free resources.
Poor quality. Wrong model for task — try alternatives. Temperature settings — adjust. Prompt quality — improve.
Connection issues with API. Firewall blocking localhost — adjust. Server not running — restart. Wrong port — check config.
Model downloading failures. Disk space — free up. Network issues — retry. Version conflicts — reinstall.
Most issues have straightforward fixes. Community forums (Reddit, Discord) are helpful resources.
Multi-user local deployments
An increasingly common pattern. A single workstation or server hosts models. Multiple users in the same organisation connect to it via API. This scales local inference beyond individual use. Typical setup: a machine with one or two high-end GPUs runs vLLM or similar production inference server. Internal network exposes API at a known endpoint. Users connect from their devices using standard OpenAI-format API calls.
Considerations for multi-user deployments. Concurrency matters — can the model serve multiple simultaneous requests? Batching improves throughput but adds latency. Queuing handles burst usage. Authentication required even on internal networks for audit and access control. Monitoring critical — you need to know when the system is saturated. Rate limiting prevents a single user from monopolising resources. These considerations resemble those for any production service, with GPU-specific wrinkles. The economics compelling for teams of 10-50 users where API costs would otherwise be significant.
Comparing inference servers
A practical comparison for deployment choices. Ollama — simplest, good for development and light production use. Performance adequate for most cases. LM Studio — GUI-focused, serves API, good for non-technical users to run local models. Not production-oriented. llama.cpp server — lightweight, C++ based, good for embedded and resource-constrained deployments. vLLM — higher performance, Python-based, designed for production throughput. More setup complexity. TGI (Text Generation Inference) — HuggingFace's production server, strong feature set. TensorRT-LLM — NVIDIA-optimised, best performance on NVIDIA hardware but platform-specific.
Selection criteria. Production scale and throughput requirements. Ease of setup and maintenance. Hardware constraints. Feature requirements (streaming, batching, specific quantisation formats). Budget for engineering effort. For most teams, Ollama suffices for individual use and small team deployment. vLLM becomes compelling for production deployments serving many users. More specialised options for specific requirements.
The local inference talent question
An organisational consideration. Running local LLMs at production scale requires specific expertise. Hardware selection and optimisation. Model selection and quantisation. Inference server configuration. Monitoring and troubleshooting. Staying current with rapidly evolving tooling. Not every organisation has this expertise or wants to develop it. The alternatives. Managed private deployment — providers offering private hosting on dedicated infrastructure. Higher cost than self-hosted but reduces expertise requirement. Hybrid approaches — self-host what you can maintain, use APIs for the rest. Consultants and professional services — bring in expertise for setup, maintain in-house for operations. Each has tradeoffs. The key insight is that "going local" is not free of ongoing cost; it shifts cost from API fees to infrastructure and expertise. Evaluating total cost of ownership, not just nominal inference pricing, drives better decisions.
Local LLMs are no longer a novelty — on a reasonable gaming PC or modern Mac, you can run genuinely useful models for privacy-sensitive or cost-sensitive work. The quality gap to frontier APIs persists but has narrowed dramatically.
The short version
Running LLMs locally in 2026 is a practical capability. Consumer hardware runs capable 7-14B models. Workstations run 30-70B. Tools like Ollama, LM Studio, and llama.cpp make setup accessible. The quality gap to frontier cloud models persists but has narrowed substantially. Good fits — privacy-sensitive work, cost optimisation at scale, offline deployment, experimentation, regulated industries. Poor fits — casual use where convenience matters more than privacy, tasks requiring frontier capability, scenarios where hardware investment does not amortise. Most users benefit from hybrid approach — local for what local does well, cloud for what cloud does well. For privacy-sensitive use cases, local inference is often the only acceptable path, and 2026 makes it accessible in ways that were impossible a few years ago.