Self-hosted LLMs: a practical guide to on-prem AI for businesses
Running your own large language model on servers you control means sensitive data never goes to a third-party API. Here's when self-hosting is worth it, what you need, and how to avoid the expensive mistakes.
By Laith Al Abbadi, Fractional CTO & AI expert · Updated 15 November 2026
Why companies self-host LLMs
- Data privacy and residency: regulated data (health, finance, legal, government) never leaves your infrastructure.
- Predictable cost at high volume: once usage is large and steady, owning the GPUs can beat per-token pricing.
- Control: pin a model version, fine-tune it, and avoid surprise deprecations or rate limits.
- Offline and edge: air-gapped sites, factories, ships and field deployments.
Choosing an open-weight model
Strong open-weight model families include Llama (Meta), Qwen (Alibaba), Mistral, Gemma (Google), DeepSeek and OpenAI's gpt-oss. Pick by:
- Task: chat and RAG, coding, extraction, or multilingual work (for Arabic, test Qwen and Llama variants on your own data).
- Size: small models (roughly 1–14B parameters) are fast and cheap and fine for classification, extraction and RAG. Large models (70B+ or mixture-of-experts) are needed for complex reasoning.
- License: check commercial-use terms. They differ between families.
Serving tools
| Tool | Best for |
|---|---|
| Ollama | The easiest start: laptops, single servers, prototypes and internal tools |
| llama.cpp | CPU and low-VRAM inference, quantised GGUF models, edge devices |
| vLLM | Production GPU serving with high throughput and an OpenAI-compatible API |
| SGLang / TensorRT-LLM | Squeezing maximum performance out of NVIDIA GPUs at scale |
| Open WebUI / LibreChat | A ChatGPT-style interface for staff on top of your models |
Hardware and GPU sizing
A rough rule of thumb for memory: about 2 bytes per parameter at 16-bit and roughly 0.5–0.6 bytes per parameter at 4-bit quantisation, plus headroom for the KV cache (context and concurrent users).
| Model size | 4-bit approx. VRAM | Typical hardware |
|---|---|---|
| 7–8B | ~5–8 GB | Single consumer or workstation GPU, Apple silicon |
| 13–14B | ~9–12 GB | 16–24 GB GPU |
| 30–35B | ~20–24 GB | 24–48 GB GPU |
| 70B | ~40–48 GB+ | 2× 24–48 GB GPUs or one 80 GB data-centre GPU |
For many concurrent users or long contexts, budget significantly more. Benchmark with your real prompts before you buy hardware. Renting cloud GPUs for a week of testing is cheap insurance.
A reference architecture
- Inference server (vLLM or Ollama) exposing an OpenAI-compatible endpoint inside your network.
- Gateway for auth, rate limits, logging and routing between local and API models.
- RAG layer: document ingestion, embeddings (also self-hosted) and a vector database such as pgvector or Qdrant.
- Apps: chat UI, internal tools and agents calling the gateway.
- Evaluation and monitoring: quality tests, latency, GPU usage and cost per request.
Common mistakes
- Buying GPUs before measuring real workloads.
- Choosing the biggest model when a small fine-tuned or well-prompted one would do.
- Skipping evaluation. Open models vary a lot by task and language.
- Forgetting operations: updates, monitoring, failover and security patching are now your job.
Frequently asked questions
Is a self-hosted LLM cheaper than an API?
Only at sustained high volume, or when compliance rules out APIs. For low or spiky usage, API models are usually cheaper once you count hardware, power and engineering time.
Can a self-hosted LLM be as good as ChatGPT or Claude?
For many focused business tasks such as extraction, classification, summarisation and RAG over your documents, well-chosen open models perform very well. For the hardest reasoning and agentic coding tasks, frontier closed models still lead.
What's the easiest way to try a local LLM?
Install Ollama, pull a small open model and connect a chat UI such as Open WebUI. You can be chatting with a local model in under an hour.
Do you help companies deploy on-prem AI?
Yes. I design and deploy self-hosted and hybrid AI stacks, from model selection and GPU sizing to RAG, security and staff rollout.