Running Local LLM Infrastructure
What changes when inference moves off a hosted API and onto GPUs you control — model serving, memory budgeting and the operational tradeoffs.
BY ASHLIN DARIUS GOVINDASAMY
Calling a hosted LLM API is a single HTTP request. Running the model yourself means owning everything that request was hiding: GPU memory budgeting, batching, quantisation tradeoffs, and the process that keeps a model server alive under load.
The stack
Memory is the constraint that decides everything
A model's parameter count sets a hard floor on VRAM before you've served a single token — and that floor moves with precision. A 7B-parameter model at FP16 needs roughly 14GB just for weights, before KV-cache and activation memory for concurrent requests are added on top. Quantisation (INT8, INT4) trades some quality for headroom; batching trades latency for throughput.
# Serve a local model with an OpenAI-compatible API
python -m vllm.entrypoints.openai.api_server \
--model /models/adg-local-7b \
--gpu-memory-utilization 0.90 \
--max-model-len 8192What you gain, and what you now own
- Data never leaves infrastructure you control — the tradeoff for that is you now own uptime, scaling and patching.
- Latency becomes a function of your hardware and batching strategy, not a third party's rate limits.
- Cost shifts from per-token billing to fixed GPU capacity — good at high, steady utilisation; wasteful at low utilisation.
None of this is a reason to avoid hosted APIs by default — it's the set of tradeoffs that matters once an application's data, latency or cost requirements make local inference the right engineering answer.
TAGS
RELATED