LLM Infrastructure for a Small Team: What You Actually Need

Most teams discover the same thing once they move past a proof of concept: getting a model to respond is the easy part. The harder part is building the system around it — the layer that meters usage, retries failed requests, keeps API keys off laptops, and queues the long jobs that would time out over a plain HTTP call. That surrounding system is what LLM infrastructure actually means in practice, and it matters whether you are calling a hosted API or running a GPU in your own environment. This article covers both the inference decision and the five operational pieces every team needs regardless of which path they choose.

Key Takeaways

• The inference decision — hosted API, managed endpoint, or self-hosted GPU — determines your ops burden, not your application architecture.
• A quantized 27B model fits on a 24 GB consumer GPU, but VRAM behavior depends on the model's attention architecture, not just parameter count.
• A single OpenAI-compatible base URL, token metering, request logging, a job queue, and server-side secrets are the five things every production stack needs.
• Most small teams should start with a hosted API and layer in self-hosting only when a specific requirement — privacy, volume, or offline access — makes it necessary.
• Common questions about GPU requirements, VRAM sizing, inference servers, and the minimum production stack are answered below.

What LLM Infrastructure Actually Includes

The term LLM infrastructure stack gets used loosely, so it helps to be specific. The model itself is one component. The infrastructure is everything else: how requests get routed, how usage gets tracked, how long jobs get queued, and how credentials stay out of the wrong hands.

There are three ways to get inference. A hosted API from Anthropic, OpenAI, or Google requires no infrastructure on your end. You send a request, you get a response, you pay per token. A managed model endpoint on AWS Bedrock or Google Vertex AI sits in the middle: the cloud provider runs the model, you control the version and region, and you pay the provider's per-token or provisioned rate. Self-hosting means running an open-weight model on your own GPU, either on a cloud VM or a physical machine, with an inference server you configure and maintain.

Hosted API, Managed Endpoint, or Self-Hosted?

The choice comes down to three questions. Do you have data that cannot leave your environment? Is your request volume high enough that per-token costs exceed fixed compute costs? Do you have someone on the team who can own infrastructure? If the answer to all three is no, a hosted API is the right starting point.

Self-hosting pays off when privacy or data-residency requirements block external API calls, when volume is high and predictable enough to justify fixed GPU costs, or when you need the model to work without internet access. It does not pay off when traffic is bursty, when you do not have dedicated infrastructure capacity, or when the quality gap between an open-weight model and a frontier model would affect your product.

Nova runs hosted Claude and OpenAI models for the product features on its managed WordPress hosting platform, where that quality gap matters. Customers' own AI agents manage their sites through the Nova MCP server. The local 27B model handles bulk, offline, and privacy-sensitive jobs: log triage, mass summarization, and first-pass code review. That split is not a permanent architecture choice. It reflects which tasks actually need frontier-model output and which ones do not. Many teams end up in the same place once they start routing by task type rather than picking a single provider for everything.

Self-Hosted LLM: The Inference Stack in Practice

Choosing the self hosted LLM path means making several decisions before your application code touches the model. You need an inference server, a quantization format, a clear picture of your VRAM budget, and a plan for how the server behaves when it is not actively processing requests.

Do You Actually Need a GPU?

llama.cpp runs on CPU, so no GPU is required. In practice, CPU throughput is too low for anything beyond single-user testing. A consumer GPU with enough VRAM is the realistic floor for inference infrastructure that can handle real workloads.

Two inference servers cover most use cases. vLLM is built for data-center GPUs and multi-user workloads. It uses continuous batching, which means it can process multiple requests at once without queuing them sequentially. llama.cpp is built for quantized models on a single consumer GPU or CPU. Both expose an OpenAI-compatible /v1 API, so switching between them does not require changes to your application code.

Quantization determines whether a model fits in your VRAM at all. Each parameter in an fp16 model takes about 2 bytes, which puts a 27B model at roughly 54 GB. At 4-bit quantization using the Q4_K_M GGUF format, that drops to about 16.5 GB. A 4-bit model is roughly a quarter the size of the fp16 version, which is what makes a 24 GB GPU viable.

Nova runs Qwen3.8-27B through llama.cpp on a single RTX 3090 as a systemd user service. Sustained throughput is about 50 to 60 tokens per second. Without speculative decoding the same setup runs at 42.7 tokens per second; a draft depth of 3 lifts that by about 39 percent, which is where the 50 to 60 figure comes from. A draft depth of 5 was slower. A 4-bit KV cache produced nearly identical throughput to f16 (43.3 versus 43.7 tokens per second) and is only worth enabling if VRAM is the constraint.

Context length adds VRAM pressure, but how much depends on the model's attention architecture, not just its size. Published VRAM tables for Qwen3.8-27B were wrong because only 16 of its 64 blocks use full attention. The other 48 use a fixed-size recurrent state. The KV cache grows only about 1.9 GB per additional 64k tokens as a result. A 96k-token context fits in about 21 GB, 128k fits in about 22 GB, and 262k does not fit in 24 GB. The only reliable approach is to measure on your hardware. Published tables reflect different architectures and different configurations.

Two setup details keep the workstation usable. The server binds to loopback only, so it has no external exposure. An idle-stop timer shuts it down 15 minutes after the last request, so a single query does not hold 21 GB of VRAM for hours. One model behavior worth knowing: thinking mode is on by default in Qwen3.8-27B and generates roughly 1,800 characters of internal reasoning before each answer. Turn it off per request when response latency matters.

The OpenAI-Compatible API and Everything Else Your Stack Needs

The OpenAI compatible API is the right interface to build against from day one, regardless of where your inference runs. Every serious inference server exposes a /v1/chat/completions endpoint in the OpenAI format. If your application targets a single base URL and reads the model name from config, you can move from a hosted API to a self-hosted model or swap providers without touching application code. Teams that wire provider-specific clients directly into their codebase make that swap into a refactoring project.

Token metering is the second requirement. Track tokens per user or account, not just per request. Set a daily spend breaker that cuts off requests before runaway usage becomes a billing problem. Nova meters AI tokens per account across all AI services and runs a platform-wide daily breaker. The implementation can be simple: a counter updated per request, checked before dispatch.

Request logging is the third requirement. Log latency, time to first token, and token counts for every request. Those three numbers tell you when a model is slowing down, which prompts are expensive, and how costs are trending. If requests contain user data, log the metadata and handle content separately. Skipping this step early means debugging production issues without any signal.

Long pipelines need a job queue. Nova's content pipeline runs about eight model calls per article and takes 13 to 14 minutes end to end. That work runs on a Redis-backed queue with retries and timeouts. A synchronous HTTP connection cannot hold that reliably. A queue handles retries, surfaces failures, and gives you visibility into where a job is without building that logic into your application.

Server-side secrets are the fifth requirement. API keys and model credentials must never reach a browser or a developer's laptop. Proxy all model calls from inside your cluster. A leaked key means someone else runs inference against your account. Store credentials in a secrets manager or runtime environment variable, and limit who can read them.

These five requirements hold regardless of your inference choice. The inference layer changes. The plumbing around it does not.

The Bottom Line

Getting inference working is a one-afternoon problem. Building a system you can operate, debug, and hand off to another engineer takes longer, and it starts with the same five pieces no matter which inference path you choose. Pick an interface your application code can target without caring what is behind it. Count tokens before they become a surprise on an invoice. Log enough to know when something breaks. Queue the long jobs. Keep the keys on the server. If you want to see how managed infrastructure handles the operational layer so your team can focus on the application, Nova's managed infrastructure services is the place to start.

FAQs

What is LLM infrastructure?

LLM infrastructure is the set of systems that make a language model usable in production. It includes the inference layer where the model runs and the surrounding stack: request routing, token metering, logging, job queues, and secret management. The model itself is one part of a larger operational system.

Do I need a GPU to run an LLM?

You do not need a GPU. llama.cpp runs quantized models on CPU. Throughput on CPU is low enough to be impractical for multi-user workloads, but it works for single-user testing or offline tasks where speed is not a constraint.

How much VRAM does a 27B model need?

A 27B model in fp16 needs about 54 GB of VRAM. At 4-bit quantization in Q4_K_M GGUF format, that drops to about 16.5 GB. A 24 GB consumer GPU can run a quantized 27B model with room for context, but actual VRAM use depends on the model's attention architecture. Measure on your hardware.

Should a small team self-host an LLM or use an API?

Start with a hosted API. Self-hosting makes sense for data-residency requirements, high and predictable request volume, or offline use. For bursty workloads or products where frontier-model quality matters, a hosted API is simpler and the economics are usually better at low volume.

What is an inference server?

An inference server loads a model into memory and serves requests over an API. vLLM handles multi-user workloads on data-center GPUs using continuous batching. llama.cpp handles quantized models on consumer GPUs or CPUs. Both expose an OpenAI-compatible /v1 endpoint so your application code does not need to change when you switch between them.