How to Deploy an LLM: Pipeline, Options and Cost

Once you have picked a model, the next question is how to get it serving real requests without it falling over. LLM deployment covers everything from a single API call to a hosted service, to running your own inference server on a GPU node you manage. This article maps the four realistic paths, then works through the self-hosted route step by step, because that is where most of the actual decisions happen.

Key Takeaways

• Four deployment paths exist: hosted API, managed endpoint, self-hosted container, and on-device. Only one requires you to own a server.
• Quantization format and weight storage location are the first two decisions in the self-hosted pipeline, and both affect every step that follows.
• Inference server choice, gateway setup, health check configuration, and Kubernetes node topology each require decisions that differ from standard web service deployments.
• GPU utilization rate, not model size, determines whether self-hosting costs less than a hosted API over time.
• Common questions about health checks, Kubernetes GPU setup, and OpenAI-compatible API differences answered below.

The LLM Deployment Pipeline and How to Choose Your Path

The llm pipeline begins with a decision that most articles skip: you do not have to deploy anything at all. There are four realistic paths, and only one involves owning a server.

The first path is a hosted API. You send requests to a provider's endpoint, pay per token, and do no infrastructure work. Idle time costs nothing. The tradeoff is no control over the model version, the hardware it runs on, or where your data goes.

The second path is a managed inference endpoint. A cloud provider hosts the model on dedicated hardware that you configure. You choose the instance type and the model weights. The provider handles the serving stack. This sits between fully managed and fully self-hosted.

The third path is a self-hosted container. You own the GPU node, the inference server, and the networking. This is what the rest of this article covers, because this is where "how to deploy an LLM" applies in practice.

The fourth path is on-device deployment. Quantized models run on laptops or edge hardware with no network dependency. This works for offline tools and privacy-sensitive applications, but it sits outside the scope of production serving at scale.

What Quantization Format Should You Choose?

Choose 4-bit GGUF for a single 24 GB consumer GPU, and fp16 or 8-bit for data-center GPUs with larger memory.

Quantization reduces the number of bits used to store each model weight. A 27-billion-parameter model in fp16 needs roughly 54 GB of VRAM. The same model at 4-bit GGUF fits in about 16.5 GB, putting it within reach of a single consumer card. Quality loss at 4-bit is acceptable for most open-weight model tasks, though results vary by model family.

The format you choose locks in your hardware requirement. Pick the quantization first, confirm you have the VRAM to load it with room left for the KV cache, then move on.

Step two is weight packaging. Model weights are gigabytes in size and change on a different schedule than your application code. Never bake them into a container image. Store them in object storage or a persistent volume and pull them at container start. This keeps your image small and your weight updates independent of your code deployments.

LLM Deployment Services and How to Run an Inference Server

Picking your llm deployment services means picking your inference server first. The server determines your hardware requirements, the API shape your application uses, and how many requests you can handle at once.

For multi-user throughput on data-center GPUs, vLLM and Text Generation Inference are the standard options. Both use continuous batching, so the GPU works on tokens from multiple requests at the same time rather than finishing one before starting the next. For quantized models on consumer GPUs, llama.cpp runs an HTTP server that handles the same job at smaller scale. For local development, Ollama wraps llama.cpp with automatic model management. All four expose an OpenAI-compatible API, so your application code does not change when you swap servers.

Put an API gateway in front of the inference server. The gateway handles authentication, rate limits, and per-key token metering. Your application targets one base URL regardless of what runs behind it, which means you can move to a different inference server or host without touching your application code.

How Do You Deploy an LLM on Kubernetes?

To deploy an LLM on Kubernetes, you need a GPU node pool, the NVIDIA device plugin on those nodes, and a hard limit of one GPU per pod.

Node selectors and taints keep model pods on GPU nodes so general workloads do not consume that capacity. Weights live on a persistent volume or get pulled from object storage at pod start.

GPU pod autoscaling does not work the same way as for stateless web pods. A second pod means a second full model copy on a second GPU. A queue in front of the inference server is the practical answer. It absorbs traffic spikes and feeds requests at a rate the GPU can handle, without spawning pods that need minutes to become ready.

Health checks need care here. A model server process starts in seconds, but loading the weights into VRAM takes minutes. A readiness probe that checks only whether the process is running will pass too early, and the load balancer will send requests to a pod that cannot serve them. The probe needs to call an endpoint the inference server marks healthy only after the model is fully loaded.

Nova's own setup shows both paths. The platform runs Go microservices on Google Kubernetes Engine behind Traefik. Product AI features call hosted Claude and OpenAI models through in-cluster services. No GPU nodes run in production. On the development workstation, a 27-billion-parameter open-weight model runs locally for jobs that need to be cheap, offline, or private.

That workstation uses llama.cpp built from source, installed as a systemd user service with linger enabled so it survives logout and reboot. The binaries could not find their shared libraries under systemd's clean environment, so an RPATH set at build time points them at their own lib directory. The server binds to loopback only. An idle-stop timer shuts it down after 15 minutes without a request, freeing VRAM. A single environment file holds the port number, and every helper script reads from it. On that hardware: about 50 to 60 tokens per second, speculative decoding at draft depth 3, and a 96k context window using about 21 GB of VRAM.

LLM Deployment Cost, Observability, and Production Pitfalls

LLM deployment cost goes beyond the GPU bill. Observability takes engineering time to build. Cold starts add latency that affects users. Reasoning models can quietly multiply your compute spend if you do not configure them carefully.

Start with the four numbers that tell you what is actually happening: time to first token, tokens per second, GPU memory utilization, and queue depth. Time to first token is how long the user waits before seeing any output. Tokens per second is the throughput your hardware delivers. Queue depth tells you whether requests are piling up faster than the GPU can process them. Without these, a slowdown looks the same whether it comes from the model, the hardware, or the traffic volume.

Rollout on GPU infrastructure needs a different approach than rolling updates. A GPU node holds one model copy. Draining a pod and starting a replacement on the same node creates a service gap. The right pattern is blue-green: load the new version on a second node, verify it is healthy, switch the load balancer, then drain the first node. Before any promotion, run an evaluation gate. Output quality regressions are harder to catch than latency regressions and tend to cause more user-facing damage.

The cost model comes down to one question: is your GPU busy enough to justify the hourly rate? Hosted APIs charge per token, so idle time costs nothing. A dedicated cloud GPU charges by the hour whether it is serving requests or not. Self-hosting pays off only when utilization is high enough that the per-hour rate beats the per-token rate at your actual volume. A used 24 GB consumer card costs once and covers models up to about 30 billion parameters at 4-bit quantization. For variable or low-volume traffic, a hosted API will usually cost less once you count the idle hours.

Four pitfalls show up repeatedly. First, cold starts: an idle-stop timer frees VRAM but means the first request after a pause waits for the model to reload, which is noticeably longer than a warm request. Plan for that delay in your application, or keep the server warm during expected usage windows. Second, context length: the KV cache grows with every token, so a 96k context window does not mean you can run many 96k contexts at once. Each one takes a proportional slice of VRAM. Third, reasoning models: some emit thousands of internal tokens before producing a visible answer. Most inference servers let you disable extended reasoning per request. Do that on any latency-sensitive path. Fourth, OpenAI-compatible does not mean identical. The chat completions endpoint is consistent across servers, but tool-calling formats and streaming event structures vary. Test against your actual inference server, not the OpenAI reference.

For teams building on managed infrastructure, Nova's managed infrastructure services and custom software development services cover the platform layer. The hosting pricing page and Kubernetes WordPress hosting pages cover what the platform supports more broadly.

The Bottom Line

The choice between a hosted API, a managed endpoint, and a self-hosted container is not a question of which is best in general. It is a question of whether your utilization justifies the operational work. At low or variable volume, a hosted API wins on cost and simplicity. At sustained high volume with privacy requirements, self-hosting on the right hardware wins on cost per token. The eight-step pipeline in this article covers the decisions that sit between those two points.

If you want to go further into the infrastructure patterns behind any of these steps, a fractional CTO engagement is a good place to continue.

FAQs

How do you deploy an LLM in production?

Choose an inference server, store model weights in object storage or a persistent volume rather than your container image, put an API gateway in front for authentication and rate limiting, and configure a readiness probe that waits for the model to finish loading before the pod accepts traffic.

Can I deploy an LLM on Kubernetes?

Yes, but it requires specific setup. You need a GPU node pool, the NVIDIA device plugin, a one-GPU-per-pod resource limit, node selectors and taints to isolate GPU nodes, and a queue in front of the inference server. Standard horizontal autoscaling does not apply because each replica needs a dedicated GPU.

What is the cheapest way to deploy an LLM?

For private workloads with models up to about 30 billion parameters, a used 24 GB consumer GPU is the cheapest option after the upfront cost. For variable or low-volume traffic, a pay-per-token hosted API costs less because you pay nothing when the model is idle.

What is an OpenAI-compatible API?

It is an API that uses the same request and response format as the OpenAI chat completions endpoint. Most open-source inference servers implement this, so your application code can switch between them without changes. Tool-calling formats and streaming event structures can still differ between servers, so test those specifically.

What should health checks look like for an LLM service?

The readiness probe should call an endpoint that only returns healthy after the model weights are fully loaded into VRAM. A probe that checks only whether the server process is running will pass too early. Model loading takes minutes, and early traffic will hit a server that cannot respond yet.