How to Deploy Production LLM Inference with vLLM in the Cloud: Architecture, GPU Optimization, and Lower Token Costs

Running a large language model in a demo environment is straightforward. Running one in production — reliably, at scale, without watching infrastructure costs climb with every thousand requests — is a different challenge entirely.

This guide covers how to deploy vLLM in a cloud environment, what a production-ready architecture looks like, and the key techniques for maximizing GPU utilization while keeping the cost per generated token under control.

What Is vLLM and Why Does It Matter for Production AI?

vLLM is an open-source inference engine designed for high-throughput serving of large language models. It was introduced by researchers at UC Berkeley and has become one of the most widely adopted solutions for organizations running open-source LLMs — including Llama, Mistral, Qwen, Gemma, and DeepSeek — in production.

The core problem vLLM addresses is GPU efficiency. Traditional serving frameworks process requests in fixed batches: the GPU waits until a batch is full before starting work, and then sits idle while the next batch is assembled. Under variable traffic, this means significant waste — expensive hardware doing nothing while users wait.

vLLM solves this through two mechanisms:

• Continuous batching — new requests are inserted into the processing pipeline dynamically as GPU capacity becomes available, rather than waiting for a fixed batch to fill up.
• PagedAttention — a memory management technique inspired by operating system virtual memory. Instead of reserving large, contiguous memory blocks for each request's KV cache, PagedAttention allocates memory in smaller, non-contiguous pages. This reduces memory waste by up to 4× compared to naive allocation and allows significantly more concurrent requests to run on the same hardware.

Together, these two features translate directly into better throughput, lower latency, and reduced infrastructure cost per generated token — which is the metric that actually matters in production.

A Production-Ready Architecture for vLLM

Getting vLLM running is easy. Getting it to handle real traffic reliably — including traffic spikes, node failures, and rolling updates — requires a structured architecture.

A typical production deployment includes the following layers.

Traffic entry point: An API Gateway handles authentication, rate limiting, and request routing. This keeps application traffic separate from inference workloads and simplifies security enforcement.

Load distribution: A load balancer distributes incoming requests across inference instances. For session-based or streaming workloads, connection persistence may need to be configured.

Orchestration layer: Kubernetes manages the deployment, scheduling, health checking, and autoscaling of vLLM pods. GPU-enabled nodes are provisioned as a separate node pool from CPU workloads to simplify scheduling and cost tracking. If you are still evaluating whether Kubernetes is the right fit for your infrastructure, our comparison of Kubernetes vs. serverless for AI workloads covers the key decision criteria.

Inference pods: Each vLLM pod runs the inference engine with the model loaded into GPU memory. Pod configuration — including memory limits, replica counts, and startup probes — should be tuned per model size.

Model storage: Model weights are stored in object storage (S3-compatible) and pulled to GPU nodes on startup. Caching at the node level reduces cold-start times for frequently used models.

Observability stack: Prometheus collects metrics from vLLM's built-in endpoint; Grafana provides dashboards. This is not optional — without visibility into GPU utilization, queue depth, and token throughput, optimization is guesswork.

A minimal deployment for a single model can be started with:

docker run --gpus all \
  -v /model-cache:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model mistralai/Mistral-7B-Instruct-v0.2 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

For Kubernetes, a Helm-based deployment with GPU node affinity, horizontal pod autoscaler configuration, and a Prometheus ServiceMonitor is the standard pattern for production environments.

Choosing the Right GPU Configuration

GPU selection is often treated as a memory problem — "does the model fit?" — but that framing misses the more important question: what does the model cost to serve at your expected concurrency?

Several factors determine the right configuration:

• Model size sets the baseline memory requirement for weights.
• Context window length determines how much KV cache memory is needed per concurrent request. A 32K context window requires significantly more memory than an 8K window, even for the same model.
• Expected concurrency determines how many requests need to run simultaneously.
• Latency requirements — specifically Time to First Token (TTFT) — can rule out certain configurations regardless of throughput.

As a starting reference:

Important: these are starting points for benchmarking, not production sizing decisions. Always benchmark with representative traffic before provisioning.

Cloud4U's NVIDIA GPU servers for AI and machine learning are available with hourly billing, making it straightforward to benchmark different configurations before committing to a production setup.

Tensor Parallelism for Large Models

For models that exceed the memory of a single GPU, vLLM supports tensor parallelism — splitting model layers across multiple GPUs so that each handles a portion of the computation. This is configured with the --tensor-parallel-size flag:

--tensor-parallel-size 2   # splits model across 2 GPUs
--tensor-parallel-size 4   # splits model across 4 GPUs

Tensor parallelism requires fast interconnects between GPUs (NVLink or high-bandwidth PCIe). On cloud GPU instances, verify the interconnect topology before assuming linear scaling.

PagedAttention and KV Cache Management

PagedAttention is the memory innovation that makes vLLM meaningfully different from earlier serving frameworks, and it deserves more than a passing mention.

During autoregressive inference — the process by which an LLM generates text one token at a time — each forward pass needs access to the key-value tensors from all previous tokens in the context. These tensors, collectively called the KV cache, grow with every generated token and must be held in GPU memory for the duration of the request.

Traditional frameworks allocate a large, contiguous memory block for each request at start time, sized for the maximum possible output length. This has two problems. First, most requests don't reach the maximum length, so allocated memory sits unused. Second, large contiguous allocations fragment GPU memory, reducing the number of requests that can run in parallel.

PagedAttention treats KV cache memory the way an operating system treats physical RAM: it divides memory into fixed-size pages and maps each request's cache to available pages wherever they exist in memory. Pages are freed as soon as a request completes.

The practical result is that vLLM can run more concurrent requests on the same GPU, translating directly into higher throughput and lower cost per token.

Quantization: The Fastest Path to Lower GPU Costs

One of the most effective — and underused — techniques for reducing inference costs is model quantization. By reducing the numerical precision of model weights, quantization shrinks memory requirements and often increases throughput with minimal impact on output quality.

vLLM supports several quantization formats natively:

For a 7B model, switching from FP16 to AWQ reduces memory from approximately 14GB to 7GB — allowing it to run on a smaller GPU, or freeing memory for more concurrent requests on the same GPU.

To load a quantized model in vLLM:

--model TheBloke/Mistral-7B-Instruct-v0.2-AWQ \
--quantization awq

For production deployments, FP8 is increasingly the preferred option on NVIDIA H100 hardware, where dedicated FP8 tensor cores deliver throughput improvements alongside the memory reduction.

Speculative Decoding for Latency-Sensitive Workloads

For applications where Time to First Token (TTFT) and per-token latency matter more than raw throughput — interactive assistants, real-time code suggestions, customer-facing chat — speculative decoding is worth evaluating.

The technique uses a small, fast "draft" model to speculatively generate several tokens at once, which a larger "target" model then verifies in a single forward pass. When the draft tokens are correct, multiple tokens are accepted per step, reducing the number of forward passes required.

vLLM supports speculative decoding via the --speculative-model flag. The throughput gains depend heavily on the domain and output characteristics; benchmark against your specific workload before committing to the added infrastructure complexity.

Improving GPU Utilization in Practice

Many production AI environments operate at 30–50% GPU utilization even under meaningful load. Common causes include:

• Request batches that are too small
• CPU bottlenecks in pre/post-processing that starve the GPU
• Slow model loading from cold storage
• Autoscaling policies that add replicas too slowly
• Single large instances instead of multiple smaller ones

The most reliable path to higher utilization:

1. Enable continuous batching — this is the default in vLLM but verify it is not being overridden by configuration.
2. Set --gpu-memory-utilization to 0.90 or higher — vLLM uses this fraction of available GPU memory for the KV cache. Lowering it unnecessarily reduces concurrency.
3. Deploy multiple replicas behind a load balancer — a single instance with a queue creates head-of-line blocking; replicas spread load and reduce P95/P99 latency.
4. Configure autoscaling on queue depth — scaling on the vllm:num_requests_waiting Prometheus metric responds faster to actual inference load than CPU or memory metrics. KEDA supports custom Prometheus metrics for this purpose. If you do not already operate a Kubernetes environment, Cloud4U's managed Kubernetes as a Service handles cluster provisioning and GPU node scheduling so your team can focus on inference configuration rather than infrastructure management.
5. Pre-load models to node local storage — pulling a 15GB model from object storage on pod startup adds 1–3 minutes to scale-out time. Daemonset-based model caching at the node level eliminates this.

Token Economics: Measuring What Actually Matters

Infrastructure teams often evaluate GPU costs by the hour. Finance teams often evaluate AI workloads by the month. Neither metric tells you whether your inference stack is efficient.

The number that connects them is cost per million output tokens — the total infrastructure spend divided by the total tokens generated. Reducing this figure is the goal of every optimization. For a broader framework covering resource tagging, spot instances, and FinOps practices across the full AI stack, see our guide to GPU cost optimization strategies for AI/ML workloads.

As a rough reference, a well-configured vLLM deployment on an NVIDIA L40S (approximately €2–3/hour on cloud GPU infrastructure) serving a quantized 7B model at moderate concurrency can deliver 1,000–3,000 tokens per second, depending on context length and batch characteristics. At that throughput, the cost per million tokens falls below €1–2 — competitive with managed API providers for high-volume workloads, with the added benefit of data privacy and no per-token pricing surprises.

The levers that move this number most:

• Quantization — same hardware, more tokens per second, lower cost per token
• Continuous batching — better GPU utilization under variable load
• Autoscaling — avoid paying for idle capacity during off-peak hours
• Right-sizing the model — a well-tuned 8B model often matches a 70B model on domain-specific tasks at a fraction of the infrastructure cost

Monitoring a Production vLLM Deployment

vLLM exposes a Prometheus-compatible metrics endpoint at /metrics. At a minimum, monitor:

• vllm:gpu_cache_usage_perc — KV cache utilization; sustained values above 95% indicate memory pressure
• vllm:num_requests_waiting — queue depth; rising values indicate the deployment cannot keep up with incoming load
• vllm:e2e_request_latency_seconds — end-to-end request latency (track P50, P95, P99)
• vllm:time_to_first_token_seconds — critical for interactive applications
• vllm:request_success_total and vllm:request_failure_total — error rate
• GPU utilization and GPU memory usage via nvidia-smi or DCGM exporter

A GPU utilization consistently below 60% during peak hours indicates untapped capacity. A queue depth that regularly exceeds zero during business hours indicates the deployment needs more capacity.

Common Deployment Mistakes

Most first production deployments encounter the same set of problems. The most consequential:

Sizing for model fit, not for concurrency. A model fitting in GPU memory is necessary but not sufficient. Without sufficient KV cache headroom for concurrent requests, throughput collapses under real traffic.

No load testing before go-live. A deployment that handles 10 concurrent users may queue at 50. Run load tests with realistic traffic shapes — including bursts — before any production launch.

Single inference instance without autoscaling. A single pod creates a single point of failure and a head-of-line blocking problem. Multiple replicas behind a load balancer are a minimum production requirement.

Ignoring the model layer. Organizations spend effort optimizing infrastructure while running an oversized FP16 model that a quantized alternative would serve equally well at half the GPU cost.

Monitoring gaps. Without visibility into KV cache utilization and queue depth, performance degradation goes unnoticed until users report it.

Conclusion: Building an Efficient AI Inference Platform

vLLM provides the technical foundation — continuous batching, PagedAttention, quantization support, and an OpenAI-compatible API — to serve open-source LLMs efficiently in production. The infrastructure layer — Kubernetes orchestration, GPU-enabled cloud nodes, autoscaling, and observability — determines whether that potential translates into reliable performance and predictable costs.

vLLM is also the inference layer that makes production Retrieval-Augmented Generation (RAG) systems practical at scale — if you are building an enterprise knowledge assistant or document search platform, that guide covers the full pipeline architecture.

Organizations that approach LLM inference as a platform engineering problem — designing for concurrency, measuring token economics, and tuning iteratively — will scale their AI services without facing proportional infrastructure cost increases.

If you are evaluating GPU cloud infrastructure for vLLM or other AI/ML workloads, Cloud4U offers NVIDIA-powered GPU servers for machine learning with hourly billing and a free trial period.