Karpenter GPU Autoscaling on Kubernetes: Dynamic Node Provisioning for LLM Inference

Kubernetes provides a flexible platform for scheduling ML and inference workloads. Karpenter automates the provisioning and removal of the underlying GPU compute capacity in response to actual demand. Together, they allow organizations to build GPU infrastructure that scales with workloads rather than running continuously regardless of utilization.

This guide covers how Karpenter works with GPU workloads, how it fits into a production LLM inference stack, and the practical configuration details that make the difference between a cost-efficient platform and an expensive one.

What Is Karpenter and Which Cloud Providers Support It?

Karpenter is an open-source Kubernetes node provisioning tool, originally developed by AWS and released under the Apache 2.0 license. It observes unschedulable pods — pods that cannot be placed on existing nodes due to insufficient resources — and automatically provisions compute capacity that satisfies their requirements. When that capacity is no longer needed, Karpenter consolidates workloads and removes idle nodes.

Unlike the older Cluster Autoscaler, which scales predefined node groups up or down, Karpenter selects instance types dynamically based on each workload's actual resource requests. This makes it significantly more flexible for GPU workloads, where the right instance type varies considerably between a small inference service and a large model training job.

Cloud provider support: Karpenter has mature, production-ready support on AWS (EKS). Google Cloud (GKE Autopilot) and Azure (AKS) have their own managed node autoprovisioning implementations that follow similar principles, but the configuration described in this guide uses Karpenter's AWS implementation. If you are running on GCP or Azure, check your provider's node autoprovisioning documentation for equivalent functionality.

Version note: This article reflects Karpenter v1 (stable API), which introduced NodePool and NodeClaim as the primary resources, replacing the earlier Provisioner API from v0.x. If your cluster is running Karpenter v0.x, the resource names and some field structures will differ.

Why GPU Autoscaling Is a Different Problem Than CPU Autoscaling

CPU-based applications are relatively straightforward to scale. Compute is inexpensive, widely available, and startup times are fast. GPU workloads present a different set of constraints.

A single NVIDIA H100 instance costs approximately 10–15 times more per hour than a comparable CPU node. At that price, idle GPU capacity is not just wasteful — it directly erodes the unit economics of every AI service running on the platform. An H100 instance sitting at 20% utilization for 20 hours a day is a significant recurring expense that better infrastructure design can eliminate.

ML workloads also have genuinely variable demand patterns. A production inference service may handle traffic peaks several times the daily average. Batch inference, model evaluation, and fine-tuning jobs require GPUs only for specific windows. Development and experimentation environments are needed intermittently. Without dynamic provisioning, all of these workloads compete for a fixed pool of GPU capacity that must be sized for peak demand — whether or not peak demand is actually occurring.

Karpenter vs. Cluster Autoscaler for GPU Workloads

Both tools scale Kubernetes nodes, but they take fundamentally different approaches.

The Cluster Autoscaler works with predefined node groups (Auto Scaling Groups on AWS). When pods are unschedulable, it scales up the appropriate node group. When nodes are underutilized, it scales them down. The instance type is fixed per node group — if your GPU node group uses p3.2xlarge instances, that is what every GPU node in that group will be.

Karpenter has no concept of node groups. It evaluates unschedulable pods, reads their resource requirements and scheduling constraints, and selects the most appropriate instance type from a broad pool — including GPU instances — at provisioning time. A single Karpenter NodePool can provision different GPU instance types for different workloads based on their actual requirements.

For GPU infrastructure specifically, this matters because the optimal instance for a 7B inference pod is very different from the optimal instance for a 70B training job. Karpenter can serve both from the same configuration; Cluster Autoscaler requires separate node groups for each.

How GPU Node Provisioning Works with Karpenter

When a GPU workload cannot be scheduled on existing nodes, the provisioning sequence looks like this:

1. A pod is submitted with a GPU resource request (nvidia.com/gpu: 1) and scheduling constraints.
2. The Kubernetes scheduler cannot find a suitable node and marks the pod as unschedulable.
3. Karpenter detects the unschedulable pod and evaluates its requirements.
4. Karpenter selects an appropriate GPU instance type and provisions a new node.
5. The NVIDIA device plugin makes the GPU available to the Kubernetes scheduler.
6. The pod is scheduled and starts running.
7. When the pod completes and the node is no longer needed, Karpenter removes it.

For LLM inference workloads, step 4 to step 6 typically takes 3–6 minutes, depending on instance type and region. A significant portion of this time is model loading — pulling a 15–30GB model from object storage onto a freshly provisioned node. This cold-start latency is the primary practical constraint on how aggressively you can scale inference infrastructure down during quiet periods.

The most effective mitigation is pre-caching model weights at the node level using a DaemonSet or node image baking, so the model is available immediately when a pod starts rather than being pulled on demand.

Configuring a GPU NodePool

A Karpenter NodePool defines the constraints within which Karpenter can provision nodes. For GPU workloads, a minimal configuration looks like this:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference
spec:
  template:
    spec:
      requirements:
        - key: karpenter.k8s.aws/instance-gpu-count
          operator: Gt
          values: ["0"]
        - key: karpenter.k8s.aws/instance-gpu-name
          operator: In
          values: ["a100", "h100", "l40s"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1
        kind: EC2NodeClass
        name: gpu-nodeclass
  limits:
    nvidia.com/gpu: 32
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m

The limits field caps total GPU capacity across all nodes provisioned by this pool — an important guardrail against runaway scaling. The disruption block controls when Karpenter is allowed to remove or consolidate nodes; WhenEmptyOrUnderutilized enables both empty-node removal and workload consolidation onto fewer nodes.

A GPU pod spec that triggers Karpenter provisioning looks like this:

resources:
  requests:
    nvidia.com/gpu: 1
    memory: "24Gi"
  limits:
    nvidia.com/gpu: 1
    memory: "24Gi"

Without an explicit GPU resource request, Karpenter has no signal that the pod requires a GPU node and will attempt to schedule it on whatever node type is available.

Pod Autoscaling vs. GPU Node Autoscaling

One of the most common sources of confusion in GPU infrastructure design is conflating two distinct scaling problems: how many inference pods are needed, and how much GPU capacity exists to run them.

These are separate layers with separate tools.

• Pod autoscaling — determines how many replicas of an inference service should be running based on request volume, queue depth, or latency. For LLM inference workloads, KEDA (Kubernetes Event-Driven Autoscaler) is the most appropriate tool here, because it can scale on custom Prometheus metrics such as vllm:num_requests_waiting — the actual inference queue depth — rather than CPU or memory, which are poor proxies for LLM serving load. Standard HPA works for simpler cases but lacks the flexibility for inference-specific metrics without custom adapters.
• Node autoscaling — determines how much GPU compute capacity the cluster has available to run those pods. This is what Karpenter handles.

In practice, the two layers work together: KEDA scales up inference replicas when queue depth rises, new pods become unschedulable if existing nodes are full, and Karpenter provisions additional GPU nodes to accommodate them. When traffic falls, KEDA scales replicas down, nodes become underutilized, and Karpenter consolidates and removes them.

This layered approach — where each layer solves a different problem — is what makes the overall system responsive without over-provisioning. For a broader discussion of when Kubernetes autoscaling fits versus serverless patterns, see our comparison of Kubernetes vs. serverless for cloud workloads.

Karpenter Consolidation and Scale-Down

Provisioning GPU capacity when demand increases is only half of the equation. The other half — removing capacity when it is no longer needed — is where many teams leave money on the table.

Karpenter's consolidation mechanism evaluates whether running workloads can be rescheduled onto fewer nodes, freeing others for removal. For GPU workloads, consolidation behavior is governed by three key settings:

• consolidationPolicy: WhenEmpty — removes nodes only when all pods have completed. Conservative; suitable for stateful or latency-sensitive inference services.
• consolidationPolicy: WhenEmptyOrUnderutilized — also consolidates underutilized nodes by evicting and rescheduling pods. More aggressive; suitable for batch workloads and development environments.
• consolidateAfter — the period Karpenter waits after a node becomes eligible for consolidation before acting. Setting this too low on inference workloads can cause unnecessary pod restarts during brief traffic lulls.

For production inference services, WhenEmpty with a generous consolidateAfter window (10–15 minutes) is usually the safer starting point. For batch inference jobs and experimentation environments, WhenEmptyOrUnderutilized with a tighter window reduces idle costs without affecting service quality.

Consider a batch inference workload that requires four GPUs for 30 minutes, running eight times per day. Without autoscaling, those four GPUs run continuously — roughly 96 GPU-hours per day. With Karpenter provisioning capacity only for each job window, actual consumption drops to 16 GPU-hours per day, a reduction of approximately 83% for that workload alone.

Spot GPU Instances for Further Cost Reduction

For workloads that can tolerate interruptions, Karpenter's flexible instance selection makes it straightforward to incorporate Spot GPU capacity. Spot instances offer discounts of 60–90% compared to on-demand pricing, at the cost of potential interruption when the cloud provider reclaims capacity.

Spot GPUs are well-suited for:

• Batch inference jobs with checkpointing
• Model evaluation and benchmarking
• Fine-tuning runs that can be resumed from a checkpoint
• Development and experimentation environments
• Non-critical or offline inference pipelines

They are less suitable for latency-sensitive production inference, where an interruption would cause request failures and pod restart time — including model reloading — would directly impact user experience.

The most practical approach for most organizations is a mixed capacity strategy: on-demand GPU nodes for production inference services, and Spot nodes for batch and development workloads. Karpenter supports this through separate NodePool definitions with different capacity-type constraints, allowing each workload type to target the appropriate capacity tier.

vLLM and Karpenter: How the Layers Work Together

For organizations running LLM inference, vLLM and Karpenter operate at different levels of the stack and solve different problems — understanding the distinction prevents misconfiguration and misaligned optimization effort.

vLLM operates inside the GPU. It uses continuous batching and PagedAttention to maximize the number of tokens each GPU produces per second, reducing the cost per generated token on whatever hardware is running. For a detailed walkthrough of vLLM's production deployment architecture, see our guide to vLLM production deployment: GPU optimization and token cost reduction.

Karpenter operates at the infrastructure level. It determines how many GPU nodes exist, provisions them when pods need them, and removes them when they don't.

The full stack, from request to hardware, looks like this:

• Incoming requests → vLLM inference engine (continuous batching, PagedAttention)
• Replica count → KEDA (scales pods based on queue depth)
• Node capacity → Karpenter (provisions and removes GPU nodes)
• Physical compute → GPU cloud infrastructure

Each layer is independently tunable. A well-optimized vLLM configuration on a poorly scaled infrastructure will still overpay for idle GPUs. Equally, perfect Karpenter configuration cannot compensate for an inference engine that uses GPU memory inefficiently. Cost reduction at scale requires both.

One practical interaction worth highlighting: because vLLM loads the full model into GPU memory on startup, cold-starting a new inference pod on a freshly provisioned Karpenter node takes meaningfully longer than cold-starting a typical stateless application. For a 30B model, total cold-start time — node provisioning plus model loading — can reach 5–8 minutes. Design your KEDA scaling thresholds and Karpenter consolidation windows with this latency in mind to avoid oscillation between scaling up and immediately scaling back down.

Choosing the Right GPU for Each Workload Type

Autoscaling does not automatically make an inefficient GPU selection cost-effective. The right instance type still needs to match the workload's actual requirements.

A useful starting framework:

• Production inference, 7B–13B models: NVIDIA L40S or A100 40GB. Cost-effective for moderate concurrency with context windows up to 16K.
• Production inference, 30B–70B models: A100 80GB or H100 80GB. Required for larger KV cache headroom at production concurrency.
• Batch inference and evaluation: T4 or L4 for smaller models; cost-optimized for throughput over latency.
• Training and fine-tuning: H100 with NVLink for large models; A100 for moderate-scale runs.
• Development and experimentation: Smallest instance that fits the model; Spot capacity where available.

The fastest GPU is rarely the most economical choice. Benchmark real workloads — including concurrency, context length, and batch characteristics — before committing to a GPU configuration. For a broader view of GPU use cases across different ML workload types, see our overview of strategic GPU use cases from graphics to AI.

For teams evaluating GPU infrastructure options, Cloud4U's GPU servers for AI and machine learning are available with hourly billing, making it practical to benchmark different instance types before settling on a production configuration.

Monitoring GPU Autoscaling in Production

GPU utilization alone is not a sufficient signal for whether an autoscaling platform is working correctly. High utilization can indicate efficient use of capacity — or it can indicate that the cluster is saturated and requests are queuing.

A complete monitoring setup for GPU autoscaling should cover three layers:

Inference engine metrics (from vLLM's /metrics endpoint, via Prometheus):

• vllm:gpu_cache_usage_perc — KV cache utilization; above 95% indicates memory pressure
• vllm:num_requests_waiting — inference queue depth; the primary signal for KEDA scaling
• vllm:e2e_request_latency_seconds — end-to-end latency at P50, P95, P99
• vllm:time_to_first_token_seconds — critical for interactive applications

GPU hardware metrics (via DCGM Exporter or nvidia-smi):

• GPU utilization percentage
• GPU memory utilization
• GPU temperature and power draw

Karpenter infrastructure metrics (via Karpenter's Prometheus endpoint):

• karpenter_nodes_total — total nodes managed by Karpenter
• karpenter_provisioner_scheduling_duration_seconds — time from unschedulable pod to node provisioning decision
• karpenter_nodes_termination_duration_seconds — time to remove a node after consolidation decision

The cost metric that ties all of this together is cost per million output tokens — total GPU infrastructure spend divided by total tokens generated. This is the number that reflects whether autoscaling, inference optimization, and GPU selection are working together effectively. For a broader discussion of GPU cost measurement and FinOps practices for AI workloads, see our guide to GPU cost optimization strategies for AI/ML workloads.

Common Mistakes in GPU Autoscaling Deployments

Using HPA instead of KEDA for inference pod scaling. HPA scales on CPU and memory, which do not reflect inference load accurately. A vLLM pod can sit at 10% CPU while its GPU is saturated and its request queue is growing. KEDA with a custom Prometheus metric on vllm:num_requests_waiting scales on the signal that actually matters.

Ignoring cold-start latency in consolidation windows. Setting consolidateAfter: 1m on an inference service that takes 5 minutes to restart will cause continuous oscillation — Karpenter removes nodes during a brief traffic lull, new pods become unschedulable when traffic returns, and provisioning starts over. Size consolidation windows to be longer than cold-start time.

No GPU resource requests in pod specs. Without explicit nvidia.com/gpu resource requests, Karpenter cannot determine that a pod needs a GPU node. The pod will be scheduled on whatever node is available, and GPU workloads may end up on CPU instances.

No GPU limits on the NodePool. Without a limits cap on the NodePool, a misconfigured workload or a traffic spike can trigger unbounded GPU provisioning. Always set a GPU limit on production NodePools.

Applying the same consolidation policy to all workloads. Batch jobs can be consolidated aggressively. Production inference services cannot. Use separate NodePools with different disruption settings for each workload type.

Building a More Efficient GPU Platform

Karpenter and Kubernetes provide the infrastructure layer for dynamic GPU provisioning, but cost efficiency at scale is a multi-layer outcome. The organizations that reduce GPU spend most significantly are those that optimize across all layers simultaneously: selecting the right GPU instance type, running an efficient inference engine, scaling pods on meaningful metrics, and dynamically provisioning infrastructure to match actual demand.

None of these optimizations is sufficient alone. An efficient inference engine on a permanently over-provisioned cluster still overpays. Perfect autoscaling cannot compensate for a GPU instance type that is 3× larger than the workload requires.

The practical path forward is iterative: instrument first, optimize the layer with the largest cost impact, then move to the next. For most teams, the largest gains come from dynamic provisioning (eliminating idle GPU capacity) and inference engine optimization (more tokens per GPU-hour) — in that order.