skills.rest
alon3153
alon3153COMMUNITY
Explore all skills by this author →
GitHub

gke-inference

Deploys and optimizes AI/ML inference workloads on GKE using GPUs, TPUs, and model servers.

Updated May 11, 2026
Search Tags:#kubernetes#gpu#gcloud#autoscaling#gke#vllm#llm inference
One-click install
npx skills add https://github.com/alon3153/upe-social-publisher --skill gke-inference-alon3153
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gke-inference
Source: https://github.com/alon3153/upe-social-publisher/tree/main/.agents/skills/gke-inference
Command: npx skills add https://github.com/alon3153/upe-social-publisher --skill gke-inference-alon3153

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Deploying large language models and AI inference workloads on Kubernetes requires choosing the right model server, GPU or TPU accelerator, and autoscaling configuration, which is error-prone when done manually. This Skill guides you through Google's Inference Quickstart to generate validated manifests and deploy them to GKE. ## Core Features & Use Cases - Model and Accelerator Discovery: List supported models and find valid GPU/TPU and model-server combinations with gcloud container ai profiles commands. - Manifest Generation: Create optimized Kubernetes manifests for servers like vLLM, TGI, Triton, and TensorRT-LLM with latency targets. - Autoscaling and Optimization: Configure HPA with GPU metrics, ComputeClasses for Autopilot, quantization, and KV cache tuning. - Use Case: Deploy Gemma 2 9B on an L4 GPU with vLLM by generating a manifest, applying it with kubectl, and monitoring the rollout. ## Quick Start Ask the AI to deploy the gemma-2-9b-it model on GKE using vLLM with an NVIDIA L4 accelerator and generate the inference manifest.

Frequently Asked Questions about gke-inference

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy an LLM on GKE with GPUs?▼

Use gcloud container ai profiles to find a valid model, server, and accelerator combination, then generate a manifest with `gcloud container ai profiles manifests create` and apply it with kubectl. Monitor the rollout with kubectl get pods and logs.

Which GPU should I choose for model inference on GKE?▼

NVIDIA L4 suits small-to-medium inference at low cost, A100 handles large models, and H100 targets frontier models with high throughput. Cloud TPUs like v5e and v6e offer cost-effective transformer inference alternatives.

Does GKE Autopilot support GPU inference workloads?▼

Yes, Autopilot supports GPU workloads through ComputeClasses and node auto-provisioning. Define a ComputeClass specifying the machine family, GPU type, and minimum resources to target GPU nodes.

Why does my inference pod fail with GPU OOM errors?▼

OOM occurs when the model exceeds accelerator memory. Fix it by selecting a larger GPU, applying quantization such as GPTQ or AWQ, or enabling tensor parallelism to split the model across multiple GPUs.

How do I autoscale LLM inference on GKE?▼

Configure a HorizontalPodAutoscaler using GPU utilization metrics like gpu_duty_cycle from DCGM monitoring. Set minReplicas to at least 1 for always-on serving and use longer scale-down stabilization windows because model loading is slow.

When should I not use this GKE inference approach?▼

Avoid it for generic batch jobs or HPC task queues, which are better handled by a batch-oriented skill. It is designed specifically for model serving and inference workloads, not offline compute pipelines.

Related Skills

Explore More Software Engineering Skills→
ssmleossmleo

gke-inference

Automate AI/ML inference deployment and optimization on GKE with GPU and TPU configurations.

Community
Advanced
wwwillottwwwillott

serving-llms-vllm

Serve scalable LLM inference with vLLM for OpenAI-compatible APIs.

Community
Advanced
coreweavecoreweave
7

setup-sunk-gpu-monitoring

Set up GPU and Slurm metrics collection on a SUNK cluster.

Official
Advanced
googlegoogle
17.1k

gke-batch-hpc

Automate batch and HPC workload deployment on Google Kubernetes Engine.

Official
Advanced