kubeflow-trainer

Configure Kubeflow Trainer v2.2 TrainJobs and TrainingRuntimes for distributed ML training on Kubernetes.

4|2|Updated Jun 1, 2026
One-click install
npx skills add https://github.com/Aidas-dev/k8s-agent-skills --skill kubeflow-trainer-aidas-dev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: kubeflow-trainer
Source: https://github.com/Aidas-dev/k8s-agent-skills/tree/main/skills/kubeflow-trainer
Command: npx skills add https://github.com/Aidas-dev/k8s-agent-skills --skill kubeflow-trainer-aidas-dev

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Running distributed machine learning training on Kubernetes requires coordinating framework-specific CRDs, gang scheduling, GPU allocation, and job orchestration, which is error-prone when done manually. This Skill provides the operational knowledge to define and manage Kubeflow Trainer v2.2 workloads using the unified TrainJob and TrainingRuntime CRDs. ## Core Features & Use Cases - Framework-agnostic training jobs: Define TrainJobs for PyTorch, TensorFlow, JAX, XGBoost, MPI, and Flux using a single CRD with TrainingRuntime execution templates. - Scheduling integration: Configure gang scheduling via coscheduling or Volcano, and queue-based admission with Kueue multikueue for shared GPU clusters. - Ecosystem integration: Orchestrate TrainJobs inside Kubeflow Pipelines v2 components and use them as trial templates for Katib hyperparameter tuning. - Use Case: A platform engineer needs to run a 4-node PyTorch training job with 2 GPUs per node on a shared cluster. The Skill provides the TrainJob manifest, a TrainingRuntime with torch mlPolicy and coscheduling, plus guidance on Kueue queue labels and common pitfalls like GPU count mismatches. ## Quick Start Ask the agent to create a Kubeflow Trainer TrainJob and TrainingRuntime for a multi-node PyTorch training workload with GPU limits and gang scheduling.

Frequently Asked Questions about kubeflow-trainer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run distributed PyTorch training on Kubernetes with Kubeflow Trainer?▼

Create a TrainJob referencing a TrainingRuntime whose mlPolicy.torch sets numGPUsPerNode, and set trainer.numNodes with GPU resource limits. The runtime configures MASTER_ADDR, WORLD_SIZE, and RANK environment variables automatically through the underlying JobSet orchestration.

What is the difference between TrainJob and TrainingRuntime in Kubeflow Trainer v2?▼

TrainJob defines the training intent including image, command, node count, and resources. TrainingRuntime defines the execution template with the ML framework policy, gang scheduling policy, and JobSet spec that determines how the job actually runs.

Does Kubeflow Trainer v2 support gang scheduling for GPU jobs?▼

Yes, TrainingRuntime supports podGroupPolicy with either coscheduling or Volcano for gang scheduling. Kueue integration via managedBy: kueue.x-k8s.io/multikueue also provides admission control, preemption, and topology-aware scheduling.

How do I migrate from PyTorchJob or TFJob to Kubeflow Trainer v2?▼

Replace each legacy CRD with a TrainJob plus a TrainingRuntime using the matching mlPolicy: torch for PyTorchJob, mpi for TFJob and MPIJob, and xgboost for XGBoostJob. Update the API group from training-operator.kubeflow.org to trainer.kubeflow.org/v1alpha1.

Why does my Kubeflow TrainJob fail after creation when I change runtimeRef?▼

The runtimeRef field is immutable after TrainJob creation, along with initializer, trainer.numNodes, and trainer.image. You must delete and recreate the TrainJob to change these fields.

Can I use Kubeflow Trainer with Katib hyperparameter tuning?▼

Yes, TrainJobs can serve as trial templates in Katib Experiments via the trialTemplate.trainJobSpec field. Hyperparameters are injected through environment variables using Katib's template substitution syntax.