rust-gpu-scheduling

What problem does it solve?

VRAM-aware GPU scheduling for ML model orchestration, balancing memory budgets with workload demands to prevent OOM and reduce idle time.

Core Features & Use Cases

• VRAM-aware scheduling with a safety margin to minimize OOM risks across multiple models (Whisper, VideoMAE, CLAP, Qwen3, DINOv3).
• Work item queues with priority handling and dependency tracking to enforce correct execution order.
• Model lifecycle management: track loaded/unloaded state and estimated load times to reduce costly reloads.
• VRAM budgeting, per-model accounting, and simple semaphore-based limiting to cap concurrent workloads.
• Guidelines and extensible architecture to integrate into ML orchestration pipelines.

Quick Start

Instantiate the GPUScheduler with your VRAM configuration and submit your first WorkItem to begin scheduling.