inference

What problem does it solve?

This skill enables fast and memory-efficient inference for large language models by leveraging a vLLM backend and unsloth, reducing latency and resource usage in both interactive and batch scenarios.

Core Features & Use Cases

• Fast_inference with vLLM backend for 2x speedups
• Model loading and merging LoRA adapters for efficient deployment
• Thinking-model output parsing and memory management for robust workflows
• Batch and interactive inference in Python environments (notebooks & apps)

Quick Start

Run a sample inference by loading a pre-quantized thinking model and enabling fast_inference to observe accelerated generation.