local-llm-orchestration

What problem does it solve?

Provides a controlled orchestration layer for running local LLMs (vLLM and Qwen3.5-9B) as a Tier 2 fallback when cloud LLM quotas are exhausted or when you need to verify VRAM capacity and fallback behavior without disrupting production.

Core Features & Use Cases

• VRAM budgeting and verification: Guidance on estimating model VRAM requirements and checking available GPU memory before loading models.
• Startup and logging checks: Instructions to monitor model load logs and safely shut down or revert to cloud tier when model loading exceeds capacity.
• Context and task assignment rules: Mitigations for the 4096 token context limit, recommendations to cap max_tokens, strip internal thinker tags, and restrict long-form generation; practical use cases include short-form research, evaluation, and scoring.

Quick Start

Run a GPU memory check, then start vLLM as a short-form research fallback and verify logs for successful model load.