inference-orchestrator

What problem does it solve?

It solves the problem of unreliable, fragmented local ML processing by providing a unified orchestration layer for OCR, STT, embeddings, and extraction.

Core Features & Use Cases

• Manage processing jobs with a structured model, including job_id, patient_id, visit_id, job_type, inputs, outputs, status, and retry policies.
• Enforce resource limits, timeouts, and concurrency controls to prevent overuse of local hardware.
• Provide deterministic caching keyed by input hashes and preprocessing config to ensure reproducible results.
• Maintain a model_registry.json with versioned models and sha256 verification to enforce model integrity.
• Offer progress updates to UI and deterministic retry strategies across job types.

Quick Start

Use the inference-orchestrator to initialize and run a batch of tasks (OCR, STT, embeddings) on the folder 'input_docs/' with caching and model verification.