model-deployment

What problem does it solve?

Export and deploy fine-tuned models to production.

Core Features & Use Cases

• GGUF export: export fine-tuned models for local or edge inference with llama.cpp/Ollama.
• Production deployment: set up and run high-throughput serving with vLLM, Ollama, or Docker-based deployments.
• Hub sharing & versioning: publish and version models on HuggingFace Hub for collaboration and reuse.
• Use Case: A medical domain team finishes fine-tuning a model and deploys it to a vLLM server behind a load balancer for 24/7 API access.

Quick Start

1. Export the fine-tuned model to GGUF, e.g., model.save_pretrained_gguf('./gguf_output', tokenizer, quantization_method='q4_k_m').
2. Deploy with Ollama or vLLM: for Ollama, ollama create my-model -f Modelfile; for vLLM, start the server with the appropriate model path.
3. Optionally push the model to HuggingFace Hub for sharing and versioning.