unsloth-gguf

What problem does it solve?

This Skill simplifies the process of exporting fine-tuned machine learning models into the GGUF format, making them compatible with popular local inference tools like llama.cpp and Ollama.

Core Features & Use Cases

• GGUF Export: Converts trained models to the efficient GGUF format.
• Quantization: Supports various quantization methods (e.g., q4_k_m, q8_0) to reduce model size and VRAM usage.
• LoRA Merging: Automatically merges LoRA adapters into the base model during export.
• Use Case: Deploy a fine-tuned LLM on your local machine for faster inference or to run it on hardware with limited VRAM.

Quick Start

Export the current model to GGUF format using the 'q4_k_m' quantization method.