unsloth-fft

What problem does it solve?

This Skill enables efficient and memory-optimized Full Fine-Tuning (FFT) of large language models, allowing for 100% exact weight updates without the limitations of low-rank adapters.

Core Features & Use Cases

• 100% Exact Weight Updates: Modifies all model weights for comprehensive behavior changes.
• Optimized Gradient Checkpointing: Significantly reduces VRAM usage, enabling larger batch sizes.
• Memory Management: Utilizes adamw_8bit and gradient accumulation for efficient training on limited hardware.
• Use Case: Ideal for base model pre-training or when significant model-wide behavioral shifts are required, and sufficient VRAM is available.

Quick Start

Initialize a full fine-tuning session for the 'unsloth/llama-3-8b' model using the provided setup script.