skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

llama-factory

Fine-tune large language models with LLaMA-Factory using LoRA, QLoRA, and WebUI workflows.

13.0k|930|Updated Nov 3, 2025
Search Tags:#llm#fine-tuning#multimodal#lora#webui#qlora#llama-factory
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill llama-factory-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-factory
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/03-fine-tuning/llama-factory
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill llama-factory-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llmtuner, torch, transformers, datasets, peft, accelerate, gradio, and includes references (resource) components.

What problem does it solve? Fine-tuning large language models typically requires writing complex training code, configuring distributed setups, and managing quantization manually. This Skill provides expert guidance for using LLaMA-Factory to fine-tune 100+ models through a no-code WebUI or YAML-based CLI, covering LoRA, QLoRA, DPO, PPO, and multimodal training. ## Core Features & Use Cases - No-Code WebUI Fine-Tuning: Launch the Gradio-based WebUI to train, evaluate, chat with, and export models without writing code. - Broad Method & Model Support: Covers full fine-tuning, Freeze, LoRA, QLoRA (2-8 bit), GaLore, BAdam, DoRA, PiSSA, plus RLHF methods like PPO, DPO, and KTO across LLaMA, Qwen, Gemma, Mistral, and multimodal models. - Distributed & Accelerated Training: Guidance for DeepSpeed ZeRO, FSDP, FlashAttention-2, Unsloth, and Ascend NPU training and inference. - Use Case: You want to fine-tune Llama-3-8B on a custom Alpaca-format dataset using 4-bit QLoRA on a single GPU, then merge the adapter and export a quantized model for deployment. ## Quick Start Ask the AI to show you how to fine-tune a Llama model on a custom dataset using LLaMA-Factory with QLoRA and then merge the adapter weights.

Frequently Asked Questions about llama-factory

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fine-tune a Llama model with LLaMA-Factory?▼

Install LLaMA-Factory with pip, prepare a dataset in Alpaca or ShareGPT format registered in dataset_info.json, then run llamafactory-cli train with a YAML config specifying stage sft, finetuning_type lora, and your model path. Alternatively, launch the WebUI for zero-code training.

How to use QLoRA 4-bit quantization in LLaMA-Factory?▼

Set quantization_bit to 4 with a quantization method such as bitsandbytes, hqq, or eetq in your training config, and use finetuning_type lora. LLaMA-Factory supports 2/3/4/5/6/8-bit QLoRA via AQLM, AWQ, GPTQ, HQQ, and EETQ.

Does LLaMA-Factory support multimodal model fine-tuning?▼

Yes, LLaMA-Factory supports multimodal fine-tuning for models like LLaVA and LLaMA-4 using image, video, and audio datasets. Datasets must include images, videos, or audio columns with paths matching the number of placeholder tokens in the text.

Can LLaMA-Factory run on Ascend NPU instead of GPU?▼

Yes, LLaMA-Factory supports Huawei Ascend 910b NPUs through torch-npu, installable via Docker or pip. Replace CUDA_VISIBLE_DEVICES with ASCEND_RT_VISIBLE_DEVICES to select NPU cards for training and inference.

How do I merge LoRA adapters after fine-tuning?▼

Run llamafactory-cli export with a merge config specifying model_name_or_path, adapter_name_or_path, template, and export directory. Do not use a quantized base model or set quantization bits when merging; quantize only after merging.

Why should I not combine LoRA with GaLore or BAdam?▼

LLaMA-Factory documentation explicitly warns against using LoRA together with GaLore or BAdam, as these are memory-efficient full-parameter optimization methods that conflict with adapter-based training. BAdam requires finetuning_type full with pure_bf16 enabled.

Related Skills

Explore More Software Engineering Skills→
ehadziabdicehadziabdic

llm-data-preparation

Generate, curate, deduplicate, and format training datasets for LLM fine-tuning and alignment.

Community
Advanced
MLT-OSSMLT-OSS
14

evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.

Official
Advanced
nishide-devnishide-dev

ml-experiment

Track ML experiment results and store resolved Hydra configurations.

Community
Intermediate
BagelHoleBagelHole
46

llm-fine-tuning

Set up QLoRA, LoRA, and full fine-tuning infrastructure with Hugging Face TRL and Axolotl.

Community
Advanced