skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

knowledge-distillation

Compress large language models into smaller student models using teacher-student distillation training.

13.0k|930|Updated Nov 3, 2025
Search Tags:#llm#knowledge distillation#temperature scaling#model compression#teacher-student#reverse kld#minillm
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill knowledge-distillation-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: knowledge-distillation
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/19-emerging-techniques/knowledge-distillation
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill knowledge-distillation-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, datasets, and includes references (resource) components.

What problem does it solve? Deploying large language models like LLaMA-70B or GPT-4 is expensive and slow, but smaller models lack their capabilities. This Skill transfers knowledge from large teacher models to compact student models, retaining most performance while cutting inference costs. ## Core Features & Use Cases - Logit Distillation: Train student models to match teacher logits using KL divergence with temperature scaling and combined soft/hard loss. - MiniLLM Reverse KLD: Use reverse KL divergence for mode-covering distillation that produces more diverse generative outputs. - Response Distillation: Generate synthetic training data from teacher models and fine-tune students to imitate them. - Use Case: Compress a LLaMA-2-70B teacher into a LLaMA-2-7B student that retains 90%+ of benchmark performance for cheaper production deployment. ## Quick Start Distill the LLaMA-2-70B teacher model into a LLaMA-2-7B student using temperature 2.0 and alpha 0.7 with the provided training script.

Frequently Asked Questions about knowledge-distillation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I distill a large language model into a smaller model?▼

Load a teacher model (e.g., LLaMA-2-70B) and a student model (e.g., LLaMA-2-7B), then train the student with a combined loss of KL divergence against teacher soft targets and cross-entropy against true labels. Use temperature 2.0 and alpha 0.7 as starting hyperparameters.

What is the difference between forward KL and reverse KL in knowledge distillation?▼

Forward KL (KL(Student || Teacher)) is mode-seeking, making the student match the teacher's average behavior, which suits classification. Reverse KL (KL(Teacher || Student)), used by MiniLLM, is mode-covering, producing more diverse generations for open-ended text tasks.

What temperature and alpha values should I use for distillation?▼

Use temperature 2.0 as a standard default, with a range of 2-5 for softer distributions. Set alpha to 0.5 for balanced training, 0.7 to emphasize teacher knowledge, or 0.9 for strong distillation.

Does knowledge distillation work with a very large teacher-student size gap?▼

Distillation works well with ratios around 10x, such as 70B to 7B or 13B to 1B. Gaps like 70B to 1B (70x) are too large and lead to ineffective knowledge transfer.

When should I use MiniLLM instead of standard distillation?▼

Use MiniLLM's reverse KL divergence for generative tasks requiring diverse, open-ended outputs, where it outperforms forward KL by roughly 10% on benchmarks like MT-Bench. Standard forward KL is better for classification or single-answer tasks.

Related Skills

Explore More Education & Research Skills→
CHENyiru3CHENyiru3

datasets

Load and preprocess datasets from Hugging Face Hub or local sources.

Community
Advanced
debug-zhuweijiandebug-zhuweijian
13

knowledge-distillation

Compress a large teacher model into a smaller student via knowledge distillation.

Community
Advanced
MedTiLabMedTiLab
5

model-merging

Merge fine-tuned language models with mergekit using SLERP, TIES, DARE, and task arithmetic.

Community
Advanced
KappTech88KappTech88

model-pruning

Reduce LLM size and accelerate inference via Wanda, SparseGPT, N:M pruning methods.

Community
Advanced