Train and fine-tune GPT-2 models with a minimalist PyTorch implementation.
npx skills add https://github.com/Clay-HHK/claude-skills --skill nanogpt-clay-hhk
Please help me install this Agent Skill. Skill: nanogpt Source: https://github.com/Clay-HHK/claude-skills/tree/main/nanogpt Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill nanogpt-clay-hhk
High-intent search queries and answers about installing and using this skill.
Run python data/shakespeare_char/prepare.py to create train.bin and val.bin, then run python train.py config/train_shakespeare_char.py. Training takes about 5 minutes on CPU or 1 minute on GPU, and you can generate text with python sample.py.
Prepare the data with python data/openwebtext/prepare.py, which takes 1-2 hours and produces about 9 billion tokens. Then launch multi-GPU training with torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py, taking roughly 4 days on 8 A100 GPUs.
nanoGPT is best for learning and experimentation since the entire model and training loop fit in ~300 lines each with no abstractions. Use HuggingFace Transformers for production with many model types, LitGPT for more architectures, or Megatron-LM for large-scale distributed training.
Yes, set init_from to gpt2, gpt2-medium, gpt2-large, or gpt2-xl in your config to load OpenAI weights automatically via the transformers library. Use a lower learning rate like 3e-5 and fewer iterations for fine-tuning on custom datasets.
CUDA OOM happens when batch size or context length exceeds GPU memory. Reduce batch_size to 1, lower block_size from 1024 to 512, and increase gradient_accumulation_steps to maintain the effective batch size.
Yes, character-level Shakespeare training runs on CPU in about 5 minutes with the default small config of 6 layers and 384 embedding dimensions. Set device to cpu and compile to False in the config for CPU training.
Supporter09Train BPE or Unigram tokenizers on raw Unicode text.
RachasumanthDeploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.
NousResearchTrack ML experiments, run hyperparameter sweeps, and version models with Weights & Biases.
KappTech88Orchestrate PPO/GRPO RL training for large language models with verl.
