long-context

Set rope_scaling in the model config with type "linear" or "yarn" and a factor of 16.0, then fine-tune for 400-1000 steps on long documents. YaRN achieves better quality with only 400 steps and 10x less training data than position interpolation.

YaRN uses NTK-aware interpolation that treats high and low frequencies differently plus attention temperature scaling, while position interpolation uniformly compresses all positions. YaRN needs 2.5x fewer steps and extrapolates better to longer contexts.

ALiBi is best for training new models from scratch since it offers unlimited context, 11% less memory, and strong extrapolation. RoPE with YaRN scaling is better for extending existing pre-trained models like LLaMA that already use rotary embeddings.

No, applying rope_scaling without fine-tuning produces poor results. Position interpolation requires roughly 1000 fine-tuning steps on long documents matching the target context length to restore model quality.

Attention memory grows quadratically with sequence length. Enable gradient checkpointing, use Flash Attention 2 via attn_implementation="flash_attention_2", reduce batch size to 1, and use bfloat16 precision to fit 32k sequences in memory.

Aggressive scaling factors like 16x require more fine-tuning data and steps, and position interpolation extrapolates poorly beyond the trained range. Incremental scaling (8k to 16k to 32k) with fine-tuning at each stage gives more stable results.