mamba-architecture

Load pretrained Mamba models with MambaLMHeadModel.from_pretrained using model names like state-spaces/mamba-2.8b, paired with the EleutherAI/gpt-neox-20b tokenizer. Available sizes range from 130M to 2.8B parameters, and generation uses standard temperature and top-p sampling.

Mamba-1 uses d_state=16 with a single-head structure, while Mamba-2 uses d_state=128 with multi-head structure, RMSNorm, and tensor parallelism support. Mamba-2 offers larger state capacity and better distributed training options.

Yes, Mamba requires Linux with an NVIDIA GPU, PyTorch 1.12+, and CUDA 11.6 or higher. The mamba-ssm library relies on custom CUDA kernels for its hardware-aware parallel scan implementation.

Slow installs usually come from building from source; use pip install mamba-ssm --no-build-isolation for binary wheels. If causal-conv1d is missing, install it separately with pip install causal-conv1d>=1.4.0.

Use Mamba for sequences over 100K tokens, memory-constrained inference without KV cache, or streaming applications needing constant per-token memory. Choose Transformers when you need best-in-class task performance and have sufficient compute.

Reduce per-GPU batch size, enable gradient checkpointing on the model, or shorten the sequence length. Mamba memory grows linearly with sequence length, so these adjustments scale predictably.