segment-anything-model

Load a SAM checkpoint with sam_model_registry, create a SamPredictor, call set_image on your RGB image, then call predict with point coordinates and labels. The predictor returns masks with quality scores, and you select the highest-scoring mask.

ViT-H is the largest at 2.4GB with the best accuracy but slowest speed, ViT-L is a medium 1.2GB option, and ViT-B is the smallest at 375MB and fastest. Use ViT-B when GPU memory or inference speed is limited.

SAM alone does not accept text prompts; it only takes points, boxes, or masks. Combine GroundingDINO, which converts text descriptions into bounding boxes, with SAM to generate masks from those boxes for text-prompted segmentation.

SAM can run on CPU by loading the model without calling .to("cuda"), though inference will be significantly slower. For limited GPU memory, use the ViT-B checkpoint, half precision, or export to ONNX for lighter deployment.

CUDA out-of-memory errors occur because ViT-H requires substantial VRAM, especially with large images. Switch to the ViT-B model, resize images to a maximum dimension, call torch.cuda.empty_cache() between images, or use half precision.

Use SAM 2 for video segmentation tasks, since it adds a streaming memory architecture that tracks objects across frames. The original SAM processes individual images only and has no temporal tracking capability.