segment-anything-model

Load a SAM checkpoint with sam_model_registry, create a SamPredictor, call set_image on your RGB image, then call predict with point coordinates and labels. Set multimask_output to True to get three mask candidates and select the one with the highest score.

ViT-H is the largest at 2.4GB with the best accuracy but slowest speed, ViT-L is a medium 1.2GB option, and ViT-B is the smallest at 375MB with the fastest inference. Use ViT-B when GPU memory or latency is constrained.

SAM alone does not accept text prompts; it only takes points, boxes, or masks. Combine GroundingDINO with SAM to convert text descriptions into bounding boxes, then feed those boxes to SAM to produce masks.

The original SAM processes individual images only and does not track objects across frames. Use SAM 2, which adds a streaming memory architecture for video segmentation and object tracking across frames.

The ViT-H checkpoint requires substantial GPU memory, especially with large images. Switch to the ViT-B model, resize images to a maximum dimension, call torch.cuda.empty_cache() between images, or use half precision.

Increase points_per_side for denser prompt grids, enable crop_n_layers for multi-scale processing, and lower min_mask_region_area. Alternatively, process the image in overlapping patches and merge the resulting masks.