modal-serverless-gpu

Decorate a function with @app.function(gpu="T4") inside a modal.App, then execute it with modal run script.py. Modal provisions the GPU container on demand and returns the result from the .remote() call.

Define a class with @app.cls(gpu=..., image=...), load the model in an @modal.enter() hook, and expose methods with @modal.method() or attach @modal.fastapi_endpoint(). Run modal deploy script.py for a persistent HTTPS endpoint.

For inference, L40S (48GB) offers the best cost-performance, while T4 or A10G suit smaller models. For training large models, use A100-80GB or H100, and specify multiple GPUs like gpu="H100:4" for distributed workloads.

Modal fits serverless, scale-to-zero workloads defined in Python. RunPod suits longer-running pods with persistent state, Lambda Labs offers reserved instances, and SkyPilot handles multi-cloud orchestration and cost optimization.

Cold starts include container launch and model loading. Mitigate by increasing container_idle_timeout, setting keep_warm=1, loading models in @modal.enter() hooks, and caching weights in a persistent Volume.

Volume writes are not persisted until you call volume.commit() after writing files. Conversely, call volume.reload() in reader functions to fetch the latest committed state from other containers.