run-experiment

Running ML training jobs across local hardware and remote SSH servers requires coordinating environment detection, code synchronization, GPU pre-flight validation, and job launching via screen for background execution with log capture. This workflow automates that deployment process end-to-end.

Deploying GPU experiments to Vast.ai and stopping idle billing involves launching training jobs with auto_destroy enabled, which automatically downloads results and logs before destroying the instance upon completion to prevent ongoing charges.

Running ML experiments on Modal serverless without managing infrastructure is supported through direct delegation via the /serverless-modal entry point, handling environment selection and job execution automatically.

GPU availability validation before launching remote training jobs works by checking nvidia-smi or MPS availability to ensure GPU resources are present, preventing wasted cycles on environments lacking hardware.

Weights & Biases logging for ML experiments is supported optionally, ensuring scripts automatically log metrics to W&B when enabled in the environment configuration for tracking experiment runs.

Synchronizing code and dependencies before running remote ML experiments requires detecting the target environment from configuration files, then coordinating code sync, dependency setup, and GPU checks prior to job execution.