What problem does it solve? When a multi-thousand-case calibration ensemble is running on Perlmutter, a crashed process looks identical to a healthy one if you only watch for progress events. This Skill ensures every long-running HPC process (auto-monitor, submitter, extractor) is detected and covered by a persistent monitor with both progress and error filters, so failures are caught instead of silently missed. ## Core Features & Use Cases - Process Detection: Uses ps -ef to find live auto-monitor, submitter, and extractor processes, cross-checked against the latest handoff dev log. - Error-Aware Monitoring: Arms persistent Monitor tasks with filters combining progress signals (TRANS_DONE, MILESTONE) and failure signatures (ERROR, Traceback, FAILED, OOM, MaxJobsExceeded). - Event-Driven Reactions: Provides a reaction table mapping events to concrete proposals, such as queue headroom math with model-dependent job multipliers and next-phase recommendations. - Zombie Job Cleanup: Detects and surgically cancels dead SLURM dependency chains that would otherwise hang completion watchers forever. - Use Case: You resume a session while a 1,140-case ELM-FATES round is in flight. The Skill detects the running auto-monitor and submitter, arms monitors on their logs in tmp/, and proposes the next batch when the queue drops below threshold. ## Quick Start Detect any in-flight A2MC ensemble processes on Perlmutter and arm persistent monitors on their logs with both progress and error filters.
