durable-hpc-gate-ecoda

Orchestrates durable Slurm gate lifecycles for ECODA pipelines on the Yggdrasil HPC cluster.

1|Updated Mar 7, 2025
One-click install
npx skills add https://github.com/carmonalab/ECODA_paper --skill durable-hpc-gate-ecoda-carmonalab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: durable-hpc-gate-ecoda
Source: https://github.com/carmonalab/ECODA_paper/tree/main/.agents/skills/durable-hpc-gate-ecoda
Command: npx skills add https://github.com/carmonalab/ECODA_paper --skill durable-hpc-gate-ecoda-carmonalab

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Long-running Slurm benchmark jobs for the ECODA scRNA-seq pipeline can be lost or left in ambiguous states when SSH sessions drop, and ad-hoc retries risk duplicate launches, corrupted artifacts, or unverified results. This Skill enforces a durable, fail-closed gate lifecycle so every ECODA compute run on Yggdrasil is prepared, launched, monitored, audited, and reviewed exactly once with full provenance. ## Core Features & Use Cases - Durable gate lifecycle: Drives the global durable-hpc-gate CLI through reconcile, prepare, launch, wait, inspect, and status subcommands with atomic manifests and event-driven waiting instead of polling. - Repository and host invariants: Validates the Yggdrasil checkout, canonical scratch/NAS roots, slurm_config.sh settings, immutable source snapshots, and run-bound runtime identities before any launch. - Serialization and lineage control: Uses explicit serialization groups as cross-manifest mutexes and reviewed dependency manifests so benchmark waves run safely in parallel or sequence. - Terminal audit and review: Performs a single strict sacct accounting query, run-scoped ecoda_run_audit.sh validation, checksum verification, and a mandatory Luna Max reviewer approval before release. - Use Case: Launch a full-cohort ECODA benchmark wave on Yggdrasil, disconnect your laptop, and later resume to a verified, reviewer-approved COMPLETED gate with audited scheduler IDs and artifact checksums. ## Quick Start Ask the agent to prepare and launch a durable ECODA gate on Yggdrasil for a specific pipeline stage using the checked-in profile and an explicit gate ID, then wait and inspect it to completion.

Frequently Asked Questions about durable-hpc-gate-ecoda

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a durable Slurm job that survives SSH disconnection?▼

Use the durable-hpc-gate CLI to prepare a manifest, launch the exact wrapper inside a persistent tmux session on the remote host, and arm one asynchronous wait on a completion event. Durable status files, not the SSH session, are the authority on job state.

How do I launch an ECODA benchmark gate on Yggdrasil?▼

Commit and push changes locally, pull the exact revision on Yggdrasil, then run reconcile, prepare, and launch with the checked-in profile.json, an explicit gate ID, and the immutable snapshot wrapper. A bounded task then runs wait and the terminal inspect with scheduler IDs.

What is the serialization_group in durable HPC gates?▼

serialization_group is an explicit cross-manifest mutex identity keyed by project, profile, and profile digest. Launch takes a deterministic lock and moves a gate to PRELAUNCH_STOP with a resource_lock_conflict if a RUNNING sibling shares the identity; it is independent of dependency lineage.

Can I edit the Yggdrasil checkout directly to fix a pipeline script?▼

No. All authoring happens on the local workstation: commit and push changes, then pull the exact committed revision on Yggdrasil. Gates execute wrappers from commit-keyed immutable source snapshots, never the mutable checkout.

What happens if a gate launch is ambiguous after a connection drop?▼

An ambiguous launch becomes PRELAUNCH_STOP: all evidence is preserved and you must reconcile before any retry. Never assume the wrapper did not start or relaunch blindly, since that risks duplicate scheduler submissions.

When is an ECODA gate result eligible for release?▼

Only after terminal durable status, one strict sacct accounting query showing all scheduler IDs COMPLETED with exit code 0:0, run-scoped ecoda_run_audit.sh validation, checksum verification, and explicit Luna Max reviewer approval via the second inspect invocation.