data-collection

Collects experiment evidence from minimal runs including W&B sync, remote logs, and Prometheus metrics.

Updated Jun 6, 2026
One-click install
npx skills add https://github.com/Kirrito-k423/AutoResearch --skill data-collection-kirrito-k423
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-collection
Source: https://github.com/Kirrito-k423/AutoResearch/tree/main/.agents/skills/07-data-collection
Command: npx skills add https://github.com/Kirrito-k423/AutoResearch --skill data-collection-kirrito-k423

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Running LLM training experiments produces scattered evidence across W&B offline runs, remote server logs, and Prometheus metrics, making it hard to reproduce results or build auditable experiment records. This Skill consolidates all of that into immutable local-first run bundles with manifests and config locks. ## Core Features & Use Cases - Minimal Experiment Execution: Runs trainer.train() for 1-2 steps via the minimal runner to generate smoke-test evidence quickly. - W&B Offline Sync: Syncs offline W&B runs locally with semantic project/run naming (e.g., verl project, run names encoding model, scale, algorithm, and sequence length), and saves runs.json plus rebuild-wandb.sh so the data bundle can rebuild the same local W&B view. - Remote Log & Prometheus Collection: Pulls remote training logs back to local storage and pushes NPU metrics to a local Pushgateway, explicitly recording which metrics (e.g., HBM/Core utilization) are missing. - Use Case: After running a formal Verl GRPO case on a remote NPU server, use this Skill to produce a self-contained evidence bundle with logs/, wandb/, prom/, and manifest.json that supports bundle-local fallback loading without relying on old absolute paths. ## Quick Start Run the data collection script with your config file, target server, and a run ID to execute a minimal experiment and gather all evidence into a local bundle.

Frequently Asked Questions about data-collection

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I collect experiment evidence from a minimal training run?▼

Run the run_minimal.py script with your config file, target server, and a run ID. It executes trainer.train() for 1-2 steps, syncs W&B offline runs, pulls remote logs, and pushes Prometheus metrics into a local evidence bundle.

How do I sync W&B offline runs to a local instance?▼

Use wandb sync on the offline run directories saved in the data bundle. The bundle includes runs.json and rebuild-wandb.sh so copying the data repository lets you rebuild the same local W&B web view.

What naming convention should W&B projects and runs use?▼

Use the code stack name as the W&B project, such as verl. Run display names should encode model, scale, algorithm, sequence length, timestamp, and key config, for example Qwen35-2B-GRPO-1Kto16K-260622d-145001s-valonly-sync-noignoreeos.

Why are Prometheus resource utilization charts empty in my report?▼

Empty charts happen when only autoresearch_npu_count is pushed without HBM or Core utilization metrics. The evidence must explicitly record which metrics were pushed and which are missing rather than implying resource curves exist.

Can the evidence bundle work without the original absolute paths?▼

Yes. Reports and recovery scripts support bundle-local fallback, loading from the bundle's own logs/, wandb/, prom/, and rows/ directories instead of relying on old absolute paths recorded in manifest.json.

What are the limitations of the current data collection?▼

Resource utilization collection is not yet complete; HBM and Core metrics require integrating an NPU exporter or npu-smi sampling. It also does not handle report rendering or pre-run stack health checks, which belong to other skills.