experiment

Run iterative optimization loops that keep only metric improvements and revert regressions.

Updated Feb 28, 2026
One-click install
npx skills add https://github.com/cosmicdreams/claude-plugins --skill experiment-cosmicdreams
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment
Source: https://github.com/cosmicdreams/claude-plugins/tree/main/research-lab/skills/experiment
Command: npx skills add https://github.com/cosmicdreams/claude-plugins --skill experiment-cosmicdreams

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill prevents guesswork by running an iterative change-measure-validate loop that keeps only improvements and automatically reverts regressions.

Core Features & Use Cases

  • Ratchet-based optimization loop: proposes changes, measures a single target metric, and keeps/discards using a ratchet pattern with strict tie handling.
  • Futility stopping and resume support: stops after consecutive unproductive iterations and can resume from existing JSONL results.
  • Git-safe experimental workflow: enforces a dedicated worktree/branch discipline, commits changes, and reverts on discard to maintain a clean lab notebook trail.
  • JSONL experiment logging: records iteration outcomes (including skips, correctness gate results, and reasons) in a structured results.jsonl.

Quick Start

Tell your PI: use research-lab:experiment with a methodology file, a dedicated worktree path (not main), and the measurement harness to iteratively improve your chosen metric and log all results to results.jsonl.

Frequently Asked Questions about experiment

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate iterative optimization to improve a measurable metric?▼

You can run an iterative optimization loop that proposes changes, measures a single target metric, and uses a ratchet pattern to keep improvements while reverting regressions. This requires a methodology file defining your numeric target, direction, and measurement protocol.

What is a ratchet-based optimization loop and how does it handle regressions?▼

A ratchet-based optimization loop commits changes, measures outcomes against a target metric, and automatically reverts regressions. It enforces strict tie handling and git commit discipline, ensuring only validated improvements are kept while maintaining a clean lab notebook trail.

How do I log experiment iterations and results to JSONL?▼

You log experiment iterations to a structured `results.jsonl` file that records outcomes including skips, correctness gate results, and reasons. This JSONL logging conforms to the iteration protocol and supports resuming experiments from existing results.

Can I run automated experiments directly on my main git branch?▼

No, you must use a dedicated worktree path rather than your main branch. The workflow enforces git-safe experimental discipline by committing changes and reverting discards, ensuring a clean lab notebook trail and protecting your main branch from experimental noise.

When should I use futility stopping rules in an optimization loop?▼

Use futility stopping to halt the optimization loop after consecutive unproductive iterations, preventing wasted computation when the metric stops improving. This prevents wasted computation when the measurable metric plateaus, while still allowing the experiment to resume from existing JSONL results later.