train-review

What problem does it solve?

Removes manual and biased evaluation of code review agents by automating blinded reviewer runs against real captured PR test cases, ensuring the agent learns to find the same signals a human reviewer recorded without contamination.

Core Features & Use Cases

• Load and interpret test cases from a local training directory, extracting PR identifiers, comment IDs, adhoc notes, and referenced issues or follow-ups.
• Resolve comment signals by fetching GitHub review comment bodies and summarizing expected signals, including handling outdated comments by anchoring to the commit state that produced the comment.
• Spawn isolated reviewer and judge sub-agents via TeamCreate so reviewers run blind, enforce contamination checks, score results into structured JSON, and iterate prompt changes until an 80% weighted-signal pass or iteration limit.
• Logging and prompt evolution with a specificity check and automatic updates to the review-pr brief, recording iterations in a training log.

Quick Start

Train and evaluate the review-pr agent on all saved test cases and iterate prompts until the reviewer reaches an 80% weighted match or stops after five iterations.