scaffold-task

What problem does it solve?

This Skill streamlines the creation of new benchmark tasks and entire benchmark suites for the CodeScaleBench project, significantly reducing the manual effort involved in setting up new evaluation scenarios.

Core Features & Use Cases

• Task Scaffolding: Generates all necessary files (task.toml, instruction.md, Dockerfile, test.sh, etc.) for both SDLC and org-scale tasks.
• Suite Creation: Facilitates the creation of new benchmark suites, including configuration files and run scripts.
• Interactive Workflow: Guides the user through a series of questions to gather all required inputs for task and suite generation.
• Use Case: A researcher wants to add a new type of cross-repository dependency tracing task to CodeScaleBench. They use this Skill to interactively define the task, specify the repositories involved, and generate all the boilerplate files, then customize the instructions and verification steps.

Quick Start

Use the scaffold-task skill to create a new SDLC task in an existing suite.