skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

evaluating-code-models

Benchmarks code generation models on HumanEval, MBPP, and MultiPL-E with pass@k metrics.

1|Updated Mar 5, 2026
Search Tags:#code generation#benchmarking#model evaluation#pass@k#humaneval#bigcode#multipl-e
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill evaluating-code-models-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-code-models
Source: https://github.com/Clay-HHK/claude-skills/tree/main/bigcode-evaluation-harness
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill evaluating-code-models-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires bigcode-evaluation-harness, transformers, accelerate, datasets, and includes references (resource) components.

What problem does it solve? Measuring how well a code generation model actually works requires executing generated code against unit tests across standardized benchmarks, which is tedious and error-prone to set up manually. This Skill provides the complete BigCode Evaluation Harness workflow for reproducible pass@k benchmarking. ## Core Features & Use Cases - Standard Benchmark Evaluation: Run HumanEval, HumanEval+, MBPP, and MBPP+ with configurable temperature, n_samples, and pass@k metric output. - Multi-Language Evaluation: Test models across 18 programming languages via MultiPL-E using a Docker-based safe execution workflow. - Instruction Model Support: Evaluate chat and instruction-tuned models with instruct-humaneval and HumanEvalPack tasks using proper instruction tokens. - Use Case: Compare StarCoder2, CodeLlama, and DeepSeek-Coder on HumanEval and MBPP, then generate a markdown comparison table of pass@1 scores for a model selection decision. ## Quick Start Ask the AI to evaluate a HuggingFace code model like bigcode/starcoder2-7b on HumanEval with 200 samples and code execution enabled, then report the pass@1 score.

Frequently Asked Questions about evaluating-code-models

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate a code model on HumanEval?▼

Run accelerate launch main.py with --model pointing to your HuggingFace model, --tasks humaneval, --n_samples 200, --temperature 0.8, and --allow_code_execution. Results including pass@1, pass@10, and pass@100 are written to the metric output JSON file.

What is the difference between HumanEval and HumanEval+?▼

HumanEval+ uses the same 164 problems as HumanEval but with 80 times more test cases per problem, catching solutions that pass original tests but fail on edge cases. It takes longer to execute due to the additional tests.

How do I evaluate code generation in multiple programming languages?▼

Use MultiPL-E tasks like multiple-py, multiple-js, and multiple-java covering 18 languages. Generate solutions on the host with --generation_only, then evaluate inside the official MultiPL-E Docker image for safe cross-language execution.

Why do my HumanEval results differ from published scores?▼

Mismatches usually come from wrong n_samples (use 200 for accurate pass@k), incorrect temperature (0.2 for pass@1, 0.8 for pass@10/100), wrong task names, or missing instruction tokens for chat models. Verify all four settings against the reference configuration.

How do I fix CUDA out of memory errors during evaluation?▼

Enable quantization with --load_in_8bit or --load_in_4bit, reduce --batch_size to 1, or set --max_memory_per_gpu to limit usage. A 7B model needs about 14GB in fp16 but only 6GB in 4-bit.

When should I use BigCode Evaluation Harness vs lm-evaluation-harness?▼

Use BigCode Evaluation Harness for code generation benchmarks requiring functional correctness testing with pass@k. Use lm-evaluation-harness for general LLM benchmarks like MMLU, GSM8K, and HellaSwag that do not involve code execution.

Related Skills

Explore More Software Engineering Skills→
automatedigitalautomatedigital
1

fine-tuning-with-trl

Automate LLM post-training and alignment workflows with the TRL library.

Official
Advanced
larsboeslarsboes

Evals

Evaluate agent workflows and prompts with deterministic tests and AI-based judges.

Community
Advanced
ar0cket1ar0cket1

evaluating-llms-harness

Run standardized language model evaluations across 60+ benchmarks.

Community
Advanced
ChasonnnnChasonnnn

terminal-bench-loop

Automate bounded Terminal-Bench test loops with diagnostics and human approval.

Community
Advanced