skills.rest
hanasho744
hanasho744COMMUNITY
Explore all skills by this author →
GitHub

evaluating-code-models

Benchmarks code generation models on HumanEval, MBPP, and MultiPL-E with pass@k metrics.

Updated Mar 23, 2026
Search Tags:#code generation#benchmarking#llm evaluation#pass@k#humaneval#bigcode#multipl-e
One-click install
npx skills add https://github.com/hanasho744/codex --skill evaluating-code-models-hanasho744
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-code-models
Source: https://github.com/hanasho744/codex/tree/main/.agents/skills/bigcode-evaluation-harness
Command: npx skills add https://github.com/hanasho744/codex --skill evaluating-code-models-hanasho744

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires bigcode-evaluation-harness, transformers>=4.25.1, accelerate>=0.13.2, datasets>=2.6.1, and includes references (resource) components.

What problem does it solve? Measuring how well a code generation model actually works requires executing generated code against unit tests across standardized benchmarks, which is tedious and error-prone to set up manually. This Skill provides the complete BigCode Evaluation Harness workflow for running HumanEval, MBPP, MultiPL-E, and 15+ other benchmarks with correct pass@k estimation. ## Core Features & Use Cases - Standard Benchmark Evaluation: Run HumanEval, HumanEval+, MBPP, and MBPP+ with proper temperature, n_samples, and code execution settings for accurate pass@k scores. - Multi-Language Evaluation: Test models across 18 programming languages via MultiPL-E using a generate-on-host, execute-in-Docker workflow for safe code execution. - Instruction Model & Custom Task Support: Evaluate chat models with instruction tokens and HumanEvalPack, or implement custom benchmark tasks by extending the Task base class. - Use Case: Compare StarCoder2, CodeLlama, and DeepSeek-Coder on HumanEval and MBPP, then generate a pass@1 comparison table to decide which model to deploy. ## Quick Start Ask the AI to evaluate a HuggingFace code model like bigcode/starcoder2-7b on HumanEval with 200 samples and code execution enabled, then report the pass@1 score.

Frequently Asked Questions about evaluating-code-models

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate a code generation model on HumanEval?▼

Run accelerate launch main.py with --model pointing to your HuggingFace model, --tasks humaneval, --n_samples 200, --temperature 0.8, and --allow_code_execution. Results are saved as JSON containing pass@1, pass@10, and pass@100 scores.

What is the difference between HumanEval and HumanEval+?▼

HumanEval+ uses the same 164 problems as HumanEval but with 80 times more test cases per problem, catching solutions that pass original tests but fail on edge cases. MBPP+ similarly extends MBPP with 35 times more tests.

How do I evaluate code models on multiple programming languages?▼

Use MultiPL-E tasks like multiple-py, multiple-js, multiple-java, or multiple-cpp covering 18 languages. Generate solutions on the host with --generation_only, then execute them inside the official MultiPL-E Docker image for safe evaluation.

Why do my HumanEval results differ from published scores?▼

Mismatches usually come from wrong n_samples (use 200 for accurate pass@k), incorrect temperature (0.2 for pass@1, 0.8 for higher k), wrong task names, or missing instruction tokens for chat models. Verify all generation parameters match the reference setup.

How do I fix CUDA out of memory errors during evaluation?▼

Enable quantization with --load_in_8bit or --load_in_4bit, reduce --batch_size to 1, set --max_memory_per_gpu, or use --precision fp16. A 7B model needs about 14GB VRAM in fp16 but only 6GB in 4-bit.

When should I use BigCode Evaluation Harness versus lm-evaluation-harness?▼

Use BigCode Evaluation Harness for code generation benchmarks requiring test execution and pass@k metrics, including multi-language evaluation. Use lm-evaluation-harness for general LLM benchmarks like MMLU, GSM8K, and HellaSwag that do not involve code execution.

Related Skills

Explore More Software Engineering Skills→
ScientiaCapitalScientiaCapital
1

model-deployment

Export fine-tuned models to GGUF and deploy with vLLM or Ollama.

Community
Advanced
Clay-HHKClay-HHK
1

evaluating-code-models

Benchmarks code generation models on HumanEval, MBPP, and MultiPL-E with pass@k metrics.

Community
Advanced
ShunsukeHayashiShunsukeHayashi

review-agent

Assess JavaScript/TypeScript code quality and security with ESLint, TypeScript, npm audit, and secret detection.

Community
Intermediate
DiologIRDiologIR

bench-task-judge

Evaluate benchmark tasks for difficulty and fairness using statistics, rubric checks, mutation tests, and solution replays.

Official
Advanced