llm-benchmark-workflow

What problem does it solve?

This Skill streamlines the process of building and executing performance benchmarks for Large Language Models (LLMs) across various backends, helping to identify and resolve issues related to benchmark execution and performance.

Core Features & Use Cases

• Benchmark Compilation: Compiles LLM benchmarking tools, including arm-llm-bench-cli, for different ML frameworks.
• Backend Configuration: Allows selection of specific backends like llama.cpp, ONNX Runtime GenAI, MediaPipe, and MNN.
• Runtime Triage: Assists in diagnosing and fixing build or runtime problems such as shared library placement, model path configuration, and thread/token settings.
• Use Case: When you modify the benchmarking code or add new performance metrics, use this Skill to ensure the benchmarks build correctly and run as expected across different LLM backends.

Quick Start

Build the benchmark binaries for the native platform by running cmake --preset=native -B build -DBUILD_BENCHMARK=ON && cmake --build ./build --parallel.