Home Security AI Benchmark

What problem does it solve?

This Skill evaluates the performance of Large Language Models (LLMs) and Vision-Language Models (VLMs) on a comprehensive suite of tests specifically designed for home security AI applications. It helps users select and optimize AI models for tasks like scene analysis, threat detection, and alert triage.

Core Features & Use Cases

• Extensive Test Suite: Covers 143 tests across 16 categories including context preprocessing, tool use, security classification, prompt injection resistance, and VLM scene analysis.
• LLM & VLM Evaluation: Assesses both text-based reasoning and visual understanding capabilities of AI models.
• Detailed Reporting: Generates an HTML report with pass/fail scores, latency charts, and token usage for easy comparison.
• Use Case: A security system integrator can use this Skill to benchmark different LLMs and VLMs to determine which models provide the best accuracy and reliability for their AI-powered security cameras before deployment.

Quick Start

Run the benchmark using the command node scripts/run-benchmark.cjs --vlm http://localhost:5405 to evaluate LLM and VLM capabilities with a VLM server running at the specified address.