SmartHome Video Anomaly Benchmark

What problem does it solve?

This Skill automates the evaluation of Visual-Language Models (VLMs) on their ability to detect anomalies in smart home video footage, providing a standardized benchmark for performance.

Core Features & Use Cases

• Video Anomaly Detection: Evaluates VLMs on identifying unusual events across 7 smart home categories (Wildlife, Senior Care, Baby Monitoring, Pet Monitoring, Home Security, Package Delivery, General Activity).
• Multi-Frame Analysis: Requires VLM understanding of video sequences, not just single frames.
• Automated Reporting: Generates detailed HTML reports with metrics, confusion matrices, and historical comparisons.
• Use Case: A researcher wants to compare how well two different VLMs can detect package theft from security camera footage. They can run this benchmark on both models and compare the accuracy and F1 scores in the generated report.

Quick Start

Run the smarthome-bench skill to evaluate a VLM at http://localhost:5405 using the default subset of videos.