locomo-benchmark

What problem does it solve?

This Skill addresses the need to quantitatively evaluate the effectiveness of long-term conversational memory systems, like cc-soul, against established academic benchmarks.

Core Features & Use Cases

• Benchmark Execution: Runs the LoCoMo benchmark suite to assess memory recall capabilities.
• Data Ingestion & Evaluation: Downloads benchmark data, ingests conversations into memory, and evaluates question-answer pairs.
• Detailed Reporting: Provides F1 scores broken down by category (Multi-hop, Single-hop, Temporal, Open-domain, Adversarial) and per conversation.
• Use Case: A developer can use this Skill to verify that recent improvements to cc-soul's memory system have indeed led to better performance on long-term conversational recall tasks, comparing results against baseline models.

Quick Start

Run the locomo benchmark for conversation 26 using the command: python3 $PLUGIN_DIR/scripts/locomo-benchmark.py conv-26