expanso-deduplicate-events

What problem does it solve?

This Skill prevents duplicate event processing by identifying and filtering out identical events based on their content.

Core Features & Use Cases

• Content Hashing: Generates a unique key based on event content for reliable deduplication.
• Cache-Based Detection: Leverages a cache to efficiently track seen events and detect duplicates.
• Configurable Strategy: Supports simple event_id based deduplication or a more complex composite key including event type and timestamp.
• Use Case: In a high-throughput system, ensure that the same event (e.g., a user login) is not processed multiple times, preventing erroneous actions or inflated metrics.

Quick Start

Run the pipeline with sample data using the provided script.