ais-deduplication

What problem does it solve?

This Skill implements a three-layer deduplication strategy to prevent processing duplicate AIS messages during ingestion and analysis.

Core Features & Use Cases

• Layer 1: Redis SET NX with a 60-second TTL to rapidly reject duplicates in volatile memory.
• Layer 2: Bloom Filter (ReBloom) for probabilistic pre-filtering to minimize database hits.
• Layer 3: PostgreSQL UPSERT to guarantee deterministic persistence by inserting only unique messages.
• Hashing Window: Messages are hashed within a 30-second window to group near-similar events.
• Use Case: In high-volume AIS streams from multiple sources, this stack ensures each unique AIS message is stored once across the system.

Quick Start

Configure your AIS ingestion to route messages through this three-layer deduplication workflow. Ensure Redis is running with a 60-second TTL, initialize the Bloom Filter, and connect to PostgreSQL for final persistence.