duplicate-detection

What problem does it solve?

This skill helps clean data by identifying exact and near-duplicate content, enabling deduplication and improved data quality.

Core Features & Use Cases

• Exact duplicate detection via content hash to quickly spot exact copies.
• Semantic similarity search to catch paraphrases and near-duplicates that hash misses.
• Use case: deduplicate FAQs, articles, tickets, and product descriptions to maintain a concise knowledge base.

Quick Start

To run this skill, provide a dataset of records and request deduplication; the tool will return unique records and a list of duplicates with similarity scores.