skills.rest
heyixue511-creator
heyixue511-creatorCOMMUNITY
Explore all skills by this author →
GitHub

S12-重复记录检测

Detects and resolves duplicate records in bibliometric datasets using DOI, title, and version matching.

Updated May 18, 2026
Search Tags:#deduplication#data cleaning#bibliometrics#literature analysis#record linkage#doi matching
One-click install
npx skills add https://github.com/heyixue511-creator/ai --skill s12-heyixue511-creator
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: S12-重复记录检测
Source: https://github.com/heyixue511-creator/ai/tree/main/My-Own-Skills-main/%E6%96%87%E7%8C%AE%E8%AE%A1%E9%87%8F%E5%88%86%E6%9E%90%E6%8A%80%E8%83%BD/S12-%E9%87%8D%E5%A4%8D%E8%AE%B0%E5%BD%95%E6%A3%80%E6%B5%8B
Command: npx skills add https://github.com/heyixue511-creator/ai --skill s12-heyixue511-creator

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Bibliometric datasets merged from multiple sources often contain duplicate records—identical DOIs, near-identical titles, or Early Access versions coexisting with formally published articles—which distort citation counts and analysis accuracy. This Skill detects, groups, and resolves these duplicates before final export. ## Core Features & Use Cases - Three-Layer Detection: Identifies exact duplicates (same DOI/UT/PMID), suspected duplicates (title similarity above 90% with same publication year), and version duplicates (Early Access vs. formally published articles). - Flexible Resolution Strategies: Supports automatic deduplication by completeness rules, manual per-group confirmation, or report-only mode without removal. - Mapping Table Integration: Marks removed records as DUPLICATE in the mapping table so downstream export steps automatically exclude them. - Use Case: After merging Web of Science and Scopus exports, run this Skill to find 15 duplicate groups, keep the most complete record from each, and ensure the final bibliometric analysis counts each paper only once. ## Quick Start Detect duplicate records in my merged bibliographic dataset and show me the duplicate groups before removing them.

Frequently Asked Questions about S12-重复记录检测

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I remove duplicate records from a bibliometric dataset?▼

Group records by source identifiers such as DOI, WoS UT, or PMID to find exact duplicates, then compare titles and publication years for suspected duplicates. Keep the most complete record in each group and mark the rest as duplicates in the mapping table.

How to detect duplicates when records have no DOI?▼

For records lacking identifiers, compare normalized titles with punctuation and articles removed, requiring both high title similarity and matching publication year. This catches suspected duplicates that exact identifier matching would miss.

What is the difference between exact and suspected duplicate detection?▼

Exact duplicates share an identical DOI, UT, or PMID and can be confirmed with certainty. Suspected duplicates rely on title similarity above 90 percent plus the same publication year, so they carry some uncertainty and may warrant manual review.

Should I keep the Early Access or published version of an article?▼

Keep the formally published version when both exist. Early Access records are preliminary versions, so the record with document type Article and a formal publication year is the authoritative one for bibliometric counting.

When should deduplication run in a data cleaning pipeline?▼

Deduplication is optional and typically runs after source data analysis reveals suspected duplicates, or just before final export. It requires the mapping table with sequence and source identifier fields to already exist.

Related Skills

Explore More Education & Research Skills→
drkamaruldrkamarul

medical-research

Search and analyze biomedical literature from PubMed, Web of Science, Scopus, and Google Scholar.

Community
Advanced
CeeKay1901CeeKay1901

daten-aufbereiten

Clean CSV and spreadsheet files into analysis-ready tables with a change log.

Community
Intermediate
NOMARJNOMARJ
5

silence-audit

Audit persona panels for under-represented groups and rate their material impact.

Official
Intermediate
SufficientDaikonSufficientDaikon
8

content-deduplication

Detect and deduplicate text content using normalization, SHA-256 fingerprinting, and Jaccard similarity.

Community
Intermediate