incremental-fetch
CommunityIngest API data incrementally and reliably.
Authorrohunvora
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Build data pipelines that never lose progress and never re-fetch existing data. Use when fetching paginated data from external APIs and need to track progress, avoid duplicates, handle rate limits, and support both incremental updates and backfills.
Core Features & Use Cases
- The Two Watermarks Pattern: Track two cursors (newest_id and oldest_id) to enable forward and backward fetching for incremental updates and backfills.
- Data vs Watermark Saving: Save data pages after each fetch for resilience; commit watermarks only at end for correctness.
- Backfill and Resume: Supports backfill mode and seamless resumption after interruptions across paginated APIs.
Quick Start
Run incremental-fetch to fetch paginated API data, saving each page and updating watermarks only after a successful run.
Dependency Matrix
Required Modules
None requiredComponents
references
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: incremental-fetch Download link: https://github.com/rohunvora/cool-claude-skills/archive/main.zip#incremental-fetch Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 223,000+ vetted skills library on demand.