historical-backfill-execution
CommunityBackfill blockchain data, avoid OOMs, stay on track.
Authorterrylica
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Automates the complex and memory-intensive task of loading multi-year historical blockchain data into ClickHouse, specifically designed to prevent Out-of-Memory (OOM) errors on Cloud Run and ensure data completeness and integrity.
Core Features & Use Cases
- Canonical 1-Year Chunking: Efficiently loads data in memory-safe 1-year chunks (approx. 2.6M blocks), avoiding Cloud Run OOM failures and optimizing execution time.
- Idempotent Backfills: Uses ClickHouse's
ReplacingMergeTreeengine for safe re-runs, automatically deduplicating data and ensuring data consistency. - Gap Filling & Verification: Integrates with automated gap detection and provides tools to verify data completeness and block range after backfill.
- Use Case: You need to backfill 10 years of Ethereum blocks (2015-2025) into ClickHouse. Use this skill to execute the backfill in ~20 minutes, ensuring it stays within Cloud Run's 4GB memory limit and automatically handles duplicates.
Quick Start
Execute a historical backfill of Ethereum blocks from 2015 to 2025 into ClickHouse. Ensure it's memory-safe for Cloud Run and verify data completeness after.
Dependency Matrix
Required Modules
None requiredComponents
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: historical-backfill-execution Download link: https://github.com/terrylica/gapless-network-data/archive/main.zip#historical-backfill-execution Please download this .zip file, extract it, and install it in the .claude/skills/ directory.