checkpoint-resume

What problem does it solve?

This Skill guarantees that data processing pipelines achieve exactly-once semantics, preventing duplicate processing of files even in the event of worker crashes.

Core Features & Use Cases

• Exactly-once processing: Ensures each file is processed precisely one time.
• Distributed coordination: Manages file claiming and status tracking across multiple workers.
• Automatic retries: Handles transient failures with configurable retry logic.
• Use Case: For large-scale batch processing jobs where data integrity is critical, this skill prevents data duplication and ensures job completion despite potential worker failures.

Quick Start

Use the checkpoint-resume skill to claim and process the file located at 's3://my-bucket/data/file-001.csv'.