telemetry-redrive

What problem does it solve?

This Skill addresses the critical need to reprocess telemetry data due to corruption, parsing errors, or schema changes. It automates the complex task of identifying and redriving specific files or entire days of data, while preventing common pitfalls like data duplication.

Core Features & Use Cases

• Flexible Redrive Options: Reprocess single raw JSONL files, multiple files via S3 prefix, or surgically redrive based on Parquet files or specific days.
• Automated Parquet Deletion: Prevent data duplication by automatically deleting old Parquet files before reprocessing, ensuring data integrity.
• Surgical Day-Based Redrive: Efficiently reprocess all data for a specific day and type, ideal for bug fixes affecting historical data.
• SQS Queue Monitoring: Includes safeguards to monitor the SQS queue, preventing overload during large redrive operations.
• Use Case: A bug fix in the telemetry parser requires reprocessing all lock type data for a specific day. Use the surgical day-based redrive to automatically delete old Parquet files and re-ingest the correct data, ensuring data integrity without manual intervention.

Quick Start

Perform a dry run to redrive a single telemetry file located at 's3://stage-signal-data-lake-raw/raw/json_telemetry/year=2025/month=11/day=24/hour=21/telemetry-signals-lake-firehose-1-2025-11-24-21-09-35-xxx.gz' in the platform-stage environment.