managing-imports

What problem does it solve?

Manually managing incoming data files, checking for duplicates, and tracking their import status is prone to errors and consumes valuable time. This skill automates the entire file staging workflow, ensuring data integrity and efficiency.

Core Features & Use Cases

• Automated File Staging: Organizes incoming files by type and academic term, moving them to appropriate staged directories for processing.
• Duplicate Detection: Prevents re-importing data by checking file hashes against a manifest in a PostgreSQL database, maintaining data cleanliness.
• Import Status Tracking: Provides a clear, real-time overview of files in various workflow stages (inbox, staged, processing, archived).
• Processed File Archiving: Safely moves successfully imported files to an organized archive structure, keeping your active directories clean.
• Use Case: When new data files arrive daily, use this skill to automatically sort them, identify any already processed, and keep a real-time status of your import queue. This ensures data integrity, saves manual oversight, and reduces complexity in your data pipeline.

Quick Start

Stage all new files from the inbox and check for duplicates.