backend-data-processor

What problem does it solve?

This Skill provides a comprehensive guide for backend data processing of vehicle insurance Excel/CSV files using Pandas. It automates file handling, data cleaning, merging, and validation, ensuring data readiness for analysis and reporting.

Core Features & Use Cases

• Data Flow Architecture: Outlines the complete process from Excel upload to merged CSV storage, including incremental updates and file versioning.
• Core Processing Functions: Details methods for reading Excel, cleaning data (missing values, date/numeric standardization), merging with existing data, and batch processing new files.
• Business Rule Implementation: Explains how to handle specific business rules like negative premiums (retained), zero commissions (retained), and date standardization.
• Performance Optimization: Offers Pandas best practices for vectorized operations, memory management, and strategies for handling large files.
• Use Case: A data engineer needs to integrate new daily Excel reports into the existing analytics platform. This skill guides them through setting up the data pipeline, ensuring new data is cleaned, deduplicated, and merged efficiently with historical records.

Quick Start

Process the new Excel file 'daily_insurance_report.xlsx' by cleaning, merging with existing data, and saving the updated dataset.