pandas-performance

What problem does it solve?

This Skill tackles the common performance bottlenecks in pandas, enabling you to process datasets that are too large or slow for standard operations, significantly reducing memory usage and execution time.

Core Features & Use Cases

• Memory Optimization: Downcasts numerical types and converts low-cardinality strings to categorical types to drastically reduce RAM footprint.
• Speed Optimization: Replaces slow row-wise operations with vectorized alternatives and leverages efficient lookup methods.
• Efficient I/O: Promotes the use of Parquet for faster reads/writes with predicate pushdown and demonstrates chunking for memory-constrained systems.
• Use Case: You have a 50GB CSV file and need to perform aggregations. This Skill guides you on how to load it efficiently using chunking or the PyArrow backend and optimize subsequent operations for speed.

Quick Start

Optimize the memory usage of your DataFrame by applying the optimize_memory function.