vaex

What problem does it solve?

This Skill introduces Vaex for out-of-core, lazy-evaluated dataframes that handle datasets far larger than RAM, enabling fast statistics, visualizations, and ML pipelines on big data.

Core Features & Use Cases

• Out-of-core dataframes: Open and analyze multi-terabyte datasets without loading entirely into memory.
• Efficient aggregations & virtual columns: Create derived features without materializing large intermediate datasets.
• ML pipelines on big data: Build scalable pipelines with Vaex ML, scikit-learn integration, and streaming I/O.

Quick Start

Open a large dataset, create a virtual column, and compute a fast mean without loading all data.