r-duckplyr

What problem does it solve?

This Skill addresses the performance bottlenecks and memory limitations encountered when using dplyr with large datasets (over 100k rows) or larger-than-memory files, by leveraging DuckDB for efficient, lazy data processing.

Core Features & Use Cases

• Lazy Evaluation: Operations are executed only when results are needed, optimizing performance for large datasets.
• Larger-than-Memory Support: Directly query Parquet, CSV, and other file formats from disk or URLs without loading them entirely into RAM.
• DuckDB Backend: Utilizes DuckDB's speed and efficiency for data manipulation tasks.
• Use Case: Process multi-gigabyte Parquet files stored on disk using familiar dplyr syntax, performing complex filtering, grouping, and summarization without running out of memory.

Quick Start

Use the r-duckplyr skill to read the file 'large_dataset.parquet' and filter rows where the 'year' column is 2024.