data-loading

What problem does it solve?

This Skill addresses the critical performance bottleneck of data loading and I/O operations, particularly during cold starts, by implementing advanced memory mapping and prefetching strategies.

Core Features & Use Cases

• Optimized Memory Mapping: Utilizes mmap with MAP_POPULATE for efficient zero-copy column access.
• Intelligent Prefetching: Employs posix_fadvise and zone-map-guided madvise to selectively load only necessary data blocks.
• NUMA-Aware Allocation: Distributes large memory allocations across NUMA nodes to avoid performance cliffs.
• Cold vs. Hot Start Optimization: Provides strategies to minimize I/O on cold starts while maintaining performance on hot starts.
• Use Case: When querying large datasets, this skill ensures that only the relevant columns and data blocks are read from disk, drastically reducing query latency.

Quick Start

Use the data-loading skill to optimize the memory mapping and prefetching for a column-oriented dataset.