numpy-io

What problem does it solve?

This Skill addresses the challenge of handling large NumPy datasets that exceed available RAM or need to be persisted efficiently to disk, streamlining data storage and retrieval.

Core Features & Use Cases

• Memory Mapping: Access and modify large arrays directly on disk as if they were in memory, ideal for datasets larger than RAM.
• Compressed Storage: Save multiple NumPy arrays into a single, compressed .npz file for efficient storage and retrieval.
• Text File Parsing: Load data from text files, including messy CSVs with missing values, using robust parsing functions.
• Use Case: Store and access multi-gigabyte simulation results without running out of memory, or bundle model weights and training data into a single, compact file.

Quick Start

Use the numpy-io skill to save two arrays, 'weights' and 'biases', into a compressed file named 'model_data.npz'.