What problem does it solve? PyArrow code often fails silently through schema drift, unexpected nulls, chunking assumptions, unsafe casts, and unverified zero-copy or pushdown claims. This Skill enforces deliberate choices about object types, batch boundaries, nullability, memory ownership, and materialization points so Arrow-based pipelines behave predictably. ## Core Features & Use Cases - Object and workflow selection: Classify the right Arrow object (Array, ChunkedArray, RecordBatch, Table, RecordBatchReader, Dataset, Scanner) from the boundary contract instead of defaulting everything to Table. - Dataset scans with pushdown: Build typed Dataset scans with projection and predicate pushdown using dataset expressions, streaming bounded batches instead of materializing entire datasets. - Boundary testing and API grounding: Test schema equality, null counts, multi-chunk inputs, and reopened Parquet/IPC artifacts, and inspect the installed PyArrow API with the included script before relying on drifting options. - Use Case: When building a pipeline that scans partitioned Parquet files, use this Skill to pin an explicit schema, push filters into the scanner, stream record batches, and verify the persisted schema after writing. ## Quick Start Ask the AI to review or write PyArrow code for scanning a partitioned Parquet dataset with an explicit schema and streaming record batches.
