skills.rest
schattenspiegel
schattenspiegelCOMMUNITY
Explore all skills by this author →
GitHub

duckdb-polars-boundaries

Guides stage ownership and data transfer between DuckDB SQL and Polars DataFrames.

Updated Aug 12, 2026
Search Tags:#data pipeline#sql#schema validation#duckdb#polars#dataframe#arrow
One-click install
npx skills add https://github.com/schattenspiegel/skill-foundry-skills --skill duckdb-polars-boundaries-schattenspiegel
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: duckdb-polars-boundaries
Source: https://github.com/schattenspiegel/skill-foundry-skills/tree/main/skills/duckdb-polars-boundaries
Command: npx skills add https://github.com/schattenspiegel/skill-foundry-skills --skill duckdb-polars-boundaries-schattenspiegel

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires duckdb, polars, and includes references (resource) components.

What problem does it solve? Python analytics code that mixes DuckDB SQL and Polars DataFrames often suffers from repeated conversions, stale registrations, lost ordering, and schema mismatches. This Skill provides decision rules for assigning each pipeline stage to one engine and crossing the boundary deliberately. ## Core Features & Use Cases - Stage Ownership Decisions: Assign relational SQL, multi-source joins, and file scans to DuckDB while keeping typed expression pipelines in Polars. - Transfer Discipline: Materialize exactly once at a consumer boundary, manage registration lifetime, and avoid repeated .collect(), .pl(), or .arrow() calls. - Schema Verification: Validate row counts, dtypes, nulls, timestamps, nested values, and ordering after every transfer. - Use Case: You filter a large Parquet dataset with Polars, register the result in DuckDB for a grouped SQL aggregation, and convert the result back to a Polars DataFrame with verified schema and grain. ## Quick Start Ask the assistant to review your script that mixes DuckDB queries and Polars transformations and restructure it so each engine owns a clear stage with a single verified transfer.

Frequently Asked Questions about duckdb-polars-boundaries

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert between DuckDB results and Polars DataFrames?▼

Use DuckDB's `.pl()` method on a query result to get a Polars DataFrame, and register a Polars DataFrame with `connection.register()` to query it in DuckDB. Materialize only once at the consumer boundary rather than converting repeatedly in a loop.

When should I use DuckDB versus Polars for a data pipeline?▼

DuckDB owns relational SQL, multi-source joins, and file scans that SQL expresses clearly. Polars owns reusable typed expression pipelines and dataframe-native transformations. Assign each stage to one engine and cross once at a deliberate boundary.

Is DuckDB to Polars conversion zero-copy?▼

Treat conversion as potentially copying or coercing data unless current APIs and buffers prove otherwise. Zero-copy is an optimization claim, not a default, so verify behavior with your installed versions before relying on it.

Why does my DuckDB query lose row order after transferring to Polars?▼

SQL result order is undefined without an explicit ORDER BY clause, and dataframe transfer does not create a durable ordering guarantee. Add ORDER BY in the owning SQL stage and re-verify ordering after transfer.

Why does my registered Polars DataFrame fail in DuckDB after the object changes?▼

DuckDB replacement scans and registered views require the source object to stay alive while referenced. Keep objects reachable until execution finishes, and unregister names when a long-lived connection would retain stale state.

Related Skills

Explore More Data & Analytics Skills→
jluo41jluo41

haipipe-data-1-source

Convert CSV, XML, Parquet, and JSON files into standardized SourceSet assets.

Community
Advanced
shushuznshushuzn
4

pathml

Convert whole-slide pathology images into ML-ready datasets with HDF5 storage.

Community
Advanced
dtabordadtaborda

contract-testing

Validate JSON contract fixtures against Zod schemas with Vitest.

Community
Intermediate
fatcrapinmybuttfatcrapinmybutt

SINGULARITY-data-dominion

Coordinate SQL workflows, DuckDB analytics, LanceDB vectors, and cross-database federation across SQLite, DuckDB, and Polars.

Community
Advanced