polars

Pandas to Polars migration maps df["col"] to df.select("col"), boolean indexing to df.filter(), assign to with_columns(), and groupby().transform() to .over() window expressions. Polars has no index, enforces strict typing, and parallelizes operations by default.

Use lazy scanning with pl.scan_csv() or pl.scan_parquet() instead of eager reads, then call collect(engine="streaming") to process data in chunks. You can also stream results directly to disk with sink_parquet().

A DataFrame executes operations immediately (eager), while a LazyFrame builds a query plan that Polars optimizes before execution. Lazy mode enables predicate pushdown, projection pushdown, and streaming, making it preferred for large datasets and complex pipelines.

Yes, Polars reads and writes Parquet and other formats directly from s3://, az://, and gs:// URIs. Use credential providers such as CredentialProviderAWS or Azure DefaultAzureCredential rather than hardcoding secrets.

Slow Polars code usually comes from Python lambdas in map_elements, eager reads of large files, or sequential pipe chains that disable parallelization. Stay within the expression API, use lazy mode, and filter and select columns early in the pipeline.

Pandas may fit better for complex index-based time series operations, libraries that only accept pandas input, or small datasets where performance is not critical. Polars is preferable for large datasets, strict typing, and parallel execution.