skill-system-eda

You can profile large CSV or Parquet datasets deterministically by using Polars-first lazy scanning, which processes data efficiently and produces profile.yaml, report.md, and contract artifacts without loading everything into memory.

Yes, Polars is used as the primary engine to generate machine-readable data contracts and conduct anomaly analysis, enforcing high-cardinality guards to ensure deterministic and reliable outputs.

The best way to detect data drift is to profile tabular files using a deterministic method that generates a profile.yaml, allowing you to compare current dataset statistics against historical profiles for drift detection.

Yes, you can profile and validate tabular datasets processed with scikit-learn and numpy, as the profiling logic applies to CSV and Parquet files of varying sizes to check data quality and generate reports.

Deterministic dataset profiling handles high-cardinality columns by enforcing specific high-cardinality guards during the Polars lazy scanning process, preventing memory overflow and ensuring stable profile generation.

Polars is preferred for exploratory data analysis on large files because it uses a lazy scanning approach that processes data in batches, avoiding the memory limitations often encountered with pandas when handling large CSV or Parquet files.