data-cleaner

Handle missing values by first dropping columns exceeding a missing fraction threshold, then imputing the rest. Numeric columns use median imputation for robustness against outliers, while categorical columns use mode. Forward-fill and linear interpolation are alternatives for sequential data.

Detect outliers with the IQR method by flagging values outside Q1 minus 1.5 times IQR and Q3 plus 1.5 times IQR, or with Z-score by flagging absolute standardized values above 3. IQR is preferred for skewed distributions common in competition data.

Capping outliers at the 1st and 99th percentiles is generally safer than removal because it preserves row count and real information. Removing rows can discard legitimate extreme values, such as genuine high-income observations, so document whichever method you choose.

Standard scaling centers data to zero mean and unit variance using StandardScaler, while min-max scaling compresses values into the 0 to 1 range using MinMaxScaler. RobustScaler is a third option that uses medians and quartiles, making it resistant to outliers.

Skip normalization when using tree-based models that are scale-invariant, or when original units carry interpretive meaning needed for the paper. Always clean missing values and outliers first, since normalizing dirty data propagates errors into the scaled output.