transbigdata-preprocess

What problem does it solve?

Provide practical, repeatable preprocessing and data quality evaluation for transportation spatiotemporal datasets so noisy, out-of-bounds, and misindexed trajectory records are identified and corrected before analysis.

Core Features & Use Cases

• Data overview and sampling interval analysis to surface missing data, sampling irregularities, and time range summaries.
• Boundary and polygon-based filtering to remove records outside study areas and improve spatial data integrity.
• ID reindexing by time gaps or distance gaps to separate long interruptions or spatial jumps into distinct trajectories.
• Use Case: Clean taxi GPS logs by filtering to the study city, analyze sampling intervals, reindex interrupted trips with a 1-hour threshold, and export a cleaned dataset for OD extraction.

Quick Start

Run a data quality summary on gps_data.csv, apply bounding-box and polygon filtering for the study area, reindex IDs using a 3600 second time gap, and save the cleaned result as gps_data_cleaned.csv.