data-collector

What problem does it solve?

Pattern for building data source collectors that fetch, transform, and deduplicate incident data from external APIs. Use when adding new data sources, fixing collector bugs, or understanding the data pipeline.

Core Features & Use Cases

• Defines a standard Collector interface with fetch() and transform(raw) methods to fetch and normalize raw data into a unified Incident shape.
• Provides guardrails for reliability: upsert to storage, dedupe keys, and error handling during fetch or transform.
• Supports common data sources (e.g., NYC FDNY, 511NY, NWS) as example implementations to illustrate integration patterns and debugging workflows.

Quick Start

Create a new collector by implementing the Collector interface, add it to the collectors registry, and run the processor to ingest and upsert incidents.