What problem does it solve? REST API data extraction pipelines built with dlt often run slowly because of thousands of sequential requests, one-request-per-parent child resources, tiny pages, or repeated connection overhead. This Skill provides a structured diagnose-fix-measure loop to raise extraction throughput. ## Core Features & Use Cases - Bottleneck Diagnosis: Measure per-resource extract time via pipeline.last_trace and identify whether the slowdown comes from sequential requests, child resources, small pages, or large responses. - Targeted Fixes: Parallelize child resources and top-level endpoints, increase page size, reuse a shared HTTP session, and narrow payloads with data_selector. - Rate-Limit Awareness: Monitor HTTP 429 responses and tune retry/backoff settings so added concurrency does not trigger persistent throttling. - Use Case: A pipeline fetching comments for every post takes hours because each parent triggers one sequential request. Apply the parallelized transformer pattern, raise per_page to the API maximum, and compare before/after timings in last_trace. ## Quick Start Ask the assistant to optimize your slow dlt REST API pipeline by naming the pipeline and describing the symptom, such as thousands of sequential requests or slow child resources.
