create-filesystem-pipeline

Creates a dlt pipeline that loads CSV, Parquet, or JSONL files from local or cloud storage into a destination.

Updated Jun 15, 2026
One-click install
npx skills add https://github.com/aminojagh/LLMZC --skill create-filesystem-pipeline-aminojagh
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: create-filesystem-pipeline
Source: https://github.com/aminojagh/LLMZC/tree/main/05_02_dlt_workshop/.claude/skills/create-filesystem-pipeline
Command: npx skills add https://github.com/aminojagh/LLMZC --skill create-filesystem-pipeline-aminojagh

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires dlt, pandas, pyarrow.

What problem does it solve? Setting up a dlt filesystem pipeline involves many decisions—backend credentials, file readers, glob patterns, table naming, and secrets configuration—that are easy to get wrong on the first attempt. This Skill scaffolds and configures a working filesystem pipeline step by step, so data flows from files into a destination without trial-and-error debugging. ## Core Features & Use Cases - Guided scaffolding: Runs dlthub pipeline init in the working directory, then replaces the demo code with a focused single-table or multi-table pipeline function. - Multi-backend support: Handles Local disk, S3, GCS, Azure, and SFTP with the correct dlt extras and per-backend credential placeholders written via MCP tools. - Safe first runs: Enumerates matched files before loading, recommends sample runs with add_limit(1), and provides a debug table for common first-run errors. - Use Case: You have a folder of CSV reports in an S3 bucket and want them in DuckDB. The Skill collects your destination, backend, and bucket URL, scaffolds the pipeline, writes placeholder credentials, and walks you through a verified sample load. ## Quick Start Ask the assistant to create a dlt filesystem pipeline that loads CSV files from your S3 bucket into DuckDB, providing your bucket URL when prompted.

Frequently Asked Questions about create-filesystem-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I load CSV files from S3 into DuckDB with dlt?▼

Use the dlt filesystem source with the read_csv reader: pipe filesystem(file_glob="*.csv") into read_csv() and run it through a dlt pipeline with destination duckdb. Install dlt[hub,s3] and pandas, then configure bucket_url in .dlt/config.toml and AWS credentials in secrets.toml.

How do I load multiple folders into separate tables in dlt?▼

Create one filesystem(...) | read_csv() pipe per folder, each with its own file_glob and a distinct .with_name("table_name") call, then pass them as a list to pipeline.run(). Without renaming, readers of the same kind collide into a single destination table.

Does the dlt filesystem source support GCS and Azure?▼

Yes, the filesystem source supports S3, GCS, Azure, SFTP, and local disk. Each cloud backend needs its dlt extra (dlt[hub,gs] or dlt[hub,az]) and matching credentials such as a GCP service-account key or Azure storage account key in secrets.toml.

Why does my dlt filesystem pipeline fail with ConfigFieldMissingException?▼

This error means credential placeholders in .dlt/secrets.toml were never replaced with real values. Open the file, fill in the fields for your backend (for example aws_access_key_id and aws_secret_access_key for S3), and rerun the pipeline.

How do I test a dlt pipeline on one file before a full load?▼

Set files_per_page=1 and call .add_limit(1) on the filesystem(...) resource so only a single file is yielded and loaded. After verifying the result, remove both settings and run again for the full load.

When should I not use the filesystem source in dlt?▼

Do not use the filesystem source for REST APIs or databases—use the rest_api or sql_database sources instead. The filesystem source is designed for reading files (CSV, Parquet, JSONL, or custom formats) from storage buckets or local disk.