dlt-connector

Load SaaS API data into DuckDB with dlt and generate a queryable Wren semantic project.

17.7k|2.0k|Updated Mar 13, 2024
One-click install
npx skills add https://github.com/Canner/WrenAI --skill dlt-connector-canner
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dlt-connector
Source: https://github.com/Canner/WrenAI/tree/main/core/wren/src/wren/skills_content/dlt-connector
Command: npx skills add https://github.com/Canner/WrenAI --skill dlt-connector-canner

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires duckdb, pyyaml, dlt, and includes scripts (resource) and references (resource) components.

What problem does it solve? Getting data out of SaaS tools like HubSpot, Stripe, Salesforce, GitHub, or Slack and into a form you can query with SQL normally requires writing custom ETL code and hand-modeling schemas. This Skill automates the full path: extract data with dlt into a local DuckDB file, introspect the schema, and generate a verified Wren semantic project that answers real SQL queries. ## Core Features & Use Cases - Guided dlt pipeline setup: Pick a verified SaaS source, configure credentials via environment variables or secrets.toml, and run the pipeline into DuckDB. - Automatic project generation: The bundled introspect_dlt.py script discovers tables, filters dlt internal columns, detects parent-child relationships from _dlt_parent_id, normalizes column types via sqlglot, and writes a complete Wren v5 YAML project. - End-to-end verification: Builds the MDL and runs real SQL queries against every model before declaring success, catching catalog naming and type errors early. - Use Case: Connect your Stripe account, load customers, charges, and subscriptions into DuckDB, then immediately ask SQL questions through a governed Wren semantic layer. ## Quick Start Ask the assistant to connect your HubSpot data to Wren using dlt and generate a queryable project from the resulting DuckDB file.

Frequently Asked Questions about dlt-connector

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I load SaaS API data into DuckDB with dlt?▼

Install dlt with the duckdb extra, write a pipeline script that imports the verified source for your SaaS tool, set credentials as environment variables, and run the script. dlt creates a local .duckdb file containing all extracted tables.

How do I generate a Wren project from a dlt DuckDB file?▼

Run the introspect_dlt.py script with --duckdb-path, --output-dir, and --project-name. It discovers tables via information_schema, filters dlt internal columns, detects relationships, and writes a complete Wren v5 YAML project.

Which SaaS sources does dlt support for data extraction?▼

dlt provides verified sources for HubSpot, Stripe, Salesforce, GitHub, Slack, Google Analytics, Google Sheets, Notion, Jira, Zendesk, Shopify, and Airtable. Each source has its own authentication pattern documented in the dlt_sources reference.

Why do I get table not found errors when querying DuckDB through Wren?▼

The most common cause is a catalog mismatch: Wren ATTACHes the DuckDB file using its filename stem as the catalog, so every model's table_reference.catalog must equal that filename stem. The introspection script sets this automatically.

Can I run a dlt pipeline while querying the same DuckDB file?▼

No. DuckDB has a single-writer limitation, so running a dlt sync while querying causes lock conflicts. For concurrent access, write to a separate file and swap it atomically after the pipeline finishes.