pytd

What problem does it solve?

This Skill provides expert assistance for using pytd (Python SDK) to query and import data with Treasure Data, streamlining Python-based data analysis, ETL pipelines, and integration with data science workflows. It helps users efficiently move data between pandas DataFrames and TD, reducing manual effort and potential errors.

Core Features & Use Cases

• Query & Load: Seamlessly query TD data into pandas DataFrames and load DataFrames back to TD tables using various writer types (bulk_import, insert_into, spark).
• Large Result Handling: Guides on using the DB-API for iterative retrieval of large result sets, preventing memory issues and timeouts in Python when processing massive datasets.
• ETL Pipeline Patterns: Provides common patterns for building complete ETL workflows, including incremental loading and multi-database operations, ensuring robust and scalable data pipelines.
• Use Case: A data scientist needs to pull a large dataset from TD, perform complex feature engineering using pandas, and then upload the transformed data back to TD for model training. This skill helps them use pytd for efficient data transfer, iterative processing, and robust error handling within their Python environment.

Quick Start

Install pytd

pip install pytd

Query data into a pandas DataFrame

import pytd import pandas as pd client = pytd.Client(database='sample_datasets') result = client.query('SELECT * FROM nasdaq LIMIT 100') df = pd.DataFrame(result['data'], columns=result['columns']) print(df.head())