bigquery-ethereum-data-acquisition

What problem does it solve?

Provides a 624x faster and cost-optimized way to acquire historical Ethereum blockchain data from Google BigQuery, eliminating the need for slow RPC polling and ensuring data is ready for ML and time-series analysis. It prevents wasted time and budget on irrelevant data.

Core Features & Use Cases

• Cost-Optimized Column Selection: Selects 11 essential columns for ML, reducing query costs by 97% (0.97 GB vs 34.4 GB for all columns) and staying well within BigQuery's free tier.
• High-Speed Data Acquisition: Download 12.44 million blocks in under an hour, significantly faster than RPC polling (which can take 26+ days), accelerating your data projects.
• Streaming to Parquet: Directly stream data to Parquet files, avoiding BigQuery storage costs and enabling efficient loading into analytical databases like DuckDB.
• Use Case: You need 5 years of Ethereum block data for a new ML model. Use this skill to quickly download the optimized 11-column dataset from BigQuery, verify its cost (0.1% of free tier), and load it into DuckDB for immediate feature engineering.

Quick Start

Download 5 years of historical Ethereum blockchain data from BigQuery. Optimize the column selection for machine learning and ensure it stays within the free tier. Save the data to a Parquet file.