universal-scraping-architect

Use the local BeautifulSoup example script to fetch the page with retries, parse the target table with pandas, normalize columns to snake_case, and validate required fields before saving to CSV. Run validate_extraction.py on JSON outputs to confirm they are well-formed and non-empty.

Use Firecrawl for public URLs, JavaScript-heavy SPAs, search-first discovery, or bulk domain crawling. Use local BeautifulSoup with pandas for static HTML pages, local files like PDF or Excel, and private data that should never leave your machine.

Load the key with os.getenv("FIRECRAWL_API_KEY") and never commit it to source control. The scripts fail fast with a clear error if the environment variable is missing, and the key value is never written to logs.

Yes, both example scripts check robots.txt using the Python stdlib urllib.robotparser before fetching and abort if the path is disallowed. They also send an identifying User-Agent, use timeouts, and retry with exponential backoff.

Validation fails when the extracted JSON is malformed, the result set is empty, or required columns are missing from the parsed table. Fix the selector or extraction logic and re-run rather than shipping unvalidated data.