using-crawl4ai-cli

Create a CSS extraction config and a JSON schema defining a baseSelector and fields, then run crwl with the -e and -s flags and -o json output. The generate_schema.py script builds schemas interactively if you prefer not to write JSON by hand.

Use a dynamic crawler config with scan_full_page set to true, wait_until set to networkidle, and a delay_before_return_html of about 2 seconds. The init_crawler.py script generates this preset with the --preset dynamic flag.

Playwright browsers are installed per Python version, so the browser may be missing for the Python that crwl uses. Run the included setup.sh script, which detects the correct Python path and installs Chromium automatically.

Yes, the -q flag sends a natural-language question about the crawled page to a configured LLM. Supported providers include openai/gpt-4, anthropic/claude-3-sonnet, and local ollama models that need no API key.

Empty results usually mean selectors are wrong or dynamic content has not loaded yet, while contamination means the baseSelector is too broad. Inspect the page with Chrome MCP first, then run validate_extraction.py on the output to detect these issues.

Avoid it for simple static HTML where plain HTTP requests suffice, and for sites requiring complex authentication flows since the CLI does not directly support login sessions. Dedicated browser automation tools fit those cases better.