What problem does it solve? Many valuable sources—official news sites, forum portals, blog index pages—offer neither a platform API nor an RSS feed, and often block bare HTTP clients or hide content behind anti-bot and JavaScript walls. This Skill provides a universal fallback adapter that fetches such pages, extracts the main article text and metadata, and enumerates article links from index pages. ## Core Features & Use Cases - Two-rung fetch ladder: Fetches pages with curl plus a real browser User-Agent and extracts main content with trafilatura; when an anti-bot or JS-render wall is detected, it escalates to Jina Reader (r.jina.ai) for server-side headless rendering. - Link enumeration with regex filtering: The enum command harvests article links from a list page, filtered by a caller-supplied --link-pattern regex matched against absolute URLs. - One-stop sync with mirror deduplication: The sync command enumerates a list page, fetches each new article, stores results in a stable per-URL mirror directory, and skips already-fetched items so re-runs are idempotent. - Use Case: Track a newspaper's digital edition that has no RSS feed—run sync with a link pattern like content_\d+\.htm to pull the latest articles into structured S1 JSON with title, author, date, and markdown body. ## Quick Start Ask the agent to fetch the article at a given URL using the generic web-sources adapter, or to sync the latest articles from a list page URL with a link pattern regex that matches its article links.
