skills.rest
RonanCodes
RonanCodesCOMMUNITY
Explore all skills by this author →
Website𝕏@ronancodesGitHub

ingest-web

Extract web article content into markdown with YAML frontmatter.

Updated Aug 23, 2026
Search Tags:#metadata#vault#web-scraping#ingest#html-to-markdown#web-extraction#image-downloading
One-click install
npx skills add https://github.com/RonanCodes/llm-wiki --skill ingest-web
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ingest-web
Source: https://github.com/RonanCodes/llm-wiki/tree/main/.claude/skills/ingest-web
Command: npx skills add https://github.com/RonanCodes/llm-wiki --skill ingest-web

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Web articles and blog posts often contain cluttered HTML, missing metadata, remote image links, or are difficult to import cleanly into a markdown-based wiki; this Skill automates extracting readable content and packaging it for ingestion into a vault.

Core Features & Use Cases

  • Readable extraction: Fetches a URL and extracts the article title, author, published date, and main body while stripping navigation, sidebars, and ads.
  • HTML-to-markdown conversion: Preserves headings, lists, blockquotes, links, code blocks, and image references when converting to clean markdown.
  • Image and asset handling: Downloads referenced images into vault/raw/assets, replaces remote URLs with local paths, and records images-downloaded in the file frontmatter.
  • Metadata-first output: Writes a YAML frontmatter including source-url, title, author, date-fetched, and images-downloaded, saving results to raw/<descriptive-slug>.md for downstream wiki workflows.
  • Fallback guidance: Notes when extraction is likely to fail (heavy JS/SPAs) and recommends using a browser clipper for better fidelity.

Quick Start

Ingest the article at https://example.com/article into vault my-research to extract content, download images, and save a markdown file in raw/.

Frequently Asked Questions about ingest-web

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert web articles to markdown for my knowledge base?▼

Converting web articles to markdown involves fetching HTML, extracting the main body content, removing sidebars, and generating a clean markdown file with YAML frontmatter metadata for wiki ingestion.

Does HTML to markdown conversion preserve images and download them locally?▼

HTML to markdown conversion downloads referenced images into vault/raw/assets, replaces remote URLs with local paths, and records images-downloaded in the YAML frontmatter for archival workflows.

What metadata is extracted during web content extraction?▼

Web content extraction captures the article title, author, and published date, then packages them into a YAML frontmatter block with source-url and date-fetched for downstream wiki workflows.

Can I use this web scraper for JavaScript-heavy single page applications?▼

For heavy JavaScript or single page applications, this extraction approach is likely to fail; using a browser clipper is recommended to achieve better content fidelity for your vault.

What is the best way to archive blog posts into a markdown vault?▼

Archiving blog posts into a markdown vault is best handled by fetching the URL, extracting readable content, downloading images locally, and saving the output as raw/<descriptive-slug>.md.

Related Skills

Explore More Content & Communication Skills→
riba2534riba2534
1.3k

feishu-cli-wiki

Fetch and export Feishu Wiki nodes to Markdown with images and assets.

Community
Intermediate
anxiety135790anxiety135790

llm-wiki

Build and maintain a markdown knowledge base with YAML frontmatter and wikilinks.

Community
Advanced
oldmangrizzzoldmangrizzz

personal-knowledge-wiki

Ingest notes and articles into an interlinked Obsidian vault wiki.

Community
Advanced
Zedit42Zedit42

web-scraping-pro

Extracts content from JavaScript-heavy and protected websites using multiple fallback methods.

Community
Advanced