skills.rest
AMC-JTC
AMC-JTCCOMMUNITY
Explore all skills by this author →
GitHub

blog-ingest

Ingest entire blogs, newsletters, and RSS feeds into deduplicated brain source pages.

Updated Aug 17, 2026
Search Tags:#knowledge base#archive#deduplication#rss#substack#feed parsing#blog ingestion
One-click install
npx skills add https://github.com/AMC-JTC/gbrain-1 --skill blog-ingest-amc-jtc
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: blog-ingest
Source: https://github.com/AMC-JTC/gbrain-1/tree/main/plugin/skills/blog-ingest
Command: npx skills add https://github.com/AMC-JTC/gbrain-1 --skill blog-ingest-amc-jtc

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Turning an entire blog or newsletter archive into structured knowledge-base pages is tedious and error-prone: feeds only show recent posts, URLs carry tracking junk, re-runs create duplicates, and paywalled or rate-limited fetches leave empty pages that pollute search results. ## Core Features & Use Cases - Feed discovery and archive walking: Finds RSS/Atom feeds via link tags, conventional paths, or sitemaps, then paginates to enumerate the full archive before fetching. - Canonical-URL dedup and idempotent re-runs: Strips tracking parameters and checks the brain before every write, so re-running an ingestion never duplicates pages. - Pacing, husk repair, and gated-post handling: Paces requests with 429 backoff, skips paywalled posts, and detects and repairs near-empty "husk" pages. - Use Case: Point it at a Substack publication to backfill every public post as a clean, normalized source page, then hand off to enrichment for entity linking. ## Quick Start Ingest this whole blog into my brain, walking its feed and writing one deduplicated source page per public post.

Frequently Asked Questions about blog-ingest

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I ingest an entire blog or newsletter archive?▼

Provide the publication URL and the skill discovers its RSS/Atom feed or sitemap, enumerates the full archive, and writes one normalized source page per public post. It deduplicates by canonical URL so re-runs never create duplicates.

How do I backfill a Substack publication into a knowledge base?▼

Substack publications are treated as ordinary feed sources: the feed lives at /feed and full-archive enumeration uses /sitemap.xml. Only public posts are ingested; gated posts are detected and skipped with a logged reason.

Does blog ingestion handle paywalled or gated posts?▼

Gated posts are detected via paywall markers or truncated preview bodies and skipped with a logged reason. The skill never uses alternate endpoints, session cookies, or credentials to reach gated content.

When should I use single-article ingestion instead of whole-publication ingestion?▼

Use single-article ingestion (idea-ingest) when the job is one URL in, one page out. Publication-scope ingestion is only for jobs that require enumerating an archive or walking a feed across many posts.

Why do re-runs of blog ingestion not create duplicate pages?▼

Every post is keyed by its canonical URL with tracking parameters stripped, and the brain is searched before each write. Existing pages are skipped, making re-runs idempotent and cheap.

What happens when a fetch returns a rate limit or empty page?▼

On HTTP 429 the skill backs off exponentially up to 30 seconds and honors Retry-After headers. Near-empty husk pages are detected after the run and either re-fetched slowly or deleted if the post turns out to be gated.

Related Skills

Explore More Content & Communication Skills→
garrytangarrytan
29.4k

blog-ingest

Ingest entire blogs, newsletters, and RSS/Atom feeds into deduplicated brain source pages.

Community
Advanced
PostergullyPostergully

signal-detector

Capture ideas and entity mentions from inbound messages into cross-linked brain pages.

Community
Advanced
zhiyuan-zhang0206zhiyuan-zhang0206
1

web-sources

Fetches web pages, RSS feeds, and YouTube content into a unified S1 item schema.

Community
Advanced
googleworkspacegoogleworkspace
30.3k

recipe-review-meet-participants

Review Google Meet conference attendance and participant session durations.

Official
Basic