ask-extract-webpage

Extract article content and APA citations from URLs, PDFs, YouTube, and DOCX.

74|11|Updated Jul 4, 2024
One-click install
npx skills add https://github.com/OpenSourceAGI/qwksearch-research-agent --skill ask-extract-webpage-opensourceagi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ask-extract-webpage
Source: https://github.com/OpenSourceAGI/qwksearch-research-agent/tree/main/skills/ask-extract-webpage
Command: npx skills add https://github.com/OpenSourceAGI/qwksearch-research-agent --skill ask-extract-webpage-opensourceagi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires extract-pdf, extract-youtube, chat-agent-toolkit, pdfjs-serverless.

What problem does it solve? Turning arbitrary web pages, PDFs, YouTube videos, and Word documents into clean article text with accurate citation metadata is error-prone: extraction returns empty bodies, authors and dates come out wrong, and PDF dependencies leak into bundles. This Skill guides you through the extract-webpage package so you can pick the right call, diagnose failures, and fix citation or extraction issues. ## Core Features & Use Cases - URL-to-cited-article pipeline: extractContent() accepts a URL, HTML/Markdown string, DOM Document, or DOCX buffer and returns the article body plus title, author, date, source, and a formatted APA citation. - Fetching and extraction control: scrapeURL handles bot detection, proxies, and redirects; Readability and Mercury extractors can be run independently; site-specific rules come from fetchScrapingRules. - Keyphrase and summary extraction: extractSEEKTOPIC produces keyphrases and top sentences via an LLM path or an n-gram/TextRank fallback, with query-biased summarization. - Use Case: A research tool needs the body text and a properly formatted citation for any link a user pastes, including PDFs and YouTube videos, without bundling pdfjs into every consumer's build. ## Quick Start Ask the assistant to extract the article content and citation from a given URL using extractContent with a raised timeout, and inspect the returned error field if the result is empty.

Frequently Asked Questions about ask-extract-webpage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract article content and citation from a URL in JavaScript?▼

Call extractContent(url, options) from the extract-webpage package. It returns the article HTML plus title, author, date, source, and a formatted citation, or an object with an error field if extraction failed.

How do I extract text from a PDF or YouTube URL with extract-webpage?▼

Pass the URL directly to extractContent. PDF URLs are detected and routed to extract-pdf via a dynamic import, and YouTube URLs are handled by extract-youtube, which accepts a languages option for transcripts.

Why does extractContent return an error or empty article body?▼

Failures are returned as { error } rather than thrown. Common causes are the default 5-second timeout, bot detection, or a non-DOCX binary buffer; for empty bodies, inspect the raw HTML with scrapeURL and try site rules via fetchScrapingRules.

Why is the author name wrong or reversed in the citation?▼

Author detection validates names against a bundled first/last-name database, which sets author_type. Names absent from the database are treated as organizations, suppressing the APA name reversal; check author_type to diagnose.

Why does pdfjs-serverless appear in my bundle unexpectedly?▼

A static PDF import was added to the package's root entry. The PDF path must remain a dynamic import inside extractContent so pdfjs-serverless is never evaluated at build time.

How do I bias SEEKTOPIC keyphrases or summaries toward a query?▼

Pass heavyWeightQuery with your query text to extractSEEKTOPIC, and set optionSkipRanking to false to also receive topSentences. Without a phrasesModel, the n-gram fallback path has no phrase statistics to score against.