skills.rest
MattSenter
MattSenterCOMMUNITY
Explore all skills by this author →
Website𝕏@MattSenterGitHub

senternet-site-robots

Creates robots.txt files that allow crawlers and reference the sitemap.

Updated May 8, 2026
Search Tags:#indexing#next.js#sitemap#seo#robots.txt#firebase hosting#web crawlers
One-click install
npx skills add https://github.com/MattSenter/senternet-site-skills --skill senternet-site-robots-mattsenter
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: senternet-site-robots
Source: https://github.com/MattSenter/senternet-site-skills/tree/main/.claude/skills/senternet-site-robots
Command: npx skills add https://github.com/MattSenter/senternet-site-skills --skill senternet-site-robots-mattsenter

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Marketing sites need a correct robots.txt so search engine crawlers can index public pages, avoid non-public paths, and discover the sitemap, but misconfigured rules can silently block indexing. ## Core Features & Use Cases - robots.txt Generation: Creates a robots.txt allowing all crawlers with a Sitemap directive pointing at the site's sitemap.xml. - Selective Disallow Rules: Adds Disallow entries only for genuinely non-public paths such as /admin/, /api/, and /_debug/. - Next.js Route Support: Implements robots.txt as an app/robots.ts MetadataRoute on the Next.js track, avoiding conflicts with a static public/robots.txt. - Use Case: After deploying a Firebase-hosted marketing site, generate robots.txt, verify it with curl, and validate it in Google Search Console's robots.txt Tester. ## Quick Start Ask the assistant to create a robots.txt for the site that allows all crawlers and points to the sitemap at the production domain.

Frequently Asked Questions about senternet-site-robots

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a robots.txt for my website?▼

Create a robots.txt with 'User-agent: *' and 'Allow: /' to permit all crawlers, then add a Sitemap line pointing to your sitemap.xml URL. Place it in the public directory so it is served at the domain root.

How do I add robots.txt in Next.js?▼

In Next.js, create app/robots.ts exporting a MetadataRoute.Robots function that returns rules and the sitemap URL. Do not also ship public/robots.txt, since the static file would shadow the route and cause the two to diverge.

Should I block staging sites with robots.txt?▼

No. Development or staging Firebase projects should be blocked from Google indexing using a noindex meta tag in the HTML, not robots.txt. Robots.txt prevents crawling but does not guarantee pages stay out of the index.

Does robots.txt block social media crawlers?▼

No, robots.txt has no effect on social media crawlers such as Twitter, Discord, or Slack. Control how those platforms render link previews using meta tags in the page HTML instead.

Why is my robots.txt not being served correctly?▼

Verify with curl that the file returns Content-Type: text/plain at the domain root. On Next.js, check that no static public/robots.txt exists alongside app/robots.ts, and confirm NEXT_PUBLIC_BASE_URL is set so the sitemap URL uses the canonical host.

Related Skills

Explore More Software Engineering Skills→
minh1609minh1609

firebase-app-hosting-basics

Deploy and manage web apps on Firebase App Hosting with apphosting.yaml.

Community
Intermediate
nowork-studionowork-studio
3.4k

sitemap-audit

Audit XML sitemaps to detect invalid URLs, crawl budget waste, and indexing gaps.

Official
Intermediate
eprouvezeeprouveze

geo-crawlers

Inspect robots.txt, meta robots tags, and HTTP headers to map AI crawler access.

Community
Advanced
kennyolofsson23-netizenkennyolofsson23-netizen

geo-crawlers

Analyzes websites' AI-crawler accessibility via headers, robots files, and page tags.

Community
Advanced