site-architecture

Add explicit User-agent blocks for GPTBot, ClaudeBot, and PerplexityBot with Allow: / rules, while disallowing private paths like /api/ and /admin/. Include a Sitemap directive pointing to your sitemap.xml so crawlers can find all indexable pages.

Create an app/sitemap.ts file exporting a default function that returns a MetadataRoute.Sitemap array. Combine static routes with dynamic entries fetched from your data source, setting url, lastModified, changeFrequency, and priority for each page.

Use Open Graph tags (og:title, og:description, og:image at 1200x630, og:url, og:type) for Facebook and LinkedIn, plus Twitter Card tags (twitter:card as summary_large_image, twitter:title, twitter:image). In Next.js, define these through the Metadata API in layout.tsx or page-level generateMetadata.

Yes. Allow GPTBot and ClaudeBot for chat-based browsing while disallowing CCBot and Google-Extended, which are used for training datasets. This lets AI assistants cite your content without contributing to model training.

Target LCP at or below 2.5 seconds, INP at or below 200 milliseconds, and CLS at or below 0.1. Achieve these by optimizing images with next/image, preloading critical assets, splitting JavaScript bundles, and reserving space for dynamic content.

Canonical tags prevent duplicate content penalties when the same page is reachable via URL parameters, HTTP/HTTPS, or www/non-www variants. In Next.js, set alternates.canonical in the metadata object so each page declares its preferred URL.