How to Create a Sitemap and robots.txt for a Next.js Website

A sitemap helps search engines discover the URLs you want them to consider, while robots.txt communicates which parts of your site crawlers may request. Neither file makes a page eligible for search results on its own, and neither replaces access controls. This tutorial shows how to create both files in a Next.js App Router project, connect them to your canonical URL strategy, and check the deployed result.

Key takeaways

• In the Next.js App Router, app/sitemap.ts and app/robots.ts can generate the corresponding files.
• Use absolute, canonical URLs in your sitemap and include only pages that are intended to be indexable.
• robots.txt manages crawler access; it does not guarantee deindexing or protect private content.
• Verify the files and page responses on the deployed site, not only in local development.

How sitemap and robots.txt work together

A sitemap is a list of URLs you want search engines to discover. It is a discovery hint, not a command to index every listed page. Search engines decide whether and how to index pages using multiple signals, including whether they can crawl a page and what its content and canonical signals say.

A robots.txt file is a set of crawler rules published at the root of a host, such as https://www.example.com/robots.txt. It can allow or disallow crawling of URL paths for compliant crawlers. It does not grant or deny access to people, and it does not ensure that a blocked URL will disappear from search results.

These files have complementary roles: the sitemap points crawlers toward preferred public URLs, while robots rules describe paths crawlers may request. Avoid listing a page in the sitemap while also disallowing its path to the crawlers you want to reach it.

Create a sitemap in the Next.js App Router

For an App Router project, create app/sitemap.ts. Next.js recognizes this file convention and serves its output as /sitemap.xml.

import type { MetadataRoute } from 'next'

const siteUrl = 'https://www.example.com'

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: siteUrl,
      lastModified: new Date('2025-01-15'),
    },
    {
      url: `${siteUrl}/about`,
      lastModified: new Date('2025-01-10'),
    },
    {
      url: `${siteUrl}/articles/nextjs-routing`,
      lastModified: new Date('2025-01-12'),
    },
  ]
}

Replace the example domain and dates with your real production domain and content data. The dates above are illustrative. Set lastModified to a trustworthy date when the page was meaningfully updated; do not change it on every build just to make a URL look fresh. If you cannot provide a reliable modification date, omit the field.

Generate entries from content data

For a site with regularly updated pages, create sitemap entries from the same data source used to render those pages. For example, if your project has a database or a content-management system, fetch the published records and map them to absolute URLs. Include only public pages that should be discoverable. Handle failures and caching according to your data source and deployment environment; the small example above does not implement those concerns.

A sitemap has protocol limits: an individual sitemap can contain up to 50,000 URLs and must not exceed 50 MB uncompressed. Larger sites need multiple sitemap files and a sitemap index. Check the current search-engine and Next.js documentation before designing a large or highly dynamic sitemap.

Set canonical URLs and decide which pages belong

A sitemap URL should agree with the canonical URL you want search engines to associate with the page. Use a consistent scheme and host—for example, HTTPS and either the www or non-www version—and avoid putting redirecting URLs, alternate tracking-parameter URLs, or duplicate variants in the sitemap.

In the App Router, you can configure a base URL for metadata and provide a canonical URL for a page:

// app/layout.tsx
import type { Metadata } from 'next'

export const metadata: Metadata = {
  metadataBase: new URL('https://www.example.com'),
}

// app/about/page.tsx
import type { Metadata } from 'next'

export const metadata: Metadata = {
  alternates: {
    canonical: '/about',
  },
}

With metadataBase, Next.js can resolve relative metadata URLs against the configured base. Make sure that base is the public production origin in the deployed environment. For pages with dynamic or parameterized URLs, generate canonical values from the same validated content data used to render the page.

A canonical is a signal about a page's preferred URL, not a way to make a page private or a guarantee that a search engine will select that URL. Keep canonical tags, internal links, redirects, and sitemap entries consistent. If you want a page excluded from search results, use an appropriate noindex directive that crawlers can access, and remove the page from the sitemap. Do not block crawling in robots.txt if the crawler needs to fetch the page to see its noindex instruction.

Create robots.txt in Next.js

Create app/robots.ts to generate /robots.txt:

import type { MetadataRoute } from 'next'

const siteUrl = 'https://www.example.com'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/admin/', '/internal-search/'],
    },
    sitemap: `${siteUrl}/sitemap.xml`,
  }
}

This example allows crawling generally and disallows two example paths. Replace them with routes that should not be crawled, if any. A disallow rule is not an access-control mechanism: someone can still request a URL, and a crawler may know about a blocked URL from links or other sources. Do not put secrets or sensitive information in public routes on the assumption that robots rules will hide them.

If you need to prevent public indexing of a page, use a noindex meta directive or an appropriate X-Robots-Tag HTTP header, and ensure search crawlers are allowed to fetch the response containing that directive. For private content, require authentication and enforce authorization on the server. A robots rule cannot secure it.

Pages Router projects

The app/sitemap.ts and app/robots.ts conventions apply to the App Router. For a Pages Router project, you can publish static files from public/sitemap.xml and public/robots.txt, or build a generation process that writes them into the public output. Ensure your deployment actually includes generated files and that they are served from the site root.

Practical workflow: generate, deploy, and verify

1. Choose the production origin. Decide the canonical HTTPS host and use it consistently in metadata, sitemap URLs, redirects, and internal links.
2. List eligible pages. Include useful, public, canonical pages that should be discoverable. Exclude private pages, redirects, error pages, duplicate variants, and pages marked noindex.
3. Add the App Router files. Create app/sitemap.ts and app/robots.ts, or use public static files for a Pages Router project.
4. Build and run the site. Check for TypeScript or build errors, then inspect the generated endpoints in the running application.
5. Check the deployed responses. Open https://www.example.com/sitemap.xml and https://www.example.com/robots.txt in a browser or request them with a command-line HTTP client. Confirm they return the expected content and are not redirected to an unintended host or blocked by deployment rules.
6. Inspect representative pages. Check their status codes, rendered canonical tags, and any noindex directives. Compare canonical URLs with sitemap entries. A URL parser or URL tools can help inspect individual URL components, but it cannot confirm that a page is indexable or replace checking the live response.
7. Review after changes. Recheck the files when changing domains, routing, content systems, deployment configuration, or indexability rules.

Common mistakes and troubleshooting

Deployment and maintenance recommendations

• Serve both files from the correct hostname and over HTTPS, with predictable successful responses.
• Keep sitemap entries aligned with page canonicals, redirects, and the site's preferred host.
• Use accurate lastModified values and omit unreliable dates rather than refreshing them mechanically.
• Keep sensitive content protected by authentication and authorization, independent of crawler rules.
• Test production and preview environments separately. Do not accidentally publish a preview host as the canonical origin or submit its URLs as production URLs.
• Treat sitemap submission in search-engine tools as a discovery aid, not a guarantee of crawling, indexing, or ranking.

Conclusion

Next.js makes it straightforward to generate sitemap.xml and robots.txt with App Router metadata files. The important work is choosing the right URLs and keeping them consistent with canonical metadata, page indexability, and deployment behavior. Use robots rules for crawler guidance—not privacy—and verify the live files and page responses after deployment.

References

• Next.js: sitemap.xml file convention
• Next.js: robots.txt file convention
• Next.js: Metadata and canonical URLs
• Google Search Central: Build and submit a sitemap
• Google Search Central: Introduction to robots.txt
• Google Search Central: Consolidate duplicate URLs
• Google Search Central: Block search indexing with noindex