How to Find and Fix Broken URLs Before Google Crawls Your Website

Broken URLs can frustrate visitors, interrupt navigation, and make it harder for search engines to discover the pages you want indexed. The challenge is not simply finding every 404: some missing pages are intentional, while others are linked from important pages or listed in a sitemap and need attention. This guide explains how to find broken URLs, decide which ones matter, and fix them before they become a persistent crawl and user experience problem.

Key takeaways

• Crawl your site and check important URLs before publishing major changes, then repeat checks regularly.
• A 404 response is not automatically a site-wide SEO problem. Prioritize URLs with useful backlinks, traffic, internal links, or sitemap entries.
• Replace broken internal links with the correct destination. Redirect a removed URL only when a relevant replacement exists.
• Keep sitemaps current and limited to canonical URLs you want search engines to consider for indexing.
• Check redirect destinations for loops, chains, and errors; a redirect is not fixed until it reaches the intended page.

What counts as a broken URL?

A URL is broken when it does not lead to the resource a visitor or crawler is meant to reach. The most visible cases are HTTP errors such as 404 Not Found and 410 Gone, but a URL can also fail because of a server error, a redirect loop, a redirect to the wrong page, or a page that loads but displays an error message.

A 404 response means the server could not find a resource at that URL. It can be the correct response when a page was intentionally removed and has no suitable replacement. Search engines may eventually remove a URL that consistently returns 404 or 410 from their results, but a missing page does not automatically mean your whole website has an SEO penalty.

A soft 404 is different: the server returns a successful status such as 200, but the page tells visitors that the content is missing or provides no meaningful content. Check both the HTTP status and what the page actually displays.

Find broken URLs across your website

Start with a crawl of your own site. A crawler can follow internal links and report URLs that return errors, redirect unexpectedly, or cannot be reached. Review the scope and settings: a crawl may not find orphaned pages that have no links pointing to them, and it may not represent what Google has already discovered.

Use several sources to build a fuller list:

1. Crawl internal links. Look for links to 4xx or 5xx responses, redirecting URLs, and pages with no clear destination.
2. Review server logs or analytics. These can help identify requests to missing URLs and pages people still try to visit. Treat each tool's data as partial; tracking may be blocked, and logs only cover recorded requests.
3. Check sitemap URLs. Compare the URLs in your XML sitemap with pages that currently load and are intended to be indexed.
4. Review search engine reports. Search Console reports can help you inspect URLs Google has encountered, but they are not a complete live crawl of every URL on your site.
5. Check important URLs individually. Use an HTTP status checker to inspect a URL's response and redirects. For larger sites, combine individual checks with a crawl or automated monitoring.

When checking a URL, record its response status, final destination, source link, and whether the page should exist. A simple list or spreadsheet makes it easier to assign and verify fixes.

Prioritize fixes that affect visitors and discovery

Do not spend equal time on every URL. A useful prioritization order is:

1. Broken pages linked from important site pages. Fix links on navigation, product pages, popular articles, and other high-value pages first.
2. URLs listed in your sitemap. A sitemap should describe URLs you want search engines to discover and consider for indexing. Remove URLs that are intentionally gone, and investigate errors for URLs that should still work.
3. Missing pages with a relevant replacement. Visitors and external links may still point to the old address; a suitable redirect can preserve a useful path to the replacement.
4. URLs receiving visits or valuable external links. Check referral data and backlinks where available before deciding whether to redirect, restore, or leave a page gone.
5. Low-value or intentionally removed URLs. If there is no relevant replacement and no meaningful user path to preserve, returning 404 or 410 can be appropriate.

A URL's presence in a report is a prompt to investigate, not proof that it needs a redirect. Avoid redirecting every missing page to the homepage: that can confuse visitors and may be treated as a soft 404 when the destination is unrelated.

Fix broken links, redirects, and stale sitemaps

Repair internal links at their source

If a page on your site links to a broken URL, update the link to the correct live destination or remove it if the link is no longer useful. This is usually better than leaving a bad link in place and relying on a redirect to catch it. Check navigation, related-content modules, templates, and older articles, not just the page where you first noticed the issue.

Use ordinary, descriptive links that point directly to the intended page. Google Search Central recommends making links crawlable, including using an anchor element with an href attribute. A link that works only after a user action or through script may not be discovered in the same way as a standard link.

Choose the right response for removed content

If the original page still belongs on the site, restore it or fix the server error. If it moved permanently and a closely related replacement exists, use a permanent redirect to that page. If it is gone with no suitable replacement, leave it as 404 or 410 rather than sending visitors somewhere unrelated.

After changing redirects, test both the old URL and the final destination. Look for loops, unnecessary chains, incorrect protocol or hostname changes, and destinations that themselves return an error. Update internal links to the final URL even when redirects are working.

Keep the sitemap accurate

Remove URLs that are permanently gone, redirect elsewhere, or are not the preferred canonical version. Include the current canonical URLs for pages you want search engines to consider. A sitemap helps communicate URLs; it does not guarantee crawling or indexing, and it does not replace sound internal linking.

When a page is temporarily unavailable, investigate the cause rather than deleting it from the sitemap automatically. For a permanent removal, update the sitemap as part of the same cleanup that removes or redirects the page.

A practical broken-URL repair workflow

1. Collect candidates. Export crawl errors, sitemap URLs, Search Console findings, and relevant server or analytics data.
2. Normalize and deduplicate. Compare URL variants carefully, including differences in host, protocol, path, query parameters, and trailing slashes. Do not assume every variant should resolve to the same page.
3. Verify the response. Check the HTTP status and follow the redirect path. Recheck intermittent errors; a single failed request may not describe the usual behavior.
4. Classify the URL. Mark it as a page to restore, a page to redirect, an intentionally removed resource, or an issue to investigate further.
5. Fix the source. Update internal links, redirect rules, page content, or the sitemap as appropriate.
6. Retest. Confirm the old URL behaves as intended, the destination loads, and internal links point directly to the correct location.
7. Monitor. Repeat checks after redesigns, migrations, URL changes, and significant publishing updates. Watch for recurring patterns such as broken template links or faulty redirect rules.

For a large site, automate checks where practical, but review the results before applying bulk redirects. URL patterns can look similar while representing different content, and an automated rule can send valid pages to the wrong destination.

Common mistakes and troubleshooting

Redirecting every 404 to the homepage

This hides the original problem without giving visitors a relevant replacement. Redirect only when the destination genuinely serves the same or closely related purpose. Otherwise, a clear 404 page with helpful navigation is more honest and useful.

Treating every 404 as urgent

Some 404s are expected, including mistyped URLs and resources intentionally removed. Prioritize based on internal links, sitemap inclusion, traffic, backlinks, and user impact rather than status code alone.

Leaving redirect chains in place

A chain may still reach a working page, but it adds unnecessary steps and can make troubleshooting harder. Update links to the final URL and simplify redirect rules when safe. Always test after editing rules, since changes can create loops.

Removing a broken URL from the sitemap but ignoring internal links

A sitemap cleanup does not repair links elsewhere on your site. Find and update the pages that point to the URL, or users and crawlers can continue encountering it.

Assuming a URL checker proves the page is healthy

A checker reports what happened during its request. It may not reveal intermittent server failures, content problems visible only after loading, or URLs it was never asked to check. Combine targeted checks with crawling and ongoing monitoring.

Conclusion

Finding broken URLs is a process of discovery, verification, and prioritization—not a mandate to eliminate every 404. Repair links that send visitors to errors, use relevant redirects for moved content, preserve accurate responses for intentionally removed pages, and keep your sitemap aligned with current canonical URLs. Recheck after site changes so problems are found while they are still easy to trace.