TidyURLs
Go to Tools
BLOG
The TidyURLsJournal

Practical guides, tutorials, and insights about URLs, SEO, web development, and online marketing. Learn, troubleshoot, and get the most of the web.

Back to the Journal
TidyURLs Journal

How to Find and Fix Broken URLs Before Google Crawls Your Website

Practical insights, clear explanations, and useful ideas for understanding the web.

October 26, 202610 min read6 topics
How to Find and Fix Broken URLs Before Google Crawls Your Website
Explore the details
#Technical SEO#Broken Links#HTTP Status Codes#Google Search#Sitemaps#Website Maintenance
October 26, 202610 min read

Broken URLs can frustrate visitors, waste crawl attention, and leave search engines with outdated signals about your pages. The good news: a 404 response is not automatically a site-wide SEO problem. The goal is to find errors that matter, fix their causes, and make sure your internal links and sitemap reflect the pages you want people and search engines to reach.

This guide explains how to find broken URLs on your website, interpret common HTTP responses, and prioritize fixes before the next crawl.

Key takeaways

  • Find broken URLs by checking your site’s links, sitemap, Search Console reports, analytics, and server logs.
  • A 404 is often the correct response for a page that no longer exists; fix links to it or redirect it only when there is a relevant replacement.
  • Update internal links at their source. Redirects are useful for changed URLs, but they should not replace fixing links you control.
  • Keep sitemaps current and limited to canonical URLs that you want indexed.
  • Prioritize errors by user impact, importance of the page, traffic, links, and whether the issue affects many URLs.

What counts as a broken URL?

A broken URL is one that fails to deliver the intended page. A browser or checking tool may report an HTTP status such as 404, 410, or 5xx, but the status alone does not tell you whether the URL needs fixing.

Response or symptomWhat it usually meansTypical response
404 Not FoundThe server cannot find a page at this URL.Restore the page, fix links to it, or leave it as a 404 if it was intentionally removed and has no suitable replacement.
410 GoneThe page has been intentionally removed.Keep this response if removal is permanent and there is no relevant replacement.
5xx server errorThe server or an upstream service failed to handle the request.Investigate availability, hosting, application, or configuration problems.
Redirect loopRedirect rules send the request around in a cycle.Correct the rules so the URL reaches a final destination.
Unexpected redirectThe URL sends users somewhere other than its intended destination.Review the target and redirect configuration.
Soft 404The server returns a success response, but the page appears to be missing or empty.Return an appropriate not-found response, or provide the real page content.

A 404 is a normal part of running a website. It can be appropriate for deleted pages, mistyped URLs, and URLs that never existed. Focus on whether the error affects a useful page or a path that your site still links to—not on eliminating every 404 report.

Find broken URLs across your website

No single report shows every problem. Combine several sources so you catch broken internal links, outdated sitemap entries, and URLs visitors or other sites still use.

1. Crawl your site’s internal links

Use a site crawler to start from your homepage and follow links on pages you control. Review URLs that return errors, then identify the page containing each link. A crawler may not discover orphaned pages that have no incoming links, so pair this check with other sources.

For an individual URL, a HTTP status checker can help you inspect the response. Check the final destination as well as the first response: a URL may redirect successfully but end at a missing page. Results can vary depending on cookies, authentication, geographic location, and the checker’s network, so confirm important issues in a browser or your own monitoring environment.

2. Review Google Search Console

Inspect Search Console’s indexing and crawl-related reports for URLs Google has encountered. Treat these reports as diagnostic clues, not a complete inventory of every broken link on your site. A report can include URLs that were removed intentionally, and it may not show a problem immediately after a change.

For each reported URL, compare the reported status with the page’s intended behavior. If a page is meant to exist, investigate the server response and page availability. If it was intentionally removed, check whether internal links or sitemap entries still point to it.

3. Check the XML sitemap

A sitemap is a discovery aid, not a list of URLs that should be kept alive at all costs. Review it for URLs that now return errors, redirect elsewhere, or are no longer canonical. Keep it focused on the canonical pages you want search engines to consider for indexing, and regenerate or update it when your site changes.

A stale sitemap can keep advertising obsolete URLs. It does not make a deleted page useful, and submitting a sitemap does not guarantee that a URL will be crawled or indexed.

4. Look at analytics and server logs

Analytics can reveal whether a broken page receives visits or is a landing page for campaigns. Server logs can show requests to missing URLs, including requests that may not appear in your analytics because the page failed before analytics code ran.

Check the referring page or referrer when available. A missing URL linked from your own navigation needs a different fix from an obsolete URL requested occasionally by an external site. Logs may contain personal or sensitive data; handle and retain them according to your organization’s privacy and security practices.

5. Check important external links and campaigns

Review links in email campaigns, paid ads, social profiles, and high-value external pages. You may not control an external site, but you can often update a campaign or create a relevant redirect on your own domain. Avoid redirecting unrelated URLs simply to make a status report look clean.

How to prioritize broken URL fixes

Fixing every reported error in the order it appears is rarely the best use of time. Use impact and intent to choose what to address first.

Priority 1: Broken pages with substantial user impact

Start with URLs linked from prominent navigation, key landing pages, current campaigns, or high-traffic pages. A broken product, service, or contact page can directly interrupt a visitor’s task.

Priority 2: Internal links to pages that still exist

If the target page is available under a different URL, update the source link to point directly to the current URL. This avoids sending visitors through an unnecessary redirect and reduces the chance that a later redirect change breaks the path again.

Priority 3: URLs with a close, relevant replacement

When a page has moved or been replaced, use a permanent redirect if the new page genuinely serves the same purpose. If there is no relevant substitute, a 404 or 410 may be more accurate than sending visitors to an unrelated page or the homepage.

Priority 4: Sitemap and canonical inconsistencies

Remove URLs that no longer belong in the sitemap and correct entries that point to redirects or non-canonical versions. Make sure the sitemap reflects the pages the site actually intends to present for indexing.

Priority 5: Low-impact, obsolete requests

Some missing URLs come from typos, old bookmarks, or external links you cannot change. If they have no meaningful traffic, no internal links, and no suitable replacement, it may be reasonable to leave them as not found rather than creating a redirect.

Consider the number of affected URLs, too. A pattern that breaks an entire directory or all URLs with a particular parameter may deserve attention before isolated errors, even if the first example looks minor.

A practical workflow for fixing broken URLs

  1. Collect the URL and its source. Record where it was found: a site crawl, Search Console, sitemap, analytics, logs, or an external campaign.
  2. Check the response and final destination. Confirm whether the URL returns an error, redirects, or serves a page that looks missing. For redirects, follow the chain far enough to identify the final response.
  3. Decide what the URL should do. Determine whether the page should exist, has moved, or was intentionally removed.
  4. Choose the right correction. Restore a page that should be available; fix the internal link; add a relevant redirect for a moved page; or leave a genuine deletion as a 404 or 410.
  5. Update every source you control. Correct navigation, in-content links, campaign URLs, canonical references, and sitemap entries as appropriate.
  6. Retest the URL and its path. Verify the response and test links from the pages that refer to it. Check both desktop and mobile user journeys if the page is important.
  7. Monitor after deployment. Recheck the affected URLs and review relevant reports or logs for recurring errors. A cache, CDN, or redirect rule can make the result differ from what you saw before deployment.

Redirect loops and redirect chains

A redirect loop happens when a request cycles between URLs without reaching a page. For example, one rule might send an HTTP URL to HTTPS while another sends the HTTPS URL back to HTTP. The browser eventually stops and reports that the page cannot be reached.

Trace each redirect one hop at a time and inspect the rules that apply at each stage, including application, server, proxy, and CDN rules. Correct the conflicting rule and verify that the URL reaches a successful final destination.

A redirect chain is different: several redirects eventually lead to a page. It may still work, but it adds unnecessary hops and can make troubleshooting harder. Where possible, update links and redirect rules so a changed URL points directly to its final destination. Avoid broad rules that redirect unrelated missing pages to the homepage; the destination should be a useful match.

Common mistakes to avoid

  • Redirecting every 404 to the homepage. Visitors may not find what they expected, and the destination may not be a meaningful replacement.
  • Treating every 404 as an SEO emergency. A deliberately removed URL can correctly return 404 or 410. Fix broken paths that matter rather than trying to erase all not-found responses.
  • Leaving internal links pointed at redirects. Update links you control to the final URL instead of depending on an extra hop.
  • Keeping outdated URLs in the sitemap. Remove obsolete entries and include canonical URLs that return the intended page.
  • Confusing a server error with a missing page. A 5xx response calls for investigation of availability or server behavior, not automatically a redirect.
  • Checking only the first response. A redirect can end at another error, and a success status can still serve a misleading or empty page.
  • Assuming one checker sees every visitor’s response. Authentication, location, cookies, caching, and bot handling can affect what a request receives.

Keep Google’s view of your site consistent

Google discovers URLs through several signals, including links and sitemaps, and may revisit URLs over time. You cannot control exactly when a particular URL will be crawled. You can make your site’s signals more consistent by linking to the correct destination, maintaining an accurate sitemap, and returning an appropriate response for removed or unavailable pages.

For a page that has permanently moved, use a redirect to the relevant replacement and update internal links. For a page that no longer exists and has no substitute, a clear not-found response is usually more accurate than an unrelated redirect. If a page should be available but returns an error, fix the underlying availability problem rather than masking it with a redirect.

Conclusion

To find broken URLs on your website, combine a site crawl with Search Console, sitemap checks, analytics, and server logs. Then determine what each URL is supposed to do before changing it. Fix internal links at their source, redirect only to relevant replacements, remove obsolete sitemap entries, and leave genuine deletions with an appropriate not-found response. This keeps paths clearer for visitors and gives search engines more consistent information about your site.