Find and Fix Broken URLs Before Google Crawls

Broken URLs can frustrate visitors, waste crawl attention, and leave search engines with outdated routes to process. Finding them early helps you repair valuable links, remove obsolete URLs from sitemaps, and make sure redirects lead to the right pages. This guide explains how to identify broken URLs, confirm what is actually wrong, and prioritize fixes—while keeping in mind that you cannot control exactly when Google crawls your site.

Key takeaways

A 404 response is not automatically an SEO problem: it can be appropriate when a page was intentionally removed and has no relevant replacement.
Fix broken internal links at their source instead of relying on redirects to compensate for them.
Redirect removed pages only when there is a genuinely relevant destination, and check that the redirect completes without a loop or chain.
Keep your sitemap limited to canonical, indexable URLs that return successful responses.
Prioritize URLs with strong internal links, external links, traffic, or business value.

What counts as a broken URL?

A URL is broken when it no longer leads to the intended, usable page. The problem can be a missing page, a faulty redirect, an incorrect link, or a server error. The HTTP status code is a useful clue, but it does not tell the whole story: a URL returning 200 OK can still show a missing-page message, while a 404 may be the correct response for content that has been permanently removed.

Common cases include:

404 Not Found: The server cannot find the requested resource. This is expected for a page that no longer exists, but it needs attention if the URL is still linked internally or listed in a sitemap.
410 Gone: The resource has been intentionally and permanently removed. Use it when that accurately describes the page’s status; do not use it as a substitute for finding a relevant replacement.
5xx server errors: The server failed to fulfill the request. These may be temporary, so investigate recurring errors before treating a page as permanently gone.
Redirect loop: A URL redirects back to itself or through a sequence that eventually returns to an earlier URL. The browser and search crawler cannot reach a final destination.
Soft 404: The response code suggests success, but the page communicates that the content is missing or provides little useful content. Check both the response and what the page displays.

Google may eventually stop indexing URLs that consistently return 404 or 410, but that does not mean every such URL requires a redirect. The right response depends on whether the content still exists, has a suitable replacement, or has been intentionally retired.

How to find broken URLs

Use more than one source. A single scan can miss pages that are not linked from your current navigation, while a crawl report can reveal old URLs that are still being requested.

1. Crawl your own site

Use a site crawler you trust to follow internal links and record response codes. Start from the homepage and important landing pages, then check that the crawler can access the sections you want audited. Export the results so you can group errors by status code and source page.

Review at least:

URLs returning 404, 410, or 5xx responses.
Redirects that fail, loop, or pass through several unnecessary hops.
The pages that link to each affected URL.
Important pages that are not reachable through internal links.

A crawler only sees what it can reach and what its configuration allows. It may miss URLs known only from external links, old campaigns, or prior versions of your site.

2. Check Google Search Console reports

Search Console can help identify URLs Google has encountered and report indexing or crawling issues. Treat those reports as investigation leads, not as a complete list of every broken link on your site. Confirm each URL on your server and check whether the page is meant to exist.

Look for patterns: a group of errors may point to a changed folder structure, a faulty template, a deployment issue, or an obsolete sitemap. If errors began after a site migration or release, compare the affected paths with the changes made at that time.

3. Review analytics and server logs

Analytics can show whether a failing URL has received visits or was a useful landing page. Server logs can reveal requests that a crawler or analytics tool does not capture, including visits to old URLs from external sites. Access logs may contain private or sensitive data, so handle them according to your organization’s retention and access policies.

4. Check individual URLs and redirect behavior

For a short list, a HTTP status checker can help inspect the response for a URL. Test important URLs individually and confirm the final destination, not just the first response. Results can vary depending on authentication, cookies, request method, geographic routing, or temporary server conditions.

For a quick command-line check, curl can show response headers and follow redirects:

curl -sS -L -D - -o /dev/null --max-redirs 10 https://www.example.com/old-page

Replace the example URL with a site you own or are authorized to test. The command displays response headers from the redirect path and stops after the configured redirect limit. A HEAD request can be useful for quick checks, but some servers handle HEAD differently from a normal page request; test with a standard GET when the result is unexpected.

Fix the cause, not just the status code

Repair internal links

If a page still exists at a new URL, update links on your site to point directly to the current destination. Fix links in navigation, footers, related-content modules, old posts, product pages, and reusable templates. This prevents visitors from taking an avoidable detour and makes your internal linking clearer.

If the URL is a typo, correct the link where it was introduced. Adding a redirect for every misspelled link can hide the underlying problem and create unnecessary maintenance.

Choose an appropriate response for removed pages

Use a permanent redirect when a page has moved or there is a genuinely equivalent replacement. A redirect to a closely related page can preserve a useful path for visitors. Do not redirect every removed URL to the homepage: a destination that does not match the old content can confuse visitors and may be treated as a soft 404.

If the content is gone and there is no relevant replacement, let the URL return an appropriate 404 or 410 response. Keep a helpful not-found page for visitors, but make sure the missing URL does not return a misleading success status.

Remove redirect loops and unnecessary chains

For each redirect, follow the entire path to a final response. Correct rules so the URL reaches its destination in a finite number of steps. Where possible, update internal links to point to the final URL instead of routing through an old address. Test both HTTP and HTTPS, and both www and non-www hostnames if your site serves or redirects between them.

Clean up stale sitemap entries

A sitemap helps communicate which URLs you want search engines to consider; it is not a command to index every listed page. Remove URLs that no longer exist, redirect elsewhere, are non-canonical, or should not be indexed. Include the preferred canonical URL for each page and ensure it returns a successful response.

After fixing a URL, check the sitemap that actually gets served—not only the file in your project. A generated sitemap can remain stale because of caching, a build process, or a plugin configuration. If your site has multiple sitemap files or a sitemap index, check the affected file and its index as well. Update lastmod only when the page has had a meaningful change; it should not be refreshed automatically just to encourage another crawl.

Submitting or updating a sitemap can help Google discover your preferred URLs, but it does not guarantee immediate crawling or indexing. Keep internal links and server behavior consistent with the sitemap.

Prioritize fixes by impact

Fixes are easier to manage when you rank them instead of treating every error as equally urgent. A practical order is:

Restore pages affected by a broader server or deployment problem. A cluster of 5xx errors or a redirect failure affecting many URLs may have a larger impact than an isolated missing page.
Repair broken links on important pages. Prioritize links from navigation, high-traffic pages, conversion paths, and other prominent internal locations.
Review URLs with external links or meaningful visits. If a useful old page has a close replacement, redirect it carefully. If it has no relevant successor, avoid sending visitors to an unrelated destination.
Remove invalid sitemap entries. Resolve URLs that should exist; otherwise remove them from the sitemap.
Address low-value, intentionally removed URLs. Confirm they have no relevant replacement and that internal links no longer point to them.

A simple tracking sheet can include the affected URL, response code, linking page, intended destination, action, owner, and verification date. This makes it easier to confirm that fixes are complete and to spot repeated errors caused by templates or publishing workflows.

Common mistakes and troubleshooting

Redirecting every 404 to the homepage: Redirect only when the destination is a meaningful replacement. Otherwise, return an honest not-found response.
Assuming every 404 is urgent: First determine whether the URL should exist, whether it is linked internally, and whether it has useful traffic or external links.
Checking only the first response: A 301 or 302 is not proof that a redirect works. Follow it and inspect the final destination for loops, errors, and relevance.
Leaving old URLs in the sitemap: A sitemap containing removed or redirected URLs sends mixed signals. Review generated output after making changes.
Trusting a successful status without viewing the page: A 200 OK response can still be an error page or an empty page. Inspect the content as well as the code.
Using a scanner as the only source: Crawlers, Search Console, analytics, and logs each provide different evidence. Check important URLs directly and investigate conflicting results.
Expecting a crawl on a specific schedule: You can improve discoverability and remove conflicting signals, but you cannot guarantee when Google will revisit or index a URL.

A repeatable URL cleanup workflow

Export broken URLs from a site crawl and gather additional candidates from Search Console, analytics, or logs.
Check the response and page content for each important URL.
Identify the cause: missing page, bad internal link, redirect problem, server error, or stale sitemap entry.
Decide whether to restore the page, update the internal link, create a relevant redirect, or leave an intentional 404 or 410.
Update the sitemap and any templates or content that still reference obsolete URLs.
Recheck the affected URLs and their final destinations after changes are deployed.
Repeat the crawl periodically and after migrations, redesigns, or significant publishing changes.

Conclusion

Finding broken URLs is a process of checking both the response and the reason behind it. Repair internal links, use redirects only for relevant replacements, keep sitemaps current, and prioritize issues that affect valuable pages or many URLs. These steps make your site easier to navigate and give search engines clearer information, even though they cannot control exactly when Google crawls it.

How to Find and Fix Broken URLs Before Google Crawls Your Website