A Guide to Fixing Google Site Crawl Errors

If you’ve ever Googled your business and it hasn't shown up, it’s because your site isn’t saved, or 'indexed' by Google. For a site to be indexed and show up in the Search Engine Results Pages (or SERPs, for short), it must be able to be viewed, or 'crawled', by Google’s web spider. Spiders ‘crawl’ sites across the web and save their information, ready to show it on relevant searches.

If a webpage is not crawled, then it can't be indexed and therefore doesn't exist in Google’s eyes. It's vital to understand how to make your website easy to crawl so that your customers can find your website when using search engines.

Making your website easy to crawl

There are several steps you can take to ensure that your site is easy to crawl. For example, make sure that your webpages link to one another. These 'internal links' show the connection between the pages on your site. They help guide crawl spiders, as well as users, through your site. Additionally, this will ensure that Google never misses any pages – which can be especially relevant if you launch a new page.

Ultimately, your goal is to ensure that every link on your site leads to an actual page. That might be via a 301 redirect (we cover what this means below), but the page at the very end of that link should always return a 200 server response.

What errors should you look out for?

Google Search Console (GSC) is a free tool that can be set up easily in most website builders or quickly by your web developer. It’s owned by Google and will help all site owners keep an eye on any issues Google might have. As well as this you can manage more advanced aspects such as sitemaps, security issues and the specific search queries people have used to find you.

The coverage feature in GSC shows some errors that Google’s spider has found on your site. It’s important to pay attention to these errors as they will be stopping your site from showing in Google's search results.

If an error is noted, you can click into the report to investigate further. Before going through crawl errors that Googlebot may have encountered, it is worth explaining that there are 4 types of errors; site errors, server errors, URL errors and rich data errors – but you don’t have to worry about the last one for now. If you’d like to see a bigger, more in-depth explanation on errors then you can find that here. 

HTTP Status Codes

We all hate clicking onto a page on a website that’s broken. Well so does Google, so it’s important to fix those. Unfortunately, there are many more issues that may occur – these return different codes, called HTTP Status Codes. Google will display these in Search Console.

We've covered some of the most common error-causing Status Codes and how to fix them below:

Not Found – 404

A 404 code means that the requested page, file, or other resource cannot be found. For example, if you link to a page that no longer exists, a 404 error code will be returned.

It sounds like a tedious task, but it is worth checking to make sure the 404 page isn’t in draft mode, or worse yet, deleted. If this is the case, make sure you set up a 301 redirect to the most relevant page.

Soft 404

A soft 404 error isn’t an official response code sent to a web browser; it’s a label Google adds to a page within their index. Essentially, it is a page that returns a 200 status code when it should return a 404 response code. If the invisible HTTP header displays a 200 code even if the web page states that the page isn’t found, the page might be indexed, which is a waste of resources for Google.

This usually occurs when there is no (or a very small amount of) content on a page. The best advice for avoiding soft 404 pages is to ensure all pages have written content on them, and best practice is over 300 words.

Otherwise, you can redirect your page, which we’ll come onto next.

Moved Permanently – 301

This status code is returned when a page has been redirected to another page. This is what you should aim for when dealing with broken (404) pages that we highlighted above. The reason we use a 301 code is because, put simply, it says to Google: “we have moved this page, it no longer exists at this location – please find it at this new location instead”.

If you’re capable of redirecting the pages on your site (all Wix, Squarespace, Shopify, WordPress and most other major site builders have this as standard, otherwise ask your developer), then redirect these pages to the most relevant page possible, ensuring they’re using a 301 code.

Mobile Issues

It’s crucial for your site to avoid mobile errors, as mobile searchers have over 51% of the search market share and Google has also now moved to a mobile-first indexing model. This means it considers your site from a mobile’s perspective before thinking about desktop. So, if you’ve got issues with your mobile site they’ll be impacting the way Google views your overall website even if you’ve got a perfect desktop site.

Most sites are now responsive but you can check yours by shrinking your browser down to mobile size, or simply by visiting your site on mobile. While you’re doing this, check your load speed, as over half of all your visitors will leave your site if it doesn’t load within 3 seconds.

On load speed, it’s also very important to compress images, as large memory size imagery can take a long time to load. You can find online tools such as TinyPNG to shrink how much memory an image takes up.

Submitting Fixed Issues

Once you’ve made amends to an error, return to the coverage report, click on the issue you’ve fixed and press 'Validate Fix'. Google will then confirm that it’s recrawling the error page and will update you when this is complete.

It's also worth bearing the above advice in mind, along with SEO best practice, when creating new pages on your company website so that your content has the best possible chance of ranking in SERPs.

About the Author

Crispin Boden-Tebbutt is a Digital Marketing Specialist at Liberty Marketing. As a SEO and content marketing specialist, Crispin has several years of experience working within the digital marketing stratosphere, and has worked on a number of high-profile client accounts whilst with Liberty Marketing.