Use Google Crawl Errors for Website Management

Use Google Crawl Errors for Website Management

If you have bigcommerce search engine ecommerce solution read my article Understanding Google Webmaster Tools you may have heard about Google crawl errors. Basically, crawl error is a condition in which Google bot fails to search and index a web page due to several reasons. It might be a URL typo, it might be a broken link, or a page that is no longer active. Almost all websites have problems with search engine crawlers. You can check how many errors you have on your website by using Google Webmaster Tools and fix them immediately.

How to Check for Crawl Errors

Some webmasters find their site errors on their Crawl Errors (CE) page in Webmaster Tools easily. It is free and pretty much straightforward so that even a first-time webmaster will be able to find issues on his or her website. Here’s how you can detect and fix some of the common issues found by webmasters by using the Google Webmaster Tools account.

1. Go to your Google Webmaster Tools account. You will find a list of menus on the left. The menus list categories such as Search Appearance, Google Index, Crawl, and Search Traffic, to name a few. Click on Crawl and then click on Crawl Errors.

2. Now you see that your workspace display Site and URL Errors. The Site Errors section shows DNS, Server Connectivity and Robots.txt Fetch data from the last 90 days. If you see green checked button on each box, then your website is doing okay.


Recommend Reading:   5 Ways to Make Money Fast on The Internet


3. Now check the URL Errors. Take a deep breath and don’t panic as some people tend to be panicked when they see a lot of error reports. The URL Errors have three categories: Web, Smartphone and Feature Phone that show where the errors come from. However, since we are discussing about website crawl error, we will click on the Web category.

4. In the Web section you will see two subsections: Server Error and Not Found, which shows errors in your server and pages that the Google bot could not find. You will find a certain number written to show you how many errors you have for each.

These are steps to navigate through Google Webmaster Tools page to specifically find the Google Crawl Errors section.

Recommend Reading: How to Increase Adsense Earnings

Fixing Page Not Found Crawl Errors

Google Crawl Errors
Sample Photo: System Status Google Crawl Error Report


If you happen to have several Not Found pages, click on it and see below a list of URLs along with their priority number, response code, and date detected. For Not Found pages, you may find 404 response codes. Consider creating a custom 404 page to help direct visitors to other pages in your site that are relevant to the page that visitors are looking for. You can also direct visitors to your web sitemap. Here are few things you can do with crawl errors.

a. Click on the URL to find out the exact problem with it. Check the error details (Last crawled, first detected), whether the URL is in your Sitemaps, and other pages that link to that URL.

b. If the URL in question is in your sitemap, you should dig deeper to know why your server is behaving the way it did.

c. If it is not, you can copy the URL and fetch as Google. Otherwise you can use the 301 redirect. You can read more about this by reading my post about Fix 404 Page Not Found Error.

By doing this, you can fix a common crawl error caused by page not found issue. Many new web owners are likely to have lots of crawl errors in their CE report. If you are one of them, relax. Think of it as a part of managing a website—just like what a building manager is supposed to do with the building he/she manages.

As you learn more about web management and put some of the theories you learn into practice, you will see lower errors in your Google Crawl Errors report and obviously have a better-optimized website.