Google’s Custom Search Engine May Not Return Some Of Your Pages

I will start this post by asking a couple of questions:

1. Are you using Google’s Custom Search Site or AdSense for Search for your web site?
2. Are you sure it’s returning ALL of your pages or content related to a particular search?
3. Are you absolutely sure?

Okay, that’s 3 questions actually, but nevermind the details.

Seriously, if you offer Google’s Custom Search Engine on your web site or web log, or if you are in the AdSense program and using the AdSense for Search feature, like so many bloggers do nowadays, you may want to reconsider, at least until they stop “hiding” or “ignoring” some of your content and pages.

Since 2005, I no longer offer Google’s Custom Search Engine on any of my web sites and I don’t intend to, anytime soon. Maybe a real example will help you understand how this could affect you and your web site:

If I search Google for [site:gidforums.com auto add slashes], i.e. restricting the search to only my web site, GIDForums, I expect it to return this page, but according to Google’s Custom Search Engine, that page doesn’t even exist! Look:

Notice that it’s not listed at all; not filtered, and not even stuck inside their infamous “supplemental results”.

But it’s an old page! In fact the page is very old — over 4 years old, actually — and Google indexed this page very soon after it was written, and even referred people to the page for at least a couple of years, or more. So why did they remove it? What happened?

If you look at the page closely, you’ll see that there’s absolutely nothing wrong with it, and although I am not proud of the content today, it’s still a web page that exists on my web site nevertheless, and “out there” for any decent search engine to find.

As you can imagine, I am now always asking myself, ‘How many other pages are “missing” like this?’

It’s one thing for Google to filter out pages when they serve their users on their regular search engine. That’s their ranking algorithm kicking in and it’s their prerogative what they believe to be relevant and what they want to return ultimately in their results pages, but it’s quite something else when they filter out your own pages when you are already restricting the search to just your site, and for them to then claim that it’s a site search engine!

I am fortunate because GIDForums uses a popular forum script that comes with a search engine of it’s own, and one that can manage user searches relatively well. So, if you are a blogger for example, and you are using WordPress to power your web site, please don’t replace the default WordPress site search form with Google’s - that would just be too stupid!

If you must use a third party site-search feature, I’d suggest using anything but Google, at least until they improve on the service.

Being curious, I decided to search other popular search engines like Yahoo and MSN. Here’s what Yahoo returns for my query. And what MSN returns. Both got it right, fortunately.

In case you don’t think this is such a huge problem for you, maybe you should think about this then: How many of your readers are leaving your site everyday convinced that a certain story, article or product, does not exist on your web site just because the Google Custom Search Site engine said so?