The 'noindex' Day

This is the story of how one check box caused 4 weeks of heartache and suffering.

The day started well, I was moving house and hired a ute to pick up the final few things from my apartment. I’d already done most of the heavy lifting with the help of my family and a truck with a tailgate lift on another day. But today was simply a few last minute things and a day off work to complete the job. Or so I thought.

Halfway through the day, whilst moving a few items, I take a panicked phone call from one of my customers. Their site had disappeared from Google. Completely. You could find it, if you looked really hard, but it was basically gone.

Previously, about a fortnight earlier, I had performed an complete upgrade of the whole website, although it was mainly to achieve a new look and feel. The content was pretty much the same. So it was quite low risk, or so I thought.

After a frantic call to my resident SEO expert, Boyter, a few more emails, a tweet to Matt Cutts of Google and yet… still no real ideas why the site had literally fallen off the edge of the Earth.

I returned the ute to the hire place and finished what moving I could do while my mind was running at 100 mph trying to work out what went wrong.

Then I struck gold.

The SEO plugin I was using called Yoast WordPress SEO on the WordPress based site had a lot of options which control how search engines crawl the site. It also had lots of check boxes. Ones that are easy to tick or leave blank as you see fit. Now it has to be said that I have nothing against this plugin - in fact it’s very good for the price (free). The issue was, as usual, between the keyboard and the chair.

I noticed that for the content type “page” had the “noindex, follow” option ticked.

That MUST be the issue!

So I turned off this option for every type of content type I could possibly find. I didn’t want any element unindexed by search engines. I then started to try and resubmit the site back to Google, Yahoo, Bing, you name it. Trying to get the site back in the index again was going to be the next challenge.

I figured the best thing to do at this point was to visit the customer and explain the mighty boo boo and what was going to happen next.

You see, the issue is that since it took 2 weeks for the site to drop out of Google, it was going to take another 2 weeks to get back in again. That’s pretty much a month of no traffic being sourced from search engines. This is going to affect the bottom line of the business, no doubt about it. Less people finding the website = less people calling up = less new customers for my customer = less turnover. Simple as that. Thankfully, the customer in question was a real champion and despite being angry, was glad to find out what went wrong, what caused it and what I was doing to correct the situation.

So the next 2 weeks were pretty painful. I began submitting content to some of my other websites, trying to fool Google into thinking it needs to reindex the missing site sooner rather than later. You really have very little control over what the crawler does and even less control over the index mechanism.

Every 4 hours I was checking Google to see if the site had appeared again… and sure enough, after about a week it started to return, with a full return to form (ranking first for certain keyword searches) after another week.

So it was all OK in the end but a lesson learnt the hard way: check your robots / noindex settings and then check them again!

Comments (0)