SEO Case Study: The disappearing, de-indexed website

Ever had the feeling when you were a kid and something really terrible happens? You get caught cheating on a test. Girl rejects you in front of a bunch of friends. Mom walks in on you…well, you get the drift.

But you know that terrible terrible feeling inside your stomach?

Well, that is the feeling I got the other day when I stumbled upon a Webmaster Tools account for a client, who also happens to be a good friend of mine who we are helping out.

webmaster tools drop in impressions

Immediately my mind starts going crazy.

Penguin?

Panda?

Negative SEO?

Hackers?

Ahhhhhhhhh!

Our team deploys and we try to get to the root of the problem. Our best men are on the job.

Diagnosing a De-indexed Site

We checked robots.txt right away to see if there were any issues in there.

We noticed some weirdness in the robots.txt and wasn’t positive of a few things so I just deleted the entire robots.txt (backed it up of course) so I could check that out later. We could always add it back and if there are crawl issues I would just rather that have Google open access than have a whack robots.txt.

After that I checked the .htaccess file just in case some tom-foolery happened in there. Nothing odd in there, just the standard WordPress permalink rewrites so I moved on from there.

Dropped back over to Google Webmaster Tools to check the index status and things got way worse.

index status dropoff

translation: you are screwed

Overnight we basically went from 422 indexed pages to 6. The fact that there was 6 rather than 0 worried me even more because that seemed like it was some sort of algorithmic action rather than a development error of some point.

At this point we know that the loss in traffic / impressions is most likely due to the fact that Google has de-indexed a great portion of our site.

Just lovely.

So we start going through the whole site.

We stripped down the whole site to keep things simple. We also took this time to delete a ton of transient files / directories in FTP that were left over from a previous install. At least 10k files and about 900Mb of data deleted.

Let me go try and fetch a few things to see what is going on in that department.

webmaster tools crawl failure

More nonsense and now I’m really starting to get nervous. Google basically hates us now.

I didn’t need to do anymore digging at this point. I knew we had a major issue that needed to be figured out.

I started checking through plugins and the WP framework. I found a bunch of unnecessary CSS and Javascript and was able to shave off a bunch of load time. I found a duplicate Sitemap plugin which was really not good.

The sitemap looked great and there was no errors. I started checking plugins for any misconfigured settings when wouldn’t you know it…

your blog is currently blocking search engines index google

We found the problem

There was an option that was causing all of our URI’s to become deindexed.

So I navigate to these “privacy settings” which takes me to a 404. Apparently WordPress  moved these settings to Settings > Reading a while back.

A few seconds later I click my way over to this portion of the dashboard and wouldn’t you know it, the old box was ticked.

discourage search engines from indexing this site

So yea WordPress, pretty please with sugar on top discourage Google from indexing this site.

The deadly part of this setting is that it is really easy to miss if you aren’t looking for it.

Rather than WordPress inserting a line into robots.txt, it inserts a robots noindex,follow into the header site wide.

<meta name='robots' content='noindex,follow' />.

From here we unchecked this menacing check box and verify the site is no longer in noindex mode.

Things are looking up from here.

We go back to Webmaster Tools to re-fetch a few pages to see what happens.

webmaster tools fetch success

So yea, it looks like that is what it was.

Lesson(s) learned

Very small and sometimes accidental mistakes can cause massive amounts of damage.

We still have no idea how this happened. It could have been one of us who checked the box by accident, it could have been the client, or the former SEO company even perhaps.

That said, this is just another argument for making sure to change your passwords on a regular basis.

I know this might not seem like a major problem to outsiders and yes it was a really obvious fix but sometimes that is just the way it is. Sometimes the smallest errors cause the most problems.

 

Patrick Coombe
Patrick Coombe is the founder and CEO of Elite Strategies Llc. Patrick takes a hands on approach to managing Elite Strategies and loves to get involved with technical projects relating to clients inbound marketing needs.
Patrick Coombe
Patrick Coombe
  6 COMMENTS
Blog Search SEO Web Development WordPress
  • Written by: Jarrett Holmes

    Great post and details Patrick. I agree with you, SEO is definitely not easy but you have to love it to be great at it and keep up with all these changes!

    • Written by: Patrick Coombe

      Thanks Jared – good to hear from you haven’t seen you around in a while. 🙂 It is definitely not for the faint of heart!

  • Written by: Sef

    It was a very sexy investigation but only a simple mistake caused all the trouble. Thanks for sharing.

    • Written by: Patrick Coombe

      Thanks Sef you are right – all that for just one little setting 🙂

  • Written by: Antonino Bologna

    Lesson learned!
    Great post Patrick

    • Written by: Patrick Coombe

      Could not have done it without you!

  • Leave a Reply

    Your email address will not be published. Required fields are marked *

    Contact Info

    900 Linton Blvd, Suite 104
    Delray Beach, FL 33444

    Phone: 561-526-8457
    Toll Free: 855-353-8730
    E-mail: info@elite-strategies.com
    Fax: 561-526-8707