In November 21, 2011, Yahoo! closed down Yahoo! Site Explorer (YSE), a service loved by many webmasters and SEOs. It has been a year since YSE went away and the SEO world is still missing YSE.
YSE was a portal where webmasters could interact with the Yahoo! search engine. Webmasters could submit sites and sitemaps for indexing. YSE also allowed webmasters to receive reports on backlinks and crawler errors. The backlinks feature was by far the most important for Yahoo! gave the most accurate backlink data for any website compared to Bing or Google. Yahoo’s backlink reports were timely (the reports were updated almost as soon as the crawler found new links), full (it showed all links to a site, not a sample), and free. Finding a replacement to YSE is hard for nothing comes close to YSE.
Backlink reports are needed by webmasters and SEOs for link analysis and reverse engineering rankings. Link analysis involves determining how a competitor’s backlink profile attributed to their rankings. This allows the webmaster to acquire similar backlinks in an attempt to acquire similar rankings as their competitor.
The analogy that is used within the SEO community is that of a poker game. The rules of the game (the algorithm) are hidden but the cards (the backlinks) are shown and the winners are known (the rankings). This allows webmasters to infer the mechanics of Google’s algorithm. Once the mechanics are known, the algorithm can be “gamed.”
For that very reason, Google has masked their backlink reports. Google allows users to view links pointing to a site with the link colon command (example link:www.google.com) but this only shows a small subsample of all backlinks. Matt Cutts, head of Google’s webspam team, stated that Google does not want competitors to reverse engineer rankings (http://www.youtube.com/watch?v=TjQ2QVQpaK0&feature=player_embedded#). Users can get an accurate backlink report with Google Webmaster Tools (GWT) but this is limited to only sites within a webmaster’s control. Comparing the results from the link colon command with the results from GWT shows that link colon only reports a small fraction of total backlinks.
Bing Webmaster Tool, the replacement for Yahoo! Site Explorer, allows users to view backlinks to one’s own site similar to GWT. For sites that are not under a webmaster’s control, the webmaster can view the site’s backlinks under Dashboard > Diagnostics & Tools > Link Explorer. However, similar to Google’s link colon command, Bing’s Link Explorer does not show all links pointing to a site.
Using Bing’s Link Explorer for google.com reports only 720 websites pointing to google.com. We know this is wrong for Google’s link colon operator reports 14,000 backlinks and, knowing that link colon only reports a sample, we know that total links pointing to google.com is well over 14,000.
Yahoo! Site Explorer was closed because Yahoo! partnered with Bing for search. As Yahoo!’s share of search gets taken over by Google, the need for Yahoo! as a search engine decreases (http://searchenginewatch.com/article/2184889/Google-U.S.-Search-Share-Hits-Record-High-as-Yahoo-Falls-to-New-Low). Yahoo! has other unique selling propositions such as Y! mail, games, chat, and content. With those factors in mind, Yahoo! partnered with Bing to provide search for Yahoo.com. This made having two webmaster tools (Bing Webmaster Tools and YSE) redundant, therefore YSE was closed.
As we discussed earlier, Bing and Google’s backlink reports were subpar compared to YSE. This increases the need for backlink reporting services. The top ones to date are Open Site Explorer, ahrefs, MajesticSEO, and SEO Spyglass.
Open Site Explorer (OSE) is a backlinks search engine from SEOmoz. The service currently has 82 billion pages indexed. Unlike YSE, OSE’s index is only periodically refreshed and, at the time of this writing, the last refresh was 3 weeks ago. This is much more slow than YSE. Another limitation to OSE is that the free trial is limited to a few queries per day and the free trial’s data are limited in scope versus the paid edition.
OSE has some features that YSE did not have. Social shares such as tweets, +1, and likes are reported. OSE also reports domain authority and page authority, how likely a site/page is to rank in the search engine results page. This metric is similar to Google’s Page Rank.
ahrefs is another SaaS similar to OSE. The difference between OSE and ahrefs is that ahref’s index is updated much more regularly. As of this writing, the latest refresh occurred 5 minutes ago. Currently ahrefs has 99.9 billion pages indexed. ahrefs Rank is a similar algorithm to OSE’s domain authority and Google’s Page Rank. ahrefs offer free trials but trials are limited like to OSE.
Another SaaS, MajesticSEO crawls the web like ahrefs or OSE and reports backlink profiles to users. As of this writing, MajesticSEO’s last index refresh was 53 minutes ago and has 53 billion pages cralwed on the fresh index and 392 billion pages crawled on the historic index. MajesticSEO also have Trust Rank signals that are similar to Page Rank. Majestic offers a free trial similar to the other SaaS tools but the trial is limited once again.
SEO Spyglass is a software developed by link-assistant. It is a one time fee software where users can view backlink data for any site. The backlink data is acquired from a variety of services and SEO Spyglass allows users to acquire additional data such as anchor text, page title, total out-bound-links, and other metrics.
Compared with ahrefs, OSE, or MajesticSEO, SEO Spyglass’s backlink reports are limited in size. While a query on OSE or ahrefs will bring up a few thousand backlinks, SEO Spyglass will only show a few hundred. SEO Spyglass also requires users to type in captchas when needed. This makes SEO Spyglass more user intensive compared to the SaaS solutions.
It is interesting to note that, with the fall of YSE, ahrefs seemed to be the biggest winner, gaining the most market share. MajesticSEO was the second biggest winner, doubled in popularity. OSE’s trend report is not an accurate reflection of OSE’s popularity for SEOmoz does a lot of publicity that will bias the graph.
YSE offered a deep and extensive backlink report that was very helpful to webmasters. The best part was that it was free. Replacement services are not as useful as YSE and have their own limitations and drawbacks such as refresh frequency, accuracy, and index size.
Webmasters must note that any index of the web is a sample of the web and, therefore, not a perfect reflection of the web. Backlink reporting tools take samples of the web and their sample will not perfectly match Google’s or Bing’s sample of the web. (see http://en.wikipedia.org/wiki/Student’s_t-test).
On top of that, even as statistical power increases, backlink reports are different than what search engines webmaster tool’s report for search engines filter out low quality pages. These include spam pages, orphan pages, hidden links, and other webpages that fail to meet quality guidelines. For what one considers a webpage can differ, backlink reports will differ.
Search engines, and most tech companies, are employing machine learning to enhance their algorithms. As machine learning improves, spam filtering abilities will also improve. This will force backlink reporting service to either replicate such machine learning or find other ways to adapt. For that reason, the highest and most accurate backlink report one can obtain is from a search engine itself and not a backlink reporting service. This makes the hurt of losing YSE even more great.