Factors That Prevent Googlebot From Crawling Or Ranking A Website
Hi everyone,
We will look at factors that may prevent the search engine spiders from crawling the pages of a site or affect a site’s rankings on the Google SERPs. The views of the top 37 SEO practitioners are collated by Rand Fishkin of SEOmoz.
1) Server Often Inaccessible to Bots:
If a site is down for more than 48 hours, then the site can be dropped like a stone from the SERPs according to Ben Pfeiffer. If a bot cannot crawl a site and index new content, then there is no point in displaying such sites on the SERPs. Web hosts that lead Googlebot on a wild goose chase have had their sites dropped from the index. This factor that is highly disputed amongst the experts.
2) Duplicate Content:
Some experts feel that it is okay to have a minimal level of duplicate content on a domain. Even though a content filter is in place, if the page manages to escape this filter, then it gets ranked because of minimal duplication. Others strongly feel that the index will have only one unique page of content and Google will filter duplicates from the index. The site does not get penalised.
There are others who feel that it is highly undesirable to have pages on a domain that is a duplicate of pages found on other domains. Such a domain is certainly going to be penalised and ends up ranking poorly on the Google SERPs. Average agreement.
3) External Links to Low Quality or Spam Sites:
Some of the experts feel that a site linking to bad neighbourhood is itself considered to be bad. Others think that the ratio of bad external links to the overall number of external links is an important factor and a high ratio spells trouble. A few others opine that in the wake of clearing up bad external links, sites moved up into top 30 positions and then steadily gained top positions on the SERPs. Average agreement.
My thinking is based on the analogy - A person is judged by the company he keeps. It is pretty logical to think this way because as a site owner, I have full control who I link to. If I am more inclined to link to spammy sites, then my intent is not good in the first place.
4) Duplicate Meta/Title Tags Sitewide:
It is a divided house. Some experts say that duplicate title tags and meta descriptions are the first sign of content duplication that raises a filter from Google perspective. Others feel that Google looks at many other factors to find out the relevancy of a page.
Some others feel that the spiders on encountering multiple duplicate title and meta description tags stop crawling the site and this in turn affects its rankings on the SERPs. Highly disputed.
Personally I feel that the title tag and the meta description tag is like a pre-cursor to the information on a given page on a site that the bots are crawling. The spiders get a fair idea of what the page is about from a good title tag and the meta description should also reflect the content of the page in totality.
5) Keyword Stuffing/Spamming:
If a page is loaded with target keyword phrases in the title, H1 tag and content, then there is a good chance of receiving a penalty from Google for over optimisation.
It is always advisable to have the target keyword once in the title, once in the H1 tag, once towards the beginning of the body copy. After that, if the copy flows naturally and the keyword modifiers or synonyms appear naturally on the page, it is a good quality page. Other experts feel that keyword spamming could impact on the rankings of the site on Google SERPs. Average agreeement.
6) Selling Links or Participation in Link Schemes:
It is a mixed bag again. Some feel that selling links is legitimate and this can also improve rankings if the right on topic link is bought. But participating in link schemes is disastrous as Google uses statistical models in its algorithms to unearth them.
A few think that linking to a low quality neighbourhood or low quality linking scheme does not matter as the links are automatically devalued because of their poor quality. Others opine that link buying and participation in link schemes can be dangerous only if found out and can result in site getting banned. Highly disputed.
7) Slow Server Response Times:
According to some experts, this is not an issue that can affect rankings in any way. Others think that this is a critical factor when the bots come to crawl a site. They are around only for a few milliseconds and they need to have access to the content in that short period of time. One thing is for sure - Repeated server downtime is going to hurt the site from being crawled and prevents its inclusion in the Google index. Average agreement.
8) Inbound Links From Spam Sites:
Thoug Google likes to say that a site owner does not have to worry about spammy inbound links over which the site owner has no control, there have been cases where a competitor has got spammy sites linking to the original site thus affecting its rankings.
Some experts feel that the ratio of good quality inbound links from trusted sites to the bad links from spammy sites is a vital number. If a site contains good quality links from trusted sites and has a few bad links, it should not affect the site’s rankings. If the site itself links to bad neighbourhood and in turn receives lots of spammy links, it is to its detriment.
Another group thinks that a trusted well established site can get away with bad inbound links and not feel any impact on its rankings on the Google SERPs. Average agreement.
9) Low Visitor Count to The Site (measured by Toolbar, Clicks on SERPs etc)
Few experts opine that this is the sign of things to come and will carry weight in future algorithms. It is a qualitative signal in addition to other signals (say time spent on the site) that are considered in the final ranking.
Others feel that it is a non-issue and has nothing to do with rankings. There is also the view that if a site ranking at the top does not attract clicks, then either it has poor meta tags or garbage content. The search engines would transgress their fundamental goal of providing good quality relevant content to users. Average agreement.
At Netconcepts, we keep up with the latest trends in the search marketing field in both organic search engine optimisation and ppc or pay per click marketing. Give us a call on (09) 476 4601 and have a chat to us on how we can provide your business the maximum exposure in the online space.