Caffeine - New Generation Google Search Infrastructure

Hi everyone,
Caffeine is the latest offering from Google which has to do with the next generation architecture for web search. Crawling large sections of the web and indexing the pages of sites is most vital for a good web search engine.

The next step would be to determine the reputation of the indexed pages and finally rank and return the most relevant pages in response to users’ search queries. Google spends a lot on resources (hardware) in carrying out crawling and indexing operations. It is quite expensive and this explains the reason why there are only 3 major search engines capapble of doing this and providing these results to secondary sites for a fee.

According to Google, a large team of developers in Googleplex have been working for several months rewriting the code to evolve the current search infrastructure into a new high tech next generation search machine that pushes the limits on dimensions like size, indexing speed, accuracy, comprehensiveness to name a few.

This under the hood technology will not appear different to most users in terms of the search experience or the results outputted. But power users and developers can notice changes. To get their valuable feedback on Caffeine, Google has opened a web developers preview to collect valuable feedback at  http://www2.sandbox.google.com/

According to the post by Vanessa Fox on the new Caffeine search index, Matt Cutts mentioned that the major difference would be the way in which Google indexes the web. This could be achieved by super efficient code driving crawling efforts vigorously both breadth and depth wide and indexing quality pages beyond the purview of the PageRank spectrum.

My pet peeve would be that currently, well established industry sites enjoy so much domain authority and trust that even for new developments in their respective industries, they dominate the top pages of the SERPs even though they do not necessarily take the effort to update their content on a regular basis. This does not refer to news sites which are hugely popular both in the online search space including social media.

The downside is that newer sites that work hard to deliver quality content to their users still do not get much benefit as they rank temporarily for a few hours or days for hot industry related developments (based on the QDF or Query Deserves Freshness algorithm) and then subside to lower rankings after a few days. This excessive importance to the domain authority and trust enjoyed by established sites can sometimes come in the way of providing relevant fresh search results.

The Google Webmasterscentral blog gives a good account of  Caffeine, Google’s new search infrastructure setup and so does Matt Cutt’s blog.

Netconcepts is a leading Auckland seo service provider offering organic seo and ppc services to its clients in New Zealand and Australia.

This entry was posted on Thursday, August 13th, 2009 at 10:28 am and is filed under Blogging, Consultants, Natural Search, Ravi Venkatesan, SEO-Search Engine Optimisation. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.