How Google Builds Its Index

We all want top search engine rankings and in order to achieve it we try really hard. We keep on optimizing our website for the related keywords but do we know how Google build its index? How it present the results? This article will talk about it.

I am not an employee of Google or its affiliated ones so don't expect me to give you all the secrets of its algorithm but my experience with SEO has taught me few things which I am always happy to share with you. In this article I will tell you something about Google's index. First of all let me make it very clear that Google never searches your query in the real time. Instead, it has a huge database of websites which it has developed by sending its robots to collect and when you enter the query then Google presents you with the results which are in its database.

Now the question is why Google require an index? The answer is quite simple but is itself a question. Why does a book require an index? I think now you got the answer. So let us proceed further. Suppose, if Google or any other search engine had to scan each and every word of say 1 million documents to find all the mentions of the word(s) you are searching for then this process will take hours. Will you wait this long to get the results? No, never. So, how to overcome this problem?

The solution of this problem lies in building an index. The aim of building an index is to optimize the speed and performance of the search process by segregating the documents according to the keywords they contain. As far as Google is concerned, it has 1 million servers across 70 data centers to hold the index which speeds up the search process on Google search site. So, in order to provide a quick result to its users Google has created a really big infrastructure and it is has to update regularly to maintain the quality. This is the reason why sometimes ranks go up and down and then return to normal state after few time.

Let us now see how Google builds its index. When Google send its robot to crawl the web and suppose it come on site ‘A'. First of all there is no guarantee that the site will be crawled. If it does then it gives a unique ID to each page that it founds on site ‘A' and passes the list to the indexing program. The list contains information about every page and the keyword associated with it. Now suppose, if a searcher enters the query "George Bush" and the word "George" may exist in documents say 1,3,6,8, 34 while the word "Bush" may exist in the documents 4, 6, 8, 14, 34, 56 then, only documents 6 and 34 would contain the words.

But here's a catch. You must have definitely heard of stop words. Google ignores certain common words such as "where", "how" and certain single digits and letters as they are not relevant to the search results. Moreover, if counted, they can slow down the search process as it will significantly increase the size of the index. However, Google is quite smart at recognizing when a stop word is being used in way that is uncommon. So, for example, a search for "the India" will be read by Google as "India". However, a search for "the New York Times" will not be modified and will be read as it is, returning results for the well known daily newspaper.
Now when Google has prepared an index, how does it present the results? When a user enters a query in the search box, Google essentially performs two tasks:

1. Finding the set of pages (from the index) that contain the user's query.

2. Ranking the matching pages in the order of importance and relevance which is measured by several factors such as quality of website, PageRank, traffic and many other things.

The second point above cannot be discussed in this article as it is a different thing altogether but I will be discussing it in my next article very soon.

At last I would like to say that Google is delivering us quality results in fraction of seconds which require robust and huge infrastructure and lots of time and efforts to maintain it and Google is doing it.

I simply love the web. According to me, it's the most happening place in the world and the best place to interact and gain knowledge. My strong attraction towards site analysis from users as well as search engines perspective made me to pursue the career in Internet Marketing. I started as SEO but now I work as an Internet Marketing Specialist in a web development company based in India specializing in web 2.0 development and providing web design and development services to the clients globally.
The company provides website design and web development services for range of websites utilizing the power of latest technologies like - .NET, PHP, Ruby on Rails (RoR), Silverlight and SharePoint to name a few.