Wow! Google now indexing One Trillion Web pages

Posted by Saad Ibrahim 27 July, 2008

The Internet is getting bigger and bigger and so is Google’s Index, According to the Official Google Blog Post Google is now indexing a mammoth 1 Trillion web pages, Yeah! even the people at Google got surprised, the Google bot has indexed 1,000,000,000,000 Unique URLs so far and this number is growing at a huge pace, The number would have been even huge if Google would have indexed the duplicate Content.

here is the exact excerpt from the Google blog post on how the indexing works:

How do we find all those pages? We start at a set of well-connected initial pages and follow each of their links to new pages. Then we follow the links on those new pages to even more pages and so on, until we have a huge list of links. In fact, we found even more than 1 trillion individual links, but not all of them lead to unique web pages. Many pages have multiple URLs with exactly the same content or URLs that are auto-generated copies of each other. Even after removing those exact duplicates, we saw a trillion unique URLs, and the number of individual web pages out there is growing by several billion pages per day.

Google is the most used search engine on the web and I sometimes wonder how are their servers able to survive so much load whereas ordinary websites cant even survive the “digg effect” but now I get my answer that is distributed infrastructure, Google has many server farms all over the world and all the servers process the data and then interact with each other making the process faster.

So head on to Google.com and start searching from those One trillion web pages if you don’t already do!