Home » Internet

All About Search Engines

Sandra Prior
Jul 14, 2008
The Hitchhiker's Guide to the Galaxy is big. You just won't believe how vastly hugely mind boggling big it is. I mean, you may think it's a long way down the road to the chemist, but that's just peanuts to space. You could say the same about the Internet. According to Inktomi, there are more than one billion Web pages and thousands more are uploaded every day.

Exploring a billion pages would take decades, and that's just text - add in MP3 music, video clips, downloadable program files and dancing hamsters, and you've got a library that would take thousands of years to explore. Because the Internet gets bigger every day, getting hold of interesting stuff is like finding a very small needle in a very large haystack.

While search engines have always been the best way to track down elusive information, a quick search often results in page after page of links that bear little or no relation to what you're looking for.

To cope with the incredible growth of the Web, search engines have become very powerful. If you know what you're doing, they'll find what you're looking for in a matter of seconds, whether you're trying to find an email address or a video clip. Many search engines now have dedicated sections for MP3 music and graphic files, while others contain a range of advanced options that you can use to get rid of irrelevant search results. Some sites even let you ask questions in plain English.

The first search engines were simple collections of links, organized into categories that made it easy to find what you were looking for. This method, known as directory or hierarchical index, is still used today by sites such as Yahoo. Using Yahoo is as simple as using the phone book. If you're looking for sites about a specific subject, a hierarchical search engine is the best place to start.

There's a good range of directory sites on the Web, many of which employ human beings to write plain English descriptions of each link. Some sites use a team of journalists who describe each site that's added to the database. The big name directories such as Yahoo attempt to categorize the entire Internet, but some sites prefer to concentrate on specific subjects.

Although directories make it easy to find sites devoted to specific subjects, browsing through categories and subcategories can be very time consuming. Many of the links are provided without explanation and there's no way of telling whether a sites is going to be useful. Hierarchical indexes can tell you where to find sites dedicated to your favorite pop star, but they won't be able to find online shops that stock specific records. For that, you'll have to ask a spider.


Search engines such as Webcrawler, Altavista, Google, get their information from spiders, not the biological kind, but special computer programs that crawl around the Web. When a Webmaster tells Google about his or her site, a spider is sent to check it out. The spider gathers all the information about the site, including the text of each page and any meta tags hidden in the HTML, and sends the information back to Google. The spider also follows any links to other sites, which is why it can take up to three weeks between telling the search engine about your site and the spider paying a visit.

Search engines that use spiders can be very effective because they know every word of every Web page they've looked at. You can carry out very specific searches without needing to work out what category a site is likely to be filed under and you can find everything from HTML pages to message board posts.

This kind of search isn't always the best way to find things, and you'll notice that Altavista now includes Yahoo style categories with its search results. Yahoo has returned the compliment, adding a spider style search engine to its directory system. New search sites also combine spiders and directories, offering you the best of both worlds.
About the Author
Subscribe to Sandra Prior's Online Newsletter and get up to date Computer Technology News delivered right to your email box for free. See website for details http://usacomputers.rr.nu and http://sacomputers.rr.nu.
Please Rate:
(Average: Not rated)
Views: 123
Print Email Share
Article Categories