Home » Internet » Search Engine Marketing

What To Do When Spiders Don't Visit Your Site

Oct 12, 2007
It sometimes happens that a search engine spider don't visit your site even after you submit it manually. Take a look at your site's log files. Look at the hostnames of the sites that visit your site. You should see something that looks like this: googlebot.com. This is Google's bot or spider. If you do not see this in your log files then the Googlebot is not visiting your site.

To get listed in Google you need the Googlebot to visit your site. The best way to do this is to get links from web sites that rate high in Google. Go to google and search a few keywords related to the subject of your web site, see what comes up. Now try the same keywords and include, "add your url" and "submit your site". As an example, if you sell vitamins search Google for, "vitamins add your url" and "vitamins submit your site".

This will bring up web sites that want to exchanges links and are highly rated by Google for keywords related to your web site. Read the instructions on their links page, add their information to your links directory and then add your information to their links page. When a spider visits a site one of the things it does is read through the sites link pages to find new sites to visit. When the Googlebot does this it will find the link to your site and hopefully visit it. Exchange links with several sites this way and keep looking for the Googlebot to show up in your log files. If it does not show up in another three weeks you need to re-submit it.

Sometimes your site can get listed on a search engine, but then taken down for some unknown reason. In either case, re-submitting isn't overly difficult. There are some guidelines you should follow in order to guarantee results. We'll be going over where you can submit sites, how to submit sites and how to re-submit sites.

Automatic submission software is necessary for submitting to the thousands of lesser search engines, but do not try it with the major search engines. No automatic submission software will replace manual submission for the major search engines.

Where you should manually submit your site

Most of the search engines traffic comes from the three major search engines Google, Yahoo! and MSN. The major search engines are the most important places to get listed. These three provide results for many other search engines, and if you are listed on these three, your site will be listed on many others too. Rankings are not always the same on other search engines, this is because the algorithm is different but the index is the same. Most web sites get into Google, Yahoo and MSN by getting spidered from other sites. If you're already linked from another sites, Google, MSN and Yahoo's spiders will eventually find you. It is still a good idea to submit to them by hand once. It is easy to do and does not take long. Submitting your web site multiple times has no effect. Before you submit your site make sure it does not appear on the search engine you are submitting to. There are hundreds of different search engines which you can submit your site to. We'll go over a few of the most popular ones here.

Google http://www.google.com/addurl/
Google has a well-deserved reputation as the top choice for those searching the web, it is used by millions of people each day. Two things you need to do to rate well in Google is to make sure that your site is full of useful, unique content and relevant sites that link back to you.

Yahoo Search Engine http://search.yahoo.com/info/submit.html
Yahoo is easy to use and has great features, you can use the tabs above the search box on the Yahoo home page to search images, Yellow Page listings or Yahoo Shopping. Yahoo is easy to submit to.

MSN Search is now Live Search http://beta.search.msn.com/docs/submit.aspx
Live Search offers users the ability to search for specific types of information using search tabs that include Web, news, images, music, desktop, local, and Microsoft Encarta.

The Open Directory Project http://www.dmoz.org/add.html
The Open Directory Project (ODP), is a large, categorized directory of web sites, which is staffed by volunteers. Getting your site listed can take a very long time. Every web site and page that is added to the directory has to be manually reviewed before it is included and they pride themselves on being highly selective. DMOZ's will only include your site if it has unique content.

Lycos: http://www.lycos.com/
A search engine and web portal centered around broadband content. This is an easy to use search engine with lots of the same features Google has. You don't have to manually submit sites here, if you have good content and good links pointing to you spider will find you. It is fast and easy to submit to.

AllTheWeb
Search results on AlltheWeb are provided by Yahoo!. To be listed with AlltheWeb you need to be listed with Yahoo! AllTheWeb is a popular search engine with quite a few features.

AltaVista
Search results on AltaVista are powered by Yahoo!. To be listed with AltaVista you need to be listed with Yahoo! AltaVista is not as popular as it once was, it still receives many of visitors.

AOL Search
AOL Search provides listings that come from Google's index. The same search on Google and AOL Search will have close to the same results. To be listed with AOL you need to be listed with Google.

Submitting advice

Submitting you site to the search engines is easy. You should submit your site not just to the well-known search engines, but also to the smaller ones. The more your site is listed, the more visitors you will get. Go to the search engine's site, click on "Submit a site" and then fill out the necessary info about your site. Here are some things you should keep in mind.

1. Don't submit your site if all the pages aren't complete. Search engines don't like incomplete sites. It is pointless to fill out the form for submission if your site isn't completely finished.

2. Make your description as brief but comprehensive as possible. People who are looking at your submission don't have time to read through a big, long description.

3. Make sure the details in your description are relevant, interesting and accurate. If yours site is about shoes make your description about the shoes on your site.

4. People are reading your submissions and approving them. The description is intended to give people a statement of what can be found in the site, and not to promote it. Don't brag too much, be modest.

How to Re-submit sites

Before you try to re-submit a site, take the following into consideration.

1. Has it been at least a three weeks since you originally submitted?

1. What kind of links do you have pointing to your site? Are they links from sites relevant to the subject of your site?

2. Was your site pulled because it had content on it that was inappropriate? If this is the case you need to change your content before you re-submit or it will only get pulled again.

To re-submit your site, all you have to do is follow the same process as you did when you submitted the site before.
About the Author
Sign up for the Web Success Weekly Email. Lean simple, step by step methods to get your business online and making money, the easy way: http://websuccess.info/

By Harvey Lew Robinson:
http://websuccess.info/seo/spiders.html
Rating:
Please Rate:
(Average: Not rated)
Views: 256
Print Email Report Share
Article Categories