Malav Shukla

Submitting to Crawlers

Crawler-based search engines automatically visit web pages to compile their listings. This means that, unlike directories, you are likely to have several if not many pages listed with them. This also means that by taking care in how you build your pages, you might rank well in crawler-produced results.

Submitting to Google

One of the most important crawler-based search engines is Google, because many people search at it, plus it “powers” the main results of several other services.

The absolute best way to get listed with Google is to build links to your web site. Indeed, this is the best way to get listed for free with all the major crawlers listed on this page. Crawlers follow links, so if you have good links pointing at your web site, the crawlers are more likely to find and include your pages.

Here’s the good news: if you submitted your site to the major directories and got listed with one of them, then Google and other crawlers will almost certainly pick up the URL that was listed. This means you may not need to do additional work to get listed with crawlers.

Google provides an Add URL page that lets you submit a URL directly to its crawler. There’s no guarantee that Google will actually include a URL submitted to it this way, however. Despite this, it makes sense to submit your home page and perhaps one or two other URLs from “inside” your web site via the Add URL page.

You really don’t need to submit more than this. The only reason for submitting some of your inside pages is in case there is a problem reaching your home page. This gives Google an alternate route into your site. From whatever page it visits, it will look for links to other pages that you have and perhaps include those. This is true for other crawlers, as well.

If you have a brand new web site, it will probably take about a month before Google lists your web pages. Because of this, you might consider making use of its paid placement program.

Submitting to Yahoo

Yahoo is an important crawler-based search engine because many people use the Yahoo site and it provides main results of several other services.

As covered with Google, building links is the best way to get listed for free. Yahoo also offers a free URL submission form. Submit according to the same instructions as for Google, above.

What if you aren’t picked up for free? Yahoo has paid inclusion programs that guarantee to add the pages you submit quickly. The downside to these programs is that you’ll be charged every time someone clicks on your listing. If you run out of money, potentially, your listing may be dropped. However, there’s still a chance that even if you run out of money, you might continue to be listed for free.

By the way, Yahoo’s crawler incorporates technology from three different crawlers that it purchased in 2002 and 2003: Inktomi, AltaVista and FAST’s AllTheWeb. Any references you hear about those crawlers are now superceded by the single Yahoo crawler that operates.

Submitting to Teoma

Teoma is an important crawler-based search engine because it powers the main of the results that appear at the popular Ask Jeeves web site. In fact, Ask Jeeves owns Teoma.

Teoma has no free Add URL page. This doesn’t mean that you can’t get listed, however. Teoma crawls the web, so if you have links pointing at your web site, you may get included naturally.

Submitting to MSN

MSN Search is an important crawler-based search engine used by many people. It is currently powered by Yahoo’s crawler-based results.