Tuesday, October 12, 2004

We Can't Live Without The Spiders



If your web pages aren't getting spidered, you're missing out. As I mentioned in a previous post, it's always a good idea to have the search engines find you instead of manually submitting to them. The concept is simply: always have your new site linked somewhere to another site that's already been spidered or is frequently spidered by the search engines.

Site maps, links that don't run too deep and are right off the top page are very helpful. Avoid flash if possible, unless your site is already so popular that people just go to it by name without searching for you. Very few people are going to Google to find amazon.com or ebay.com, unless they are looking for something "related" to those sites such as "amazon marketing" or "ebay business".

I am looking for a good site map making program so if you know of one, drop me a note. I don't like the one I currently have. It's clumsy at best. The following are some resources you can check out regarding the search engine spiders. The more familiar you become with them, the more you can set up your site for traffic and more sales, which go hand-in-hand:

Resources

SpiderSpotting - Search Engine Watch

http://searchenginewatch.com/webmasters/spiders.html

Robotstxt.org

List of robots and protocols for setting up a robots.txt file.

http://www.robotstxt.org/

Spider-FoodTutorials, forums and articles about Search Engine spiders and Search Engine Marketing.

http://spider-food.net/

Spiderhunter.com

Articles and resources about tracking Search Engine spiders.

http://www.spiderhunter.com/

Sim Spider Search Engine Robot SimulatorSearch Engine World has a spider that simulates what the Search Engine robots read from your website.

http://www.searchengineworld.com/cgi-bin/sim_spider.cgi