What Are Crawlers, Spiders, Bots & Robots?

Crawler bots and robots are same. This is an application written in programming language. This application is used to fetch new and existing websites, web page from net. All search engines have their own crawlers. Some search engines used same crawler to fetch all types of data and some have different crawlers for different data types. Google, Yahoo and MSN have different crawlers for images, video, text and for mobile.

Crawler Names:

Google: Google Bot

Yahoo: Slurp

Inktomi: Slurp

Bing/MSN: MSN Bot

Alta Vista: Scooter

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.