What are Bots, Crawlers or Web Spiders?

Bots, cawlers or web spiders are automated programs that download web pages and extract information. Generally, web crawlers are used to index web pages for efficient search, to index web pages they index URLs of web pages. The well known application of  web crawling  is search engines. Further, web crawlers have different components, such as,  web page downloader, a database and a link scheduler. More components will be added according to need.

It will be easy to understand for you if I explain crawlers by examples.
The first example I am going to expalin about GoogleBot.

GoogleBot- GoogleBot is Google’s search engine crawler  which indexes URLs of the web pages and meta data and when a user type a key word Google shows the results. If you use Google’s Webmaster ( Search Console )  and you submit a URL manually, GoogleBot checks whether it is complaint with Google’s policy and indexex in its database. In this example, GoogleBot does not crawl automaticaly. Instead, you provide URL of your web page.
In automated crawling, Google will have your web address in dadabase and whenever, you you post an article and GoogleBot find it, then URL of the article will be indexed. It can take lot of time to be indexed.

BingBot, YandexBot, BaiduBot and Yandex osearch engines crawlers work on the same principle with little bit variation in algorithm and crawler  components.
Search engines are not only the applications of web crawlers there are variety of application.