Spiders

A spider is an automated software that searches the pages of a website. A spider is a kind of bot that automatically checks every line of content on a website and follows dofollow hyperlinks to another page. 

Through spiders, new pages and changed content are indexed so that they can be found in the search results of a search engine. Therefore, having a clear internal link structure and a sitemap is very important for search engine optimisation. 

The terms spiders and crawlers are often used interchangeably. However, there is a difference. A spider is used to search a website, whereas a crawler is used to search entire search engines.