...
Social Media Crawler - These crawlers scan the content of your website in order to know how to display content when someone links to your website from their platform, such as a twitter Twitter card.
Note that there are other types of crawlers, but these are the most common that you will see when viewing your website's access logs.
...
Even when allowing specific crawler(s), it is important to slow down the rate at which they crawl your site, so they do not impact the performance of the server and therefore your website. Several of the most common and legitimate web crawlers hit your website one to two times per second. Depending on the type of content being served and the resources allocated, this could impact the server performance and contribute to slower website load times.
...
Crawl-delay - The amount of time between page requests that the crawler the rule applies to should wait between each page request.
...
If your website is on a Windows server running IIS web server and you need to block specific user-agents, this can be done easily through the IIS GUI or directly in your website's web.config file. We'll walk through the steps depending on which method (using the GUI or web.config file directly) you choose:
...
5. In the right-side actions pane, click the option labeled 'Add Rule(s)...', then choose Blank Rule under the Inbound rules, and then click OK.
...