...
Note that there are other types of crawlers, but these are the most common that you will see when viewing your website's access logs.
...
Even when allowing specific crawler(s), it is important to slow down the rate at which they crawl your site, so they do not impact the performance of the server and therefore your website. Several of the most common and legitimate web crawlers hit your website one to two times per second. Depending on the type of content being served and the resources allocated, this could impact the server performance and contribute to slower website load times.
...
Crawl-delay - The amount of time between page requests that the crawler the rule applies to should wait between each page request.
...
5. In the right-side actions pane, click the option labeled 'Add Rule(s)...', then choose Blank Rule under the Inbound rules, and then click OK.
...