Toggle navigation
Join GoodFirms to access
  • Write your answer
  • Follow this question
  • Request answers
  • Upvote or downvote answers
  • Write your comment

What is a crawl delay?

Write Answer
Answer
Follow
request
  Request
Downvote
Downvote
Share
user image
A file with robots.txt is one of the main ways to inform the search engine where it crawls through on the website. In essence, it is a text file containing the instructions that are interpreted by search engine spiders, often called robots, to obey a...read more
Upvote
Upvote  ‧ 1
Downvote
Downvote
Share
Share