Recon 00

This exercise covers the robots.txt file

easy diffculty Easy difficulty
average completion time icon
Less than an hour average completion time
number of users completed icon
11770 users completed this exercise

For this challenge, your goal is to retrieve the robots.txt from the main website for

The robots.txt file

The robots.txt file is used to tell web spiders how to crawl a website. To avoid having confidential information indexed and searchable, webmasters often use this file to tell spiders to avoid specific pages. This is done using the keyword Disallow. You can find more about the robots.txt file by reading Robots exclusion standard