SHARE

The Robots.txt File

The robots.txt file is a text file you should put in the root directory (folder) of your website (http://www.yourwebsite.com/robots.txt), telling search engines such as Google what to scan and what should not be scanned on your website. It is a good practice to configure this file.

However, some webmasters add the location of the control panel on the list of files not to be scanned by the search engine. Since the robots.txt file is public, anyone can open it to check if there is any unusual directory (folder) listed under “Disallow.”

Consider the real example presented in Figure 2. Why is the/Comment/NewComment directory listed under “Disallow?” That is definitely a place a hacker would open to see what is there. Opening this directory on this particular website produces the login screen shown in Figure 3. Bingo!

Robots.txt fileFigure 2: Robots.txt file

A login screen found through the robots.txt fileFigure 3: A login screen found through the robots.txt file

Therefore, you must not add the directory (folder) of your control panel in the robots.txt file.

1
2
3
4
5
6
7

Gabriel Torres is a Brazilian best-selling ICT expert, with 24 books published. He started his online career in 1996, when he launched Clube do Hardware, which is one of the oldest and largest websites about technology in Brazil. He created Hardware Secrets in 1999 to expand his knowledge outside his home country.