Secret ingredients to quality software

Do you use your Robots.txt file effectively?

Last updated by SSW.Rules.SharePointExtractor on 22 Feb 2021 02:13 am (9 months ago) See History

If you decide to you use the redirect method when linking to external pages from your site, it's a good idea to have a robots.txt  file in your root directory. In the robots.txt file, you specify that the robot (or spider as they're sometimes known) should not look in the redirects folder. This will avoid the problem that can sometimes occur, where a Google search will incorrectly display content from another site as if it was from your site.

Also, this avoids incorrect hits on your redirects, mucking up your statistics which is one of the main reasons you would use redirects in the first place!

Adam CoganAdam Cogan

We open source. Powered by GitHub