Adding Robot.txt To Cpanel Websites

It is well known that Bing bot and MSN bot could kill a website because of their unmetered bot requests. This is a simple shell script to reduce the crawling rate of bing and msn bots for the cpanel websites.

Features

  • Used for adding robot.txt file for cpanel websites to reduce crawl delay.
  • Prevents bots from creating load in the server.
  • No need of any installations.
  • Just copy the code and execute it in your command line.

Downloads

Create robot.txt file and copy the below code in to the file,
#---------- Script by hscripts.com ---------------# #---------- Copyright of HIOXINDIA ---------------# #---------
More scripts @www.hscripts.com -------# User-agent: bingbot Crawl-delay: 60
User-agent: msnbot Crawl-delay: 60
User-agent: msnbot-UDiscovery/2.0b Crawl-delay: 60
User-agent: * Crawl-delay: 20 #--------- Script by hscripts.com -----------#

Create a shell file named, something like 'hiox-robot-block.sh' file and copy the below code in to the file,
#---------- Script by hscripts.com ---------------# #---------- Copyright of HIOXINDIA ---------------# #---------
More scripts @www.hscripts.com -------#
cat /etc/httpd/conf/httpd.conf | grep 'DocumentRoot' | awk '{print $2}' > docroots.txt nolines=`cat docroots.txt | wc -l` lineno=1 while [ $lineno -lt $nolines ] || [ $lineno -eq $nolines ] do
docroot=`sed -n "${lineno}p" docroots.txt`
if [ -d $docroot ]; then
if [ ! -f $docroot/robots.txt ]; then
echo 'robots.txt file updated in' $docroot
#cp /root/cpanel3-skel/public_html/robots.txt $docroot/.
fi
fi
lineno=`expr $lineno + 1` done rm -f docroots.txt #--------- Script by hscripts.com -----------#

  • Release Date - 23-04-2014
  • Get free version without ©copyright link for just $10/-
  • For customization of this script or any script development, mail to support@hscripts.com

Usage

  • Place both the files under the same folder in your website.
  • Run the script in a command prompt as "sh hiox-robot-block.sh"

License

Other Scripts


Ask Questions

Ask Question