Adding Robot.txt To Cpanel Websites
It is well known that Bing bot and MSN bot could kill a website because of their unmetered bot requests. This is a simple shell script to reduce the crawling rate of bing and msn bots for the cpanel websites.
Features
- Used for adding robot.txt file for cpanel websites to reduce crawl delay.
- Prevents bots from creating load in the server.
- No need of any installations.
- Just copy the code and execute it in your command line.
Downloads
Create robot.txt file and copy the below code in to the file,
Create a shell file named, something like 'hiox-robot-block.sh' file and copy the below code in to the file,
#---------- Script by hscripts.com ---------------# #---------- Copyright of HIOXINDIA ---------------# #---------
More scripts @www.hscripts.com -------# User-agent: bingbot Crawl-delay: 60
User-agent: msnbot Crawl-delay: 60
User-agent: msnbot-UDiscovery/2.0b Crawl-delay: 60
User-agent: * Crawl-delay: 20 #--------- Script by hscripts.com -----------#
More scripts @www.hscripts.com -------# User-agent: bingbot Crawl-delay: 60
User-agent: msnbot Crawl-delay: 60
User-agent: msnbot-UDiscovery/2.0b Crawl-delay: 60
User-agent: * Crawl-delay: 20 #--------- Script by hscripts.com -----------#
Create a shell file named, something like 'hiox-robot-block.sh' file and copy the below code in to the file,
#---------- Script by hscripts.com ---------------# #---------- Copyright of HIOXINDIA ---------------# #---------
More scripts @www.hscripts.com -------#
cat /etc/httpd/conf/httpd.conf | grep 'DocumentRoot' | awk '{print $2}' > docroots.txt nolines=`cat docroots.txt | wc -l` lineno=1 while [ $lineno -lt $nolines ] || [ $lineno -eq $nolines ] do
docroot=`sed -n "${lineno}p" docroots.txt`
if [ -d $docroot ]; then
if [ ! -f $docroot/robots.txt ]; then
echo 'robots.txt file updated in' $docroot
#cp /root/cpanel3-skel/public_html/robots.txt $docroot/.
fi
fi
lineno=`expr $lineno + 1` done rm -f docroots.txt #--------- Script by hscripts.com -----------#
More scripts @www.hscripts.com -------#
cat /etc/httpd/conf/httpd.conf | grep 'DocumentRoot' | awk '{print $2}' > docroots.txt nolines=`cat docroots.txt | wc -l` lineno=1 while [ $lineno -lt $nolines ] || [ $lineno -eq $nolines ] do
docroot=`sed -n "${lineno}p" docroots.txt`
if [ -d $docroot ]; then
if [ ! -f $docroot/robots.txt ]; then
echo 'robots.txt file updated in' $docroot
#cp /root/cpanel3-skel/public_html/robots.txt $docroot/.
fi
fi
lineno=`expr $lineno + 1` done rm -f docroots.txt #--------- Script by hscripts.com -----------#
- Release Date - 23-04-2014
- Get free version without ©copyright link for just $10/-
- For customization of this script or any script development, mail to support@hscripts.com
Usage
- Place both the files under the same folder in your website.
- Run the script in a command prompt as "sh hiox-robot-block.sh"
License
- Click Here for detailed license information.
