It is well known that Bing bot and MSN bot could kill a website because of their unmetered bot requests. This is a simple shell script to reduce the crawling rate of bing and msn bots for the cpanel websites.
Features
Used for adding robot.txt file for cpanel websites to reduce crawl delay.
Prevents bots from creating load in the server.
No need of any installations.
Just copy the code and execute it in your command line.
Downloads
Create robot.txt file and copy the below code in to the file,
#---------- Script by hscripts.com ---------------# #---------- Copyright of HIOXINDIA ---------------# #--------- More scripts @www.hscripts.com -------# User-agent: bingbot Crawl-delay: 60 User-agent: msnbot Crawl-delay: 60 User-agent: msnbot-UDiscovery/2.0b Crawl-delay: 60 User-agent: * Crawl-delay: 20 #--------- Script by hscripts.com -----------#
Create a shell file named, something like 'hiox-robot-block.sh' file and copy the below code in to the file,
#---------- Script by hscripts.com ---------------# #---------- Copyright of HIOXINDIA ---------------# #--------- More scripts @www.hscripts.com -------# cat /etc/httpd/conf/httpd.conf | grep 'DocumentRoot' | awk '{print $2}' > docroots.txt nolines=`cat docroots.txt | wc -l` lineno=1 while [ $lineno -lt $nolines ] || [ $lineno -eq $nolines ] do docroot=`sed -n "${lineno}p" docroots.txt` if [ -d $docroot ]; then if [ ! -f $docroot/robots.txt ]; then echo 'robots.txt file updated in' $docroot #cp /root/cpanel3-skel/public_html/robots.txt $docroot/. fi fi lineno=`expr $lineno + 1` done rm -f docroots.txt #--------- Script by hscripts.com -----------#