Reaching out to others! Free & Open Source Software, Kannada, L10n, L18n Data Science, Cloud Computing & more…

Fix CSF blocking Google bots

If you install on your server along with you might see it blocking google bot for some specific rules. At times you might find it offensive to remove some of the rules from the configuration just to allow Google bots.

 

Here is what you should be doing to make Google Bots crawl through your sites again.

Just add the following rules to mod_security configuration file and restart apache.

# GoogleBot by user-agent…
SecRule HTTP_USER_AGENT “Google” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot” nolog,allow
SecRule HTTP_USER_AGENT “GoogleBot” nolog,allow
SecRule HTTP_USER_AGENT “googlebot” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot-Image” nolog,allow
##
SecRule HTTP_USER_AGENT “AdsBot-Google” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot-Image/1.0″ nolog,allow
SecRule HTTP_USER_AGENT “Googlebot/2.1″ nolog,allow
SecRule HTTP_USER_AGENT “Googlebot/Test” nolog,allow
SecRule HTTP_USER_AGENT “Mediapartners-Google/2.1″ nolog,allow
SecRule HTTP_USER_AGENT “Mediapartners-Google*” nolog,allow
SecRule HTTP_USER_AGENT “msnbot” nolog,allow

Enjoy!

Tags: , , , ,