If you install CSF on your server along with Mod_security you might see it blocking google bot for some specific rules. At times you might find it offensive to remove some of the mod_security rules from the configuration just to allow Google bots.
Here is what you should be doing to make Google Bots crawl through your sites again.
Just add the following rules to mod_security configuration file and restart apache.
# GoogleBot by user-agent…
SecRule HTTP_USER_AGENT “Google” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot” nolog,allow
SecRule HTTP_USER_AGENT “GoogleBot” nolog,allow
SecRule HTTP_USER_AGENT “googlebot” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot-Image” nolog,allow
##
SecRule HTTP_USER_AGENT “AdsBot-Google” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot-Image/1.0″ nolog,allow
SecRule HTTP_USER_AGENT “Googlebot/2.1″ nolog,allow
SecRule HTTP_USER_AGENT “Googlebot/Test” nolog,allow
SecRule HTTP_USER_AGENT “Mediapartners-Google/2.1″ nolog,allow
SecRule HTTP_USER_AGENT “Mediapartners-Google*” nolog,allow
SecRule HTTP_USER_AGENT “msnbot” nolog,allow
Enjoy!
Tags: cpanel, csf, GoogleBots, mod_security, SEO
