Do you Disallow Ahrefs/Semrush and similar bots?
by @ms (4326), 2 years ago
Tricky question SEOs.
Do you Disallow those bots (which according to their docs respect robots.txt)? If yes, why? If not, why?
For example, to prevent your competition from getting data about your site/business...
by @binayjha (5694), 2 years ago
No, I have never blocked those bots. I never felt any harm in letting them give such SEO data.
by @ms (4326), 2 years ago
I personally use Ahrefs so I don't block their bot, however I do block Semrush as I don't use it. Why allow it to consume my server resources just to give my competitors extra data about my website..
by @Gamerseo (1330), 2 years ago
We also never block any bots.