Robots.txt Issues in Google Search Console
Hello SEOs,
I found multiple robots txt URLs in my Google search console that I didn't even create. I don't know from where GSC is fetching these URLs?
1https //example(dot)com/robots(dot)txt 2https //subdomain(dot)example(dot)com/robots(dot)txt 3http //www.example(dot)com/robots(dot)txt 4https //www.example(dot)com/robots(dot)txt 5http //example(dot)com/robots(dot)txt
The main version of my website is the first one (https //example(dot)com/robots(dot)txt). I don't know how to remove other robots txt URLs. Need help on this.
Moreover, in Google search console >> settings >> Crawl stats >> Hosts
I can see three different URLs of my site 1example(dot)com 2subdomain-example(dot)com 3www(dot)example(dot)com
The website is on WordPress. I worked on a lot of websites and never faced such issues. Can anybody tell me if these are the technical issues? The website has more than 900 pages and only 10 are indexed. Google is not crawling my site's pages. Content on the website is related to healthcare and its 100% AI-generated.
What should I do in order to make Google crawl my website and index its pages.
Robots.txt Issues in Google Search Console typically occur when Googlebot encounters problems accessing your site's robots.txt file. This file tells search engines which pages to crawl and which to avoid. Common issues include:
Blocked Pages: If important pages are mistakenly blocked by your robots.txt file, they won’t be indexed. Incorrect Syntax: Errors in the file’s syntax can prevent search engines from reading it properly. File Not Found: If Google can't find your robots.txt file, it may default to crawling all pages. Too Restrictive: Overly strict directives may block search engines from accessing valuable content, negatively impacting SEO.