Adding NoIndex to SiteSearch
Hi there,
I have an ecommerce client who launched their online store without a robot.txt file to block the crawling of filters and site search. Due to this, they have thousands "Site Search" URLs indexed. I want to noindex these results before blocking crawling in the rpbots.txt file.
Does anyone know how you can do this with Magento? Or other suggestions on how to get these results removed before adding it to the robots.txt file?
Indexed URLs can be removed using Webmasters Tools or Search Console of search engines. For example, the Removals section can be used in the Google Search Console to remove all URLs with a specific prefix. After removing the URLs, restrict for crawling.
Hi @binayjha
Thanks for your reply, but I believe this tool only "filters" these pages/URLs from the results for 6 months. However, it might be worth attempting, as I don't think we can noindex these site search results/