Adding NoIndex to SiteSearch
I have an ecommerce client who launched their online store without a robot.txt file to block the crawling of filters and site search. Due to this, they have thousands "Site Search" URLs indexed. I want to noindex these results before blocking crawling in the rpbots.txt file.
Does anyone know how you can do this with Magento? Or other suggestions on how to get these results removed before adding it to the robots.txt file?
Indexed URLs can be removed using Webmasters Tools or Search Console of search engines. For example, the Removals section can be used in the Google Search Console to remove all URLs with a specific prefix. After removing the URLs, restrict for crawling.