Google is looking to make changes in the 30 years old robots.txt to accommodate mainly AI/ML. Here are the details:
https://blog.google/technology/ai/ai-web-publisher-con...
Let me share the latest Google Update news.
https://blog.google/products/search/google-search-update-march-2024/
And a little bit more info:
https://developers.google.com...