binayjha

Googlebot Documentation has been updated today, what's your take?

by @binayjha (2549), 7 months ago

It says: > Googlebot can crawl the first 15MB of content in an HTML file or supported text-based file. After the first 15MB of the file, Googlebot stops crawling and only considers the first 15MB of content for indexing. Other crawlers may have different limits.

https://developers.google.com/search/docs/advanced/crawling/googlebot

355 Views
1 Upvote
4 Replies
3 Users
Sort replies:
ms
by @ms (3182), 7 months ago

15MB is a LOT of data.

Imagine 15 million characters X number of sites available on the Internet...

binayjha
by @binayjha (2549), 7 months ago

Yes @ms, as it is the size of HTML document data and doesn't include the file size of images or videos etc.

mit_midastouch
by @mit_midastouch (200), 7 months ago

I think 15MB is the maximum space. I saw some e-commerce websites also crawl unwanted through the Googlebot.

binayjha
by @binayjha (2549), 7 months ago

@mit_midastouch Can you please explain it a bit more! I want to understand it.

Join the forum to unlock true power of SEO community

You're welcome to become part of SEO Forum community. Register for free, learn and contribute.

Log In Sign up