The 5,000-product problem: why AI crawlers never read most of your catalog

Here's a number that should bother you if you run a store with more than a few hundred SKUs: most AI crawlers read somewhere between 5,000 and 8,000 tokens per document fetch. Not per site. Per file.

That's not a lot. A token is roughly three-quarters of a word, so we're talking about something in the range of 4,000 to 6,000 words before the crawler stops reading and moves on. Enough for a focused document about, say, one category of products. Nowhere close to enough for an entire catalog.

Now picture a store with 5,000 products, all described in a single flat file (which is exactly what happens if you never touch the content layer we described in what llms.txt actually is). The crawler opens the file. It starts reading from the top. Somewhere around product 200, maybe 300 if the descriptions are short, it hits the edge of its context window and stops. Everything after that point might as well not exist, as far as that AI system is concerned.

The part that makes this dangerous

There's no error message. Nothing breaks. Your file loads fine, the crawler fetches it successfully, your server logs show a 200 response. Every signal you'd normally check says everything is working.

And 94% of your catalog was never actually read.

We picked that number for a reason: if a crawler reads the first 300 products out of 5,000 and stops, that's roughly 6% coverage. Ninety-four percent of what you sell is invisible to that AI system, and there is no dashboard, no warning, no log line anywhere in standard Shopify analytics that tells you this happened. You'd have to know to go looking for it, and most merchants don't, because why would you assume a file loading correctly is quietly failing at its actual job?

Why this happens even to stores doing everything "right"

This isn't a content-quality problem. You can write beautiful, keyword-rich, genuinely helpful product descriptions for every single item you sell, and it won't matter if they're all sitting in one document that's too long for a crawler to finish reading. The chunking problem is structural, not editorial. It's about file architecture, not writing quality.

It's also not a Shopify problem specifically. Any single-file approach to AI-readable product content hits this same wall, because the limit lives on the crawler's side, not the file format's side. A 5,000-product catalog crammed into one document is too long for an AI system to fully digest no matter how well it's written or which platform generated it.

What actually fixes it

The fix isn't "write shorter descriptions" or "pick your best 300 products and only describe those." That just moves the ceiling, it doesn't remove it, and it means most of your catalog stays permanently unrecommendable.

The fix is chunking: instead of one file trying to hold everything, split the catalog into collection-sized files, roughly 100 products each, so a crawler fetching the "hiking gear" chunk gets a complete, self-contained document about hiking gear, well within its reading window, instead of a fragment of an oversized file. A navigation layer on top (a structured table of contents) tells crawlers which chunk to go to for which category, so nothing gets missed and nothing gets truncated mid-read.

This is the architecture OtoRank builds automatically. Every collection becomes its own file at a predictable URL, sized for how AI crawlers actually consume documents, with the header, product descriptions, FAQs, and variant data a crawler needs to fully read that category in one pass. A 5,000-product store with this structure in place has, in principle, 100% of its catalog readable by AI. The same store with one flat file has a small fraction.

We also sort every chunk by AEO score, meaning your most AI-ready products (the ones with real descriptions and FAQs already published) show up first within each file, right where a crawler with limited reading budget is most likely to actually get to them. If you want the mechanics of how that score works and why ordering matters this much, that's worth its own read, but the short version is: the products most prepared for AI discovery get the best odds of being seen at all.

The question worth asking yourself

Not "do I have good product content," but "how much of my catalog is actually reachable by an AI system in one fetch." For a lot of stores above a couple hundred SKUs, the honest answer is uncomfortable.

If you want to see where your own catalog stands, an AEO score walks through exactly this, product by product, no guesswork. It's a quick look, not a sales pitch, and if you'd rather talk it through with a person than click around a dashboard alone, you're welcome to book a short call and we'll pull up your store together.