{"id":1377,"date":"2026-07-16T06:34:56","date_gmt":"2026-07-16T06:34:56","guid":{"rendered":"https:\/\/maxaeo.ai\/blog\/ai-crawlers-infinite-scroll\/"},"modified":"2026-07-16T06:34:56","modified_gmt":"2026-07-16T06:34:56","slug":"ai-crawlers-infinite-scroll","status":"publish","type":"post","link":"https:\/\/maxaeo.ai\/blog\/ai-crawlers-infinite-scroll\/","title":{"rendered":"Can AI Crawlers See Infinite Scroll? Pagination Patterns That Keep Evidence Discoverable"},"content":{"rendered":"

Can AI crawlers see infinite scroll? In most cases, no.<\/strong> The crawler receives only the first server-rendered batch of items\u2014often 12 to 24\u2014and never discovers the hundreds more that a human unlocks by scrolling. The bots that feed ChatGPT, Perplexity, and Claude do not run JavaScript at all, so the scroll event that loads the rest of your catalog never fires. This guide shows exactly what breaks, how to test your own archives in ten minutes, and which pagination patterns keep every product and post retrievable.<\/p>\n

\"How<\/figure>\n

What happens when an AI crawler hits an infinite-scroll page?<\/h2>\n

An AI crawler fetches the raw HTML your server returns, extracts the links and text inside it, and leaves. It does not scroll, click, or wait for lazy-loaded batches.<\/strong> So an infinite-scroll archive exposes only its first viewport of content\u2014whatever the server put in the initial response.<\/p>\n

That single behavior explains the whole problem. Infinite scroll works by listening for a scroll event, then fetching the next batch over a background request and appending it. A human triggers dozens of those events. A crawler triggers zero. Everything below the first fold lives behind an interaction the bot will never perform\u2014so your deep inventory (old posts, page-40 products, the case study from 2023) is effectively unpublished as far as retrieval is concerned.<\/p>\n

Why infinite scroll fails twice for AI search, but only once for Google<\/h2>\n

Infinite scroll is a single problem for Googlebot and a double problem for AI crawlers. Googlebot can render JavaScript but chooses not to scroll; AI crawlers neither render JavaScript nor scroll.<\/strong> That distinction is the most overlooked fact in this topic, and it changes your fix.<\/p>\n

Googlebot runs a modern rendering engine. It can execute the JavaScript that builds your page, so a client-rendered layout is at least visible<\/em> to it. What Googlebot won't do is simulate a user scrolling to fire lazy-load events. So for Google, infinite scroll fails once: rendering works, discovery of deeper batches does not.<\/p>\n

AI crawlers fail on both counts. Vercel's analysis of AI crawler traffic<\/a> found GPTBot spent roughly 11.5% of its fetches on JavaScript files and ClaudeBot about 23.84%\u2014yet neither executed a line of it<\/strong>. In the month Vercel measured, GPTBot alone generated 569 million requests across its network and rendered none of them. So an AI crawler never runs the app and<\/em> never scrolls it. If your first batch is itself injected by client-side JavaScript, the crawler may see an empty shell\u2014which is why understanding whether AI crawlers render JavaScript at all<\/a> matters before you touch pagination.<\/p>\n\n\n\n\n\n\n\n\n
Crawler<\/th>\nRenders JavaScript?<\/th>\nFires scroll events?<\/th>\nSees deep archive on infinite scroll?<\/th>\n<\/tr>\n<\/thead>\n
Googlebot<\/td>\nYes (with a render queue)<\/td>\nNo<\/td>\nOnly if a crawlable paginated fallback exists<\/td>\n<\/tr>\n
GPTBot \/ OAI-SearchBot<\/td>\nNo<\/td>\nNo<\/td>\nOnly what's in raw HTML<\/td>\n<\/tr>\n
ClaudeBot \/ Claude-SearchBot<\/td>\nNo<\/td>\nNo<\/td>\nOnly what's in raw HTML<\/td>\n<\/tr>\n
PerplexityBot<\/td>\nNo<\/td>\nNo<\/td>\nOnly what's in raw HTML<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The takeaway: optimizing infinite scroll for Google is not enough for AI search.<\/strong> A pattern that leans on Googlebot's rendering will still starve the answer engines.<\/p>\n

A repeatable test: can AI crawlers see your archive?<\/h2>\n

Measure your exposure in about ten minutes with a terminal and a browser. The method compares what a non-rendering crawler receives (raw HTML) against what a human sees (the rendered DOM). The gap is your invisible inventory.<\/strong> Run it on any product or article archive you care about.<\/p>\n

Here is the four-step test, in order:<\/p>\n

    \n
  1. Fetch the raw HTML a crawler gets.<\/strong> Request the page with a bot user agent and no JavaScript execution:
    \n curl -A "GPTBot" https:\/\/yoursite.com\/blog\/ > raw.html<\/code><\/li>\n
  2. Count the items in that raw response.<\/strong> Count your repeating item markup\u2014<article<\/code>, a product-card class, a listing <li><\/code>\u2014directly: grep -c '<article' raw.html<\/code>. If the archive holds 240 products but the count is 12, a crawler sees 12.<\/li>\n
  3. Look for a crawlable next-page link.<\/strong> Grep the same file for an anchor to page two: grep -o 'href="[^"]*page[^"]*"' raw.html<\/code>. A real <a href="...?page=2"><\/code> means a discovery path exists. A <button><\/code> or an onclick<\/code> handler does not.<\/li>\n
  4. Compare against the rendered page.<\/strong> Open the archive in a browser, scroll to the bottom, and count the fully loaded items. The difference between the browser count and the raw.html<\/code> count is exactly what AI crawlers cannot reach.<\/li>\n<\/ol>\n
    \"Terminal<\/figure>\n

    Representative results from running this test on two common setups look like the table below. Treat these as illustrations of the method\u2014your own numbers will differ, which is the point of testing.<\/p>\n\n\n\n\n\n\n\n
    Archive type<\/th>\nPattern<\/th>\nItems in raw HTML<\/th>\nItems after scroll<\/th>\nCrawlable <a href><\/code> to next page?<\/th>\nCrawler-visible share<\/th>\n<\/tr>\n<\/thead>\n
    Product listing<\/td>\nInfinite scroll, JS-injected<\/td>\n12<\/td>\n240<\/td>\nNo<\/td>\n~5%<\/td>\n<\/tr>\n
    Article archive<\/td>\n"Load more" <button><\/code><\/td>\n10<\/td>\n130<\/td>\nNo<\/td>\n~8%<\/td>\n<\/tr>\n
    Product listing<\/td>\nPaginated URLs (?page=<\/code>)<\/td>\n24<\/td>\n24 per page<\/td>\nYes<\/td>\n100%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    The pattern is stark: infinite scroll without a linked fallback exposes single-digit percentages of a large archive to AI crawlers.<\/strong> For a deeper walkthrough of raw-HTML versus rendered-DOM differences, see the comparison of server-side versus client-side rendering for AI visibility<\/a>.<\/p>\n

    Four pagination patterns, tested against crawlers<\/h2>\n

    Not all "modern" pagination is equal. The deciding factor is one question: does the raw HTML contain a real anchor tag pointing to the next set of items at a distinct URL?<\/strong> If yes, crawlers can walk your archive. If no, they stop at batch one. Here is how the four common patterns score.<\/p>\n\n\n\n\n\n\n\n\n
    Pattern<\/th>\nReal <a href><\/code> in raw HTML?<\/th>\nDistinct URL per page?<\/th>\nGooglebot reaches deep pages<\/th>\nAI crawlers reach deep pages<\/th>\n<\/tr>\n<\/thead>\n
    Pure infinite scroll (scroll event \u2192 fetch)<\/td>\nNo<\/td>\nNo<\/td>\nNo<\/td>\nNo<\/td>\n<\/tr>\n
    "Load more" button (JS onclick<\/code>)<\/td>\nNo<\/td>\nNo<\/td>\nNo<\/td>\nNo<\/td>\n<\/tr>\n
    "Load more" link (anchor to ?page=2<\/code>)<\/td>\nYes<\/td>\nYes<\/td>\nYes<\/td>\nYes<\/td>\n<\/tr>\n
    Numbered paginated URLs<\/td>\nYes<\/td>\nYes<\/td>\nYes<\/td>\nYes<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Two of these patterns look almost identical to users but behave oppositely for machines. A "load more" button<\/strong> wired as a <button><\/code> with a JavaScript click handler is invisible to crawlers\u2014the same trap as content buried in tabs and accordions, which we cover in whether AI crawlers see hidden interactive content<\/a>. A "load more" control implemented as an <a href="\/blog\/?page=2"><\/code><\/strong> that JavaScript intercepts for a smooth experience is fully crawlable, because the anchor and its destination URL exist in the HTML before any script runs.<\/p>\n

    The rule to remember: crawlers follow anchors to URLs, not clicks on widgets.<\/strong> Progressive enhancement\u2014ship the links, then upgrade the interaction\u2014is what separates a discoverable archive from an invisible one.<\/p>\n

    The crawlable fallback pattern that actually works<\/h2>\n

    The reliable pattern is a paginated series of real URLs underneath your scroll experience: every scroll position corresponds to a component page a crawler can request directly. This is Google's long-standing recommendation, and it happens to be the only pattern AI crawlers can follow.<\/strong> Build the URLs first; layer the scroll on top.<\/p>\n

    Follow these steps to implement it:<\/p>\n

      \n
    1. Give every batch its own URL.<\/strong> Expose \/blog\/page\/2\/<\/code>, \/blog\/page\/3\/<\/code>, and so on\u2014each returning that batch's items in server-rendered HTML, readable without JavaScript.<\/li>\n
    2. Put real anchor links in the markup.<\/strong> Include <a href><\/code> navigation between component pages in the initial HTML, not injected after load.<\/li>\n
    3. Keep the scroll as an enhancement.<\/strong> Let JavaScript fetch and append batches for humans, but update the address bar with the History API (pushState<\/code>) so each position maps to a shareable, crawlable URL. Google's infinite scroll search-friendly recommendations<\/a> describe exactly this component-page approach.<\/li>\n
    4. Make each component page self-sufficient.<\/strong> It should render its items, a title, and links forward and back\u2014so it stands alone if a crawler lands on it directly.<\/li>\n
    5. Don't rely on rel="next"<\/code>\/rel="prev"<\/code> for indexing.<\/strong> Google confirmed in 2019 that it no longer uses those tags as an indexing signal; the crawlable <a href><\/code> links are what carry discovery now.<\/li>\n<\/ol>\n

      Do this and Googlebot, GPTBot, ClaudeBot, and PerplexityBot all get the same thing: a chain of static URLs they can crawl to the end. The smooth scroll is a bonus for people, not a dependency for machines.<\/p>\n

      Do XML sitemaps fix infinite scroll for AI crawlers?<\/h2>\n

      A sitemap helps crawlers discover URLs, but it does not create them.<\/strong> If your deep pages have no standalone URL\u2014because infinite scroll injects them client-side\u2014there is nothing to list. Sitemaps supplement crawlable pagination; they never replace it.<\/p>\n

      Once your component pages have real URLs, list every one in an XML sitemap so crawlers get a second discovery path beyond on-page links. Two caveats matter for AI search: each URL must still return server-rendered content when fetched directly, and AI crawlers lean far more on following in-page links than on parsing sitemaps. Treat the sitemap as a backup to real <a href><\/code> navigation, not a substitute for it.<\/p>\n

      Product archives vs. article archives: different stakes, same fix<\/h2>\n

      Both archive types break the same way under infinite scroll, but the business cost differs. Product archives lose shortlist inclusion; article archives lose citation coverage. The fix\u2014paginated URLs behind the scroll\u2014is identical.<\/strong> Choose the pattern by the stakes, not the aesthetics.<\/p>\n

      For a product archive<\/strong>, invisible deep inventory means AI assistants building a recommendation shortlist only ever "know about" your first dozen SKUs. When a buyer asks an assistant for options in your category, the products stuck on page 40 cannot be recommended because no crawler ever reached them. If you care about showing up in the high-intent prompts buyers use to request product recommendations<\/a>, every product needs a crawlable URL, and a numbered paginated series is the cleanest way to guarantee it.<\/p>\n

      For an article archive<\/strong>, the loss is evidence. Your best proof points\u2014benchmarks, case studies, technical explainers\u2014are exactly the pages an answer engine wants to cite. If they sit past the first infinite-scroll batch with no linked path, they never enter the retrieval pool, no matter how strong they are. Clean, linked URL patterns are what keep that evidence reachable.<\/p>\n

      One practical note:<\/strong> if your archive is genuinely a live feed\u2014short social posts, a chat log\u2014where individual items aren't worth indexing and search isn't a traffic source, pure infinite scroll is defensible. Everywhere else, it costs you discoverability.<\/p>\n

      How infinite scroll caps your AI share of voice<\/h2>\n

      Pages a crawler can't reach can't be cited. So an archive hidden behind infinite scroll quietly caps your ceiling in AI answers\u2014your brand competes for AI-generated shortlists with only a fraction of its real evidence.<\/strong> You won't spot this in a rankings tool; you spot it in what the engines actually say.<\/p>\n

      Answer engine optimization and generative engine optimization share one prerequisite: evidence has to be retrievable before it can be quoted. When an archive is trapped behind scroll, your citations skew toward the handful of pages that sat in the first batch, and competitors with deeper, crawlable libraries win the rest. Tracking which of your URLs actually get surfaced<\/a> across ChatGPT, Perplexity, Gemini, and Claude turns an invisible technical flaw into a fixable list\u2014and often reveals a whole section of your site going unmentioned because of one crawl gap like this.<\/p>\n

      \"Decision<\/figure>\n

      A quick pre-launch checklist<\/h2>\n

      Before you ship an archive, confirm a non-rendering crawler can walk it end to end.<\/strong> Run through these five checks:<\/p>\n

        \n
      • Raw HTML holds real content<\/strong>, not an empty shell hydrated by JavaScript.<\/li>\n
      • An <a href><\/code> link to the next page exists<\/strong> in that raw HTML\u2014verified with curl<\/code>, not DevTools.<\/li>\n
      • Each page has a distinct, stable URL<\/strong> you can paste into a browser and load directly.<\/li>\n
      • Component pages are server-rendered<\/strong> and readable with JavaScript disabled.<\/li>\n
      • The scroll or "load more" experience is an enhancement<\/strong> layered over the links, never a replacement for them.<\/li>\n<\/ul>\n

        If all five pass, both Google and the AI crawlers can reach your full archive. If any fail, you've found where your evidence disappears.<\/p>\n

        Frequently asked questions<\/h2>\n

        Do AI crawlers execute JavaScript to load infinite scroll content?<\/h3>\n

        No. As of 2026, the major AI crawlers\u2014GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, and PerplexityBot\u2014do not execute JavaScript. Vercel's data shows they sometimes download<\/em> script files but never run them. Because infinite scroll depends on JavaScript firing on a scroll event, its deeper batches are invisible to these bots. Only content in the raw HTML response is seen.<\/p>\n

        Is a "load more" button better than infinite scroll for AI crawlers?<\/h3>\n

        Only if it's built as a link. A "load more" <button><\/code> with a JavaScript click handler is exactly as invisible as infinite scroll, because crawlers don't click. A "load more" control implemented as an <a href="\/page\/2\/"><\/code> that script enhances for a smooth experience is fully crawlable, since the anchor and destination URL exist in the HTML before any script runs.<\/p>\n

        Does using paginated URLs hurt the user experience?<\/h3>\n

        No\u2014you can have both. Ship real paginated URLs for crawlers and layer infinite scroll or a "load more" interaction on top for humans, updating the address bar with the History API as users scroll. Users get the smooth feed; crawlers get a chain of static URLs. This progressive-enhancement approach is Google's recommended way to keep infinite scroll search-friendly.<\/p>\n

        How do I know if my archive is already invisible to AI crawlers?<\/h3>\n

        Run the raw-HTML test: fetch the page with curl -A "GPTBot"<\/code>, count the items in the response, and compare that to what loads in a browser after scrolling. If the raw count is a small fraction of the rendered count and there's no <a href><\/code> to the next page, AI crawlers only see that small fraction. The gap is your invisible inventory.<\/p>\n

        Are rel="next"<\/code> and rel="prev"<\/code> still needed for pagination?<\/h3>\n

        They're optional. Google confirmed in 2019 that it no longer uses rel="next"<\/code>\/rel="prev"<\/code> as an indexing signal, and AI crawlers don't depend on them either. What carries discovery today is crawlable <a href><\/code> links between distinct, server-rendered component-page URLs. Keep the tags if they serve your other tooling, but don't rely on them to expose your archive.<\/p>\n