llms.txt and robots.txt for Shopify: What Actually Affects AI Crawlers

You have probably been told to add an llms.txt file to your Shopify store. You may also have been told to block AI crawlers in robots.txt before they steal your content. Both pieces of advice arrive with total confidence and almost no evidence.

This is the sober version. What these two files are, which AI crawlers actually read them, what a Shopify store can and cannot change, and the one decision that matters commercially. If you sell online in the UK and your customers are now asking ChatGPT, Claude or Perplexity what to buy, getting this wrong removes you from the conversation.

llms.txt on Shopify: cheap to add, not a ranking factor

llms.txt is a proposed standard from late 2024. The idea is a plain Markdown file at the root of your domain that gives language models a curated summary of your site: what you sell, your key pages, and where to find clean product information. It is the AI-era cousin of a sitemap, except that nobody with a large crawler has agreed to use it.

That last point is the important one. Google's Gary Illyes said in mid-2025 that Google does not support llms.txt and has no plans to, and that AI Overviews rely on ordinary SEO. As of 2026, none of OpenAI, Anthropic or Meta has publicly committed to reading it in production. An SE Ranking study of around 300,000 domains in 2026 found roughly one in ten sites had adopted it, with adoption heavily skewed towards SaaS and technical documentation, not retail. Server log studies of AI bot traffic show requests for llms.txt as a rounding error.

So should you bother? Only if it costs you nothing. Shopify does not let you drop arbitrary files at the domain root, so you need a redirect from /llms.txt or a proxy through an app or edge layer. If that is a ten-minute job for your developer, fine. If anyone is charging real money for it, or selling it as a visibility lever, decline. Put the effort into the things that demonstrably feed AI answers, which we covered in GEO for Shopify: how to optimise your store for AI search.

robots.txt on Shopify: the file that actually matters

Every AI crawler that behaves itself checks robots.txt. That makes it the only file where your instructions to AI companies carry real weight. Shopify generates a sensible default for every store: it blocks checkout, cart, account, admin, internal search and a handful of filtered collection URLs, and it points crawlers at your sitemap. For most stores that default is correct and should be left alone.

To change it you create a file called robots.txt.liquid in the templates folder of your live theme. It is a Liquid template that loops over Shopify's default groups and lets you append rules for named user agents at the end. That means you need code access to the theme, someone comfortable in Liquid, and a plan for keeping the change when the theme is next updated. Shopify's own documentation on editing robots.txt.liquid is blunt: this is an unsupported customisation, Shopify Support will not help with it, and incorrect use can result in loss of all traffic.

That warning is not theatre. A wildcard intended for one bot can block Googlebot from every collection, and nobody notices until sales drop. Treat this file like a checkout change: version it, test it before deploying, and check Search Console coverage the following week.

The AI crawlers you will see in your logs

The mistake most guides make is to treat AI bots as one thing. They are not. Each major provider now runs separate crawlers for separate jobs, and each can be allowed or blocked on its own line in robots.txt.

GPTBot (OpenAI): collects content for training future models. Blocking it has no effect on whether ChatGPT can cite you today.OAI-SearchBot (OpenAI): builds the index that ChatGPT search uses to find and link sources. This is the one that decides whether you appear in answers.ChatGPT-User (OpenAI): fetches a live page when a user asks a question that needs it. User-initiated, not a crawl.ClaudeBot (Anthropic): training data collection. Claude-SearchBot indexes for search quality, and Claude-User fetches pages on a user's request. Same split, same logic.PerplexityBot and Perplexity-User: Perplexity's declared crawler and its live-fetch agent. Worth knowing that in August 2025 Cloudflare delisted Perplexity as a verified bot after finding it fetched pages under a disguised browser user agent when its named bot was blocked.Google-Extended: controls whether Google uses your content for Gemini training and grounding. It does not remove you from AI Overviews or AI Mode. Those draw on the normal search index via Googlebot, so the only way out of them is to leave Google Search entirely.CCBot (Common Crawl): a non-profit archive whose monthly snapshots feed the training sets behind most open models. Blocking it opts you out of dozens of future models at once, but archives already published stay published.

OpenAI publishes its crawler documentation with the exact user agent strings and IP ranges, and Anthropic does the same. Use the official lists, not a blog post, when you write rules.

Training versus retrieval: the distinction that matters

Here is the commercial question hiding behind all the technical detail. Blocking training crawlers and blocking retrieval crawlers are two different decisions, and most robots.txt advice collapses them into one.

Blocking training (GPTBot, ClaudeBot, CCBot, Google-Extended) means your product descriptions and blog posts do not go into the next model's weights. For a publisher whose words are the product, that can be a principled position. For a retailer, the practical effect is close to zero. Nobody is going to buy your jacket from a language model's memory of your copy.

Blocking retrieval (OAI-SearchBot, Claude-SearchBot, the -User agents) is a different thing entirely. Those are the systems that find your product page when a shopper asks which waterproof jacket is best for a Highland walk, and that link back to it in the answer. Block them and you are absent from that answer while your competitor is quoted with a link. Cloudflare's own data from mid-2026 shows search-purpose crawling is a small slice of AI bot traffic, around a tenth of declared requests, with training taking the largest share. So blocking retrieval saves you very little server load and costs you the only AI traffic that actually converts.

For most retail brands, blocking retrieval crawlers is the wrong trade. You give up citations that send buyers to your store and get almost nothing in return.

A default posture for a Shopify store that wants to be found

If your business is selling products rather than selling content, this is the position we recommend to clients in Glasgow and everywhere else.

Leave Shopify's default robots.txt rules in place. They already keep crawlers out of checkout, cart and account pages.Explicitly allow the retrieval and search agents: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. A named Allow line makes your intent clear and survives a later edit.Make a deliberate decision on training crawlers. If you have no strong view, allow them too. If you have a view, block GPTBot, ClaudeBot and CCBot by name and understand that is a statement rather than a revenue move.Do not block Google-Extended expecting to leave AI Overviews. It will not do that.Skip crawl-delay for AI bots. Most ignore it, and rate limiting belongs at the CDN or firewall, not in a text file.Add llms.txt only if it is trivially cheap, and expect nothing from it.

A crawler that is allowed in but finds thin descriptions and no structured data has nothing worth citing. That is a page problem, not a robots problem, and it is the same one that keeps your human conversion rate flat. Our technical SEO checklist for Shopify covers the groundwork.

How to verify what is actually hitting your site

Shopify does not give you raw server logs, which is the honest reason most store owners have never seen an AI crawler. You have three practical routes.

First, if your domain sits behind Cloudflare, its bot analytics break down verified AI crawler requests by name and purpose. Second, check GA4 referrals from chatgpt.com, perplexity.ai and claude.ai. Those visits are what retrieval crawlers eventually produce, and they are the number that matters. Third, ask the assistants directly whether your brand is cited for your category. We covered a repeatable process for this in GEO reporting: how to measure your store's visibility in AI search.

If the referrals exist and the visitors bounce, you have confirmed something more useful than any crawler stat. The AI channel is delivering people and your pages are not converting them. That is a conversion rate optimisation problem, and it is fixable.

The takeaway

llms.txt is a nice idea that nobody important reads yet. robots.txt is the file that counts, and on Shopify it is powerful enough to remove you from Google if you get a line wrong. Block training crawlers if that reflects your values. Do not block retrieval crawlers unless you are happy to disappear from the place your customers are now asking questions.

Most Shopify brands we meet do not have a traffic problem. They have a conversion problem, and AI search is about to send them more visitors with the same unanswered questions. If you want a senior team to set your crawler rules properly, check your theme changes do not break anything, and then work on what those visitors see when they land, that is what our Conversion Growth Retainer is for. Get in touch and we will start with a look at what is actually reaching your store.