Shopify SimGym Review: What AI Shopper Testing Tells You Before a Redesign (and Where It Is Wrong)

You have a redesigned theme sitting in drafts. It took weeks, it looks better, and nobody can tell you whether it will sell better. The usual answer is a real A/B test, which means more weeks and enough traffic to reach significance. So when Shopify offers a tool that runs AI shoppers through both themes and gives you a verdict in about 15 minutes, it is tempting to treat that verdict as the answer.

This Shopify SimGym review is written from the agency side. We have spent 12 years building and optimising Shopify stores from Glasgow, and we have used SimGym in real redesign workflows. It is useful. It is also wrong often enough that you need to know exactly what it measures before you let it make decisions for you.

What Shopify SimGym actually is

SimGym is a free Shopify app, currently labelled an AI Research Preview. Shopify opened it to all eligible merchants in March 2026 with no waitlist. You pick your live theme and a draft theme, and SimGym sends a set of AI shopper agents through both. Each agent is modelled on a human-like profile with a shopping goal. The report tells you which theme the agents were more likely to add products to cart on, where they got stuck navigating, and a list of recommendations with the shopper feedback behind each one. You can also run a single theme on its own for qualitative feedback.

Eligibility is narrower than the marketing suggests. According to Shopify's SimGym documentation, your store needs a Liquid storefront (Hydrogen and headless builds are not supported), an active plan, Shopify Network Intelligence switched on, at least one published product with a price above zero, and no permanent password page. For a comparison you also need a draft theme alongside the published one.

What the AI shopper simulation measures (and what it does not)

The output is directional. SimGym tells you whether the draft theme makes it more or less likely that a shopper adds to cart, and it flags navigation friction. That is the whole scope. It does not predict conversion rate, revenue per visitor or average order value. It does not test pricing, discounts or offers. And because it compares whole themes, you cannot test one hero section in isolation without duplicating the theme and changing only that section.

Shopify's own research gives a sense of how far to trust the direction. Across 50 stores, SimGym agreed with the direction of real shoppers' add-to-cart change about 77% of the time, with a stated range of roughly 66% to 87%, and that figure covers visual theme changes only. Read that the right way: on roughly one redesign in four, the simulation points the wrong way. That is a lot better than a coin toss and a lot worse than a properly run test.

SimGym is a friction detector, not a forecast. Treat the add-to-cart verdict as a hypothesis and the shopper feedback as the useful part.

Where SimGym gets it wrong

The app store rating at the time of writing sits around 2.8 stars from a few dozen reviews, which is low for a Shopify-built app. The complaints are specific and they line up with what we have seen in our own runs:

Agents misreading variant selectors and reporting a purchasable variant as out of stock.Feedback on elements that were not changed between the two themes, including claims that prices were hard to find on a page where nothing moved.At least one merchant running a live A/B test alongside SimGym reported that the simulated result contradicted real traffic.Reliability issues in the preview: crashes and session replays that would not play.

Expect the same trouble anywhere your theme relies on hover states or JavaScript-driven components. Mega menus and drawer carts are the obvious candidates. The agents navigate the DOM, not the screen the way a person sees it, so a menu that opens on hover or a cart that slides in from the side can register as friction that your real customers never feel. If your draft theme introduces either of those, discount the navigation score and read the shopper transcripts instead.

One more structural limit: the simulated shoppers are built from common behaviour profiles. If you sell to a niche audience, a trade customer or a market with different browsing habits, the agents are not your customers. Our clients in Scotland and across the UK often sell to buyers who know exactly what they want and go straight to search. A generic AI shopper browsing collections tells you very little about them.

SimGym vs A/B test: when each one earns its place

The honest comparison is not SimGym versus A/B testing. It is SimGym before A/B testing. The two tools answer different questions, and the cost difference is the point.

Use SimGym when

You have a full redesign in drafts and want to catch obvious friction before anyone sees it.You do not have enough traffic to reach significance in a reasonable time. Many stores under a few thousand orders a month fall into this group.You want qualitative feedback on a draft theme quickly, with no risk to live revenue.

Run a real test when

The change affects pricing, offers, shipping thresholds or anything in checkout. SimGym cannot see these.You need a number you can take to the board: conversion rate or revenue per visitor, not an add-to-cart direction.The simulation result is close or contradicts your own data. A 77% hit rate means a close call is noise.

Shopify now has native A/B testing through Rollouts, which copies your published theme, lets you edit the copy, and splits live traffic between the two while comparing conversion rate and revenue. That is the tool that settles the question. We cover how to run one properly in our practical guide to A/B testing on Shopify, including the traffic you need and the mistakes that make results meaningless.

What a SimGym run costs

Installing the app is free and Shopify gives eligible stores a small allocation of free simulation credits. Beyond that, reports on pricing conflict, which is normal for a research preview. The most consistent picture is one credit per single-theme analysis, two credits per comparison, and roughly $10 per credit once the free allowance is gone. Some sources quote much lower per-run prices. Check the billing screen inside the app before you budget, and treat the number as likely to change.

At anything near $10 a run, SimGym is cheap compared with the cost of shipping a redesign that quietly lowers add-to-cart rate for a month. It is not cheap compared with doing nothing, so use it with intent: one comparison per meaningful iteration, not a run after every colour tweak.

How we use SimGym in a redesign workflow

On theme design and development projects we slot SimGym in at two points. The first is after the structural build, before visual polish: navigation, collection filtering, product page layout and cart flow. A comparison run here surfaces the friction that is expensive to fix later. We read every shopper transcript and ignore the headline score. If three agents could not find the size selector, that is a real finding. If the score says the draft loses by a few points, that is a hypothesis.

The second run is on the near-final theme, and its job is to confirm we have not introduced anything new. Then the theme goes live behind a Rollout at a small traffic percentage, and the real test decides. The simulation shortens the list of things we need to test with real traffic. It never replaces the test.

This matters because most Shopify brands we talk to do not have a traffic problem, they have a conversion problem. A redesign that looks better and converts worse is the most expensive version of that problem, and SimGym is a reasonable early warning system for it. If you are not sure the redesign is needed at all, read our piece on when to redesign versus optimise your Shopify theme first. Often the answer is a programme of smaller conversion rate optimisation changes, each one tested on real traffic.

The takeaway

SimGym is worth installing if you run a Liquid theme and you are about to change it. Use it to find friction and to read what simulated shoppers struggled with. Do not use it to decide which theme wins, and never use it to test anything involving price, offers or checkout. It is right about the direction roughly three times in four. Your real traffic is right every time, it just takes longer.

If you have a redesign in drafts and want a senior team to run the simulation, read the transcripts, and then set up a proper live test, that is the work we do every month inside our Conversion Growth Retainer. Get in touch and we will tell you honestly whether the redesign is the right move.