Why no Shopify A/B testing app can edit your theme — and what that means for your tests

Protected scopes, plainly

Reading a Shopify theme is unrestricted. Any app can request `read_themes` and get it. Writing is different: `write_themes` — the scope behind the Asset API's `PUT` and `DELETE` and the `themeFilesUpsert` mutation — is a protected scope. An App Store app must apply for an exemption and be granted it before it may write a single theme file.

Shopify publishes which kinds of app the exemption is for. There are four categories:

• Page builders that replace all layout and template files.
• Backup and restore apps.
• Apps that add ratings, buttons or badges to recurring resource lists.
• SEO, content-locking and developer-tooling apps.

How the rule is actually enforced

Rules on a documentation page are one thing; review decisions are another. Two public threads on Shopify's developer community show the rule being applied to apps that look a great deal like this category.

In the first, a developer describes an app that identifies conversion issues — broken forms, mispositioned calls to action — proposes targeted fixes to existing theme files, and has the merchant review a diff and approve each individual change. That is about as careful and merchant-controlled as a theme write can be made. The exemption was denied.

The same thread contains the follow-up question every developer in this category eventually asks: is a merchant-approved, single-file write ever approvable? Shopify staff answered: no — a merchant-approved write to an existing theme file still requires approved access to `write_themes`. The recommended path was theme app extensions and app embeds, or collaborator access.

Creating a new theme does not escape it

The obvious workaround is to leave the merchant's theme alone and create a fresh one: duplicate the live theme, apply the variant to the copy, let Shopify serve whichever copy the visitor should see. Several apps in the category work broadly this way, and it is a good design on paper.

A second public thread records what happened to an app that tried it through the front door. It had `themeCreate` and `themePublish` approved on the exemption form — and was then rejected at app review. The reviewer's position was that the exemption covers editing existing themes through the Assets API, not creating new ones, and that distributing a new theme was a separate store-requirements violation.

So both halves close: writing into your theme needs an exemption this use case does not qualify for, and creating a theme instead has been rejected at review even with the form approved.

Two more constraints worth knowing

Two smaller platform facts shape what any theme-based testing app can do, and merchants are rarely told either one before they buy.

• Theme budget. A Shopify store can hold 20 themes on Basic, Grow and Advanced, and 100 on Plus. Your own themes — the live one, the one from the last redesign, the seasonal one nobody deleted — consume the same budget. An app that creates one theme per variant is spending a resource you also need.
• No webhook fires when theme files change. Shopify's `themes/create`, `themes/update` and `themes/delete` webhook topics each state explicitly that they do not fire when theme files are created, updated or deleted, and `themes/publish` fires only when a main-role theme is published. There is no event for “the merchant edited their theme code.” Any app that claims to detect theme drift is polling for it, because polling is the only mechanism that exists.

What this means for you

Strip away the platform vocabulary and one practical fact remains. An app that never writes to your theme cannot leave code behind, cannot break your theme, and uninstalls completely — not because its developers are careful, but because it has nothing installed in your theme to fail at removing.

That is worth more than it sounds. Across the 73 negative reviews we coded in this category, 7% describe code left behind after uninstall and 5% describe an app breaking the store or theme. Twelve percent of everything merchants complain about is an app changing something it should not have. Flicker — the failure mode this category actually markets against — is one review out of 73.

The rule that frustrates vendors is, from where you sit, a consumer protection.

What it costs

Now the honest half, because an app that cannot touch your theme genuinely gives things up.

The compliant delivery surface is a theme app extension — an app embed block that renders alongside your theme, in the first byte of HTML, but never in place of your theme's own markup. The documented ceilings are real: an app embed can target `head`, `body` and `compliance_head` only, cannot render on checkout step pages, works within 100 KB of Liquid across all its files, and is held to 10 KB of JavaScript measured uncompressed — 10,000 raw bytes, at Error severity. There is also a step vendors rarely mention in an install flow: an app embed block is deactivated by default after install, and the merchant has to switch it on in theme settings before it renders anything at all.

The consequence for what a test can change is mechanical. Our variants are built from six operations. Two of them — hiding an element and restyling one — become CSS rules that ship in the first byte and are applied by the browser's own style engine at parse time. The other four — changing text, replacing markup, and inserting content before or after an element — modify markup that the theme already rendered, so they execute in the browser after the HTML arrives. And none of the six can place new content at an arbitrary slot in your theme; only a merchant-placed app block can do that, and that is a different artifact.

In plain terms: whole-theme testing and template testing are off the table for us, and if that is the test you want to run, an app that integrates with your theme is the correct purchase. What you get instead is an app with no ability to damage the thing it is testing.

The question to ask any vendor

You do not need to memorise scope names to use any of this. One question does most of the work: does this app write to my theme files, and if so, what exactly does it write and what removes it?

A vendor who writes to your theme should be able to answer precisely and should say so without hedging — it is a legitimate design with real advantages. A vendor who cannot answer precisely has not thought about your cleanup, which is the specific failure 12% of this category's negative reviews are about.

Sources

• Shopify — Asset API (legacy) restrictions and protected scopes
• Shopify community — “Theme API write exemption denied: is single-file, merchant-approved writing ever approvable?”
• Shopify community — clarification on the theme API exemption for creating vs. editing themes
• Shopify Help Center — theme limits per plan
• Shopify — WebhookSubscriptionTopic (theme topics and their exclusions)
• Shopify — theme app extension framework and its limits