Platform
What actually causes A/B test flicker on Shopify — and what a theme app extension can and cannot fix
Cem Bilen, Founder · August 1, 2026 · 8 min read
Flicker is the flash of your original content before a test's variant replaces it. Every vendor in this category benchmarks against it, and it accounts for one review out of the 73 negative reviews we coded across the category. We are writing about it anyway, because the mechanism is genuinely interesting and because the honest version of the story has a seam in it that most vendors do not mention.
This one is for developers and technical merchants. If you just want to know whether an app will break your store, the uninstall post is the more useful read.
Where the milliseconds go
A client-side A/B test does four things between the page arriving and the visitor seeing the variant: fetch the testing script, fetch the experiment configuration, decide which arm this visitor is in, and apply the change to the DOM.
Only the last of those changes anything visible, and by the time it runs the page has usually already painted. So testing tools cover the gap with an anti-flicker hide: a rule that makes the page — or the affected part of it — invisible until the change has been applied, with a timeout so a failure does not leave a blank store.
Ours hides the document and is time-boxed at 1.5 seconds worst case. That number deserves to be said out loud rather than buried: on a slow connection, an anti-flicker hide is not free. It converts a visible flash into a visible delay. It is the better of two bad options, not a solution.
The config round trip is the actual cause
Of the four steps, the one that costs the most is fetching the experiment configuration. It is a cross-origin request to the vendor's API, it cannot start until their script has loaded, and its latency is whatever the network decides today.
Remove it and most of the problem goes with it. That is what a theme app extension makes possible: the approved variant payload is written into app-owned metafields, and the extension's Liquid inlines it into the page's first byte. The browser never asks anyone for the configuration, because it already has it. No round trip, no waiting, no reason to hide anything while waiting.
This is the real argument for delivering a testing app through a Shopify app rather than a pasted script, and it is a good one.
The seam: the content is server-rendered, the assignment is not
Here is the part that gets skipped. Removing the config fetch does not make the split server-side.
Liquid's `request` object exposes `host`, `origin`, `path`, `page_type`, `locale`, `design_mode` and `visual_preview_mode` — and no cookies, no headers, no query parameters, no visitor identity of any kind. A theme app extension therefore cannot know who the visitor is, which means it cannot decide which arm they belong to. Shopify's own Rollouts can, because Shopify is in front of that constraint. An app is not.
So the payload arrives server-rendered and the choice between arms still has to be made in the browser. No app in this category can honestly claim that Shopify itself picks the arm, ours included. What is true is narrower: the content ships in the first byte, and the selection runs on the visitor's machine.
And Shopify closes the obvious way to do it
The natural fix for a browser-side selection is to run it as early as possible: a render-blocking script in `head`, resolving the arm before the body is parsed. That door is closed, and it is worth being specific about how, because it is closed at build time rather than at review.
- `ParserBlockingScript` is an Error-severity Theme Check, enabled by default, and it flags any script tag carrying `src` without `defer` or `async`. Worth knowing precisely: `shopify app build` reports the offense but does not fail — it writes Theme Check output to stdout and exits zero. Enforcement happens server-side at deploy, where the extension silently declines to update. A render-blocking external script in `head` is therefore not a reviewer's judgment call; it is a rule that stops your extension shipping.
- The sanctioned route gives you no control over the tag. Scripts declared through an app block's schema are injected by Shopify with `defer`. You do not author the tag, so you cannot omit the attribute.
- `defer` is too late for the mechanism that would be needed. Deferred scripts run after document parsing completes, so a MutationObserver started then has nothing left to intercept — every node it wanted to rewrite before paint is already parsed.
- Built for Shopify caps the damage from the other side: an app must not reduce the storefront Lighthouse score by more than 10 points, measured before and after install as a weighted average across home, product and collection pages.
And Shopify's App Store quality guidance treats flicker itself as a defect — UI is buggy if content flickers, repeatedly loads in and out, or causes surrounding content to shift. Which leaves apps in this category between a rule that forbids the pre-paint hook and a rule that penalises the flicker the hook would prevent.
There is one gap left, and we would rather describe it accurately than sell it. Theme Check only lints script tags that include a `src` attribute, so an inline script inside the app embed's own Liquid is outside the check — and an inline script in `head` runs before the body is parsed, which is precisely the window that is needed. That is an absence of a rule, not a permission. We have not verified it on a development store, we do not know whether it survives review, and until we do we are not making that claim on the strength of it.
The documented ceilings
Independent of the flicker question, an app embed has hard limits that determine what a test can do at all:
- Targets are limited to `head`, `body` and `compliance_head`. `content_for_header` and `content_for_layout` are off limits, as are most section properties.
- Blocks cannot render on checkout step pages.
- 100 KB of Liquid across all the extension's files, and 100 locale files.
- 10 KB of JavaScript, measured uncompressed — 10,000 raw bytes, at Error severity. Bucketing, exposure logging and applying the change all have to fit inside that, which is a much tighter budget than it sounds when you are used to thinking in gzipped kilobytes.
- An app embed cannot modify existing theme Liquid. It renders alongside your theme, never in place of it.
- App embed blocks are deactivated by default after install. Until the merchant activates the block in theme settings, the extension renders nothing — which means an experiment can read as running while serving nobody, and any honest install flow has to check for it rather than assume it.
What that means for what a test can change
Our variants are built from exactly six operations, and the delivery mechanism each one needs is a function of the operation, not of the page:
| Operation | How it reaches the page |
|---|---|
| Hide an element | A CSS rule in the first byte. Applied by the browser's style engine at parse time. |
| Restyle an element | A CSS rule in the first byte. Same mechanism. |
| Change text | Executes in the browser against markup the theme already rendered. |
| Replace markup | Executes in the browser. |
| Insert before an element | Executes in the browser. |
| Insert after an element | Executes in the browser. |
Two of six ship as CSS in the first byte, which is the only mechanism here that deserves to be called server-delivered. The other four modify markup that already exists, so they run on the visitor's machine. And none of the six can place new content at an arbitrary position in your theme — only a merchant-placed app block does that, and none of our operations model it.
Which produces an uncomfortable but important detail: changing a headline, the most common CRO edit there is, is in the second group. The common case is not server-rendered content. Anyone in this category telling you otherwise is describing the two easy operations and calling it the product.
Why we wrote this anyway
Flicker is one of 73 complaints in this category. Merchants are not lying awake about a 200-millisecond flash; they are lying awake about an app breaking their store. We wrote this because it is true and because the engineering is worth documenting, not because it is what anyone is afraid of.
If you are evaluating apps on flicker benchmarks, the more useful question is the one two posts over: what does this app write into my theme, and what removes it?
Sources
See it on your own store
Looplift runs this methodology on your site automatically: a free audit, three ready-to-launch experiment proposals, peeking-safe results. You approve every launch.
Run a free audit