Case Study: A/B Testing with Google Optimize

A/B Testing is the best way to test your website re-design hypotheses. Let’s look at how to process looks in practice using Google Optimize for a Magento 2 store. You’ll get to see the practical steps that needs to be take as well as common issues that you might run into.

Note: This is written as it was developed for a Magento 2 store, but the idea can be applied anywhere.

Recently a company requested to redesign their PDP page and to find a solution for A/B testing the new implementation of the page. If you’ve missed what is A/B testing for some reason — it’s a technique of increasing sales or other metrics on the website by testing multiple versions of the same page, one being the current, existing page, the other one being the one with changes to be tested. Client traffic is then spread in an even and random fashion between both page versions and on these versions conversion or the metric of choice is being tracked separately.

We decided to share this, as it might be useful for you, or for a client you are working with.

The company had already done some A/B testing using Google Optimize’s style injections, for example, “Add to cart” button style testing. However, in this case, simple CSS injections were not enough as layout modification was also required. Dropdowns were to be converted into clickable buttons, picture gallery configurations had to be changed, some blocks had to be modified, removed and/or added, and lots of elements required new styling.

The entire page had to be reworked. Obviously, just JS/CSS injections wouldn’t have cut it, this required some thought put into it.

A/B Testing with Google Optimize: Project Requirements

• Ability to test two different versions of pages
• Page versions must differ not only with CSS & JS (native Google Optimize experience) but also may require layout & functionality modifications
• Must work properly on the existing multi-instance infrastructure
• A/B traffic splitting must feel jitter-free

Figuring Out A/B Testing Specifics

It became obvious quite soon, the server will have to be the one to track and switch page versions with all the necessary changes.

We knew Magento 2 Enterprise edition had a native integration with some Google services to enable A/B testing. We considered a more barebones custom implementation of that feature, which might’ve been nice, but the required time to implement this was too much for us in this case.

Another idea we were considering — splitting traffic right on the servers and handle slightly modified versions of the application. However, that might cause issues with the multi-instance server setup. And that created more issues — we would have to create a custom way of tracking the activity and gather the results.

After some back and forth with their dedicated developer, we came up with a simple concept — we use Google Optimize to target the Product Details Page and redirect half of the traffic to the same URL, but with an additional query parameter. On the server side, it was enough to check for additional query parameter to decide on which layout to load.

Developer’s note: On the server side we can do much more than just reload layouts, however, it was absolutely enough for the task and caused no problems from the Magento side. Conveniently there’s an event called ‘layout_load_before’, where we can append the layout changes we want. Even more conveniently, we use Magento’s layout file-handles to load the appropriate layout files, which is really handy.

Thereby we eliminated the need to reimplement traffic splitting or any data analysis tools and we get a lot of fun delivering the request.

Example of A/B Testing Implementation in Google Optimize

1. The developer creates a layout e.g. catalog_product_view_v_orange.xml, where changes to the PDP layout can be added (this works just like when extending core layouts, we just append a set of changes).
2. The developer can, for example, load another CSS file in the layout, where he sets the color of all text elements to be orange.
3. The marketing department targets the PDP page (Google Optimize offers all kind of ways to properly mark request: query params, URL regex matching, checking cookies, you name it!
4. The marketing department splits the traffic 50/50 and redirects 50% of the customers to the same page with an additional query parameter, e.g “..?v=orange”.

Is the A/B Testing Implementation in Google Optimize the answer to everything?

Nope, issues can still be encountered in this process, but if everything is done right, one can get an A/B test integration tool working just fine.

Possible A/B Testing Issues in Google Optimize

Google Optimize redirection capabilities are quite limited — it can not just attach a query param, so the initial URL has to be carefully tracked to properly apply the argument (? vs &)

In case a page layout uses configuration values from a database, one might have to hardcode them for one of the A/B test parts because two values for the same config can’t be held in the database (although this can be easily implemented or creatively worked around).

One must remember that after the testing a switch from the already existing “default” page to the new tested page might have to be performed. If there are a lot of changes one needs to think ahead about how these changes can be written so that they can easily be set as the default when the client decides that the test was successful.

Need help running A/B tests for your website? Feel free to drop us a line at [email protected] or check out our Conversion Rate Optimization!

Related articles:

• Live-mode dashboard for A/B Test results with Data Studio
• How to approach A/B testing & CRO for a B2B website
• What are conversion optimization experts talking about besides A/B tests?

Share on: