In Short:  Online testing tools typically analyze test data against all traffic that has been exposed to a test experience whether test entrants actually see the tested area or not.  It’s sometimes good to additionally analyze the data against only those that see (scroll to) the tested component within the experience.  We take a deep dive to understand where and when to do that and touch on a couple of platforms that make this easy to do.

Analyzing the holistic test experience by default

Since the early days of AB and multivariate split testing being applied to online marketing, many platforms that facilitate such experimentation have popped up with great success.  Some have bells and whistles that others don’t have, but one of the commonalities they all have is that the basis for KPI measurement is usually on the number of visitors that have been served an overall experience, rather than those that have actually seen a specific component (that is getting tested, for example).

Let’s take an example from an experiment we are running for a brand we currently work with.  We are testing content on their home page, where the content is well below the page fold.  While it’s always best to focus testing on content that sits above that page fold – the area that visitors can instantly see, without having to scroll – there are times where the underlying content is also important, and therefore must be experimented with.  Now using tools like Contentsquare, Hotjar, Crazy Egg and others, it is quick and easy to see how the visitor populations in these lower page depths decrease, the farther down the page you go.  In our case, we used Hotjar to see that the area we wanted to test had less than a quarter of home page visitors scrolling to the component we set out to test.

What this means is that, based on how testing tools function by default, in this scenario, over 3 in 4 of the visitors that are analyzed in the test’s dataset do not even have a chance to see what is being tested.  At quick glance this sounds counterintuitive. Right?  Yet it is important to see both the big picture and how all of the pieces and parts of an experience interact with each other to produce a result.  Alexandre Anquetil, who manages North American customer engagements for our partner AB Tasty, puts it well: “Testing platforms voluntarily base KPIs on test exposure rather than content interaction. It is necessary to measure the overall impact to identify halo and side effects. If you narrow down tracking to what you expect only, you may miss lots of insights.”

In the current test we are doing, we decided to look at both approaches so we can get an understanding if the noise caused by the majority of visitors not being able to see the tested changes, is really that meaningful, or not.  What we are finding is that the results tied to the metrics that focus on click engagement around the tested component are very consistent when comparing the two uses cases – one being where the data set is all home page desktop visitors and the other is only desktop visitors that actually scroll and see the tested component.  Where there are differences is when you focus on the website’s north star metric – in our case, contact sales form submissions.  When we take a step back, this all makes sense.  Data that is specific to a particular part of the page will likely be similar in the two scenarios.  But with these north star metrics on websites, whether it is orders, revenue, email sign ups or leads generated, it is important to take caution when trying to evaluate cause and effect between a micro change on a low lying component and that uber metric for the website.  Our exercise of looking at data through these different lenses reinforces this point.

Taking a more filtered approach

What they see is what you measure.  Or is it? Sometimes it’s best to take a look at the entire data set and narrow your dataset to those that have a chance to see your content.  Pick your spots to do just that so you can understand the differences.  Both AB Tasty and Convert, another partner (and the tool we are using for our current experiment), have well-documented, turn key solutions to filter out the noise and focus on only those that see your tested component.

Lower lying content on home pages – pages that tend to get a lot of traffic overall but have significant dips based on different scroll depths – are ripe opportunities for the dual exploration.

When testing on pop up forms where only the content within the form pop up is being tested, we like to use the unique form views as the basis for analysis, instead of any unique visitors for the underlying page(s) that the form was spawned from.  In this sense, the test is really on the form and the unique visitors are those that invoke the form.

Pick your spots for more refined filtering when it makes sense to in your testing program – ask your testing vendor if they have a recommended filtering approach that works with their platform.