In Short:  Pop ups are increasingly “popping up” – for Marketers, they often provide the conversion juice they are seeking. But there is a flip slide to it – and one that does not get measured as well – visitor angst.  Here we take a deep dive into both sides of the pop up equation.

We’ve all likely seen an uptick in “pop ups” in our online journeys in recent years.  From newsletter sign ups and first-time customer discounts to free shipping and various other offerings, we are increasingly getting presented with these spontaneous pop-up windows.

Pop ups are distinct from the more passive offers that are built into the web pages, in that they are usually unexpected, and can therefore come across as disruptive.  Despite the annoyances, we see them more, in part, because they are deemed to be successful by digital marketers, who focus primarily on what engagement and value are driven from the pop ups.

Conversion juice on the one hand

As my team helps brands optimize their websites to extract maximum value from them, we often probe on how to best utilize various tactics, including pop ups.  Probing often means A/B split testing (experimentation) or looking at analytics data.

For example, we recently ran an experiment for a customer where we tested the timing logic to trigger the pop up.  In this case, the pop up actually had a short form embedded into it, offering a research report to site visitors and acting as a lead generation mechanism for the brand we were conducting the test for. In one test variation we tested, the pop up triggered by a 5-second delay (time spent on test page), while another test variation triggered the same pop up when the visitor scrolled 50% down the page.

One finding was that the 5-second delay tactic invoked the pop up 33% more than the tactic that relied on the user scrolling halfway down the page.  That’s a nice increase in form views alone. What’s more, the 5-second delay yielded 157% more form submissions per test page visitor than the variation that triggered the pop up by page scroll.  Not only did the more aggressive approach get the form in front of more visitors, but when the form did appear, the form completion rate was 91% greater when doing the more immediate pop up.  Perhaps some added angst from having the pop up appear while the visitor was actively doing something (scrolling) got in the way.  In any case, these are some gaudy numbers from an engagement and conversion perspective.

Just like the cooking kit informercials that never seem to end, “there’s more.”  While we were testing the pop up, we also had the same offer that leads to a form page, built more passively into the same test page (about halfway down the page).  While there was some cannibalization at play, where the pop ups stole attention away from the offer in the underlying page, two things were still very clear from the data.

One, the offer on the page that linked to the form with a button, yielded far fewer overall form submissions than either of the tested pop-up tactics.  Two, the form completion rate, for the offer built into the page, was much higher.  This all makes sense as the pop ups are more aggressive and therefore yield more conversions.  The pop ups are also not user initiated so the rates of form completions that are triggered by the pop ups are naturally going to be lower.  After all, the pop-up form was displayed whether the user wanted to go there or not.

The main takeaway here is that being aggressive with pop ups will often pay dividends from an engagement and conversion perspective. If tested again and again, in different scenarios, the same general finding would likely be true.  Getting things in front of people more often will almost always drive more of whatever that thing is promoting.  But aggressive can also means annoying.

Angst on the other hand

For those that don’t find the pop up offer useful – either based on its content and/or poor timing in their journey – negative sentiment “pops up” in return.  And split testing tools aren’t good at measuring sentiment.  Bounce rates, which happened to be even between the two tested experiences noted above, isn’t great at measuring feelings either. People may not bounce or exit, even if they’re annoyed.  And those that do, may not actually be annoyed.

To better understand this annoyance side of the equation, I turned to some research that is more qualitative in nature – a survey.  To gauge whether someone gets ticked off about something, the reliable way to find out is to just ask them.

With that in mind, unrelated to the aforementioned experiment, I recently surveyed several people across different geographic regions, ages and occupations about how pop ups impact, or do not impact, their feelings about the website and brand they are engaging with.  The exact question: “To what extent does a spontaneous ‘pop up’ offer on a website impact your feelings about engaging with the website/brand?”

The question was presented in a matrix with a few flavors of pop ups probed on – one was an instant pop up; another was a delayed pop up and another was a pop-up on site exit.  One key takeaway from the data is that the vast majority of responders feel pop ups are annoying.  A whopping 89% of the 100 respondents feel the instant pop up is annoying.  Delaying pop ups lower that percentage, only slightly, to 85%.  When the pop up is invoked the pop up until the user show’s intent to exit the site, the annoyance indicator is lowered slightly still, to 73%.

The balance of responders stated that they do not find the pop ups annoying at all.  That would be just 11% for the instant pop ups, 15% for the delayed pop ups and 27% for the pop ups on exit.

So, what does this all mean?  Pop ups can drive great engagement and business value for a brand, but there is clearly a flip side to it.  People don’t like them.  Now the question in the minds of many marketers, is how annoying are the pop ups?  Are they palatable?  Will the visitor want to engage further with the website and stay loyal to the brand behind the website?  Or will they lose desire to engage with the brand?

The survey explored this as well.  We had responders distinguish between being annoyed, but not to the point of affecting their desire to engage with the site/brand and being so annoyed that it would negatively impact their desire to engage with the site/brand.  The results here were interesting.

When experiencing a pop up that appears instantly, 48%, nearly half of all respondents, indicated that they not only get annoyed, but they become less inspired to engage with the site/brand.  Delayed pop ups bring this figure down to 37%.  With the pop-up upon site exit, the figure comes down to 30%.  When we consider the number of visitors that brands subject their pop ups to, these figures translate into a large-scale brewery of negative sentiment.

Finding the balance

How is the right balance arrived at?  Here are some tips, some of which were raised in the survey comments:

As with everything digital, take a measured approach to getting pop ups right.  Maximize engagement and conversion, without being an annoyance to the audience. Understand both sides to the pop-up equation.