Experimentation / CRO

Experimentation and Testing Programs acknowledge that the future is uncertain. These programs focus on getting better data to product and marketing teams to make better decisions.

Research & Strategy

We believe that research is an integral part of experimentation. Our research projects aim to identify optimization opportunities by uncovering what really matters to your website users and customers.

Data and Analytics

90% of the analytics setups we’ve seen are critically flawed. Our data analytics audit services give you the confidence to make better decisions with data you can trust.

Internal Promotions vs Third-Party Banner Advertisements [Original Research]

We were asked recently about the effects of using internal promotions (e.g., a discounted product sold within the site) vs. third-party (from an outside business) banner advertising on web site clarity and visitor perceptions.

Our first study used the five-second test to examine whether ads on website homepages distract visitors from understanding a site’s purpose. This follow-up study looks for differences in user perceptions between ad types: internal promotions versus third-party ads.

Results summary

The five-second tests yielded some interesting results:

  • Respondents recalled seeing third-party ads more often than internal promotions.
  • The webpage with internal promotions was rated as higher quality and more attractive than the web page with third-party ads.

The effects seen here are limited to the treatments used. We're looking soon to expand this test across hundreds of ads to normalize for ad type and presentation.

Background

If a website homepage is cluttered with irrelevant information, first-time visitors may become distracted or overwhelmed. More time is spent trying to simply figure out the purpose of a site and, if the page is confusing enough, visitors will abandon a website when this task becomes too difficult. This issue can lead to short and long-term decreases in sales.

While we can now safely say that third-party advertisements are considered “clutter”, we wonder whether internal promotions are also seen as advertisements.

Internal promotions, like a storewide 30% discount, are advertisements for the website the visitor is already on. They don’t intend to take users to a different site, but to encourage their current exploration.

However, internal promotions often look identical to third-party ads. Our question is this: Do users distinguish between internal promotions and third-party ads, or do they look at internal promotions and see an ad?

Study Report

Data Collection Methods and Operations:

We used the homepage of Colonial Candle as we thought it provided the perfect test subject in terms of not being well-known to a broad audience and with a relatively vague homepage.

The thought here was that we didn’t want it to be too easy to gather what the website was selling with just 5 seconds to view the page.

Here are the two variations tested where you can see the addition of large, easily seen banner ads in the right sidebars:

Variation with 2 ‘internal’ ads, promoting a product within the site.
Variation with 2 ‘external’ ads, promoting services or products external to the website.

Participants – Using UsabilityHub.com, we ran five-second tests on 800 participants (400/variation).

We asked three questions:

  1. What is this website selling?
  2. How many advertisements did you see?
  3. Rate the quality and attractiveness of this page 1 (worst) and 5 (best).

These questions were intended to gauge the difference in perceptions of third-party ads versus internal promotions, and to see if either had an effect on perceptions of the entire web page.

Findings

1. “What is this website selling?”

A Chi-square goodness of fit test told us that there was no difference between variations in the percentage of people able to identify ‘candles’ as the primary product class being sold by the website: 59% for the internal promotion variation and 60% for the third-party ad variation.

2. “How many advertisements did you see?”

We compared mean responses between the variations using a t-test:

number of ads results
Visualization of the respondent data for the survey question, “How many advertisements did you see?”

The two third-party ads increased the number of ads that respondents recalled seeing after the 5-second test.

The results for this survey question reveal that participants considered the internal promotions to be advertisements. Out of the two internal website promotions displayed, participants reported seeing an average of 1.76 ads.

3. “Rate the quality and attractiveness of this page 1 (worst) and 5 (best).”

We compared mean response between the variations using a t-test:

quality_results
Visualization of the respondent data for the survey question, “Rate the quality and attractiveness of this page 1 (worst) and 5 (best)”

Participants who saw the homepage with third-party ads found the page to be less attractive than the homepage with internal promotions.

Limitations

This test is very limited in its ability to extrapolate findings, as we tested just two ads on one website. Survey question three on the quality and attractiveness of the web page would surely fluctuate with different websites and ads.

Maybe people really don’t like American Express. Perhaps they found the third-party ads unattractive or tacky. There’s a possibility that different ads —or different products being advertised— would elicit different results.

We’re looking forward to community ideas on how we can develop this study for a follow-up.

While the majority of participants considered internal promotions to be advertisements, it’s possible that they answered this way simply because the survey question asked about advertisements. If the question had asked about promotions, perhaps participants would’ve responded that they saw two promotions.

Conclusion

While there could be many confounding variables, we found that, in this case, people recalled viewing more ads when they were third party vs. internal promotions. However, they rated the variations with internal promotions as more attractive.

Related Posts

Who's currently reading The Experimental Revolution?