High Score Labs News   •   Sep 27, 2022

There are two major objectives when running tests in Optimizely; to experiment or to optimize.

• Experimenting attempts to validate a claim that a variation is unique or fundamentally different based on its statistical significance, with the intent to apply the results of the experiment to future deployments or experiments.
• Optimization involves generating as much lift as possible from a set of variations within a set time frame. The process is bound by a set-it-and-forget-it algorithm and ignores statistical significance in favor of maximizing impact temporarily.

Multi-armed Bandits and Stats Accelerator are two of the most common testing platforms out there. Both generate wonderful insight and impacts when used right. Thus, it’s important to know when to use each exactly. Let’s briefly examine their major differences.

Multi-armed Bandits

Multi-armed bandits help you maximize the performance of your most effective variation by dynamically re-directing traffic to that variation. In the past, website owners had to manually and frequently readjust traffic to the current best performing variation.

However, MAB does this automatically throughout the experiment lifecycle. The better a variation does; the more traffic MAB sends its way.

Stats Accelerator

Stats Accelerator uses a set of algorithms to determine which variation has the greatest effect size, i.e., the furthest difference from baseline, in much less time than most other programs, while keeping false discovery rate (FDR) below a set threshold. Basically, it reduces the time it takes to find the variation with the most statistical significance.

Now let’s examine some specific use cases for MAB and Stats Accelerator.

When to Use Multi-armed Bandits

MAB answers the question “Which variation is showing the largest reward?” Its goal is to maximize conversion within a short period. Therefore, consider using Multi-armed Bandits for:

• Promotions

Promos and special offers are short-lived campaigns to boost conversion rates, so it’s obvious why MAB is the best fit here. During the duration of the campaign, MAB will consistently direct more traffic to the variations that convert best.

• Events

MAB will boost signups for your webinars, free classes, and other events by experimenting with variations of your landing pages and driving more traffic to the versions that prove most effective.

• Headline testing

Headlines occupy prominent space on your website for a short period. If you’re a news website, for instance, a headline will only be relevant for about a day, less at times. MAB lets you maximize the power of your headlines during their lifespan.

When to Use Stats Accelerator

Stats Accelerator basically answers the question, “Which variation is most different from baseline, and has the greatest potential for performance?” It’s best suited for:

• Driving traffic to specific pages

You can increase click rates simply by changing the position of a button or section. Stats Accelerator lets you test these little tweaks and reap actionable insights.

• Landing Pages

Different variations of CTAs, landing page copy, design, and concept can be tested at once. The most effective combinations can then be used to increase conversion rates.

• Add-to-cart rate
• CTAs
• Search results optimization

Conclusion

Multi-armed Bandits and Stats Accelerator are amazing A/B testing tools. However, you’ll get the best results when you apply each to the right case. Use the information provided in this guide to decide on your next project.

Contact us today!