Road to A/B testing

Road to A/B testing

Brought to you by Oleksii Vasyliev

What is A/B testing?

First reaction

Rails solutions

• Split
• Vanity
• Field Test
• A/Bingo (dead)
• Bandit (dead)
• Absurdity (dead)
• SevenMinuteAbs (dead)

Split

Variants

ab_test(:user_variations, 'new', 'old', 'strange')
ab_test(:homepage_design, {'Old' => 18}, {'New' => 2})
      

Conversion

ab_finished(:user_variations)

Split Web interface

gem 'split', require: 'split/dashboard'
mount Split::Dashboard, at: 'split'

Split Extensions

• Split::Export - Easily export A/B test data out of Split.
• Split::Analytics - Push test data to Google Analytics.
• Split::Mongoid - Store experiment data in mongoid (still uses redis).
• Split::Cacheable - Automatically create cache buckets per test.
• Split::Counters - Add counters per experiment and alternative.
• Split::Cli - A CLI to trigger Split A/B tests.

Demo for client

Small changes from client

• Support SPA applications (React, Vue.js, Marionette.js)
• Cover only some percentage of visitors to target an experiment
• I need determines when the experiment is shown: geo data, browser/mobile device, third-party cookies, query parameter, etc
• Running simple experiments (text/image variations, etc) without development and deployment
• Support Multivariate and Multi-Armed Bandit testing

Multivariate tests (MVT)

Reaction

Ploblems

• I need better admin system for experiments (editor, settings, etc)
• I need collect info about customers (geo, device, etc) and do not forget about GDPR/Data Processing Addendum/etc
• It will create load on system
• Your customers not need this feature
• You as developer not need this code
• It is only need for marketing/business analytic team

What about PaaS?

• Optimizely
• ABTasty
• VWO
• Omniconvert
• Adobe Target
• etc

PaaS pricing

Me

Google Optimize Integration

ga('create', 'UA-234234-1', 'auto');
ga('require', 'GTM-234234');
ga('send', 'pageview');

Google Optimize Integration

<style>.async-hide { opacity: 0 !important} </style>
<script>(function(a,s,y,n,c,h,i,d,e){s.className+=' '+y;h.start=1*new Date;
h.end=i=function(){s.className=s.className.replace(RegExp(' ?'+y),'')};
(a[n]=a[n]||[]).hide=h;setTimeout(function(){i();h.end=null},c);h.timeout=c;
})(window,document.documentElement,'async-hide','dataLayer',4000,
{'GTM-234234':true});</script>

Google Optimize JS API

function implementExperimentA(value) {
    if (value === '2') {
      $( document ).ready(function() {
        $('#my_element').text('Hello from variant 2');
      })
    }
    }

    gtag('event', 'optimize.callback', {
      name: 'experiment_id_A',
      callback: implementExperimentA
    });

A/B test results

MVT setup

It is Free (with some limits)

Conclusion

• If you need PaaS for A/B testing - try Google Optimize
• Free version limits: 5 experiments, 10 personalizations, 16 MVT combinations. You can update to Optimize 360 if needed
• Ability to do experiments without development
• Works on top of a Google Analytics
• Integration with Google Tag Manager provide additional flexibility in settings