Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AB Testing with Local Storage and Google Analytics

I'm running a sitewide AB test on my ecommerce site. After a visitor lands, I assign them a local storage key/value:

function isLocalStorageNameSupported() {
  var testKey = 'test', storage = window.localStorage;
  try {
    storage.setItem(testKey, '1');
    storage.removeItem(testKey);
    return true;
  } catch (error) {
    return false;
  }
}

$(function() {

  if(isLocalStorageNameSupported()){
      var version = Cookies.get("version");

      if (version == null) {

        if (Math.random() >= 0.5){
            Cookies.set("version", "A");  
            var version = "A"
        }else{
            Cookies.set("version", "B");  
            var version = "B"
        }

        ga('create', 'UA-XXXXXXXX-1', 'auto');
        ga('send', {
          hitType: 'event',
          eventCategory: 'Pricing Experiment',
          eventAction: 'New Unique Visit',
          eventLabel: version,
          eventValue: 0
        });       
    } 
  } 



});

After a visitor checks out, I check which version they are on and send another Google Analytics Event.

My checkout conversion events are showing up just fine. But I am only getting about 25% of the "New Unique Visit" events. Analytics is showing 12000 visits to the site but I only have 3000 of my custom events.

Which part of my code is causing this discrepancy and how can I fire an event on all visits?

like image 716
Jackson Cunningham Avatar asked Jun 20 '17 03:06

Jackson Cunningham


1 Answers

This is somewhat a shot in the dark since I've ran your code and the event would trigger each time (I'm assuming your Cookies object/functions are working), but what I'm wondering is if the 12,000 number may not be correct...

Is the snippet here exactly what your site is running? Where are you sending the GA pageview? I ask because I see the event is immediately after your create function. The other potential reporting issue I see is that the event is interactive- meaning it will affect your bounce rate if you are sending a pageview too (which I'm assuming you are).

What metric specifically do you mean when you say 12,000 visits? Sessions?

Sending events before pageviews will also give you a bunch of (not set) Landing Page dimensions. That means that the event will not be associated with any landing page, and when the pageview is sent it also increments your user count... So if you're seeing more users than sessions that would indicate this is a problem.

Some things to try:

  • Send the pageview before the event.
  • Set the event to be a non-interaction event using {nonInteraction: true}.
  • Another thing I would do is only put the event within the conditional- the create and pageview send should always happen regardless of the localstorage condition.
  • Edit: You're also waiting for DOM ready to create. I'd move that to the <head> tag so the event is the only thing waiting for the document ready.

<head>
    <script>
	    //Your preferred method of loading analytics.js here...
	    ga('create', 'UA-XXXXXXXX-1', 'auto');		
	    ga('send', 'pageview');
    </script>
</head>

function isLocalStorageNameSupported() {
	var testKey = 'test', storage = window.localStorage;
	try {
		storage.setItem(testKey, '1');
		storage.removeItem(testKey);
		return true;
	} catch (error) {
		return false;
	}
}

$(function() {
	
	if(isLocalStorageNameSupported()){
		var version = Cookies.get("version");
		if (version == null) {
			if (Math.random() >= 0.5){
				Cookies.set("version", "A");  
				var version = "A"
			}else{
				Cookies.set("version", "B");  
				var version = "B"
			}
			
			ga('send', 'event', 'Pricing Experiment', 'New Unique Visit', version, 0, {nonInteraction: true});			
		} 
	} 

});

See if that gets your metrics closer together, and then consider this alternative way of tracking the data:

Perhaps an event here isn't the best solution altogether. What you're doing sounds like an excellent candidate for using a custom dimension scoped to the user.

I'd recommend creating a custom dimension called "Pricing Experiment" with a scope of User (because with GA Experiments, the user will always see the same variation upon successive sessions) and then after your create function, replace the event with this: ga('set', 'dimension1', version); (be sure to replace dimension1 with your actual number.

Finally, send a pageview after setting the dimension (custom dimensions use hit types to be transported to GA).

<head>
	<script>
		//Your preferred method of loading analytics.js here...
		ga('create', 'UA-XXXXXXXX-1', 'auto');
		
		if(isLocalStorageNameSupported()){
			var version = Cookies.get("version");
			if (version == null) {
				if (Math.random() >= 0.5){
					Cookies.set("version", "A");  
					var version = "A"
				}else{
					Cookies.set("version", "B");  
					var version = "B"
				}
				
				ga('set', 'dimension1', version);			
			} 
		} 
		
		ga('send', 'pageview');
		
		function isLocalStorageNameSupported() {
			var testKey = 'test', storage = window.localStorage;
			try {
				storage.setItem(testKey, '1');
				storage.removeItem(testKey);
				return true;
			} catch (error) {
				return false;
			}
		}
		
	</script>
</head>

Now when you view any other report, you can apply a Secondary Dimension of "Pricing Experiment" and you'll be able to show which version they saw.

This way you'll be able to get the contextual data without messing with hit types that alter reporting metrics.

Again, I apologize if this isn't perfectly illuminating your issue- I'm taking the snippet you provided literally and filling in any blanks as best I can.

like image 88
GreatBlakes Avatar answered Nov 07 '22 18:11

GreatBlakes