Statistical significance in online-marketing

Dr. Torge Schmidt has a PhD in Mathematics and works as a Data Scientist at Akanoo. He is responsible for the development of new prediction models and statistical analysis.
He loves talking about statistics and significance, so feel free to reach out to him, if you have questions or simply want to discuss: torge@akanoo.com

What is statistical significance and why do we need it ?

If you want to increase the performance of your website by introducing a new measure (like a discount campaign), you will want to confirm that this method is actually effective. The best way to do that is to split your traffic randomly into two parts A and B and compare the performance.

Let us assume that A stands for the old version and B stands for the version with the new measure enabled. After one day of analyzing the traffic we get the following data:

Version Number of Visitors Number of Conversions Conversion Rate (cr)
A 49 4 8.2%
B 51 5 9.8%

This data seems to indicate that the new version performs better than the old one, because crB– crA is greater than 0, i.e. crB> crA. But is this really enough to prove this thesis? Imagine that one more visitor arrives on version A and converts, then we get the following result:

Version Number of Visitors Number of Conversions Conversion Rate (cr)
A 50 5 10%
B 51 5 9.8%

Now version A seems to be better than version B. Therefore based on this data we can not reliably conclude that version A or B is better. Naturally we need more data to prove this! So let us assume we compare both versions for a longer time and get the following data:

Version Number of Visitors Number of Conversions Conversion Rate (cr)
A 5023 421 8.4%
B 5012 549 10.9%

Now we can say with high probability that version B is better than version A. But why is that? Could we not simply have had bad luck in our choice of who gets to see version A and version B? It is of course possible, but very unlikely. How unlikely it actually is, can be shown by applying a statistical significance test (hypothesis test).

There are two possible explanations for the observations in the table above:

  • Version B performs better than version A
  • Version B does not perform better than version A, we just had bad luck selecting the visitors

This leads to the question: How likely is it that the observations happened by chance?

Let us assume, for now, that the observations did in fact happen by chance and therefore both versions have the same conversion rate (8.4%), if we simply collect enough data.

We now repeat the experiment using 5000 visitors in every version, but instead of using real data, we just toss a coin to decide whether a visitor buys or not (well, not a 50/50 coin but a 91.6 / 8.4 coin). In the end we compute the conversion rate for both versions and take a look at the difference crB– crA.

We expect them to be 0, as both versions should perform the same (since they have the same conversion rates), but due to chance in the coin-tossing, we also get results that are bigger or lower than 0. If we repeat this multiple times and draw the results we arrive at the following chart, the distribution of crB– crA.

The X-Axis denotes the difference crB– crA.. The higher the curve, the more often crB– crA has the corresponding value. This chart is the basis for hypothesis testing.

We can see, based on this chart, how probable getting a certain value crB– crA is. The further we go away from the zero, the less likely it is to observe this result. If we go to the left, version A performs better, if we go to the right, version B performs better.
Now we know that values to the far right appear less often than in the middle, the probability of those appearing randomly is low. But when do we say that something is probable and something is improbable? There is no exact definition, so we have to create one ourselves. Let us say that we define the rightmost 5% of the graph as improbable, while the remaining 95% are probable:

At this point 95% of all cases lie on the left and 5% on the right.
This implies that 5 in 100 random experiments results in a difference value that lies in the green area.

The corresponding percentage, 5% (or 0.05), is called the significance value (often denoted as alpha) and it is normally set before starting an experiment. It defines how certain we want to be,  when we decide that version B performs better than version A.

Note: Obviously we could do the same on the left side but since the experiment indicated crB– crA>0 we do not test if the cr values are different, but instead if crB> crA or not.We now place our own value from our observation above in this chart (add 10.9%-8.4% = 2.5% or 0.025):

We see that our observation lies on the right side of our threshold,
therefore this result appears in less than 5% of all random experiments.

This means that it is very unlikely that our results above happened by chance.As a last example we present the same analysis using the first observation (i.e. only 50 visitors in each group and a smaller cr difference of 1.6%) using a significance value of 5%:

Note that our observation lies on the far left of our significance value and probably happened randomly.

Rule of thumb:

The smaller the significance value is, the harder it is to prove that two versions are different. This gets easier by either having more data or having a larger effect (i.e. difference in conversion rates).

 

Meet our intern Karsten

Karsten has joined our team in June as an intern for data science. We asked him a few questions that you can get a good impression of what he is doing here.

What have you done before joining us?

I am currently studying at the University of Hamburg for my master’s degree in business administration.

What are your responsibilities at Akanoo?

Everything data science and statistics related.

What do you enjoy most about your work?

I really like the variety of tasks and the steep learning curve in the data science field.

And what do you usually do after work?

Sports or recording music.

Which colleague would you take with you to a lonely island and why?

Bendix. I’m confident he would be able to start a farm and develop a watering system for all the plants.

Which attraction/part of Hamburg do you think you should definitely visit and why?

One of the lakes in and around Hamburg for a refreshing swim.

Which apps could you never do without and why

Google Maps and HVV to find the fastest way to my next destination and Audible and iTunes U to keep me entertained and learn something new while travelling.

On the web you can find me here:

Xing 

New customer uhrzeit.org

We are happy to welcome uhrzeit.org as our new customer for Akanoo onsite marketing.

Created as the result of a coincidence during a scavenger hunt, uhrzeit.org was the first German website showing the exact time in cooperation with the technical institute for physics in Braunschweig.
In 2003 the online shop was founded and since then the e-commerce company from Hamburg has become the most visited online shop for watches in Germany.

Akanoo’s real-time tracking & targeting with succesful campaigns at exactly the right time seem to be a perfect match and we are looking forward to creating many exciting campaigns together.

Meet our new Senior Sales Manager Sebastian

 
Sebastian has joined our team as a Senior Sales Manager in May. To give you a better impression of what he is doing here at Akanoo we asked him to answer a few questions.
What did you do before you came to us ?
I have worked for Groupon, Yelp and an Internet agency for Google AdWords.
What are your responsibilities at Akanoo?
I am employed as a Senior Sales Manager. My responsibilities are:
  • Cold call, direct Email and perform other lead generation activities
  • Develop and follow up on business leads
  • Pipeline management
  • Closing the deals
What do you like most about your work?
Speaking with my clients about their specific goals. I love the whole sales process.
And what do you usually do after work?
I like to walk around the Alster after work.
Which colleague would you take with you to a deserted island and why?
Just me, the girl of my dreams, with a yacht on the water and a knife in my pocket (to open a coconut in case ), suncream and a helicopter.
What attraction / part of Hamburg do you think you should visit and why?
Akanoo Mittelweg 121…and be my guest.
What apps can you never do without and why?
  • UFC
  • Netflix
  • Spotify
  • WhatsApp
On the Internet you can find me here:

New customers for Akanoo: Erwin Müller and Sport-Thieme


We are happy to announce our new collaborations for our onsite marketing with Erwin Müller and Sport-Thieme.

Founded in 1951 Erwin Müller is a specialist mail order supplier focusing on home textiles and decorations. Their offer includes unique customization service and is available to international clients as well. We are looking forward to support them with successful onsite campaigns.

Sport-Thieme is the number 1 for institutional sport with more than 65 years of offering quality sports goods for fair prices. You can find over 13.000 items and accessories for over 70 different sports in their online shop! We will be happy to increase the number of 250.000 happy clients so far.

We heartily welcome our new customers at Akanoo!

Meet our new working student Kevin

Kevin has joined our team as a working student for Pre-Sales in June. To give you a better impression of him we asked him to answer our questionnaire for new team members.

What did you do before you came to us?

In fact, the same as now – study and run my own business.

What are your responsibilities at Akanoo?

My responsibility is to talk to customers and find out if our service can help them to increase their revenue.

What do you like most about your work?

The team is really great. It is less “just a job”, but more fun to work with everyone. Everyone helps each other and it’s a nice atmosphere.

And what do you usually do after work?

Usually also work, but I equally like to go out with friends, read, go to the cinema and do athletics.

Which colleague would you take with you to a deserted island and why?

Actually, I would make a lottery, because I like everyone alike.

What attraction / part of Hamburg do you think you should visit and why?

Depends why and when – but now in the summer I would recommend Planten& Blomen before and at sunset.

What apps can you never do without and why?

Actually the Google app because I have access to all the information at any time.

On the Internet you can find me here: Xing

New customers for Akanoo Onsite Marketing: Eterna

We are happy to welcome Eterna as a new customer for Akanoo Onsite Marketing. Eterna is a fashion label with a long tradition. Since 1863 Eterna has been selling high-quality shirts and blouses. Today, the company is the European market leader with a yearly revenue of about 101.7 million Euro.

We are looking forward to helping Eterna generating more revenue with personalized onsite campaigns!

New Data Insights Case Study with Deichmann published

We are happy to announce that a new case study with our client Deichmann is available for download now. The case study shows you how Deichmann, a European market leader in the footwear retail sector, uses our Data Insights benchmarks to optimize its online shop and thus increase its conversion rate and revenue.

Akanoo Insights analyses more than 70 million shop visitors per month for its customers and based on this data identifies for every shop the most promising avenues for onsite improvement. The case study with Deichmann gives you a deeper understanding how such improvements look like.

You can read and download the Deichmann case study here: http://akanoo.com/casestudy-deichmann

We thank Deichmann for the successful collaboration and we are looking forward to identify even more optimization potential in the future.

How to run JavaScript QUnit Tests upon Deployment using Jenkins & PhantomJS

Check out the first part of the JavaScript Testing Series: Unit Testing Self-Invoking JavaScript Functions

Unit Testing is great. However, the real benefit of unit testing is only achieved when the tests are run before each deployment. In continuous integration (CI) it makes sense to run the tests automatically in the CI tool.

At Akanoo we are using Jenkins which can be extended by various plugins for several use cases. Unfortunately, there is no plugin for QUnit test results. So we have to utilize the existing plugin for JUnit. Two steps need to be done:

  • Find a way to run the tests in Jenkins.
  • Find a way to output the QUnit results as JUnit results.

Running QUnit Tests in Jenkins

Jenkins offers no service to open web pages upon deployment. I didn’t know of any plugin that offers such a thing. So I went out googling. I found a guide in the repository of the HTML5 boilerplate on how to set up QUnit with Jenkins that suggested to use PhantomJS with a QUnit test runner.

Following that lead, the first thing I did was to download PhantomJS and try to run my test HTML file locally. PhantomJS can only run JavaScript files, so I needed a test runner. I took a look at the one mentioned above but found it a little too bold for our needs, so I came with my own solution.

var system = require('system');
var fs = require('fs');
var page = require('webpage').create();

// argument 0 is always the file which is called (this)
if (system.args.length === 1) {
    console.log('Pass the path/to/testfile.js as argument to run the test.');
    phantom.exit();
} else {
    // path is relative to where phantomjs is started
    var url = system.args[1]; // e.g. 'test/unit/tests.html'
    console.log("Opening " + url);
}

page.open(url, function (status) {
    console.log("Status: " + status);
    if (status === "success") {
        setTimeout(function () {
            var path = 'results.xml';
            var output = page.evaluate(function () {
                return document.output;
            });

            fs.write(path, output, 'w');
            console.log("Wrote JUnit style output of QUnit tests into " + path);

            console.log("Tests finished. Exiting.");
            phantom.exit();
        }, 3000);
    } else {
        console.log("Failure opening" + url + ". Exiting.");
        phantom.exit();
    }
});

What does the runner? It takes an argument with the path to the HTML QUnit test file that is to be opened by PhantomJS. If the argument is missing, we can use console.log() to print the result into the console running PhantomJS. The main part of the script opens the page. If the given file doesn’t exist an error message is logged and PhantomJS terminated. If the file can be opened the JavaScript variable document.output of the test page is evaluated and written into a file called results.xml. The evaluation is done after a timeout of three seconds – the time the tests never exceeded on my local machine.

Output QUnit Results in JUnit Format

In the next step we need to make sure the QUnit results can be interpreted by the Jenkins JUnit plugin. Luckily, there is already a plugin for QUnit to produce the results in a JUnit-style XML report. I installed the plugin and configured it to write the results in the document.output variable that we’ve already seen in the PhantomJS runner above.

The current setup is running fine on my local machine: PhantomJS is installed, can be started via shell to execute the runner script, opening the QUnit test HTML file and saving the JUnit-style report into results.xml.

Creating the Jenkins Pipeline

Let’s make sure the job is also running in Jenkins. At Akanoo, Jenkins lies inside a docker image, so I edited the Dockerfile to download and unpack PhantomJS. Use the correct version (32bit or 64bit) — I first used 32bit on a 64bit machine and wondered why it didn’t work. Make sure to add PhantomJS to your PATH variable.

Jenkins allows by default to define multiple build steps for one build. But we want to achieve that the full build is terminated as soon as one step fails. Jenkins offers the Pipeline plugin to define multiple stages of a build. So I installed the pipeline and the JUnit plugin and restarted Jenkins.

I have configured several stages in the pipeline:

  1. Checkout the latest version of code from Git.
  2. Run the tests in PhantomJS, archive the test results and report results to the JUnit plugin.
  3. Build, if the previous step didn’t fail.

To make the build fail if an error in the unit tests occured, we can utilize a try-catch-block. The Groovy script in the pipeline also allows to run shell scripts which we need to run PhantomJS. I came up with the following script:

node {
    stage('Version Control') {
        // checkout the latest version from Git
    }
    stage('Test') {
        try {
            // run PhantomJS
            sh 'cd ${JENKINS_HOME}/path/to/unit/tests && phantomjs phantomjs-runner.js tests.html'
 
            // move result file into workspace
            sh 'mv ${JENKINS_HOME}/path/to/unit/tests/results.xml ${JENKINS_HOME}/workspace/${JOB_NAME}'
 
            // archive test results with relative path from ${JENKINS_HOME}/workspace
            step([$class: 'JUnitResultArchiver', testResults: '**results.xml'])
 
            // report to JUnit with relative path from ${JENKINS_HOME}/workspace
            junit '**results.xml'
        } catch(err) {
            throw err
        }
    }
    stage('Build') {
        // I would build now if the test didn't fail
    }
}

Let’s discuss the script line-by-line. At first, we have the Version Control stage. I assume you know how to checkout from Git. You may also omit this stage if the script is stored on the same machine as Jenkins.

In the Test stage a shell script executes PhantomJS with two parameters: the phantomjs-runner.js file we discussed above and the QUnit HTML test file. The results of the test are stored in a file called results.xml in the same folder the tests lie in. In the next line we move it into the Jenkins workspace of the current job. The step command is used to store the test results using the JUnitResultArchiver to be able to analyse the results of all tests later. We also send the results to the JUnit plugin to check for errors. This step will throw an exception if errors are found that is caught by the try-catch-block and re-thrown to stop the build before starting the Build step.

In the Build step the actual build would run. This step depends on what you want to achieve. In our case we run a Groovy script.

Conclusion

We managed to configure a Jenkins build pipeline that checkouts the current version from Git (or any other version control system), runs the QUnit tests in a PhantomJS headless browser, returns the test results in JUnit-style format, archives the results and only builds if the tests were successful.

It took me a couple of hours to figure out the single steps and bring everything together. I hope you found this useful. If you have any questions or ideas for optimization, please leave a comment below.

Meet our new Senior Corporation Manager Monica

We are happy to welcome Monica as part of our Sales team. In her interview, she will tell you a bit more about her background, her responsibilities at Akanoo and what she is doing in her free time:

What have you done before joining us?

After studying Architecture/ Design and later Electrical Engineering at NYU, I landed at Mühlbauer AG as a Sales Engineer and then followed the same Sales path for the past 12 years. Moved back to Germany in 2010, after 3 years working at Oracle and have worked for Dell/ EMC, Mailjet and Webtrekk in Munich, Paris and Berlin before joining Akanoo.

What are your responsibilities at Akanoo?

I’m the Senior Corporation Manager at Akanoo meaning the Sales point to our major target markets DACH and also responsible for expanding Akanoo internationally and winning new great customers.

What do you enjoy the most about your work?

I love Sales and have been expanding German, French and American companies in Europe, North America and also MENA. Sales is very emotional to me and I love offering Akanoo solutions to companies that struggling with communicating to their non-buyers and help turn them into buyers and increase their turnover.

And what do you usually do after work?

Yoga, Kick-Boxing, Jogging, Fashion and Interior Design, I believe I am a little bit of both intro- and extrovert and always try to keep the balance by spending some quiet time drawing sketches, reading books and watching movies!

Which colleague would you take with you to a lonely island and why?

I’d take Gwen with me, we will talk only in French so that I can remember some forgotten French language skills of mine ;-), She’s very patient with my French and a perfect teacher!

Which attraction/part of Hamburg do you think you should definitely visit and why?

As an architect/ designer myself, I am very looking forward to visiting Elphi (Elbphilharmonie) as it is one the largest and most acoustically advanced concert halls in the world. And of course the chocolate museum (yam yam yam)!

Which apps could you never do without and why?

I can’t live up without my online banking Sparkasse, Facebook, Instagram, Twitter, WhatsApp, Telegram apps.

These blogs/websites belong to my daily reading:

Twitter, Facebook & Insta, Online Marketing, Internet World, …

On the web you can find me here:

LinkedIn