sample size for A/B fisher test significance

Question

Given the results for a simple A / B test...

        A   B
clicked 8   60
ignored 192 1940

( ie a conversation rate of A 4% and B 3% )

... a fisher test in R quite rightly says there's no significant difference

> fisher.test(data.frame(A=c(8,192), B=c(60,1940)))
...
p-value = 0.3933
...

But what function is available in R to tell me how much I need to increase my sample size to get to a p-value of say 0.05?

I could just increase the A values (in their proportion) until I get to it but there's got to be a better way? Perhaps pwr.2p2n.test [1] is somehow usable?

[1] http://rss.acs.unt.edu/Rdoc/library/pwr/html/pwr.2p2n.test.html

Answer 1

power.prop.test() should do this for you. In order to get the math to work I converted your 'ignored' data to impressions by summing up your columns.

> power.prop.test(p1=8/200, p2=60/2000, power=0.8, sig.level=0.05)

     Two-sample comparison of proportions power calculation 

              n = 5300.739
             p1 = 0.04
             p2 = 0.03
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

That gives 5301, which is for each group, so your sample size needs to be 10600. Subtracting out the 2200 that have already run, you have 8400 "tests" to go.

In this case:

• sig.level is the same as your p-value.
• power is the likelihood of finding significant results that exist within your sample. This is somewhat arbitrary, 80% is a common choice. Note that choosing 80% means that 20% of the time you won't find significance when you should. Increasing the power means you'll need a larger sample size to reach your desired significance level.

If you wanted to decide how much longer it will take to reach significance, divide 8400 by the number of impressions per day. That can help determine if its worth while to continue the test.

You can also use this function to determine required sample size before testing begins. There's a nice write-up describing this on the 37 Signals blog.

This is a native R function, so you won't need to add or load any packages. Other than that I can't say how similar this is to pwr.p2pn.test().