Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Any benefit/disadvantages to open beta vs. closed beta for webapp [closed]

This isn't a question about code, but it's programming related. We have a web app that's ready for beta testing. Has anyone noticed any difference between open beta vs. closed beta in terms of the quality or quantity of feedback the testers give or any other factors?

like image 950
Kamo Avatar asked Jun 18 '10 01:06

Kamo


People also ask

What is the difference between open beta and closed beta?

In an open test, anyone can use the product and is usually presented with some messaging that the product is in beta and given a method for submitting feedback. In closed beta, the testing is limited to a specific set of testers, which may be composed of current customers, early adopters, and/or paid beta testers.

What is the purpose of an open beta?

An open beta is a beta-level release of a game which its maker exposes to the outside world. The reasons for doing this are to get player feedback early, to use the players as a test group to find bugs, and finally to create a participatory community around the game that carries its makreting story.

What are the benefits of a beta program?

Beta testing is a tool that tests customer satisfaction and provides you with an early insight into releases. It also enables you to have your say in helping shape the products you use. We do this by letting existing users trial new features and feedback to us.

How long should beta testing last?

We generally recommend tests that are no shorter than two weeks and no longer than twelve, with most beta tests having between four and eight weeks of test time. Below are a four key factors to consider when deciding on the duration of your test.


2 Answers

With a closed beta, you can limit the number of users

This might not seem like a big deal but, consider this...

Closed Beta:

  • the user base is selected by requiring them to write a proposal in how they expect to use the app
  • you release a highly anticipated app to 100 users with no extra invites for the first month
  • those users use the program on a regular basis and pick through most of the really common bugs that slipped through the pre-beta release QC.
  • those users feel privileged to use the app so they brag to everybody about how much they love it (tons of free PR) and are less inclined to trash it because it's still a 'closed beta for a reason'
  • most of the commonly occurring bugs are identified in the first month and a limited number of invites are given to the first set of beta testers to progress into step 2 of beta
  • or, there are still a nightmarish number of bugs in the app and further invites are postponed for review until the next release cycle.

Open Beta:

  • you feel like your app is sufficiently polished so you release it to the masses as a public beta
  • the more objective users start to find and report bugs
  • unfortunately, the sheer number of bug submissions bloats the bug tracker so finding bugs becomes sufficiently difficult
  • since it's hard to find bugs, duplicates start popping up and the bug tracker bloats even more
  • you spend x amount of effort wasted trying to keep the bug tracker clean while also trying to fix bugs in the code
  • the less objective users give the app a shot and discover that 'not everything' works perfectly or works in a the way they expected (intuitive)
  • the less objective users run off to their little blogs and start making posts like 'OMG WTF srsly, [appName] sucks for reason [x] and reason [y] and [z]
  • all the little 'buzz bloggers' trample all over the name of your app because it makes them feel 'empowered' to publicly rant about any/everything
  • google indexes all of the blog rants because they contain a lot of indicative keywords related to your product so the top 2 pages that come up when you enter your apps name in google usually involve something along the lines of '[appName] sucks'.

One of the biggest benefits of a 'closed beta' is, you have the ability to control your work load based on how many users you allow and what types of users you allow.

You need an army of 'objective' users to back you up against the 'subjective' users because the ladder group's main purpose in life is to troll the web looking for an app to trash and create a lot of sensational anti-hype over; all in the name of attracting more traffic to their blog.

If you want a really good example of how to successfully run a closed beta look to Google.

  • with gmail they had a strictly limited closed beta to start
  • they fixed any obvious bugs discovered by the first round of beta testers
  • more invites are given out
  • then they start collecting new ideas of features to implement in their webapp from the beta users
  • they incorporate features while simultaneously fixing bugs
  • they dole out more invites
  • continue tracking features requests and bug submissions
  • when it's sufficiently polished they release it to open beta
  • GMail remains in open beta for 3 years

Why? Google is smart. If some random obscure bug pops up 2 years into open beta nobody can really trash google for it because it's still 'beta'. It's like Google's little way of saying, it's good but we're not completely satisfied with just good. Even if they didn't touch the codebase for the last 2 years of the beta, it still gives the impression of 'they're still perfecting it'.

Which leads me to the single most import point of why you'd want to limit the beta...

Once created, you can't change people's perceptions about your product

Watch this, "How to Ignore Marketing and Become Irrelevant in Two Easy Steps" to see what I mean. It's easily one of the most intriguing presentations I have seen.

Note: I've personally participated in multiple 'closed' betas. Namely, GMail, Google Wave, Boxee, Songbird, and a few others.

like image 148
Evan Plaice Avatar answered Oct 05 '22 13:10

Evan Plaice


With a closed beta the people generally want to be there. They've either waited patiently in line to get an invite after signing up, or they've forgotten about it by the time it starts.

With an open beta, you get more people but they tend to be the "hey, this website is neat, I think I'll... OOH SHINY! runs off after a piece of tin foil" type.

like image 43
Ignacio Vazquez-Abrams Avatar answered Oct 01 '22 13:10

Ignacio Vazquez-Abrams