Running Closed Betas: Which Users, and How Long?

A closed beta is an excellent feedback loop. It lets you see what works well in your application, and it helps you understand the jobs your customers are trying to do. However—like any system—if you put garbage in, you get garbage out. Your beta users and their feedback are massively influential, so picking users at random isn’t wise, as it can lead to a one-size-fits-none product.

Beta users must be focused

Your beta users must be a focused group of potential customers experiencing the same problem, or the same set of problems. If this isn’t the case then inevitably one of two things will happen:

• Unhappy customers: You only pay attention to feedback from certain customers, meaning your product will get mixed reviews. A large number of your beta customers use your product for the wrong reasons, and they’ll feel like you’re ignoring their requests.
• One-size-fits-none: You attempt to keep everyone happy and build the sum of all total features. A collection of under-developed features to meet the idle daydreams of a user base. A Swiss Army knife.

Beta users will guide you toward an MVP

As you map out all of the areas that could benefit from further iterations (spoiler: usually all of them), you’ll realize you need guidance on where to focus. This is where beta users come in. If your possible features are a map then your users are your compass. Best make sure they’re all pointing the same direction.

But they won’t tell you when you’re done

Early adopters desire the new functionality you claim to provide. They are willing to accept instability, poor usability, limited integrations and other faults just to gain this new technology.

Their tolerance and enthusiasm is a double-edged sword. New—but incomplete—features will be welcomed with open arms, always. The nature of early adopters means they’ll never press you for polish or finesse, and will always be excited by new features regardless of the state.

In my experience, there are only two ways you leave beta.

Exit via scope: a defined feature set

At some point in beta you decide “We’re launching with these 3 features”, or in Jobs To Be Done speak, “Users should hire our product to do the following three jobs”. Along with a closed scope, you should maintain a quality level. For some apps that might be “stunning visual design on all screens”, for other apps it could be simply “clear and usable”. Ryan Singer has good advice on maintaining a good user experience while searching for an MVP.

Exit via time: an unmissable deadline

Events such as SXSW, Demo Day, Disrupt, etc. provide start-ups with a compelling event. If they have a live, open, web application they can capitalize on the traffic they will draw. If they fail, they will receive thousands of visitors, most of whom leave, often never to return (see Here’s what you should know about private betas). There is a danger here of shipping junk on time. Deadlines you can’t afford to miss can leave you with products you can’t afford to ship.

What about exit via budget? This is like testing out a car’s brakes by speeding towards a wall. Design and development continue well after launch, not to mention new costs such as advertising, support, etc.

Launching your product because you’re almost out of money isn’t a strategy, it’s a death knell. You don’t want to hear it.