How I use beta tests to rapidly optimize new products
September 30th, 2009 - 34 Comments
A couple days ago, I asked if you guys wanted to hear about the business side of iwillteachyoutoberich, and with 375+ comments, the response was an overwhelming yes.
Today, I’m going to share how I use beta tests to get rapid feedback on in-depth projects I’m creating. This will show you what I’m doing behind the scenes, and how you can use these same techniques for your entrepreneurial projects.
What you’ll find in this guide
- How I decide which products to build
- Why a beta test? Using the 85% Solution when building products
- The logistics of testing (and: Charging for a beta test? Are you crazy?)
- Mistakes I’ve made in beta testing
“Rich” is about more than money
Although I Will Teach You To Be Rich sounds like a get-rich-quick scam, you know it’s not — it’s about living a rich life, with money as part of that. Since the early days of this blog, I’ve always emphasized that rich is about more than just money.
My book, I Will Teach You To Be Rich, crystallized my automation system and philosophy about money.
But there are so many areas that I want to cover, some of them dealing with money, some of them dealing with other ways to live a rich life. (As an interesting data point, the guest post on becoming a full-time traveler for $14,000/year was my most popular guest post — ever).
So I’ve been working on specific, in-depth projects that I hope will help you live a rich life. For example, they’ll include automation techniques and scripts you can use to earn more, save more, negotiate, get better service, etc. These are far more complex than a simple blog post. They involve significant research, months of writing, interviews and case studies, custom design, and lots of testing. And while it’s a lot of work, it’s too easy to be a commodity blogger who simply uses cheap tricks (5 posts/day!! Top 10 ways to…!! Look at the new ING interest rate!!) instead of taking the slow path to building something that you’ll be proud of in 10 years.
Now that I’ve set the context, let’s get to the part about building a beta program to test these.
How I decide which products to build
There are an infinite number of in-depth projects I could do. I could talk more about automation, walk people through my book and help them to change money attitudes/behaviors, help people negotiate, focus on debt reduction, show people the similarities between food and personal finance, and on and on.
Just like the infinite number of financial decisions we have when we wake up each morning — which invariably cause us to get overwhelmed and do nothing — I prefer to limit the scope to a few Big Wins.
I take two approaches when deciding which products to create.
- Listening to users. I get dozens of emails from iwillteachyoutoberich readers each day and I read every single one, trying to see what the patterns are. For example, people LOVE the scripts in my book…interesting. On the other hand, I get a lot fewer emails about debt reduction (probably because debt isn’t really my focus, if you have a lot of debt, there are better sites than iwillteachyoutoberich to handle it). Again, another data point.
- Intuition. After 5+ years of blogging, emailing, and meeting you guys in person, I think I have a pretty good idea of what you’ll respond to.
I admit that this is a luxury of having a large user base, but don’t commit the Shrug Effect and think you can’t do the same thing. I use this blog as a laboratory and test frequently to hone my intuition.
Once I come up with an idea, I don’t keep it secret — my idea is never good enough to keep secret. I start asking people around me what they think. Is it a problem for them? I’ll tweet it. I’ll search and see what else is out there.
The truth is, because iwillteachyoutoberich has grown so much, I could basically release anything and get some sales and interest from people. The point is not to release anything, though. I try to pick projects that will (1) optimize my returns and (2) make me proud of what I’ve built by helping a lot of people — and not just iwillteachyoutoberich readers, but outside people.
See also: My bookmarks for customer research.
Why a beta test? Using the 85% Solution when building products
Beta testing is a great way to test your product and get real feedback from people to let you iterate (or improve) on what you’ve got. Here’s how I do it.
Once I’ve decided on a product, I focus on building it. (More on that process later.) I do check in with a close group of friends along the way, but mostly it’s just heads-down to create something that I think will be useful. Whether it’s an ebook that I have to write, or a video product that I structure and record, my goal is build a skeleton that’s 85% complete.
Basically, it should be good enough that it can be used by people and get them very impressive results — but not be polished in terms of design, testimonials, outbound marketing, or even content, because I can do that later. Sometimes, the positioning may even be wildly off, but it’s good enough to send out to a small group for testing.
The point of testing is to validate the following:
- Is this useful?
- Is this better than anything on the market
- Where do people get stuck? What’s their favorite part?
- What kind of feedback/testimonials/impressive stats can I collect from people?
Usually by this stage, you don’t get a lot of catastrophic feedback that your product sucks. If so, something has gone very, very wrong in the product-creation and micro-testing process. But I ALWAYS get feedback on areas that are broken, not easy to understand, or can be improved.
The other thing I ask people is to go through the product, try every step of it, and give me explicit feedback on what works. In this stage, I’m looking for awesome testimonials I can use to put in the product so others can see how well it works.
So, bottom line: I’m trying to identify the weaknesses in the product, as well as get feedback on the strengths of the product. This is a classic product/marketing exercise that helps you stay on track and comes in very, very handy when it’s time to launch the product.
See also: My bookmarks on testimonials.
The logistics of testing (and: Charging for a beta test? Are you crazy?)
Here’s the fun stuff: How do you actually run these tests?
I’ve done it in a variety of ways. For my book launch, I created a pre-launch community where people could join a private community, get PDFs of the book before anyone else, and go through my 6-week bootcamp with a group of similarly aggressive people. Check out the original announcement.
For other projects, I’ll have people buy the 85%-complete product at a steep discount, work through it, and survey them with very specific questions on what works and what doesn’t.
The commonality is this: For beta testers, I always look for qualified people. This means paying in some way, whether pre-ordering my book or buying the product at a large discount.
Why do I charge for beta tests?
Because I’m looking for the type of people who will pay for value — these are the same types of people who will pay for the product when it launches at full price. If I opened it up to anyone for free testing, I’d get 100x the response rate, but the type of tester would be totally different than the real customers. In other words, people who pay are far more likely to actually open, use, and give feedback on a beta test. I have statistics to back this up by unimaginably huge margins, which I’ll share later.
The beta price actually doesn’t matter — usually I’ll offer steep discounts for the beta product when it will end up costing many times that on launch day, and I always offer generous money-back guarantees if it doesn’t work. The point is simply to separate the serious people from curious looky-loos who will never buy the product, anyway.
I also intentionally limit the number of testers to keep it small yet ensure I get enough responses. (For example, if I need 10 testimonials over 5 different topics, most people won’t give you really superb quotes, so you need a lot more than 10 people.)
In exchange for paying a dramatically reduced price, they get early access to a product that’s 85% complete (and presumably works very well), a chance to give feedback and shape the final polish of the product, and we all get a chance to interact via webcast and share feedback about ways to make it better. I usually send them a copy of the final product in exchange for their help.
Bottom line: In exchange for their feedback, beta testers get early access to the product, a dramatically reduced price, and the ability to help shape the final version — and usually the final version of the product for no additional charge.
More about logistics
I usually get beta testers from my newsletter list. This is because of (1) qualification strategy and (2) I can track clicks, opens, and the entire engagement funnel more closely. More on these in future posts.
When newsletter subscribers join the beta program, I add them to a separate email list where I send them automatic followup messages to get their feedback about the product via email, survey, and even video or Twitter. I share some of the best feedback with others because collecting feedback is difficult — people often don’t know how to articulate their feedback, so showing examples helps.
I collect people’s name/email address because there are often a few people — usually 2%-5% — whose feedback is so outstanding that I personally reach out to them to chat with them over the phone, put together an interview, etc. This helps not only the product development, but marketing for the launch.
Mistakes I’ve made
I’ve made a lot of stupid mistakes in running beta tests. The most common is simply working on too many projects, so I end up with 4 of them stuck at 70%, while nothing is getting completed. To help that, I’ve brought someone on to help me project-manage and kick my ass to get things out the door and prioritize better.
Other mistakes I’ve made with beta testing:
- Not planning the beta test in advance of launch. If you try to do run a beta program without planning how it will flow beforehand, you will run into lots of problems and wish you were dead. Also, you’ll be so busy polishing the product that you won’t have time to build the beta program on the fly, and your feedback will suffer. Now, I do it beforehand and automate it as much as possible.
- Not creating buckets for feedback. One dumb mistake was when I forgot to specify where the feedback should go. It all went to my email inbox, which was a mess. Now, responses go to specific buckets. For example, I might collect “day 5″ feedback in one Surveymonkey, and I’ll collect “day 10″ in another Surveymonkey. This makes it really easy to find the right feedback/testimonials when we’re ready to iterate.
- Not thanking the beta testers! It’s thrilling to get real feedback on the project you’ve been working on for so long. Once you get beta responses, it’s off to the races to complete it and launch. I’ve forgotten to take some time to thank the beta testers for taking a risk and helping with feedback, and that was a mistake. Always be sure to thank the people who helped get you the final 20%
So, that’s an overview of how I run beta tests with new projects coming out on I Will Teach You To Be Rich.
Coming up: Beta invites for 2 new domination guides
I’ll be releasing a number of beta products in the next few days. I send beta announcements out via my email list, so if you want access to the beta tests, sign up for the free list below.
I decided to take a last-minute ski trip this weekend. Here’s a quick email I sent to my friends: ...Read More
Let’s say you meet a new person at a party this weekend. Isn’t it interesting how you might ...Read More