How I use beta tests to rapidly optimize new products

September 30th, 2009 - 34 Comments

A couple days ago, I asked if you guys wanted to hear about the business side of iwillteachyoutoberich, and with 375+ comments, the response was an overwhelming yes.

Today, I’m going to share how I use beta tests to get rapid feedback on in-depth projects I’m creating. This will show you what I’m doing behind the scenes, and how you can use these same techniques for your entrepreneurial projects.

Guinea pigs

What you’ll find in this guide

  • How I decide which products to build
  • Why a beta test? Using the 85% Solution when building products
  • The logistics of testing (and: Charging for a beta test? Are you crazy?)
  • Mistakes I’ve made in beta testing

“Rich” is about more than money

Although I Will Teach You To Be Rich sounds like a get-rich-quick scam, you know it’s not — it’s about living a rich life, with money as part of that. Since the early days of this blog, I’ve always emphasized that rich is about more than just money.

My book, I Will Teach You To Be Rich, crystallized my automation system and philosophy about money.

But there are so many areas that I want to cover, some of them dealing with money, some of them dealing with other ways to live a rich life. (As an interesting data point, the guest post on becoming a full-time traveler for $14,000/year was my most popular guest post — ever).

So I’ve been working on specific, in-depth projects that I hope will help you live a rich life. For example, they’ll include automation techniques and scripts you can use to earn more, save more, negotiate, get better service, etc. These are far more complex than a simple blog post. They involve significant research, months of writing, interviews and case studies, custom design, and lots of testing. And while it’s a lot of work, it’s too easy to be a commodity blogger who simply uses cheap tricks (5 posts/day!! Top 10 ways to…!! Look at the new ING interest rate!!) instead of taking the slow path to building something that you’ll be proud of in 10 years.

Now that I’ve set the context, let’s get to the part about building a beta program to test these.

How I decide which products to build

There are an infinite number of in-depth projects I could do. I could talk more about automation, walk people through my book and help them to change money attitudes/behaviors, help people negotiate, focus on debt reduction, show people the similarities between food and personal finance, and on and on.

Just like the infinite number of financial decisions we have when we wake up each morning — which invariably cause us to get overwhelmed and do nothing — I prefer to limit the scope to a few Big Wins.

I take two approaches when deciding which products to create.

  1. Listening to users. I get dozens of emails from iwillteachyoutoberich readers each day and I read every single one, trying to see what the patterns are. For example, people LOVE the scripts in my book…interesting. On the other hand, I get a lot fewer emails about debt reduction (probably because debt isn’t really my focus, if you have a lot of debt, there are better sites than iwillteachyoutoberich to handle it). Again, another data point.
  2. Intuition. After 5+ years of blogging, emailing, and meeting you guys in person, I think I have a pretty good idea of what you’ll respond to.

I admit that this is a luxury of having a large user base, but don’t commit the Shrug Effect and think you can’t do the same thing. I use this blog as a laboratory and test frequently to hone my intuition.

Once I come up with an idea, I don’t keep it secret — my idea is never good enough to keep secret. I start asking people around me what they think. Is it a problem for them? I’ll tweet it. I’ll search and see what else is out there.

The truth is, because iwillteachyoutoberich has grown so much, I could basically release anything and get some sales and interest from people. The point is not to release anything, though. I try to pick projects that will (1) optimize my returns and (2) make me proud of what I’ve built by helping a lot of people — and not just iwillteachyoutoberich readers, but outside people.

See also: My bookmarks for customer research.

Why a beta test? Using the 85% Solution when building products

Beta testing is a great way to test your product and get real feedback from people to let you iterate (or improve) on what you’ve got. Here’s how I do it.

Once I’ve decided on a product, I focus on building it. (More on that process later.) I do check in with a close group of friends along the way, but mostly it’s just heads-down to create something that I think will be useful. Whether it’s an ebook that I have to write, or a video product that I structure and record, my goal is build a skeleton that’s 85% complete.

Basically, it should be good enough that it can be used by people and get them very impressive results — but not be polished in terms of design, testimonials, outbound marketing, or even content, because I can do that later. Sometimes, the positioning may even be wildly off, but it’s good enough to send out to a small group for testing.

The point of testing is to validate the following:

  • Is this useful?
  • Is this better than anything on the market
  • Where do people get stuck? What’s their favorite part?
  • What kind of feedback/testimonials/impressive stats can I collect from people?

Usually by this stage, you don’t get a lot of catastrophic feedback that your product sucks. If so, something has gone very, very wrong in the product-creation and micro-testing process. But I ALWAYS get feedback on areas that are broken, not easy to understand, or can be improved.

The other thing I ask people is to go through the product, try every step of it, and give me explicit feedback on what works. In this stage, I’m looking for awesome testimonials I can use to put in the product so others can see how well it works.

So, bottom line: I’m trying to identify the weaknesses in the product, as well as get feedback on the strengths of the product. This is a classic product/marketing exercise that helps you stay on track and comes in very, very handy when it’s time to launch the product.

See also: My bookmarks on testimonials.

The logistics of testing (and: Charging for a beta test? Are you crazy?)

Here’s the fun stuff: How do you actually run these tests?

I’ve done it in a variety of ways. For my book launch, I created a pre-launch community where people could join a private community, get PDFs of the book before anyone else, and go through my 6-week bootcamp with a group of similarly aggressive people. Check out the original announcement.

For other projects, I’ll have people buy the 85%-complete product at a steep discount, work through it, and survey them with very specific questions on what works and what doesn’t.

The commonality is this: For beta testers, I always look for qualified people. This means paying in some way, whether pre-ordering my book or buying the product at a large discount.

Why do I charge for beta tests?
Because I’m looking for the type of people who will pay for value — these are the same types of people who will pay for the product when it launches at full price. If I opened it up to anyone for free testing, I’d get 100x the response rate, but the type of tester would be totally different than the real customers. In other words, people who pay are far more likely to actually open, use, and give feedback on a beta test. I have statistics to back this up by unimaginably huge margins, which I’ll share later.

The beta price actually doesn’t matter — usually I’ll offer steep discounts for the beta product when it will end up costing many times that on launch day, and I always offer generous money-back guarantees if it doesn’t work. The point is simply to separate the serious people from curious looky-loos who will never buy the product, anyway.

I also intentionally limit the number of testers to keep it small yet ensure I get enough responses. (For example, if I need 10 testimonials over 5 different topics, most people won’t give you really superb quotes, so you need a lot more than 10 people.)

In exchange for paying a dramatically reduced price, they get early access to a product that’s 85% complete (and presumably works very well), a chance to give feedback and shape the final polish of the product, and we all get a chance to interact via webcast and share feedback about ways to make it better. I usually send them a copy of the final product in exchange for their help.

Bottom line: In exchange for their feedback, beta testers get early access to the product, a dramatically reduced price, and the ability to help shape the final version — and usually the final version of the product for no additional charge.

More about logistics
I usually get beta testers from my newsletter list. This is because of (1) qualification strategy and (2) I can track clicks, opens, and the entire engagement funnel more closely. More on these in future posts.

When newsletter subscribers join the beta program, I add them to a separate email list where I send them automatic followup messages to get their feedback about the product via email, survey, and even video or Twitter. I share some of the best feedback with others because collecting feedback is difficult — people often don’t know how to articulate their feedback, so showing examples helps.

I collect people’s name/email address because there are often a few people — usually 2%-5% — whose feedback is so outstanding that I personally reach out to them to chat with them over the phone, put together an interview, etc. This helps not only the product development, but marketing for the launch.

Mistakes I’ve made

I’ve made a lot of stupid mistakes in running beta tests. The most common is simply working on too many projects, so I end up with 4 of them stuck at 70%, while nothing is getting completed. To help that, I’ve brought someone on to help me project-manage and kick my ass to get things out the door and prioritize better.

Other mistakes I’ve made with beta testing:

  • Not planning the beta test in advance of launch. If you try to do run a beta program without planning how it will flow beforehand, you will run into lots of problems and wish you were dead. Also, you’ll be so busy polishing the product that you won’t have time to build the beta program on the fly, and your feedback will suffer. Now, I do it beforehand and automate it as much as possible.

  • Not creating buckets for feedback. One dumb mistake was when I forgot to specify where the feedback should go. It all went to my email inbox, which was a mess. Now, responses go to specific buckets. For example, I might collect “day 5″ feedback in one Surveymonkey, and I’ll collect “day 10″ in another Surveymonkey. This makes it really easy to find the right feedback/testimonials when we’re ready to iterate.
  • Not thanking the beta testers! It’s thrilling to get real feedback on the project you’ve been working on for so long. Once you get beta responses, it’s off to the races to complete it and launch. I’ve forgotten to take some time to thank the beta testers for taking a risk and helping with feedback, and that was a mistake. Always be sure to thank the people who helped get you the final 20%

So, that’s an overview of how I run beta tests with new projects coming out on I Will Teach You To Be Rich.

Coming up: Beta invites for 2 new domination guides

I’ll be releasing a number of beta products in the next few days. I send beta announcements out via my email list, so if you want access to the beta tests, sign up for the free list below.

facebooktwittergoogle_plus

Related Articles

How to find love using psychology

If there is one thing I have learned in my decades on this earth, it is this: Beware of a ...

Read More

How to get an amazing job and build your career -- with Ryan Holiday

I believe you can learn from ANYONE -- even if you don’t agree with 100% of their views. Years ...

Read More

34 Comments

 

Comments

  1. Wow, this is *just* the sort of stuff that has to do with the business side of things, but is extremely applicable for many different types of entrepreneurs.

    I’m only one of 375+ comments that were a resounding, “yes”, but after seeing this type of thing, I’m excited to see what else you have in store.

    If I can be totally selfish, I’m super interested in experience you have hiring VA’s are your “ass-kicking” person that you alluded to above. This sort of thing has been covered before in other places, but I’m interested in your specific experience. :-)

    I’ll be using this for my own business projects in the coming months!

  2. Nice post Ramit

    Keep it coming

  3. So you beta-test. Interesting!

  4. Great stuff. I’ve noticed some book publishers are releasing beta or draft versions of their books for readers to purchase and provide feedback as they are being written.

  5. so 400+ posts begging for information and once it finally arrives (all of 3 days later) there’s only 4 comments after 24 hours? What the hell? Insanity.

    There’s a bunch of very good info in this post. I like how you tie in using a lot of the things you’ve been writing about with “how I do things backstage.” This might shut up some of the naysayers. One can only hope.

  6. I very much agree with your 85% to completion launch plan. Everyone has an idea but the people that actually get right to work and execute are few and far between. Unless it is a super top secret idea that must be held back from potential competitors until just the right time then almost often it is best to buckle down, develop until a workable model is ready to test (85% done in your case), and then get feedback and iteratively complete until you are completely satisfied (which may be never). Looking forward to more of these entrepreneurship/business type posts. Thanks!

  7. Ramit, I haven’t read a lot of your material, but I’ve heard of your book. After reading this post, I no longer think your book is a joke. Solid methods on testing and launching a product, thanks for putting it out there. Good job on all your success in the last couple years…hope to join you soon with similar entrepreneur tactics. The long term money saving advice you give is a great retelling of all the smart tried and true actions to build long term wealth. Awesome.

  8. Ramit,

    Great approach to a SWOT Analysis. By the way, first time blogger on this site but I deeply appreciate the hard work you do. I’ve received several emails from you (newsletter) and I can see the effort. Thank you.

    Keep it up and let’s see where this subject takes us. I’m in.

  9. @Dave who said “So 400+ posts begging for information and once it finally arrives (all of 3 days later) there’s only 4 comments after 24 hours?”

    You are right ! I was thinking the exact same thing. It goes back to psychology of people and how Ramit has spoken about this many times. In one of his older posts (gems) he says how over 100 people ask for business cards at meetings, but less than 10 ever contact him the next day.

    Maybe that explains why the Lottery Industry is so profitable (to the sellers of the Lottery tickets !)

    • Yep, there is a LOT going on in the huge comment count on the last post vs. the lack of comments on this one. I’m deciding whether to write something up on this or not. Maybe.

  10. Lots of great info to be honest. It was a lot to take in at first read. I think for a lot its just a matter of using these ideas into practice.. how to take this information and apply it for a specific product they want to create..

    anyways, taking advantage of that feeback is huge. i’ve been only recently doing this is in a different way.. for a retail company sending a feeback letter, after the sale, and man.. wondering why they didn’t take advantage of this earlier? you need that two way communication to keep improving and we have those tools today more than ever.. mitigates your risk tremendously…anyways yeah love it, this makes me wanna launch things!

  11. I’m excited about this series, so I hope it continues. I am working on a site, and before this, my “beta testing” (if you could have called it that, would have consisted of my husband and maybe one other person looking at the site.

    I’m really new at this, and I would like to learn more about it.

  12. Hey Ramit,

    Showing up to BizTechDay in SF? Tim Ferris featured it on his blog, hope to see you there.

  13. Thanks for sharing, especially the “mistakes” section. The things that “go wrong” are often key learning indicators and great teaching tools.

    And it seems that the mistakes that you’ve highlighted are pretty universal–make sure you have even a simple infrastructure/process for managing the stages of development and say “thank you” to those who help you at the final stages to get to the finish line.

  14. Glad you posted this. Definitely relevant to anyone doing creative work. When I work on a new blog project, I give a handful of people a preview before telling everyone about it, because simple feedback like “ugh your colors look awful together” sometimes makes a huge difference in whether people find it appealing.

  15. Ramit, don’t give up on us ;-)
    Maybe some people had expected something magical. Also see my last comment.

  16. Well I will add my comment as one of the 400 people who requested this info – I found this useful and relevant.

    Interestingly for me, it raised more questions than answers
    e.g.
    - how to build a community you have that credibility with,
    - where to start,
    - how do you care about this group while still being a real human-being, not too self-helpish. And how to ask the group to help you.

    Anyway, thanks for the information. Personally I would request info on how people should get moving with their ideas, how to test them, and how to most effectively find people to help. Maybe what entrepreneurialism means to you.
    have a great day

  17. Ramit,

    Amazing job. I think you successfully separated the true entrepreneurs with the “get rich quick” folks. I am very satisfied with your detailed and technical sharing of information. I have to say I am not surprised at the amount of work required to get a idea out (that may or may not work).

    Using the 80/20 (80/10) rule; if you got 400 comments before then 40-80 this go around means you tapped the right audience. I know you probably have prioritize but, many of us respect and follow your model.

    Serve and be served.

  18. Hey Ramit!

    This is such a great post! Very detailed and thoroughly written… probably scared a lot of people off when they realized it takes a lot of work – most people don’t want to do that ;)

    Thanks for the info! Keep up the great work!

  19. Hey Ramit,

    Quality over quantity. And the easiest way to achieve quality is to have people pay.

    The difference between free and 1 cent is huge. I love that you have paying beta-testers. Since Teddy had to pay to get into something, he’s MUCH more invested in the thing – rather than being a casual passer-by.

    You get way less people, but they are much more relevant to the customers you’ll want for the product, and will provide ridiculously better feedback. It will come from a genuine desire to have the thing they’re using be better, rather than a it’s-something-to-do or I-want-the-freebie-I-was-promised-for-joining attitude.

    Thanks for sharing your behind-the-scenes business operations. Very useful to budding entrepreneurs and life-maximizers like us,
    Oleg

  20. [...] product development and minimum viable product. The second is a post by Ramit Sethi on his blog, I Will Teach You To Be Rich, and focuses specifically on beta [...]

  21. I’m confused. How is CHARGING $49 and $75 to BETA TESTERS a good thing for anyone helping you out? Perhaps I’m missing the business model here but either way, you’re selling your books. Sure maybe not at the “premium” price so this is really nothing more than a discount for the first 50 people who BUY the books.

    I’ve beta tested MANY products in the past, and all the companies I’ve done beta testing for ALL either compensated me or let me get the products for free. You’re asking your readers to BUY your books and try them to see if YOUR theories work?

    I may be in the minority here but this seems akin to selling your books and asking for PAID reviews. Are you going to then turn around and compensate your beta testers in some way?

    • I mentioned this in my emails. I want beta testers who will be the same type of people as the purchasers: willing to pay for value, not just trying to get something free. By charging, I’m intentionally using a barrier to screen out those people, and I’m testing the value of the product.

      In exchange, beta users get a steeply discounted beta price, the full product for no additional charge when it’s completed, and more personal attention from me and my co-authors.

      If this isn’t your cup of tea, no problem — you can either:
      * Wait for the final product
      * Not buy it at all

      I’m getting lots of orders so I feel pretty comfortable with this strategy.

  22. Ramit,

    I will only make a single response, don’t want to clutter up your website with disagreement for your marketing model.

    I’m just confused, is it only for 50 “select” first buyers or “lots of orders”? Is this a beta test that should be closed (as about the 10 or so I’m currently participating in for hardware and software products) or is this just an advanced sale to anyone willing to buy the product at what is claimed to be a “beta discount price”?

    Good for you that it’s working out well.

    • It’s not just early sales, it’s a pretty comprehensive beta test behind the scenes. We’re already chatting by email so let’s just take more questions up there. Thanks, Sanjay.

  23. Ramit,
    I got your email about the beta test. I’m very interested in hearing more about reducing my medical insurance costs since I’m an IT contractor and pay $400/month for my family plan but I’m not willing to pay $75 for that information. You mentioned the final product would be much more than the beta price. Yes I pay $4800 per year for insurance but I’m not will to pay $75 for the possibility that I “might” be able to reduce it. I guess I’m not comfortable with spending that much money without a gurantee. Not sure if others feel that same way as well.
    PS: I have 2 duplexes and I hope none of my tenants buy your renter’s guide!!!!

  24. Ben,

    Without knowing what the advice is in the “renter’s guide” how would you know if they are detrimental or beneficial. Unless that is you’re willing to pay to play to find out.

  25. Sanjay,
    The renters guide is all about how to negotiate your rent. I’m a landlord. How could my tenants trying to negotiate their rent possibly help me???

  26. Ben,

    It can’t but at least knowing what their strategy is could help you head them off at the pass. Or at least know when it’s coming.

  27. Solid information again. Yes, I’d make this its own blog post–it’s subsumed under your original post, the one with 400+ comments.

  28. I think it is a good strategy to improve more the product you are going to launch or sell.

  29. Beta tests are considered customer focused technique on marketing (that involves product, price, placement and promotion). Thus, it is a great way to collect data from the consumers to constantly improve the product and continue to satisfy them. In this case you should invest more customer relationship, having the stronger bonding between you and your consumers. And in the long-run, will benefit both – profit for you, satisfaction for your consumers.

  30. Thanks Ramit for this post. I have had strategies of my own that I have followed when looking into products but there are so more which you have mentioned which I was unaware of. Thanks for the info.

  31. [...] How I use beta tests to rapidly optimize new products » AKPC_IDS += "1181,"; [...]