Inside test results from IWT “Labs”

September 28th, 2012 - 8 Comments

What do you guys do for fun? Go out to bars? See a movie?

What about run carefully designed, iterative multi-variate tests to study human behavior?

What? Is that weird?

Check out this test data from I Will Teach You To Be Rich Labs, where we’re running dozens of tactical and theoretical tests at any given time.

I wanted to try a really simple test — two completely different designs to see which would produce more email subscriptions. So we designed two totally different pages for a recent contest I ran:

Contest A

Contest B

With over 20,000 data points and statistical confidence, which do you think worked better?

Data Table

Yes, this is a table of data. Yes, this will remind you of college when you didn’t understand the data there, either.

ANSWER: No change!

Despite being totally different designs, the results were basically identical.

One of the surprising secrets of testing is that MOST tests produce no change. Online marketers don’t tell you this because it’s not sexy and kind of depressing. It’s like guys going out to bars, predator style, then the next morning, telling their friends last night was “CRAZY”…but neglecting to mention the fact that they didn’t really meet any girls.

In my experience running tests, once in awhile, you’ll find a small change (3%), and once in a blue moon, you’ll get a huge huge win — a total twist that changes everything. To put things in perspective, I find about 2 HUGE WINS every year….out of hundreds of tests. But they are big enough to pay for me running IWT Labs and giving you dirty jokes every couple days on my email list.

Btw, people who follow me on twitter.com/ramit will know of one that I recently wrote about that transformed my business. If you’re not following me there, and you want to see me make fun of some of the really dumb comments I get, check it out.

Anyway, back to you. I could spend all day talking about testing theory, but we both know you don’t care. Most people are like ZOMG RAMIT GIMME THE GOOD TESTS PLS~!~##

Tests That Produced Big Results
I know you’d rather hear about the Big Win tests, so I figured I’d share a few good ones with you.

  • My friend Noah Kagan, founder of AppSumo, ran one simple price test that skyrocketed his conversions and revenue while simultaneously lowering his returns. (Can you guess what the test was?)
  • Dating expert Nicholas Kho used testing to discover the one text message that will get 90% of VIPs (and women) to respond to you.
  • Primal nutrition expert Mark Sisson used testing to change his body composition with just one 20-minute workout a week.

If you’re curious about the exact test results, I created a new ebook that includes 15 little life experiments you can use today.

It’s totally free. No upsells, no nothing. Get your copy today, my treat: http://littlelifehacks.com

P.S. If you click through to that page, you’ll also see that I’m giving away 3 iPhone 5s to kick off the launch of this ebook. Right now there aren’t a lot of participants, so if you participate you’ll have pretty good odds of winning. Here’s the link again: http://littlelifehacks.com

facebooktwittergoogle_plus

Related Articles

Sex vs. salary: Which would you rather talk about?

My friend, a perpetual Manhattan bachelor, just got into a relationship with a sweet girl. So naturally, we invited them ...

Read More

What’s the best thing you learned from your parents?

One of my students asked this amazing question, and I loved it so much that I want to share it ...

Read More

8 Comments

 

Comments

  1. So true. Marketers not talking about null-results seems similair to the reporting bias in science.

    But they simply give people what they want to hear. Just compare the buzz about your positive results test to the response of this null-results test.

    Well played Ramit, you succesfully tested that talking about null-results sucks.

  2. I think Im subscribed to the wrong mailing list because I rarely ever get dirty jokes from you.

    • I don’t get dirty jokes from the e-mail list I’m on, either. Hmmmm…must be a multi-variate test to see if females are offended by dirty jokes. Guess what? Some of us aren’t.

  3. One of the ways to find your wins – and we use this- is to find a plausible failure. Meaning: you might try to find a variant that reduces conversion/lift. we have had better luck finding variance…this way.

  4. Love your truth telling Ramit!

    Most people that make a living investing in the stock market are NOT winners. Of the people who are winners – the top 5-15% – they make all of their money on only a few of their trades. So out of a 100 trades they made for that year, the majority of their money will have come from 5-10 trades they made where they hit it out of the park. Most of their other trades will have lost them money.

    Same thing holds true for poker – most of your winnings will have come from a few hands. Most of the hands you get will be garbage. Most of the hands you play will actually LOSE you money.

    This is the opposite of what your chimpanzee wiring wants you to do.

    We’re addicted to instant gratification because we live in an instant gratification driven society. When you smell pizza, you don’t go inside your mind and imagine how drained you’re going to feel 30-60 minutes from the time you shoveled the pizza into your pie hole. You don’t project 20 years ahead and imagine having 47 pounds of accumulated sludge that’s made up of old pizza, ice cream, potato chips, and gummy bears in your colon that’s lodged firmly in place that’s poisoning you from the inside making it all but impossible to take a nice healthy dump. Nope.

    What you do is imagine thing you want RIGHT NOW and then you run towards it. Thinking long term is something we don’t come wired with and have to train ourselves to do with mantras like, “80+% of what I try is going to fail”.

    This mindset gives you permission to be the perfect non-perfectionist. It makes it okay for you to shut down a test that’s not working and move on to the next experiment.

  5. in my experience I have noticed that although change does make an impact (thus testing works), it works short term mainly because of the variation (readers are not used to seeing a different look and feel).

    what many marketers will not reveal however is the post test results as they tend to normalize. thus, few months/years later, another test is necessary to shake things up, only to see results go back to the norm again.

    I feel it’s similar to the whole night club/bar concept. the venue doesn’t change, but the name changes every so often. one club shuts down just for the other to open up in the same place by the same owner.

    this is very interesting to observe

    • Great point, Sunil, and very true about results becoming attenuated over time. Almost nobody talks about that (party because nobody wants to hear it).

  6. “Despite being totally different designs, the results were basically identical.”

    Ramit, I respectfully submit (as someone who has overseen more than 1,000 tests and driven the optimization programs for Adobe and Intuit) that there are no *meaningful* differences between these two pages. Visual differences, sure, but in the *minds* of the 20,000 visitors who saw these, what impact did you expect these changes to have? What was your original hypothesis?

    The only hypothesis that I can reverse engineer is that the funny image of the dog would get more people to take the next step — and if that was your original hypothesis, now you know that it doesn’t work.

    And that’s a great learning, my friend — you actually did get something real from this test!

    “One of the surprising secrets of testing is that MOST tests produce no change.”

    This is so true — but again, respectfully, it is almost always the fault of the tester. Who’s driving the bus anyway? ;-)

    And the focus of testing should not be to see change — but to learn more about your visitors so that you can apply new learning to invoke change. Learn first; change later.

    This is great stuff and I hope you’ll continue to post more examples…

    Lance