Measure, Release, Measure Again

In the world of product development, there is far too little mentioned about the need and use of data in decision making. We’re easily pleased with ideas that seem logical to us, but may be terrible for the users of our product. We add new features, but we never check to see if the investment we made (in resources and hours spent) have actually paid off.

Instead of assuming, let’s take a look at how data can be easily gathered and utilized to inform our decision making process.

The Problem

“Why aren’t we gaining more users?” someone asked at our monthly meeting with management. “Do we need to spend more money on marketing?”

A simple “yeah, sure” would have been easier for me. But blowing tons of money on marketing usually masks other issues. Sure, it has its place, but why invest that kind of money if another issue is the root of the problem.

“I don’t know,” I said. (the 3 hardest words in the English language) But I said it with confidence. I honestly had not yet considered this question and knew I didn’t have data one way or the other. But I assured management I would spend the week trying to see if we had a leaky funnel somewhere – were users not coming to the site? Or were they coming but not signing up?

Google Analytics was an easy tool to reach for here. I spent some time digging through user patterns – seeing where they went after hitting the homepage. After only about an hour, I found the problem: a 65% conversion rate on our signup page. I knew we could do better.

Hypothesis

Now that we had a baseline set of data, we could begin to experiment. The first thought from our team was that the form was too long. We had our entire terms of service document as an iframe that was visible before clicking the submit button.

“Let’s just chop off that entire section and place a link to our TOS instead.” But we had another issue at hand – we wanted to capture more information about the user on sign up. So for our first test, we decided to split the difference: we added a few more fields, but we completely removed the TOS iframe.

I certainly had my doubts about this theory. But that’s just it, they were my doubts. It’s easy to sit in a meeting and argue about subjectivity. Making product decisions from the gut can be a great move, but you have to remember that it’s impossible for you to fully and accurately represent your entire user base.

After a week, we checked the numbers. Our conversion rate had dropped from 65% to 57%. The addition of the extra fields were scaring off more people than the TOS iframe had been. So we implemented another change: a simpler form to validate the theory that our original form was too intimidating. Our form needed to say: “Just give us the bare mimimum and get in here!” So we pulled it all back and had the form only ask the bare minimum to use the site successfully.

Lo and behold, our numbers jumped to 71% the very next week. We continued for 2 more weeks to make changes that eeked out another 4% to get us to a respectable 75% conversion. The need for the extra user info was still there – we still wanted to capture that data somehow. But we all agreed that increasing our conversion was more critical than getting that extra data at this step.

Next Steps

After seeing how easy it was to perform these tests, we feel in love with Optimizely – a great tool made for A/B testing. We looked for other ways to make small but meaningful changes to our marketing sites: smaller headers, larger headers, different copy in the navbar, different colored buttons. All of these were small changes to make, but we saw huge improvements in our metrics.

For you, it may not be a marketing homepage. Acquisition may not be the target – maybe it’s retention. So where’s the drop-off? Why are customers leaving? What pain points might your users be having that they’re not telling you about, but your data can reveal to you? Use the data you have – or create more through logging – but make your data work for you. Invest your time and money in features that you know will pay off; not just ones that “feel” right, but ones backed by solid data.