A new analytics platform crops up every day, each one more slick than the next. Don’t get me wrong- I’m thrilled to see attention paid to great UX design, but…
Beware of shiny reports!
Bait and Switch
When I first started using Omniture’s Site Catalyst product a few years ago, I stumbled on the Events Funnel report.
Holler! Just what I was looking for…or so I thought.
I was working in a subscription model business with a four-step trial signup flow, and we were driving traffic into the funnel from various campaigns; the first step/landing page varied by channel, but all users passed through the same subsequent steps (steps 2-4). I set up my event report, selecting our homepage as ‘step one’. I ran the report, already envisioning revealing the funnel visual to my boss, beaming as I’d paint the picture of how easy it would be for us to track fallout between signup steps for our campaigns.
But then, something didn’t add up. Where I’d expected to see fallout between the first and second steps of the signup full, the number of users who completed step two greatly exceeded the number that had completed step one. How could this be?! Here’s the thing about Ominiture’s Event Funnel Report- the steps in the report don’t have to occur in sequential order. This logic is missing entirely from the report. Step 2 was capturing traffic from all of our marketing campaigns, not just traffic that had entered through the homepage. What at first seemed like a brilliantly simple way to track signup funnel fallout was actually totally worthless to us.
Then, last month, I hit the jackpot; I found the funnel report I’ve been whining didn’t exist the last few years. KISSmetrics really excels in tracking funnel fallout. Simply set up the steps in your funnel, and bam!, you’ve got a funnel fallout report.
The problem with this? Unless you’ve identified the steps in the funnel you want to track, you’re taking a stab in the dark. For fixed funnels like the trial signup flow I was tracking in my old gig, this report is hotsauce. Want to track cart abandonment? KISSmetrics rocks at this. That said, it can’t bubble up the steps you’re users are taking or content they’re consuming prior to signing up that’s triggering conversion (Performable is crushing this, btw). Instead, it’s up to you to build a bunch of funnels based on your own hypotheses about the flows users are engaging in, and test them out in the tool. Question is, what might you be missing?
Superhot, but Not Very Bright
Then, there are the reports that look really cool and insightful, but aren’t at all actionable. During a demo, have you ever heard a sales guy say, “Your exec team will really love this report”? Bing, bing, bing! Red flag! This is sales lingo for, “This report looks totally sexy, but it doesn’t actually put out.”
Take the time to figure out what you’re looking for before you take the plunge. You might be okay with the report you get when you see what’s under the paint. Likewise, sometimes a one-trick pony will get you what you’re looking for. What’s usually guaranteed though, is that while ‘superhot but not bright’ might be fun for a few days, you’re probably not going to want to put a ring on it…