Sarah Hodges

Objects in the analytics dashboard may appear closer than they are…

In Web Analytics on February 14, 2011 at 10:00 am

A new analytics platform crops up every day, each one more slick than the next. Don’t get me wrong- I’m thrilled to see attention paid to great UX design, but…

Beware of shiny reports!

Bait and Switch

When I first started using Omniture’s Site Catalyst product a few years ago, I stumbled on the Events Funnel report.

Holler! Just what I was looking for…or so I thought.

I was working in a subscription model business with a four-step trial signup flow, and we were driving traffic into the funnel from various campaigns; the first step/landing page varied by channel, but all users passed through the same subsequent steps (steps 2-4). I set up my event report, selecting our homepage as ‘step one’. I ran the report, already envisioning revealing the funnel visual to my boss, beaming as I’d paint the picture of how easy it would be for us to track fallout between signup steps for our campaigns.

But then, something didn’t add up. Where I’d expected to see fallout between the first and second steps of the signup full, the number of users who completed step two greatly exceeded the number that had completed step one. How could this be?! Here’s the thing about Ominiture’s Event Funnel Report- the steps in the report don’t have to occur in sequential order. This logic is missing entirely from the report. Step 2 was capturing traffic from all of our marketing campaigns, not just traffic that had entered through the homepage. What at first seemed like a brilliantly simple way to track signup funnel fallout was actually totally worthless to us.

One-trick Pony

Then, last month, I hit the jackpot; I found the funnel report I’ve been whining didn’t exist the last few years. KISSmetrics really excels in tracking funnel fallout. Simply set up the steps in your funnel, and bam!, you’ve got a funnel fallout report.

The problem with this? Unless you’ve identified the steps in the funnel you want to track, you’re taking a stab in the dark. For fixed funnels like the trial signup flow I was tracking in my old gig, this report is hotsauce. Want to track cart abandonment? KISSmetrics rocks at this. That said, it can’t bubble up the steps you’re users are taking or content they’re consuming prior to signing up that’s triggering conversion (Performable is crushing this, btw). Instead, it’s up to you to build a bunch of funnels based on your own hypotheses about the flows users are engaging in, and test them out in the tool. Question is, what might you be missing?

Superhot, but Not Very Bright

Then, there are the reports that look really cool and insightful, but aren’t at all actionable. During a demo, have you ever heard a sales guy say, “Your exec team will really love this report”? Bing, bing, bing! Red flag! This is sales lingo for, “This report looks totally sexy, but it doesn’t actually put out.”

Take the time to figure out what you’re looking for before you take the plunge. You might be okay with the report you get when you see what’s under the paint. Likewise, sometimes a one-trick pony will get you what you’re looking for. What’s usually guaranteed though, is that while ‘superhot but not bright’ might be fun for a few days, you’re probably not going to want to put a ring on it…

  1. Great post!

    Love this – “This report looks totally sexy, but it doesn’t actually put out.”


  2. I second Rudi’s comment!

    The perennial problem for technology vendors and their sales teams is that their platform is a hammer, and they see all the world as a nail. They struggle — understandably, as it’s just how the human brain works — to see how someone would have any need for which their tool is not perfectly suited. If such a question is posed to them, the immediate reaction is, “That’s not a valid question. Rather, you should be looking at .”

    No analytics platform actually provides: “Easy and intuitive access to all the data…after simply adding a couple of lines of Javascript to your site.” Unfortunately, all of the vendors pitch that they do just that.

    • Great comment. I agree- too often vendors develop platform capabilities based on an ‘if you build it, they will come,’ mentality, rather than truly addressing customer need and the pain points their prospects are currently experiencing. Some of the newer players are doing this quite well (I mention Performable in the post); I’ve also been encouraged by Omniture’s recent attempts to incorporate community feedback into its product roadmap. Vendors who rapidly evolve their platforms by getting in front of customers and distilling feedback into the product features that truly drive value will emerge on top.

  3. Great point. However, Omniture does provide a sequential page funnel report in Omniture Discover though it doesn’t quite help the Site Catalyst-only customers.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: