Organizations have a personality, and it bleeds into everything from executive reporting to product offerings. A recent Fortune article entitled Microsoft without Gates offers this wonderful tidbit about Steve Ballmer, CEO of Microsoft:

Even though he never was a serious computer programmer, by all accounts Ballmer is just as good at math as Gates is. He lives and breathes data. “Steve has a computer in his head,” says Bob Muglia, a 20-year company man who heads the Server and Tools division. Ballmer expects his subordinates to be adept in math as well. He distributes 11-by-17 sheets filled with numbers detailing the progress of various operations. The numerals are so small that executives use transparent magnifier rulers to see them. But there are never any columns showing percentage changes. Ballmer believes people ought to do that in their heads. It saves space on the paper for more numbers.

Wow. If it is as bad as the author describes, Ballmer has designed the anti-dashboard.

The Presentation Zen blog offers another great example of organization culture as displayed in business artifacts.

Gates here explaining the Live strategy. A lot of images and a lot of text...Good graphic design guides the viewer and has a clear hierarchy or order so that she knows where to look first, second, and so on. What is the communication priority of this visual? It must be the circle of clip art, but that does not help me much.

Does it get more "Zen" than this? "Visual-Zen Master," Steve Jobs, allows the screen to fade completely empty at appropriate, short moments while he tells his story.

We frequently get requests to review and write about analytics-related solutions. I’ve put off most of these requests because it sounded like a lot of work. Then I had an 4-hour-work-week-style epiphany: most new users only give a new product or service a few minutes before they make up their mind. Why can’t I make the same snap judgement and call it an expert opinion?

First up is Open Flash Charts, pointed out to us by Matt Bear. This is an open source project started by John Glazebrook to provide flash charts that can be embedded in web pages. I love John’s explanation for taking on this project:

“Once upon a time I had to deal with a company who sell flash charting components, their component had a bug that I needed fixing, so I emailed them about it asking when it’d be fixed. (Remember that I had paid real money for this software.) They were so incompetent, rude and obnoxious that after three or four weeks of emails I thought to myself “I could learn Flash and Actionscript and write my own charting component, release it as Open Source, host it on sourceforge and build up a community of helpful coders faster than they can fix a single bug.” And that is what I did. And that is why it is free. I guess the moral of the lesson is: don’t piss off your customers.”

Great lesson. Great attitude. There are a bunch of vendors in this space (Fusion Charts, AnyChart, ILOG, PHP/SWF Charts, amCharts, Corda) and the going price seems to start at $500 for a developer's license up to $5,000 for an enterprise license. (Apparently that doesn’t always come with customer service.)

Open Flash Charts isn’t as flashy as any of these products, but that tends to be a good thing for charting components. Here’s a column chart from Fusion charts (notice how each bar is a separate color, for no good reason)

Here’s the Open Flash Charts

Open Flash Charts does a number of things well:

• It seems to be easy to implement. Basically, you just copy the Open Flash Chart SWF file into your web server, then start embedding flash charts into your HTML and point to either static or dynamic data on your server.
• You can configure data labels, background, number formats, on-click events, tooltips, etc.
• All the basic chart types are available (bar, line, area, pie, scatter).
• The help forum seem both lively (multiple messages a day) and supportive (a generally polite tone with lots of code posted).

On the negative side, Open Flash Charts doesn’t totally succeed in terms of data visualization fundamentals. The default charts have some contrast issues, odd color choices, and a little excess chartjunk. And when the charts get some “pizzazz,” things get worse:

I know… it is an open source project, so I should step up and fix the things I don’t like. I would, but I just ran out of my 10 minutes.

There is a tendency with reporting, and dashboards in particular, to cram as much information on the page as possible. It is a problem that Avinash describes with typical candor:

“This one of the core reasons why most dashboards are 'crappy', i.e. they are data pukes that provide little in terms of context and even less in terms of actionable value.”

In the past, we have offered tools to make data presentation as clear as possible (chart chooser, Excel chart cleaner). Sometimes clean isn’t enough; a more dramatic approach is needed.

One alternative is to shift the focus from the full data to changes in the most critical data points. By pulling out the important exceptions, you can make it easier for your audience to digest what matters and take action.

Stephen Few says in his book Information Dashboard Design:

“The best way to condense a broad spectrum of information to fit onto a dashboard is in the form of summaries and exceptions…given the purpose of a dashboard to help people monitor what’s going on, much of the information it presents is necessary only when something unusual is happening; something that falls outside the realm of normality, into the realm of problems and opportunities. Why make someone wade through hundreds of values when only one or two require attention? We call these critical values exceptions.”

Alerts are one mechanism to turn the focus to the exceptions, outliers and data highlights. Whether embedded in the dashboard or presented separately, alerts can be the extra layer of abstraction that make a dashboard useful. Unfortunately, they are hard to get right. I’ve arrived at four C’s for effective alerts—context, cogency, communication, control. Here’s a checklist to consider as you build alerts into a dashboard or report:

Context: Users need to understand how an alert is defined and how it fits into the larger picture.

• Are the parameters well defined? An alert is commonly defined by the following factors: metric (e.g. revenue), dimension (e.g. time), delta (e.g month over month change), scope (e.g. Northeast region, Peanut-product line), threshold (e.g. increase or decrease of 10%).
• Is the timing of the alerts actionable? One client explained to us that fluctuations in many of their metrics make monthly alerts too frequent—it would unnecessarily alarm people when, from their perspective, no significant trend had been established.
• Is the change statistically significant? This is of particular importance when you are measuring deltas. A doubling of traffic from a referring site doesn’t mean much when it is moving from one to two visitors.

Cogency: An alerting system needs to avoid causing unnecessary alarm while delivering easy-to-understand information that can be acted upon.

• Can the alerts be described in simple terms that even an executive can understand? Alerts should have a real-world meaning that users are familiar with. If an alert is based on a complex metric, for example, users will be confused as to the implications.
• Is the alert actionable? In the best cases, alerts should point users to both the drivers of the alert and the actions that can address the situation. This system does neither: ![terror warning system]
• Are the alerts so granular and/or frequently triggered that users will get alert fatigue? Excessive use of alerts will undermining their credibility. We saw this happen at one client where an IT-designed system threw off alerts like they were going out of style. The application went out of style the next year when users decided it was more distracting than useful. Here’s another example of a system that seems designed to raise blood pressure.

(It appears that a 5% increase in brand attribute performance isn’t good enough to get you out of the yellow.)

Communication: Alerts must be designed to effectively capture attention and inform.

• Is the alert placed in context? Google Finance does a nice job of putting news alerts within the stock chart.
• Is it clear what the user should do next? Give the user a clear path to more information so they can understand the full context of the alert.
• Does the sophistication of your alerts match the sophistication of your audience? I’ve found that it is better to start with some simple alerts so your audience can begin to learn what they mean and how to react. Over time, these alerts can become more refined and focused to capture complex situations.
• Does the alert draw the eye without being visually overwhelming or annoying? Here’s a article about how to “reduce visual noise” in dashboards.
• Is color used appropriately? Red means bad. Yellow is sorta bad. Green means good (but “good” things don’t need to be alerts). It isn’t particularly fair for color blind folks, but these conventions are deeply rooted.
• Have you found the best mechanism for presenting alerts? Alerts can be sent through e-mail, as SMS message, blasted over the office intercom system, or posted to the wall in the bathroom. What is the most convenient and appropriate medium?

Control: Advanced alert system should give users the ability to customize and manage alerts.

• Can the user identify the important alerts for them, and avoid the others? As hard as you may try in designing the dashboard or report, you aren’t in the shoes of the users. They will learn what they want to pay attention to and what information is extraneous.
• Can the user adjust the parameters? With more sophisticated dashboards, you want to give users the ability to adjust parameters to hone in on the exceptions that really require action.
• Can the user analyze alert frequency and trends? I’ve never seen a system that does this, but having the ability to view and analyze alert history seems critically important to getting a holistic view of performance.

After deep introspection, Stephen Few has determined that graphs with dual-scaled axes are fundamentally flawed. Rather than risk the potential for confusion, he believes that there are superior graphing approaches for situations where related data series have different units or magnitudes. His measured and thorough analysis concludes:

“It is inappropriate to use more than one quantitative scale on a single axis, because, to some degree, this encourages people to compare magnitudes of values between them, but this is meaningless.”

I commend Stephen for the courage to start down this path, but he hasn’t gone far enough. Here at Juice, we must often take controversial positions. You may remember that we were among the first to criticize Microsoft’s “databars”, the first to take on the powerful Dashboard Gauge lobby, and the first to challenge the applicability of Tom Davenport’s “Competing on Analytics” sales machine.

While it is true that the second axis can be deceptive, let’s not let the first axis off without asking some tough questions. It is the confusion—nay, the collusion—of the two that causes trouble—who is to say which is the bad seed? We must ask ourselves, do not axes belong in the “Axis of Evil”?

The problem is broader than Stephen suggests: axes are just the tip of the iceberg when it comes to graphic bling that potentially distract or confuse readers:

Take data labels, for example. They encourage users to consider specific values rather than focusing on relative sizes or placement of graph lines or bars.

Legends draw the reader’s eye away from the central storyline of a graphic.

Gridlines… please don’t waste my time with these flat faux-series. One wouldn’t put pinstripping on a Ferrari.

Place your graph in proper context and titles become redundant.

Minimalism is in. Extraneous graph decoration is out. Look no further than Tufte’s sparkline: no excessive graph decoration there.

The world cries out for a new charting aesthetic. One that champions elegance and casts down gaudiness. Let us evoke the pure visual essence of the data. Let us find a pure form to evoke the emotion and hidden meaning of the data. Now is the time for Naked graphs—stripped to the essentials (TM).

Our argument is simple: the visualization of information is the message. The data is but an intermediary form of that visualization. Therefore, any residue from the raw data should be scrubbed from your final graph. Only when you achieve this unadulterated state will the meaning of the graphic burn its way into your consciousness.

Here’s an example of an analysis that casts light on both the relationship of the Fed to hedge funds while simultaneously answering your question about what happened with last month’s sales in the Newark division.

Truly here we see the words of Mark 9:43 made real:

If your hand causes you to stumble, cut it off; it is better for you to enter life crippled, than, having your two hands, to go into hell, into the unquenchable fire.

Gaze in awe, viewers, and find wisdom on this very foolish day.

Sometimes a simple metric isn’t enough. It can’t fully describe a behavior or performance of a system. That’s when you need a Franken-measure: a made-up metric monster that creates a comprehensive composite to capture complex concepts.

Franken-measures go by many names—indexes, scales, ratings, composite or compound measures—and show up in all sorts of places:

Web analytics has an ongoing discussion about a measure of visitor engagement; the famous Google PageRank measures the “importance” of sites using a complex and mysterious algorithm.

Sports have embraced Franken-measures to evaluate player and team performance, e.g. passer ratings, Rating Percentage Index for college basketball, and judging of Olympic events like gymnastics, ski jumping, and ice dancing.

Economists loves indexes, e.g. Consumer Price Index, Consumer Confidence Index, Gross Happiness Index.

Marketers use “scores” to simplify their lives, e.g. Q scores measure the familiarity and appeal of popular culture entities and credit scores judge your value as human being.

Why would I want a Franken-measure?

You are probably already up to here with measures, so why would you want another one—much less one that is going to need extra effort and explanation? Here are a few things Franken-measures can offer:

A short-hand way to communicate about a complex concept. For example, a concept like customer loyalty may encompass everything from share-of-wallet to frequency of interactions to average sales amount.

A mechanism to operationalize a complex concept. Systems can take action on a single number more easily than an array of variables.

A definitive weighting of factors. Rather than constantly bickering about the relative importance of various measures, a Franken-measure can lock down the weighting, avoiding individual biases (in exchange for a systematic bias).

A balance of components. By combining multiple measures, variation in one measure doesn’t unduly bias the results.

What does it take to design an useful Franken-measure?

Not all Franken-measures are effective at achieving these benefits. There are at least four elements that contribute to a good design: completeness, concision, measurability, and independence. These factors can be combined into the Franken-measure Effectiveness Index (FEI) using Juice’s proprietary weighting model.

Completeness. Modeling all relevant performance factors to provide a holistic measurement of the concept.

Concision. A calculation that is as simple and straightfoward as possible, making it understandable and logical to users.

Measurability. Using direct performance data rather than relying too heavily on proxies or subjective measures. And from a practical perspective, if you can’t reliably gather valid data, the exercise is futile.

Independence. The components of the measure need to be independent so that variation in one component doesn’t directly drive another.

What can go wrong?

Finally, here are a few of the pitfalls to avoid when setting out to create your perfect Franken-measure:

Complexity. A complex calculation can confuse and infuriate your audience because it is hard to understanding what is driving performance and why the measure is moving. Leigh Steinberg, famous NFL agent, said of the NFL passer rating: “Other than one attorney in our office, I am unaware of a single human being who has the capacity to figure a quarterback rating.” The formula isn’t quite as inpenetrable as that, but it isn’t for the weak of heart:

Changing the baseline. There will be inevitable pressure to change the franken-measure formula which automatically invalidates historical performance.

In search of comprehensiveness. A desire to be comprehensive can hamstring the effort. Take Eric T. Peterson’s Engagement Model. He is clearly striving for completeness but at the risk of feasibility, in my opinion.

Black box and credibility. For the people impacted by a Franken-measure, it is important to understand what is going on under the covers. And if it is impossible to share the algorithm or approach, credibility of the creator is all that remains. PageRank succeeds to the extend that people trust that Google has an objective, well-intentioned algorithm. A whiff of agenda or bias would undermine it in the eyes of the audience. Take the National Review’s “Liberal Rankings” which have managed to label the last two Democratic Presidential nominees as the “Most Liberal Senators.” Coincidences like that can undermine credibility.

For more information:

• Compound Measures, University of South Carolina
• Composite Measures, Wikipedia
• A Primer on Metrics, Gary T. Smith
• Choosing the Right Metric, Juice Analytics
• 5 Rules for Successful Success Metrics, Juice Analytics