Internet Time KnowledgeBase: Metrics

The shortest presentation on metrics you will ever hear, a twelve-minute overview of where I'm coming from on metrics and measurements. Macromedia Breeze.

First presented at Queens School of Business, Kingston, Ontario, in May 2004.

"It is time, once and for all, to drive a stake through the heart of traditional accounting, which is draining the life from business." Tom Stewart, The Wealth of Knowledge

The title of my next presentation was Decision-Making Memes: Putting a Value on Learning. I explained that while I've been in the training business for nearly 30 years, before that I was a mainframe salesman, Army officer, and Harvard MBA student. I'm a business guy. I understand how business people make decisions. ROI doesn't have a heck of a lot to do with it. I'm not going to recount the story here. Later I'll put it on the web in narrated form. For now, as promised, here's the slide deck I used in my presentation. For more on this topic, visit the Metrics Page in the KnowledgeBase here.

Four years ago I attended a how-to-ROI presentation at a major eLearning event and found it so misleading that I began writing about how companies really evaluate project potential and after-the-fact results.

Recently I've noticed ROI Workshops popping up. Spend a couple of days and the better part of a thousand dollars. Get a certificate. Such a deal. Unfortunately, neither the workshops nor the conference presentations cover the things I deem important:

Metrics are in the eye of the beholder. They are not simply the application of a rote formula or accounting rule. They are subject to interpretation. This is what makes them worthy of discussion.
The internal customer for metrics is your sponsor, also known as the person who pays the bills. When you talk with an executive, you need to talk about execution, not training.
The only valid metrics for corporate learning are business metrics. To converse in business terms, it helps to be fluent with the concepts of trade-offs, risk assessment, expected value, focusing on core, changing perspective, the 80/20 rule, and the bottom line.
Business goals. Strategic initiatives. Quarterly objectives. New product introductions. Figure out what matters in your organization. Then show the connection between what you do and what matters. It will make you an insider instead of an outcast.
Kirkpatrick's four levels are a history lesson, not a guide to action. Imagine telling your sales manager that the sales force was well prepared ("Levels 1 & 2") but simply hadn't sold anything ("Levels 3 & 4"). Good luck in your next job.
Most of a company's value resides in the know-how and relationships of its people. Traditional accounting assigns these intangibles a value of zero. Hence, traditional ROI has little credibility with enlightened executives.

Rather than update my various white papers and articles, I have consolidated my thoughts into a single one hundred-page eBook called Metrics. Check it out.

Read Carl Binder's review of Metrics in Performance Improvement.

Metrics 1.0 was a screed against traditional ROI and conceptual argument in favor of a broader approach. I think my readers want something more concrete. In my spare time, I'm looking into Balanced Scorecards, capital-flow assessments, managerial economics, and methods for evaluating intangibles.

My anger at seeing time wasted looking at the wrong end of ROI equations (i.e. cutting costs as opposed to taking advantage of opportunity) that I've been writing about this topic for a while. Some of the favorites:

Balanced Scorecard Collaborative - Strategy, not numbers.

Power Shift

eLearning infrastructure decisions are climbing up the corporate ladder. A few years ago, eLearning was pigeonholed as a cheaper, faster way to train employees. By default, eLearning decisions fell to the director of training or HR.

Now, functional managers are using eLearning to meet business objectives. Managers look beyond employees to customers, suppliers, and distribution channels -- everyone benefits from seeding eLearning throughout the value chain. This is where we are now, with eLearning decisions seesawing back and forth between can-do functional managers anxious to get on with it, and CIOs/CLOs who want to go the next step to enterprise solutions. Still rare but perhaps the next step in this evolution is the CEO who looks at eLearning as a competitive weapon, the way to create a nimble organization, improve customer service, move quickly, and stay ahead of the pack.

Knowledge Advisors 2005 Learning Analytics Symposium: Microsoft, DAU, Nextel, Bersin

ROI

January 10, 2003. Those of you who've read my thoughts on ROI know that I believe cost/benefit analysis is manditory and most ROI calculations are utterly worthless. Thus, I was delighted to come upon Enough Already! Getting Off the ROI Bandwagon by Kevin Kruse (mistakenly identified as Kevin Kenexa) in the current issue of Chief Learning Officer magazine.

Kevin writes that:

First came the articles, then the books, and now I see that an entire conference is devoted to the ROI of training. Obviously we're seeing a backlash against the orgy of IT spending of the late 1990s, and against e-learning initiatives that fell short of expectations. Personally, I think it's all hype, and I've had enough.

First, many senior executives don't care about ROI. In Jack Welch's book, "Straight From the Gut," he tells of his decision to invest millions in GE's new Crotonville training facility, even while undertaking massive layoffs. He didn't have an ROI spreadsheet to tell him training was a good investment; he just knew that investing in talent was critical to GE's future.

Second, ROI is an imperfect science that often involves making educated guesses at potential savings and gains. Senior executives know this, and they also know that there are many variables that can't be captured by a formula.

Third, ROI guesstimates are often a cop-out for tougher measurements of results. How about measuring employee engagement scores before and after management training, or doing pilot studies of sales training programs that measure closing ratios and time-to-close?

Systems Changes

Traditional ROI has suckered corporations into evaluating learning initiatives on a project-by-project basis, and this has lead to supporting each new approach as if it existed in isolation. The Meta-Learning Lab is developing ways to improve the overall learning process.

Take the old cliché of "Give a man to fish and he won't be hungry today. Teach a man to fish and he will never be hungry again." (Excuse the sexism; this dates back several thousand years.) The Meta-Learning Lab's goal is to teach fishermen how to improve their catch.

Chuck Fred and I are both obsessed with time. Chuck's a former competitive runner and the "breakaway" of his book's title is that point when the winners pull ahead of the also-rans. It worked for Jesse Owens and it works for Wal*Mart. The name of this site is a reflection of my view of time. Time has become the prime business metric. How soon can our team reach proficiency? How can we get there faster? How can we stay ahead of the game? How can we speed things up? How soon will we be ready to execute?

The genesis of Chuck's book, Breakaway, was interviews with 300 CEOs. He promised them absolute confidentiality in return for their candor. He maintains these relationships to this day.

Late last year, Chuck asked the CEOs about their levels of confidence in the ROI presentations made in suport of training expenditures. Specifically, he asked about purchases of off-the-shelf courseware, training technology & infrastructure, and training-related advisory services.

Nine out of ten CEOs said they had no confidence in the ROI of training as presented to them. You can reach Chuck at Breakaway Group.

Scientific rigor: The Baloney Detection Kit

How to draw boundaries between science and pseudoscience, or between useful metrics and pure hype. From Scientific American

1. How reliable is the source of the claim? 2. Does this source often make similar claims? 3. Have the claims been verified by another source? 4. How does the claim fit with what we know about how the world works? 5. Has anyone gone out of the way to disprove the claim, or has only supportive evidence been sought?

6. Does the preponderance of evidence point to the claimant's conclusion or to a different one? 7. Is the claimant employing the accepted rules of reason and tools of research, or have these been abandoned in favor of others that lead to the desired conclusion? 8. Is the claimant providing an explanation for the observed phenomena or merely denying the existing explanation? 9. If the claimant proffers a new explanation, does it account for as many phenomena as the old explanation did? 10. Do the claimant's personal beliefs and biases drive the conclusions, or vice versa? "Clearly, there are no foolproof methods of detecting baloney or drawing the boundary between science and pseudoscience. Yet there is a solution: science deals in fuzzy fractions of certainties and uncertainties, where evolution and big bang cosmology may be assigned a 0.9 probability of being true, and creationism and UFOs a 0.1 probability of being true. In between are borderland claims: we might assign superstring theory a 0.7 and cryonics a 0.2. In all cases, we remain open-minded and flexible, willing to reconsider our assessments as new evidence arises. This is, undeniably, what makes science so fleeting and frustrating to many people; it is, at the same time, what makes science the most glorious product of the human mind."

Corporate Learning Strategies by Dan Tobin. "If you start and end all of your learning efforts by focusing on your organization's goals, you will never be asked to do an ROI analysis to justify your budget."

There's no cookbook approach to measuring the ROI of training. Fred Nichols is so right about this. Because the definition and perception of value varies from person to person, so do the purposes of evaluation. Moreover, the various audiences for evaluation frequently act as their own evaluators. If you look carefully about you, or if you reflect upon your own experiences as a "trainee," you will quickly discover that training is being evaluated every day, but by trainees, managers, and executives -- and in accordance with their criteria and purposes.

Payback

Technology-enabled learning creates value by speeding things up. Business-school professors compare making big corporate changes to turning around the Queen Mary. Turn the rudder and in a few miles, the ship changes course. These days, organizations that lack the agility to turn on a dime can only go about as far as the Queen Mary (which is moored in cement alongside a pier in Long Beach, California.)

A Fortune 50 company used eLearning, knowledge management, and collaboration to bring new-hire sales people up to speed in six months instead of fifteen. Nine months x 1400 new hires/year x $5 million quota = $5 billion incremental revenue. To be sure, better products, sales campaigns, and a host of factors contributed to the gain but a tiny faction of $5 billion still yields a significant ROI. (Here are the details: New-hire training at Sun Microsystems.)

Ten thousand consultants at a Fortune 100 technical services company earned professional certifications via eLearning. The result? Less attrition, better esprit de corps, and $100 million revenue/year attributable to higher billing rates.

A software firm launches a new system into a $250 million global market with eLearning and virtual meetings. This accelerates time-to-market by two months, gives them first-mover advantage over a major competitor, builds a more confident and enthusiastic sales force, and gets the channel up to speed at the same time as the direct sales force. Gain? $80 to $100 million incremental revenue.

A very large retailer of personal computers realizes that customers are frustrated with their products because they don?t understand the software that accompanies them. The company offers customers free admission to an online learning community created by SmartForce. More than 100,000 customers sign up to learn Windows, Word, and Office apps online. Value of increased customer loyalty? Conservatively, $20 million in repeat business over three years.

Often an e-Learning initiative pays for itself right off the bat by eliminating travel and facility costs, but that misses the point, because in comparison, upside gains dwarf cost savings.

The eLearning Emperor Has No Clothes

Go to any major conference for trainers and you'll find many sessions on evaluating results and measuring performance. If you're a line manager with no training background, you will at first be confused when participants make statements like, "We evaluate 100% at Level 1, 80% and Level 2, and 40% at Level 3. We're going to shoot for some Level 4 next year."

The "levels" come from a taxonomy developed by a budding academic, Donald Kirkpatrick, as his Pd. D. thesis more than forty years ago. Level 1 evaluates trainee reaction (generally via evaluation forms derisively known as "smile sheets.") Level 2 checks retention (can they pass the test?) Level 3 looks at whether theydo what they were trained for. Level 4 is whether the learning creates meaningful results for the organization.

Picture this. A national sales manager is reviewing quarterly sales performance with his boss. He tells her the new sales trainees scored 95% on Level 1, 82% on Level 2, and 9% on Level 3. Unfortunately, Level 4 improvement was infinitesimal. So the sales force loved the sales training, the majority passed the test, and nearly four out of five could demonstrate great sales behavior in a role-play. The only trouble is Levels 3 & 4: they aren't selling. How long would the sales manager keep her job? In business, Level 4 is, in fact, the only thing that matters. No wonder senior managers question the value of training.

The only valid measure of training is business metrics, not training metrics.

As the Godfather said, "This is business." If you can't see a benefit, don't do it.

Jack Zigon's list of performance measurement sites

Excerpt from Ed Trolley's Running Training Like a Business

Evaluating e-Learning by Dorman Woodall

Jay's notes on making the business case, new ROI challenges

The trouble with the "four levels" is that they falter when they go outside of the limited context of training. What happens outside the box is what counts inside the box. You can guess how I see this.

BNH on ROI. Their software models simlify complex ROI calculations.

Discussion group: ROInet

The Business Case website

The Fallacy of ROI Calculations

Measuring the Success of Training

Baruch Lev

Measuring the ROI of Training, CIO

Economic Value Added (EVA)

Metrics and Web Services

Today, somewhere over Texas, I was reading John Hagel's Out of the Box, a wonderful description of the power of Web Services. The "box" of the title is actually a series of boxes, and the "most insidious box of all ... is the box that we all create in terms of the mind-sets we bring to our businesses."

Web Services = overlaying legacy systems with interoperable Internet-style concepts to enable computers to understand one another without human intervention. The next step in the evolution of computing.

Business managers are stuck in their ruts. And largely unaware of it. Sweating bullets but not knowing why. Web Services are part of the way out.

I'm also in the midst of rewriting Metrics, and I found these lines of Hagel's so appropos that it stunned me:

The advocates of business process reengineering challenged conventional business practices, but at the end of the day, they remained firmly within a mechanistic mental model. Even the language they used shaped, and revealed, their outlook. Reengineering -- could one possibly choose a more mechanistic, top-down, deterministic view of business activities?

Hagel (and I hope he pronounces it "Hegel" and not "haggle") points out that when reegineering types talked about end-to-end, the end of the world was the wall of the enterprise silo. End-to-end didn't encompass raw materials at one end and customers at the other. We don't need no stinking value chain.

Web Services are captivating because they can be adopted for demonstrable short-term gains, all the while laying the foundation for radically more malleable business models.

All business executives understand financial leverage. Use somebody else's money alongside your own, and you grow faster. You don't leverage yourself to the hilt, for that's risky. But if you don't leverage yourself at all, that's foolish.

The flexibility brought on by Web Services creates the opportunity for operational leverage. If I want to grow my business, why shouldn't I have somebody else's assets alongside my own?

I flew across the country today to meet with a major client. They requested I book my travel through their travel department. American Express. Why? Because for my client, the travel business would be a diversion. It's not something they would ever get out-sized returns from. So they farm it out and have more assets to put behind their core operations.

When Web Services are widely adopted a couple of years hence, companies will be able to swap a lot more than travel administration under or out from under their umbrellas. Hagel suggests that the largest gains will be from transferring major business processes such as maintaining customer relationships, managing infrastructure, and creating & commercializing new products.

How does our engineering mind-set manager adapt to that? It's not the old scenario of "draw the blueprint and then manage activities to it". This is more like rewriting the blueprint whenever you see it's to your advantage to do so. We have a name for people who stick to their old plans when new plans would take them further; we call them losers.

Now I'm pondering three sorts of >strong>value that make business worth doing. You can measure the first without leaving the enterprise silo. This is value from operations. You boost value by increasing revenues or decreasing costs. You can increase revenue by selling more stuff or selling at higher prices, by selling more through agents and partners, by adding new products or by increasing prices. You can decrease costs by being more efficient, achieving higher quality/fewer rejects, and leveraging intangibles (such as customer loyalty, employee retention, effective work processes, team experience).

Shareholders are more interested in a second form of value, market capitalization, i.e. the value of the stock. Share prices are set by the market, based on investors' perceptions of the firm's earnings potential, discounted for risk and time. This in turn rests on competitive advantage as evidenced by innovation, patents, social capital, executive smarts, reputation, market position, confidence, and inspirational management. Market cap is only loosely coupled with profitability.

Hagel started me contemplating another sort of value that traverses an enterprise's traditional boundaries. Back to operational leverage. If my company's operations are interoperable with other companies', I can pick and choose what I want to focus on. If I'm truly interoperable, I can farm out just about any business process and use the capitalI would have spent there leveraging my primary business. Or I could consciously set out to capture the highest margin segments of my entire value chain.

A company whose IT is built on Web Standards by definition has more options to leverage operations than one saddled with proprietary systems. In time, the equity markets will pay a premium for such adaptability.

Three sources of corporate value are:

value from how well the business has operated in the past
value from investors' perceptions of how well the business will do in the future
value from changing what the business does -- or having the flexibility to do so

This all sounded a lot more exciting when Hagel wrote about it, but he filled a book rather than a blog entry in the telling.

Learning at home

Training magazine, the March 2000 issue: Train on your own time, not "during work."

Sure, moving training from the classroom to the Web can mean reduced travel costs, less learning time away from the job, and certainly lower delivery costs. But most corporate training doesn't require travel, says Paul Reali, president of CyberSkills Computer Training Centers in Winston Salem, NC. And, he points out, no valid study has yet shown that online delivery significantly reduces learning time - actual time spent mastering a skill or acquiring knowledge-compared with instructor-led training of similar quality.

"No one wants to tell you that the 'anytime' of online learning is supposed to be after work and that the 'anyplace' is at home," he says. Another reason: Despite yellow crime-scene tape barriers and "do not disturb" signs, the cubicle is a tough place to have a quality learning experience. And it's almost impossible to reserve the necessary time and concentration without broad organizational support--and the backing of trainees' immediate managers for regular learning time-outs.

This is so true and so short-sighted. CapitalWorks

Capitalworks' logic and findings around measuring impract are the best I know of. They inspired my understanding of informal learning and metrics. The Capitalworks material is so compact yet so eloquent that it's almost poetry. Let me amend that. It's poetry if you're conversant with the concepts of finance.

Jeff and his partners get it. Jeff contends that "Learning is the single greatest contributor in all enterprises to superior operating performance and robust value creation."

Capitalworks stalks Learning Effectiveness, defined as:

The performance of an organization's applied learning portfolio in contributing to operating performance and value creation. Applied learning includes formal learning (training) and informal learning occurring naturally in social practice.

Why is learning vital?

Learning enables flows and exchanges of knowledge through diverse intra- and inter-enterprise interactions.
Learning transcends hierarchical constraints.
Learning connects demand drivers.
Learning accelerates systemic effects.

Learning is the great enabler of flows and exchanges of knowledge. With flow, you are primed. Everyone has workarounds. Workarounds are really positive. Learning transcends hierarchical constraints. Organizations are not optimized to connect demand drivers. In fact, we're living with obsolete, 19th century organizational structures created for an illiterate workforce long before the advent of computers. Jeff points out that "Optimizing dimensions, dynamics and drivers of learning are natural means of transforming costs of coordination in all enterprises and their ecosystems." Learning itself is the ultimate workaround.

Learning is one of our primary earning assets and we should manage it that way. Looking at the flows, here's the Value Creation Circulatory System:

It's nonlinear, continuous. Process orientation. Feedback loops are critical. A single measure doesn't get us there. (Emergence, emergence....)

What enables flow? Self-study contributed as much to job proficiency as instructor-led training programs. Own volition. Regard selves as professionals. Informal learning dynamics contributed 70-to-80% of operating performance. Cohesion of social practice contributed to learning effectiveness, with informal learning as an enabler. Conversations are the primary conduit:

Read this one twice if you need to; it's important. "We see contributions by learning, like other intangibles, through value drivers. They enable us to depict causal relationships in the interactions associated with transactions, decision flows, procedures and other normal activities. Value drivers interact in clusters and sets throughout organizational work practices."