How to Measure Innovation?

I’ve just read a July 2008 report of a National Science Foundation conference, that brought together a panel of experts to try to figure out a way to measure innovation (click here for the full report).  They started with the observation that we don’t know very much about how to measure the outcomes of innovation, and we don’t know very much about what characteristics make the difference between a successful or failed innovation.

Briefly, they identified four areas that businesses and governments could measure.

  • Measure what innovation is. Determine the rates of return over a project’s lifecycle, and how this looks different for different kinds of innovation.
  • Study how and why innovation takes place, by examining the organizational structures and cultures of successful innovations (this is the focus of my own research).
  • Figure out how to capture the complete range of consequences of innovation, including the downsides of “creative destruction”– unemployment, regional winners and losers, and social responsibility.
  • We need a better understanding of the complete environment of innovation, from globalization to government IT law to tax policy to technological change, all is interconnected.

And then the panel had four recommendations:

  • Pursue new directions in data gathering measuring the net outcomes of innovation. This includes data on innovation processes within organizations.
  • Pursue new directions in measuring the net inputs (costs) associated with innovation.
  • Gather data on innovation processes that span across organizations.  (I like this one, because it acknowledges my claim about “collaborative webs”: that innovation is rarely bounded within the walls of one company.)
  • New information gathering and management techniques could allow us to confidentially aggregate data from many companies, and share it with everyone.

I looked at the list of participants and they all seem very qualified–with expertise focusing on economics, consulting, workforce and human resources, computer science, and business strategy. I didn’t see any of the top creativity or innovation experts, however, which was a bit puzzling. Innovation experts absolutely must be deeply involved in all of the steps recommended by this report, I’m not sure why they weren’t involved at this early stage.

Also see my post of March 4, 2008 about the www.innovationmetrics.gov panel of experts.

Measuring Innovation

A long-awaited report on how to measure innovation in the U.S. economy has just been released by the U.S. Commerce Department. The report is called “Innovation Measurement: Tracking the State of Innovation in the 21st Century Economy”. I first learned about this high-profile initiative last October; a press release revealed that a panel of CEOs and academics had met in Washington DC to discuss how to measure innovation in the U.S. economy. When I say “high profile” I mean folks like Microsoft CEO Steve Ballmer, Medtronic Chairman and CEO Art Collins, IBM CEO Samual Palmisano, and Harvard economist Dale Jorgenson.  The original press release said that the panel’s recommendations would be published in November; perhaps only an innovation junkie like me would be checking every week since then!

To measure the impact of innovation on the economy, analysts often use a measure called Total Factor Productivity (TFP). Any growth in TFP is assumed to result from innovation. Of course, the problem is that productivity could grow for other, non-innovation, reasons (for example, if existing innovations are diffused more broadly, TFP would grow even without new innovations). Other common measures of a country’s innovation have their own problems. You could count up the number of patents; but, patents alone don’t translate into successful innovation. You could count up the number of professionals working in R&D and university research labs; but as with patents, that’s a crude measure that doesn’t directly track successful innovation.

In the end, the panel’s report doesn’t tell us exactly what to do.  Panel member Ashis Arora, Professor of Economics and Public Policy at Carnegie Mellon, said that “The current advisory panel did not opt to recommend an index, because there is no serious evidence on how different measures of innovation should be combined, either at the organizational level or at the aggregate national level.”  However, Commerce Secretary Gutierrez outlined a plan for moving forward: a better measure of the impact of high-tech goods and services (to be developed by the Bureau of Economic Analysis and the Bureau of Labor Statistics); a better way to measure productivity increases that result from innovation investment (to be developed by the BEA); and new data collection efforts to measure the role of basic research (spearheaded by the National Science Foundation).

A longstanding problem has been getting different government agencies to share data with each other.  The stumbling block has always been confidentiality concerns.  Secretary Gutierrez announced his intention to work aggressively with the relevant agencies to try to find a way to share the relevant data while addressing confidentiality concerns.  That’s going to require working with a wide range of agencies including the Office of Management and Budget, the Council of Economic Advisors, the Census Bureau, and the Securities and Exchange Commission.  That’s a pretty tall order, but if that could happen it would result in a much better picture of national innovation.

Quote of the week:

“I don’t think people appreciate how much money, time and good technical research goes into what we do. Sometimes, people think the idea is the thing. I think the idea can be the easy part.” Dr. Darryle Schoepp, of Eli Lilly, in an interview about new drug development quoted in the New York Times (Sunday, February 24, 2008, Business section p. 10).