I’m writing a series of posts about Generalizing Apdex. This is #4.
I ended my earlier post on Core Apdex Qualities with this observation: “I believe that a core benefit of Apdex is its usefulness as a management indicator (such as a Key Performance Indicator, or KPI). I don’t think the current spec really captures that aspect properly, so I’m going to devote my next post to that subject”.
Since then I have done extensive research on the subject of performance indicators, resulting in a collection of relevant papers, articles, and blog posts–see my later post on Metrics and Performance Indicators: A Bibliography. I have also been extending my original list of 20 references, appending several newer discoveries of relevant online resources as comments. But much as I enjoy doing research, at some point I have to stop and draw some conclusions, however tentative. So here’s a bit more detail about the role of Apdex as a (key) performance indicator.
Metrics, Performance Indicators, and KPIs
If you spend more than a few minutes learning about Key Performance Indicators, you will surely read words like these: All KPIs are metrics, but not all metrics are KPIs BAUE04a, BECH06, ECKE05, HALL04 . Furthermore, for the adjective Key in “KPI” to have any real meaning, it must also be true that not all performance indicators are KPIs. Jonathan Becher writes: While metrics can be a measure of just about anything, KPIs are the measures that matter most BECH06. So we have defined a three level hierarchy: metrics, performance indicators, and KPIs.
An Apdex index is certainly a metric, and–based on the Apdex definition alone–it is also a performance indicator, because an Apdex score indicates the degree to which individual measurements of actual performance meet previously defined targets. This applies regardless of the data being summarized in the Apdex score. But can an Apdex index also be used as a Key Performance Indicator, or KPI? The answer depends on how you define “KPI”, a term with no single universally accepted meaning. Even so, many authors have proposed definitions, so I will evaluate Apdex against a list of proposed KPI characteristics found in my bibliography.
Table 1 below presents my analysis and conclusions. It quotes text by several authors describing KPI characteristics, assigns each distinct characteristic a short name, and (in the rightmost column) shows my conclusion as to whether an Apdex score possesses that characteristic.
When reviewing the table, remember that I am considering the potential applicability of a generalized version of the Apdex index. So my evaluation is not limited by the properties and usage of the response time metrics defined in the present Apdex spec. For some of the proposed KPI characteristics, the suitability of an Apdex index will depend on the nature of the underlying data to be reported. In such cases, I use “POTENTIALLY” rather than “YES” to show that an Apdex index is capable of possessing the desired characteristic, depending on the data being reported.
|Author||Reference||Description of KPI characteristic (See note 1)||Characteristic name(s)||Apdex property?|
|Kent Bauer||BAUE04a||KPIs are quantifiable metrics which reflect the performance of an organization in achieving its goals and objectives.||Target-based; Outcome-oriented||YES|
|Jonathan Becher||BECH06||The first—and perhaps most critical—criterion a metric must meet to be deemed a KPI is that it must be outcome-oriented. Metrics that track inputs (the amount of financial and nonfinancial resources applied to providing a service or producing a product) or outputs (the quantity of service or products produced) are just metrics. A KPI tracks outcomes that measure progress toward a defined goal so that you can understand impacts. Another way of looking at this is that a KPI is explicitly tied to an objective.||Outcome-oriented||YES|
|The second point to remember when evaluating whether a metric is truly a KPI is that KPIs are more than just numbers. In order to provide a meaningful gauge of progress toward organizational objectives, a metric must have context. Milestone-based targets provide this context.||Target-based||YES|
|Associating a rating system with KPIs provides a quick and easy-to-understand reading of whether a particular KPI status is good or bad, how on or off target it is. For example, you can use the traditional “letter grade” system, in which A, B, C, D, or F grades are assigned based on the calculated gap.||Rated or Graded||YES|
|Wayne Eckerson||ECKE05 (See note 2)||#1. KPIs Reflect Strategic Value Drivers: KPIs reflect and measure key drivers of business value. Value drivers represent activities that, when executed properly, guarantee future success. Value drivers move the organization in the right direction to achieve its stated financial and organizational goals. Examples of value drivers might be “high customer satisfaction” or “excellent product quality.”||Strategic value driver||POTENTIALLY (See note 2)|
|#4. KPIs Are Based on Corporate Standards: The only way cascading KPIs work is if an organization has established standard measurements. This is deceptively hard. It can take organizations months if not years to hash out the meaning of key measures or entities, such as “net profit” or “customer.”||Standardized||POTENTIALLY (See note 3)|
|#5. KPIs Are Based on Valid Data: Knowing what to measure and actually measuring it are two different things. Before executives finalize a KPI, they need to ask a technical analyst if the data exists to calculate the metric and whether it’s accurate enough to deliver valid results.||Valid||POTENTIALLY (See note 4)|
|#6. KPIs Must Be Easy to Comprehend: Employees must know what’s being measured, how it’s being calculated, and, more importantly, what they should do (and shouldn’t do) to positively affect the KPI.||Understandable||YES
(See note 5)
|#7. KPIs Are Always Relevant: To ensure that KPIs continually boost performance, you need to periodically audit the KPIs to determine usage and relevance. If a KPI isn’t being looked at, it should probably be discarded or rewritten.||Relevant||POTENTIALLY (See note 6)|
|#8. KPIs Provide Context: Metrics always show a number that reflects performance. But a KPI puts that performance in context. It evaluates the performance according to expectations. The context is provided using thresholds, targets, or benchmarks.||Target-based||YES|
|Curt Hall||HALL04||KPIs are metrics whose measurement functionality are designed to be applied enterprise wide–to impart standardized definitions and meanings that cut across different departmental and organizational boundaries–so that everyone in the company can operate from the same set of definitions.||Standardized||POTENTIALLY (See note 3)|
|KPIs are meant to gauge progress toward vital, strategic objectives usually defined by upper management, as opposed to the more generic metric used to measure a more mundane (i.e., less strategic) process. The goal is to foster greater visibility, better execution of strategy, faster reaction to opportunities and threats, and improved collaboration and coordination across key business operations. Suitable examples include KPIs for measuring and achieving better customer satisfaction, organizational learning, improved product quality, or overall contact center efficiency.||Target-based||YES|
|David Parmenter||PARM10||KPIs tell staff and management what to do to increase performance dramatically. For example, British Airways implemented a “late plane” KPI, which communicated immediately to everybody that there needed to be a focus on recovering the lost time. Cleaners, caterers, ground crew, flight attendants, and liaison officers with traffic controllers would all work some magic to save “a minute here and a minute there” whilst maintaining or improving service standards.||Prescriptive||NO
(See note 7)
|KPIs should be monitored and reported 24/7, daily and a few perhaps weekly. How can a KPI be measured monthly? This surely is “shutting the barn door well after the horse has truly bolted”. KPIs are “current” or “future” measures as opposed to “past” measures. When you look at most organisational measures, they are very much past indicators measuring events of the last month or quarter. These indicators cannot be and never were a KPI. A satisfaction percentage (e.g. 65%) from a customer satisfaction survey performed every six months can never be a KPI.||Current||POTENTIALLY (See note 8)|
- In each case, the quoted text is drawn from the cited publication, but my version may have been shortened for clarity. I have also added emphasis in a few places, to highlight where Apdex applies.
- Eckerson defined ten characteristics, stating: “When developing KPIs for scorecards or dashboards, you should keep in mind that KPIs possess 10 distinct characteristics. Although metrics may exhibit some of these characteristics, good KPIs possess all of them”. I omitted #2. KPIs Are Defined by “Executives”, #3. KPIs Cascade throughout an Organization, #9. KPIs Empower Users, and #10. KPIs Lead to Positive Action. To find out why, see the section below on Limitations of Apdex as a Key Performance Indicator.
- Apdex provides a standard method for computing and reporting metrics; an organization using Apdex must still standardize their definitions of the underlying measures being reported as Apdex scores.
- The Apdex scoring method can be implemented in a way that corrects for random measurement error, or for some types of systematic bias in underlying measurement data. A future post will address the subject of accuracy and precision in Apdex scores in greater detail.
- Because of its simplicity, Apdex satisfies the first part of this definition. In my view, the second part (employees knowing what to do) is a management responsibility, not a KPI characteristic. See the section below on Does a KPI Prescribe Corrective Action?.
- Apdex scores cannot be computed until appropriate Apdex thresholds (T and F) have been defined. This forces organizations using Apdex to focus on the meaning and relevance of an Apdex score. In an ideal world, organizations would be less likely to devote resources to defining Apdex thresholds for measures that are not proving to be relevant. In reality, nothing prevents organizations from producing Apdex scores for any measurement data, relevant or not.
- I find this definition by David Parmenter to be misleading at best. I do not believe that KPIs themselves need to be prescriptive. I discuss this immediately below.
- Apdex was designed to be a quality indicator for a set of individual measurements (samples in Apdex terminology). Today most Apdex implementations report on current (or the most recent) measurements of response time. Only the choice of measurements to be summarized (the report group in Apdex terminology) determines whether an Apdex index satisfies this requirement.
Does a KPI Prescribe Corrective Action?
David Parmenter PARM10 and Wayne Eckerson ECKE05, #6 believe it should. In my opinion, a corrective action that is taken when a KPI threshold triggers an alert is a property of a business rule associated with a KPI, not a characteristic of the KPI itself. This may seem like a fine distinction, but I believe it is one that helps us to think clearly about the nature of KPIs and the role of metrics like an Apdex index.
It may certainly be possible to find examples of business metrics and thresholds which–like a fuel gauge or an engine temperature warning light–signal the need for immediate and obvious corrective actions. But first, this does not mean that all KPI’s should possess that characteristic, and second, I strongly doubt that the British Airways “late plane” KPI cited by David is even illustrative of such a KPI.
Explaining how the “late plane” KPI determined staff actions, David writes: The BA airport manager at the relevant airport knew that if a plane was delayed beyond a certain “threshold”, they would receive a personal call from (the CEO). It was not long before BA planes had a reputation for leaving on time.
This explanation, while amusing, glosses over all the business rules and/or procedures that BA airport managers had to put into effect to correct the lateness problem. In actuality, I’m sure it was those calculated management actions, not the KPI itself, that produced the desired staff actions that ultimately made the difference. Someone could perhaps argue that the KPI was prescriptive because it caused management to take the necessary actions to reduce lateness. But in my view, “fix the problem” does not qualify as a prescription for action; if it did, then every metric that signals a problem is a prescriptive metric, by definition.
I conclude that, while some KPIs may embody an obvious prescription for corrective action, being prescriptive is not an essential KPI characteristic.
Using Apdex as a Key Performance Indicator
Dissecting the literature like this was quite revealing. It showed me that authors writing about KPIs find many different ways to describe just a few concepts. In fact, the three core concepts are those laid out clearly by Jonathan Becher, namely that KPI’s are Outcome-oriented, Target based, and Rated or graded. [For more details, I recommend reading Jonathan’s excellent 4-page paper, Mitigating Metrics Madness: How to Tell KPIs from Mere Metrics BECH06.]
An Apdex score, in combination with the optional Apdex rating scheme, satisfies all three core requirements of a KPI. Stepping outside the three core characteristics, KPIs exhibit a few other properties that are a consequence of the particular data being reported. These are Strategic value driver, Standardized, Valid, Understandable, Relevant, and Current. Depending on the underlying measures it summarizes, an Apdex score is capable of fulfilling every one of these additional requirements.
This is quite encouraging, considering that our mission is to generalize the definition of Apdex to make it more widely applicable. But it is not the whole story, because there are KPI characteristics that Apdex cannot address.
Limitations of Apdex as a Key Performance Indicator
Many definitions of a KPI, and many lists of KPI characteristics, involve aspects of business measurement and management that are not directly influenced by the use of Apdex as a scoring methodology. For example, following the KPI definition that I included in the first row of Table 1, Kent Bauer continues:
KPIs reflect strategic value drivers rather than just measuring non-critical business activities and processes. KPIs align all levels of an organization (business units, departments and individuals) with clearly defined and cascaded targets and benchmarks to create accountability and track progress. KPIs accelerate seamless and collaborative planning across the organization to ensure that everyone is operating from the same playbook.
I chose not to list these (or similar characteristics) in Table 1, because in practice, success or failure in achieving these admirable characteristics of effective KPIs is not going to be determined by a decision to use an Apdex scoring methodology. Yes, Apdex is simple and easy to comprehend, and (as I showed in Table 1) has other qualities that can make it suitable for widespread use. But it is not a panacea that will automatically deliver management success. Viewed within the broader context of business performance management, Apdex is merely a useful reporting tool, to be employed where appropriate.
Depending on the data being reported, an Apdex score may be used as a KPI, but I particularly want to avoid overstating any claims for its use in that role. Some business metrics important enough to be used in a KPI are not suitable for presentation as Apdex scores. And some Apdex scores may be suitable for use as performance indicators, but not be important enough to qualify as KPIs. I will discuss these subjects further in future posts.
- KPIs – The Metrics that Drive Performance Management, Kent Bauer, 2004. [Information Management Magazine, September 2004]
- Mitigating Metrics Madness: How to Tell KPIs from Mere Metrics, Jonathan Becher, 2006. [From Cutter IT Journal Vol. 19 No. 4, April 2006, pp13-16]
- Ten Characteristics of a Good KPI, Wayne Eckerson, TDWI, 2005. [The Data Warehousing Institute]
- Key Performance Indicators and Metrics, Curt Hall, Senior Consultant, Cutter Consortium, 2004. [The Cutter Edge, 23 November 2004]
- The New Thinking on KPIs, David Parmenter, 2010. [Extract from “Implementing winning KPIs”, 19 April 2010]