I’m writing a series of posts about Generalizing Apdex. This is #10.
In my post on Separately Configurable Thresholds in Apdex-G, I began work on a general description of Apdex zones and thresholds. The language I proposed would allow Apdex to be used to report on metrics for which “quality” increases or decreases monotonically with the value of the metric. Response times and VOIP MOS scores are typical monotonic metrics: smaller response times are always better, higher MOS scores are always better. Many metrics have this property. Arthur Schneiderman writes:
The first requirement for a good metric is that it should be a reliable proxy for stakeholder satisfaction. In other words, improvement in the metric should link directly to improved stakeholder satisfaction. This linkage should be clear and uncomplicated. It should also be what mathematicians call monotonic—i.e., improvement in the metric should always produce improved stakeholder satisfaction …
–Arthur M. Schneiderman SCHN96
In mathematical terms, if Q(x) denotes the quality of a value x (however quality is measured), then we could describe the relationship of quality and MOS scores or response times as follows:
- for MOS scores: Q(a)>Q(b) for all a>b (monotonically increasing quality)
- for response times: Q(a)>Q(b) for all a<b (monotonically decreasing quality)
The corresponding Apdex zone alignments for these two cases can be written as follows:
2. Monotonic Decreasing: Satisfied < Tolerating < Frustrated
But as I pointed out in Which Apdex Features Can Be Generalized?, not every metric has a monotonic distribution of quality. Arthur Schneiderman again:
–Arthur M. Schneiderman SCHN96
Schneiderman suggests that non-monotonic metrics can often be replaced by monotonic variants:
Excess cycle time (i.e., actual minus optimum cycle time) transforms cycle time into a monotonic metric. Also, combined with metrics that characterize the other side of the tradeoff, cycle time can be a useful component of a system of metrics.
–Arthur M. Schneiderman SCHN96
This seems to be good advice. However, in generalizing the Apdex spec, we cannot rule out the possibility that someone will want to use Apdex to report on a non-monotonic metric. We should design Apdex-G to accommodate as many quality distributions as possible, subject only to the constraint that Apdex assigns each measurements to one of three non-overlapping performance zones: Satisfied, Tolerating, and Frustrated.
The following sections discuss examples of metrics whose zone configurations are not monotonic, but which are otherwise compatible with the Apdex approach of partitioning measures into three zones. Therefore my goal is to create an Apdex-G specification that accommodates these metrics.
Goals for Service Quality
An earlier post [Putting Apdex in Context] introduced the SERVQUAL gaps model, and the various ways of assessing service quality. The service quality domain also uses three-way classification schemes and terminology similar to Apdex:
- Measurements of customer service quality are classified as more than acceptable, acceptable, and unacceptable.
- Customer satisfaction levels are classified as delighted, satisfied, and dissatisfied.
- The middle zone–below the level of the service customers desire, but above the minimum they will tolerate–is called “The Zone of Tolerance” (ZoT).
For the best introduction to these concepts, see Robert Johnston’s paper, The Zone of Tolerance JOHN95. The same concepts are also explained (with more elaborate diagrams, like the one below) in Chapter 4 of his book on Service Operations Management JOHN08.
You might imagine that measurements falling above and within the ZoT would receive Apdex scores 1 and ½ respectively. But this is not necessarily true; different organizations adopt different stances toward the ZoT. While some aim to Delight their customers with service outcomes that are rated better than “Desired Service”, others aim only to Satisfy customers, keeping their service level scores within the ZoT. They reason that doing more would be too costly, or would raise customer expectations unduly, leading to customer dissatisfaction tomorrow.
For an organization adopting this approach, the Apdex zones for the metric Service Quality Score would be ordered:
In other words, high service quality scores would merit only ½ an Apdex point, while middling scores would get the full point. This Apdex zone alignment may seem odd, but–even though the original scores may have originated as customer evaluations of service outcomes–the Apdex metric reflects the organization’s performance against its service goals, not customers’ evaluations of service outcomes.
For further discussion of this distinction, see the conclusion of Robert Johnston’s paper JOHN95 pp13-14 and the lively and insightful debate in the comments following two posts in the blog CustomerThink CUST04, CUST10. I especially enjoyed this contribution by Bob Thompson:
In the tour operating company where I started my career (we) surprised our customers by putting baskets of fruit or a bottle of wine and a hand-written card into their room when they came to the tourist destination. Everybody got extremely happy, because nobody expected it and they all thought it was a kind of individual service to them. The year after, our advertising manager, who must not have been related to Einstein, wrote in the brochure, “You should know that when you travel with our company, there’s always a surprise waiting for you in the room.”
–Bob Thompson CUST10
In the approach discussed above, an organization’s service quality goals are deliberately set at levels below the best possible. Furthermore, the best possible outcomes, although tolerated, are actually less desirable to the business, and need to be measured and controlled. If, in such a business environment, we imagine a metric based on service times instead of service scores, then we can also imagine the need to support the following Apdex zone alignment:
Avoiding a Range of Values
Having made the case for supporting four zone alignments, only two further possibilities remain. These would be needed to report on metrics for which the goal was to avoid a particular range of values, with a preference for higher (or lower) values:
6. Avoid, Preferably Below: Satisfied < Frustrated < Tolerating
But do such metrics actually exist? I was having trouble coming up with examples until I turned on the TV and watched some of the British Open golf tournament. Then I realized that drive length would be such a metric, whenever there is a hazard to be avoided. Sometimes it’s preferable to drive the ball beyond the hazard (Tolerating < Satisfied), sometimes it’s safer to lay up (Satisfied < Tolerating). But in either case, a golfer whose drives land in the hazard will be Frustrated.
I can’t picture anyone (a golf statistician? a swing coach? a TV analyst?) actually using Apdex to evaluate golfers. But maybe practitioners in other measurement domains deal with similar metrics, and would find the Apdex method useful. If you know of any, post a comment below. In any case, I believe that Apdex-G should support all six possible zone alignments enumerated above.
Process Control Applications
Generalizing further, Apdex-G could accommodate zones that are not continuous. In Statistical Process Control applications, closeness to a target value is desirable, and the degree of satisfaction with a measurement decreases with its distance above or below the target. For example, Jim Smith SMIT09 describes a pre-control chart that partitions measurements into three zones, as illustrated by this example:
The chart legend is singularly lacking in information content. A better one might note that measurements in the central green zone (±50% of the specification tolerance) signal management to continue running the process, measurements in two yellow zones (each 25%) signal caution, and measurements falling outside the yellow zones, in two red zones, signal stop the process. The “traffic-light” colors are probably sufficient for users familiar with the purpose of the chart.
Before applying Apdex to measurements of this type, some organizations might take the advice of Arthur Schneiderman and convert the data into a monotonic variant, combining the zones on either side of the target by computing |actual−target|, the absolute value of actual minus target. But if that approach is not feasible, or not adopted for any reason, Apdex-G should accept discontinuous zone definitions.
Creating an Apdex-G specification to support all six possible alignments of three continuous zones, and also to accept discontinuous zone definitions, introduces two complications. Both concern the current spec’s use of just two thresholds, T and F, to define the three Apdex zones. We must reconsider:
- The relationship between thresholds and zones
- The number of thresholds required
In my next post [Generalizing the Apdex Thresholds] I will discuss these issues, suggest a solution, and draft the corresponding language for section [2.5] of Apdex-G.
- What Does It Take To Delight Customers? Christopher Carfi, 2004. [CustomerThink, June 2004]
- About Them Customers’ Expectations, Esteban Kolsky, 2010. [CustomerThink, Feb 2010]
- The Zone of Tolerance, Robert Johnston, 1995. [48Kb pdf ]
- Service Operations Management (3rd Edition), Robert Johnston and Graham Clark, Prentice Hall, 2008. [Amazon Books]
- Metrics for the Order Fulfillment Process (Part I), Arthur M. Schneiderman, 1996. [Journal of Cost Management Vol.10 No.2 Summer 1996, pp30-42]
- Pre-Control May be the Solution, Jim L. Smith, 2009. [Quality Magazine, Sept 2009]