LinkedIn

See our LinkedIn profile, click on this button:

APDEX

Configurable Zone Alignment in Apdex-G

I’m writing a series of posts about Generalizing Apdex. This is #10.

In my post on Separately Configurable Thresholds in Apdex-G, I began work on a general description of Apdex zones and thresholds. The language I proposed would allow Apdex to be used to report on metrics for which “quality” increases or decreases monotonically with the value of the metric. Response times and VOIP MOS scores are typical monotonic metrics: smaller response times are always better, higher MOS scores are always better. Many metrics have this property. Arthur Schneiderman writes:

Properties of good metrics

The first requirement for a good metric is that it should be a reliable proxy for stake­holder satisfaction. In other words, improvement in the metric should link directly to improved stakeholder satisfac­tion. This linkage should be clear and uncomplicated. It should also be what mathematicians call monotonic—i.e., improvement in the metric should always produce improved stakeholder satisfaction …

–Arthur M. Schneiderman SCHN96

In mathematical terms, if Q(x) denotes the quality of a value x (however quality is measured), then we could describe the relationship of quality and MOS scores or response times as follows:

  • for MOS scores: Q(a)>Q(b) for all a>b (monotonically increasing quality)
  • for response times: Q(a)>Q(b) for all a<b (monotonically decreasing quality)

The corresponding Apdex zone alignments for these two cases can be written as follows:

1. Monotonic Increasing: Frustrated < Tolerating < Satisfied
2. Monotonic Decreasing: Satisfied < Tolerating < Frustrated

Non-monotonic Metrics

But as I pointed out in Which Apdex Features Can Be Generalized?, not every metric has a monotonic distribution of quality. Arthur Schneiderman again:

Truly monotonic metrics are often difficult to define. In practice, there can be too much of a good thing. For example, cycle time—a key metric of the manufacturing process—can be reduced to a point where it produces increased value to customers. Beyond that point, however, throughput declines, delivery performance suffers, and value is destroyed.

–Arthur M. Schneiderman SCHN96

Schneiderman suggests that non-monotonic metrics can often be replaced by monotonic variants:

Lead time defined as “ship date minus order date” … (is not a good metric) … because customers will generally tell their supplier when they want the product. Shipping it early to them (a shorter lead time) simply increases their inventory. It is becoming more common for customers to penalize their suppliers for early as well as late shipments. A better met­ric is excess lead time, which is defined as “supplier?quoted minus customer?requested lead time.” Improving this metric always leads to increased customer satisfaction.

Excess cycle time (i.e., actual minus optimum cycle time) transforms cycle time into a monotonic metric. Also, combined with metrics that characterize the other side of the tradeoff, cycle time can be a useful component of a system of metrics.
–Arthur M. Schneiderman SCHN96

This seems to be good advice. However, in generalizing the Apdex spec, we cannot rule out the possibility that someone will want to use Apdex to report on a non-monotonic metric. We should design Apdex-G to accommodate as many quality distributions as possible, subject only to the constraint that Apdex assigns each measurements to one of three non-overlapping performance zones: Satisfied, Tolerating, and Frustrated.

The following sections discuss examples of metrics whose zone configurations are not monotonic, but which are otherwise compatible with the Apdex approach of partitioning measures into three zones. Therefore my goal is to create an Apdex-G specification that accommodates these metrics.

Goals for Service Quality

An earlier post [Putting Apdex in Context] introduced the SERVQUAL gaps model, and the various ways of assessing service quality. The service quality domain also uses three-way classification schemes and terminology similar to Apdex:

  • Measurements of customer service quality are classified as more than acceptable, acceptable, and unacceptable.
  • Customer satisfaction levels are classified as delighted, satisfied, and dissatisfied.
  • The middle zone–below the level of the service customers desire, but above the minimum they will tolerate–is called “The Zone of Tolerance” (ZoT).

For the best introduction to these concepts, see Robert Johnston’s paper, The Zone of Tolerance JOHN95. The same concepts are also explained (with more elaborate diagrams, like the one below) in Chapter 4 of his book on Service Operations Management JOHN08.

The Zone of Tolerance (ZoT)

You might imagine that measurements falling above and within the ZoT would receive Apdex scores 1 and ½ respectively. But this is not necessarily true; different organizations adopt different stances toward the ZoT. While some aim to Delight their customers with service outcomes that are rated better than “Desired Service”, others aim only to Satisfy customers, keeping their service level scores within the ZoT. They reason that doing more would be too costly, or would raise customer expectations unduly, leading to customer dissatisfaction tomorrow.

For an organization adopting this approach, the Apdex zones for the metric Service Quality Score would be ordered:

3. Service Quality Scores: Frustrated < Satisfied < Tolerating

In other words, high service quality scores would merit only ½ an Apdex point, while middling scores would get the full point. This Apdex zone alignment may seem odd, but–even though the original scores may have originated as customer evaluations of service outcomes–the Apdex metric reflects the organization’s performance against its service goals, not customers’ evaluations of service outcomes.

For further discussion of this distinction, see the conclusion of Robert Johnston’s paper JOHN95 pp13-14 and the lively and insightful debate in the comments following two posts in the blog CustomerThink CUST04, CUST10. I especially enjoyed this contribution by Bob Thompson:

It’s difficult to sustain a strategy of always exceeding expectations and “delighting” customers. Especially if the marketing folks start promoting that the company will now be exceeding expectations.

In the tour operating company where I started my career (we) surprised our customers by putting baskets of fruit or a bottle of wine and a hand-written card into their room when they came to the tourist destination. Everybody got extremely happy, because nobody expected it and they all thought it was a kind of individual service to them. The year after, our advertising manager, who must not have been related to Einstein, wrote in the brochure, “You should know that when you travel with our company, there’s always a surprise waiting for you in the room.”

–Bob Thompson CUST10

In the approach discussed above, an organization’s service quality goals are deliberately set at levels below the best possible. Furthermore, the best possible outcomes, although tolerated, are actually less desirable to the business, and need to be measured and controlled. If, in such a business environment, we imagine a metric based on service times instead of service scores, then we can also imagine the need to support the following Apdex zone alignment:

4. Controlled Service Times: Tolerating < Satisfied < Frustrated

Avoiding a Range of Values

Having made the case for supporting four zone alignments, only two further possibilities remain. These would be needed to report on metrics for which the goal was to avoid a particular range of values, with a preference for higher (or lower) values:

5. Avoid, Preferably Above: Tolerating < Frustrated < Satisfied
6. Avoid, Preferably Below: Satisfied < Frustrated < Tolerating

But do such metrics actually exist? I was having trouble coming up with examples until I turned on the TV and watched some of the British Open golf tournament. Then I realized that drive length would be such a metric, whenever there is a hazard to be avoided. Sometimes it’s preferable to drive the ball beyond the hazard (Tolerating < Satisfied), sometimes it’s safer to lay up (Satisfied < Tolerating). But in either case, a golfer whose drives land in the hazard will be Frustrated.

I can’t picture anyone (a golf statistician? a swing coach? a TV analyst?) actually using Apdex to evaluate golfers. But maybe practitioners in other measurement domains deal with similar metrics, and would find the Apdex method useful. If you know of any, post a comment below. In any case, I believe that Apdex-G should support all six possible zone alignments enumerated above.

Process Control Applications

Generalizing further, Apdex-G could accommodate zones that are not continuous. In Statistical Process Control applications, closeness to a target value is desirable, and the degree of satisfaction with a measurement decreases with its distance above or below the target. For example, Jim Smith SMIT09 describes a pre-control chart that partitions measurements into three zones, as illustrated by this example:

A Pre-Control Chart

The chart legend is singularly lacking in information content. A better one might note that measurements in the central green zone (±50% of the specification tolerance) signal management to continue running the process, measurements in two yellow zones (each 25%) signal caution, and measurements falling outside the yellow zones, in two red zones, signal stop the process. The “traffic-light” colors are probably sufficient for users familiar with the purpose of the chart.

Before applying Apdex to measurements of this type, some organizations might take the advice of Arthur Schneiderman and convert the data into a monotonic variant, combining the zones on either side of the target by computing |actual−target|, the absolute value of actual minus target. But if that approach is not feasible, or not adopted for any reason, Apdex-G should accept discontinuous zone definitions.

Conclusions

Creating an Apdex-G specification to support all six possible alignments of three continuous zones, and also to accept discontinuous zone definitions, introduces two complications. Both concern the current spec’s use of just two thresholds, T and F, to define the three Apdex zones. We must reconsider:

  1. The relationship between thresholds and zones
  2. The number of thresholds required

In my next post [Generalizing the Apdex Thresholds] I will discuss these issues, suggest a solution, and draft the corresponding language for section [2.5] of Apdex-G.

References

CUST04
What Does It Take To Delight Customers? Christopher Carfi, 2004. [CustomerThink, June 2004]
CUST10
About Them Customers’ Expectations, Esteban Kolsky, 2010. [CustomerThink, Feb 2010]
JOHN95
The Zone of Tolerance, Robert Johnston, 1995. [48Kb pdf ]
JOHN08
Service Operations Management (3rd Edition), Robert Johnston and Graham Clark, Prentice Hall, 2008. [Amazon Books]
SCHN96
Metrics for the Order Fulfillment Process (Part I), Arthur M. Schneiderman, 1996. [Journal of Cost Management Vol.10 No.2 Summer 1996, pp30-42]
SMIT09
Pre-Control May be the Solution, Jim L. Smith, 2009. [Quality Magazine, Sept 2009]

1 comment to Configurable Zone Alignment in Apdex-G

  • […] In my previous post in this series, I discussed non-monotonic metrics. These are metrics for which improvement in the metric does not always always correspond to an improvement in quality. To accommodate such metrics, the Apdex-G specification must support six distinct orderings of the three Apdex performance zones (Satisfied, Tolerating, and Frustrated). It should also accept discontinuous zone definitions. For the full story, see Configurable Zone Alignment in Apdex-G, […]

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  


*