LinkedIn

See our LinkedIn profile, click on this button:

APDEX

Separately Configurable Thresholds in Apdex-G

I’m writing a series of posts about Generalizing Apdex. This is #8. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

To compute an Apdex score for a set of response-time measurements, you count how many measurements fall into each of three performance zones. Apdex calls these zones Satisfied, Tolerating, and Dissatisfied but, as Neil Gunther observed, the labels … are unimportant. You could just as easily call them Good, Bad and Ugly GUNT09.

Indeed, while investigating other potential applications of Apdex in other measurement domains, I found many other examples of measured outcomes being classified using three categories. Of course, many other labels are being used for those categories. For example:

  • Business triage involves categorizing desired outcomes/goals and the processes that support those outcomes/goals based on their relative importance to achieving a stated, measurable goal or outcome. Using the same triage categories employed by … medical services, business processes are categorized as essential/critical (red), important/urgent (yellow), or optional/supportive (green).
  • Most BPM dashboards employ red, yellow, and green visualizations to indicate how well performance metrics are conforming to defined targets or thresholds, with green signaling satisfactory, red signaling unsatisfactory, and yellow signaling an “in-between” state requiring some caution or extra attention.
  • Measurements of customer service quality, and customer satisfaction are often reported using three categories: service quality can be more than acceptable, acceptable, and unacceptable; satisfaction levels are delighted, satisfied, and dissatisfied JOHN08 Chapter 4. The middle zone–below the level of the service customers desire, but above the minimum they will tolerate–is called The Zone of Tolerance JOHN95.
  • In manufacturing, a variant of Statistical Process Control called a pre-control chart partitions measurements into three groups. Measurements in the central green zone (±50% of the specification tolerance) signal management to continue running the process, measurements in two yellow zones (each 25%) signal caution, and measurements falling outside the yellow zones, in two red zones, signal stop the process.

For some monitoring tasks, it may be more effective to assign measurements to five categories, or more. Nonetheless, there is no disputing the popularity of three-category (good/neutral/bad) partitioning schemes, or the widespread appeal of traffic light colors (green/yellow/red) as a universally recognizable way to visualize those schemes. So, since Apdex already employs a three-category partitioning scheme, our aim when generalizing the Apdex spec is to accommodate as many other instances as possible of the three-category approach, in other measurement domains.

But, you may ask, could Apdex ever be used for (say) process control? That is an interesting question, and one that we should probably investigate sometime. But in the context of generalizing the spec, we do not need to concern ourselves with the question of whether an Apdex metric would be useful in a particular measurement domain; all we need to do is to make sure the Apdex specification is applicable. Then a practitioner working in that domain can choose to use Apdex if it adds value.

Thresholds

Today, the three Apdex performance zones are defined by two thresholds, T and F, where T < F (faster response times are better) and F = 4 x T. The left-hand column below shows Section §2.5 of the current spec, where these relationships are specified in the context of application response times. Apdex-G must remove these constraints and neutralize the language. It must permit both T and F to be specified separately, and make no assumptions about which is the larger value. In some measurement domains, larger values are more desirable. The right-hand column below suggests possible replacement text. Consider this text to be a first draft of Apdex-G Section [2.5].

2.5 How Users Interpret Application Response Times.

Users have a finite set of reactions or views by which they characterize application response time. Each such group of time durations is called a performance zone. Performance zones are defined by two thresholds – times where the zone begins and ends. Apdex defines three such performance zones:

Satisfied
Response times that are fast enough to satisfy the user, who is therefore able to concentrate fully on the work at hand, with minimal negative impact on his/her thought process.
Tolerating
Responses in the tolerating zone are longer than those of the satisfied zone, exceeding the threshold at which the user notices how long it takes to interact with the system, and potentially impairing the user’s productivity. Application responses in this zone are less than ideal but don’t by themselves threaten the usability of the application.
Frustrated
As response times increase, at some threshold the user becomes unhappy with slow performance entering the frustrated zone. With response times in the frustrated zone, a casual user is likely to abandon a course of action and a production user is likely to cancel a Task.

The three zones are defined by two thresholds: T and F, in seconds, as follows.

  • Satisfied Zone = zero to T.
  • Tolerating Zone = Greater than T to F.
  • Frustrated Zone = Greater than F

The value of F is four times the value T. For example, if users perceive response time as tolerable beginning at 4 seconds then they will be frustrated at greater than 16 seconds. The research that supports this model is described in Reference 1. Further background information is available in References 2-4.

[2.5] Performance Zones

Apdex assigns each measurement to a performance zone based on its value, where a performance zone is a range of values. Apdex defines three non-overlapping performance zones:

Satisfied
This zone reflects the desired level of performance for the application or process being reported. Measurements in the satisfied zone usually indicate that the application or process being measured is meeting its targets and needs little or no attention.
Tolerating
This zone reflects a level of performance below the desired level, but which can be tolerated. Measurements in the tolerating zone may signal the need for caution or greater attention to the application or process being measured.
Frustrated
This zone reflects an unsatisfactory level of performance for the application or process being reported. Measurements in the frustrated zone may be associated with undesirable outcomes, and lead to remedial or abnormal actions. For example, a customer may abandon an unsatisfactory transaction, an executive may assign more resources to a critical business process that is not meeting its assigned targets, or a supervisor may halt a manufacturing process whose product is outside the specified tolerance.

[2.5.1] Thresholds

The three performance zones are defined by two threshold values: the tolerating threshold (T) and the frustrated threshold (F). In a typical zone alignment, the tolerating zone lies between the satisfied zone and the frustrated zone, and:

  • The value T is contained in the satisfied zone, and is the boundary between the satisfied zone and the tolerating zone.
  • The value F is contained in the tolerating zone, and is the boundary between the tolerating zone and the frustrated zone.

[2.5.2] Zone Alignments

Notes on Apdex-G

  • It seems best to make the section title more specific, since there are so many possible applications.
  • The rule that the value of F is four times the value T is a convenient simplification that works well for response-time measures. But it does not apply outside the application responsiveness domain, and must be moved to Apdex-R.
  • These threshold definitions apply when the performance zones have the default ordering, namely Satisfied|Tolerating|Frustrated. This is the simplest case, which corresponds to Apdex as it is defined today. So I wanted to deal with generalizing it before introducing the complications of other zone alignments.
  • Possible alternative zone alignments will be discussed in a future post. Some simple diagrams may be helpful when discussing zone alignments in Section [2.5.2]. I will see what I can come up with.
  • It may also be useful to introduce mathematical notation to describe relationships among the thresholds and zones. I will also discuss this when drafting language for Section [2.5.2].
  • The Apdex-G spec for reporting T and F will be dealt with later, when we discuss generalizing Section §5.
  • I decided not to capitalize terms of art, to reduce visual clutter and improve readability.

As usual, these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

References

GUNT09
The Apdex Index Revealed, Neil J. Gunther, February 2009. [CMG MeasureIT]
JOHN95
The Zone of Tolerance, Robert Johnston, 1995. [48Kb pdf]
JOHN08
Service Operations Management (3rd Edition), Robert Johnston and Graham Clark, Prentice Hall, 2008. [Amazon Books]

2 comments to Separately Configurable Thresholds in Apdex-G

  • Chris Loosley

    I revised the section titles, changing [2.5] from ‘Apdex Performance Zones’ to ‘Performance Zones’, and [2.5.2] from ‘Alternative Zone Alignments’ to ‘Zone Alignments’. In both cases, the adjectives were superfluous. Everything in the spec is about Apdex, and section [2.5.2] should discuss all possible zone alignments.

  • […] I have begun work on the details (see Generalizing the Apdex Language and Separately Configurable Thresholds in Apdex-G), a mental framework for the structure and content of each document has begun to emerge. This post […]

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  


*