LinkedIn

See our LinkedIn profile, click on this button:

APDEX

Generalizing the Apdex Thresholds

I’m writing a series of posts about Generalizing Apdex. This is #11. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

In my previous post in this series, I discussed non-monotonic metrics. These are metrics for which improvement in the metric does not always always correspond to an improvement in quality. To accommodate such metrics, the Apdex-G specification must support six distinct orderings of the three Apdex performance zones (Satisfied, Tolerating, and Frustrated). It should also accept discontinuous zone definitions. For the full story, see Configurable Zone Alignment in Apdex-G,

These requirements introduce two complications, both involving the current spec’s use of just two thresholds, T and F, to define the three Apdex zones. In this post, I will review:

  1. The relationship between thresholds and zones
  2. The number of thresholds required

The Relationship Between Thresholds and Zones

To explain the first problem, the Table 1 below lists the six possible orderings of three continuous zones, grouping them according to whether low or high data values are preferred. Reading from left to right, each row shows the alignment of the three Apdex performance zones, low to high, within the measurement domain. It also shows where the thresholds T and F would be placed in Apdex-G, to delimit the zones.

Preference [1] Number [2] Metric Type [3] Low Zone [5] Low Threshold [6] [7] [8] Middle Zone [5] High Threshold [6] [7] [8] High Zone [5]
High Values Preferred 1 Monotonic Increasing Frustrated [F] Tolerating [T] Satisfied
3 Service Quality Scores Frustrated [F] Satisfied [T] Tolerating
5 Avoid, Preferably Above Tolerating [T] Frustrated [F] Satisfied
Low Values Preferred 2 Monotonic Decreasing [4] Satisfied [T] Tolerating [F] Frustrated
4 Controlled Service Times Tolerating [T] Satisfied [F] Frustrated
6 Avoid, Preferably Below Satisfied [F] Frustrated [T] Tolerating
Table 1. Six possible alignments of continuous zones, using [T] and [F] as thresholds

Explanatory Notes

  1. When the Satisfied Zone is above (to the right of) the Frustrated Zone, higher values are preferred, and vice versa.
  2. These numbers refer to the previous post. They were used when describing the different types of metrics.
  3. These types were identified in the previous post.
  4. The Monotonic Decreasing metric type corresponds to the present Apdex spec.
  5. For each metric type, Low Zone < Middle Zone < High Zone
  6. I use the notation [T] and [F] to denote the Apdex-G versions of the two thresholds.
  7. In each row I placed the thresholds so that they are always adjacent to their “corresponding” zones. [T] is always adjacent to the Tolerating Zone; [F] is always adjacent to the Frustrated Zone. Indeed, in each row there is only one sensible way to position a pair of thresholds having those labels; reversing their positions always separates one of the thresholds from its corresponding zone.
  8. In the current Apdex spec, thresholds are defined to fall within the lower of two adjacent zones. For example, T is the upper bound of the Satisfied Zone, and the Tolerating Zone is defined as “Greater than T to F”. There is an argument for reversing this convention, so that a threshold falls within the higher of the two zones. Then T would fall within the Tolerating Zone, which (to me) seems more natural. However, whether or not we retain or change that convention has no bearing on the placement of thresholds in this table.

Problems with Using [T] and [F]

In the current Apdex spec, T and F have natural meanings: above T begins the Tolerating Zone, above F begins the Frustrated Zone. But when we shuffle the zones, and continue to use T and F as delimiters, we can’t hold onto these natural meanings. Even though I have assigned the roles of [T] and [F] in the only possible way in each row, the result still requires their meanings to vary from row to row. Sometimes they mark the preferred end of their corresponding zone, sometimes the opposite.

As a result, if we adopt the approach shown in Table 1, there will be no simple and natural way to describe the roles of [T] and [F] in Apdex-G. The only ways would be either to reproduce Table 1, or my explanation of why [T] and [F] occupy those positions in Table 1, or both. Since our aim is to keep Apdex-G as simple as possible, these are not attractive propositions.

An alternative approach would be to use thresholds called [S], [T], and [F], picking two as appropriate for each metric type, so as to retain the natural relationship between threshold names and roles that exists in the current spec. Table 2 illustrates how this idea might be applied:

Preference Number Metric Type Low Zone Low Threshold Middle Zone High Threshold High Zone
High Values Preferred 1 Monotonic Increasing Frustrated [T] Tolerating [S] Satisfied
3 Service Quality Scores Frustrated [S] Satisfied [T] Tolerating
5 Avoid, Preferably Above Tolerating [F] Frustrated [S] Satisfied
Low Values Preferred 2 Monotonic Decreasing Satisfied [T] Tolerating [F] Frustrated
4 Controlled Service Times Tolerating [S] Satisfied [F] Frustrated
6 Avoid, Preferably Below Satisfied [F] Frustrated [T] Tolerating
Table 2. Six possible alignments of continuous zones, using [S], [T], or [F] as thresholds.

This approach has the merit of removing any confusion over threshold roles: above [S] is the Satisfied Zone, above [T] is the Tolerating Zone, above [F] is the Frustrated Zone. That much is natural and easy to specify. However, we have now introduced a complication into the Apdex-G spec: which pair of thresholds is appropriate for any given metric type? To answer that, we would still need to include a version of Table 2 in the spec. And if we have to do that, does this solution even qualify as a “generalization” of Apdex?

Generic Threshold Names

I therefore conclude that it would be better not to introduce the names [S], [T], and [F] when defining the purpose of thresholds in Apdex-G. Instead we should use generic threshold names that contain no implied references to the names of the Apdex performance zones.

An addendum (like Apdex-R, for response time metrics) can still specify domain-specific threshold names (like [T] and [F]) to replace the generic ones. But in Apdex-G, we would use names like [LT] and [HT], signifying the Low and High Threshold respectively, or completely neutral threshold labels like [T1] and [T2]. This last approach seems to work best when describing discontinuous zones, which I discuss next. But first, here’s Table 3, showing this conclusion:

Preference Number Metric Type Low Zone Low Threshold Middle Zone High Threshold High Zone
High Values Preferred 1 Monotonic Increasing Frustrated [T1] Tolerating [T2] Satisfied
3 Service Quality Scores Frustrated [T1] Satisfied [T2] Tolerating
5 Avoid, Preferably Above Tolerating [T1] Frustrated [T2] Satisfied
Low Values Preferred 2 Monotonic Decreasing Satisfied [T1] Tolerating [T2] Frustrated
4 Controlled Service Times Tolerating [T1] Satisfied [T2] Frustrated
6 Avoid, Preferably Below Satisfied [T1] Frustrated [T2] Tolerating
Table 3. Six possible alignments of continuous zones, using [T1] and [T2] as thresholds.

The Number of Thresholds

As I discussed in my previous post, to support typical process control applications, Apdex-G must accommodate bifurcated performance zones. The classic control diagram is symmetric, with both the Yellow/Tolerating and Red/Frustrated Zones being split equally on either side of the central Green/Satisfied Zone.

We can also imagine cases in which measurements of a process to be reported by Apdex are strongly skewed, so that the Frustrated Zone occurs on only one side of the target value. Both positive and negative skews are common in real-world measurement data, when the process or application being measured has a natural lower or upper bound. For example:

  • timings are often postively skewed, because times cannot be less than zero, but may have no upper bound
  • bandwidth data may be negatively skewed, because network technology imposes an upper limit, but no lower limit

With those possibilities in mind, Table 4 shows the three types of process control metrics. As in Tables 1-3, reading from left to right, each row shows the alignment of the Apdex performance zones, low to high, within the measurement domain. The only difference is that the Frustrated and Tolerating Zones are now split into two regions, one above and one below the Satisfied Zone. And in the case of the Frustrated Zone, one of those regions may be absent.

No. [1] Metric Type [2] Low Red Zone [3] Thr. 1 [4] Low Yellow Zone Thr. 2 Green Zone Thr. 3 High Yellow Zone Thr. 4 High Red Zone [5]
7 Symmetric Frustrated [T1] Tolerating [T2] Satisfied [T3] Tolerating [T4] Frustrated
8 Negative Skew Frustrated [T1] Tolerating [T2] Satisfied [T3] Tolerating
9 Positive Skew Tolerating [T2] Satisfied [T3] Tolerating [T4] Frustrated
Table 4. Three possible alignments of bifurcated zones for process control applications

Explanatory Notes

  1. To enumerate the different types of metrics and zone alignments, these numbers continue the sequence begun in Table 3.
  2. These names reflect the likely shape of the data distribution for each metric type.
  3. The names and colors reflect the example.
  4. I use the abbreviation “Thr” for “Threshold” in this table.
  5. For each metric type, Low Red Zone < Low Yellow Zone < Green Zone < High Yellow Zone < High Red Zone

Introducing Performance Intervals

There are basically two ways to get from my earlier draft of section [2.5] of Apdex-G to a version that accommodates the nine zone alignments listed in Tables 3 and 4. Either we have to add detail, enumerating the nine metric types, or we have to be less specific about the exact composition of the three performance zones, the number of thresholds, and their names. I much prefer the second approach, because any generic rule specified in Apdex-G can be further refined and focused by a domain-specific Addendum like Apdex-R.

I believe we can retain the essential feature of Apdex–three distinct performance zones–while permitting more flexibility in their alignment. As illustrated in the diagrams below, we can do that by introducing an additional concept: a performance interval, which is defined to be a range of values within the measurement domain that share a common satisfaction or quality level (i.e. ‘Satisfied’, ‘Tolerating’, or ‘Frustrated’). Thresholds now define the boundaries of the performance intervals, not the performance zones, which are now simply a collection (union) of one or more intervals. The Apdex-G diagram illustrates Metric Type #7, Symmetric Process Control measurements.

Conceptual Models of Apdex and Apdex-G

This is not an entirely new concept. First, the proposed performance intervals can be defined as mathematical intervals, which will be convenient when we work on section [5] Reporting. Second, the current performance zones are already identical to performance intervals, so that the current model is just a trivial case of the general model. But by allowing subdivision of the measurement domain into an arbitrary number of intervals, then collecting those intervals into three performance zones, we permit Apdex-G to support all the different metric types we discussed above–and more if the need arises.

Drafting New Definitions

Below I propose language to describe these concepts. The left hand column shows how the current Apdex spec describes performance zones and thresholds; draft descriptions of the corresponding concepts in Apdex-G appear on the right. My goal in this draft is to define the new rules correctly, so that they are both precise and general. And since the concepts to be described in section [2.5.1] must also be spelled out systematically in section [7], the Glossary, I have chosen to work out the details using the Glossary format. When those definitions are finalized, I will edit them into a narrative format to create section [2.5.1].

Current spec:
§2.5 How Users Interpret Application Response Times.

Users have a finite set of reactions or views by which they characterize application response time. Each such group of time durations is called a performance zone. Performance zones are defined by two thresholds – times where the zone begins and ends. Apdex defines three such performance zones:

Satisfied
Response times that are fast enough to satisfy the user, who is therefore able to concentrate fully on the work at hand, with minimal negative impact on his/her thought process.
Tolerating
Responses in the tolerating zone are longer than those of the satisfied zone, exceeding the threshold at which the user notices how long it takes to interact with the system, and potentially impairing the user’s productivity. Application responses in this zone are less than ideal but don’t by themselves threaten the usability of the application.
Frustrated
As response times increase, at some threshold the user becomes unhappy with slow performance entering the frustrated zone. With response times in the frustrated zone, a casual user is likely to abandon a course of action and a production user is likely to cancel a Task.

My first draft, slightly modified:
[2.5] Performance Zones

Apdex assigns each measurement to a performance zone based on its value, where a performance zone is a range of values. Apdex defines three non-overlapping performance zones:

Satisfied Zone
This zone reflects the desired level of performance for the application or process being reported. Measurements in the satisfied zone usually indicate that the application or process being measured is meeting its targets and needs little or no attention.
Tolerating Zone
This zone reflects a level of performance below the desired level, but which can be tolerated. Measurements in the tolerating zone may signal the need for caution or greater attention to the application or process being measured.
Frustrated Zone
This zone reflects an unsatisfactory level of performance for the application or process being reported. Measurements in the frustrated zone may be associated with undesirable outcomes, and lead to remedial or abnormal actions. … currently this is followed by some examples of abnormal actions …

Current spec (heading added):
Thresholds
The three zones are defined by two thresholds: T and F, in seconds, as follows.

  • Satisfied Zone = zero to T.
  • Tolerating Zone = Greater than T to F.
  • Frustrated Zone = Greater than F

The value of F is four times the value T. For example, if users perceive response time as tolerable beginning at 4 seconds then they will be frustrated at greater than 16 seconds. The research that supports this model is described in Reference 1. Further background information is available in References 2-4.

First draft (to be replaced):
[2.5.1] Thresholds

The three performance zones are defined by two threshold values: the tolerating threshold (T) and the frustrated threshold (F). In a typical zone alignment, the tolerating zone lies between the satisfied zone and the frustrated zone, and:

  • The value T is contained in the satisfied zone, and is the boundary between the satisfied zone and the tolerating zone.
  • The value F is contained in the tolerating zone, and is the boundary between the tolerating zone and the frustrated zone.

Current spec:
§7. Glossary

Performance Zone
A range of time (between two time values) that a user waits for an application to respond during which his/her perception of the application’s responsiveness does not change.
Satisfied Zone
The range of application response times between 0 and T in which the user is not affected by the response time.
Tolerating Zone
The range of application response times between T and F in which the user is negatively affected by response time.
Frustrated Zone
Any application response time above F in which the user is very negatively affected by response time.
Threshold
The response time value that defines a boundary between performance zones.

First draft of proposed new terminology:
[7] Glossary

Addendum
A supplemental Apdex specification containing rules that apply only to applications of the Apdex method within a specific domain.
Addendum Name
An addendum is named Apdex-X, where X is a suffix appropriate to a particular domain. Examples might be Apdex-R for response time, Apdex-V for VOIP quality, and so on.
Tool
The measurement and reporting system (devices, software, etc.) that generates Apdex values.
Measurement Domain
The set of all possible values from which the input to an Apdex calculation is drawn. Unless limited by a domain-specific Addendum, this will be the real number domain (-∞, ∞)
Satisfaction Level
One of three categories (Satisfied, Tolerating, or Frustrated) that reflects how an organization using Apdex evaluates the performance of the application or process being reported. For the Apdex method to be applicable to a Measurement Domain, it must be possible to assign a Satisfaction Level to any value within the Measurement Domain.
Quality Level
An alternative term for Satisfaction Level.
Satisfied
A Satisfaction Level that reflects the desired level of performance for the application or process being reported. Measurements classified as Satisfied usually indicate that the application or process being measured is meeting its targets and needs little or no attention.
Tolerating
A Satisfaction Level that reflects a level of performance below the desired level, but which can be tolerated. Measurements classified as Tolerating may signal the need for caution or greater attention to the application or process being measured.
Frustrated
A Satisfaction Level that reflects an unsatisfactory level of performance for the application or process being reported. Measurements classified as Frustrated may be associated with undesirable outcomes, and lead to remedial or abnormal actions.
Performance Interval
A partition of the Measurement Domain defined by Thresholds. All measurement values within a Performance Interval have the same Satisfaction Level, which is therefore considered to be an attribute of the Performance Interval. There must be at least three Performance Intervals, one for each of the three Satisfaction Levels. When the context is unambiguous, the term ‘Performance Interval’ may be shortened to Interval.
Performance Interval Name
Performance Intervals have user-assigned names. Tools may assign default generic names (such as PI1, PI2, … or PI1, PI2, …) or names signifying the Satisfaction Level of the Interval (such as PIS, PIT1, PIT2, PIF1, … or PIs, PIt1, PIt2, PIf1 …). Default names are assigned to Intervals based on their order, from low to high. An Addendum may use domain-specific names and orderings.
Threshold
One of N distinct values within the Measurement Domain, which partition the Measurement Domain into N+1 contiguous non-empty Performance Intervals in such a way that measurement values within a Performance Interval have identical Satisfaction Levels. Each Threshold lies within one and only one Performance Interval. Unless otherwise specified by a domain-specific Addendum, a Threshold lies within the Performance Interval for which it acts as the upper bound.
Threshold Name
Threshold have user-assigned names. Tools may assign default generic names (T1, T2, … or T1, T2, …). Default names are assigned to thresholds based on their order, from low to high. An Addendum may use domain-specific names and orderings.
Satisfied Zone
The union of all Performance Intervals that have the Satisfaction Level of ‘Satisfied’.
Tolerating Zone
The union of all Performance Intervals that have the Satisfaction Level of ‘Tolerating’.
Frustrated Zone
The union of all Performance Intervals that have the Satisfaction Level of ‘Frustrated’.
Performance Zone
The Satisfied Zone, the Tolerating Zone, or the Frustrated Zone.

Next Steps …

The definitions above may be clumsy in places. Nevertheless, I will add them to the Extensible Apdex Glossary once this is posted. I expect that refinements will emerge as the complete document takes shape. Note also that the list of proposed Apdex-G definitions above appears in an order that eliminates most forward references, to maximize its readability. In the Glossary, of course, terms will be listed be in alphabetical order.

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

3 comments to Generalizing the Apdex Thresholds

  • Chris Loosley

    I edited the definitions of ‘Performance Interval Name’ and ‘Threshold Name’, simplifying the wording and referring to ‘Tools’, the definition of which I also included (unchanged) from the current spec, for reference.

  • […] In my previous post in this series, I introduced a conceptual model for Apdex-G. Compared to the current Apdex model, the key addition is the Performance Interval, a range of values within the measurement domain that shares a common Satisfaction Level of ‘Satisfied’, ‘Tolerating’, or ‘Frustrated’. Thresholds now form the boundaries of performance intervals, rather than Performance Zones–which are now defined as a collection (union) of one or more performance intervals. For the full story, see Generalizing the Apdex Thresholds. […]

  • […] In my previous posts in this series, I have stated that the notion of classifying all measurements into one of three performance zones is a core feature of Apdex that should be retained, because three-category classification schemes are common to many measurement and reporting domains. I have also made the case for allowing more thresholds, and thereby defining each performance zone as the union of one or more distinct performance intervals. For the conclusion of that discussion, see Generalizing the Apdex Thresholds. […]

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  


*