LinkedIn

See our LinkedIn profile, click on this button:

APDEX

Apdex-G Section [4] Calculating the Index

I’m writing a series of posts about Generalizing Apdex. This is #18. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

I have been working systematically through the process of generalizing the current Apdex spec. Along the way, I have been skipping over parts of the current spec to work on the paragraphs that presented challenges. I am now going back to fill in those less contentious paragraphs, posting updated drafts of each section of the Apdex-G spec. The first three are Section [1] Introduction, Section [2] Index Overview, and Section [3] Calculation Inputs; today I continue with Section [4] Calculating the Index.


Summary

The first draft of this section of Apdex-G was posted as Configurable Scoring in Apdex-G. In this section, I have generalized and reorganized the material in Apdex section §4 as follows:

  • The introduction generalizes the language in the current spec.
  • Section [4.1] corresponds to Apdex section §4.1. I have generalized the language, removing references to response times. Since the first draft, I have revised the references to earlier sections, making them more complete.
  • Section [4.1.1] corresponds to the last paragraph of Apdex section §4.1. I have rewritten this section, generalizing the language and adding the second paragraph. I am not sure if the third paragraph is required.
  • Section [4.1.2] is new. It reflects the discussion of alternative scoring methods published with the first draft, and extends the conclusion with a rule for the implementation of a sliding scale.
  • Section [4.2] corresponds to Apdex section §4.2. I have generalized the language, and tried to enumerate the possible classes of errors in a general way. This section was discussed briefly with the first draft.

Because this material includes some drafts posted previously, some of which I have edited, everything in this series of posts is considered the second draft of Apdex-G. After the second draft is posted, I will begin work on Apdex-R, the addendum addressing measurements of response times. Apdex-R will cover all the domain-specific content of the current spec that has been excised from Apdex-G.

Apdex, current spec:

Apdex-G, second draft:

§4. Calculating the Index
The Apdex does not entail new measurements – rather it is a new way to represent existing measurements, calculated by counting the measurement samples in each of the performance zones.

§4.1 The Apdex Formula
The Apdex is calculated for each report group using the following equation:

Given,

    A report group that defines a set of measurement samples, and a target threshold T (seconds) between the satisfied-tolerating zones of performance

Where,

    F defines the threshold between the tolerating-frustrated zones of performance,
    and
    F = 4T

Such that,

    There are counts of response time measurement samples within the above defined performance zones of:
    Satisfied_Count = number of satisfied response time samples,
    Tolerating_Count = number of tolerating response time samples,
    Total_Samples = number of all samples in the report group

Then,

    Apdex Formula

[4] Calculating the Index

The Apdex does not entail new measurements, rather it is a new way to represent an existing set of measurements, reflecting the degree to which those measurements achieve designated targets.

[4.1] The Standard Apdex Formula

The Apdex index is calculated as follows. Given:

    Measurement data that meets the requirements of section [3.1]

    A report group comprising measurement samples, defined as specified in sections [3.2] and [3.3]

    Three performance zones (Satisfied, Tolerating, Frustrated), defined as specified in sections [2.5], [2.5.1], [2.5.2], and [3.4]

    An allocation process that assigns each sample to a performance zone, and counts all samples, so that:

    Total_Samples is the number of all samples in the report group

    Satisfied_Count is the number of report group samples in the Satisfied Zone

    Tolerating_Count is the number of report group samples in the Tolerating Zone

Then the Apdex index for the report group is:

    Apdex Formula

Note that measurements in the frustrated zone are counted in the number of total user samples in the denominator. To achieve the optimal Apdex value of 1.00, all users must experience satisfactory performance. If some users see tolerating or frustrating performance, then the index rapidly dips below 1.00. For example, if 80% of users are satisfied and 10% are tolerating, while the remaining 10% are frustrated, the index is 0.85.

[4.1.1] The Standard Formula in Action

Note that measurements in the Frustrated zone are counted in Total_Samples, the denominator of the formula, but do not contribute to the numerator.

Another way to describe the Apdex index is as the weighted proportion of satisfactory samples in the report group. Samples in the Satisfied Zone have weights of 1, those in the Tolerating Zone have weights of ½, and those in the Frustrated Zone have weights of 0.

[ TBD: ] Therefore to achieve the optimal Apdex value of 1.00, all samples must represent the Satisfaction Level of ‘Satisfied’. If any samples reflect Tolerating or Frustrated performance, the index drops below 1.00. For example, if 80% of samples are Satisfied and 10% are Tolerating, while the remaining 10% are Frustrated, the index is 0.85.

[4.1.2] Alternative Scoring in the Tolerating Zone

If the characteristics of a particular measurement and reporting domain justify using graduated weights within the Tolerating zone, an Addendum may substitute an alternative scoring function for the factor Tolerating_Count/2 in the standard formula. Any alternative scoring function (such as a sliding scale) must have the property that sample values (or value ranges) closest to the satisfied zone receive the greatest weight, in accordance with the index objectives (see [2.1] item 6).

§4.2 Dealing with Exceptions

User aborts are factored into the above equation. A user abort occurs when a user enters a new inquiry before the system responds with the original inquiry. A user-generated abort stops the timing of the Task. Therefore, user aborts can fall into any of the satisfied, tolerating, frustrated zones. If a tool can detect a clear server-generated abort, then it is handled differently. Server aborts (e.g., TCP closes within a Task) are counted as a frustrated sample regardless of the Task time measurement.

Some tools may have the optional capability to interpret the application to a greater level of detail than the minimal Task boundary. For example, they may be able to detect user relevant information at the layer of the application logic. If the tool can detect Task errors, then these application errors (e.g. Web page 404 replies) are counted as frustrated samples.

[4.2] Dealing with Exceptions

When a Report Group contains samples marked as errors or exceptions, tools performing the Apdex calculation should classify those measurements as follows:

User Error: Exceptions caused by user actions may be classified as Satisfied, Tolerating, or Frustrated in the same way as any other measurement, if the necessary field(s) are present in the sample. If the field(s) required to perform classification are absent, user-generated exceptions should be classified as Satisfied.

Warning: Exceptions indicating abnormal system or process behavior may be classified as Tolerating if the system or process returned immediately to normal operation without requiring abnormal intervention.

Failure: Exceptions indicating a system or process failure that is experienced by the user should be classified as Frustrated.

Measurement Error: Exceptions indicating measurement errors should be discarded from the Report Group.

Unknown: Exceptions not amenable to classification should be discarded from the Report Group. However, tool creators must attempt to minimize the number of error types assigned to this category.

Because all exceptions are specific to a particular measurement domain, an addendum may specify domain-specific refinements to these general guidelines.

Public Comment

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

1 comment to Apdex-G Section [4] Calculating the Index

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  


*