LinkedIn

See our LinkedIn profile, click on this button:

APDEX

Apdex-R Section [4] Calculating the Index

I’m writing a series of posts about Generalizing Apdex. This is #23. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec documents, Apdex-G and Apdex-R, are enclosed in square brackets and prefixed by the document type, like this: [G 1] and [R 1].

Drafts of the Apdex-G spec were posted as Section [1] Introduction, Section [2] Index Overview, Section [3] Calculation Inputs, Section [4] Calculating the Index, Section [5] Reporting, and Section [6] References.

The first draft of Apdex-R began with Sections 1 and 2, then Section 3. This post covers Section 4.

Apdex-R specifies the Apdex rules for reporting response-time measurements, therefore my first draft draws heavily on the present Apdex spec. My goal is to reformat the present spec’s rules within the context established by Apdex-G. To allow both documents to be viewed together, I show Apdex-G (draft #2, plus a few subsequent amendments) on the left, and Apdex-R (draft #1) on the right.

Differences between Apdex-R and the corresponding sections of the present spec are highlighted, as follows:

  • text deleted from the present spec This text is covered in Apdex-G, or elsewhere in Apdex-R, or extraneous.
  • new text added to Apdex-R (plus a few additions to Apdex-G draft #2)
  • new text created for Apdex-G, now being repeated in Apdex-R for readability
  • notes, questions, issues to be resolved

While major deletions, additions, and changes to the present spec are marked, many minor wording changes to the present spec’s text, to improve continuity and clarity of meaning, are not marked. I believe those changes do not substantially alter the meaning and intent of the present spec.

Apdex-R, first draft:

Apdex-G, second draft:

[R 4] Calculating the Index

See [G 4] for a general introduction to the Apdex index calculation.

The Apdex does not entail new measurements – rather it is a new way to represent existing measurements, calculated by counting the measurement samples in each of the performance zones.

[G 4] Calculating the Index

The Apdex does not entail new measurements, rather it is a new way to represent an existing set of measurements, reflecting the degree to which those measurements achieve designated targets.

[R 4.1] The Apdex Formula

See [G 4.1] for the Apdex formula, and corresponding input requirements:

    Apdex Formula

The following sections address Apdex-R input requirements:

    Measurement data: [R 3.1]

    Report group: [R 3.2] and [R 3.3]

    Performance zones: [R 2.5], [R 2.5.1], [R 2.5.2], and [R 3.4]

Given,

    A report group that defines a set of measurement samples, and a target threshold T (seconds) between the satisfied-tolerating zones of performance

Where,

    F defines the threshold between the tolerating-frustrated zones of performance,
    and
    F = 4T

Such that,

    There are counts of response time measurement samples within the above defined performance zones of:
    Satisfied_Count = number of satisfied response time samples,
    Tolerating_Count = number of tolerating response time samples,
    Total_Samples = number of all samples in the report group

[G 4.1] The Standard Apdex Formula

The Apdex index is calculated as follows. Given:

    Measurement data that meets the requirements of section [G 3.1]

    A report group comprising measurement samples, defined as specified in sections [G 3.2] and [G 3.3]

    Three performance zones (Satisfied, Tolerating, Frustrated), defined as specified in sections [G 2.5], [G 2.5.1], [G 2.5.2], and [G 3.4]

    An allocation process that assigns each sample to a performance zone, and counts all samples, so that:

    Total_Samples is the number of all samples in the report group

    Satisfied_Count is the number of report group samples in the Satisfied Zone

    Tolerating_Count is the number of report group samples in the Tolerating Zone

Then the Apdex index for the report group is:

    Apdex Formula

[R 4.1.1] The Standard Formula in Action

See [G 4.1.1] for a general overview of the Apdex formula in action.

Note that measurements in the frustrated zone are counted in the number of total user samples in the denominator. To achieve the optimal Apdex value of 1.00, all users must experience satisfactory performance. If some users see tolerating or frustrating performance, then the index rapidly dips below 1.00. For example, if 80% of users are satisfied and 10% are tolerating, while the remaining 10% are frustrated, the index is 0.85.

[G 4.1.1] The Standard Formula in Action

Note that measurements in the Frustrated zone are counted in Total_Samples, the denominator of the formula, but do not contribute to the numerator.

Another way to describe the Apdex index is as the weighted proportion of satisfactory samples in the report group. Samples in the Satisfied Zone have weights of 1, those in the Tolerating Zone have weights of ½, and those in the Frustrated Zone have weights of 0.

[ TBD: ] Therefore to achieve the optimal Apdex value of 1.00, all samples must represent the Satisfaction Level of ‘Satisfied’. If any samples reflect Tolerating or Frustrated performance, the index drops below 1.00. For example, if 80% of samples are Satisfied and 10% are Tolerating, while the remaining 10% are Frustrated, the index is 0.85.

[R 4.1.2] Alternative Scoring in the Tolerating Zone

See [G 4.1.2] for the rules for optional alternative scoring. Apdex-R permits tool creators to substitute an alternative scoring function in the tolerating zone, provided that sample values (or value ranges) closest to the satisfied zone receive the greatest weight.

[G 4.1.2] Alternative Scoring in the Tolerating Zone

If the characteristics of a particular measurement and reporting domain justify using graduated weights within the Tolerating zone, an Addendum may substitute an alternative scoring function for the factor Tolerating_Count/2 in the standard formula. Any alternative scoring function (such as a sliding scale) must have the property that sample values (or value ranges) closest to the satisfied zone receive the greatest weight, in accordance with the index objectives (see [G 2.1] item 6).

[R 4.2] Dealing with Exceptions

See [G 4.2] for general rules for dealing with exceptions. The following additional notes may apply to measurements of Task (and Task Chain) times.

User aborts: User aborts are factored into the above equation. A user abort occurs when a user enters a new inquiry before the system responds with the original inquiry. A user-generated abort stops the timing of the Task. Therefore, user aborts can fall into any of the satisfied, tolerating, frustrated zones.

Server failure: If a tool can detect a clear server-generated abort, then it is handled differently. Server aborts (e.g., TCP closes within a Task) are counted as a frustrated sample regardless of the Task time measurement.

Application failure: Some tools may have the optional capability to interpret the application to a greater level of detail than the minimal Task boundary. For example, they may be able to detect user relevant information at the layer of the application logic. If the tool can detect Task errors, then these application errors (e.g. Web page 404 replies) are counted as frustrated samples.

[G 4.2] Dealing with Exceptions

When a Report Group contains samples marked as errors or exceptions, tools performing the Apdex calculation should classify those measurements as follows:

User Error: Exceptions caused by user actions may be classified as Satisfied, Tolerating, or Frustrated in the same way as any other measurement, if the necessary field(s) are present in the sample. If the field(s) required to perform classification are absent, user-generated exceptions should be classified as Satisfied.

Warning: Exceptions indicating abnormal system or process behavior may be classified as Tolerating if the system or process returned immediately to normal operation without requiring abnormal intervention.

Failure: Exceptions indicating a system or process failure that is experienced by the user should be classified as Frustrated.

Measurement Error: Exceptions indicating measurement errors should be discarded from the Report Group.

Unknown: Exceptions not amenable to classification should be discarded from the Report Group. However, tool creators must attempt to minimize the number of error types assigned to this category.

Because all exceptions are specific to a particular measurement domain, an addendum may specify domain-specific refinements to these general guidelines.

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  


*