LinkedIn

See our LinkedIn profile, click on this button:

APDEX

Apdex-G Section [3] Calculation Inputs

I’m writing a series of posts about Generalizing Apdex. This is #17. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

I have been working systematically through the process of generalizing the current Apdex spec. Along the way, I have been skipping over parts of the current spec to work on the paragraphs that presented challenges. I am now going back to fill in those less contentious paragraphs. Over the next week, I plan to post updated drafts of each section of the Apdex-G spec. The first two are Section [1] Introduction and Section [2] Index Overview; today I continue with Section [3] Calculation Inputs.


Summary

In this section of Apdex-G, I have generalized and reorganized the material in Apdex section §3 as follows:

  • The introduction repeats the current spec, adding a note about the possibility of a domain specific addendum.
  • Section [3.1] corresponds to Apdex section §3.1. I have generalized the language, and removed the discussion of Web response time measurements, which will be reviewed later for inclusion in Apdex-R.
  • Section [3.2] corresponds to Apdex section §3.2. I have rewritten this section to improve the flow, added the Measurement Subtype filter (to accommodate the distinction between Task and Task Chain in the current spec, and later in Apdex-R, for example).
  • Section [3.3] corresponds to Apdex section §3.4. I reversed the order of sections §3.3 and §3.4, to improve the flow of ideas. For the same reason, I moved content about report group size in §3.2 to [3.3].
  • Section [3.4] corresponds to Apdex section §3.3, generalizing response time terminology, and adding a paragraph about adjusting thresholds to compensate for differences between tools.

Because this material includes some drafts posted previously, some of which I have edited, everything in this series of posts is considered the second draft of Apdex-G. After the second draft is posted, I will begin work on Apdex-R, the addendum addressing measurements of response times. Apdex-R will cover all the domain-specific content of the current spec that has been excised from Apdex-G.

Apdex, current spec:

Apdex-G, second draft:

§3. Apdex Calculation Inputs

A wide variety of tools and methods may be used to gather the necessary information for an Apdex calculation and report. The following are the minimal requirements for such a tool.

Each tool vendor may add features and capabilities above these minimal requirements.

[3] Apdex Calculation Inputs

Apdex does not favor any tool or method; a wide variety of tools and methods may be used to gather the necessary information for an Apdex calculation and report.

This section specifies the minimal requirements governing input to any Apdex calculation. A tool creator may add features and capabilities above these minimal requirements. An addendum for a particular measurement type may also specify domain-specific rules governing the methods and tools used to obtain measurements of that type.

§3.1 Response Time Measurements

Tool vendors must identify which applications their tool is capable of interpreting at the Task level.

[Examples of Web page response-time measurement]

Implementations of Apdex-based reporting may vary, provided that individual measurements are assigned to a performance zone that is factored into an Apdex calculation and reported as further defined in this specification.

[3.1] Measurements
Tool creators must identify the measurement types their tool is capable of analyzing. When a tool analyzes a measurement type that is covered by an addendum, the tool creator must apply the rules specified in that addendum, in addition to those specified in this document.

Implementations of Apdex-based reporting may vary, provided that individual measurements are assigned to a performance zone that is factored into an Apdex calculation and reported as further defined in this specification.

§3.2 Defining a Report Group

The report group is a specified set of individual measurement samples (of Task Time or Task Chain Time) that will form the foundation for an Apdex calculation. A report group’s set of measurement samples may be unique or may have overlapping measurement samples with other report groups. Report groups can be defined in many ways, but the following are required:

Type
Task or Task Chain; measurements of Task Time and Task Chain Time may not be combined in one report group.
Application
An application as selected by the technician. At a minimum, this is the application that the tool can interpret to the Task level (see above).
User Group
The technician must be able to define various user groups of an application (e.g., geography, organization).
Time Period
The technician must be able to define time of day periods for which the index will be calculated.

The report group is one of the fundamental controls available to a technician. The report group may be defined as broadly as all of the samples for an application, or as narrowly as a single sample. Single samples are useful for diagnostic purposes.

[3.2] Report Groups

The report group is a named set of measurement samples that will be input to an Apdex calculation. Tools must provide a way for the technician to specify a report group.

Report groups can be defined in many ways, but tools must allow the technician to specify values for the following filters:

Measurement Type
Required if the tool analyzes measurements of more than one type. Measurements of different types may not be included in the same report group.
Measurement Subtype
Required if the measurement type is covered by an addendum that defines distinct subtypes, and the tool analyzes measurements of more than one subtype. Measurements of different subtypes may not be included in the same report group.
Application(s) or Process(es)
Required if the tool analyzes measurements of more than one application or process. Measurements of more than one application or process may be included in the same report group if they are subject to the same thresholds.
Usage Segment(s)
Required if the tool analyzes measurements of more than one usage segment. A usage segment is any identifiable subset of the measurements of an application or process, such as a demographic segment, a geographic segment, an organizational segment, etc. Measurements of different usage segments may be included in the same report group if they are subject to the same thresholds.
Time Period(s)
Required. The technician must be able to define the time period(s) for which the index will be calculated.

Filters can be applied alone and in combination with others, to select a report group from a larger database of measurement data. An addendum governing a particular measurement type may define additional filters to be provided for measurement of that type.

§3.4 Number of Measurement Samples

There must be at least one sample in the report group in order to calculate an Apdex value. Single sample Apdex results are permitted but discouraged. The primary reason to have a small sample size is to cover low traffic periods or provide data for diagnostic reasons. These are not considered good indicators of application performance.

The minimum sample size for normal reporting of the index is 100 samples within a report group. Special handling is required for smaller report groups; Section 5.2 specifies the method for reporting an Apdex result for a report group of fewer than 100 samples.

[3.3] Report Group Size

A set of measurement samples comprising a report group may be unique or may overlap with the measurement samples in other report groups. A report group may be defined as broadly as all of the samples for an application, or as narrowly as a single sample.

For normal reporting, the default minimum sample count for a report group is 100 samples; addenda may specify domain-specific minima. Any report group containing fewer than the specified minimum sample count is considered a small group.

Small groups may not be reliable as performance indicators, but may occur when reporting low traffic periods, or when reporting diagnostic measurements. Special handling is required for reporting small groups; this is specified in section [5.2].

A report group must contain at least one sample for an Apdex value to be calculated. Single samples report groups may be useful for diagnostic purposes. Single sample Apdex results are also permitted when reporting low traffic periods, but discouraged.

§3.3 Threshold Settings

The technician (operator of the tool) must be able to set the target threshold, T (satisfied-tolerating boundary) and no other Apdex threshold for each application. This threshold is a positive decimal value in seconds, having no more than two significant digits of granularity.

This means that the following types of values are permitted:

0<T<10
Greater than 0 seconds and below 10 seconds can be defined to a tenth of a second.
Examples: 0.5, 1.2, 5.8, 9.9
10<T<100
Equal to or greater than 10 seconds but below 100 seconds can be defined to one second.
Examples: 10, 19, 56, 85, 99
100<T<1,000
Equal to or greater than 100 seconds but below 1000 seconds can be defined to 10 seconds.
Examples: 100, 190, 560, 850, 990
T>1,000
Values equal to or greater than 1,000 seconds follow the same two significant digits restriction.

The threshold must be consistent for all samples of a report group.

All tools will ship with a default setting of T that will be selected by the tool vendor. The default enables the tool to begin supplying information with minimal set-up by the technician. It is recommended that the default target threshold value, T, be set to 4 seconds. Technicians have the ability to change this default setting as defined above.

[3.4] Thresholds

Tools must provide a way for the technician to specify the thresholds that define the boundaries of performance intervals to be used when calculating an Apdex index for a report group. A single set of thresholds is applied to all samples in a report group.

Thresholds are decimal values having no more than two significant digits of precision. This means that the following types of values are permitted, for any measurement unit, where a ‘unit’ refers to the dimension of the measurement type, such as seconds, meters, kilograms:

0<T<10
Values greater than 0 units and below 10 units can be defined to a tenth of a unit.
Examples: 0.5, 1.2, 5.8, 9.9
10<T<100
Values equal to or greater than 10 units but below 100 units can be defined to one unit.
Examples: 10, 19, 56, 85, 99
100<T<1,000
Values equal to or greater than 100 units but below 1000 units can be defined to 10 units.
Examples: 100, 190, 560, 850, 990
T>1,000
Values equal to or greater than 1,000 units follow the same two significant digits restriction.

A tool that supports a specific measurement type will ship with default threshold values selected by the tool creator, which technicians can change. These defaults enable the tool to begin supplying information with minimal set-up by the technician. An addendum may specify additional rules governing default threshold values, relationships among threshold values, and the required precision of threshold values, as appropriate for a specific measurement type.

A goal of Apdex is to operate so that different Apdex tools will report similar index values for user experiences having comparable levels of quality (see [2.1] item 8). However, tools use different measurement methods, and measure from different vantage points within a system. As a result, data purporting to measure the same quantity can have different values. If such differences are systematic, the technician can adjust the Apdex thresholds used with measurements from different tools so that the Apdex calculation produces comparable index values.

Public Comment

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  


*