LinkedIn

See our LinkedIn profile, click on this button:

APDEX

Apdex-R Section [3] Calculation Inputs

I’m writing a series of posts about Generalizing Apdex. This is #22. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec documents, Apdex-G and Apdex-R, are enclosed in square brackets and prefixed by the document type, like this: [G 1] and [R 1].

Drafts of the Apdex-G spec were posted as Section [1] Introduction, Section [2] Index Overview, Section [3] Calculation Inputs, Section [4] Calculating the Index, Section [5] Reporting, and Section [6] References.

The first draft of Apdex-R began with Sections 1 and 2. This post covers Section 3.

Apdex-R specifies the Apdex rules for reporting response-time measurements, therefore my first draft draws heavily on the present Apdex spec. My goal is to reformat the present spec’s rules within the context established by Apdex-G. To allow both documents to be viewed together, I show Apdex-G (draft #2, plus a few subsequent amendments) on the left, and Apdex-R (draft #1) on the right.

Differences between Apdex-R and the corresponding sections of the present spec are highlighted, as follows:

  • text deleted from the present spec This text is covered in Apdex-G, or elsewhere in Apdex-R, or extraneous.
  • new text added to Apdex-R (plus a few additions to Apdex-G draft #2)
  • new text created for Apdex-G, now being repeated in Apdex-R for readability
  • notes, questions, issues to be resolved

While major deletions, additions, and changes to the present spec are marked, many minor wording changes to the present spec’s text, to improve continuity and clarity of meaning, are not marked. I believe those changes do not substantially alter the meaning and intent of the present spec.

Apdex-R, first draft:

Apdex-G, second draft:

[R 3]. Apdex Calculation Inputs

See [G 3] for a general introduction to Apdex calculation inputs.

A wide variety of tools and methods may be used to gather the necessary information for an Apdex calculation and report. The following are the minimal requirements for such a tool.

Each tool creator may add features and capabilities above these minimal requirements.

[G 3] Apdex Calculation Inputs

Apdex does not favor any tool or method; a wide variety of tools and methods may be used to gather the necessary information for an Apdex calculation and report.

This section specifies the minimal requirements governing input to any Apdex calculation. A tool creator may add features and capabilities above these minimal requirements. An addendum for a particular measurement type may also specify domain-specific rules governing the methods and tools used to obtain measurements of that type.

[R 3.1] Measurements: Tasks and Task Chains

See [G 3.1] for a general rules regarding measurements.

[R 3.1.1] Task Measurements

Tool creators must identify which applications their tool is capable of interpreting at the Task level. For example, having the ability to mark the start and end of a Web page by interpreting the HTTP and HTML protocols is sufficient for Web-based applications. The tool must clearly identify the application by protocol (e.g., HTTP/HTML), or by software maker and product name (e.g., Microsoft Exchange), that it is capable of properly interpreting Tasks and measuring their response times.

Implementation of the Task measurement will vary. However, Task measurements must be a reasonable approximation of response time measurements that could be performed by the user with a stopwatch.

[R 3.1.2] Task Chain Measurements

Because Task Chain Time is defined to be the sum of a set of Task Times, and does not include any Think Time between Tasks, obtaining a Task Chain Time measurement will probably necessitate measuring and accumulating individual Task times. However, the implementation is left to the measurement tool developer, provided that the result is a reasonable approximation of a measurement of Task Chain Time a user could obtain by measuring and summing the component Task Times.

Implementations of Apdex-based reporting may vary, provided that individual measurements are assigned to a performance zone that is factored into an Apdex calculation and reported as further defined in this specification.

[G 3.1] Measurements

Tool creators must identify the measurement types their tool is capable of analyzing. When a tool analyzes a measurement type that is covered by an addendum, the tool creator must apply the rules specified in that addendum, in addition to those specified in this document.

Implementations of Apdex-based reporting may vary, provided that individual measurements are assigned to a performance zone that is factored into an Apdex calculation and reported as further defined in this specification.

[R 3.2] Report Groups

The report group is a named set of measurement samples that will be input to an Apdex calculation. Tools must provide a way for the technician to specify a report group. See [G 3.2] for general rules regarding the specification of report groups. Tools implementing Apdex-R must also allow the technician to specify the following values for filters:

The report group is a specified set of individual measurement samples (of Task Time or Task Chain Time) that will form the foundation for an Apdex calculation. A report group’s set of measurement samples may be unique or may have overlapping measurement samples with other report groups. Report groups can be defined in many ways, but the following are required:

Measurement Type (R)
If the tool analyzes measurements of more than one type, it must provide a filter that limits a report group to response time data only. The value “R” (signifying “response time data”) may be the default measurement type.
Measurement Subtypes
Task and Task Chain must be distinguished; measurements of Task Time and Task Chain Time may not be combined in one report group.
Application
An application as selected by the technician. At a minimum, this is the application that the tool can interpret to the Task level (see above).
User Group
The technician must be able to define various user groups of an application (e.g., geography, organization).
Time Period
The technician must be able to define time of day periods for which the index will be calculated.

The report group is one of the fundamental controls available to a technician. The report group may be defined as broadly as all of the samples for an application, or as narrowly as a single sample. Single samples are useful for diagnostic purposes.

[G 3.2] Report Groups

The report group is a named set of measurement samples that will be input to an Apdex calculation. Tools must provide a way for the technician to specify a report group.

Report groups can be defined in many ways, but tools must allow the technician to specify values for the following filters:

Measurement Type
Required if the tool analyzes measurements of more than one type. Measurements of different types may not be included in the same report group.
Measurement Subtype
Required if the measurement type is covered by an addendum that defines distinct subtypes, and the tool analyzes measurements of more than one subtype. Measurements of different subtypes may not be included in the same report group.
Application(s) or Process(es)
Required if the tool analyzes measurements of more than one application or process. Measurements of more than one application or process may be included in the same report group if they are subject to the same thresholds.
Usage Segment(s)
Required if the tool analyzes measurements of more than one usage segment. A usage segment is any identifiable subset of the measurements of an application or process, such as a demographic segment, a geographic segment, an organizational segment, etc. Measurements of different usage segments may be included in the same report group if they are subject to the same thresholds.
Time Period(s)
Required. The technician must be able to define the time period(s) for which the index will be calculated.

Filters can be applied alone and in combination with others, to select a report group from a larger database of measurement data. An addendum governing a particular measurement type may define additional filters to be provided for measurement of that type.

[R 3.3] Report Group Size

See [G 3.3] for rules regarding report group size.

There must be at least one sample in the report group in order to calculate an Apdex value. Single sample Apdex results are permitted but discouraged. The primary reason to have a small sample size is to cover low traffic periods or provide data for diagnostic reasons. These are not considered good indicators of application performance.

The minimum sample size for normal reporting of the index is 100 samples within a report group. Special handling is required for smaller report groups; Section 5.2 specifies the method for reporting an Apdex result for a report group of fewer than 100 samples.

[G 3.3] Report Group Size

A set of measurement samples comprising a report group may be unique or may overlap with the measurement samples in other report groups. A report group may be defined as broadly as all of the samples for an application, or as narrowly as a single sample.

For normal reporting, the default minimum sample count for a report group is 100 samples; addenda may specify domain-specific minima. Any report group containing fewer than the specified minimum sample count is considered a small group.

Small groups may not be reliable as performance indicators, but may occur when reporting low traffic periods, or when reporting diagnostic measurements. Special handling is required for reporting small groups; this is specified in section [G 5.2].

A report group must contain at least one sample for an Apdex value to be calculated. Single samples report groups may be useful for diagnostic purposes. Single sample Apdex results are also permitted when reporting low traffic periods, but discouraged.

[R 3.4] Thresholds

See [G 3.4] for rules regarding threshold settings. For tools supporting Apdex-R, it is recommended that the default target threshold value, T, be set to 4 seconds.

The technician (operator of the tool) must be able to set the target threshold, T (satisfied-tolerating boundary) and no other Apdex threshold for each application. This threshold is a positive decimal value in seconds, having no more than two significant digits of granularity.

This means that the following types of values are permitted:

0<T<10
Greater than 0 seconds and below 10 seconds can be defined to a tenth of a second.
Examples: 0.5, 1.2, 5.8, 9.9
10<T<100
Equal to or greater than 10 seconds but below 100 seconds can be defined to one second.
Examples: 10, 19, 56, 85, 99
100<T<1,000
Equal to or greater than 100 seconds but below 1000 seconds can be defined to 10 seconds.
Examples: 100, 190, 560, 850, 990
T>1,000
Values equal to or greater than 1,000 seconds follow the same two significant digits restriction.

The threshold must be consistent for all samples of a report group.

All tools will ship with a default setting of T that will be selected by the tool creator. The default enables the tool to begin supplying information with minimal set-up by the technician. Technicians have the ability to change this default setting as defined above.

[G 3.4] Thresholds

Tools must provide a way for the technician to specify the thresholds that define the boundaries of performance intervals to be used when calculating an Apdex index for a report group. A single set of thresholds is applied to all samples in a report group.

Thresholds are decimal values having no more than two significant digits of precision. This means that the following types of values are permitted, for any measurement unit, where a ‘unit’ refers to the dimension of the measurement type, such as seconds, meters, kilograms:

0<|T|<10
Values greater than 0 units and below 10 units can be defined to a tenth of a unit.
Examples: 0.5, 1.2, 5.8, 9.9
10<|T|<100
Values equal to or greater than 10 units but below 100 units can be defined to one unit.
Examples: 10, 19, 56, 85, 99
100<|T|<1,000
Values equal to or greater than 100 units but below 1000 units can be defined to 10 units.
Examples: 100, 190, 560, 850, 990
|T|>1,000
Values equal to or greater than 1,000 units follow the same two significant digits restriction.

A tool that supports a specific measurement type will ship with default threshold values selected by the tool creator, which technicians can change. These defaults enable the tool to begin supplying information with minimal set-up by the technician. An addendum may specify additional rules governing default threshold values, relationships among threshold values, and the required precision of threshold values, as appropriate for a specific measurement type.

A goal of Apdex is to operate so that different Apdex tools will report similar index values for user experiences having comparable levels of quality (see [G 2.1] item 8). However, tools use different measurement methods, and measure from different vantage points within a system. As a result, data purporting to measure the same quantity can have different values. If such differences are systematic, the technician can adjust the Apdex thresholds used with measurements from different tools so that the Apdex calculation produces comparable index values.

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  


*