LinkedIn

See our LinkedIn profile, click on this button:

APDEX

Apdex-R Section [5] Reporting

I’m writing a series of posts about Generalizing Apdex. This is #24. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec documents, Apdex-G and Apdex-R, are enclosed in square brackets and prefixed by the document type, like this: [G 1] and [R 1].

Drafts of the Apdex-G spec were posted as Section [1] Introduction, Section [2] Index Overview, Section [3] Calculation Inputs, Section [4] Calculating the Index, Section [5] Reporting, and Section [6] References.

The first draft of Apdex-R began with Sections 1 and 2, Section 3, and Section 4. This post covers Section 5.

Apdex-R specifies the Apdex rules for reporting response-time measurements, therefore my first draft draws heavily on the present Apdex spec. My goal is to reformat the present spec’s rules within the context established by Apdex-G. To allow both documents to be viewed together, I show Apdex-G (draft #2, plus a few subsequent amendments) on the left, and Apdex-R (draft #1) on the right.

Differences between Apdex-R and the corresponding sections of the present spec are highlighted, as follows:

  • text deleted from the present spec This text is covered in Apdex-G, or elsewhere in Apdex-R, or extraneous.
  • new text added to Apdex-R (plus a few additions to Apdex-G draft #2)
  • new text created for Apdex-G, now being repeated in Apdex-R for readability
  • notes, questions, issues to be resolved

While major deletions, additions, and changes to the present spec are marked, many minor wording changes to the present spec’s text, to improve continuity and clarity of meaning, are not marked. I believe those changes do not substantially alter the meaning and intent of the present spec.

Apdex-R, first draft:

Apdex-G, second draft:

[R 5] Reporting the IndexIn this section, the terms Report(s) and Reporting refer to any representation of an Apdex index, whether in print, on a computer screen, or elsewhere. The index is a decimal value between 0 and 1. The Apdex representation on all reports (screen, print, or otherwise) must adhere to the following rules.
[G 5] Reporting the IndexIn this section, the terms Report(s) and Reporting refer to any representation of an Apdex index, whether in print, on a computer screen, or elsewhere.

[R 5.1] Reporting the Apdex ValueSee [G 5.1] for general rules about reporting the Apdex value. Apdex-R adds the following rule for reporting performance zones:

All Apdex-R values are calculated using two thresholds, T and F, which must be reported in association with the corresponding Apdex score for a report group. The value of T must always be reported. If the specification of the report group applied a default rule for the value of F (see [R 2.5.1]), the value of F may be omitted.

The index always starts with a 0, followed by a decimal point, followed by the fractional value for the calculation to two decimals, and the value of T is clearly associated with the index presentation as defined below. There is a special case of the value 1.00 that starts with the digit of 1. A 1.00 is presented when the formula produces a result that can be rounded arithmetically to 1.00 (results equal to or greater than 0.995 round to 1.00).

Apdex may not be reported in granularity smaller than one one-hundredth. Decimal values of 3 or more digits are not permitted.

Tools in a computer screen or in printed reports always identify it as an Apdex value. When there are many values of the index presented in tabular form representing many reporting groups, then the Apdex label should appear with the appropriate column or row.

All Apdex values are calculated with a particular target threshold, T. The value of T must be clearly displayed in association with the Apdex score. Furthermore, in order to facilitate exporting Apdex values from a tool and then importing them to other analysis tools, all tools must support at least one uniform output as shown below.

[G 5.1] Reporting the Apdex ValueReports must adhere to the following rules:

  1. The index is a decimal value between 0 and 1, rounded to a precision of two decimal places. Values equal to or greater than 0.995 round to 1.00.
  2. Unless its value is 1.00, the index is always reported with a zero in the tens place, followed by a decimal point, followed by no more than two decimal places.
  3. Apdex values must be identified as such. When several Apdex values are presented in tabular form, a single Apdex identifier can be identify a group of values in a row, column, or table.
  4. All Apdex values are calculated based on specified performance zones. The performance zone specification must be clearly displayed in association with the Apdex score.

[R 5.1.1] Describing General CasesSee [G 5.1.1] for rules about describing unspecified Apdex thresholds.

General Apdex value discussions that reflect any target of T are written with “T” as shown in the following examples.

Uniform Output: “Everyone should understand that 0.90 [T] is a better value than 0.80 [T].”
Subscripted Output: “Everyone should understand that 0.90T is a better value than 0.80T.”

[G 5.1.1] Describing General CasesIn general discussions of Apdex values, references to unspecified performance thresholds are written using the notation “[T]”, as shown in the following examples.

“Everyone should understand that 0.90 [T] is a better value than 0.80 [T].”
“Apdex scores in the range 0.85 to 0.93 [T] are rated Good.”

Sections [G 5.3] and [G 5.4] also use this notation when referring to threshold reporting.

[R 5.2] Uniform Output File (Mandatory)To support the exchange of index values between Apdex analysis and reporting tools, all Apdex analysis tools must support the Apdex Uniform Output format specified in [G 5.2]. In Apdex-R, the following additional rules apply:

[R 5.2.1] Bracketed Output Format (Mandatory)

Tools implementing Apdex-R must be capable of reporting Apdex-R values using the bracketed format described in Table 2. When the value of F is set using a default rule (see [R 2.5.1]) elements 5 and 6 in Table 2 may be omitted, so that only the value of the threshold T accompanies the reported Apdex value.

Element Number Definition Type Content
1 Apdex index value Number Decimal in range [0.00, 1.00]
2 Space Literal
3 Left bracket designating start of threshold(s) Literal [
4 Tolerating threshold, T Number Decimal
5 Comma Literal ,
6 Frustrated threshold, F Number Decimal
7 Right bracket designating end of threshold(s) Literal ]
8 Small Group Indicator Literal * or NS
Table 2. Bracketed Output Format

Examples of bracketed output format are: 0.85 [5.5], 1.00 [8.0]*, 0.90 [4.0], and 0.77 [450].

[R 5.2.2] Subscripted Output Format (Optional)

When the value of F is set using a default rule (see [R 2.5.1]), tools may display Apdex values in which the value of T is shown as a mathematical subscript. For example, an Apdex value of 0.75 that is based upon a T value of 4 seconds is shown as 0.754.

When T has a decimal component, as defined in section [G 3.4], then the exact value of T must be shown (example: 0.754.5). To increase readability, tools may drop the decimal portion of T if it is zero (example: 0.754.0 becomes 0.754).

[G 5.2] Uniform Output File (Mandatory)Separate tools may be used to calculate Apdex index values from measurement data and to report Apdex index values. To support the exchange of index values between Apdex analysis and reporting tools, all Apdex analysis tools must support the Apdex Uniform Output format. A tool shall display, print, and export Apdex-G values to an ASCII file having a comma-separated values (CSV) format. For this purpose, Apdex-G incorporates the IETF specification of CSV published in RFC 4180 (see section [G 6] References).

A Uniform Output File is composed of a uniform output header record, followed by one or more uniform output data records. The header record contains the same number of comma-separated fields as all data records in the file. Fields in the header record contain fixed literal values, or names that correspond to each field in the data records. The content of a uniform output header and data records are described in Tables 2 and 3 below. In Table 2, each element except the last is followed by a comma.

Element Number Definition Header Record Data Record
Type Content Type Content
1 Apdex data record identifier Literal Apdex Header Literal Apdex
Report Group Metadata Section Header Record Data Record
2 Apdex Report Group section identifier Literal ARG Literal ARG
3 Report Group Name Literal Report Group Name Defined by tool or user
4 Report Group Description Literal Description Text String Defined by tool or user. See note 1 below
5 Measurement Type Literal Type Name G, or addendum type (e.g. R for Apdex-R)
6 Measurement Subtype Literal Subtype Name Values specified in an addendum for this measurement type
7 Application Literal Application Name Defined by tool or user
8 User Group Literal User Group Name Defined by tool or user
9 Time Period Start Literal Start Time Timestamp ISO 8601: [YYYY][MM][DD]T[hh][mm][ss]Z
10 Time Period End Literal End Time Timestamp ISO 8601: [YYYY][MM][DD]T[hh][mm][ss]Z
Input Data Summary Section Header Record Data Record
11 Apdex Data Summary section identifier Literal ADS Literal ADS
12 Sample Count Literal Total Samples Number Integer
13 Satisfied Zone Count Literal Satisfied Count Number Integer
14 Tolerating Zone Count Literal Tolerating Count Number Integer
15 Frustrated Zone Count Literal Frustrated Count Number Integer
16 Earliest Sample Timestamp Literal First Sample Timestamp ISO 8601: [YYYY][MM][DD]T[hh][mm][ss]Z
17 Latest Sample Timestamp Literal Last Sample Timestamp ISO 8601: [YYYY][MM][DD]T[hh][mm][ss]Z
Apdex Index Section Header Record Data Record
18 Apdex IndeX section identifier Literal AIX Literal AIX
19 Apdex index value Literal Apdex Index Number Decimal in range [0.00, 1.00]
20 Satisfied Zone Identifier Literal S Literal S
21 Satisfied Thresholds(s) Group Performance Interval Names Group Interval Group, see Table 3
22 Tolerating Zone Identifier Literal T Literal T
23 Tolerating Thresholds(s) Group Performance Interval Names Group Interval Group, see Table 3
24 Frustrated Zone Identifier Literal F Literal F
25 Frustrated Thresholds(s) Group Performance Interval Names Group Interval Group, see Table 3
26 Small Group Indicator Literal SGI Literal * or NS
Table 2. Layout of a Uniform Output Data Record

Note 1: To avoid ambiguity, tool creators must ensure that Report Group description fields do not contain commas, because commas separate fields in the Uniform Output format.

Performance Interval Names

If a tool has obtained user-specified names for performance intervals, those names should be included in the corresponding fields of the header record (elements 21, 23, and 25 in Table 2). If user-specified names are not available, distinct default names must be supplied for each performance interval field in the header record. See also section [G 2.5.1] Thresholds and Performance Intervals.

Performance Interval Groups

In Table 2, each Interval Group (elements 21, 23, and 25) is composed of one or more Performance Interval elements, separated by commas. Table 3 shows the structure of a single Performance Interval element, therefore the components shown in Table 3 are not separated by commas.

Component Definition Header Record Data Record
Type Content Type Content
1 Opening backet Name Performance Interval Name Literal ( or [
2 Lower threshold Number Decimal number — see [G 3.4]
3 Colon Literal : See note 2 below
4 Upper threshold Number Decimal number — see [G 3.4]
5 Closing backet Literal ) or ]
Table 3. Components of a Performance Interval within a Uniform Output Data Record

Note 2: Interval Notation, described in References [G 6.8], uses a comma to separate the upper and lower bounds of an interval. To avoid ambiguity Apdex substitutes a colon, because commas separate fields in the Uniform Output format.

Optional Sections

The Uniform Output File format comprises three sections, each having its own identifier (fields 2, 11, and 18). The Report Group Metadata Section and the Apdex Index Section are required, because they contain the necessary data for reporting Apdex scores. The Input Data Summary Section is optional, but tool creators are encouraged to produce this data at the analysis stage, and to pass it to Reporting tools. Reporting tools can use this data to provide additional context for Apdex scores. If the Input Data Summary Section is omitted, fields 11 through 17 must be omitted from the header and data records.

Examples of the uniform output

Examples of the uniform output are:

Apdex Header,ARG,,,,,,,,,AIX,Apdex Index,S,Green,T,Yellow,F,Red,SGI CRLF user labels
Apdex,ARG,,,,,,,,,AIX,0.72,S,[0:4],T,(4:16],F,(16:INF) CRLF
Apdex,ARG,,,,,,,,,AIX,0.85,S,[0:6],T,(6:20],F,(20:INF),* CRLF
Apdex,ARG,,,,,,,,,AIX,0.58,S,[0:3],T,(3:10],F,(10:INF) CRLF
Apdex,ARG,,,,,,,,,AIX,0.91,S,[0:10],T,(10:30],F,(30:INF) CRLF

Apdex Header,ARG,,,,,,,,,AIX,Apdex Index,S,PI3,T,PI2,PI4,F,PI1,PI5,SGI CRLF generic labels
Apdex,ARG,,,,,,,,,AIX,0.88,S,(10:12],T,(6:10],(12:16],F,[0:6],(16:INF) CRLF
Apdex,ARG,,,,,,,,,AIX,0.96,S,(9:15],T,(6:9],(15:18],F,[0:6],(18:INF) CRLF

[ More details and examples will be added here ]

[R 5.3] Indicating Sample SizeSee [G 5.3] for rules about reporting the sample size of an Apdex report group.

Apdex values are calculated based upon a set of measurements (samples) in the report group. If there are a small number of samples, the tool must still present a result. However, a result for such a small report group must be clearly marked.

A small report group is defined as any number of samples between 0 and 99. Apdex tools will clearly indicate that the result is based upon one of the following scenarios:

No Samples
The Apdex calculation could not be performed because there were no samples (NS) within the report group. Where the calculated Apdex value would normally appear, the tool will show an output of NS. Examples: NS [4.0], NS4
Small Group
When an Apdex value is the output of a small group (1 to 99) calculation, an asterisk (*) must be appended to that value. Examples: 0.80 [4.0]*, 0.804*.

[G 5.3] Indicating Sample SizeApdex values are calculated based upon a set of measurements (samples) in the report group. If there are a small number of samples, the tool must still present a result. However, a result for such a small report group must be clearly marked.

A small report group is defined as one having fewer than 100 samples. An addendum may modify this definition to be appropriate for a particular measurement domain. Apdex tools will clearly indicate that the result is based upon one of the following scenarios:

No Samples
The Apdex calculation could not be performed because there were no samples (NS) within the report group. Where the calculated Apdex value would normally appear, the tool will show an output of NS [T], where [T] is the normal threshold display (see section [G 5.1.1]).
Small Group
When an Apdex value is the output of a small group calculation, an asterisk (*) must be appended to that value, for example: 0.84* [T], where [T] is the normal threshold display (see section [G 5.1.1]).

[R 5.4] Apdex Quality RatingsSee [G 5.4] for rules about assigning and reporting Apdex quality ratings.

In Apdex-R, when the value of F is set using a default rule (see [R 2.5.1]), the target threshold T may be shown as a subscript, as illustrated in Table 3. The table shows examples where T is 4 seconds.

Some tool creators may wish to add graphical aids to report the Apdex value. This is an optional feature, but, if implemented, it must follow these guidelines.

Two forms of alternative representations are permitted: the rating (a word), and a color indication. The following table shows the fixed set of alternative modes of representing the Apdex.

The color indication can be determined by the creator in line with their existing product set, however a legend must clearly indicate which color represents each Apdex rating.

Table 3 – Apdex Qualitative Reporting Rules
(Examples where T=4)

Apdex Value Range Rating Color Indication
0.944 to 1.004 Excellent4 Determined by creator (with a 4 plus a color indication)
0.854 to 0.934 Good4 Determined by creator (with a 4 plus a color indication)
0.704 to 0.844 Fair4 Determined by creator (with a 4 plus a color indication)
0.504 to 0.694 Poor4 Determined by creator (with a 4 plus a color indication)
0.004 to 0.494 Unacceptable4 or UNAX4 Determined by creator (with a 4 plus a color indication)
Low Sample Cases
NS4 NoSample4 Determined by creator (with a 4 plus an NS inside color indication)
0.854* Good4* Determined by creator (with a 4 plus an * inside color indication)
Table 3. Apdex Qualitative Reporting Rules

Note: In the current specification, the ‘Apdex Value Range’ column of Table 3 contains a typo in the ‘NoSample’ row, which reads ‘0.NS4‘. That cell should read ‘NS4‘ (as shown above).

[G 5.4] Apdex Quality RatingsSome tool creators may wish to assign quality ratings to Apdex value ranges, and to present those ratings graphically. This is an optional feature, but, if implemented, it must follow these guidelines.

Two alternative representations are permitted for quality ratings: a rating word or a color indication. Table 6 below lists the value ranges to be used when assigning a rating to an Apdex value. The table shows examples for the target threshold [T], where [T] is the normal threshold display as described in section [G 5.1.1].

Colors may be selected by the creator for consistency with other products, or based on user-supplied preferences. However a legend must clearly indicate which color represents each Apdex rating.

Apdex Value Range Rating Word Color Indication
0.94 to 1.00 [T] Excellent [T] Determined by creator (with [T] plus a color indication)
0.85 to 0.93 [T] Good [T] Determined by creator (with [T] plus a color indication)
0.70 to 0.84 [T] Fair [T] Determined by creator (with [T] plus a color indication)
0.50 to 0.69 [T] Poor [T] Determined by creator (with [T] plus a color indication)
0.00 to 0.49 [T] Unacceptable [T] or UNAX [T] Determined by creator (with [T] plus a color indication)
Low Sample Cases
NS [T] NoSample [T] Determined by creator (with [T] plus an NS inside color indication)
0.85* [T] Good* [T] Determined by creator (with [T] plus an * inside color indication)
Table 6. Rules for Apdex Quality Ratings

Note: An addendum may specify alternative colors [TBD: and/or value ranges?] to be applied when reporting Apdex scores for a particular measurement domain.

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  


*