I’m writing a series of posts about Generalizing Apdex. This is #24. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec documents, Apdex-G and Apdex-R, are enclosed in square brackets and prefixed by the document type, like this: [G 1] and [R 1].
Drafts of the Apdex-G spec were posted as Section [1] Introduction, Section [2] Index Overview, Section [3] Calculation Inputs, Section [4] Calculating the Index, Section [5] Reporting, and Section [6] References.
The first draft of Apdex-R began with Sections 1 and 2, Section 3, and Section 4. This post covers Section 5.
Apdex-R specifies the Apdex rules for reporting response-time measurements, therefore my first draft draws heavily on the present Apdex spec. My goal is to reformat the present spec’s rules within the context established by Apdex-G. To allow both documents to be viewed together, I show Apdex-G (draft #2, plus a few subsequent amendments) on the left, and Apdex-R (draft #1) on the right.
Differences between Apdex-R and the corresponding sections of the present spec are highlighted, as follows:
text deleted from the present specThis text is covered in Apdex-G, or elsewhere in Apdex-R, or extraneous.- new text added to Apdex-R (plus a few additions to Apdex-G draft #2)
- new text created for Apdex-G, now being repeated in Apdex-R for readability
- notes, questions, issues to be resolved
While major deletions, additions, and changes to the present spec are marked, many minor wording changes to the present spec’s text, to improve continuity and clarity of meaning, are not marked. I believe those changes do not substantially alter the meaning and intent of the present spec.
Apdex-R, first draft:
Apdex-G, second draft:
[R 5.1] Reporting the Apdex ValueSee [G 5.1] for general rules about reporting the Apdex value. Apdex-R adds the following rule for reporting performance zones:
All Apdex-R values are calculated using two thresholds, T and F, which must be reported in association with the corresponding Apdex score for a report group. The value of T must always be reported. If the specification of the report group applied a default rule for the value of F (see [R 2.5.1]), the value of F may be omitted.
The index always starts with a 0, followed by a decimal point, followed by the fractional value for the calculation to two decimals, and the value of T is clearly associated with the index presentation as defined below. There is a special case of the value 1.00 that starts with the digit of 1. A 1.00 is presented when the formula produces a result that can be rounded arithmetically to 1.00 (results equal to or greater than 0.995 round to 1.00).
Apdex may not be reported in granularity smaller than one one-hundredth. Decimal values of 3 or more digits are not permitted.
Tools in a computer screen or in printed reports always identify it as an Apdex value. When there are many values of the index presented in tabular form representing many reporting groups, then the Apdex label should appear with the appropriate column or row.
All Apdex values are calculated with a particular target threshold, T. The value of T must be clearly displayed in association with the Apdex score. Furthermore, in order to facilitate exporting Apdex values from a tool and then importing them to other analysis tools, all tools must support at least one uniform output as shown below.
[G 5.1] Reporting the Apdex ValueReports must adhere to the following rules:
- The index is a decimal value between 0 and 1, rounded to a precision of two decimal places. Values equal to or greater than 0.995 round to 1.00.
- Unless its value is 1.00, the index is always reported with a zero in the tens place, followed by a decimal point, followed by no more than two decimal places.
- Apdex values must be identified as such. When several Apdex values are presented in tabular form, a single Apdex identifier can be identify a group of values in a row, column, or table.
- All Apdex values are calculated based on specified performance zones. The performance zone specification must be clearly displayed in association with the Apdex score.
[R 5.1.1] Describing General CasesSee [G 5.1.1] for rules about describing unspecified Apdex thresholds.
General Apdex value discussions that reflect any target of T are written with “T” as shown in the following examples.
Uniform Output: “Everyone should understand that 0.90 [T] is a better value than 0.80 [T].”
Subscripted Output: “Everyone should understand that 0.90T is a better value than 0.80T.”
[G 5.1.1] Describing General CasesIn general discussions of Apdex values, references to unspecified performance thresholds are written using the notation “[T]”, as shown in the following examples.
“Everyone should understand that 0.90 [T] is a better value than 0.80 [T].”
“Apdex scores in the range 0.85 to 0.93 [T] are rated Good.”
Sections [G 5.3] and [G 5.4] also use this notation when referring to threshold reporting.
[R 5.2] Uniform Output File (Mandatory)To support the exchange of index values between Apdex analysis and reporting tools, all Apdex analysis tools must support the Apdex Uniform Output format specified in [G 5.2]. In Apdex-R, the following additional rules apply:
[R 5.2.1] Bracketed Output Format (Mandatory)
Tools implementing Apdex-R must be capable of reporting Apdex-R values using the bracketed format described in Table 2. When the value of F is set using a default rule (see [R 2.5.1]) elements 5 and 6 in Table 2 may be omitted, so that only the value of the threshold T accompanies the reported Apdex value.
| Element Number | Definition | Type | Content |
|---|---|---|---|
| 1 | Apdex index value | Number | Decimal in range [0.00, 1.00] |
| 2 | Space | Literal | |
| 3 | Left bracket designating start of threshold(s) | Literal | [ |
| 4 | Tolerating threshold, T | Number | Decimal |
| 5 | Comma | Literal | , |
| 6 | Frustrated threshold, F | Number | Decimal |
| 7 | Right bracket designating end of threshold(s) | Literal | ] |
| 8 | Small Group Indicator | Literal | * or NS |
Examples of bracketed output format are: 0.85 [5.5], 1.00 [8.0]*, 0.90 [4.0], and 0.77 [450].
[R 5.2.2] Subscripted Output Format (Optional)
When the value of F is set using a default rule (see [R 2.5.1]), tools may display Apdex values in which the value of T is shown as a mathematical subscript. For example, an Apdex value of 0.75 that is based upon a T value of 4 seconds is shown as 0.754.
When T has a decimal component, as defined in section [G 3.4], then the exact value of T must be shown (example: 0.754.5). To increase readability, tools may drop the decimal portion of T if it is zero (example: 0.754.0 becomes 0.754).
[G 5.2] Uniform Output File (Mandatory)Separate tools may be used to calculate Apdex index values from measurement data and to report Apdex index values. To support the exchange of index values between Apdex analysis and reporting tools, all Apdex analysis tools must support the Apdex Uniform Output format. A tool shall display, print, and export Apdex-G values to an ASCII file having a comma-separated values (CSV) format. For this purpose, Apdex-G incorporates the IETF specification of CSV published in RFC 4180 (see section [G 6] References).
A Uniform Output File is composed of a uniform output header record, followed by one or more uniform output data records. The header record contains the same number of comma-separated fields as all data records in the file. Fields in the header record contain fixed literal values, or names that correspond to each field in the data records. The content of a uniform output header and data records are described in Tables 2 and 3 below. In Table 2, each element except the last is followed by a comma.
| Element Number | Definition | Header Record | Data Record | ||
|---|---|---|---|---|---|
| Type | Content | Type | Content | ||
| 1 | Apdex data record identifier | Literal | Apdex Header | Literal | Apdex |
| Report Group Metadata Section | Header Record | Data Record | |||
| 2 | Apdex Report Group section identifier | Literal | ARG | Literal | ARG |
| 3 | Report Group Name | Literal | Report Group | Name | Defined by tool or user |
| 4 | Report Group Description | Literal | Description | Text String | Defined by tool or user. See note 1 below |
| 5 | Measurement Type | Literal | Type | Name | G, or addendum type (e.g. R for Apdex-R) |
| 6 | Measurement Subtype | Literal | Subtype | Name | Values specified in an addendum for this measurement type |
| 7 | Application | Literal | Application | Name | Defined by tool or user |
| 8 | User Group | Literal | User Group | Name | Defined by tool or user |
| 9 | Time Period Start | Literal | Start Time | Timestamp | ISO 8601: [YYYY][MM][DD]T[hh][mm][ss]Z |
| 10 | Time Period End | Literal | End Time | Timestamp | ISO 8601: [YYYY][MM][DD]T[hh][mm][ss]Z |
| Input Data Summary Section | Header Record | Data Record | |||
| 11 | Apdex Data Summary section identifier | Literal | ADS | Literal | ADS |
| 12 | Sample Count | Literal | Total Samples | Number | Integer |
| 13 | Satisfied Zone Count | Literal | Satisfied Count | Number | Integer |
| 14 | Tolerating Zone Count | Literal | Tolerating Count | Number | Integer |
| 15 | Frustrated Zone Count | Literal | Frustrated Count | Number | Integer |
| 16 | Earliest Sample Timestamp | Literal | First Sample | Timestamp | ISO 8601: [YYYY][MM][DD]T[hh][mm][ss]Z |
| 17 | Latest Sample Timestamp | Literal | Last Sample | Timestamp | ISO 8601: [YYYY][MM][DD]T[hh][mm][ss]Z |
| Apdex Index Section | Header Record | Data Record | |||
| 18 | Apdex IndeX section identifier | Literal | AIX | Literal | AIX |
| 19 | Apdex index value | Literal | Apdex Index | Number | Decimal in range [0.00, 1.00] |
| 20 | Satisfied Zone Identifier | Literal | S | Literal | S |
| 21 | Satisfied Thresholds(s) | Group | Performance Interval Names | Group | Interval Group, see Table 3 |
| 22 | Tolerating Zone Identifier | Literal | T | Literal | T |
| 23 | Tolerating Thresholds(s) | Group | Performance Interval Names | Group | Interval Group, see Table 3 |
| 24 | Frustrated Zone Identifier | Literal | F | Literal | F |
| 25 | Frustrated Thresholds(s) | Group | Performance Interval Names | Group | Interval Group, see Table 3 |
| 26 | Small Group Indicator | Literal | SGI | Literal | * or NS |
Note 1: To avoid ambiguity, tool creators must ensure that Report Group description fields do not contain commas, because commas separate fields in the Uniform Output format.
Performance Interval Names
If a tool has obtained user-specified names for performance intervals, those names should be included in the corresponding fields of the header record (elements 21, 23, and 25 in Table 2). If user-specified names are not available, distinct default names must be supplied for each performance interval field in the header record. See also section [G 2.5.1] Thresholds and Performance Intervals.
Performance Interval Groups
In Table 2, each Interval Group (elements 21, 23, and 25) is composed of one or more Performance Interval elements, separated by commas. Table 3 shows the structure of a single Performance Interval element, therefore the components shown in Table 3 are not separated by commas.
| Component | Definition | Header Record | Data Record | ||
|---|---|---|---|---|---|
| Type | Content | Type | Content | ||
| 1 | Opening backet | Name | Performance Interval Name | Literal | ( or [ |
| 2 | Lower threshold | Number | Decimal number — see [G 3.4] | ||
| 3 | Colon | Literal | : See note 2 below | ||
| 4 | Upper threshold | Number | Decimal number — see [G 3.4] | ||
| 5 | Closing backet | Literal | ) or ] | ||
Note 2: Interval Notation, described in References [G 6.8], uses a comma to separate the upper and lower bounds of an interval. To avoid ambiguity Apdex substitutes a colon, because commas separate fields in the Uniform Output format.
Optional Sections
The Uniform Output File format comprises three sections, each having its own identifier (fields 2, 11, and 18). The Report Group Metadata Section and the Apdex Index Section are required, because they contain the necessary data for reporting Apdex scores. The Input Data Summary Section is optional, but tool creators are encouraged to produce this data at the analysis stage, and to pass it to Reporting tools. Reporting tools can use this data to provide additional context for Apdex scores. If the Input Data Summary Section is omitted, fields 11 through 17 must be omitted from the header and data records.
Examples of the uniform output
Examples of the uniform output are:
Apdex Header,ARG,,,,,,,,,AIX,Apdex Index,S,Green,T,Yellow,F,Red,SGI CRLF user labels
Apdex,ARG,,,,,,,,,AIX,0.72,S,[0:4],T,(4:16],F,(16:INF) CRLF
Apdex,ARG,,,,,,,,,AIX,0.85,S,[0:6],T,(6:20],F,(20:INF),* CRLF
Apdex,ARG,,,,,,,,,AIX,0.58,S,[0:3],T,(3:10],F,(10:INF) CRLF
Apdex,ARG,,,,,,,,,AIX,0.91,S,[0:10],T,(10:30],F,(30:INF) CRLF
Apdex Header,ARG,,,,,,,,,AIX,Apdex Index,S,PI3,T,PI2,PI4,F,PI1,PI5,SGI CRLF generic labels
Apdex,ARG,,,,,,,,,AIX,0.88,S,(10:12],T,(6:10],(12:16],F,[0:6],(16:INF) CRLF
Apdex,ARG,,,,,,,,,AIX,0.96,S,(9:15],T,(6:9],(15:18],F,[0:6],(18:INF) CRLF
[ More details and examples will be added here ]
[R 5.3] Indicating Sample SizeSee [G 5.3] for rules about reporting the sample size of an Apdex report group.
Apdex values are calculated based upon a set of measurements (samples) in the report group. If there are a small number of samples, the tool must still present a result. However, a result for such a small report group must be clearly marked.
A small report group is defined as any number of samples between 0 and 99. Apdex tools will clearly indicate that the result is based upon one of the following scenarios:
No SamplesThe Apdex calculation could not be performed because there were no samples (NS) within the report group. Where the calculated Apdex value would normally appear, the tool will show an output of NS. Examples: NS [4.0], NS4Small GroupWhen an Apdex value is the output of a small group (1 to 99) calculation, an asterisk (*) must be appended to that value. Examples: 0.80 [4.0]*, 0.804*.
[G 5.3] Indicating Sample SizeApdex values are calculated based upon a set of measurements (samples) in the report group. If there are a small number of samples, the tool must still present a result. However, a result for such a small report group must be clearly marked.
A small report group is defined as one having fewer than 100 samples. An addendum may modify this definition to be appropriate for a particular measurement domain. Apdex tools will clearly indicate that the result is based upon one of the following scenarios:
- No Samples
- The Apdex calculation could not be performed because there were no samples (NS) within the report group. Where the calculated Apdex value would normally appear, the tool will show an output of NS [T], where [T] is the normal threshold display (see section [G 5.1.1]).
- Small Group
- When an Apdex value is the output of a small group calculation, an asterisk (*) must be appended to that value, for example: 0.84* [T], where [T] is the normal threshold display (see section [G 5.1.1]).
[R 5.4] Apdex Quality RatingsSee [G 5.4] for rules about assigning and reporting Apdex quality ratings.
In Apdex-R, when the value of F is set using a default rule (see [R 2.5.1]), the target threshold T may be shown as a subscript, as illustrated in Table 3. The table shows examples where T is 4 seconds.
Some tool creators may wish to add graphical aids to report the Apdex value. This is an optional feature, but, if implemented, it must follow these guidelines.
Two forms of alternative representations are permitted: the rating (a word), and a color indication. The following table shows the fixed set of alternative modes of representing the Apdex.
The color indication can be determined by the creator in line with their existing product set, however a legend must clearly indicate which color represents each Apdex rating.
Table 3 – Apdex Qualitative Reporting Rules
(Examples where T=4)
| Apdex Value Range | Rating | Color Indication |
|---|---|---|
| 0.944 to 1.004 | Excellent4 | Determined by creator (with a 4 plus a color indication) |
| 0.854 to 0.934 | Good4 | Determined by creator (with a 4 plus a color indication) |
| 0.704 to 0.844 | Fair4 | Determined by creator (with a 4 plus a color indication) |
| 0.504 to 0.694 | Poor4 | Determined by creator (with a 4 plus a color indication) |
| 0.004 to 0.494 | Unacceptable4 or UNAX4 | Determined by creator (with a 4 plus a color indication) |
| Low Sample Cases | ||
| NS4 | NoSample4 | Determined by creator (with a 4 plus an NS inside color indication) |
| 0.854* | Good4* | Determined by creator (with a 4 plus an * inside color indication) |
Note: In the current specification, the ‘Apdex Value Range’ column of Table 3 contains a typo in the ‘NoSample’ row, which reads ‘0.NS4‘. That cell should read ‘NS4‘ (as shown above).
[G 5.4] Apdex Quality RatingsSome tool creators may wish to assign quality ratings to Apdex value ranges, and to present those ratings graphically. This is an optional feature, but, if implemented, it must follow these guidelines.
Two alternative representations are permitted for quality ratings: a rating word or a color indication. Table 6 below lists the value ranges to be used when assigning a rating to an Apdex value. The table shows examples for the target threshold [T], where [T] is the normal threshold display as described in section [G 5.1.1].
Colors may be selected by the creator for consistency with other products, or based on user-supplied preferences. However a legend must clearly indicate which color represents each Apdex rating.
| Apdex Value Range | Rating Word | Color Indication |
|---|---|---|
| 0.94 to 1.00 [T] | Excellent [T] | Determined by creator (with [T] plus a color indication) |
| 0.85 to 0.93 [T] | Good [T] | Determined by creator (with [T] plus a color indication) |
| 0.70 to 0.84 [T] | Fair [T] | Determined by creator (with [T] plus a color indication) |
| 0.50 to 0.69 [T] | Poor [T] | Determined by creator (with [T] plus a color indication) |
| 0.00 to 0.49 [T] | Unacceptable [T] or UNAX [T] | Determined by creator (with [T] plus a color indication) |
| Low Sample Cases | ||
| NS [T] | NoSample [T] | Determined by creator (with [T] plus an NS inside color indication) |
| 0.85* [T] | Good* [T] | Determined by creator (with [T] plus an * inside color indication) |
Note: An addendum may specify alternative colors [TBD: and/or value ranges?] to be applied when reporting Apdex scores for a particular measurement domain.
Leave a Reply