LinkedIn

See our LinkedIn profile, click on this button:

APDEX

Configurable Reporting in Apdex-G

I’m writing a series of posts about Generalizing Apdex. This is #12. To minimize confusion, section numbers in the current spec are accompanied by the section symbol, like this: §1. The corresponding section numbers in the generalized spec, Apdex-G, are enclosed in square brackets, like this: [1].

In my previous post in this series, I introduced a conceptual model for Apdex-G. Compared to the current Apdex model, the key addition is the Performance Interval, a range of values within the measurement domain that shares a common Satisfaction Level of ‘Satisfied’, ‘Tolerating’, or ‘Frustrated’. Thresholds now form the boundaries of performance intervals, rather than Performance Zones–which are now defined as a collection (union) of one or more performance intervals. For the full story, see Generalizing the Apdex Thresholds.

This generalization allows Apdex-G to accommodate arbitrary orderings of satisfaction levels within the measurement domain, including discontinuous performance zones. But it also complicates Apdex reporting. The current Apdex model, which employs just two thresholds, T and F, makes it easy to report the threshold(s) alongside the value of an Apdex score. And doing so is a mandatory requirement of Apdex:

All Apdex values are calculated with a particular target threshold, T. The value of T must be clearly displayed in association with the Apdex score.

— Apdex spec, section §5 Reporting the Index, introduction

How can we retain this essential Apdex feature in Apdex-G, which must accommodate reporting on arbitrary alignments of performance intervals?

Holding to Einstein’s Razor

The short answer is, of course, that we can’t make all Apdex-G scores as simple to display and read as the current Apdex metric. In the immortal words of Albert Einstein, we should “make everything as simple as possible, but no simpler”. Interestingly, those words are just a shortened version of what Einstein actually said, which turns out to be particularly relevant in this context:

It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.
–“On the Method of Theoretical Physics” The Herbert Spencer Lecture, delivered at Oxford on June 10, 1933

This principle is sometimes called Einstein’s razor by people who view it as a counterpoint to Occam’s razor, namely “entities must not be multiplied beyond necessity”. Both principles can be applied to Apdex-G. First, I believe I have followed Occam’s razor when extending Apdex. Apdex-G does introduce a new entity–a performance interval–but that is necessary to accommodate arbitrary alignments of satisfaction levels. Second, we must follow Einstein’s razor to ensure that when reporting an Apdex-G score, we do not “surrender the adequate representation” of the configuration of those performance intervals.

Introducing Interval Notation

Clearly, Apdex-G must define a standard notation for describing any perfomance interval. This notation must show the lower and upper thresholds that define the interval, and whether or not they are contained within the interval. For example, in the current Apdex spec, we could describe the tolerating zone (Zt say), using the following inequality:

T < Zt ≤ F

where in this example the symbols T and F signify threshold values. The choice of inequalities, < and ≤, show that T, the lower threshold, is not contained within the zone, while F, the upper threshold, is included.

Interval notation ZONAW1, SOSMW1, WIKIW1 is an alternative way convey the same information without using inequality symbols; Roberta Bloom has created a handy one-page summary BLOO07. For the Apdex tolerating zone we would write:

Zt: (T, F]

The round bracket (or parenthesis) signifies exclusion from the interval, the square bracket signifies inclusion. So we could describe the Apdex index like this:

Apdex: [0.00, 1.00]

To describe a tolerating zone composed of two intervals (for example, Metric Type #7, Symmetric Process Control in Table 4 of my previous post), we would write something like this:

Zt: (t1, t2] ∪ (t3, t4]

where the ti (i=1..4) are threshold values, and the mathematical “cup” symbol ∪ signifies the union of the two intervals. The full description of Metric Type #7 would be:

Zs: (t2, t3]
Zt: (t1, t2] ∪ (t3, t4]
Zf: (−∞, t1] ∪ (t4, ∞)

This choice of round and square brackets in this example follow the Apdex-G rule that “Each Threshold lies within one and only one Performance Interval. Unless otherwise specified by a domain-specific Addendum, a Threshold lies within the Performance Interval for which it acts as the upper bound”. The final bracket is round by convention; in this notation, infinity symbols are always accompanied by round brackets.

I propose that we adopt interval notation to describe performance intervals and performance zones within the Apdex-G Uniform Output spec. Granted, the 3-line example given above is a lot more cumbersome than the current Apdex requirement of a simple suffix displaying a single T value. But note that:

  • the example illustrates how to report a more complex metric type that employs five performance intervals
  • an addendum (like Apdex-R) can specify simpler alternative notation (like the current subscript method) if the metric type warrants it
  • in addition to requiring that all tools support the Uniform Output specification, the spec allows tools to use other ways to display Apdex scores

These matters are addressed in Section §5 of the Apdex spec, which is shown in the left-hand column in the following paragraphs. On the right are my proposals for the corresponding portions of Apdex-G.

General Reporting Rules

I have generalized these rules, tightened up the language a little, and removed the last sentence. It has been incorporated into the introduction to section [5.1.1].

Current spec:
§5. Reporting the Index

The index is a decimal value between 0 and 1. The Apdex representation on all reports (screen, print, or otherwise) must adhere to the following rules.

§5.1 Displaying the Apdex Value

The index always starts with a 0, followed by a decimal point, followed by the fractional value for the calculation to two decimals, and the value of T is clearly associated with the index presentation as defined below. There is a special case of the value 1.00 that starts with the digit of 1. A 1.00 is presented when the formula produces a result that can be rounded arithmetically to 1.00 (results equal to or greater than 0.995 round to 1.00).

Apdex may not be reported in granularity smaller than one one-hundredth. Decimal values of 3 or more digits are not permitted.

Tools in a computer screen or in printed reports always identify it as an Apdex value. When there are many values of the index presented in tabular form representing many reporting groups, then the Apdex label should appear with the appropriate column or row.

All Apdex values are calculated with a particular target threshold, T. The value of T must be clearly displayed in association with the Apdex score. Furthermore, in order to facilitate exporting Apdex values from a tool and then importing them to other analysis tools, all tools must support at least one uniform output as shown below.

First draft:
[5] Reporting the Index

In this section, the terms Report(s) and Reporting refer to any representation of an Apdex index, whether in print, on a computer screen, or elsewhere.

[5.1] Reporting the Apdex Value

Reports must adhere to the following rules:

  1. The index is a decimal value between 0 and 1, rounded to a precision of two decimal places. Values equal to or greater than 0.995 round to 1.00.
  2. Unless its value is 1.00, the index is always reported with a zero in the tens place, followed by a decimal point, followed by no more than two decimal places.
  3. Apdex values must be identified as such. When several Apdex values are presented in tabular form, a single Apdex identifier can identify a group of values in a row, column, or table.
  4. All Apdex values are calculated based on specified performance zones. The performance zone specification must be clearly displayed in association with the Apdex score.

[5.1.1] Uniform Output

First some housekeeping: In Apdex-G, I have numbered the principal subsections, and removed the optional subscripted format for output. This format is not applicable in Apdex-G, and will be reviewed later for inclusion in Apdex-R.

To support Apdex-G’s more complex performance zones, comprising an unknown number of performance intervals, we have to abandon the fixed format specification for Uniform Output provided in the current spec. I propose that we replace it with a more flexible comma-separated values (CSV) file. For compatibility with existing Apdex implementations, we can retain the current fixed format as an option in the Apdex-R spec.

There is no universally accepted standard specification for CSV files, but to provide a degree of rigor in the Apdex-G specification, we can incorporate the public domain specification for CSV files published as RFC 4180 RFC4180.

Section [5.1.2] is a placeholder for a new proposal which would permit (or maybe require) tools to supply additional Report Group identifiers in Uniform Output files.

Current spec:
Uniform Output (Mandatory)

The tool shall display, print, and export to an ASCII file each Apdex value with the following fixed format.

Table 2. Uniform Output Definition

Element Position Definition Example/Range
1 Value whole number digit 0 or 1
2 Decimal point .
3 Value tenths digit 0 through 9
4 Value hundredths digit 0 through 9
5 Space
6 Left bracket designating start of T [
7 First digit of T 0.1 or greater
N Rest of T as defined in section 3.3 Numeric digits
N+1 Right bracket designating end of T ]
N+2 Small group indicator (see 5.2 below) * if present
Table 2. Uniform Output Definition

Examples of the uniform output are: 0.85 [5.5], 1.00 [8.0]*, 0.90 [4.0], and 0.77 [450].

Subscripted Output (Optional)

Tools may display Apdex values with a T shown as a mathematical subscript. For example, an Apdex value of 0.75 that is based upon a T of 4 seconds is shown as 0.754.

When T has a decimal component, as defined in section 3.3, then the exact value of T must be shown (example: 0.754.5). In order to make the index easy to read, tools may drop the decimal portion of T if it is zero (example: 0.754.0 becomes 0.754).

First draft:
[5.1.1] Uniform Output File (Mandatory)

To support the exchange of index values between Apdex analysis and reporting tools, all Apdex analysis tools must support the Apdex Uniform Output format. A tool shall display, print, and export Apdex-G values to an ASCII file having a comma-separated values (CSV) format. For this purpose, Apdex-G incorporates the IETF specification of CSV published in RFC 4180 (see section [6] References).

A Uniform Output File is composed of a uniform output header record, followed by one or more uniform output data records. The content of a uniform output data record is described in Tables 2, 3, and 4 below.

Element Number Definition Type Content
1 Apdex metric identifier Literal Apdex
2 Apdex Index Number Decimal in range [0.00, 1.00]
3 Satisfied Zone Identifier Literal S
4 Satisfied Thresholds(s) Group Interval Group, see Table 3
5 Tolerating Zone Identifier Literal T
6 Tolerating Thresholds(s) Group Interval Group, see Table 3
7 Frustrated Zone Identifier Literal F
8 Frustrated Thresholds(s) Group Interval Group, see Table 3
Table 2. Layout of a Uniform Output Data Record

In Table 2, each element except the last is followed by a comma. In addition, each Interval Group (elements 4, 6, and 8 in Table 2) is composed of one or more Performance Interval elements, which are also separated by commas. Table 3 shows the components of a single Performance Interval element. The components shown in Table 3 are not separated by commas.

Component Definition Type Content
1 Opening backet Literal ( or [
2 Lower threshold Number Decimal number — see [3.3]
3 Comma Literal ,
4 Upper threshold Number Decimal number — see [3.3]
5 Closing backet Literal ) or ]
Table 3. Components of a Performance Interval within a Uniform Output Data Record

Uniform Output Header Record

In an output file having the uniform output format, the first line is a header record that contains the same number of comma-separated fields as all data records in the file. Fields in the header record contain fixed literal values, or names that correspond to each field in the data records, as specified in Table 4.

Element Number Definition Type Content
1 Apdex header identifier Literal Apdex Header
2 Apdex Index Literal Apdex Index
3 Satisfied Zone Identifier Literal S
4 Performance Intervals Group Performance Interval Names
5 Tolerating Zone Identifier Literal T
6 Performance Intervals Group Performance Interval Names
7 Frustrated Zone Identifier Literal F
8 Performance Intervals Group Performance Interval Names
Table 4. Layout of a Uniform Output Header Record

Performance Interval Names: If a tool has obtained user-specified names for performance intervals, those names should be included in the corresponding fields of the header record (elements 4, 6, and 8 in Table 4). If user-specified names are not available, distinct default names must be supplied for each performance interval field in the header record. See also section [2.5.1] Thresholds and Performance Intervals.

Examples of the uniform output

Examples of the uniform output are:

Apdex Header,Apdex Index,S,Green,T,Yellow,F,Red CRLF user labels
Apdex,0.72,S,[0,4],T,(4,16],F,(16,INF) CRLF
Apdex,0.85,S,[0,6],T,(6,20],F,(20,INF) CRLF
Apdex,0.58,S,[0,3],T,(3,10],F,(10,INF) CRLF
Apdex,0.91,S,[0,10],T,(10,30],F,(30,INF) CRLF

Apdex Header,Apdex Index,S,PI3,T,PI2,PI4,F,PI1,PI5 CRLF generic labels
Apdex,0.88,S,(10,12],T,(6,10],(12,16],F,[0,6],(16,INF) CRLF
Apdex,0.96,S,(9,15],T,(6,9],(15,18],F,[0,6],(18,INF) CRLF

Proposal for future discussion:
[5.1.2] Report Group Section

More on Section [5] Reporting …

I debated whether to call this post configurable reporting, because the configuration options that most affect reporting are the selection of Apdex-G performance intervals and thresholds. But the rules for reporting do have to accommodate those options, which in turn forces us to introduce more flexibility throughout the reporting specifications. And we have more decisions to consider. This post has addressed the first half of the reporting area; in a further post, I will draft the remaining paragraphs of Apdex-G section [5]:

    [5.1.2] Report Group Section The current spec requires tools to report Apdex scores in Uniform Output format, but does not require any identifiers or context to be reported with those scores. This strikes me as an omission. I will explore the question of what data should be made available to an independent reporting tool that queries an Apdex-based data analysis service, and discuss the relationship of those sections to section [3.2] Report Groups.

    [5.2] Sample Size We must consider whether we need any configuration options in the rules for Apdex sample sizes.

    [5.3] Describing General Cases This section of the current spec can be retained in Apdex-G with minor modifications.

    [5.4] Quality Ratings We must consider whether we need any configuration options for the display of Apdex ratings or grades.

As usual, all these proposals are open for public discussion. Please use the comment form below to contribute any comments, suggestions, or questions.

References

BLOO07
Interval Notation Roberta Bloom, 2007. [pdf]
RFC4180
Common Format and MIME Type for Comma-Separated Values (CSV) Files Y. Shafranovich, 2005. [RFC 4180, The Internet Society, October 2005]
SOSMW1
The Real Number Line, Interval Notation and Set Notation Helmut Knaust, S.O.S. MATHematics [S.O.S. MATHematics Website]
WIKIW1
Interval (mathematics) [Wikipedia]
ZONAW1
Interval Notation Zonaland Education. [Zonaland Website]

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  

  

  


*