Apdex is dead simple. It is a standard way to convert many response time measurements into a single numerical value that always stays within the range of 0 to 1 where 0 is a disaster and 1 is perfect performance delivery. Dozens of vendors measure response time, so you would think you could readily get an Apdex report. Alas it is not so.
Last December, the Apdex Alliance asked vendors to submit the names of their products that generate Apdex reports to the Apdex Tools Directory. The directory has entries from the following six vendors: Compuware, Gomez, ip-label.newtest (formerly Auditec), WildPackets, and Xa Systems. However, when you search for information about Apdex at their websites, you get responses like, “No Pages were found containing Apdex”, or “Forbidden: Your client does not have permission to get URL.” WildPackets is the only vendor that has any information about their Apdex features. The WildPackets search on Apdex yields 51 documents with descriptions, FAQs, and tips.
So if you are interested in Apdex, you had better be prepared to roll your own reports. The good news is that you can easily create Apdex reports in a spreadsheet. We also find it useful to use a simple database like Access to organize your measurements. An Access query can pull out the measurements associated with the report you are creating, like all California measurements for yesterday. You can program the Apdex formula directly into queries and show them in a variety of reports.
Regardless of your report generation method, you will need input data. This turns out to be the tricky part. Many measurement tools do not make it easy for you to export measurement samples in a useful format. Most vendors want you to stay within the confines of their product. In fact, they help you import data from other sources but once there, they really don’t want you to export.
The same Apdex Tools Directory lists a dozen vendors that supply data which can be used to generate Apex reports. The most valuable part of the directory is the explanation of how to export the data and what kind of format the data will be when exported. This gives you a guide to match the measurement tool and your report generation tool.
Measurement vendors should make all of this much simpler. For example, they can add a generic report design capability where the user can define what data they want to see and how they want it processed (like the Apdex formula), and then how the report should look. If vendors continue to try and force their tool users of to see only reports that they designed then they must provide a wide range of reports including Apdex.
We are seeing an interesting pattern among enterprises that have created the following solution to the problem. They generate production (ongoing) Apdex reports using their own software using input data from various vendors. They use their own data format and conventions as a way to “normalize” the measurements from the various vendors. This then becomes the common data set which a variety of reports can use.
Enterprises that are serious about Apdex describe three benefits to this approach. First, they do not have to sort out differences in how the vendors generated their unique reports. They simply don’t use the vendor reports. Second, they can integrate data feeds from a variety of vendors into a single report. This would be nearly impossible with any of the vendors that have designed their product thinking they are the only measurement and reporting tool at an enterprise. Their reaction is, “You mean you use my tool and another company’s tool?” Finally, they can now add and remove measurement vendors as needed without changing their reporting system. They call this the “vendor isolation” feature.
If you are a measurement or management vendor you should be worried. This approach reduces your value to the enterprise to commodity status. Vendor isolation also means easy vendor replacement.
We think vendors have two choices. Increase value to your customers with better and more flexible reporting capabilities so your customer can build his own Apdex reports–or slide down the slippery slope to a commodity low-price data feed role. Oh, yes, there is a third choice: add Apdex reports directly into your product.
If you are a vendor that generates Apdex reports or can supply measurement data into a customer’s Apdex reporting engine, we want to know about it. Please post a comment to this blog or send an email to firstname.lastname@example.org.