LinkedIn

See our LinkedIn profile, click on this button:

APDEX

Many Uses of Apdex

Apdex is a simple formula that converts many performance values into an easy to understand 0-to-1 performance index. Defining the target application response time T and accurately interpreting the result (the score) requires some methodology–and that methodology should reflect how you plan to use Apdex. There are three ways to use Apdex: to do tactical diagnostics, to support process, and to link performance to your business. These three uses govern how you make Apdex parameter choices.

Tactical Diagnosis
This is the simplest use of Apdex.  The Apdex parameters are completely under your control without requiring agreement from any other group.  It allows you to experiment with T values to see the sensitivity of your applications to the Apdex formula.  Once you have a T value set you can use the Apdex scores to sort many measurements to determine which applications need attention.  This is an alternative to alarm triggers that are often too sensitive and provide false alarms.

When used for diagnostics the Apdex T usually floats as needed to get the job done within a specific investigation.  The Apdex scores provide first level performance problem diagnostics.  Of course the tool that supplies the real-time data must have drill-down capabilities to continue on the diagnostic path. 

Process Support
Sophisticated performance management requires process.  The four fundamental ITIL-based processes that apply to application performance are: incident management, availability management, capacity management, and finally service assurance.  When applied to these processes Apdex takes on management properties that require you to involve other groups in your organization so Apdex scores will have meaning across groups.

When supporting incident management, the Apdex T and the “trigger” score should be meaningful (show actual incidents), yet not too aggressive (which creates false positives).  This said, we find that organizations tend to run Apdex parameters “hot” so Apdex scores vary a lot over a day.

By the time you progress to performance assurance, Apdex is incorporated in executive reporting and/or possibly is the foundation for SLAs.  For this use the Apdex T and “acceptable” score should be more relaxed.  Apdex scores should not swing wildly over the course of a day if in fact performance was consistent in the view of the users.

Think of it as similar to tracking human health.  Doctors in a hospital emergency room want monitoring instruments to be sensitive to the slightest patient changes.  In contrast, doctors at the U.S. government’s Centers for Disease Control (CDC) need to assess the health of the entire US population, so their parameters must be set to discover trends.  If parameters are too “hot” the results become very “noisy” and it is very hard to see long term trends.

Business/Performance Linkage
The most advanced use of Apdex is to report performance as it relates to the business.  Here Apdex is an element of dialog among IT staff, business managers, user representatives, and executive staff.  Apdex parameters are now carefully analyzed, presented with supporting evidence that shows business linkage, agreed upon by appropriate members of the organization, and documented.  The Apdex methodology brings structure, context, and an open standard to the dialog among key participants.  Apdex helps achieve consensus across groups about how to link application performance to the needs of the business.

Once company executives rely on Apdex reports then longer term performance management issues arise. For example how should Apdex parameters evolve over time?  This is like how Dow Jones decides to replace one of the 30 companies in the Dow Jones Industrial Average or showing the price of a stock before and after a share split.  The basis has changed.  How should you make the change?  Such a change shifts Apdex scores on the day the change is implemented.  How do you communicate the change in charts going forward?  Eric Goldsmith, Operations Architect at AOL has dealt with these issues, and described them in a fascinating presentation on the topic in the 2007 Apdex Symposium.

In a nutshell, you can use Apdex for simple tactical purposes within the network group as well as complex strategic executive level purposes.  The good news is that you can walk before you run, starting with simple uses and working your way up to sophisticated uses, leveraging your Apdex experience along the way.

Apdex Vendor Challenge: Show Me the Data

Apdex is dead simple. It is a standard way to convert many response time measurements into a single numerical value that always stays within the range of 0 to 1 where 0 is a disaster and 1 is perfect performance delivery. Dozens of vendors measure response time, so you would think you could readily get an Apdex report. Alas it is not so.

Last December, the Apdex Alliance asked vendors to submit the names of their products that generate Apdex reports to the Apdex Tools Directory. The directory has entries from the following six vendors: Compuware, Gomez, ip-label.newtest (formerly Auditec), WildPackets, and Xa Systems. However, when you search for information about Apdex at their websites, you get responses like, “No Pages were found containing Apdex”, or “Forbidden: Your client does not have permission to get URL.” WildPackets is the only vendor that has any information about their Apdex features. The WildPackets search on Apdex yields 51 documents with descriptions, FAQs, and tips.

So if you are interested in Apdex, you had better be prepared to roll your own reports. The good news is that you can easily create Apdex reports in a spreadsheet. We also find it useful to use a simple database like Access to organize your measurements. An Access query can pull out the measurements associated with the report you are creating, like all California measurements for yesterday. You can program the Apdex formula directly into queries and show them in a variety of reports.

Regardless of your report generation method, you will need input data. This turns out to be the tricky part. Many measurement tools do not make it easy for you to export measurement samples in a useful format. Most vendors want you to stay within the confines of their product. In fact, they help you import data from other sources but once there, they really don’t want you to export.

The same Apdex Tools Directory lists a dozen vendors that supply data which can be used to generate Apex reports. The most valuable part of the directory is the explanation of how to export the data and what kind of format the data will be when exported. This gives you a guide to match the measurement tool and your report generation tool.

Measurement vendors should make all of this much simpler. For example, they can add a generic report design capability where the user can define what data they want to see and how they want it processed (like the Apdex formula), and then how the report should look. If vendors continue to try and force their tool users of to see only reports that they designed then they must provide a wide range of reports including Apdex.

We are seeing an interesting pattern among enterprises that have created the following solution to the problem. They generate production (ongoing) Apdex reports using their own software using input data from various vendors. They use their own data format and conventions as a way to “normalize” the measurements from the various vendors. This then becomes the common data set which a variety of reports can use.

Enterprises that are serious about Apdex describe three benefits to this approach. First, they do not have to sort out differences in how the vendors generated their unique reports. They simply don’t use the vendor reports. Second, they can integrate data feeds from a variety of vendors into a single report. This would be nearly impossible with any of the vendors that have designed their product thinking they are the only measurement and reporting tool at an enterprise. Their reaction is, “You mean you use my tool and another company’s tool?” Finally, they can now add and remove measurement vendors as needed without changing their reporting system. They call this the “vendor isolation” feature.

If you are a measurement or management vendor you should be worried. This approach reduces your value to the enterprise to commodity status. Vendor isolation also means easy vendor replacement.

We think vendors have two choices. Increase value to your customers with better and more flexible reporting capabilities so your customer can build his own Apdex reports–or slide down the slippery slope to a commodity low-price data feed role. Oh, yes, there is a third choice: add Apdex reports directly into your product.

If you are a vendor that generates Apdex reports or can supply measurement data into a customer’s Apdex reporting engine, we want to know about it. Please post a comment to this blog or send an email to peter@apdex.org.

How Fast is Fast Enough?

Modern life is about speed-rushing to keep up. Your business application had better run fast, faster, fastest. Conventional wisdom dictates that application response time must always be faster. Whatever speed your users experience today, it had better be half that next year. Stop! Think about the consequences.

I once met with a client with a serious business problem-an application used by partners to support the company was too slow. So slow in fact, that frustrated partners were defecting to competitors. Everyone agreed that the response time was too slow. But when I asked, “How fast should it go?” chaos ensued with 20 people arguing over the target time. 

In the end they agreed that it should go faster as soon as possible and keep getting faster indefinitely. Peter replied, “This is good news since you just hired our firm for an indefinite contract that will have an infinite price!” The room went quiet, then people asked how they should determine how fast is fast enough.

There is no shortage of experts on this topic. James Gleick wrote a fascinating book entitled “Faster: The acceleration of just about everything.” His book is replete with examples of how the world has sped up as technology has enabled faster transportation, production, and communication. His thesis is that processes will go faster until we live a uniformly nanosecond world. When he wrote the book in 1999 he thought that day was nigh. But note that one of the great speed icons of that day-the Concord-was retired soon thereafter in 2003. There are no supersonic commercial flights available today. How can that be? Faster hit a limit.

No doubt some things will continue to happen faster, especially if is the speedup has a direct tangible benefit. The tangible benefit to business is improved productivity (more for the same money) or higher revenue (more money for the same effort). Did you notice the word money appeared twice? The first business rule of speedup is that it must improve the bottom line. The second rule is that the money it contributes (income or savings) must be greater than the cost.

Many projects that improve application response times are not cost effective. All technology improvements reach a point of diminishing returns, and business application speed is no exception.

I have had many animated conversations with colleague Peter Christy at the Internet Research Group over the years about the push and pull between “it’s never fast enough” and “speed has practical human-computer interface limits.”  The last time they had this conversation Peter Christy cited research that the human nervous system transmits cell-to-cell signals at about 1 millisecond. [Note that the brain and nervous system do that many times in parallel so the bandwidth must be big.] Peter Christy postulated that we won’t reach the limit of application response time speed until the computer interface matches human capabilities.

Many speed targets have come and gone over time. There was “the Web must respond in 8 seconds” rule promulgated by Zona Research at the turn of the millenium. Jupiter Research recently replaced that with 4 seconds as the new threshold of acceptability for retail web page response times.  And there is the old IBM 1 second rule from the 1970s. All of these rules were proposed by organizations with a vested interest in speed. By the way, they all told us that the world as we knew it would end if these speed rules were not achieved immediately. Has Microsoft gone bust because nothing on Windows happens in less than a second, or is Amazon just a mirage since most of their web pages load in more than 4 seconds?

The upshot is that if you believe that faster is always better, you are likely to spend too much, and however alluring one inflexible “fast enough” speed may be, it is a bad idea. The “right” speed depends on context. That context is a complex fusion of what the user needs to accomplish and how the application is designed. Taking this context into account is key to the Apdex methodology. Enterprises must figure out how fast is fast enough for application users to have a satisfactory experience.

I will be explaining Apdex approaches to determining the proper target time “T” in future posts.

But for now I have run out of time.

Gomez Showcases Apdex

At a Gomez users’ meeting held after Web Experience Forum on October 16 in Boston, users provided great commentary about how they use Gomez services to ensure good performance-including how they are using Apdex.  Imad Mouline, Gomez’ CTO, described where all of the Gomez application performance measurement services fit into an application management maturity scale of availability, response time, and consistency (good performance for all users all the time).  The Gomez Business Pulse XF portal integrates the views and leverages web standards (e.g., XML and SOAP) built into its measurement services to make it easy to tailor your own web service performance view.

As fans of Apdex, a Business Pulse XF feature that caught our attention was that it produces Apdex scores right from the Gomez measurement platforms.  The Apdex reporting tool is a simple-to-configure widget in the Business Pulse XF dashboard.

Norm Morrison of GSI Commerce, an e-commerce service provider, gave an excellent presentation on how GSI uses Apdex to track the performance delivered to all its customers.  GSI Commerce is a billion dollar company that provides design, hosting and operations support to about 100 major businesses like Toys”R”Us, Radio Shack and CBS Sports.  The company has been on a multi-year quest to improve how they define and deliver service quality to their customers.

GSI started defining and tracking quality using several metrics. Rigorous measurement and reporting of these metrics uncovered many small, previously unnoticed issues with capacity, product configurations, and vendor bugs.  This new understanding gave them confidence to press on to write service quality into their contracts.

GSI then struggled to expand the service metrics, searching for a simple response time metric they could link to customer business needs.  They tried and adopted Apdex, and their customers now get Apdex reports that uncover issues that remained unseen in simple response time averages or availability reports.

GSI is a great example of a company that is constantly improving quality.  The fact that they adopted Apdex as a better, more aggressive way to measure and report quality and share the results with their customers is a clear commitment to continuous improvement.

What is your company doing to truly foster continuous service quality improvement?  We are looking for similar stories.  If you have an Apdex story pass it on to peter@apdex.org so we can share it with the Apdex community.

Apdex Symposium Call for Presentations

We are seeking speakers for the third annual Apdex Symposium. The Apdex standard and methodology are supported and promoted by the Apdex Alliance. The 800 alliance members include vendors that support the effort and individuals interested in adopting Apdex within their enterprise. A goal of the alliance is educational outreach and building community around application performance management, and the Apdex Symposium is key to that effort. 

The Alliance will hold its third annual Apdex Symposium–a conference within the CMG Conference in Las Vegas, Nevada Tuesday, December 9th. We encourage you to attend, and if you have experience applying Apdex to share what you have learned as a speaker.

If you have some Apdex data, case studies, examples, products, a home-grown implementation, or ideas about how to improve Apdex–let the rest of the community know. Topics will include, but are not limited to:

  • Comprehensive Performance Management
  • Using Apdex to Manage Performance
  • Performance Management Tools
  • Apdex Case Studies
  • Apdex Enhancements

The Apdex sessions are always well attended and the feedback is consistently positive. You will only need to provide a 20-minute presentation. No lengthy paper is required! If you are interested, please contact the symposium chair (peter@apdex.org). Take a look at last year’s symposium program and presentations at the Apdex website to get a better idea of what it’s about. The 2007 Apdex Symposium archive is here.

The International CMG ’08 Conference is in Las Vegas, December 7-12 at the Paris Hotel. The CMG Conference is an extremely educational venue for anyone interested in application performance management, and Apdex Symposium participation is included in the CMG conference fee. 

How Apdex Works

Apdex describes how to report application response time measurements so even business managers can understand. The methodology is measurement-tool agnostic–you can use synthetic agents or measure the real user experience. You can measure in the data center, out on the net, or on the user’s desktop. However you gather the data, after an hour or a day you will have thousands of values to work with. Apdex converts all those values into a simple index that allows you to make apples-to-apples performance comparisons across applications.

The process starts with defining a ‘report group’ that the index value will represent. This is the first step in reducing vast numbers of measurement samples into a meaningful data subset. Report group parameters may include such things as user group and time of day.

The index is based on three application responsiveness zones:

Satisfied:     The user is fully productive. This represents the time value (T seconds) below which users are not impeded by application response time.

Tolerating:   The user notices performance lagging within responses greater than T, but continues the process.

Frustrated:   Performance with a response time greater than F seconds is unacceptable, and users may abandon the process.

The two thresholds of T (the tolerating threshold, also known as the target time) and F (the frustration threshold) define three performance buckets into which all the samples of a report group can be placed: zero to T, T to F, and >F. The index calculation is a weighted sum of the percentages of samples that fall into each performance zone.

The Apdex formula is the number of satisfied samples plus half of the tolerating samples divided by all the samples. Samples from frustrated users do not count.

Apdex[T] = (Satisfied + Tolerating/2)/Total

This ratio is always directly related to the users’ perception of satisfactory application responsiveness. To understand the full meaning of the ratio, it is always presented as a decimal value with the target time T in brackets (or subscript without brackets). For example, if there are 100 samples with a target time of 3 seconds where 60 samples are below 3 seconds, 30 are between 3 and 12 seconds, and the remaining 10 are above 12 seconds, the Apdex formula result is:

0.75[3] = (60 + 30/2)/100

The Apdex score is always between zero and one, with one the best score possible. Like school grades, a unit-less index makes intuitive sense and reflects the way most managers think–i.e., bigger is better. An application that performs better today compared to yesterday has smaller response time numbers but its Apdex score is higher. It may sound simplistic, but trust us–you are more likely to get budget approval to “get an A” than to lower a bunch of measurement values.

Also the Apdex score operates like availability which is well understood by non-IT managers. It is expensive to improve availability from 0.96 to 0.98 and it is even more expensive to reach 0.99 availability. An Apdex score slips seamlessly into the same management discussion.

So those are the Apdex basics. We encourage you to export some response time measurements from your favorite tool into a spreadsheet and apply the Apdex formula. Let us know how it goes. We will be explaining more of the methodology in upcoming blogs.

Apdex: A Standard for Reporting Application Performance

Imagine a simple numerical measure of user satisfaction with the performance of enterprise applications. It exists and it’s catching on. It’s called an Apdex score. Here’s why it is and what it is.

Why Apdex? For two decades CIO magazine has polled IT industry leaders to understand their burning issues, and every year a top goal has been to align IT and business performance. In this year’s survey 82 percent of respondents said aligning IT and business was their number one activity. Think about it. CIO’s have not solved their number one problem for the past 20 years!

For those 20 years CIOs have applied management technology as a nostrum to align application delivery technology to the business. Now more than 50 performance measurement tool vendors vie with each other on the accuracy and detail of their data. Enterprises often use more than one of these vendors and add home-grown tools to the mix. The result is a sea of numbers which fail to shed light on the user’s actual experience.

To solve this dilemma, NetForecast organized a cadre of vendors to develop and specify a new way to report on performance using measurement capabilities that already exist. The result is the Application Performance Index or Apdex. Apdex is an open standard so you can use it freely.

What is Apdex? Apdex is a numerical measure of user satisfaction with application performance that provides a uniform way to report on the user experience. It converts many measurements into one number on a zero to one scale (0 = no users satisfied, 1 = all users satisfied). It can be applied to any source of end-user performance measurements. If you have a measurement tool that gathers timing data like a user could gather with a stopwatch, then you can use it.

Apdex translates many individual response times measured at the user task level, into a single number. A task is an individual interaction with the system within a larger process. Task response time is the elapsed time from when a user does something (mouse click, hits enter or return, etc) to when the system (client, network, server) responds so the user can proceed with the process. This is the time during which the human waits for the system. These individual waiting periods define the “responsiveness” of the application to the user.

Apdex is governed by the Apdex Alliance, a non-profit organization governed by its members. The Alliance promotes the Apdex standard and associated methodology, and also runs an education and community outreach program. The Alliance is supported by about a dozen companies and more than 800 individual supporting members.

In upcoming postings we will explain the Apdex standard and supporting methodologies. We will describe how to report application performance so business managers understand, and we will help you reach that holy grail of business/IT alignment that has eluded CIOs for the past 20 years.