Beware: You Get What You Measure!

There’s an old expression that says “you get what you measure”.  This may be a little clichéd, but it generally holds true within organizations and processes. It’s human nature to shoot for the objectives that have been set, and it is human nature to pay attention to the metrics which are being measured. If metrics and performance indicators are set incorrectly, process and systems break. Sometimes KPIs that were set with the intention of improving a situation actually have the direct opposite effect. And what’s worse, this poor customer experience gets “hidden” because the wrong data gets captured.

The problem: Targets are often blunt and misinterpreted

An emotive example of this phenomenon can be seen in health care.  In the UK, there is a state-funded national health system known as the NHS.  A few years ago, a target was set that meant that anyone should be able to get a non-emergency (routine) appointment with their family doctor within 48 hours.  The objective was to ensure timely access to medical advice, and on the face of it this sounds sensible.

Image of a tape measure

Beware: You get what you measure!

The problem with targets like these is that they are blunt instruments, and are open to misinterpretation.  After this target had been implemented, there were reports of a minority of doctors surgeries refusing to allow patients to book appointments in advance.  Patients were forced to phone every day, wait on congested phone lines and keep ringing back until an appointment slot was available that was within the 48 hour target.  In this case, the Service Level Agreement (SLA) looked “Green” – the data showed that every patient was being seen within 48 hours, but the reality was that a patient may have actually been trying to get an appointment for days or weeks before that!  The targets were wrongly understood and implemented, and this led to poor customer experience.

Sanity-checking SLAs and targets

As Business Analysts, we have the opportunity to challenge targets and SLAs of this type when we define or re-engineer processes.  As practitioners of organizational change, we should challenge targets and SLAs to ensure they are having the desired effect.  Before targets are set it’s important to understand what customer and business value is being generated by the system or process. There are several questions we can ask to provoke these thoughts:

1. Understand the problem: Long before a process, system or target is changed, the problem space has to be fully understood.  Only by avoiding the temptation to jump straight to a solution can we ensure that the root problem or root cause can be addressed.  By understanding this, we can ensure any targets are aligned to resolving the problem.  We can also ensure that the solution is appropriate.

2. Understand demand: Before changing a process or system, it’s necessary to understand demand. What do the customers or users want? How often do they want it, and which parts of their requests vary? What is their worldview? How are they interacting with the current system or process, and what would be considered an “improvement” from their perspective?  By knowing this, we can ensure targets and SLAs are properly aligned.

3. Work on systems not processes: Changing an individual business process in isolation will often move a constraint or problem to a different point in the system.  The only way to ensure you’re going to solve your problem is to understand the end-to-end experience the customer will have, by tracking their request through the entire system.  This top-level understanding will ensure you don’t cause new problems down-stream.

4. Understand business value: When working on any project or business improvement initiative, it’s important to understand the value that the business wants to obtain, as well as the value the customer wants to obtain. There might be tensions or contradictions between the two, and airing these early can be helpful.  Targets should be aligned with both business value and customer value.

5. Provide mechanisms for customer feedback: After implementing a system or process change, provide mechanisms for customer feedback.  If the system and processes aren’t working, revisit, iterate and improve.

On a personal level, I’ve never been a fan of targets and SLAs. Controversially, I believe that they should be used sparingly – each SLA or KPI should have a clear and compelling reason.  If it can’t be justified it should be scrapped.  Generally speaking, I’ve always found it’s better to have the right people employed andthe mutually good relationships with suppliers. An organization that manages its staff or suppliers by SLA is already in trouble!

>> Learn to Ask the Right Questions – And Discover What Is Really Needed

Interested in receiving a comprehensive set of questions you can ask in almost any project context? Want to feel more confident asking questions in a new domain? The Requirements Discovery Checklist Pack includes over 700 questions, categorized and cross-referenced so you can prepare for your next elicitation session with a sense of ease and confidence.

Click here to learn more about the Requirements Discovery Checklist Pack

There’s an old expression that says “you get what you measure”.This may be a little clichéd, but it generally holds true within organizations and processes. It’s human nature to shoot for the objectives that have been set, and it is human nature to pay attention to the metrics which are being measured. If metrics and performance indicators are set incorrectly, process and systems break. Sometimes KPIs that were set with the intention of improving as situation actually have the direct opposite effect. And what’s worse, this poor customer experience gets “hidden” because the wrong data gets captured.

An emotive example of this phenomenon can be seen in health care. In the UK, there is a state-funded national health system known as the NHS.A few years ago, a target was set that meant that anyone should be able to get a non-emergency (routine) appointment with their family doctor within 48 hours.The objective was to ensure timely access to medical advice, and on the face of it this sounds sensible.

The problem with targets like these is that they are blunt instruments, and are open to mis-interpretation.After this target had been implemented, there were reports of a minority of doctors surgeries refusing to allow patients to book appointments in advance.Patients were forced to phone every day, wait on congested phone lines and keep ringing back until an appointment slot was available that was within the 48 hour target.In this case, the Service Level Agreement (SLA) looked “Green” – the data showed that every patient was being seen within 48 hours, but the reality was that a patient may have actually been trying to get an appointment for days or weeks before that! The targets were wrong, and this led to unintended outcomes leading to poor customer experience.

As Business Analysts, we have the opportunity to challenge targets and SLAs of this type when we define or re-engineer processes.As practitioners of organizational change, we should challenge targets and SLAs to ensure they are having the desired effect.Before targets are set it’s important to understand what customer and business value is being generated by the system or process There are several questions we can ask to provoke these thoughts:

1.Understand the problem: Long before a process or system is changed, the problem space has to be fully understood.Only by avoiding the temptation to jump straight to a solution can we ensure that the root problem or root cause can be addressed.By understanding this, we can ensure any targets are aligned to resolving the problem.We can also ensure that the solution is appropriate.

2.Understand demand: Before changing a process or system, it’s necessary to understand demand.What do the customers or users want? How often do they want it, and which parts of their requests vary? What is their worldview? How are they interacting with the current system or process, and what would be considered an “improvement” from their perspective?By knowing this, we can ensure targets and SLAs are properly aligned.

3.Work on systems not processes:Changing an individual business process in isolation will often move a constraint or problem to a different point in the system.The only way to ensure you’re going to achieve your problem is to understand the end-to-end experience the customer will have, by tracking their request through the entire system.This top-level understanding will ensure you don’t cause new problems down-stream.

4.Understand business value:When working on any project or business improvement initiative, it’s important to understand the value that the business wants to obtain as well as the value the customer wants to obtain.There might be tensions or contradictions between the two, and airing these early can be helpful.Targets should be aligned with both business value and customer value.

5.Provide mechanisms for customer feedback:After implementing a system or process change, provide mechanisms for customer feedback.If the system and processes aren’t working, revisit, iterate and improve.

On a person

There’s an old expression that says “you get what you measure”.  This may be a little clichéd, but it generally holds true within organizations and processes. It’s human nature to shoot for the objectives that have been set, and it is human nature to pay attention to the metrics which are being measured. If metrics and performance indicators are set incorrectly, process and systems break. Sometimes KPIs that were set with the intention of improving as situation actually have the direct opposite effect. And what’s worse, this poor customer experience gets “hidden” because the wrong data gets captured.

An emotive example of this phenomenon can be seen in health care.  In the UK, there is a state-funded national health system known as the NHS.  A few years ago, a target was set that meant that anyone should be able to get a non-emergency (routine) appointment with their family doctor within 48 hours.  The objective was to ensure timely access to medical advice, and on the face of it this sounds sensible.

The problem with targets like these is that they are blunt instruments, and are open to mis-interpretation.  After this target had been implemented, there were reports of a minority of doctors surgeries refusing to allow patients to book appointments in advance.  Patients were forced to phone every day, wait on congested phone lines and keep ringing back until an appointment slot was available that was within the 48 hour target.  In this case, the Service Level Agreement (SLA) looked “Green” – the data showed that every patient was being seen within 48 hours, but the reality was that a patient may have actually been trying to get an appointment for days or weeks before that!  The targets were wrong, and this led to unintended outcomes leading to poor customer experience.

As Business Analysts, we have the opportunity to challenge targets and SLAs of this type when we define or re-engineer processes.  As practitioners of organizational change, we should challenge targets and SLAs to ensure they are having the desired effect.  Before targets are set it’s important to understand what customer and business value is being generated by the system or process There are several questions we can ask to provoke these thoughts:

  1. 1. 1. Understand the problem: Long before a process or system is changed, the problem space has to be fully understood.  Only by avoiding the temptation to jump straight to a solution can we ensure that the root problem or root cause can be addressed.  By understanding this, we can ensure any targets are aligned to resolving the problem.  We can also ensure that the solution is appropriate.
  2. 2. 2. Understand demand: Before changing a process or system, it’s necessary to understand demand. What do the customers or users want? How often do they want it, and which parts of their requests vary? What is their worldview? How are they interacting with the current system or process, and what would be considered an “improvement” from their perspective?  By knowing this, we can ensure targets and SLAs are properly aligned.
  3. 3. 3. Work on systems not processes: Changing an individual business process in isolation will often move a constraint or problem to a different point in the system.  The only way to ensure you’re going to achieve your problem is to understand the end-to-end experience the customer will have, by tracking their request through the entire system.  This top-level understanding will ensure you don’t cause new problems down-stream.
  4. 4. 4. Understand business value: When working on any project or business improvement initiative, it’s important to understand the value that the business wants to obtain as well as the value the customer wants to obtain. There might be tensions or contradictions between the two, and airing these early can be helpful.  Targets should be aligned with both business value and customer value.
  5. 5. 5. Provide mechanisms for customer feedback: After implementing a system or process change, provide mechanisms for customer feedback.  If the system and processes aren’t working, revisit, iterate and improve.

On a personal level, I’ve never been a fan of targets and SLAs. Controversially I believe that they should be used sparingly – each SLA or KPI should have a clear and compelling reason.  If it can’t be justified it should be scrapped.  Generally speaking, I’ve always found it’s better to have the right people employed and the mutually good relationships with supplier. An organization that manages is staff or suppliers by SLA is already in trouble!

What are your views on targets and SLAs?  Do you have any experiences or additional thoughts? I’d love to hear from you. Please feel free to contact me directly, or add a comment below:

al level, I’ve never been a fan of targets and SLAs. Controversially I believe that they should be used sparingly – each SLA or KPI should have a clear and compelling reason.If it can’t be justified it should be scrapped.Generally speaking, I’ve always found it’s better to have the right people employed and the mutually good relationships with supplier.An organization that manages is staff or suppliers by SLA is already in trouble!

What are your views on targets and SLAs?Do you have any experiences or additional thoughts? I’d love to hear from you. Please feel free to contact me directly, or add a comment below:

Free Training - Quick Start to Success

(Stop the frustration and earn the respect
you deserve as a business analyst.)

Click here to learn more

By signing up, you agree to our Privacy Policy.

Comments

  1. Hi Adrian,

    Yes there are unintended consequences in many of the metrics we use. The more you study this area, the more you start to see it everywhere.

    I wrote a light-hearted blog on it a few days ago after reading this post () and having a particularly faff-filled week.

    Jason, you have it spot on! It is the ethos under which the metric is formed which is most important. When everyone’s on board and wants to improve then you’re more than half way there.

    Mel

  2. Jason, you make a very valid point – engagement is key. People need to understand *why* and *how* they are being measured, and need to know how this contributes to the overall value that the organisation delivers to its customers.

    That is an angle I hadn’t considered, so thank you very much for raising it!

    Adrian.

  3. Hi Mel,

    Many thanks for your comment.

    Yes – I agree – a measure of “how many people are we curing” would be much more appropriate. The key is to align the measure with things the *customer* actually values!

    However, there are still hazards to be avoided. For example, if a metric of “how many people are we curing” was used, there could be an incentive to work on the easy cases, and ignore the hard cases…. So if you have a simple cut that needs a bandage, you’ll be seen the same day (as it’s quick), but more complex cases are deferred….

    It’s an extremely interesting and complex area, and one that could be debated for hours. I think the discipline of systems thinking is a useful lens for this debate.

    Thanks again for your comments Mel, much appreciated.

    Adrian

  4. Jason Ang says:

    Good day Adrian,

    I agree with you that its the “spirit” of the SLA or KPI that’s important. I’ve used KPIs and SLAs quite extensively because it provides an idea/guage on how to plan in that it provides a sense of scale. I do think that there will always be people who try to “game” the system, so that’s where engaging with the people whose activities are measured, establishing trust with them as well as having good feedback mechanisms from the end-client are important.

    Thanks for the article, very useful.

  5. Hi Adrian,

    I completely agree. If a measure isn’t telling you how well you are doing, and how to improve, it’s a waste of time.

    Good example too. Getting an appointment at my doctors is a bit like playing the lottery now, if the lottery only sold tickets between 8:30 and 8:33am each day before running out. You can just imagine the potentially life-threatening failure demand that this is creating! A better measure could be “How many people are we curing?”. I’d like to see more of these.

  6. Hi Mark,

    Many thanks for the comment. You are so right!

    I often think that it’s the “spirit of the SLA” rather than the “letter of the SLA” that is important. What really matters is the *intention* rather than the specific wording/semantics. The trouble is that often people are measured by a strict interpretation of the rules, and as you quite rightly point out, it isn’t possible to mandate for morality or ethics. I’d go as far as to say it isn’t possible to legislate or mandate common sense(!). This leads to occasions where people would rather “act by the book” instead of being empowered to use discretion and common sense.

    I’m sure we’ve all seen such examples – from the fast-food counter clerk who won’t serve breakfast because it’s 10:02 (and breakfast finishes at 10:00) right through to a supplier who will take *exactly* ten days to produce an estimate for each piece of work (however trivial) because that’s what’s stated in the contract…..

    I agree fully with your final paragraph. In a mutually trusting relationship, measurements can be useful. As you quite accurately point out, this allows feedback and adjustment — the key being that feedback is used for learning and enlightenment, not as a contractual weapon!

    Kind regards,
    Adrian.

  7. Great article! We’ve all seen misguided measurements lead to the wrong behaviors…a very frustrating thing!

    I especially like this comment: “Generally speaking, I’ve always found it’s better to have the right people employed and the mutually good relationships with suppliers.”

    I think that’s a very important essence. You cannot legislate/regulate/mandate ethics, morality, or “heart.” Relationships with mutual trust are so important. It is unfortunate that many of the huge systems (like the healthcare example you gave) end up having to try to manage ethics/morality/heart by mandate.

    That being said, in the context of a trusting relationship, a few simple measurements can be helpful in order to ensure everyone is achieving what they set out to do. With a positive (as opposed to punitive) mindset, this can give the opportunity for early feedback and adjustments to ensure everyone is successful.

    Mark

  8. Michelle Swoboda says:

    Adrian, it is always great to read your posts. I learn from all of them!

  9. Hi Curtis,

    This is a really good question. I certainly agree with the statement that “a rough measure of the right thing is better than a precise measure of the wrong thing”. Measuring the wrong thing will be misleading at best, and at worst will drive and reward the wrong types of behaviour amongst staff!

    However, you raise a very interesting point about “scale”. I hadn’t specifically thought about it from this angle, but it’s certainly true that some SLAs would benefit from taking more of a macro-view.

    For example, take a typical call-centre. It’s quite possible that each call-centre agent is incentivized to ensure that their average call length doesn’t exceed 2 minutes (or whatever). This drives the wrong behaviour, as some will people do the *quickest* thing, not the *right* thing. This creates failure demand… which creates more calls…. Which are rushed, leading to more failure demand (and the cycle continues!)

    A better measure would be at the macro level – e.g. “Is the call centre answering every call within x minutes” and “Are our customers satisfied – measured by a survey”. These are higher level goals, but are less likely to lead to the wrong types of outcome.

    Thanks again for your comment. Let me know what you think of my reply!

    All the best,

    Adrian.

  10. Curtis Michelson says:

    Adrian, I enjoyed the article. thank you. I’m reminded of another quote, not sure who said it, “a rough measure of the right thing is better than a precise measure of the wrong thing”. I wonder if one of your concerns with SLAs is that they tend to measure at the wrong ‘scale’? Would love your comments on that.