How to Use Performance Measures to Fix Business Analysis Problems

Author: Adriana Beal

It is well and good to opine or theorize about a subject, as humankind is wont to do, but when moral posturing is replaced by an honest assessment of the data, the result is often a new, surprising insight.

–Steven D. Levitt and Stephen J. Dubner, in Freakonomics

Many people like to stay away from discussions about individual performance measurement, dismissing the entire subject as a source of trouble. As we all know, performance measures can drive behavior in both bad and good directions. When not enough thought is given to the selection of measures, often they will present a hidden downside, or fail to drive the desired outcomes. (This clip from Jamie Oliver’s Food Revolution offers a good example of how measuring performance even against well-intentioned targets can easily subvert the purpose of the system).

Poorly designed performance measurement systems inhibit creativity, cause sub-optimization, and lead to gaming, creative accounting, and fraud. Those effects can be frequently seen in organizations that follow the command-and-control thinking and use performance measurement to control behavior and to punish or reward people. A comment left at this Bridging the Gap article illustrates well how a dysfunctional measure can cause more damage than not measuring anything:

If we have one set of measurements for everyone, then we will end up asking some teams to produce documentation just so we can measure it, and not because the documentation is needed for the project! I have personally experienced this (unfortunately it has been often in recent years), where I was REQUIRED to produce documentation that was useless for the project, but had to exist so that measurements could be made of my team’s effectiveness.”

Oh, how I would have liked to meet the designers of this performance metric! There are so many questions I would like to ask them, starting from, “do you really think that this measure tells you anything about the achievement of purpose?”.

The fact is that measuring compliance with standards and procedures is rarely needed. Adherence to processes and policies–when they actually make sense for high performance–can be easily achieved through training, coaching, and mentoring, without the need to control individual behavior. As I wrote in my just released eBook, Measuring the Performance of Business Analysts,

If there is a solid reason to measure adherence to documentation standards, one way to avoid distortions is to have different sets of documentation requirements for projects according to their size, level of complexity, organizational impact, and other relevant factors. A better option may be to let the decision of the required documentation happen during the inception phase of each project. This allows the adherence to documentation standards to be measured against a particular agreement, rather than against a generic set of standards that may not be applicable in all cases.

There is, though, a much better use for performance measurement systems than to ensure compliance with existing stardards: to produce insight over what is happening in our areas of responsibilities, identify performance problems and opportunities, determine priorities, and help us take action to improve processes and results.

Good performance measures remove any questions about what matters most. They clarify what specific results or problems you are responsible for, remove any ambiguity that may exist about your role and responsibilities, and provide the necessary context and information you need to decide where to focus when you are caught between conflicting or competing initiatives.

Here is an example: imagine that data collected from projects over the years helped management determine that requirements artifacts with up to 3 minor defects per 100 requirements statements (found during requirements review sessions or later by the development team) do not cause any significant impact on project schedule or effort, while a higher number of even small defects constitutes an issue that affects the health of the project. A target of a maximum of 3 non-severe defects per 100 requirements statements could then be established and monitored. If I, as a BA, have an average of 7 defects per 100 requirements, my knowledge about the desired vs. actual results will help me focus on developing skills and adopting behaviors that will take me closer to the existing goal. On the other hand, if I’ve been consistently delivering requirements artifacts with zero severe defects and no more than 3 minor defects per 100 requirements, it will make sense to direct my improvement efforts to other areas where performance gains can generate a higher return to the organization.

As long as performance measures are providing a complete and balanced picture of performance (so individuals are not overlooking important parts of the overall process in the attempt to maximize their own functional silos), and not being used to control behavior or punish people, there are huge benefits to be achieved from collecting performance data.

As Karl Wiegers wrote in his book Practical Project Initiation: A Handbook with Tools,

When my small software group at Kodak established a work effort metrics program in 1990 we found it to be more helpful than we expected. One team member expressed it well: “It was interesting to compare how I really spent my time with how I thought I spent my time and how I was supposed to spend my time.”

If you use measures in your organization, do any of them tell you about the achievement of purpose? Do they tell you what needs changing in BA activities to improve performance? Help you predict what would be the consequences of an action taken to change business analysis processes? Describe how well your business analysis techniques are working from the stakeholders point of view?

If the answer to these questions is no, regardless of how many measures you are collecting, performance measurement is not being used for what really matters to achieve top performance in your organization. BAs who can answer yes for those questions will find it much easier to recognize the broad assortment of benefits that a good performance measurement system can produce, from gaining a better understanding of what factors affect your performance most, to the opportunity to study performance problems in a methodical manner so that data, not opinions, can drive actions.

>>Learn More About Measuring BA Performance

Measuring the performance of business analystsAdriana has put together a thoughtful e-book on how to create a BA measurement program. You’ll learn how to establish the right metrics and use this new information to improve the performance of your business analyst practice.

I’d recommend this book highly to anyone in a position to oversee the work of business analysts, including BA managers, IT managers, operational managers, or project managers. It’s also a great resource for the senior business analyst looking to showcase thought leadership around BA performance measurement within their organization.

Click here to learn more about Measuring the Performance of Business Analysts

Free Training - Quick Start to Success

(Stop the frustration and earn the respect
you deserve as a business analyst.)

Click here to learn more

By signing up, you agree to our Privacy Policy.

Comments

  1. Thank you, Anup, I’m glad you liked the article and the reference to Oliver’s video.

    I completely agree, measures such as lines of code written or number of requirements written are examples of ill-conceived metrics that are irrelevant for the purpose of achieving high performance, but sadly are still found in the measurement system of some organizations.

  2. Anup Mahansaria says

    sorry for my typo …
    I meant counter productive measure …

  3. Anup Mahansaria says

    Hi Adriana,

    Wonderful article! Jamie Oliver’s video is so relevant. By the way, I really like Jamie Oliver. He is a revolution himself!
    Another example of counter productive which is common IT world: Lines of codes written.

  4. Linda, I decided not to wait until my next BTG article to answer your question, so here is my answer for you: http://bit.ly/cNolnK

    I hope you like the answers, and thank you for keeping the conversation going on such important (and frequently overlooked) topic.

  5. Your comments are always appreciated too, Linda! I think that you are absolutely right, and that the main reason performance measurement frequently fails is that many managers start from the question “what can I measure?” instead of “what are our improvement goals, and what are the performance questions related to these goals that I need answers for?”

    I like your question very much, and will use it as the topic for my next article here at Bridging the Gap. Thank you for jumping in!

  6. I always love to read your thoughts Adrianna. Great article and well said about defining metrics that measure achievement of purpose. I find that many BA or BA managers struggle with this topic because of the difficutlies faced with defining the purpose. Defining metrics without knowing the purpose/clear objectives is what lead organization to define metrics that are not useful and dysfunctional. It may be the case that those folks who desire to implement measures, are not so aware of what to look for because of the lack of clear objectives. Hence, they want to see anything that can be produced (which sounds like what happened in your example above). Understanding the Basics of our profession – What should folks look for when implementing a BA practice in terms of effectiveness? Is there a benchmark or standards? What does it mean to define clear objectives and how to derive useful metrics based on those objectives? – can be a great starting point for those folks. It will be nice to hear your thoughts on this.

Before you go, would you like to receive our absolutely FREE workshop?

(No formal experience required.)

21689
21690

Quick Start to Success
as a Business Analyst

By signing up, you agree to our Privacy Policy.