Hi, My manager asked me to define KPI for the business analyst job. I’ve tried do to some research but everybody seems to have a bit of a problem around this. Do you have any ideas from where to start?
Finding a Meaningful KPI for a Business Analyst
Trying to measure what appears to be an intangible item (BA Performance) is truly difficult. It takes some imagination and thought to figure out whether there are actually tangible objects that can be measured. If the conversation was about development performance, we could measure defects per lines of code. For project management, we could track cost overruns or schedule slips.
For business analysis, what exactly is there available to measure?
We have to start with our products that we deliver – and look at the value of business analysis. Since most of what we actually deliver is a service, what could be considered an actual product that is created? The only thing that comes to mind is the requirement itself, which is either captured in BRD for Waterfall, or as an individual or small set of requirements grouped together for Iterative/Agile. Either way, since it is produced, it should be able to be measured for quality, right? Kind of.
Measuring Business Analysis Performance Through Peer Reviews
I’ve found that it’s not a simple comparison, but rather a multi-step process that offers an illustration of quality over time. What I do is mandate for my projects that documentation is exposed to the peer review process .
A peer review comprises of a review of the documentation against pre-defined acceptance criteria of quality characteristics. The BABOK 2.0 discusses these criterion at length starting on page 115. The review process should capture a percentage of overall requirements (maybe 2% of the total number of requirements) and should result in feedback that highlights flaws in requirements.
For instance, if I request a review of a BRD from five peer analysts with direction to review a random set of say 20 requirements each, I would receive feedback on 100 +/- reqs and indication of what is wrong with each. The initial review provides a picture of how well the analyst is writing requirements and the feedback can be used for rolling improvements. I then keep that feedback even after I correct the flaws.
Measuring Business Analysis Performance Through Evaluating Defects
The second piece is to run a report of defects for the release during the UAT cycle and perform some analysis to try to determine if any of the defects map directly back to requirements (or missed requirements). The number of requirement related defects can then be compared against the number of peer review requirements flaws for an overall percentage that can be applied toward a performance KPI.
The kicker here is the time it takes to perform all this comparison analysis and usually results in this second piece not getting done. Simple cost/benefit analysis usually indicates that the first part is fairly adequate and adds in the needed quality where it has the most impact.
How do you measure your business analysis performance? Have you found a truly valuable KPI to evaluate a business analyst?
Learn How to Measure BA Performance
Adriana Beal has address this challenging topic in Measuring the Performance of Business Analysts, a practical guide to finding meaningful KPIs that can be measured without unnecessary overhead.
Click here to learn more about Measuring the Performance of Business Analysts
21 thoughts on “How Do I Define a KPI for a Business Analyst?”
I’m back just to add a couple of links that may be of interest of people who get to this page through search:
Using Performance Objectives and Measures to Improve Requirements Processes
Measuring the Performance of Business Analysts
OK, now that I had a bit more time to compose my answers, I’d like to address in more detail some of questions and comments here.
“If you measure the wrong things or only part of the picture you skew behaviors into feedback lops that continuously degrade your outcomes and make everyone frustrated.”
I couldn’t agree more, Craig.
“What standard are you going to measure against? Good grammar, adherence to formatting rules, completeness, correctness? Good grammar is only important for meaning, but if the document is meaningful, I do not care how beautifully it is written.”
Good grammar would make a terrible metric, in my opinion, heh.
Take a look at this table: http://bealprojects.com/metrics.png. It is an excerpt from a paper from Daniel Powell called “Requirements Evaluation Using Behavior Trees – Findings from Industry”. Requirements incompleteness, redundancy, inaccuracy, ambiguity, etc., are all characteristics that can be measured in a requirements artifact. Results from this type of metric can then be used to improve the process, identify competence gaps so the BA can receive appropriate training, etc. See below examples of how to objectively define those defects.
“I could care less about documentation standards because that tells me absolutely nothing about how effective the BA is in his/her job.”
I agree that measuring adherence to standards is typically a bad idea. If management is doing a good job explaining why following certain standards is important (e.g., because clients are expecting to receive documents with the quality and look and feel they’ve been accustomed to), the level of adherence should be high anyway, and there wouldn’t be any reason to measure it.
“How can you measure completeness if it is an Agile project, so that requirements are not complete until the end of the project?”
It all depends on how you define “completeness”. In an Agile project, it would be possible, for example, to verify the completeness of a set of user stories. Example: The BA is in charge of completing all “Preferred Customer” user stories. He produces “Create Preferred Customer” (As a customer, when I purchase more than $10K in goods since my first purchase, I become a Preferred Customer), “Set Preferred Customer Discount” (As a Preferred Customer, I receive 10% discount …) and declares the job done.
If it had been communicated by the project stakeholders that preferred customers also have the privilege of signing up for an exclusive promotions newsletter, there’s clearly one story missing from the set (the measurement would then reflect 1 incomplete story set).
“And even then, whether or not the requirements are complete depends a LOT on who is receiving them. Are the requirements complete enough for the person receiving them?”
What are you calling “completeness” here? To me it’s only a matter of clearly defining it. It is entirely possible to determine in an objective way whether a set of requirements is complete or not based on a clear definition (in the Agile example above, doesn’t have any user story missing; for a project using use cases, doesn’t have missing/omitted/implied behavior, and so on).
“quantified and objective measurements do not work due to factors that are beyond the BA’s control”
That’s not a valid argument for not measuring the performance of BAs. There are many ways to isolate the performance problems for which the BA is accountable, vs. things that are consequence of poor processes, lack of stakeholder involvement, etc. The attribution problem is not exclusive to the BA world; many other types of professionals, like salespeople, are subject to the same issue. Sales revenue production performance results are dependent on multiple departments that salespeople have no authority to control. The ability to close sales is highly dependent upon having satisfied customer references available, continuing flow of new prospects from advertising and other marketing efforts, general state of the economy, etc. Nevertheless, lots of organizations are able to successfully create metrics that provide objective basis for managers to isolate the performance problems for which the salesperson is accountable vs. external circumstances that are responsible for unusually bad (and good!) results.
Why go through this effort, though, of finding the right metrics, defining the standards against what people are going to be measured, etc.?
Performance measurement allows managers to set expectations regarding what constitutes effective BA work, identify where they need to focus their coaching efforts or allocate more resources, and generally use the knowledge gained to achieve substantial performance improvement.
Lots of good observations here, but for lack of time to reply individually, I’ll just add a few comments:
As some have already mentioned, you need to start from what you are trying to achieve (“which strategic element do we want to assess?”). Once you have clariﬁed your strategic objectives, you can start designing KPQs (key performance questions) based on what matters in your organization.
From there you define appropriate indicators that can help you answer the KPQs, do not invite cheating, for which you can collect data, etc.
In his book on performance measurement, Douglas Hubbard provides four useful assumptions when it comes to performance indicators and performance assessment:
1. Your problem is not as unique as you think.
2. You have more data than you think.
3. You need less data than you think.
4. There is a useful indicator that is much simpler than you think.
I’d add that we also don’t need “perfect” indicators (which don’t really exist). As long as measurement is an improvement on prior knowledge and helps us reduce uncertainty, then it is valid.
I am a little late to this party, and I think I can only speak to the impact of Business Analysis overall. One type of consulting I participate in is introducing requirements practices to organizations that have been doing requirements badly. The main thing that tells them this is the number of change requests on a project after requirements are done that are because of incorrect or missing requirements (as described in the original post). This then is the primary metric to monitor organization effectiveness when new practices are being used; this number will drop, and our experience is that it will drop by 75% or more.
How can this be used for evaluating one person doing Business Analysis? Certainly the goal of a BA is to have no change requests, but an acceptable amount, relative to project size, can be defined based on the experience of the organization. This acceptable amount can be reduced over time to reflect the overall improvement in requirements work in an organization.
So, any project that experiences a number of change requests above the acceptable level will be the subject of some analysis (often in a “Lessons Learned” activity). Some of the increase may be due to the nature of the project, such as those that are for a business function that is new to the organization. However, it should be identified when BA error is the underlying cause, and use that for remedial or advanced training to reduce the changes on the BA’s next project.
So, I agree that evaluating a BA’s effectiveness through peer reviews, for example, does not provide a measurable metric but can provide feedback to a BA, especially one who is just starting out and is looking to improve.
In the end, KPIs are useful, but only as one aspect of evaluating BAs. I speak from the experience of being evaluated annually as a BA for over 20 years, and never being really satisfied with the process. A KPI like reduced change requests might in fact have helped me (presuming that it would be a positive metric for me!). However, I have also never been a people manager charged with evaluating people, and I do not envy such managers, mainly because they have to try and evaluate feisty people like me!
This is fuzzy and difficult because we’re starting from a lousy requirements definition … we don’t know for sure if the manager who initiated this is looking for a KPI (or KPIs) to:
(a) serve as part of the job evaluation of the BA analyst, and/or
(b) serve as a value measure of the BA function to the organization, and/or
(c) serve as a “everything’s working well (or not)” process measure for the employee’s (or the BA function’s) in-box vs out-box.
Re (a), if the manager is looking for input to a periodic evaluation, he/she should be able to sit quietly for 3 minutes and be able to come to a judgment of whether the employee is an A / B / C / D / E.
If he/she can’t do that, then that manager is either not involved enough in the day-to-day work and projects, supervises way too many people, or is simply afraid to make a judgment call. (Quant people are sometimes exceedingly reluctant to rely on nothing but their own judgment; that could be caused by an underlying “I interpret data, I don’t just make it up” mentality, perhaps.)
Here’s a startling insight: evaluating people’s job performance IS inherently subjective, unless the job is as simple as “run 100 meters faster than anybody else.” You can dress it up, throw in any number of purportedly objective or standardized measures, but 9 times out of 10 the evaluating mgr is filling out the scorecards, after the fact, to justify the gestalt impression he’s made during the year. There are some valid reasons for doing that exercise, but let’s not fool ourselves into thinking that’s an entirely objective measurement. Or that it should be.
The more pseudo metrics you throw in the evaluation process, the more likely it is that your employees will work towards those metrics, at the expense of trying to maximize their real value to the organization as they prioritize their efforts on the day-to-day opportunities that arise.
I happen to think that is a very counter-productive approach for BA knowledge-workers. (But I work in a fast-moving dept. where today’s work is often a question that no-one in the organization had ever thought of until yesterday. Flexibility and the willingness to work on whatever unexpected thing might fall out of the coconut tree onto us is a key to a good chunk of what what we do … but that is not necessarily true in all areas of business analytics. )
I’ll leave discussion of the other flavors of KPIs listed above for someone else to run with, or for another time.
But not before saying that, if you are coming from a “competing with analytics as a core strategic capability” mentality, then you arguably don’t need performance KPIs as much as you need to be looking at “do we have ENOUGH BA resources to maximize on all the opportunities we could be exploiting, and do we have them working on the RIGHT things?” …. both of which are SUBJECTIVE, high-level mgmt questions.
Why does trying to get at “knowledge worker” productivity always seem to get back to issue of “what are you trying the measure here”? I think the simple answer is something on the order “if you don’t have numbers, you can’t measure it.”
It seems like there isn’t much agreement on what your trying to measure either.
Just what are those KPI’s ? (how about are we making money yet? 🙂
Measure what is important. If you measure the wrong things or only part of the picture you skew behaviors into feedback lops that continuously degrade your outcomes and make everyone frustrated.
(Measuring estimates versus actuals is a road to pain.)
Think about the higher level goals you are trying to achieve and measure progress towards those.
Looking for a framework? Start with the balanced scorecard.
– Are you getting value for the work?
– Are you continually learning and adapting to keep up with your changing environment?
– How are your project and systems stakeholders? Do they love you or hate you? Why?
– Are you working with the corporate processes, improving them or ignoring them? How does that affect others and overall performance?
Dave, I’m sure there are plenty of discussions happening about this topic. I receive lots of requests to write articles about it, and have declined most of them because of the risk of people taking your ideas in the wrong way and implementing poorly conceived measures that will be as likely to cause performance to decrease as to increase.
I am preparing a workshop (which may be presented in a future IIBA event) that hopefully will allow a more in-depth exchange of ideas and provide good context-dependent examples to help BA managers select the right indicators to use.
By the way, I disagree with the conclusion that you can’t have quantified and objective measurements because of factors beyond the BA’s control. This is a common complaint that I see in various knowledge-intensive fields and I think is unfounded. Measuring performance the right way can be challenging, may require some creativity, and will probably involve some trial and error, but it can always be done.
There have been many online discussions related to this topic. Here is the link to just one other example … http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&gid=92583&discussionID=11148888&sik=1277301103051&split_page=2&goback=%2Eana_92583_1277301103051_3_1
There was also a symposium panel discussion at the BA World Conference in Toronto last month related to this topic. My notes from that session indicate that even the panel of experts agreed that you need to define why the measurements are needed and what they will be used for and that and that quantified and objective measurements do not work due to factors that are beyond the BA’s control.
Excellent continued discussion. I appreciate all of your contributions. To further agitate you, I came across the following discussion that has been active since 2006 over on Requirements Network and thought I’d share.
Geri, thank you for the excellent example of why measurements need to be defined in context.
I had the same problem once working for a large bank — at some point my supervisor had to create a conference call with the PMO (who worked in a separate location) so I could directly explain to him why the indicators we were required to produce were not appropriate for our case, and were simply creating unnecessary paperwork.
An effective performance measurement system will adjust KPIs to take into account the disparities caused by differences in product/service segments, business environments, markets, technologies, software development methodologies, etc.
I like your article Adriana – particularly I like that you emphasize that each company has to think about what they want to achieve by the measurements, and that they need to review what measurements they are using as the organization changes.
One of my big concerns is the idea that every organization can be measured the same way. Even within an organization, measurements that make sense for a traditional team (such as number of change requests) may be completely meaningless for a Scrum team (which does not have a formal change management process).
If we have one set of measurements for everyone, then we will end up asking some teams to produce documentation just so we can measure it, and not because the documentation is needed for the project! I have personally experienced this (unfortunately it has been often in recent years), where I was REQUIRED to produce documentation that was useless for the project, but had to exist so that measurements could be made of my team’s effectiveness.
Thank you for reminding us that we need to think about what we are doing before putting policies into place.
I think you did a great job summarizing a topic that is too complex to discuss in such a short space. I’m working on a presentation on the topic of measuring the performance of BAs, and it’s really hard to discuss this topic without establishing a good foundation on performance measurement first.
For the reader asking where to start, I’d add the following advice: make sure you are selecting KPIs that are useful for learning and improving, rather than for controlling purposes. Perhaps you will find this article I wrote some time ago for ComputerWorld UK also relevant:
Mark and Geri:
Thanks for your comments. Mark, your starting measurements make complete sense and I was wondering if you could update this post down the road with what you are finding regarding the measurement results and their effectiveness.
On another note, I took a gander at http://www.smartkpis.com and there are a few measurements that have been posted regarding the requirements portion of technology. There are currently KPIs for % Requirements satisfied in the initial design, # Person days needed for rework due to requirements capturing inefficiency, % Defects removal efficiency , and % Requirements changed….all interesting things to take a look at.
Hi Doug –
Check out “first, break all the rules” by Marcus Buckingham and Curt Coffman. It is a very interesting study of effective managers and what they do to be effective. 🙂
We have just gone through our first round of performance measurement review. At this stage we are just looking to get a baseline, but we are targeting the following data points.
1. Pre-proposal work (time spent in Enterprise Analysis type activities)
Start Date, End Date and hours
2. Business Requirements
Start Date, projected hours, end date, revised end date, actual hours
3. Functional Requirements
Start Date, projected hours, end date, revised end date, actual
We have also started to review a Quality program, aimed at reporting on requirements traceability, as well as looking at how many change controls and requirements ellaboration exercises we are doing. This is all in addition to process tracking of peer reviews and making sure that projects actually have BA’s 😉
As a manager I have also been reviewing the utilization of my BA’s and where they are spending their time by business unit.
Hope that helps,
Pingback: Best Nine Brainiac’s Experiment in Microwaves | PCB Electronics Designing
Your thoughts are great Geri. I’m very glad you posted, because this questions continue to gnaw at me. I couldn’t agree with you more, but I approached the answer from a perspective in which if forced to produce SOME measurement, the recommendation may be a way to do so.
I understand the need to measure and dissect everything under the sun in order to assess performance, but it’s not always that easy or straightforward…..or worthwhile.
Thanks again, Geri
I find looking at requirements documents pretty meaningless. It seems like a logical thing to do, but here are the problems:
What standard are you going to measure against? Good grammar, adherence to formatting rules, completeness, correctness? Good grammar is only important for meaning, but if the document is meaningful, I do not care how beautifully it is written. I could care less about documentation standards because that tells me absolutely nothing about how effective the BA is in his/her job. How can you measure completeness if it is an Agile project, so that requirements are not complete until the end of the project? And even then, whether or not the requirements are complete depends a LOT on who is receiving them. Are the requirements complete enough for the person receiving them? How can you measure correctness unless you are a member of the project team yourself?
It is hard to measure BA Effectiveness – it requires WORK on the part of the reviewer. It requires that the person measuring the BA be closing monitoring the BA’s work. This means involved management, not the hands off style I see so many places. And it requires that the manager knows what the BA should be doing.
Here is what I look for in the people I have worked with who I would hire again:
1. Can the person work effectively with almost anybody on the planet? By effectively I mean that the person gets his or her work done while maintaining a cordial working relationship with all concerned. It is a combination of being self-motivated, assertive without being aggressive, and being sensitive to when it is time to push and when it is time to back off.
2. Does the person collect the right information at the right point in time? This is an evaluation of the person’s judgment. There are no hard rules here – right information at the right point in time is completely dependent on the project and the people involved.
3. Does the person effectively communicate what he or she has learned with the project team? In this case, he or she is effective if the team understands what they need to do with the information and it is the right information at the right time for the job. I also evaluate if the person writes well and speaks well, because a weakness in those areas leads to poor communication.
On top of all that, the reviewer has to be sensitive to the challenges inherent in the projects the BA is part of. For example, maybe the BA was actively prevented from collecting some information in a timely manner. You cannot blame the BA for that. So the reviewer has to know the situation.
BTW – I think it is equally difficult to effectively evaluate a Project Manager or Developer. We like measurements and think they prove something, but in our knowledge based world there are too many intangibles for me to put my faith in those measurements.
Just my thoughts. I welcome discussion.
Thank you Geri. I found this helpful as well.
Pingback: Tweets that mention How do I define KPI for a business analyst? -- Topsy.com