Manage Quality and Metrics – Techniques
Gather Subjective Metrics with Client Satisfaction Surveys
Gathering metrics is important because it allows you to see how you are performing against the expectations of your clients. If the world were perfect, all of the metrics you collect would be factual, relevant and accurate. However, in many cases it is impractical or cost-prohibitive to try to gather exact and quantitative numbers.
One way to supplement quantifiable metrics is with client satisfaction surveys. For instance, instead of trying to measure the exact response time of an application against some service-level standard, you could simply ask your main users how satisfied they were with the application response time. It should make sense that if you are rated a 4.5 out of 5.0 (5 being the highest), you are probably doing a pretty good job in this area. However, if your survey results come back as a 1.8 out of 5.0 (5 being the highest) then it should be obvious that you are missing the mark. You did not need a complicated system, metric to tell you that. A simple customer satisfaction survey questions gets quickly to the same result.
In the same way, let’s say you want to gather metrics that indicate the time it takes to resolve client problems. This could involve tracking when the initial request comes in, when you first responded to the client and when the request was resolved. On the other hand, you could simply send out surveys that ask your clients if they were satisfied with the time it took to resolve the problem.
Surveys are by their nature qualitative; that is, they reflect the opinion of the person being surveyed. Therefore, you would not necessarily want to base your entire project success criteria on survey metrics. Some results are more easily obtained quantitatively. For instance, there is usually no reason to send out a survey to the finance department to ask them if your spending is within budget. You should have the facts available to you. However, for many other types of metrics, a qualitative survey question can be used as a substitute for the quantitative metric.
Collecting Project Metrics and Demographics Can Assist on Future Projects
Gathering and reporting a consistent set of metrics at the end of a project can help your organization see the trends for delivering projects over a period of time. The metrics should show how well project teams are meeting their commitments in terms of quality, cost and duration. As more and more projects report the metrics, a baseline will be established that will allow your organization to see how it is improving, or slipping, over time.
If you collect metrics on an organization-wide basis, you should collect some project demographics in addition to the actual metrics. Project demographics are just predefined project characteristics that provide a description on each particular project. If you store the demographics and metrics in a file or database, they can be analyzed to show the overall trends in a more discreet and granular way.
If you do not collect demographics, you can still compare actual cost to deliver versus the estimated cost. The advantage of gathering some project demographics is that you can compare similar projects. For instance, you can compare the cost and effort associated with delivering mainframe development projects versus web development projects. Or you could compare how well projects from the Sales organization meet their deadlines and budgets versus projects from the Finance area.
The other benefit of gathering project demographics is that you can use the information as input into future project estimates. For instance, let’s say you have marketing campaigns for different products. Let’s also say that you have been collecting project estimations and the actual results for cost and duration. If you had this information, you could estimate the cost and duration of a current product marketing campaign by comparing your project with similar projects that were completed in the past. This may be of help for estimating your project. The demographics you capture from completed projects can be used to search on later for new projects. (A sample worksheet to capture project demographics and metrics is available in the Template Library.)
Make Sure Your Metrics Tell a Complete Story
In many instances the project team publishes the results of a metric in a way that does not allow the reader to fully understand whether the results are good or bad. This is because the reader sees a metric, but they do not understand the target and they do not always understand what the metric is trying to achieve. The project manager and project team may know what a given metric is telling them, but other readers of the information may not. One way to help is to always report the metric along with the target. For instance, if you report your current expenditures to date, also include your expected expenditures at this point in the project. If you report that your project has spent $100,000 so far and your total budget is $150,000, the reader still does not have the context to know whether this is a good or bad situation. Sure you are under budget, but the work is not complete either. The better way to report this information is to state that you have spent $100,000 to date and that according to your estimate you should have spent $110,000 at this point in the project. If the trend continued, you estimate the final cost of the project to be $135,000 versus your budget of $150,000. If you report the metrics with this context, your readers can understand what the numbers are saying.
