The Template Collective Blog

Manage Quality and Metrics – Techniques

Gather Subjective Metrics with Client Satisfaction Surveys

Gathering metrics is important because it allows you to see how you are performing against the expectations of your clients. If the world were perfect, all of the metrics you collect would be factual, relevant and accurate. However, in many cases it is impractical or cost-prohibitive to try to gather exact and quantitative numbers.

One way to supplement quantifiable metrics is with client satisfaction surveys. For instance, instead of trying to measure the exact response time of an application against some service-level standard, you could simply ask your main users how satisfied they were with the application response time. It should make sense that if you are rated a 4.5 out of 5.0 (5 being the highest), you are probably doing a pretty good job in this area. However, if your survey results come back as a 1.8 out of 5.0 (5 being the highest) then it should be obvious that you are missing the mark. You did not need a complicated system, metric to tell you that. A simple customer satisfaction survey questions gets quickly to the same result.

In the same way, let’s say you want to gather metrics that indicate the time it takes to resolve client problems. This could involve tracking when the initial request comes in, when you first responded to the client and when the request was resolved. On the other hand, you could simply send out surveys that ask your clients if they were satisfied with the time it took to resolve the problem.

Surveys are by their nature qualitative; that is, they reflect the opinion of the person being surveyed. Therefore, you would not necessarily want to base your entire project success criteria on survey metrics. Some results are more easily obtained quantitatively. For instance, there is usually no reason to send out a survey to the finance department to ask them if your spending is within budget. You should have the facts available to you. However, for many other types of metrics, a qualitative survey question can be used as a substitute for the quantitative metric.

Collecting Project Metrics and Demographics Can Assist on Future Projects

Gathering and reporting a consistent set of metrics at the end of a project can help your organization see the trends for delivering projects over a period of time. The metrics should show how well project teams are meeting their commitments in terms of quality, cost and duration. As more and more projects report the metrics, a baseline will be established that will allow your organization to see how it is improving, or slipping, over time.

If you collect metrics on an organization-wide basis, you should collect some project demographics in addition to the actual metrics. Project demographics are just predefined project characteristics that provide a description on each particular project. If you store the demographics and metrics in a file or database, they can be analyzed to show the overall trends in a more discreet and granular way.

If you do not collect demographics, you can still compare actual cost to deliver versus the estimated cost. The advantage of gathering some project demographics is that you can compare similar projects. For instance, you can compare the cost and effort associated with delivering mainframe development projects versus web development projects. Or you could compare how well projects from the Sales organization meet their deadlines and budgets versus projects from the Finance area.

The other benefit of gathering project demographics is that you can use the information as input into future project estimates. For instance, let’s say you have marketing campaigns for different products. Let’s also say that you have been collecting project estimations and the actual results for cost and duration. If you had this information, you could estimate the cost and duration of a current product marketing campaign by comparing your project with similar projects that were completed in the past. This may be of help for estimating your project. The demographics you capture from completed projects can be used to search on later for new projects. (A sample worksheet to capture project demographics and metrics is available in the Template Library.)

Make Sure Your Metrics Tell a Complete Story

In many instances the project team publishes the results of a metric in a way that does not allow the reader to fully understand whether the results are good or bad. This is because the reader sees a metric, but they do not understand the target and they do not always understand what the metric is trying to achieve. The project manager and project team may know what a given metric is telling them, but other readers of the information may not. One way to help is to always report the metric along with the target. For instance, if you report your current expenditures to date, also include your expected expenditures at this point in the project. If you report that your project has spent $100,000 so far and your total budget is $150,000, the reader still does not have the context to know whether this is a good or bad situation. Sure you are under budget, but the work is not complete either. The better way to report this information is to state that you have spent $100,000 to date and that according to your estimate you should have spent $110,000 at this point in the project. If the trend continued, you estimate the final cost of the project to be $135,000 versus your budget of $150,000. If you report the metrics with this context, your readers can understand what the numbers are saying.

Remember is that it is impossible to categorize everyone within a profession.  You can make some general assumptions about technical people, but this does not mean that the assumptions apply to everyone.  As a manager, you must ultimately have multiple techniques that you can apply to different people in different circumstances.  One technique will not work for all people at all times. 

That being said, let’s make some generalizations about managing technical staff on your project.

• They tend to be introverts.  Generally speaking, the definition of an introvert is one who is primarily more comfortable with an inward focus in life while an extrovert is generally more comfortable with an outward focus.  For example, when introverts receive a lot of new information, they tend to want to think for a while before speaking or drawing conclusions.  Extroverts, on the other hand, are more comfortable expressing ideas to others.  If they jump to the wrong conclusions, they just change their minds.  Basically extroverts are comfortable thinking out loud.  Introverts would rather think through the “rough drafts” in their minds and then talk when they think they have a coherent and logical position. 
• They tend to think more logically than emotionally.  This tendency should be obvious.  Technical staffers typically are not motivated by a lot of “rah-rah” speeches.  In fact, they tend to be cynical of this type of motivation.  They will usually listen politely (perhaps even snickering to themselves), but the effects are short-term.  On the other hand, they can be persuaded and motivated by a logical argument.  If the logical argument can be combined with some motivational techniques, you might have a chance to actually get them excited. 
• They tend to be problem solvers.  This is a great strength of technical staff as well as one of their weaknesses.  Most technical people love nothing better than to be confronted with a problem.  They get excited and they immediately start to apply their problem-solving skills.  The weakness comes in because there is a tendency to jump on a problem without fully understanding it first.  This often can lead to being less than optimal in the use of resources. This is also a reason why technical people don’t like to spend as much time on project planning. They have a disposition to jump in and execute the project instead of spending time to understand the nature of the project first.
• They tend to be technically creative.  This may seem like a contradiction.  Your first thought might be that the sales and marketing staffs are the creative people.  In fact they are – in the sales and marketing areas.  They will also be the first to tell you so – because they are extroverts.  However, the technical discipline requires a fair degree of creativity as well.  This is especially true in the IT world.  In many cases, there is not one best solution to a business problem.  In the development field, for instance, analysts need creativity when they are defining a solution with the business clients.  Designers need to be creative applying technology in the best manner.  Programmers need to be creative as well in trying to apply the best techniques to build the most elegant solution.

Understanding these general characteristics is the place to start if you are a manager of technical staff.  Once you begin to understand how people work and how they are motivated, you can start to think of the best way to manage them.  Now that you have gone through many ideas, here is a summary of some of the general points you should consider as you manage your staff.

• Try to establish an environment where people feel they have what they need to do their jobs.  This includes having appropriate hardware, software, policies, procedures, etc. 
• Technical people like to understand the work processes in the group, and then they like to be creative in working within that structure.  So, set the high-level rules, but don’t micromanage the details. 
• Give people as much information as they need to do their jobs.  Technical staff tends to reflect on this information.  Ask for their ideas and opinions, but give them time and ample opportunities.  Don’t expect them to react immediately.
• Shield the team from office politics and all of the distractions that can abound in a large company.  Tell people what they need to know (see prior point), but don’t get them bogged down in the organization muck. 
• Give people continuous opportunities to learn.  This includes encouraging people to invest the time to learn, but also helping with some opportunities.  There are many creative ways to learn new things.  Once a person has mastered a certain skill or aspect of his job and he starts to become bored, look for ways he can cross-train and learn new areas of the group. 
• Be there when needed and respond to problems and concerns.  Not all problems can be fixed, but many times the simple act of listening and trying is enough.  People will give you credit for trying, even if the ultimate resolution to a problem is not available. 

You might note that many of these management techniques are not unique to technical staff in general or IT staff in particular.  It is true that many of the techniques can be used in other areas as well.  However, they are particularly applicable to the IT staff.

Many people find quality management to be one of the more difficult project management processes to implement. This is because quality is hard to define, and formal quality management requires you to collect metrics to validate the state of quality. The following process will help create a framework for the quality management process.

1. Create a Quality Management Plan

Develop a Quality Management Plan to identify the major deliverables, completeness and correctness criteria, quality control activities and quality assurance activities. The Quality Management Plan also describes how you will ensure that the client’s quality requirements are achieved. It is the place to describe the processes and activities that will be put into place to ensure that quality deliverables are produced.

2. Determine the customer requirements for quality

Work with your customer to determine their requirements for quality. The high-level characteristics of quality can be uncovered during the project definition process. The detailed quality requirements should be uncovered when you gather business requirements.

3. Define a set of metrics to validate quality requirements are met

Identify a set of metrics that will provide insight into the quality of the deliverables. The project manager should already be capturing overall financial and duration metrics. The quality-related metrics need to be more sophisticated. There are two areas where you are trying to manage quality – in your project work processes and in the actual deliverables you are building. You should try to capture metrics that will measure each.

4. Execute quality control activities

Quality control refers to activities that validate the quality of your deliverables. It is also referred to as “inspection”. Ensure that the quality control activities for every deliverable are performed while the project is underway.

5. Execute quality assurance activities

Quality assurance refers to the processes used to build deliverables. It is also referred to as “prevention”. Having good processes should results in good quality deliverables.

6. Monitor and resolve deliverable quality

You need to validate the quality of your deliverables on an ongoing basis. When quality problems are found, implement a process to determine the cause and to make improvements in the process.

Using this process will help you understand, plan for and manage the state of quality on your project.

Have you ever been asked how far along you were on a project? Of course you have. If you do not have a valid schedule, or if you are not keeping the schedule up-to-date, you know that your answer is pretty much a guess. If you have a good schedule and you are keeping it up-to-date, you should have a sense for how much work is remaining and what the projected end-date is. But are you 50% complete? Or 90% complete? It is not always easy to know.

Earned value metrics were established to remove the guess work from determining where you are at in relation to a baseline. Using it allows a project manager to know precisely how far along he is, how much work is remaining, what the expected cost will be, and all sorts of other interesting information.

Are you using earned value on your project today? Probably not. You are not using earned value because your organization has not adopted it. Implementing earned value on your project requires a tremendous level of discipline and a common set of processes. It is hard to apply earned value one project at a time, since no one else would understand what you are doing and why. It required an organization focus.

History

Earned value has not been around for hundreds of years. You can actually trace its beginning to the late 1800s and early 1900s, as managers attempted to make the factory floor and the production line as efficient as possible. The drive for efficiency requires a foundation in metrics and earned value was a way to measure things more precisely.

In the 1960s, the US Department of Defense began to mandate the use of earned value on defense related projects. As you might expect, if the government is contracting out projects worth hundreds of millions or billions of dollars, they want project progress updates to consist of more than “we seem to be on target.” Earned value calculations can provide a better sense for exactly where the project is against the baseline and provide an early warning if the trends indicate that the project would be overbudget or over its deadline.

EVM is based on just three core values.

• Actual cost (AC). The actual cost of work completed as of a point in time.
• Planned Value (PV). The budgeted cost of work you planned to complete as of the same point in time.
• Earned Value (EV). The budgeted cost of the work actually completed as of the same point in time.

Earned Value is calculated by adding up the budgeted cost of every activity that has been completed. (Remember, this is not the actual cost of the work activities. This is thebudgeted cost.) Look at the following example:

 Today’s Date is March 31

Let’s say that as of March 31 you have actually completed activity A, B, C and D. Let’s calculate AC, PV and EV.

• AC is the actual cost of the work completed. This is 55 (20 + 5 + 20 + 10).
• PV is the budgeted cost of the work planned to be completed. This is 45 (20 + 10 + 15). Note that Activity D is not counted since it was not planned to be completed as of March 31.
• EV is the budgeted cost of the work completed. This is 50 (20 + 10 + 15 + 5).

These three numbers seem to be interesting, but by themselves they do not tell you too much. So, we need to combine and compare the values to determine our status against schedule and budget. Sorry, but this next bit of information must wait until next week’s email.

Procurement refers to obtaining goods and services from outside companies. This specifically refers to vendors and suppliers. It does not refer to other internal organizations within your own company. (For the purposes of this discussion, “purchasing” and “procurement” are equivalent terms.) This is an area that project managers definitely need to understand at some level, and it is an area into which the project manager will give input. However, in many, and perhaps most companies, procurement is an area that the project manager does not own. The project manager normally does not have the authority to enter into contracts on behalf of the company, and he normally is not asked to administer the contracts once they are in place. (In some organizations the project managers have this authority, but my perception is that in most organizations they don’t.) 

If you are purchasing goods or services on your project, you should determine whether you will simply follow the procurement contracts and plans that are already established by your company or your organization.

Examples

• You may purchase hardware from companies using a pre-existing company contract.
• You may acquire contactors using a pre-selected preferred vendor list.

In some cases, the vendor identification and selection processes occur at an organization level.

For instance, your company may purchase a Customer Relationship Management (CRM) system based on high-level management requirements. This CRM system would then be used on all subsequent projects – regardless of the specific needs of each individual project.

If your project team is actually conducting the vendor identification and selection process, you have some flexibility on when it is done. Many project teams perform the vendor identification and selection processes during the project Analysis Phase. This would be the case if you need to better understand your business requirements before you determine the vendor that can best meet the requirements.

Once the vendor is chosen, there are many procurement processes that are part of project management monitoring and control. This includes monitoring the vendor progress, answering questions, validating invoices, paying invoices, managing contractual issues, etc. Ultimately the procurement process concludes when the project is completed and all project contracts are closed.

In general the processes and techniques for procurement are not so hard, but it is an area where many project managers don’t have a lot of experience. In fact, many project managers only acquire procurement knowledge as a way to pass the PMP Exam.  

Document management is a part of communication management. It is important for the project manager to recognize the stages that a document must go through from creation to completion. This knowledge allows the project manager to understand the overall status of a document at any given time and helps ensure adequate time is allocated for the completion of the document. For instance, when a team member says they can complete a document in two weeks, are they saying that the document will be ready to circulate in two weeks or that the document will be completed and totally approved in two weeks? Not all documents need to go through all the stages of document creation and approval. However, depending on the document, one or more of the steps will be required.

Plan the document

Sometimes you can sit down and immediately start writing your document. Other times you need to think, prepare and plan first. This is especially true as your document gets larger and more complex. Preparation and planning, which includes outlining the content and structuring the sections, will help you get started.

Create the initial document draft

In this step, the document draft is created. If there are no subsequent reviews and approvals, this step results in the creation of the final deliverable. Most of the effort associated with the document is used in this step. Subsequent steps may take a long duration, but they do not take nearly as much effort.

Circulate document for feedback
Modify as appropriate

These two steps involve circulating the document for initial review and feedback. The document is updated based on the review comments. Depending on the particular document, this may be an iterative step. A document may have an internal review, followed by a stakeholder review, followed by a management review. After each of these reviews, the document is subsequently modified based in the feedback and sent to the next step.

Gain document approval

When the document has been circulated for feedback and subsequently updated, it will be ready for final approval. Some documents should be formally approved in writing. Others are simply considered complete after the final round of feedback is received.

Like all completed deliverables there may be subsequent updates or enhancements that may require their own mini-document life cycle as well.

The Procurement Management Plan is a part of the overall Project Management Plan. The document describes how items will be procured during the project and the approach you will use to managing vendors on the project. Specific areas to describe include:

Procurement process. This section provides a brief overview of the process requirements necessary to manage procurement of the identified needs. This process should include:

• Initiating a request
• Development of requirements (technical, timing, quality, constraints)
• Request approval
• Purchasing authority
• Bid / proposal review
• Contract management responsibility
• Contract closure requirements
• Procurement process flowchart

Roles and responsibilities. This section describes the various roles on the project that have some connection to procurement. This section should describe who can request outside resources, who can approve the requests, any secondary approvers, etc.

Identified procurement needs. This section details the material, products or services identified for outside procurement.  Each listed item should include a justification statement explaining why this should be an outside purchase if there is the possibility of inside sourcing (make vs. buy decision).

Timing. This section will describe the timeframe that resources are needed. This will provide a better sense for when the procurement process needs to be started for each item.

Change review and approval process. Describe how changes are made to procurement documents to ensure the changes are valid, understood and approved by the appropriate people.

Vendor processes. Describe the processes that the vendors should use for timesheet approval, invoice processing, contract renegotiation, status reporting, scope change requests, etc.

There may be additional information in the plan as well to ensure the procurement process is understood and managed effectively.

Issues are more than just common problems. They are problems that meet specific criteria. An issue is a formally-defined problem that willimpede the progress of the project and cannot be totally resolved by the project manager and project team without outside help. The processes used to manage issues can be simpler or more rigorous depending on the size of the project. Use the following process to manage issues on large projects.

1. Identify the problem and document on the Issues FormSolicit potential issues from any project stakeholders, including the project team, clients, sponsors, etc. The issue can be surfaced through verbal or written means, but it must be formally documented using an Issues Form.
2. Determine if the problem is really an issueThe project manager determines whether the problem can be resolved or whether it should be classified as an issue.
3. Enter the issue into the Issues LogIf it is an issue, the project manager enters the issue into the Issues Log.
4. Determine who needs to be involved in resolving the issueThe project manager determines who needs to be involved in resolving the issue. The sponsor may be involved, or the sponsor may not have the expertise to assist in the resolution process. For instance, the problem may be contractual and require resolution from the Purchasing Department. However, at some point the alternatives will be discussed and a resolution will be made. It is important to understand up-front who needs to be involved in making this final issue resolution.
5. Assign to team member for analysis and alternativesThe project manager assigns the issue to a project team member for investigation (the project manager could assign it to himself or herself). The team member will investigate options that are available to resolve the issue. For each option, the team member should also estimate the impact to the project in terms of budget, schedule and scope.
6. Gain agreement on resolutionThe various alternatives and impact on schedule and budget are documented on the Issues Form. The project manager should take the issue, alternatives and project impact to the appropriate stakeholders (from step 6 above) for discussion and resolution. The project manager may want to make a recommendation from among the alternatives as well.
7. Close the Issues LogThe project manager documents the resolution or course of action on the Issues Log.
8. Close the Issues FormThe project manager documents the issue resolution on the Issues Form and then closes and files this document.
9. Add action plan to the scheduleOnce a resolution is agreed upon, the appropriate corrective activities are added to the schedule to ensure the issue is resolved.
10. Communicate through the Status ReportThe project manager communicates issue status and resolutions to project team members and other appropriate stakeholders through the methods established in the Communication Management Plan, including the project Status Report.

Smaller projects do not need all of these steps. For instance, the issue can be documented directly in the Issues Log without the need for the separate Issues Form.

When you create a schedule you generally don’t know enough to enter all of the detailed activities the first time though. Instead, you identify large chunks of work first, and then break the larger chunks into smaller pieces. These smaller pieces are, in turn, broken down into still smaller and more discrete activities. This technique is referred to as creating a Work Breakdown Structure (WBS).

A question people ask is how small the activities should be before they do not need to be broken down further. This is referred to as your “estimating threshold”. Work can be broken down into smaller activities than the estimating threshold, but normally no work would be left at a higher level. The threshold can be different based on the size of your project and how well the work is understood.

You can use the following criteria as a guide. For a typical large project (say 5000 effort hours or more) the activities should be no longer than two weeks. Medium and small projects (say 1000 effort hours or less) should have activities no larger than one week. Remember that this threshold is an upper limit. You can break the activities down further if you want.

Assigning work that is smaller than your threshold allows the work to be more manageable. Think about it. When you assign work to a team member you don’t know for sure how he is progressing until the due date (or the completion date if it comes first). (Yes, you can track progress if the work proceeds linearly – like painting a wall. But many of us work in the knowledge business (IT, Sales, Finance) and it is not easy to know exactly how the work is progressing.)

Let’s look at an example. If you assign a team member work that is due in four weeks, you are not going to know for sure whether the work is on time until the four-week deadline. The worker may tell you things are on track, but you don’t really know for sure until the due date. If the work is completed you will know you are on track. If the work is late you will know it then as well. However, four weeks (or longer) is too long to wait to know if the work is on track. A better approach is to break the four-week activity into four one-week activities. Then you will know after the first week if the work is on time or not.

If at all possible you want to try to have schedule work completed within two weeks. If you give someone work that takes four weeks or longer there is just too much time before you really know if things are on-track or not. It is much better for you if you can keep the schedule feedback loop to no more than two weeks.