Accurately estimating the scope, costs, and priorities of development projects requires a collaborative effort between IT and management.

You ask IT how long it will take to get a small project completed, and you are told that it will take about three weeks. After much frustration and numerous phone calls, your project is finally completed six months later! The IT department doesn’t seem overly concerned about late project delivery, and no one on the management team seems to know why every project takes approximately 10 times longer than promised.

No one wants to confront IT about the problem in case it results in bad feelings and future projects being delivered even later. Lack of ownership for finding a solution to the problem means the scenario repeats itself over and over again. Management thinks it is the IT department's fault; the IT department thinks it is management's fault. Month after month, year after year, this wedge of misunderstanding pushes management expectations and IT deliverables further and further apart. Sometimes, this results in the development of a "them and us" situation that further exacerbates the problem. Working relationships start to break down, mutual respect dwindles, and communication between management and IT becomes even less effective.

So why is it so difficult to accurately estimate project timelines? The majority of the time, the difficulty does not stem from poor estimation skills; rather, it is the result of poorly defined project scope combined with an ineffective prioritization process. To understand how these problems occur, you need insight into the problem from both IT and management perspectives. Consider the following scenario:

Jenny is a marketing assistant and has been tasked with finding out more about the organization's online customers. The information is needed to help her department design marketing strategies for current products and to identify potential opportunities for new products. Not much customer data is currently available. Jenny talks to Rob, the guy in IT who always helps her with computer problems, and asks him how long it would take to add some new fields to the customer database. He tells her that it will most likely take two or three weeks. Jenny asks him to go ahead and add six new fields to the database. She goes back to her desk confident that in about three weeks she will have all the data she needs for her marketing assignment. Unknown to both Jenny and Rob, the disconnect has begun.

From Rob's perspective, he has accurately answered the question that Jenny posed. The requested changes to the database will take about three weeks. What he didn't tell her, and perhaps because he didn’t know he needed to, is that there are quite a few other tasks that would need to be completed to collect and store the customer data to populate those fields. Jenny doesn't understand the technology well enough to ask the right questions of Rob. Rob doesn’t understand Jenny's business need well enough to offer advice on a solution or to ask additional questions.

Let's continue the journey of Jenny's IT request. Rob passes Jenny's request on to the IT manager, who adds it to his big list of IT projects. It ends up about halfway down the list, which means someone will be able to start working on it in about two or three months. Jenny doesn’t ask where her request is on the "big list" because she doesn’t know there is a "big list." Based on her conversation with Rob, she assumes that her project will be finished in three weeks, starting from the day she submitted the request.

Three months go by, and Rob's manager finally asks him to add those fields to the database. Three weeks later, Rob lets Jenny know that the database fields are there. She asks him when she will be able to get access to the customer data. Rob says he doesn’t know because all he does is make the change to the database, so she will have to ask the website developer about when those fields will be populated with data. Jenny is confused! She didn’t know she needed to talk to the website developer as well. She talks to John, the website developer, and asks him how long it will take for that customer data to be transferred from the website to the database. He tells her that he will have to make changes to the website to collect that data, and then he asks lots of questions that she is not sure how to answer, such as, "Do you need aggregate data or personally identifiable data?" and "Are we sampling or do you need all the data?" He also asks if she has spoken to the systems engineering team about the possible performance hit to the webservers from the data collection. Now she is even more confused and ends up going back and forth between numerous folks for the next two weeks. From those discussions, she learns that collecting the data will slow down the website unless they add an additional webserver, so she must decide which she prefers. After a couple more weeks of discussion with her manager and various IT folks, decisions are made about how to proceed. A month later, the data is being collected and deposited into the database.

Jenny goes back to Rob and asks how she can get access to the customer data. He tells her she can't get access to the production database, so she will need to dump a copy of the data to a separate system. What? She still can't look at the data? Jenny goes back to the systems engineering team and is asked a lot more questions that she can't answer like "How many weeks of data do you need to store?" "Will you purge old data or do you need to keep all historical data?" and "How often do you need the data copied to the marketing database?" She discovers that the cost of the system she needs will vary considerably based on how much data she needs to store. By now, poor Jenny is not enjoying this assignment much at all, but there is even more angst in her future.

A few weeks and a lot more dollars later, the marketing system is ready and is populated with customer data. Now she needs to get some reports. She asks the marketing administrative assistant if she can create some reports for her. The admin says she has no idea how to do that. Luckily for Jenny there is someone else on her team, Beth, who knows how to create the reports and she has offered to help her. Beth asks Jenny what specific information she needs in the reports. Jenny has no idea. Neither does her boss. They thought if they had access to more data they would just magically know more about their customers. By now, a three-week project has ended up being an eight-month project that has cost over $15,000, and no one knows what to do with all that data! Jenny's boss believes that IT underestimated the project, and IT believes that the marketing department should have known what they wanted before asking IT to do the work. At no point did the marketing department consider discussing with IT the business problem they were trying to solve. They assumed that the solution they came up with was the right one even though they lacked IT knowledge and expertise. IT is not blameless in this scenario. At no point did anyone in the IT department explain that adding fields to the database would not give Jenny access to any data without a significant amount of additional IT work. The IT manager should have been aware that the marketing department lacked the IT knowledge to know precisely what IT work would need to be performed to meet their business need.

This example might seem rather extreme, but it is not an unusual scenario. Some IT projects are less complicated than this, and others are more complicated, but they would all benefit from early input and advice from the technical folks. It is possible that someone in IT has, in the past, asked management why they wanted a specific project and management expressed displeasure at being questioned about business matters by the IT department. If someone complained about you to your boss, you would make sure that in the future you just did what you were told without asking questions. Many people in IT do just that for those very reasons. It is also possible that management has asked IT for advice in the past and IT has either confused and embarrassed them with complicated techspeak or has been very evasive with providing information. Some IT departments have the attitude that management should know exactly what they need and if they do not, then it is not IT's job to figure it out for them. If you have been intimidated or made to feel foolish by IT in the past, you are unlikely to ask them for advice in the future.

A project should be created and prioritized based on the importance to the business of solving the problem, not on how well a business person can define a technical solution to it. Therefore, it should not be a project requirement that you already have a solution identified before submitting a project request. If you involve your IT folks in the problem-solving part of the process, it will result in much better solutions. If Jenny had discussed marketing's business problem with IT, and they had been willing to help her scope out the project, she would have received a more accurate time and cost estimate and would have saved herself a lot of frustration!

Root Causes of Problems

Lack of communication, misunderstandings, and reluctance to work collaboratively are the root causes of project scope definition and project prioritization issues. Why? Because you have two vastly different groups of employees who are thinking about problems from two very different perspectives: IT—focused on solving technical problems; and Management—focused on solving business problems. When management requests a specific task from IT without any discussion about the business problem they are trying to solve, IT is not in a position to offer technical advice, guidance, or alternative solutions to solving the problem. In addition, IT cannot offer a realistic estimate if the full scope of the project is not discussed and understood. When IT gives an estimate for a project without discussion about the priority of that project over other work, their estimate is based purely on the number of hours the solution will take to implement and not on the calendar time it will take to complete it. If management is not aware that work on a project will not start immediately, management expectations are set too high right from the start. The expectation will not be met, which will result in disappointment and frustration at what management will perceive as a late delivery of an IT project. The IT folks think it should be obvious to management that they can't stop work on everything else each time a new project is requested. IT is often unaware that an expectation has been set that every project will start immediately. If IT is operating in a bubble where it is disinterested and/or disconnected from business priorities, it will not operate effectively. IT management is responsible for ensuring that business priorities are understood and taken into account when scheduling IT work. This means that IT management and business management need to be communicating regularly.

Aside from the challenges of determining project scope and accurately estimating time and costs, insufficient project requirements may be resulting in IT spending time working on projects that are an unnecessary use of time and resources. If management presents IT with tasks that are not based on proper business requirements, neither IT nor management will be unable to identify unnecessary projects, duplicate projects, or business problems that could be solved with a non-IT solution.

For example:

• Someone else in the organization has solved the same problem and has a solution that could be shared with no further involvement from IT.
• There exists a current solution that could be tweaked to meet your requirements rather than building a completely new system.
• A current solution exists that would meet 90% of your requirements at 10% of the time and cost of implementing your proposed solution.
• The solution requested will not solve the business problem.
• Another department has IT currently working on a project that solves the same business problem.
• Someone may have tried to solve a similar business problem using exactly the same solution at some point in the past, and though the project was implemented successfully, the business application was a failure.
• An off-the-shelf solution is available for $10 and you could buy it and use it today.

Lack of a well-defined process or inconsistent use of a process will create estimation and prioritization problems. Even if you have a team of management and IT folks who work well together, respect each other, and try their best to do the right thing, you will still run into problems if you start projects without following a pre-defined process. Skipping the pre-defined process because you think your project is too small for it is not advisable. How can you be sure your project is small if it has not been properly defined? Jenny thought her project was small!

Solving the Problem

To determine project scope, estimation and prioritization you need a well-defined process. It does not have to be a complicated or time-consuming process. The time you spend on it will likely be less than the time you currently spend chasing down IT to find out what happened to your overdue projects.

When it comes to process design, the most important element is simplicity! An overly complicated or time-consuming process will quickly be abandoned and you will quickly end up back where you started. The hardest part about introducing a new process is making sure everyone follows it. If some people circumvent the process, it will eventually fall apart completely. A streamlined and easy-to-follow process will not add more work. It will front-load some of the effort so that projects are estimated accurately and are completed on time. A process needs the following three elements:

• Assigned responsibilities
• Standard documentation
• Defined process steps

Assigned Responsibilities

Your project process needs an owner. Your process owner should be someone from the business side of your operation. Ideally, your process manager will have some project or program management experience. At the very least, select someone with good organization skills who is able to take an unbiased approach to project prioritization. It is unreasonable to expect IT to understand, or be able to make informed decisions about, business priorities or budgets. If ownership of this process has fallen to your IT department in the past, then it is no wonder that you have run into problems. The chances are that projects were prioritized based on the importance of the person making the request rather than the importance of the project to the business. If IT has no clear direction on business priorities, they have no choice but to make decisions themselves.

In addition to a business process owner, you need a business project coordinator (which could be the same person) and an IT project coordinator. These two folks will work together to ensure that requirements are sufficiently detailed so IT can accurately estimate time and costs. They will also work together to set priorities and schedule project start dates.

Standard Documentation

As with the process, keep project documentation as simple as possible. At a minimum, you need a basic project request document and a system for tracking the projects and priorities, which could be a database, a spreadsheet, or a webpage.

If you want IT to build solutions that meet business requirements, you must document business requirements and expectations clearly. Small projects may need just a few sentences describing what is needed. Larger projects may require more detailed information. Soliciting input from IT early in the documentation process will enable you to make full use of their expertise in documenting requirements and selecting the best technical solution.

You can create a project-tracking tool that is as simple as an Excel spreadsheet or as complicated as a portfolio management system. If you currently have no system, start with the spreadsheet so you can get up and running quickly. You can always upgrade to more sophisticated tracking tools later.

A sample project request document and project tracking spreadsheet are available for download here.

Defined Process Steps

Whether it is a three-day, three-month, or three-year project, the same process should be used. The scale will be bigger for more complex projects, but the steps should be the same. You may decide that you need a weekly or monthly formal management meeting to review project requests, or you may elect to have the IT and management project coordinators work together to approve and prioritize projects. In practical terms, it may make sense for smaller or urgent projects to go through an expedited process and for more complex projects to go through a more detailed review. The high-level steps will look something like this:

1.  Document the request.

2.  Review the request with IT to solicit input into potential solutions and to estimate time and costs.

3.  Submit the project request for approval and prioritization (to process owner, project coordinator, or project approval committee) and add to the project-tracking spreadsheet.

4.  If approved, assign a priority and schedule a start date (based on IT availability).

5.  Review the project-tracking spreadsheet weekly or monthly and revise project start and end dates based on new or changed priorities.

Depending on the size of the projects, you may be able to combine steps 2-5 into a weekly one-hour meeting. Just because there are multiple steps does not mean you have to spend a lot of time completing them.

Accurately estimating the scope, costs, and priorities of development projects requires a collaborative effort between IT and management. Rather than pointing fingers and apportioning blame for problems, focus on working together to find solutions. If you learn from mistakes, you will be able to quickly move on from them while using what you learned to continually improve the process. Even with the best process in the world, you will run into problems from time to time. With clear and timely communication, you can minimize disruptions and reduce frustrations for those affected by those problems.