The hidden cost of skipping requirements

A common complaint about software development is that projects often go over-budget and take longer than originally estimated.

The sad truth is: the hourly effort of software development in almost all cases is much higher than anticipated, and certainly a lot higher than any programmer would ever admit.

While in some cases the problem is caused by the developer's incompetence and lack of experience, in most cases the problem has nothing to do with the developer. The problem is caused by the fact that there is hardly any billable effort allocated to properly assess the requirements prior to giving a "quote".


While most small to medium clients view the work as hours spend on the actual programming - building a web application, just like building a house, involves a lot more effort than simply coding or putting bricks together. In fact, coding in most cases is less than 40% of the total effort.

A typical web application development process consists of several phases, each of them requiring time (and money):
-- Planning (also called "Discovery", "Specification Development" or "Requirements Gathering")
-- Structural design
-- Implementation
-- Testing

(The phases may repeat over and over, depending on client's solidity in original requirements, market changes, new fads in the industry, and new cool ideas seen on a competitor's site).

Unfortunately there is hardly ever enough budget allocated for the first 2 phases. The client tends to be willing to just jump right into the project without giving it enough thought ("the sooner we start - the better"), and feed random ideas later, throughout the development cycle.

In the meantime, the vendor is too focused on getting to the Implementation phase, as anything prior to that is viewed as part of getting the deal.

However, the first two phases of the development cycle are often the most labor-intensive and expensive ones, as they not only involve commitment by multiple parties, but also - brain picking and time spent by the most "expensive" participants: business stake holders, project management and senior development personnel to properly understand the requirements, develop the idea and lay it down into a blueprint that best fits the client's needs as well as time and budget.

If this stage in the process is done correctly, development (implementation) is a breeze.
However, it is extremely hard to get the small-to-medium business client to pay for the pre-implementation phases. Clients simply don't want to pick up the tab for the preparation time spent prior to implementing their custom solution. And unfortunately, there are plenty of desperate vendors out there who will spend hours of unpaid time on requirements gathering, research and idea generation BEFORE the client commits to a billable project. These vendors throw the expectations out of balance and create a situation where other vendors are also expected to develop a business process and other intellectual property for free in hopes of getting to the billable implementation phase. So what happens what you are forced into completing a good chunk of work for free in order to get that bid? Most likely you are not going to put forth enough effort to property assess all the details. As a result, the estimated project cost is likely to be just a number you gamble on.

Ideally, in today's world where choices and possibilities are overwhelming, and you have to be top-notch or else - a vendor should be paid a good chunk of money to review the project and do the proper research to sketch out the best solution and brain-storm on ideas before committing to any budget. In other words, the vendor should be paid for responding to an RFP.

Yet, most IT deals in the small to medium business world work in the pattern of rushed "guestimates".

Guess what? If you are not paying the vendor to do job one, but there is a potential to get job two, the vendor will complete job one for free with as little effort as possible, just to get that job two. Once the vendor gets job two and realizes that the original "guestimate" is way below the real cost, he has several choices:
-- insert hidden costs to make up for time spent on job one behind the scenes
-- complete the job two according to the vendor standards and resent any input from the client in an "out of scope" verdict fashion
-- inform the client half-way into the project that the work is going to take a lot longer and cost a lot more

The IT industry has been around long enough to establish methodologies, coding practices, etc. Yet, unlike the building industry, the idea that you have to develop (and pay for!) a blueprint for your IT solution before commiting to any budget has not sunk in for many clients. Would you ever ask a construction company to propose a budget on a new house without a blueprint? Would a construction company even talk about budget without assigning an architect to the site first?
What seems to be ridiculous in an established industry is still a common practice in the IT world that serves small to medium businesses.

What about the larger businesses? Once the deal is signed, the vendor is not going to be shy about blowing up the scope once any small change request creeps in. Larger businesses are more likely to easily accept (and pay for) scope changes, which gives the vendor a chance to get paid for all the hidden work.