The cost of poor quality software

Issue Date: 01-11-98
Issue Number: 9.06
Category: SOFTWARE

Quality assurance in software development is a problem that has been kicking around for almost a half century. The first bug was an insect caught in an early, 1940s computer. Things have hardly gotten better since then. A recent task force headed up by Howard Rubin and including Capers Jones, chairman of Software Productivity Research of Burlington, Mass., found that fewer than 6% of organizations have clearly defined software management processes in place. Would any other industry get away with treating quality in this way? The current chronic shortage of software systems personnel though, together with the scramble to achieve Y2K readiness (probably the biggest fix in software history), is beginning to lift the issue of quality higher up the agenda.

In a separate paper published earlier this year, Capers Jones analyzed the link between poor quality and the number of canceled projects with the software labor shortage. In addition to that 60% estimate of the amount of time developers spent on software repair, he found that at any one time, 15% of the global software workforce is involved in projects that will never be deployed. Last year, we published a two-part series which looked at the role of automated software quality testing tools, and how they impact the quality assurance process. In this report, we take a broader look at the quality process itself, examining the figures in the Capers Jones study, and its implications for how to improve the process.

The process of defect removal is long and labor intensive. And it's not flashy. All too often, project teams focus on the 'visible sides' of software projects: the functionality and user interface, and developers are compensated accordingly. Yet few reward good quality. All too often, quality-conscious developers are accused of working too slowly.

The problem is: software quality assurance remains labor-intensive, even with automated tools that eliminated a portion of the legwork. Things only get worse with size; as projects get bigger, they get more complex, providing a fertile breeding ground for bugs. Capers Jones estimates that projects of 100,000 function points in size (large enterprise wide systems) have a failure rate of some 65%.

The Capers Jones study uses data gathered during software assessments and benchmark studies that are the meat and gravy of Software Productivity Research (SPR). In addition to working on projects that never see the light of day, software personnel are involved in defect removal from projects that are eventually completed and defect repairs during the routine maintenance of applications already running. Table 1 shows the work pattern for a typical software engineer after vacations, training, sick days and all other activities other than time spent actually working on software projects are factored out. Jones estimates that only 47 working days of a full calendar year are available for actually developing or enhancing software applications. In contrast a total of 150 workdays are devoted to testing, defect repair or Y2K repair work.

Table 1. Software engineering effort by task
(Source: Capers Jones).

Jones admits that these figures have a high margin of error but they do serve to illustrate the scale of the problem. A very large number of personnel would be available if software quality could be more rigorously controlled. And highly publicized programmer labor shortages would be less severe.

Looking at the figures by occupation grouping within the spread of the many specialized occupations that comprise the overall software domain yields results that border the absurd. Table 2 compares the relative amounts of time available for new projects versus time spent on defect repairs and canceled projects. For occupations such as testing specialist and quality assurance, almost all of the effort is focused on defect repairs. The skillset of these specialists don't come cheap and the return for their efforts really ought to be realized through the prevention of defects emerging in the first place. Had they been in place-and being utilized, as they ought to be-when legacy systems were being coded with two digit date fields, how much would they have saved? Even for occupations such as development programmer or software architect the time spent dealing with software error rates, and canceled projects absorbs approximately half of the available personnel.

Table 2. US software work force
(Source: Capers Jones).

The overall conclusion that can be drawn from Table 2 is that 'wastage' and defect repairs absorb almost two-thirds of the US software workforce, leaving only about one-third for productive work on new projects. Once again it has to be asked, what other industry would get away with treating quality in this way?

Of course the figures in the Jones study are averages, with organizations having varying levels of quality practices. Jones compares the best with the worst, to illustrate the scope for improvement. Table 3 shows a range of results for lagging, average, and leading software projects from among the SPR client base. Errors in software requirements and software design documents are more frequent than errors in the source code itself. Not only are requirements and design errors more numerous, but they are also more severe and more difficult to remove. Front-end errors in requirements and design cannot be found and removed via testing, but instead need pre-test reviews and inspections. The figures suggest that there is ample room for improvement. The occurrence rate of bugs in average organizations is almost twice that found in best of breed shops. Further by improvements to both the occurrence rate of defects and the defect removal process there is scope for a reduction by a factor of five in the number of defects actually delivered by average organizations.

Note: Data in defects per function point.

Table 3. Software defect removal efficiency
(Source: Capers Jones).

What's important is that it's the combination of high levels of potential defects and low levels of defect removal efficiency that contributes to both canceled projects and to the dominance of error-related work patterns in the software community. More than twice as much effort is associated with defect removal than with actual product development.

Areas for improvement fall into four categories, in the following order of importance: People. Process. Methods. Tools.

'Top priority is to hire the best people, pay and support them well, motivate them, give them responsibility, and trust them,' according to Tom Welsh, a ComputerWire analyst and OO development specialist. He continues, 'Next is to have a solid and appropriate process [lifecycle, project management, etc.] then to use methods for analysis, design, testing, validation, and maintenance. Last is to acquire suitable software tools [IDEs, 4GLs, databases, compilers, networking, OS, etc.].' However attention is almost completely limited to the latest (hence, mostly unproven) tools, with a little interest in methods (e.g. UML, patterns). Best in class organizations tend to draw upon a common core of best of breed methodologies and tools and these include:

Risk analysis. As mentioned earlier, the Y2K fix is the most dramatic example of the wastage of software labor and is one which could have been avoided if risk analysis had been used to determine the likelihood of legacy applications enduring as long as they have done. Capers Jones sat upon the influential Airlie council task force set up by the US Department of Defense to draw up a list of best practices to ensure quality in software development. One of the top recommendations the council proposed was that a formal risk analysis report should be drawn up as part of the initial requirements of any project. This report should include amongst other things: information on the application's projected life expectancy and details of the application's connections to other applications.

Quality measurements. Jones notes in his study that one of the most striking differences between leading organizations and lagging ones is that the leaders collect statistics about their quality levels and user satisfaction levels. These quality measurements usually include the following:

• Software defect volumes measured from the requirements stage through the entire development lifecycle and beyond.
• Defect severity levels.
• Defect origins are measured so that problems with requirements, design, code and documents are known.

This data is collected rigorously on a daily, monthly, quarterly and annual basis in order that trend statistics can be analyzed.

Quality methods. What distinguishes leading organizations in terms of the methods used is made up of many years of working on the problem by those organizations. Two of the main methods that have proved to be highly effective though are:

• The use of formal inspections of design, code, and other deliverables to prevent and remove software defects.
• Software quality assurance groups (often >5% of total staff).

Quality culture. It is common practice for firms to hire young, inexperienced programmers at minimum rates, drive them hard and give them undue discretion. They also tend to have a focus that is almost purely quantative: generate as many windows or function points as possible. Taking too much time to resolve defects is considered wasteful and is not rewarded.

This is hardly conducive to fostering high quality products. Top management knows little or nothing about software, let alone software quality, while those who do have virtually no power to set policy. Add to this the prevalence of short-term contract developers who have every motivation to produce error-filled and badly structured code. This is in part because more errors (especially those which do not surface at an early stage after roll-out) mean more work. It is also because contract developers are often given unrealistic time scales for completion of a project; under these circumstances the difficulty in producing any product overwhelms the quality goals.

Quality tools. In our August and September issues last year Computer Finance looked into the value and effectiveness of a variety of automated test tools, which attacked the following processes:

• Predictive quality estimation
• Defect and quality measurement
• Test planning
• Software reliability predictive models
• Statistical analysis and reporting

We found that tools alone cannot do the job; they require processes, such as planning, careful test definition, and regression analyses, to produce results. Yet, more often than not, organizations that embraced testing tools tended to have above average development process.

Tools That Manage Tools. A new generation of management tools help development organizations manage the tools they already have. The need for them arises from the fact that having all the best tools that money can buy is no guarantee that they will get used, or used effectively. For instance, with developers under mounting pressure to meet rapid fire deadlines, they may not take the time to learn some of the less familiar tools in their arsenal, they might forget that their organization had a particular tool, or they may not have identified the right tool for the right quality test.

These tools, which are best described as 'software to manage software,' are used to guide development teams to the right tools. For instance, if a developer is creating a new module, it puts it in the correct libraries, names it accordingly, compiles, links and binds it the proper way, and basically prevents people from shooting themselves in the foot. Most tools or standards can be plugged in at some sensible point in the change cycle, in the same way. They include tools that restructure code, analyze complexity, debug, compare and merge code, or cleanse Y2K problems. Using this 'framework' software, when a module is being updated, it can automatically invoke code analyzers, or convert the code for COBOL 370, or whatever the developer wants to do. Similarly for Y2K renovation, tools can be plugged in, so the code gets inspected for compliance, error reports get generated, and renovation tools like VIA/Insight or VIA/Alliance, for instance, get used.

Developers don't have to know about these tools, or how to invoke them as they get called up automatically. Everyone from new hires to contractors is forced to do business the same way, and the company gets quality assurance and standards (and audit) enforcement proactively.