Bank of Montreal
In the US alone, up to 60% of software developers are involved in fixing errors. We examine results from a recent Capers Jones study and discuss the implications for improving the quality of the software quality process.
Issue Date: 01-11-98
Issue Number: 9.06
Category: SOFTWARE
Quality assurance in software development is a problem that has been kicking around for almost a half century. The first bug was an insect caught in an early, 1940s computer. Things have hardly gotten better since then. A recent task force headed up by Howard Rubin and including Capers Jones, chairman of Software Productivity Research of Burlington, Mass., found that fewer than 6% of organizations have clearly defined software management processes in place. Would any other industry get away with treating quality in this way? The current chronic shortage of software systems personnel though, together with the scramble to achieve Y2K readiness (probably the biggest fix in software history), is beginning to lift the issue of quality higher up the agenda.
In a separate paper published earlier this year, Capers Jones analyzed the link between poor quality and the number of canceled projects with the software labor shortage. In addition to that 60% estimate of the amount of time developers spent on software repair, he found that at any one time, 15% of the global software workforce is involved in projects that will never be deployed. Last year, we published a two-part series which looked at the role of automated software quality testing tools, and how they impact the quality assurance process. In this report, we take a broader look at the quality process itself, examining the figures in the Capers Jones study, and its implications for how to improve the process.
AVOIDABLE LABOR
The process of defect removal is long and labor intensive. And it's not flashy. All too often, project teams focus on the 'visible sides' of software projects: the functionality and user interface, and developers are compensated accordingly. Yet few reward good quality. All too often, quality-conscious developers are accused of working too slowly.
The problem is: software quality assurance remains labor-intensive, even with automated tools that eliminated a portion of the legwork. Things only get worse with size; as projects get bigger, they get more complex, providing a fertile breeding ground for bugs. Capers Jones estimates that projects of 100,000 function points in size (large enterprise wide systems) have a failure rate of some 65%.
The Capers Jones study uses data gathered during software assessments and benchmark studies that are the meat and gravy of Software Productivity Research (SPR). In addition to working on projects that never see the light of day, software personnel are involved in defect removal from projects that are eventually completed and defect repairs during the routine maintenance of applications already running. Table 1 shows the work pattern for a typical software engineer after vacations, training, sick days and all other activities other than time spent actually working on software projects are factored out. Jones estimates that only 47 working days of a full calendar year are available for actually developing or enhancing software applications. In contrast a total of 150 workdays are devoted to testing, defect repair or Y2K repair work.
| Activities | Workdays | Percent |
| Testing and defect repairs | 70 | 35.53% |
| Year 2000 and similar repairs | 50 | 25.38% |
| Time on canceled projects | 30 | 15.23% |
| Productive time on projects | 47 | 23.86% |
| Total | 197 | 100% |
Table 1. Software engineering effort by task
(Source: Capers Jones).
Jones admits that these figures have a high margin of error but they do serve to illustrate the scale of the problem. A very large number of personnel would be available if software quality could be more rigorously controlled. And highly publicized programmer labor shortages would be less severe.
Looking at the figures by occupation grouping within the spread of the many specialized occupations that comprise the overall software domain yields results that border the absurd. Table 2 compares the relative amounts of time available for new projects versus time spent on defect repairs and canceled projects. For occupations such as testing specialist and quality assurance, almost all of the effort is focused on defect repairs. The skillset of these specialists don't come cheap and the return for their efforts really ought to be realized through the prevention of defects emerging in the first place. Had they been in place-and being utilized, as they ought to be-when legacy systems were being coded with two digit date fields, how much would they have saved? Even for occupations such as development programmer or software architect the time spent dealing with software error rates, and canceled projects absorbs approximately half of the available personnel.
| Software occupation groups | Number employed | Staff for repairs | Staff for new work | Percent of total |
| Programmer/analyst | 400,000 | 240,000 | 160,000 | 40.00% |
| Programmer (maintenance) | 350,000 | 297,500 | 52,500 | 15.00% |
| Programmer (development) | 275,000 | 151,250 | 123,750 | 45.00% |
| Project manager (1st level) | 225,000 | 123,750 | 101,250 | 45.00% |
| Software engineer (systems) | 200,000 | 130,000 | 70,000 | 35.00% |
| Testing specialist | 125,000 | 112,500 | 12,500 | 10.00% |
| Systems analyst | 100,000 | 40,000 | 60,000 | 60.00% |
| Software engineer (real-time) | 75,000 | 45,000 | 30,000 | 40.00% |
| Software technical writer | 75,000 | 15,000 | 60,000 | 80.00% |
| Software engineer (embedded) | 70,000 | 45,500 | 24,500 | 35.00% |
| Data administration specialist | 50,000 | 25,000 | 25,000 | 50.00% |
| Project manager (2nd level) | 35,000 | 17,500 | 17,500 | 50.00% |
| Software Quality Assurance | 25,000 | 22,500 | 2,500 | 10.00% |
| Configuration control specialist | 15,000 | 9,000 | 6,000 | 40.00% |
| Performance specialists | 7,500 | 5,250 | 2,250 | 30.00% |
| Project manager (3rd level) | 5,000 | 2,500 | 2,500 | 50.00% |
| Software Architect | 1,500 | 600 | 900 | 60.00% |
| TOTAL | 2,034,000 | 1,282,850 | 751,150 | 36.93% |
Table 2. US software work force
(Source: Capers Jones).
The overall conclusion that can be drawn from Table 2 is that 'wastage' and defect repairs absorb almost two-thirds of the US software workforce, leaving only about one-third for productive work on new projects. Once again it has to be asked, what other industry would get away with treating quality in this way?
WHERE THE EFFORT GOES
Of course the figures in the Jones study are averages, with organizations having varying levels of quality practices. Jones compares the best with the worst, to illustrate the scope for improvement. Table 3 shows a range of results for lagging, average, and leading software projects from among the SPR client base. Errors in software requirements and software design documents are more frequent than errors in the source code itself. Not only are requirements and design errors more numerous, but they are also more severe and more difficult to remove. Front-end errors in requirements and design cannot be found and removed via testing, but instead need pre-test reviews and inspections. The figures suggest that there is ample room for improvement. The occurrence rate of bugs in average organizations is almost twice that found in best of breed shops. Further by improvements to both the occurrence rate of defects and the defect removal process there is scope for a reduction by a factor of five in the number of defects actually delivered by average organizations.
| Task | Leading | Average | Lagging |
| Requirements | 0.55 | 1.00 | 1.45 |
| Design | 0.75 | 1.25 | 1.90 |
| Coding | 1.00 | 1.75 | 2.35 |
| User manuals | 0.40 | 0.60 | 0.75 |
| Bad fixes | 0.10 | 0.40 | 0.85 |
| Total | 2.80 | 5.00 | 7.30 |
| Removal % | 95% | 85% | 75% |
| Delivered | 0.14 | 0.75 | 1.83 |
Note: Data in defects per function point.
Table 3. Software defect removal efficiency
(Source: Capers Jones).
What's important is that it's the combination of high levels of potential defects and low levels of defect removal efficiency that contributes to both canceled projects and to the dominance of error-related work patterns in the software community. More than twice as much effort is associated with defect removal than with actual product development.
THE ROAD TO RECOVERY
Areas for improvement fall into four categories, in the following order of importance: People. Process. Methods. Tools.
'Top priority is to hire the best people, pay and support them well, motivate them, give them responsibility, and trust them,' according to Tom Welsh, a ComputerWire analyst and OO development specialist. He continues, 'Next is to have a solid and appropriate process [lifecycle, project management, etc.] then to use methods for analysis, design, testing, validation, and maintenance. Last is to acquire suitable software tools [IDEs, 4GLs, databases, compilers, networking, OS, etc.].' However attention is almost completely limited to the latest (hence, mostly unproven) tools, with a little interest in methods (e.g. UML, patterns). Best in class organizations tend to draw upon a common core of best of breed methodologies and tools and these include:
Risk analysis. As mentioned earlier, the Y2K fix is the most dramatic example of the wastage of software labor and is one which could have been avoided if risk analysis had been used to determine the likelihood of legacy applications enduring as long as they have done. Capers Jones sat upon the influential Airlie council task force set up by the US Department of Defense to draw up a list of best practices to ensure quality in software development. One of the top recommendations the council proposed was that a formal risk analysis report should be drawn up as part of the initial requirements of any project. This report should include amongst other things: information on the application's projected life expectancy and details of the application's connections to other applications.
Quality measurements. Jones notes in his study that one of the most striking differences between leading organizations and lagging ones is that the leaders collect statistics about their quality levels and user satisfaction levels. These quality measurements usually include the following:
- Software defect volumes measured from the requirements stage through the entire development lifecycle and beyond.
- Defect severity levels.
- Defect origins are measured so that problems with requirements, design, code and documents are known.
This data is collected rigorously on a daily, monthly, quarterly and annual basis in order that trend statistics can be analyzed.
Quality methods. What distinguishes leading organizations in terms of the methods used is made up of many years of working on the problem by those organizations. Two of the main methods that have proved to be highly effective though are:
- The use of formal inspections of design, code, and other deliverables to prevent and remove software defects.
- Software quality assurance groups (often >5% of total staff).
Quality culture. It is common practice for firms to hire young, inexperienced programmers at minimum rates, drive them hard and give them undue discretion. They also tend to have a focus that is almost purely quantative: generate as many windows or function points as possible. Taking too much time to resolve defects is considered wasteful and is not rewarded.
This is hardly conducive to fostering high quality products. Top management knows little or nothing about software, let alone software quality, while those who do have virtually no power to set policy. Add to this the prevalence of short-term contract developers who have every motivation to produce error-filled and badly structured code. This is in part because more errors (especially those which do not surface at an early stage after roll-out) mean more work. It is also because contract developers are often given unrealistic time scales for completion of a project; under these circumstances the difficulty in producing any product overwhelms the quality goals.
Quality tools. In our August and September issues last year Computer Finance looked into the value and effectiveness of a variety of automated test tools, which attacked the following processes:
- Predictive quality estimation
- Defect and quality measurement
- Test planning
- Software reliability predictive models
- Statistical analysis and reporting
We found that tools alone cannot do the job; they require processes, such as planning, careful test definition, and regression analyses, to produce results. Yet, more often than not, organizations that embraced testing tools tended to have above average development process.
Tools That Manage Tools. A new generation of management tools help development organizations manage the tools they already have. The need for them arises from the fact that having all the best tools that money can buy is no guarantee that they will get used, or used effectively. For instance, with developers under mounting pressure to meet rapid fire deadlines, they may not take the time to learn some of the less familiar tools in their arsenal, they might forget that their organization had a particular tool, or they may not have identified the right tool for the right quality test.
These tools, which are best described as 'software to manage software,' are used to guide development teams to the right tools. For instance, if a developer is creating a new module, it puts it in the correct libraries, names it accordingly, compiles, links and binds it the proper way, and basically prevents people from shooting themselves in the foot. Most tools or standards can be plugged in at some sensible point in the change cycle, in the same way. They include tools that restructure code, analyze complexity, debug, compare and merge code, or cleanse Y2K problems. Using this 'framework' software, when a module is being updated, it can automatically invoke code analyzers, or convert the code for COBOL 370, or whatever the developer wants to do. Similarly for Y2K renovation, tools can be plugged in, so the code gets inspected for compliance, error reports get generated, and renovation tools like VIA/Insight or VIA/Alliance, for instance, get used.
Developers don't have to know about these tools, or how to invoke them as they get called up automatically. Everyone from new hires to contractors is forced to do business the same way, and the company gets quality assurance and standards (and audit) enforcement proactively.
| A number of vendors have products of this sort either on the market or soon to be released. Compuware's QADirector (formerly known as Acqua, acquired from Centerline) is due for release this month. Meanwhile, Softlab's Enabler suite has recently been shipped. On the other hand, Benchmark Technologies' ISPW (Integrated Software Processing Workframe) has been around for almost ten years and some user feedback is available. These tools aren't cheap, however; for instance, ISPW is priced at $30,000 for a 25 user license. That said, the savings can be significant: A detailed cost-benefit analysis carried out by one ISPW user estimated that it saved an application team of 90 in excess of $1 million per year of operation. Excluding ISPW, management tools for test deployment are brand new; in a future report, we will look at user experiences. For this report, we focus on the experiences of one ISPW user, Bank of Montreal, to provide an idea of potential savings.
The bank, one of Canada's largest, with around 1,200 branches and a 500-person IS division, has used ISPW for almost eight years. Mark Neilson, Systems Specialist at the bank, who oversaw the introduction of the system at that time, says that ISPW has proved to be faster and more reliable than manual methods. For example, for configuration management, the bank used CA's Endeavor (a tool that tracks and manages different versions of an application), but at that time the system had a lot of problems with compatibility. 'We are now able to maintain the system internally with non-technical personnel, representing a saving on the CA solution that required a systems manager at a rate of around $50,000,' said Neilson. Also administration costs for the ISPW system have been reduced since it too can be operated using non-technical personnel. Overall the bank has seen a headcount reduction of 11 application administration staff representing a 69% reduction. The bank also found that new releases of applications to accommodate structural changes to banking products could be brought to market in only 22 hours, where previously it could take have taken a couple of weeks. At one time this represented significant competitive advantage to the bank. But, Neilson reported, the rest of the field has caught up. 'Most competitor banks have now using the same or similar tools,' he said. Although we found that few organizations had even heard of 'tools that manage tools', banks do generally have a better reputation for being ahead of the curve. In fact, in our previous reports on quality testing tools, the overwhelming majority of users came from the financial industry. Neilson notes that training requirements are fairly modest; for instance, new users only require about an hour to get up to speed since they don't need to know about every single type of test file or application. At present the bank is midway through a major reorganization, moving toward a business unit from a departmental model. Since ISPW developers are faced with the same front end, regardless of the technology, the savings are substantial. Developers do not have to retrain on unfamiliar systems when they move to a new location. Neither do they have to learn a new methodology as the system automatically invokes the right tool at the right time. Today at the bank, ISPW acts as a framework that supports the use of many development tools. 'Maintenance would not be possible for a group with as many modules as we have,' said Nielson. The bank has tens of thousands of written modules and tens of millions lines of code. 'It would be impossible to manage without an internally-developed tool, or ISPW,' he said adding 'The ISPW system has become one of our critical systems that has to be up and running; otherwise, we could not do development.' |
YOU GET WHAT YOU PAY FOR
The Capers Jones' findings demonstrate the degree of waste that currently permeates the software development life cycle. It reveals that waste reduction is a tantalizingly fat target; what's surprising is that, with IT organizations facing either shortage of skilled labor pools or arbitrary headcount caps, that more have not tried attacking the efficiency of the core process first. In an upcoming report, we'll look at another strategy to improve development productivity: use of component-based development techniques. Again, we won't be surprised to discover similar findings: that strapped IT organizations generally shy away from making structural investments that in the long run could improve productivity.
Then again, may be we might be surprised.
? 1998 ComputerWire Inc
