Committee on Certifiably Dependable Software Systems & Computer Science and Telecommunications Board 
Software for Dependable Systems [PDF ebook] 
Sufficient Evidence?

The focus of Software for Dependable Systems is a set of fundamental principles that underlie software system dependability and that suggest a different approach to the development and assessment of dependable software.Unfortunately, it is difficult to assess the dependability of software. The field of software engineering suffers from a pervasive lack of evidence about the incidence and severity of software failures; about the dependability of existing software systems; about the efficacy of existing and proposed development methods; about the benefits of certification schemes; and so on. There are many anecdotal reports, whichalthough often useful for indicating areas of concern or highlighting promising avenues of researchdo little to establish a sound and complete basis for making policy decisions regarding dependability. The committee regards claims of extraordinary dependability that are sometimes made on this basis for the most critical of systems as unsubstantiated, and perhaps irresponsible. This difficulty regarding the lack of evidence for system dependability leads to two conclusions: (1) that better evidence is needed, so that approaches aimed at improving the dependability of software can be objectively assessed, and (2) that, for now, the pursuit of dependability in software systems should focus on the construction and evaluation of evidence.The committee also recognized the importance of adopting the practices that are already known and used by the best developers; this report gives a sample of such practices. Some of these (such as systematic configuration management and automated regression testing) are relatively easy to adopt; others (such as constructing hazard analyses and threat models, exploiting formal notations when appropriate, and applying static analysis to code) will require new training for many developers. However valuable, though, these practices are in themselves no silver bullet, and new techniques and methods will be required in order to build future software systems to the level of dependability that will be required.