Software Reliability Using Inverse Sampling
Dr. Balwant Singh, Dr. Roman Viveros, Dr. David Parnas
Software Engineering Research Group
CRL, McMaster University,
Hamilton, Ontario, Canada L8S 4K1
This paper addresses one of the perpetual questions faced by
Software developers, "How much testing is enough testing?".
Software testing accounts for a substantial portion of software
development costs, but releasing software with unacceptable reliability
is also very costly. We begin with a discussion of how classical notions
of reliability can be applied to deterministic software and explain that
the reliability of software is as much a function of the way that it is used
as of quality of the software. We then illustrate how simple operational
profiles can be used to characterize usage patterns. Finally, we show how
the reliability requirements can be used to determine how much testing is
and whether or not the software is acceptable.
Using the method of inverse sampling, also called negative binomial
sampling, we develop an efficient statistical procedure for quantifying
the reliability of
a piece of software. The procedure allows substantial reductions in the
average number of executions run over the traditional binomial testing.
Other issues such as the calculation of upper confidence bounds for software
failure rate under both binomial and negative binomial sampling are also
addressed. The results obtained are illustrated numerically and graphically
on several cases arising in practice. Some issues for further work, namely
the use of sequential testing, the computer implementation of the methods
developed and the testing of continuously-run software, are also discussed.