Realistic Expectations of Random Testing

Ms. Denise M. Woit
(DWOIT@scs.Ryerson.CA)

Software Engineering Research Group
CRL, McMaster University,
Hamilton, Ontario, Canada L8S 4K1


Abstract

There are varying opinions in the literature regarding the usefulness of random testing in the software development process. Several studies are available exploring random testing's effectiveness in general, and as compared to systematic testing. However, the results of these studies are diverse, and no clear conclusions about random testing are agreed upon by all. In this report, we attempt to consolidate the information available in the current random testing literature. We classify many of the random testing reports into various categories depending on the nature of the studies, and then evaluate the studies based upon their underlying theories and based upon their relation to reality. We form several hypotheses about the uses and usefulness of random testing in real-life, and use our evaluation to determine the extent to which these hypotheses are supported by the evidence available in the literature. Based upon our explorations, we are able to draw some final conclusions about the effectiveness of random testing in reality, and we are able to identify some directions for future research in this area.