Skip to main content
Journal of the Brazilian Computer Society
Download PDF

The marginal value of increased testing: An empirical analysis using four code coverage measures

Journal of the Brazilian Computer Society volume 12, pages 13–30 (2006)Cite this article

Abstract

This paper presents an empirical comparison of the growth characteristics of four code coverage measures, block, decision, c-use and p-use, as testing is increased. Due to the theoretical foundations underlying the lognormal software reliability growth model, we hypothesize that the growth for each coverage measure is lognormal. Further, since for a given program the breadth and the depth of the different coverage measures are similar, we expect that the parameters of the lognormal coverage growth model for each of the four coverage measures to be similar. We confirm these hypotheses using coverage data generated from extensive testing of an application which has 30 KLOC. We then discuss how the lognormal coverage growth function could be used to control the testing process and to guide decisions about when to stop testing, since it can provide an estimate of the marginal testing effort necessary to achieve a given level of improvement in the coverage.

References

  1. J. Aitchison and J.A.C. Brown, The Lognormal Distribution, Cambridge University Press, NY, 1969.

    Google Scholar 

  2. H. Akaike, Prediction and Entropy, MRC Technical Summary Report #2397, NTIS, Springfield, VA, 1982.

    Google Scholar 

  3. P. Bishop and R. Bloomfield, Using a Log-normal Failure Rate Distribution for Worst Case Bound Reliability Prediction. InProceedings of International. Symposium on Software Reliability Engineering, pages 237–245, 2003.

  4. L. Briand and D. Pfahl, Using simulation for assessing the real impact of test coverage on defect coverage,IEEE Transactions on Reliability, 49(1): 60–70, 2000.

    Article  Google Scholar 

  5. M. H. Chen, M. R. Lyu and W. E. Wong, Effect of code coverage on software reliability measurement,IEEE Transactions on Reliability, 50(2): 165–170, 2001.

    Article  Google Scholar 

  6. L. Clarke, A. Podgurski, D. J. Richardson and S. J. Zeil, A formal evaluation of data flow path selection criteria,IEEE Transactions on Software Engineering, 15(11): 1318–1332, 1989.

    Article  Google Scholar 

  7. E.L. Crow and K. Shimizu, ed., Lognormal Distributions: Theory and Applications, Marcel Dekker, NY, 1988.

    MATH  Google Scholar 

  8. S. R. Dalal, J. R. Horgan and J. R. Kettenring, Reliable software and communications: Software quality, reliability and safety. InProceedings of International. Conference on Software Engineering, pages 425–435, 1993.

  9. E. Diaz, J. Tuya and R. Blanco, A modular tool for automated coverage in software testing. InProceedings of Eleventh Annual International Workshop on Software Technology and Engineering Practice, pages 241–246, 2003.

  10. P. Frankl and P. J. Weiss, An experimental comparison of the effectiveness of branch testing and data flow testing,IEEE Transactions on Software Engineering, 19(8): 747–787, 1993.

    Article  Google Scholar 

  11. A. L. Goel and K. Okumoto, Time-dependent error detection rate models for software reliability and other performance measures,IEEE Transactions on Reliability, R-28(3): 206–211, 1979.

    Article  Google Scholar 

  12. S. Gokhale and R. Mullen, From test count to code coverage using the Lognormal. InProceedings of International Symposium on Software Reliability Engineering, pages 295–395, 2004.

  13. S. Gokhale and K. S. Trivedi, A time/structure based software reliability model,Annals of Software Engineering, 8: 85–121, 1999.

    Article  Google Scholar 

  14. M. Grottke, A vector Markov model for structural coverage growth and number of failure occurrences. InProceedings of International Symposium on Software Reliability Engineering, pages 304–315, 2002.

  15. H. Hirose, Estimation of threshold stress in accelerated life-testing,IEEE Transactions on Reliability, 42(4): 650–657, 1993.

    Article  MATH  MathSciNet  Google Scholar 

  16. M. Hutchings, T. Goradia and T. Ostrand, Experiments on the effectiveness of control-flow and data-flow based test adequacy criteria. InProceedings of International Conference on Software Engineering, pages 191–200, 1994.

  17. J.R. Horgan and S.A. London, Dataflow Coverage and the C Language. InProceedings of Fourth International Symposium on Testing, Analysis and Verification, pages 87–97, 1991.

  18. L. Johnson, S. Kotz, and N. Balakrishnan, Continuous Univariate Distributions, vol. 1, Wiley, New York, 1994.

    MATH  Google Scholar 

  19. J. A. Jones and M. J. Harrold, Test-suite reduction and prioritization for modified condition/decision coverage,IEEE Transactions. on Software Engineering, 29(3): 195–209, 2003.

    Article  Google Scholar 

  20. P. A. Keiller and D. R. Miller, On the use and performance of software reliability growth models,Reliability Engineering and Systems Safety, 32: 95–117, 1991.

    Article  Google Scholar 

  21. M. R. Lyu, J. R. Horgan and S. London, A coverage analysis tool for the effectiveness of software testing,IEEE Transactions on Reliability, 43 (4): 527–535, 1994.

    Article  Google Scholar 

  22. Y. K. Malaiya, N. Li, J. Beiman and R. Karcich, Software reliability growth with test coverage,IEEE Transactions on Reliability, 51(4): 420–426, 2002.

    Article  Google Scholar 

  23. Y. Malaiya, N. Li, J. Beiman et al., The relationship between test coverage and reliability,In Proceedings of International Symposium on Software Reliability Engineering, pages 186–195, 1994.

  24. D.R. Miller, Exponential Order Statistic Models of Software Reliability Growth, NASA Contractor Report 3909, NTIS, Springfield, VA 22161, 1985.

    Google Scholar 

  25. S. Misra, Evaluating four white-box test coverage methodologies. InProceedings of IEEE Canadian Conference on Electrical and Computer Engineering, pages 1739–1743, 2003.

  26. R.E. Mullen, The Lognormal distribution of software failure rates: Application to software reliability growth modeling. InProceedings of International Symposium on Software Reliability Engineering, pages 134–142, 1998.

  27. R.E. Mullen, The Lognormal distribution of software failure rates: Origin and evidence. InProceedings of International Symposium on Software Reliability Engineering, pages 124–133, 1998.

  28. J. D. Musa, A theory of software reliability and its application,IEEE Transactions on Software Engineering, SE-1(1): 312–317, 1975.

    Google Scholar 

  29. R. Mullen and S. Gokhale, Software defect rediscoveries: A Discrete Lognormal model. InProceedings of International Symposium on Software Reliability Engineering, pages 203–212, 2005.

  30. R. Mullen and S. Gokhale, A Discrete Lognormal model for software defects affecting QoP. In Quality of Protection: Security Measurements and Metrics, D. Gollmann and F. Massacci and A. Yautsiukhin (Eds.), pages 37–48, Advances in Information Security Series, Springer Verlag, 2006.

  31. W. Nelson, Accelerated Testing, Wiley, New York, 1990.

    Book  Google Scholar 

  32. P. Netisopakul, L. J. White, and J. Morris, Data coverage testing. InProceedings of Asia Pacific Software Engineering Conference, pages 465–472, 2002.

  33. P. Piworaski, M. Ohba and J. Caruso, Coverage measurement experience during functional test. InProceedings of International Conference on Software Engineering, pages 287–293, 1993.

  34. S. Rapps and E. J. Weyuker, Selecting test data using data flow information,IEEE Transasctions on Software Engineering, SE-11(4): 367–375, 1985.

    Article  Google Scholar 

  35. Y. Sakamoto, M. Ishiguro, and G. Kitagawa, Akaike Information Criterion Statistics, D.Reidel, Holland, 1986.

    MATH  Google Scholar 

  36. R. A. Sahner, K. S. Trivedi and A. Puliafito, Performance and Reliability Analysis of Computer Systems: An Example-Based Approach Using the SHARPE Software Package, Kluwer Academic Publishers, Boston, 1996.

    MATH  Google Scholar 

  37. S. K. S. Sze and M. R. Lyu, ATACOBOL — a COBOL test coverage analysis tool and its applications. InProceedings of International Symposium on Software Reliability Engineering, pages 327–335, 2000.

  38. K. S. Trivedi, Probability and Statistics with Reliability, Queuing and Computer Science Applications, John Wiley and Sons, New York, 2001.

    Google Scholar 

  39. M. Vouk, Using reliability models during testing with non operational profiles. InProceedings of Second Bellcore/Purdue workshop on Issues in Software Reliability Estimation, pages 103–111, 1992.

  40. W. E. Wong, J. R. Horgan, S. London and A. P. Mathur, Effect of test set size and block coverage on fault detection effectiveness. InProceedings of International Symposium on Software Reliability Engineering, pages 230–238, 1994.

Download references

Author information

Authors and Affiliations

  1. Dept. of Computer Science & Engineering, Univ. of Connecticut, 06269, Storrs, CT

    Swapna S. Gokhale

  2. Cisco Systems, New England Development Center, 1414 Massachusetts Avenue, 01719, Boxborough, MA

    Robert E. Mullen

Authors
  1. Swapna S. Gokhale
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert E. Mullen
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Gokhale, S.S., Mullen, R.E. The marginal value of increased testing: An empirical analysis using four code coverage measures. J Braz Comp Soc 12, 13–30 (2006). https://doi.org/10.1007/BF03194493

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF03194493

Keywords