Skip to main content
Journal of the Brazilian Computer Society
Download PDF

Muta-Pro: Towards the definition of a mutation testing process

Journal of the Brazilian Computer Society volume 12, pages 49–61 (2006)Cite this article

Abstract

Mutation Testing originated from a classical method for digital circuit testing and today is used at program and specification levels. It can be used either to generate or to assess the quality of test sets. In spite of being very effective in detecting faults, Mutation Testing is usually considered a high cost criterion due to: i) the large number of generated mutants; ii) the time-consuming activity of determining equivalent mutants; and iii) the mutant execution time. Many initiatives aiming at reducing the Mutation Testing application cost have been conducted, most of them addressing one of the drawbacks mentioned above.

In this paper, we identify and summarize some of the most relevant researches and results related to Mutation Testing cost reduction, e.g., Constrained-Mutation, Constraint-Based Testing and Bayesian Learning. Moreover, we propose a Mutation Testing process, namedMuta-Pro, that synergetically integrates the related approaches and mechanisms. This process is intended to be incremental and tailorable to a specific application domain such as C programs or finite state machine models. The main ideas in this paper are illustrated using a UNIX utility program.

This process is being integrated in a Mutation Testing environment, based on the authors’ previous experience on implementing theProteum Family tools, aiming at promoting the technology transfer to industry and providing the basis for improving theMuta-Pro process itself.

References

  1. A. T. Acree.On Mutation. PhD thesis, Georgia Institute of Technology, Atlanta, GA, 1980.

    Google Scholar 

  2. H. Agrawal, R. A. DeMillo, R. Hathaway, W. Hsu, W. Hsu, E. W. Krauser, R. J. Martin, A. P. Mathur, and E. H. Spafford. Design of mutant operators for the C programming language. Technical Report SERC-TR41-P, Software Engineering Research Center, Purdue University, West Lafayette, IN, Mar. 1989.

    Google Scholar 

  3. A. V. Aho, R. Sethi, and J. D. Ullman.Compilers: Principles, Techniques and Tools. Addison Wesley, 1996.

  4. E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. Towards the determination of sufficient mutant operators for C.STVR — Software Testing, Verification and Reliability, 11(2):113–136, June 2001.

    Article  Google Scholar 

  5. J.M. Bieman, S. Ghosh, and R. T. Alexander. A technique for mutation of Java objects. In16th IEEE International Conference on Automated Software Engineering, pages 23–26, San Diego, CA, Nov. 2001.

  6. L. Bottaci. A genetic algorithm fitness function for mutation testing. InSEMINAL’2001 — First International Workshop on Software Engineering using Metaheuristic INnovative ALgorithms, Toronto, Ontario, Canada, May 2001.

  7. P. M. S. Bueno and M. Jino. Automated test data generaton for program paths using genetic algorithms. In13th International Conference on Software Engineering & Knowledge Engineering — SEKE’2001, pages 2–9, Buenos Aires, Argentina, June 2001.

  8. L. A. Clarke. A system to generate test data and symbolically execute programs.IEEE Transactions on Software Engineering, 2(3):215–222, Sept. 1976.

    Article  Google Scholar 

  9. M. Delamaro and J. Maldonado. Interface mutation: Assessing testing quality at interprocedural level. In19th International Conference of the Chilean Computer Science Society (SCCC’99), pages 78–86, Talca — Chile, Nov. 1999.

  10. M. E. Delamaro and J. C. Maldonado. Proteum — a tool for the assessment of test adequacy for C programs. InConference on Performability in Computing Systems (PCS’96), pages 79–95, East Brunswick, NJ, July 1996.

  11. M. E. Delamaro, J. C. Maldonado, and A. P. Mathur. Interface mutation: An approach for integration testing.IEEE Transactions on Software Engineering, 27(3):228–247, Mar. 2001.

    Article  Google Scholar 

  12. M. E. Delamaro, J. C. Maldonado, and A. M. R. Vincenzi. Proteum/IM 2.0: An integrated mutation testing environment. InMutation 2000 Symposium, pages 91–101, San Jose, CA, Oct. 2000. Kluwer Academic Publishers.

    Google Scholar 

  13. R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer.IEEE Computer, 11(4):34–43, Apr. 1978.

    Google Scholar 

  14. R. A. DeMillo and A. J. Offutt. Constraint based automatic test data generation.IEEE Transactions on Software Engineering, 17(9):900–910, Sept. 1991.

    Article  Google Scholar 

  15. J. W. Duran and S. C. Ntafos. An evaluation of random testing.IEEE Transactions on Software Engineering, 10(4), July 1984.

  16. S. C. P. F. Fabbri, J. C. Maldonado, P. C. Masiero, and M. E. Delamaro. Mutation analysis testing for finite state machines. In5th International Symposium on Software Reliability Engineering (ISSRE’94), pages 220–229, Monterey-CA, Nov. 1994. IEEE Computer Society Press.

    Google Scholar 

  17. S. C. P. F. Fabbri, J. C. Maldonado, P. C. Masiero, and M. E. Delamaro. Mutation analysis applied to validate specifications based on petri nets. InFORTE’95 — 8th IFIP Conference on Formal Descriptions Techniques for Distribute Systems and Communication Protocols, pages 329–337, Montreal, Canada, Oct. 1995. Kluwer Academic Publishers.

    Google Scholar 

  18. A. D. Friedman.Logical Design of Digital Systems. Computer Science Press, 1975.

  19. S. Ghosh and A. P. Mathur. Interface mutation.STVR — Software Testing, Verification and Reliability, 11(4):227–247, Dec. 2001. (Special Issue: Mutation 2000 — A Symposium on Mutation Testing. Issue Edited by W. Eric Wong).

  20. M. Harman, R. Hierons, and S. Danicic. The relationship between program dependence and mutation analysis. InMutation 2000 Symposium, pages 5–12, San Jose, CA, Oct. 2000. Kluwer Academic Publishers.

    Google Scholar 

  21. R. M. Hierons, M. Harman, and S. Danicic. Using program slicing to assist in the detection of equivalent mutants.STVR — Software Testing, Verification and Reliability, 9(4):233–262, 1999.

    Article  Google Scholar 

  22. D. Jackson and E. J. Rollins. Chopping: A generalization of slicing. Technical Report CMU-CS-94-169, School of Computer Science — Carnegie Mellon University, Pittsburgh, PA, July 1994.

    Google Scholar 

  23. B. F. Jones, D. E. Eyres, and H. H. Sthamer. A strategy for using genetic algorithms to automate branch and fault-based testing.The Computer Journal, 41(2):98–107, 1998.

    Article  Google Scholar 

  24. R. F. Jorge, A. M. R. Vincenzi, M. E. Delamaro, and J. C. Maldonado. Teste de mutaço: Estratégias baseadas em equivalência de mutantes para reduço do custo de aplicaço. InCLEI’2001 — XXVII Latin-American Conference on Informatics, Mérida — Venezuela, May 2001. (available in CD-ROM: article number — a202.pdf).

  25. M. Kim. Design of a mutation testing tool for C. Department of Computer Sciences, Purdue University, Apr. 1992.

  26. S. Kim, J. A. Clark, and J. A. Mcdermid. The rigorous generation of Java mutation operators using HAZOP. In12th International Conference on Software & Systems Engineering and their Applications (ICSSEA’99), Dec. 1999.

  27. S. Kim, J. A. Clark, and J. A. Mcdermid. Class mutation: Mutation testing for object-oriented programs. InObject-Oriented Software Systems — OOSS, 2000. Disponvel em: http://www.cs.york.ac.uk/~jac/. Acesso em: 01/03/2004.

  28. B. Korel. Automated software test data generation.IEEE Transactions on Software Engineering, 16(8):870–879, Aug. 1990.

    Article  Google Scholar 

  29. A. P. Mathur. Performance, effectiveness and reliability issues in software testing. In15th Annual International Computer Software and Applications Conference, pages 604–605, Tokio, Japan, Sept. 1991. IEEE Computer Society Press.

    Google Scholar 

  30. C. Michael and G. McGraw. Opportunism and diversity in automated software test data generation. Technical Report RSTR-003-97-13, RST Corporation, Sterling, VA, Dec. 1997.

    Google Scholar 

  31. T. Mitchell.Machine Learning. McGraw-Hill, New York, NY, 1997.

    MATH  Google Scholar 

  32. E. Mresa and L. Bottaci. Efficiency of mutation operators and selective mutation strategies: an empirical study.The Journal of Software Testing, Verification and Reliability, 9(4):205–232, Dec. 1999.

    Article  Google Scholar 

  33. A. J. Offutt and W. M. Craft. Using compiler optimization techniques to detect equivalent mutants.STVR — Software Testing, Verification and Reliability, 4:131–154, 1994.

    Article  Google Scholar 

  34. A. J. Offutt, Z. Jin, and J. Pan. The dynamic domain reduction approach to test data generation.Software Practice and Experience, 29(2):167–193, Jan. 1999.

    Article  Google Scholar 

  35. A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. An experimental determination of sufficient mutant operators.ACM Transactions on Software Engineering Methodology, 5(2):99–118, Apr. 1996.

    Article  Google Scholar 

  36. A. J. Offutt and J. Pan. Detecting equivalent mutants and the feasible path problem. InCOMPASS’96— In Annual Conference on Computer Assurance, pages 224–236, Gaithersburg, MD, June 1996. IEEE Computer Society Press.

    Chapter  Google Scholar 

  37. A. J. Offutt and J. Pan. Automatically detecting equivalent mutants and infeasible paths.STVR — Software Testing, Verification and Reliability, 7(3):165–192, Sept. 1997.

    Article  Google Scholar 

  38. R. P. Pargas, M. J. Harrold, and R. Peck. Test-data generation using genetic algorithms.STVR — Software Testing, Verification and Reliability, 9(4):263–282, 1999.

    Article  Google Scholar 

  39. R. S. Pressman.Software Engineering — A Practitioner’s Approach. McGraw-Hill, 5 edition, 2001.

  40. R. L. Probert and F. Guo. Mutation testing of protocols: Principles and preliminary experimental results. InIFIP TC6 — Third International Workshop on Protocol Test Systems, pages 57–76. North-Holland, 1991.

  41. S. Rapps and E. J. Weyuker. Selecting software test data using data flow information.IEEE Transactions on Software Engineering, 11(4):367–375, Apr. 1985.

    Article  Google Scholar 

  42. D. T. Ross. Structured analysis (sa): A language for communicating ideas.IEEE Transactions on Software Engineering, 3(1):16–34, Jan. 1977.

    Article  Google Scholar 

  43. A. S. Simo and J. C. Maldonado. MuDeL: A language and a system for describing and generating mutants. InXV SBES — Simpsio Brasileiro de Engenharia de Software, pages 240–255, Rio de Janeiro, Brasil, Oct. 2001.

  44. A. S. Simo, A. M. R. Vincenzi, J. C. Maldonado, and A. C. L. Santana. Software product instrumentation description. Technical Report 157, Instituto de Cincias Matemáticas e de Computaço — ICMC-USP, Mar. 2002.

  45. T. Sugeta. Proteum-rs/st: Uma ferramenta para apoiar a validaço de especificações statecharts baseada na análise de mutantes. Master’s thesis, ICMC-USP, So Carlos, SP, Nov. 1999.

    Google Scholar 

  46. T. Sugeta, J. C. Maldonado, and W. E. Wong. Mutation testing applied to validate SDL specifications. In16th IFIP International Conference on Testing of Communicating Systems — TestCom2004, pages 193–208, Oxford, United Kingdom, Mar. 2004. Springer.

  47. N. Tracey, J. Clark, K. Mander, and J. McDermid. Automated test-data generation for exception conditions.Software Practice and Experience, 30:61–79, 2000.

    Article  Google Scholar 

  48. R. Untch, M. J. Harrold, and J. Offutt. Mutation analysis using mutant schemata. InInternational Symposium on Software Testing and Analysis, pages 139–148, Cambridge, Massachusetts, June 1993.

  49. A. M. R. Vincenzi, J. C. Maldonado, E. F. Barbosa, and M. E. Delamaro. Unit and integration testing strategies for C programs using mutation-based criteria.STVR — Software Testing, Verification and Reliability, 11(4):249–268, Dec. 2001.

    Article  Google Scholar 

  50. A. M. R. Vincenzi, E. Y. Nakagawa, J. C. Maldonado, M. E. Delamaro, and R. A. F. Romero. Bayesian-learning based guidelines to determine equivalent mutants.International Journal of Software Engineering and Knowledge Engineering — IJSEKE, 12(6):675–689, Dec. 2002.

    Article  Google Scholar 

  51. H. Zhu, P. Hall, and J. May. Software unit test coverage and adequacy.ACM Computing Surveys, 29(4):366–427, Dec. 1997.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Instituto de Informática, Universidade Federal de Goiás, Goiânia, GO, Brazil

    A. M. R. Vincenzi

  2. Inst. de Ciências Mat. e de Computação, Universidade de São Paulo, São Carlos, SP, Brazil

    A. S. Simão & J. C. Maldonado

  3. Centro Universitário Eurípides de Marília Marília, São Paulo, Brazil

    M. E. Delamaro

  4. Auri Marcelo Rizzo Vincenzi, Bloco IMF I, sala 239 — Campus II — Samambaia, Caixa Postal 131, 74001-970, Goiânia, GO, Brazil

    A. M. R. Vincenzi

Authors
  1. A. M. R. Vincenzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. S. Simão
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. E. Delamaro
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. J. C. Maldonado
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Vincenzi, A.M.R., Simão, A.S., Delamaro, M.E. et al. Muta-Pro: Towards the definition of a mutation testing process. J Braz Comp Soc 12, 49–61 (2006). https://doi.org/10.1007/BF03192394

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF03192394

Keywords