- Invited Paper
- Open access
- Published:
An empirical comparative evaluation of requirements engineering methods
Journal of the Brazilian Computer Society volume 16, pages 3–19 (2010)
Abstract
Requirements Engineering (RE) is a relatively young discipline, and still many advances have been achieved during the last decades. In particular, numerous RE approaches are proposed in the literature with the aim of understanding a certain problem (e.g. information systems development) and establishing a knowledge base that is shared between domain experts and developers (i.e. a requirements specification). However, there is a growing concern for empirical validations that assess RE proposals and statements. This paper is related to the assessment of the quality of functional requirements specifications, using the Method Evaluation Model (MEM) as a theoretical framework. The MEM distinguishes the actual efficacy and the perceived efficacy of a method. In order to assess the actual efficacy or RE methods, the conceptual model quality framework by Lindland et al. can be applied; in this paper, we focus on the completeness and granularity of requirements models and extend this framework by defining four new metrics (e.g. degree of functional encapsulations completeness with respect to a reference model, number of functional fragmentation errors). In order to assess the perceived efficacy, conventional questionnaires can be used. A laboratory experiment with master students has been carried out, in order to compare (using the proposed metrics) two RE methods; namely, Use Cases and Communication Analysis. With respect to actual efficacy, results indicate greater model quality (in terms of completeness and granularity) when Communication Analysis guidelines are followed. With respect to perceived efficacy, we found that Use Cases was perceived to be slightly easier to use than Communication Analysis. However, Communication Analysis was perceived to be more useful in terms of determining the proper business processes granularity. The paper discusses these results and highlights some key issues for future research in this area.
References
España S, Condori-Fernández N, González A, Pastor Ó (2009) Evaluating the completeness and granularity of functional requirements specifications: a controlled experiment. In: 17th IEEE international requirements engineering conference (RE’09), Atlanta, GA, USA. IEEE, New York, pp 161–170
Boehm B, McClean RK, Ufrig DB (1975) Some experience with automated aids to the design of large-scale reliable software. IEEE Trans Softw Eng 1(1):125–133
Wieringa RJ, Heerkens JMG (2004) Evaluating the structure of research papers: a case study. In: 2nd international workshop in comparative evaluation of requirements engineering, Kyoto, Japan. IEEE Press, New York, pp 41–50
Wieringa RJ, Heerkens JMG (2006) The methodological soundness of requirements engineering papers: a conceptual framework and two case studies. Requir Eng 11(4):295–307
Höfer A, Tichy WF (2007) Status of empirical research in software engineering. In: Empirical software engineering issues. Critical assessment and future directions. LNCS, vol 4336. Springer, Berlin, pp 10–19
Iivari J, Kerola P (1983) A sociocybernetic framework for the feature analysis of information systems development methodologies. In: Olle TW, Sol HG, Tully CJ (eds) Information systems methodologies: a feature analysis. North-Holland, Amsterdam, pp 87–139
Dieste O, Lopez M, Ramos F (2008) Updating a systematic review about selection of software requirements specification techniques. In: 11th workshop on requirements engineering (WER 2008), Barcelona, Spain
Siau K, Rossi M (1998) Evaluation of information modeling methods—a review. In: 31st Hawaii international conference on systems science (HICSS 1998), Kohala Coast, USA, vol 5. IEEE, New York, pp 314–322
Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in Software Engineering: an introduction. Kluwer, Dordrecht
Cockburn A (2000) Writing effective use cases. Addison-Wesley, Reading
España S, González A, Pastor Ó (2009) Communication Analysis: a requirements elicitation approach for information systems. In: 21st international conference on advanced information systems, Amsterdam, The Netherlands. LNCS. Springer, Berlin
Wieringa RJ (1996) Requirements engineering: frameworks for understanding. Wiley, New York
Dobing B, Parsons J (2006) How UML is used. Commun ACM 49(5):109–113
Simons AJH (1999) Use cases considered harmful. In: Technology of object-oriented languages and systems, Nancy, France. IEEE Computer Society, Los Alamitos, pp 194–203
Dano B, Briand H, Barbier F (1997) A use case driven requirements engineering process. In: Third IEEE international symposium on requirements engineering (RE’97), Antapolis, MD. IEEE, New York
Rolland C, Achour CB (1998) Guiding the construction of textual use case specifications. Data Knowl Eng J 25(1–2):125–160
Constantine LL, Lockwood LAD (1999) Software for use: a practical guide to the models and methods of usage-centered design. Addison-Wesley, Reading
Cockburn A (1997) Structuring use cases with goals. J Object-Oriented Program
Phalp KT, Jonathan V, Cox K (2007) Improving the quality of use case descriptions: empirical assessment of writing guidelines. Softw Qual Control 15(4):383–399
Cox K, Phalp K, Shepperd M (2001) Comparing use case writing guidelines. In: 7th international workshop on requirements engineering: foundations for software quality (REFSQ’2001), Interlaken, Switzerland
Pastor O, González A, España S (2007) Conceptual alignment of software production methods. In: Conceptual modelling in information systems engineering. Springer, Heidelberg, pp 209–228
González A, España S, Pastor O (2008) Towards a communicational perspective for enterprise information systems modelling. In: IFIP WG 8.1 working conference on the practice of enterprise modeling, Stockholm, Sweden. LNBIP. Springer, Berlin
González A, España S, Pastor O (2009) Unity criteria for business process modelling: a theoretical argumentation for a software engineering recurrent problem. In: 3rd international conference on research challenges in information science (RCIS 2009), Fes, Morocco. IEEE, New York
Moody DL (2003) The Method Evaluation Model: a theoretical model for validating information systems design methods. In: Proceedings of the 11th European conference on information systems (ECIS 2003), Naples, Italy, 16–21 June 2003
Yadav SB, Bravoco RR, Chatfield AT, Rajkumar TM (1988) Comparison of analysis techniques for information requirement determination. Commun ACM 31(9):1090–1097
Lindland OI, Sindre G, Sølvberg A (1994) Understanding quality in conceptual modeling. IEEE Softw 11(2):42–49
Moody DL, Sindre G, Brasethvik T, Solvberg A (2003) Evaluating the quality of information models: empirical testing of a conceptual model quality framework. In: 25th international conference on software engineering (ICSE 2003), Portland, USA, pp 295–305
Moody DL, Sindre G, Brasethvik T, Solvberg A (2002) Evaluating the quality of process models: empirical testing of a quality framework. In: 21st international conference on conceptual modeling. Springer, Berlin
Larman C (1997) Applying UML and patterns. Prentice Hall, New York (see p 53)
Övergaard G, Palmkvist K (2004) Use Cases: patterns and blueprints. Addison-Wesley, Reading (see Part V)
Kulak D, Guiney E (2000) Use cases: requirements in context. Addison-Wesley, Reading, pp 94–95
Langlois RN (2002) Modularity in technology and organization. J Econ Behav Organ 49(1):19–37
Reijers H, Mendling J (2008) Modularity in process models: review and effects. In: 6th international conference on business process management (BPM 2008), Milan, Italy. LNCS, vol 5240. Springer, Berlin, pp 20–35
Rescher N (1977) Methodological pragmatism: systems-theoretic approach to the theory of knowledge. Basil Blackwell, Oxford
Davis FD, Bagozzi RP, Warshaw PR (1989) User acceptance of computer technology: a comparison of two theoretical models. Manag Sci 35(8):982–1003
Davis AM, Overmyer S, Jordan K, Caruso J, Dandashi F, Dinh A, Kincaid G, Ledeboer G, Reynolds P, Sitaram P, Ta A, Theofanos M (1993) Identifying and measuring quality in a software requirements specification. In: 1st international software metrics symposium, pp 141–152
Pohl K (1994) The three dimensions of requirements engineering: a framework and its applications. In: 5th international conference on advanced information systems engineering, Paris, France. Pergamon, New York
Krogstie J, Sindre G, Jorgensen H (2006) Process models representing knowledge for action: a revised quality framework. Eur J Inf Syst 15(1):91–102
Falkenberg E, Hesse W, Lindgreeen P, Nilsson B, Oei JLH, Rolland C, Stamper RK, VanAssche F, Verrijn-Stuart A, Voss K (1998) FRISCO. A framework of information systems concepts. IFIP WG 8.1 report
Krogstie J, Lindland OI, Sindre G (1995) Towards a deeper understanding of quality in requirements engineering. In: 7th international conference on advanced information systems engineering. Springer, Berlin, pp 82–95
Moody DL, Shanks GG (1994) What makes a good data model? Evaluating the quality of entity relationship models. In: 13th international conference on the entity-relationship approach, Manchester, UK. Springer, Berlin, pp 94–111
Schuette R, Rotthowe T (1998) The guidelines of modeling—an approach to enhance the quality in information models. In: Conceptual modeling (ER’98), Singapore. LNCS, vol 1507. Springer, Berlin, pp 240–254
Schuette R (1999) Architectures for evaluating the quality of information models—a meta and an object level comparison. In: 18th international conference on conceptual modeling (RE 1999). Springer, Berlin, pp 490–505
Shanks GG, Darke P (1997) Quality in conceptual modelling: linking theory and practice. Asia-Pacific conference on information systems (PACIS 1007), Brisbane, pp 805–814
Moody DL (2005) Theoretical and practical issues in evaluating the quality of conceptual models: current state and future directions. Data Knowl Eng 55(3):243–276
Lockemann PC, Mayr HC (1986) Information system design: techniques and software support. Information processing, vol 86. North-Holland, Amsterdam
Zave P, Jackson M (1997) Four dark corners of requirements engineering. ACM Trans Softw Eng Methodol 6(1):1–30
Jacobson I (2004) Use cases—yesterday, today, and tomorrow. Softw Syst Model 3(3):210–220
Kroll P (2004) Dr Process how many use cases should you have in a system? IBM Rational developerWorks documentation. Accessed 02-2009
Juristo N, Moreno A (2001) Basics of software engineering experimentation. Kluwer, Boston
Achour CB, Rolland C, Maiden NAM, Souveyet C (1999) Guiding use case authoring: results of an empirical study. In: IEEE symposium on requirements engineering. IEEE, Los Alamitos
Cox K, Phalp K (2000) Replicating the CREWS use case authoring guidelines experiment. Empir Softw Eng 5(3):245–267
Anda B, Sjøberg DIK, Jørgensen M (2001) Quality and understandability of use case models. In: 15th European conference on object-oriented programming (ECOOP 2001). LNCS, vol 2072. Springer, Berlin, pp 402–428
Fortuna M, Werner C, Borges M (2007) Um modelo integrado de requisitos com casos de uso. In: Workshop de ingeniería de requisitos y ambientes software IDEAS 2007
Jacobson I (1987) Object-oriented development in an industrial environment. In: Conference on object-oriented programming systems, languages and applications, Orlando, FL, USA. ACM, New York, pp 183–191
OMG (2009) OMG unified modeling language (OMG UML), superstructure, V2.2. http://www.omg.org/cgi-bin/doc?formal/09-02-02. Accessed 02-2010
Basili V, Rombach HD (1988) The TAME project: towards improvement-oriented software environments. IEEE Trans Softw Eng 14(6):758–773
Garson D (1998) Scales and standard measures. North Carolina State University. Updated September 2008. http://www2.chass.ncsu.edu/garson/pa765/standard.htm
Runeson P (2003) Using students as experiment subjects—an analysis on graduate and freshmen student data. In: 7th international conference on empirical assessment in software engineering, pp 95–102
Author information
Authors and Affiliations
Corresponding author
Additional information
This paper revises and extends previous work that has been published in the 17th IEEE International Requirements Engineering Conference (RE’09) [1].
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Cite this article
España, S., Condori-Fernandez, N., González, A. et al. An empirical comparative evaluation of requirements engineering methods. J Braz Comput Soc 16, 3–19 (2010). https://doi.org/10.1007/s13173-010-0003-5
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13173-010-0003-5