Automated Checking of Conformance to Requirements Templates Using Natural Language Processing
Open accessThe DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
Templates are effective tools for increasing the precision of natural language requirements and for avoiding ambiguities that may arise from the use of unrestricted natural language. When templates are applied, it is important to verify that the requirements are indeed written according to the templates. If done manually, checking conformance to templates is laborious, presenting a particular challenge when the task has to be repeated multiple times in response to changes in the requirements. In this article, using techniques from natural language processing (NLP), we develop an automated approach for checking conformance to templates. Specifically, we present a generalizable method for casting templates into NLP pattern matchers and reflect on our practical experience implementing automated checkers for two well-known templates in the requirements engineering community. We report on the application of our approach to four case studies. Our results indicate that: (1) our approach provides a robust and accurate basis for checking conformance to templates; and (2) the effectiveness of our approach is not compromised even when the requirements glossary terms are unknown. This makes our work particularly relevant to practice, as many industrial requirements documents have incomplete glossaries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 1×
“Cohen’s Kappa [38], for the independent inspections conducted by the first two authors.”
BackgroundClassification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Introduction to Modern Information Retrieval
Salton, G., & McGill, M. J. (1983) · Virtual Defense Library (Ministerio de Defensa) · Cited by 11,546
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Introduction to modern information retrieval
Dillon, M. (1983) · Information Processing & Management · doi:10.1016/0306-4573(83)90062-6 · Cited by 5,909
Jurafsky, D. S., & Martin, J. (2000) · Cited by 4,132
Natural language processing with Python
Bird, S., Klein, E., & Loper, E. (2009) · CERN Document Server (European Organization for Nuclear Research) · Cited by 3,988
Text Chunking Using Transformation-Based Learning
Ramshaw, L. A., & Marcus, M. P. (1999) · Text, Speech and Language Technology · doi:10.1007/978-94-017-2390-9_10 · Cited by 1,460
The relationship between Recall and Precision
Buckland, M., & Gey, F. (1994) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199401)45:1<12::aid-asi2>3.0.co;2-l · Cited by 961
Requirements engineering: fundamentals, principles, and techniques
(2011) · Choice Reviews Online · doi:10.5860/choice.48-3304 · Cited by 609
How Do Users Like This Feature? A Fine Grained Sentiment Analysis of App Reviews
Guzman, E., & Maalej, W. (2014) · 2014 IEEE 22nd International Requirements Engineering Conference (RE) · doi:10.1109/re.2014.6912257 · Cited by 590 · cited 1×
“Guzman and Maalej [70] employ part-ofspeech tagging, stemming and sentiment analysis for synthesizing user opinions about the requirements of mobile applications.”
MethodBuilding a Large Annotated Corpus of English: The
MarcusMitchell, P., Ann, M., & SantoriniBeatrice (1993) · Computational Linguistics · doi:10.5555/972470.972475 · Cited by 489
Software traceability: trends and future directions
Cleland-Huang, J., Gotel, O. C. Z., Huffman Hayes, J., Mäder, P., & Zisman, A. (2014) · Future of Software Engineering Proceedings · doi:10.1145/2593882.2593891 · Cited by 309 · cited 1×
“[60] provide detailed reviews of the state-of-the-art on requirements traceability, including the range of NLP-based techniques used in this area.”
BackgroundEasy Approach to Requirements Syntax (EARS)
Mavin, A., Wilkinson, P., Harwood, A., & Novak, M. (2009) · 2009 17th IEEE International Requirements Engineering Conference · doi:10.1109/re.2009.9 · Cited by 300 · cited 4×
“While our approach can be tailored to work with a variety of existing templates, we ground our work in this article on two templates, namely Rupp's and the EARS templates [5], [16].”
BackgroundMethodInfluentialFrom Contract Drafting to Software Specification: Linguistic Sources of Ambiguity
Berry, D. M., Kamsties, E., & Krieger, M. (2003) · Cited by 254
Reasoning about inconsistencies in natural language requirements
Gervasi, V., & Zowghi, D. (2005) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/1072997.1072999 · Cited by 193 · cited 1×
“Gervasi and Zowghi [54] use part-of-speech tagging and parsing to automatically transform requirements into propositional logic and identify inconsistencies through logical reasoning.”
BackgroundIdentifying Nocuous Ambiguities in Natural Language Requirements
Chantree, F., Nuseibeh, B., Roeck, A. N. D., & Willis, A. (2006) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2006.31 · Cited by 177 · cited 1×
“[61] use morphological analysis for detecting nocuous requirements ambiguities.”
MethodA systematic review of transformation approaches between user requirements and analysis models
Yue, T., Briand, L. C., & Labiche, Y. (2011) · Requirements Engineering · doi:10.1007/s00766-010-0111-y · Cited by 170 · cited 1×
“[64] present a systematic literature review on the approaches for transforming NL requirements into analysis models, further providing insights about how NLP is utilized in this context.”
BackgroundSelf-citationA natural language approach for Requirements Engineering
Rolland, C., & Proix, C. (1992) · Notes on Numerical Fluid Mechanics and Multidisciplinary Design · doi:10.1007/bfb0035136 · Cited by 167 · cited 1×
“, by Rolland and Proix [40], by Rupp and”
BackgroundKiyavitskaya, N., Zeni, N., Mich, L., & Berry, D. M. (2008) · Requirements Engineering · doi:10.1007/s00766-008-0063-7 · Cited by 167 · cited 1×
“[62] use parsing for finding various potential syntactic and semantic ambiguities in requirements.”
BackgroundIndependence and commitment: assumptions for rapid training and execution of rule-based POS taggers
Hepple, M. (2000) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1075218.1075254 · Cited by 153 · cited 1×
“Step 3 (3 alternatives): OpenNLP POS Tagger [31], Stanford POS Tagger [32], ANNIE POS Tagger [33].”
BackgroundRequirements engineering tools: Capabilities, survey and assessment
de Gea, J. M. C., Nicolás, J., Fernández‐Alemán, J. L., Toval, A., Ebert, C., & Vizcaíno, A. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.04.005 · Cited by 125 · cited 1×
“Support for requirements templates now exists in some commercial requirements authoring and management tools [11], indicating a general industrial interest in templates and, naturally, the quality assurance activities surrounding templates.”
BackgroundFalessi, D., Cantone, G., & Canfora, G. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.122 · Cited by 118 · cited 1×
“[69] apply and empirically compare dif-”
Improving automated requirements trace retrieval: a study of term-based enhancement methods
Zou, X., Settimi, R., & Cleland‐Huang, J. (2009) · Empirical Software Engineering · doi:10.1007/s10664-009-9114-z · Cited by 117 · cited 1×
“Furthermore, the literature suggests that glossaries may remain incomplete throughout the entire development process [14], thus providing only partial coverage of the glossary terms.”
BackgroundAutomating the Extraction of Rights and Obligations for Regulatory Compliance
Kiyavitskaya, N., Zeni, N., Breaux, T. D., Antón, A. I., Cordy, J. R., Mich, L., & Mylopoulos, J. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-87877-3_13 · Cited by 115 · cited 1×
“[66] use part-of-speech tagging and parsing for generating various kinds of markup information over regulatory requirements.”
BackgroundREQUIREMENTS TRACEABILITY: A SYSTEMATIC REVIEW AND INDUSTRY CASE STUDY
Torkar, R., Gorschek, T., FELDT, R. H., Svahnberg, M., Raja, U. A., & Kamran, K. (2012) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819401250009x · Cited by 113 · cited 1×
“[59] and Cleland-Huang et al.”
Assessing traceability of software engineering artifacts
Sundaram, S. K., Hayes, J. H., Dekhtyar, A., & Holbrook, E. A. (2010) · Requirements Engineering · doi:10.1007/s00766-009-0096-6 · Cited by 101 · cited 1×
“[57] utilize similarity measures in a manner analogous to the above to improve requirements traceability detection.”
MethodBig Ears (The Return of "Easy Approach to Requirements Engineering")
Mavin, A., & Wilkinson, P. (2010) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2010.39 · Cited by 78 · cited 1×
“[16], [17], [18] and discussed in Section 2.1.”
BackgroundInfluentialRapid requirements checks with requirements smells: two case studies
Femmer, H., Méndez, D., Juergens, E., Klose, M., Zimmer, I., & Zimmer, J. (2014) · doi:10.1145/2593812.2593817 · Cited by 64 · cited 1×
“In addition to checking template conformance, we use NLP for detecting and warning about several potentially problematic constructs, also called requirements smells [28], that”
MethodArora, C., Sabetzadeh, M., Briand, L., & Zimmer, F. (2014) · IEEE International Workshop on Requirements Patterns (RePa) · doi:10.1109/repa.2014.6894837 · Cited by 38 · cited 2×
“In fact, based on our experience [13], practitioners tend to identify and define the glossary terms after the requirements have sufficiently matured and are thus less likely to change.”
BackgroundSelf-citationArora, C., Sabetzadeh, M., Briand, L., Zimmer, F., & Gnaga, R. (2013) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.13 · Cited by 30 · cited 2×
“The article further provides substantial new empirical evidence to support the effectiveness of our approach, complementing the single case study in our previous work [19] with three new case studies.”
BackgroundResultSelf-citationTeaching finite state machines with case method and role play
Joseph, S., Schumm, M., Rummel, O., Soska, A., Reschke, M., Mottok, J., Niemetz, M., & Schroll-Decker, I. (2013) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon.2013.6530275 · Cited by 12 · cited 1×
“These templates have been used both in academia and in industrial settings for specifying the requirements of complex systems [21], [41], [42], [43].”
MethodAutomatic Extraction for Creating a Lexical Repository of Abbreviations in the Biomedical Literature
Song, M., Song, I., & Lee, K. J. (2006) · Lecture notes in computer science · doi:10.1007/11823728_37 · Cited by 2 · cited 1×
“Second, text chunking is more robust than parsing [25], in the sense that it produces results in the large majority of cases; whereas parsing may fail to generate a parse tree, particularly when faced with unfamiliar input.”
Background
Cited by
Requirements engineering for artificial intelligence systems: A systematic mapping study
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107176 · Cited by 132
Self-citation
Metadata
- Authors
- Arora, C., Sabetzadeh, M., Briand, L., & Zimmer, F.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2015-10-01
- Volume
- 41
- Issue
- 10
- Pages
- 944-968
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W1978721560
- Cited by
- 168
- References
- 34
- FWCI
- 35.14
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.