CoChat
Sign Up
Cited by 50Software Engineering Techniques and PracticesInformation and Cyber SecurityWeb Application Security Vulnerabilities

Prioritization based taxonomy of DevOps security challenges using PROMETHEE

Open access
Rafi, Saima, Yu, Wu, Akbar, Muhammad Azeem, Alsanad, Ahmed, Gumaei, Abdu·2020·IEEE Access
DOI 10.1109/access.2020.2998819
Open this paper in CoChat — free accountOpen PDF

Abstract

DevOps is a combination of collaborative and multidisciplinary efforts of an organization to control continuous delivery and updates of new software while guaranteeing their reliability and correctness. In the software industry, the implementation of DevOps (development and operations units) faces many challenges that are specifically associated with the security. The objective of this study is to identify and develop a prioritization based taxonomy of DevOps security challenges. The total of eighteen DevOps security challenges were extracted using systematic literature review approach and were further evaluated with experts using questionnaire survey study. Finally, the multi criteria decision making PROMETHEE-II approach was used to prioritize and develop the taxonomy of identified factors and their categories. The implications of PROMETHEE-II approach are novel in this research domain as it has been used successfully in various other domains e.g. medical, banking, internet techniques and management etc. The contribution of this study is not limited to develop the taxonomy based structure of DevOps security challenges, but also the proper prioritization of these challenges by introducing PROMETHEE-II approach in the research field of DevOps. The study results will assist the practitioners to remove the uncertainty and vagueness in the opinion of DevOps experts to secure DevOps implementation for better and continuous software development process.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper identifies 18 DevOps security challenges through a systematic literature review, validates them with a practitioner survey, and uses PROMETHEE-II to prioritize them within a CAMS-based taxonomy. It ranks “lack of automated testing tools” (CH1) as the most critical challenge, with “lack of secure coding standards” (CH16) and “neglecting change control in security” (CH18) also highlighted as high-priority.

Key findings

  • The SLR started from 110 studies and selected 40 for data extraction, yielding 18 DevOps security challenges; 65% of selected studies scored more than 60% in quality assessment, and inter-rater agreement for extraction was Kendall’s W = 0.84 (p = 0.003).
  • The questionnaire survey collected 88 responses, of which 78 complete responses were analyzed; most challenges and categories had >60% positive agreement, CH18 (“neglecting change control in security”) was the highest reported challenge at 96%, and “Culture” (C1) was the highest category at 91%.
  • PROMETHEE-II ranked CH1 (“lack of automated testing tools”) as the most critical challenge; CH16 (“lack of secure coding standards”) and CH18 (“neglecting change control in security”) were also identified as critical/high-priority.
  • In the CAMS taxonomy, “security manual testing and performance configuration” (CH2) ranked 4th overall but 1st in the Measurement category, while “abundance of information is a serious threat to secure data” (CH13) ranked 9th overall but 1st in Sharing; category weights were Culture w = 0.35, Automation w = 0.25, and Measurement w = 0.25.

Methods

  • Three-phase design: (1) SLR following Kitchenham and Charters guidelines with a defined search string, inclusion/exclusion criteria, tollgate selection, and quality assessment; (2) questionnaire survey to validate challenges from industry/academic experts; (3) PROMETHEE-II multi-criteria decision-making to prioritize challenges and categories.
  • SLR sources included IEEE Xplore, ACM DL, Wiley, Springer, Google Scholar, and IET Digital Library; data extraction involved the first three authors plus three external experts, with inter-rater reliability checks.
  • Survey used Google Forms, pilot-tested with three external experts, distributed via ResearchGate, emails, and LinkedIn using snowball sampling from November 2019 to January 2020; 78 complete responses were analyzed, and a second survey of 25 experts provided pairwise comparisons for PROMETHEE-II, using the usual preference function and treating automation as a non-beneficial criterion.

Limitations

  • The paper states possible incomplete literature coverage and omitted relevant studies; researcher bias was addressed through inter-rater reliability tests; the n = 78 survey sample “might not be strong enough” to justify validity, though it is argued to be justifiable based on prior software engineering research.
  • The paper states most survey respondents were from Asian countries, limiting generalization to other regions; mapping-scheme biases were addressed through coding, empirical validation, and inter-rater reliability; ranking threats were addressed by using PROMETHEE-II net outranking flows.
  • The paper also notes the second survey’s sample size of 25 is small and “may limit our research to some extent,” though it justifies this with prior small-sample studies; evident from the text, the open-ended survey questions yielded no additional challenges, and the study focuses only on negative security challenges, with positive factors left for future work.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/access.2020.2998819

Cites

79 of 79 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 41 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The total of eighteen DevOps security challenges were extracted using systematic literature review.

    “The total of eighteen DevOps security challenges were extracted using systematic literature review approach and were further evaluated with experts using questionnaire survey study.”
    • Also stated in: § b: DATA EXTRACTION AND SYNTHESIS · plain
      “After synthesizing the data, we have total eighteen security challenges of DevOps.”
    • Also stated in: § A. IDENTIFIED CHALLENGES VIA SYSTEMATIC LITERATURE REVIE · plain
      “The total of 18 challenges were identified and presented in Table 1.”
    Quote in the paperSubstantial claim
  • Findingown data

    65% of selected studies scored more than 60% in quality assessment.

    “The 65 % of studies score more than 60% (Appendix C) which shows that the selected studies are significant enough to answer the research question of this study.”
    Quote in the paperSubstantial claim
  • Findingown data

    Culture (C1) was the highest category of investigated factors at 91%.

    “and ''Culture'' i.e. (C1 = 91%) is the highest category of investigated factors.”
    Quote in the paperSubstantial claim
  • Findingown data

    CH18 (neglecting change control in security) was the highest reported challenging factor by survey respondents at 96%.

    “We further noted that CH18 (neglecting change control in security 96%) is the highest reported challenging factor by survey respondents”
    Quote in the paperNot a substantial claim
  • Findingown data

    CH18 'neglecting change control in security' is the second highest ranked challenge for securing DevOps.

    “Furthermore, CH16 ''lack of secure coding standards'' and CH18 ''neglecting change control in security'' are the second highest ranked challenges while securing DevOps for successful implementation.”
    Quote in the paperSubstantial claim
  • The quote only partly states this claim. 3
  • Classified as probably not a substantial claim. 22
  • May combine two claims. 16

  • The quote only partly states this claim.

    Findingown data

    No additional DevOps security challenge was reported by survey participants in the open-ended questions.

    “However, no additional challenge was marked by the participants.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. The quote only partly states this claim.

    Findingown data

    A second survey was conducted with 25 participants who had participated in the first survey.

    “A total of 25 participants agreed to give a response on second survey Table 4.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The selected studies were published from 2011 to 2020, showing an increasing trend of DevOps research.

    “The frequency analysis shows that the selected studies are from year 2011 to 2020 showing the increasing trend of research in the field of DevOps.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The questionnaire survey collected 88 responses, of which 10 were incomplete and 78 complete responses were evaluated.

    “We have collected total 88 responses.All responses were checked manually and 10 of them were incomplete.We didn't consider the incomplete responses for data analysis as a requirement of the questionnaire is not accomplished fully in those 10 incomplete responses.Total 78 responses were evaluated for further processing.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The SLR initially collected 110 studies after inclusion/exclusion criteria, and 40 studies were selected for data extraction.

    “In the initial stage after performing inclusion and exclusion criteria we have collected 110 studies.All the steps of tollgate approach were performed carefully and total of 40 studies were collected for data extraction (Figure 2).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [67]

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    The majority of survey respondents had a positive opinion about the identified DevOps security challenges, with most challenges and categories having >60% positive response.

    “The summarized results Table 3 shows that the majority of the survey respondents have a positive opinion about security challenges in DevOps showing the negative effect of factors on DevOps implementation.The frequency analysis shows that percentage of most of the challenges and their categories is > 60%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The survey respondents are representative of the broader population of DevOps practitioners.

  • The quote only partly states this claim.

    Findingcitation

    The eighteen identified DevOps security challenges were mapped into four main categories of the CAMS model (Culture, Automation, Measurement, Sharing).

    “The reported challenges were further divided into four main categories of CAMS model, presented by Edwards and Willis [25].”
    • Also stated in: § A. IDENTIFIED CHALLENGES VIA SYSTEMATIC LITERATURE REVIE · plain
      “By applying this approach, all the challenges were mapped as shown in Table 2.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [25]

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Automation and Measurement were weighted 0.25 and ranked as the second and third most important CAMS categories.

    “Automation 'w = 0.25' and Measurement 'w = 0.25' ranked as the second and third most important categories of investigated DevOps security challenges.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    CH13 'Abundance of information is a serious threat to secure data' ranked GR=9 in the overall ranking but LR=1 in the Sharing category.

    “Similarly, ''Abundance of information is a serious threat to secure data'' CH 13 ranked GR = 9 in overall ranking but ranked, LR = 1 in Sharing category.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    CH2 'Security manual testing and performance configuration' ranked 4th in the overall ranking (GR) but ranks 1st (LR) in the 'Measurement' criteria of the CAMS model.

    “Further, we analyzed that ''Security manual testing and performance configuration'' CH 2 ranked 4th in the overall ranking (GR) but ranks 1st (LR) in ''Measurement'' criteria of CAMS model.”
    • Also stated in: § D. PRIORITIZATION BASED TAXONOMY OF DEVOPS SECURITY CHALLENGES (RQ4) · plain
      “We have investigated that the ''security manual testing and performance configuration'' CH 2 is ranked, i.e.GR = 4 but in Measurement category it is ranked, i.e.LR = 1 which shows the significance of this challenge with respect to category.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The small sample size of the second survey may limit the research.

    “The sample size is small which may limit our research to some extent.But small sample size has been considered previously in other research studies also [45]- [48].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45]- [48]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The questionnaire was pilot-tested with three external experts from academia and industry to check understandability and suitability.

    “The pilot assessment of questionnaire was performed with aim to check the understandability and suitability of designed questionnaire [36].”
    • Also stated in: § 2) PILOT ASSESSMENT OF QUESTIONNAIRE SURVE · plain
      “The designed questionnaire was sent to three external experts of academia and industry for evaluation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The systematic literature review followed Kitchenham and Charters guidelines for planning, conducting, and reporting the review.

    “The guidelines provided by Kitchenham and Charters [64] was adopted to explore literature which include i) planning the review, ii) conducting the review and iii) reporting the review.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [64] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The study used Grounded theory approach coding steps for mapping challenges.

    “All the team members of the group were using key steps of coding scheme, i.e. code, sub-categories, categories and framework/ theory, of Grounded theory approach [28], [85] as followed by other researchers in their studies [30], [86], [87].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [28] source

      Not checked in the free run.

    • cites [85]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    PROMETHEE II provides a complete ranking of alternatives and avoids incomparability between alternatives.

    “Another advantage of using PROMETHEE II is that it avoids incomparability between alternatives.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The PROMETHEE II method was selected because it is interactive, able to classify and order complex alternatives, and has stability, clarity, and simplicity.

    “The PROMETHEE II was selected for this study because it is interactive and is able to classify and order alternatives which are complex and difficult to compare.This approach in addition has some other characteristics like stability, clarity and simplicity [21].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The weights of the CAMS categories were validated by three external DevOps experts, and after their approval the weighted categories were used for further calculation.

    “The weights of categories were further investigated by sending them to three external DevOps experts (one expert from Chongqing University, China and two DevOps experts from Soft-Tech organization Pakistan).After their approval, we have used the weighted categories for further calculation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The authors followed a formal mapping approach to categorize the challenges, as applied in other research studies.

    “We have followed a formal mapping approach to categories the challenges, applied in other research studies as well [31]- [33], [44].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31]- [33], [44]

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The mapping scheme was validated by three DevOps experts, and based on their recommendations some factors were rearranged.

    “For validation we send our mapping scheme to three DevOps experts (one expert from Chongqing University China and two experts from Soft-Tech organization Pakistan) section 4.2.4.Based on their recommendations, we rearrange the position of some factors, the final mapping is presented in Table 3.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    The PROMETHEE II technique provides a complete ranking of all alternatives.

    “This technique provides the complete ranking of all alternatives which will assist the DevOps team to focus on areas having security related issues.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The questionnaire survey was conducted to identify and validate the factors from industrial practitioners having knowledge about DevOps.

    “The questionnaire survey was conducted to identify and validate the factors from industrial practitioners having knowledge about DevOps.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The mapping scheme biases were addressed by using a coding scheme, empirical study, and inter-rater reliability test.

    “The mapping scheme also has biases to some extent, to validate this concern we have used coding scheme to develop a mapping concept and validate it by empirical study (questionnaire survey) and considering inter-rater reliability test.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The taxonomy designed using the CAMS model will show the significance of factors and categories for further perfection of DevOps practices.

    “The taxonomy designed using CAMS model will show the significance of factors and categories for further perfection of DevOps practices.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Integrating security with DevOps will resolve several issues.

    “The DevOps team focus more on coordination of development and operational team, integrating security with DevOps will resolve several issues.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Most survey respondents are from Asian countries, so the outcomes cannot be generalized to other regions.

    “In this study most of the survey respondents are from Asian countries and we will unable to generalize the outcomes from the other regions perspective.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The identified challenges were further mapped into key categories and verified from industrial and academic experts using questionnaire survey approach.

    “The identified challenges were further mapped into key categories and verified from industrial and academic experts using questionnaire survey approach.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The PROMETHEE II approach was used to prioritize factors to avoid the threat of ranking.

    “There is another threat of ranking the identified and reported challenges, to avoid this threat we prioritize factors through PROMETHEE II approach.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Search strings were developed to collect data from targeted digital libraries and QA steps were performed to verify the quality of research material.

    “To reduce this threat, we have developed the search strings to collect data from targeted digital libraries and performed QA steps to verify the quality of research material.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The factors and their categorization were empirically investigated by questionnaire survey approach.

    “The factors and their categorization were empirically investigated by questionnaire survey approach.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisereasoning

    The PROMETHEE II technique provides better understanding of multi criteria decision making problems.

    “This technique provides better understanding of multi criteria decision making problems by considering the DevOps security challenges and their mapped categories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Researchers and academic experts agreed to integrate security with DevOps implementations.

    “The researchers and academic experts agreed to integrate security with DevOps implementations [4], [6]- [8], [59].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4], [6]- [8], [59]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductnone given

    The data collected from the survey includes responses from different regions and is sufficient to validate the factors.

    “However, the data collected from survey consist of response from different regions and we believe that this sample data is sufficient to validate the factors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The given taxonomy of categories and factors will contribute to enhance the areas which required more security practices for improvement.

    “The given taxonomy of categories and factors will contribute to enhance the areas which required more security practices for improvement.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The ranking of identified challenges of DevOps security will assist practitioners by considering the significance of factor within the process area and as a whole.

    “This ranking of identified challenges of DevOps security will assist the practitioners by considering the significance of factor within the process area and as a whole.”
    Quote in the paperNot a substantial claim
34 of 34

  • The paper relies on this without support.

    The survey respondents are representative of the broader population of DevOps practitioners.

    “The total of 78 complete responses was collected from participants (DevOps experts) Table 3.”

  • The paper relies on this without support.

    The 25 practitioners who provided pairwise comparisons are sufficiently knowledgeable about DevOps security to produce valid comparisons.

    “We collected data from 25 practitioners to measure the pairwise comparison of identified challenges using PROMETHEE II approach.”

  • The paper relies on this without support.

    The three external DevOps experts are qualified to validate the weights and mapping of the challenges.

    “The weights of categories were further investigated by sending them to three external DevOps experts (one expert from Chongqing University, China and two DevOps experts from Soft-Tech organization Pakistan).”
    “For validation we send our mapping scheme to three DevOps experts (one expert from Chongqing University China and two experts from Soft-Tech organization Pakistan) section 4.2.4.”

  • The paper relies on this without support.

    The survey respondents are representative of industrial practitioners with knowledge about DevOps.

    “The questionnaire survey was conducted to identify and validate the factors from industrial practitioners having knowledge about DevOps.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Various prioritization-based studies identify C18-Lack of mature tools for automation and security and rank it in the top tier.

    “Various prioritization-based studies (Akbar et al., 2022; Rafi et al., 2020) also identify this challenge and rank it in the top tier”

    From Identifying the primary dimensions of DevSecOps: A multi-vocal literature review (2024) · cites this paper as Rafi et al., 2020

    In “Prioritization of DevOps Security Challenges (RQ3)” (page unavailable), Rafi et al. rank lack of automated testing tools as their most critical challenge. The literature-review section also identifies immature automated deployment tools, but the copy does not show that broader challenge ranked in the top tier; its CH18 is instead neglecting change control in security. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • C02 reveals that isolation between developers, operations and InfoSec causes lack of collaboration, communication and coordination among teams, leading to friction, conflicts and mistrust.

    “This challenge reveals that the isolation between developers, operation and InfoSec would cause the lack of collaboration, communication and coordination among teams, further lead to friction, conflicts and mistrust (Rafi et al., 2020).”

    From Identifying the primary dimensions of DevSecOps: A multi-vocal literature review (2024) · cites this paper as Rafi et al., 2020

    The Background of DevOps Security says poor coordination between development and operations causes poor information and communication flow. The Identified Challenges section also links lack of coordination between security and DevOps teams to problems and says communication and coordination help develop trust. Neither passage states that isolation among all three teams leads to friction, conflicts and mistrust; page numbers are unavailable in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Without addressing C01 at the beginning, projects may not start off well.

    “Without addressing this issue at the beginning, projects may not start off well (Rafi et al., 2020; Tomas et al., 2019).”

    From Identifying the primary dimensions of DevSecOps: A multi-vocal literature review (2024) · cites this paper as Rafi et al., 2020

    The Introduction says security must be addressed from the forefront for successful DevOps adoption and warns that teams working without security coordination may not achieve the desired output. That supports the general concern about addressing security early, but the paper does not identify an issue called C01 or say that failing to address it will cause projects to start poorly (no page numbers available in our copy). (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Ranking rule favors the flow described as weakness

    Major

    The paper defines net flow as entering flow minus leaving flow, calls entering flow weakness and leaving flow strength, then says the highest net flow is best. Under those definitions, the ranking rule rewards weakness and penalizes strength.

  • Organization-size categories overlap at 200 employees

    Minor

    An organization with exactly 200 employees meets both the stated medium and large definitions, so the categories do not separate organizations at that boundary.

  • CAMS model attribution conflicts with reference 25

    Minor

    The text attributes the cited CAMS model to Edwards and Willis, but reference 25 identifies S. Guthrie as its author.

  • Figure 10 is cited as a ranking but captioned as organization size

    Minor

    The text directs readers to Figure 10 for the ranking of alternatives, while the Figure 10 caption identifies it as organization-size analysis.

  • Final mapping is assigned to the responses table

    Minor

    The text says the final mapping appears in Table 3, but Table 3 is labeled as survey responses; the mapping table is labeled Table 2.

5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 9

  1. lewis-beck, 2003, Semi structured interview, Sage Encyclopedia Social Sci Res Methods, 1
  2. willis, 2012, DevOps Culture (Part 1)
  3. humble, 2011, Why enterprises must adopt devops to enable continuous delivery, Cutter IT J, 24, 6
  4. guthrie, 2019, DevOps Principles—The CAMS Model
  5. de vries, 2014, Continuous security testing in a DevOps world, Proc OWASP AppSec Eur
  6. technologies, 2014, DevOps The Worst-Kept Secret to Winning in the Application Economy
  7. siahaan, 2017, Fadlina—Best student selection using extended promethee II method, International Journal of Recent Trends in Engineering
  8. yaghoobi, 2018, Prioritizing key success factors of software projects using fuzzy AHP, Journal of Software: Evolution and Process, 30, 1891e
  9. wen-ying, 2009, Application of ahp analysis in risk management of engineering projects, J Beijing Univ Chem Technol (Social Sci Ed ), 1, 46

Cited by

Sorted by citations

Related

Authors
Rafi, S., Yu, W., Akbar, M. A., Alsanad, A., & Gumaei, A.
Affiliations
  • Chongqing University of Posts and Telecommunications (CN)
  • Nanjing University of Aeronautics and Astronautics (CN)
  • King Saud University (SA)
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2020
Volume
8
Pages
105426-105446
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2020.2998819
OpenAlex
W3033639614
Cited by
50
References
79
Citations by year
2026: 3 · 2025: 13 · 2024: 11 · 2023: 8 · 2022: 9 · 2021: 6
FWCI
5.41
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Access (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Social Science Foundation of China (17XFX013)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
  • Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
Keywords
DevOps security, security challenges, taxonomy (general classification scheme), challenge prioritization, PROMETHEE II, multi-criteria decision making, expert opinion, software security

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review