STRAM: Measuring the Trustworthiness of Computer-Based Systems
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Various system metrics have been proposed for measuring the quality of computer-based systems, such as dependability and security metrics for estimating their performance and security characteristics. As computer-based systems grow in complexity with many subsystems or components, measuring their quality in multiple dimensions is a challenging task. In this work, we tackle the problem of measuring the quality of computer-based systems based on the four key attributes of trustworthiness we developed: security, trust, resilience, and agility. In addition to conducting a systematic survey on metrics, measurements, attributes of metrics, and associated ontologies, we propose a system-level trustworthiness metric framework that accommodates four submetrics, called STRAM (S ecurity, T rust, R esilience, and A gility M etrics). The proposed STRAM framework offers a hierarchical ontology structure where each submetric is defined as a sub-ontology. Moreover, this work proposes developing and incorporating metrics describing key assessment tools, including vulnerability assessment, risk assessment, and red teaming, to provide additional evidence in the measurement and quality of trustworthy systems. We further discuss how assessment tools are related to measuring the quality of computer-based systems and the limitations of the state-of-the-art metrics and measurements. Finally, we suggest future research directions for system-level metrics research toward measuring fundamental attributes of the quality of computer-based systems and improving the current metric and measurement methodologies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Resilience, Adaptability and Transformability in Social–ecological Systems
Walker, B., Holling, C. S., Carpenter, S. R., & Kinzig, A. P. (2004) · Ecology and Society · doi:10.5751/es-00650-090205 · Cited by 7,821
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Colquitt, J. A., Scott, B. A., & LePine, J. A. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.4.909 · Cited by 2,606
Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002) · Communications of the ACM · doi:10.1145/505248.506010 · Cited by 1,724
What makes Web sites credible?: a report on a large quantitative study
Fogg, B. J., Marshall, J., Laraki, O., Osipovich, A., Varma, C., Fang, N., Paul, J., Rangnekar, A., Shon, J., Swani, P., et al. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365037 · Cited by 685 · cited 1×
“Fogg et al. (2001) used the term “trustworthiness” to describe the key element of information credibility, such as “wellintended, truthful, and unbiased,” in terms of the perceived goodness or morality of the source in the context of evaluating the credibility of websites on the Internet.”
MethodTowards a Conceptual Framework for Resilience Engineering
Madni, A. M., & Jackson, S. (2009) · IEEE Systems Journal · doi:10.1109/jsyst.2009.2017397 · Cited by 649 · cited 2×
“While it is intuitive to define resilience in engineering systems as the ability of a system to adapt to disruptions (Haimes 2009a; Madni and Jackson 2009; Woods and Hollnagel 2006), definitions vary in different domains.”
Background
Cited by
Trust in AI: progress, challenges, and future directions
Afroogh, S., Akbari, A., Malone, E., Kargar, M., & Alambeigi, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04044-8 · Cited by 507
A Survey on Ethereum Systems Security: Vulnerabilities, Attacks, and Defenses
Chen, H., Pendleton, M., Njilla, L., & Xu, S. (2021) · ACM Computing Surveys · doi:10.1145/3391195 · Cited by 480
Self-citationAlvarado, R. (2023) · Science and Engineering Ethics · doi:10.1007/s11948-023-00451-3 · Cited by 137
Human Cognition Through the Lens of Social Engineering Cyberattacks
Montañez, R., Golob, E., & Xu, S. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01755 · Cited by 106
“…and discussed above, we stress the importance of defining and quantifying social engineering cyberattack metrics, which are largely missing and will become an indispensable component of the broader family of cybersecurity metrics as discussed in Pendleton et al. [2016], Cho et al. [2019], Xu [2019].”
BackgroundSelf-citationWhat kind of trust does AI deserve, if any?
Alvarado, R. (2023) · AI and Ethics · doi:10.1007/s43681-022-00224-x · Cited by 60
Respite for SMEs: A Systematic Review of Socio-Technical Cybersecurity Metrics
van Haastrecht, M., Yigit Ozkan, B., Brinkhuis, M., & Spruit, M. (2021) · Applied Sciences · doi:10.3390/app11156909 · Cited by 45
“Our final research question covered the consideration of adaptability: “the state of being able to change to work or fit better” [18].”
BackgroundMethodInfluentialTrust and Trustworthiness in AI
Durán, J. M., & Pozzi, G. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00843-2 · Cited by 47
An overview of key trustworthiness attributes and KPIs for trusted ML-based systems engineering
Mattioli, J., Sohier, H., Delaborde, A., Amokrane-Ferka, K., Awadid, A., Chihani, Z., Khalfaoui, S., & Pedroza, G. (2024) · AI and Ethics · doi:10.1007/s43681-023-00394-2 · Cited by 22
Metadata
- Authors
- Cho, J.-H., Xu, S., Hurley, P. M., Mackay, M., Benjamin, T., & Beaumont, M.
- Venue
- ACM Computing Surveys
- Published
- 2019-02-04
- Volume
- 51
- Issue
- 6
- Pages
- 1-47
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2914051825
- Cited by
- 94
- References
- 7
- FWCI
- 13.12
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.