A study on the prevalence of human values in software engineering publications, 2015 -- 2018
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Failure to account for human values in software (e.g., equality and fairness) can result in user dissatisfaction and negative socio-economic impact. Engineering these values in software, however, requires technical and methodological support throughout the development life cycle. This paper investigates to what extent top Software Engineering (SE) conferences and journals have included research on human values in SE. We investigate the prevalence of human values in recent (2015 -- 2018) publications in these top venues. We classify these publications, based on their relevance to different values, against a widely used value structure adopted from the social sciences. Our results show that: (a) only a small proportion of the publications directly consider values, classified as directly relevant publications; (b) for the majority of the values, very few or no directly relevant publications were found; and (c) the prevalence of directly relevant publications was higher in SE conferences compared to SE journals. This paper shares these and other insights that may motivate future research on human values in software engineering.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Schwartz Values Structure [31] (adopted from [8]). Words in black boxes are values categories, each subdivided into values.](https://figures.cochat.ai/paper-figures/6f3ed17a70e04d5b9caea84976b8c06d/a-study-on-the-prevalence-of-human-values-in-software-engineering-publications-figure-1.png)








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A consideration of human values is largely lacking in software engineering [23]”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [23]
In the citing paper
“A literature review found four values to be the most frequently cited in 1,350 recently published software engineering publications: security, benevolence, universalism, and self-direction [23].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 3×
“The results of the Kappa measure are interpreted according to the agreement strengths introduced by Landis and Koch [19].”
MethodInfluentialSchwartz, S. H. (1992) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60281-6 · Cited by 14,619 · cited 3×
“classification not only in the social sciences but also in other disciplines [11, 40]. In SVS, Schwartz introduced 10 motivationally-distinct value categories recognized across more than 30 cultures [27]. Each value category has underlying distinct motivational goals (see Table 1) which relate to three fundamental nee…”
BackgroundMethodInfluential(1973) · DigitalGeorgetown (Georgetown University Library) · Cited by 10,144
Are There Universal Aspects in the Structure and Contents of Human Values?
Schwartz, S. H. (1994) · Journal of Social Issues · doi:10.1111/j.1540-4560.1994.tb01196.x · Cited by 5,793 · cited 2×
“tegories (Universalism, Conformity and Tradition). This verifies the tensions discussed in the Schwartz Values Structure about the competing and contradicting nature of these bipolar value categories [28]. It is important to note that SVS served as an appropriate yet not ideal scheme for classifying human values in SE…”
BackgroundAn overview of the Schwartz theory of basic values
Schwartz, S. H. (2012) · Online Readings in Psychology and Culture · doi:10.9707/2307-0919.1116 · Cited by 4,147 · cited 2×
“We followed a methodology similar to that of prior classification work in SE Table 1: Value categories and descriptions [32]”
MethodEvaluating the structure of human values with confirmatory factor analysis
Schwartz, S. H., & Boehnke, K. (2004) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00069-2 · Cited by 1,092 · cited 1×
“Figure 1: Schwartz Values Structure [30, 33] (adopted from [17]).”
BackgroundLes valeurs de base de la personne : théorie, mesures et applications
Schwartz, S. H. (2006) · Revue française de sociologie · doi:10.3917/rfs.474.0929 · Cited by 970 · cited 1×
“Figure 1: Schwartz Values Structure [30, 33] (adopted from [17]).”
BackgroundFriedman, B. (1996) · Interactions · doi:10.1145/242485.242493 · Cited by 966
Requirements engineering paper classification and evaluation criteria: a proposal and a discussion
Wieringa, R., Maiden, N., Mead, N., & Rolland, C. (2006) · Requirements Engineering · doi:10.1007/s00766-005-0021-6 · Cited by 940 · cited 1×
“[43] developed a classification scheme to identify papers that belong to Requirements Engineering as a subdomain in SE.”
Artificial Intelligence and the ‘Good Society’: the US, EU, and UK approach
Cath, C., Wachter, S., Mittelstadt, B., Taddeo, M., & Floridi, L. (2017) · Science and Engineering Ethics · doi:10.1007/s11948-017-9901-7 · Cited by 733 · cited 1×
“Meanwhile, the rapid popularization of artificial intelligence (AI) and its potential negative impact on society have raised the awareness of human values in AI research [5, 9, 23].”
BackgroundA survey of controlled experiments in software engineering
Sjoeberg, D. I. K., Hannay, J. E., Hansen, O., Kampenes, V. B., Karahasanovic, A., Liborg, N.-K., & Rekdal, A. C. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.97 · Cited by 682 · cited 1×
“[35] surveyed SE papers in nine journals and three conferences from 1993 to 2002 with the aim to characterize controlled experiments in SE by characterizing the topics of the experiments and their subjects, tasks, and environments.”
BackgroundMeta-research: Evaluation and Improvement of Research Methods and Practices
Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002264 · Cited by 495 · cited 1×
“n the call for papers of top-tier SE conferences and journals included in the Journal Citation Reports and the instructions to authors of relevant journals and published works. Also, Ioannidis et al. [18] categorized the meta-research discipline into five main thematic fields corresponding to how to conduct, report, v…”
BackgroundResearch in software engineering: an analysis of the literature
Glass, R. L., Vessey, I., & Ramesh, V. (2002) · Information and Software Technology · doi:10.1016/s0950-5849(02)00049-6 · Cited by 391
Research in Information Systems: An Empirical Study of Diversity in the Discipline and Its Journals
Vessey, I., Ramesh, V., & Glass, R. L. (2002) · Journal of Management Information Systems · doi:10.1080/07421222.2002.11045721 · Cited by 391 · cited 5×
“[42]: to report their findings, the authors categorized samples of SE papers published from 1995 to 1999 in six journals based on topic, method, and approach.”
BackgroundResultInfluentialIncorporating Ethics into Artificial Intelligence
Etzioni, A., & Etzioni, O. (2017) · The Journal of Ethics · doi:10.1007/s10892-017-9252-2 · Cited by 388 · cited 1×
“Meanwhile, the rapid popularization of artificial intelligence (AI) and its potential negative impact on society have raised the awareness of human values in AI research [5, 9, 23].”
BackgroundHuman Values, Ethics, and Design
(2002) · The Human-Computer Interaction Handbook · doi:10.1201/9781410606723-48 · Cited by 270
Writing good software engineering research papers
Shaw, M. (2003) · 25th International Conference on Software Engineering, 2003. Proceedings. · doi:10.1109/icse.2003.1201262 · Cited by 213 · cited 6×
“used in similar classification studies in SE [4, 15, 34].”
BackgroundMethodInfluentialExperimental validation in software engineering
Zelkowitz, M. V., & Wallace, D. (1997) · Information and Software Technology · doi:10.1016/s0950-5849(97)00025-6 · Cited by 202 · cited 1×
“Also, Zelkowitz and Wallace [45] classified, according to a 12-model classification scheme, around 600 SE papers published over a period of three years to provide insights on the use of experimentation within SE.”
BackgroundDeveloping a meta‐inventory of human values
Cheng, A., & Fleischmann, K. R. (2010) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.14504701232 · Cited by 175 · cited 2×
“While these two value structures remain the most well recognized ways of representing values, there are at least ten other value classifications [9].”
BackgroundOvercoming open source project entry barriers with a portal for newcomers
Steinmacher, I., Conte, T. U., Treude, C., & Gerosa, M. A. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884806 · Cited by 156 · cited 1×
“for OSS projects is to provide ways to support newcomers during their first contribution. In this paper, we propose and evaluate FLOSScoach, a portal created to support newcomers to OSS projects. ... [37] Protecting the Environment ... The battery power limitation of mobile devices has pushed developers and researcher…”
BackgroundValues at play: design tradeoffs in socially-oriented game design
Flanagan, M., Howe, D. C., & Nissenbaum, H. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055076 · Cited by 149 · cited 1×
“Human values with an ethical and moral import such as Equality, Privacy and Fairness have been studied in technology design and human-computer interaction for more than two decades [12–14].”
BackgroundOn the success of empirical studies in the international conference on software engineering
Zannier, C., Melnik, G., & Maurer, F. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134333 · Cited by 126 · cited 1×
“[44] to investigate the improvement of the quantity and quality of empirical evaluations conducted within ICSE papers over time.”
BackgroundValue-based requirements engineering: method and experience
Thew, S., & Sutcliffe, A. (2018) · Requirements Engineering · doi:10.1007/s00766-017-0273-y · Cited by 113 · cited 3×
“In this paper, we use SVS, which is the most cited and most widely applied classification not only in the social sciences but also in other disciplines [11, 40].”
BackgroundMethodEARMO: An Energy-Aware Refactoring Approach for Mobile Apps
Morales, R., Saborido, R., Khomh, F., Chicano, F., & Antoniol, G. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2757486 · Cited by 85
Measuring human values in software engineering
Winter, E., Forshaw, S., & Ferrario, M. A. (2018) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3239235.3267427 · Cited by 83 · cited 1×
“For example, Schwartz was used to incorporate values in the SE decision making process [14], to measure the values of software developers [47] and to predict movie genres for certain personality types [27].”
MethodFerrario, M. A., Simm, W., Newman, P., Forshaw, S., & Whittle, J. (2014) · Companion Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2591062.2591121 · Cited by 81 · cited 1×
“In this paper, we use SVS, which is the most cited and most widely applied classification not only in the social sciences but also in other disciplines [11, 40].”
MethodSelf-citationOperationalizing human values in software: a research roadmap
Mougouei, D., Perera, H., Hussain, W., Shams, R., & Whittle, J. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264843 · Cited by 77 · cited 2×
“Ignoring human values while engineering software may result in violating those values [10, 22] and subsequent dissatisfaction of”
BackgroundSelf-citationBasic Human Values: theory, methods, and application
Schwartz, S. H. (2007) · doi:10.1400/92271 · Cited by 68 · cited 1×
“emes for human values in SE. Therefore, we take inspiration from the social sciences. Social scientists have been searching for the most useful way to conceptualize basic human values since the 1950s [31]. In 1973, Rockeach captured 36 human values and organized them into 2 categories [25]. In 1992, Schwartz introduce…”
BackgroundValues-first SE: research principles in practice
Ferrario, M. A., Simm, W., Forshaw, S., Gradinar, A., Smith, M. T., & Smith, I. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2889219 · Cited by 60 · cited 3×
“Ignoring human values while engineering software may result in violating those values [10, 22] and subsequent dissatisfaction of”
BackgroundUsing Stories to Teach Human Values to Artificial Agents
Riedl, M. O., & Harrison, B. E. (2016) · National Conference on Artificial Intelligence · Cited by 52
Analyzing a decade of Linux system calls
Bagherzadeh, M., Kahani, N., Bezemer, C., Hassan, A. E., Dingel, J., & Cordy, J. R. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9551-z · Cited by 47
Software engineering article types: An analysis of the literature
Montesi, M., & Lago, P. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2007.11.723 · Cited by 45 · cited 4×
“ture [20, 34, 39, 42] as a way of providing insights on trends and directions in SE research.”
BackgroundMethodInfluentialHow healthy are software engineering conferences?
Vasilescu, B., Serebrenik, A., Mens, T., van den Brand, M., & Pek, E. (2014) · Science of Computer Programming · doi:10.1016/j.scico.2014.01.016 · Cited by 41 · cited 1×
“[41] by proposing a wider collection of metrics to assess the health of 11 SE conferences over a period of more than 10 years.”
BackgroundBarceló, J. A., Castillo, F. D., del Olmo, R., Mameli, L., Quesada, F. J. M., Poza, D. J., & Vilà, X. (2013) · Social Science Computer Review · doi:10.1177/0894439313511943 · Cited by 31
Prove it! Inferring Formal Proof Scripts from CafeOBJ Proof Scores
Riesco, A., & Ogata, K. (2018) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3208951 · Cited by 25
A categorization scheme for software engineering conference papers and its application
Bertolino, A., Calabrò, A., Lonetti, F., Marchetti, E., & Miranda, B. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.11.048 · Cited by 18 · cited 9×
“Alignedwith previous studies [4], we calculated inter-rater agreement using Fleiss’ Kappa, once attempts at resolving disagreements”
BackgroundMethodResultInfluentialDesign for Values in Software Development
Aldewereld, H., Dignum, V., & Tan, Y. (2014) · Springer eBooks · doi:10.1007/978-94-007-6994-6_26-1 · Cited by 16
A holistic overview of software engineering research strategies
Stol, K.-J., & Fitzgerald, B. F. (2015) · Arrow@dit (Dublin Institute of Technology) · Cited by 13
Writing Good Software Engineering Research Papers: Revisited
Theisen, C., Dunaiski, M., Williams, L. A., & Visser, W. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-c.2017.51 · Cited by 11 · cited 1×
“Classification based on abstracts is a well-accepted method in the SE literature [35, 41].”
MethodModel-driven software engineering in practice: privacy-enhanced filtering of network traffic
van Dijk, R., Creeten, C., van der Ham, J., & van den Bos, J. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3117777 · Cited by 3 · cited 1×
“otherwise sensitive information, ... We present Privacy-Enhanced Filtering (PEF), a model-driven prototype framework that relies on declarative descriptions of protocols and a set of filter rules ... [8] Helpful ... However, newcomers face many barriers when making their first contribution to an OSS project, leading i…”
Research works printed but not linked: 7
- Nick Baker. 2019. Molly Russell: Instagram Bans Graphic Self-Harm Images After Suicide of UK Teen. https://www.sbs.com.au/news/molly-russell-instagram-ba ns-graphic-self-harm-images-after-suicide-of-uk-teen
- Common Cause. 2011. The Common Cause Handbook-A Guide to Values and Frames for Campaigners, Community Organisers, Civil Servants, Fundraisers, Educators, Social Entrepreneurs, Activists, Funders, Politicians, and everyone in between. Public Interest Research Centre, Machynlleth, Wales (2011).
- ICSE. 2018. ICSE -Software Engineering in Society. https://www.icse2018.org/t rack/icse-2018-Software-Engineering-in-Society
- Md. Saddam Hossain Mukta, Euna Mehnaz Khan, Mohammed Eunus Ali, and Jalal Mahmud. 2017. Predicting Movie Genre Preferences from Personality and Values of Social Media Users. In Eleventh International AAAI Conference on Web and Social Media.
- Shalom H. Schwartz. 2005. Basic Human Values: Their Content and Structure Across Countries. Valores e Comportamento nas Organizações (2005), 21-55.
- Shalom H Schwartz. 2006. Basic Human Values: An Overview. Recuperado de http://www. yourmorals. org/schwartz (2006).
- Courtney Williams, Margaret-Anne Storey, Neil A Ernst, Alexey Zagalsky, and Eirini Kalliamvakou. 2019. Methodology Matters: How We Study Socio-Technical Aspects in Software Engineering. arXiv preprint arXiv:1905.12841 (2019). · arXiv
Cited by
Socio-Technical Grounded Theory for Software Engineering
Hoda, R. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3106280 · Cited by 204
“ues [44], [60], and the human-in-the-loop in order to”
BackgroundOperationalising ethics in artificial intelligence for healthcare: a framework for AI developers
Solanki, P., Grundy, J., & Hussain, W. (2023) · AI and Ethics · doi:10.1007/s43681-022-00195-z · Cited by 157
“Of the ten total human values in this theory, we reference only the four found to be most cited in software engineering literature [23].”
BackgroundSelf-citationTowards a roadmap on software engineering for responsible AI
Lu, Q., Zhu, L., Xu, X., Whittle, J., & Xing, Z. (2022) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3522664.3528607 · Cited by 103
“But these efforts are still limited to a small subset of human values [40].”
BackgroundSelf-citationHuman Values in Software Engineering: Contrasting Case Studies of Practice
Hussain, W., Perera, H., Whittle, J., Nurwidyantoro, A., Hoda, R., Shams, R. A., & Oliver, G. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3038802 · Cited by 59
Self-citationAI and Ethics—Operationalizing Responsible AI
Zhu, L., Xu, X., Lu, Q., Governatori, G., & Whittle, J. (2021) · Springer eBooks · doi:10.1007/978-3-030-72188-6_2 · Cited by 86
Self-citation
Metadata
- Authors
- Perera, H., Hussain, W., Whittle, J., Nurwidyantoro, A., Mougouei, D., Shams, R. A., & Oliver, G.
- Affiliations
- Monash University (AU)
- ORCID
- Venue
- Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2020-06-27
- Pages
- 409-420
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W2959714675
- Cited by
- 63
- References
- 40
- Citations by year
- 2026: 3 · 2025: 8 · 2024: 7 · 2023: 12 · 2022: 13 · 2021: 12 · 2020: 6
- FWCI
- 8.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- Figshare (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- human values, software engineering, equality, fairness, software engineering conferences, value structure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.