On the Use of Grey Literature: A Survey with the Brazilian Software Engineering Research Community
Abstract
Background: The use of Grey Literature (GL) has been investigated in diverse research areas. In Software Engineering (SE), this topic has an increasing interest over the last years. Problem: Even with the increase of GL published in diverse sources, the understanding of their use on the SE research community is still controversial. Objective: To understand how Brazilian SE researchers use GL, we aimed to become aware of the criteria to assess the credibility of their use, as well as the benefits and challenges. Method: We surveyed 76 active SE researchers participants of a flagship SE conference in Brazil, using a questionnaire with 11 questions to share their views on the use of GL in the context of SE research. We followed a qualitative approach to analyze open questions. Results: We found that most surveyed researchers use GL mainly to understand new topics. Our work identified new findings, including: 1) GL sources used by SE researchers (e.g., blogs, community website); 2) motivations to use (e.g., to understand problems and to complement research findings) or reasons to avoid GL (e.g., lack of reliability, lack of scientific value); 3) the benefit that is easy to access and read GL and the challenge of GL to have its scientific value recognized; and 4) criteria to assess GL credibility, showing the importance of the content owner to be renowned (e.g., renowned author and institutions). Conclusions: Our findings contribute to form a body of knowledge on the use of GL by…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Widespread availability of generative AI does not imply competent use.
“However, widespread availability does not imply competent use [15].”
From Students' Practices and Skills in the LLM-Era: "You Can't Outsource the Struggle and Still Get the Skill" (2026) · cites this paper as [15]
Research in grey literature (GL) has credibility and serves as empirical evidence in software engineering (SE), even if it has not undergone peer review.
“Furthermore, research in GL has credibility and serves as empirical evidence in SE, even if it has not undergone peer review [15, 16].”
From Students' Practices and Skills in the LLM-Era: "You Can't Outsource the Struggle and Still Get the Skill" (2026) · cites this paper as [15]
Cites
(2005) · PubMed · Cited by 7,011 · see citing works
Systematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,074 · see citing works
Systematic Reviews in the Social Sciences: A Practical Guide
Petticrew, M., & Roberts, H. (2006) · doi:10.1002/9780470754887 · Cited by 3,586 · see citing works
Linnenluecke, M. K. (2017) · International Journal of Management Reviews · doi:10.1111/ijmr.12076 · Cited by 1,682 · see citing works
Gray literature: An important resource in systematic reviews
Paez, A. (2017) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12266 · Cited by 1,459 · see citing works
Adams, R. J., Smart, P., & Huff, A. S. (2017) · International Journal of Management Reviews · doi:10.1111/ijmr.12102 · Cited by 995 · see citing works
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831 · see citing works
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486 · see citing works
The pains and gains of microservices: A Systematic grey literature review
Soldani, J., Tamburri, D. A., & van den Heuvel, W. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.09.082 · Cited by 438 · see citing works
Stack Overflow Considered Harmful? The Impact of Copy&Paste on Android Application Security
Fischer, F., Bottinger, K., Xiao, H., Stransky, C., Acar, Y., Backes, M., & Fahl, S. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.31 · Cited by 280 · see citing works
Sharma, T., & Spinellis, D. D. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.12.034 · Cited by 270 · see citing works
Storey, M.-A., Zagalsky, A., Filho, F. F., Singer, L., & German, D. M. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584053 · Cited by 221 · see citing works
Garousi, V., Felderer, M., & Mäntylä, M. V. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916008 · Cited by 210 · see citing works
The (R) Evolution of social media in software engineering
Storey, M., Singer, L., Cleary, B., Filho, F. F., & Zagalsky, A. (2014) · Future of Software Engineering (FOSE) · doi:10.1145/2593882.2593887 · Cited by 198 · see citing works
Using CMMI together with agile software development: A systematic review
Silva, F. S., Soares, F. S. F., Peres, A. L., de Azevedo, I. M., Vasconcelos, A. P., Kamei, F., & Meira, S. R. D. L. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.09.012 · Cited by 155 · see citing works
Self-citationGuidelines for Conducting Surveys in Software Engineering
Linåker, J., Sulaman, S. M., de Mello, R., & Höst, M. (2015) · Lund University Publications (Lund University) · Cited by 151 · see citing works
Software engineering at the speed of light: how developers stay current using twitter
Singer, L., Filho, F. F., & Storey, M. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568305 · Cited by 131 · see citing works
On using grey literature and google scholar in systematic literature reviews in software engineering
Yasin, A., Fatima, R., Wen, L., Afzal, W., Azhar, M., & Torkar, R. (2020) · IEEE Access · doi:10.1109/access.2020.2971712 · Cited by 85 · see citing works
How modern news aggregators help development communities shape and share knowledge
Aniche, M. F., Treude, C., Steinmacher, I., Wiese, I., Pinto, G., Storey, M., & Gerosa, M. A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180180 · Cited by 78 · see citing works
Self-citationCode, Camera, Action: How Software Developers Document and Share Program Knowledge Using YouTube
MacLeod, L., Storey, M., & Bergen, A. (2015) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2015.19 · Cited by 75 · see citing works
Kitchenham, B. A., Pearl Brereton, A., Turner, M., Niazi, M., Linkman, S., Pretorius, R., & Budgen, D. (2009) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2009.5314238 · Cited by 60 · see citing works
Training Software Engineers Using Open-Source Software: The Students' Perspective
Pinto, G., Ferreira, C., Souza, C., Steinmacher, I., & Meirelles, P. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet.2019.00024 · Cited by 60 · see citing works
Self-citationChoosing the Right Test Automation Tool
Raulamo-Jurvanen, P., Mäntylä, M., & Garousi, V. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084252 · Cited by 46 · see citing works
Cartaxo, B., Pinto, G., Vieira, E. R., & Soares, S. (2016) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2961111.2962603 · Cited by 44 · see citing works
Self-citationMultivocal literature reviews in software engineering: Preliminary findings from a tertiary study
Neto, G. T. G., Santos, W. B., Endo, P. T., & Fagundes, R. A. A. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870142 · Cited by 29 · see citing works
Mining Questions Asked about Continuous Software Engineering
Zahedi, M., Rajapakse, R. N., & Babar, M. A. (2020) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3383219.3383224 · Cited by 29 · see citing works
An evidence-based inquiry into the use of grey literature in software engineering
Zhang, H., Zhou, X., Huang, X., Huang, H., & Babar, M. A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380336 · Cited by 29 · see citing works
Toward the use of blog articles as a source of evidence for software engineering research
Williams, A. H., & Rainer, A. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084268 · Cited by 15 · see citing works
Using Blog Articles in Software Engineering Research: Benefits, Challenges and Case–Survey Method
Rainer, A., & Williams, A. H. (2018) · Australasian Software Engineering Conference (ASWEC) · doi:10.1109/aswec.2018.00034 · Cited by 14 · see citing works
Williams, A. H., & Rainer, A. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319013 · Cited by 11 · see citing works
Clarke, M. J., & Li, Y. (2017) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12247 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Grey Literature in Software Engineering: A critical review
Kamei, F., Wiese, I., Lima, C., Polato, I., Nepomuceno, V., Ferreira, W., Ribeiro, M., Pena, C., Cartaxo, B., Pinto, G., et al. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106609 · Cited by 70 · see citing works
Self-citationWhat Evidence We Would Miss If We Do Not Use Grey Literature?
Kamei, F., Pinto, G., Wiese, I., Ribeiro, M., & Soares, S. (2021) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3475716.3475777 · Cited by 21 · see citing works
Self-citationStudying the Interplay Between the Durations and Breakages of Continuous Integration Builds
Ghaleb, T. A., Hassan, S., & Zou, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3222160 · Cited by 14 · see citing works
Organizing Empirical Studies as Learning Iterations in Design Science Research Projects
Barcellos, M. P., Santos, G., Conte, T. U., Trinkenreich, B., & Matsubara, P. (2022) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3571473.3571474 · Cited by 7 · see citing works
de Quadros, E. L. L., Lisboa, A., de Souza, M. Q. B., Prikladnicki, R., & Chaves, M. S. (2022) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3571473.3571498 · Cited by 6 · see citing works
Grey Literature in Software Engineering: A Critical ReviewPreprint
Kamei, F., Wiese, I., Lima, C., Polato, I., Nepomuceno, V., Ferreira, W., Ribeiro, M., Pena, C., Cartaxo, B., Pinto, G., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13435 · Cited by 6 · see citing works
Self-citationHuman in the loop chain of code prompting for deterministic tool development with generative AI
Atkinson, C. F. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00704-z · Cited by 3 · see citing works
Characterizing Toolkits for Platform Independent Chatbot Development
Leifheit, B. R., Basso, F. P., & Silva, W. (2023) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3592813.3592885 · Cited by 3 · see citing works
Factors that Affect Merge Conflicts: A Software Developers’ Perspective
Costa, C. D. S., Menezes, J. W., Trindade, B., & dos Santos, R. P. (2021) · Brazilian Symposium on Software Engineering · doi:10.1145/3474624.3474641 · Cited by 3 · see citing works
Atkinson, C. F. (2025) · doi:10.21203/rs.3.rs-5702366/v1 · Cited by 3 · see citing works
Nazário, M., Bonifácio, R., de Souza, C. R. B., Kamei, F., & Pinto, G. (2025) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2025.4378 · Cited by 1 · see citing works
Self-citationAtkinson, C. F. (2024) · Research Square · doi:10.21203/rs.3.rs-4007725/v1 · Cited by 1 · see citing works
de Freitas, E. R. S., Pinto, G., & Monteiro, D. (2026) · arXiv · doi:10.48550/arxiv.2607.29519
“Furthermore, research in GL has credibility and serves as empirical evidence in SE, even if it has not undergone peer review [15, 16].”
Self-citationWhat Evidence We Would Miss If We Do Not Use Grey Literature?Preprint
Kamei, F., Pinto, G., Wiese, I., Ribeiro, M., & Soares, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05792
Self-citationde Lima, L. F., & Peres, L. M. (2022) · SBC Reviews on Computer Science · doi:10.5753/reviews.2022.2724
Vojičić, N., Sekulski, D., & Urošević, V. (2025) · Smart innovation, systems and technologies · doi:10.1007/978-981-96-3077-6_25
Mahmoudi, B., Chenail-Larcher, Z., Cherief, H. A., Stiévenart, Q., Moha, N., & Avellaneda, F. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793304
- Authors
- Kamei, F., Wiese, I., Pinto, G., Ribeiro, M., & Soares, S.
- Affiliations
- Instituto Federal de Educação, Ciência e Tecnologia de Alagoas (BR)
- Universidade Tecnológica Federal do Paraná (BR)
- Universidade Federal do Pará (BR)
- Universidade Federal de Alagoas (BR)
- Universidade Federal de Pernambuco (BR)
- Venue
- Proceedings of the XXXIV Brazilian Symposium on Software Engineering
- Publisher
- ACM
- Published
- 2020-10-21
- Pages
- 183-192
- OpenAlex
- W3114035905
- Cited by
- 15
- References
- 31
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 1 · 2023: 1 · 2022: 4 · 2021: 5
- FWCI
- 1.90
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Brazilian Symposium on Software Engineering (publishedVersion)
- Funders
- Fundação de Amparo à Ciência e Tecnologia do Estado de Pernambuco (APQ-0388-1.03/14, APQ-0399-1.03/17)
- Conselho Nacional de Desenvolvimento Científico e Tecnológico (465614/2014-0, 309697/2019-0)
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (88887.136410/2017-00)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Business, Management and Accounting › Management Information Systems › Optics and Image Analysis
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- grey literature, software engineering research, researcher practices, credibility assessment, information sources, blogs, community websites, scientific values
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
