The DOI matches its Crossref record; 178 of the 178 references it cites were found in a registry or an index.
A survey of the use of crowdsourcing in software engineering
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Software Engineering Economics
Boehm, B. W. (2001) · Pioneers and Their Contributions to Software Engineering · doi:10.1007/978-3-642-48354-7_5 · Cited by 5,981
The New Science of Management Decision
Hershauer, J. C., & Simon, H. A. (1978) · The Academy of Management Review · doi:10.2307/257591 · Cited by 3,200
The new science of management decision.
Simon, H. A. (1960) · Harper & Brothers eBooks · doi:10.1037/13978-000 · Cited by 2,437
Crowdsourcing as a Model for Problem Solving: An Introduction and Cases
Brabham, D. C. (2008) · Convergence: The International Journal of Research into New Media Technologies · doi:10.1177/1354856507084420 · Cited by 1,943
Towards an integrated crowdsourcing definition
Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012) · Journal of Information Science · doi:10.1177/0165551512437638 · Cited by 1,858
Information systems outsourcing: a survey and analysis of the literature
Dibbern, J., Goles, T., Hirschheim, R., & Jayatilaka, B. (2004) · ACM SIGMIS Database: the DATABASE for Advances in Information Systems · doi:10.1145/1035233.1035236 · Cited by 1,381
Predicting protein structures with a multiplayer online game
Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z., & players, F. (2010) · Nature · doi:10.1038/nature09304 · Cited by 1,368
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104
Self-citationGenProg: A Generic Method for Automatic Software Repair
Le Goues, C., Nguyen, T., Forrest, S., & Weimer, W. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,085
Farrell, J., & Rabin, M. (1996) · Journal of Economic Perspectives · doi:10.1257/jep.10.3.103 · Cited by 1,012
An architecture-based approach to self-adaptive software
Oreizy, P., Gorlick, M. M., Taylor, R. N., Heimhigner, D., Johnson, G., Medvidovic, N., Quilici, A., Rosenblum, D. S., & Wolf, A. L. (1999) · IEEE Intelligent Systems · doi:10.1109/5254.769885 · Cited by 988
Quality management on Amazon Mechanical Turk
Ipeirotis, P. G., Provost, F., & Wang, J. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837906 · Cited by 973
Guide to the Software Engineering Body of Knowledge - SWEBOK
Abran, A., Bourque, P., Dupuis, R., & Moore, J. W. (2001) · Cited by 934
Search-based software engineering
Harman, M., & Jones, B. F. (2001) · Information and Software Technology · doi:10.1016/s0950-5849(01)00189-6 · Cited by 914
Self-citationCrowdroid: behavior-based malware detection system for Android
Burguera, I., Zurutuza, U., & Nadjm-Tehrani, S. (2011) · Proceedings of the 1st ACM workshop on Security and privacy in smartphones and mobile devices · doi:10.1145/2046614.2046619 · Cited by 910
Feedback-Directed Random Test Generation
Pacheco, C., Lahiri, S. K., Ernst, M. D., & Ball, T. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.37 · Cited by 863
Pex–White Box Test Generation for .NET
Tillmann, N., & de Halleux, J. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-79124-9_10 · Cited by 791
Leveraging Crowdsourcing: Activation-Supporting Components for IT-Based Ideas Competition
Leimeister, J. M., Huber, M., Bretschneider, U., & Krcmar, H. (2009) · Journal of Management Information Systems · doi:10.2753/mis0742-1222260108 · Cited by 745
Longitude: the true story of a lone genius who solved the greatest scientific problem of his time
(1996) · Choice Reviews Online · doi:10.5860/choice.33-3880 · Cited by 698
On testing non-testable programs
Weyuker, E. J. (1982) · The Computer Journal · doi:10.1093/comjnl/25.4.465 · Cited by 606
How Do Users Like This Feature? A Fine Grained Sentiment Analysis of App Reviews
Guzman, E., & Maalej, W. (2014) · 2014 IEEE 22nd International Requirements Engineering Conference (RE) · doi:10.1109/re.2014.6912257 · Cited by 590
Crystal structure of a monomeric retroviral protease solved by protein folding game players
Foldit Contenders Group, Khatib, F., Foldit Void Crushers Group, DiMaio, F., Cooper, S., Kazmierczyk, M., Gilski, M., Krzywda, S., Zabranska, H., Pichova, I., et al. (2011) · Nature Structural & Molecular Biology · doi:10.1038/nsmb.2119 · Cited by 540
User feedback in the appstore: An empirical study
Pagano, D., & Maalej, W. (2013) · 2013 21st IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2013.6636712 · Cited by 540
Open-Source Software Development and Distributed Innovation
Kogut, B. (2001) · Oxford Review of Economic Policy · doi:10.1093/oxrep/17.2.248 · Cited by 527
AR-miner: mining informative reviews for developers from mobile app marketplace
Chen, N., Lin, J., Hoi, S. C. H., Xiao, X., & Zhang, B. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568263 · Cited by 517
Motivation in software engineering: A systematic literature review
Beecham, S., Baddoo, N., Hall, T., Robinson, H., & Sharp, H. (2008) · Information and Software Technology · doi:10.1016/j.infsof.2007.09.004 · Cited by 506
Lin (2012) · doi:10.1145/2370216.2370290 · Cited by 451
CrowdForge: crowdsourcing complex work
Kittur, A., Smus, B., & Kraut, R. (2011) · CHI '11 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1979742.1979902 · Cited by 434
SemFix: program repair via semantic analysis
Nguyen, H. D. T., Qi, D., Roychoudhury, A., & Chandra, S. (2013) · doi:10.5555/2486788.2486890 · Cited by 398
A Survey of Crowdsourcing Systems
Yuen, M.-C., King, I., & Leung, K.-S. (2011) · 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing · doi:10.1109/passat/socialcom.2011.203 · Cited by 392
CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools
Sen, K., & Agha, G. (2006) · Lecture Notes in Computer Science · doi:10.1007/11817963_38 · Cited by 389
Quality Control in Crowdsourcing Systems: Issues and Directions
Allahbakhsh, M., Benatallah, B., Ignjatovic, A., Motahari-Nezhad, H. R., Bertino, E., & Dustdar, S. (2013) · IEEE Internet Computing · doi:10.1109/mic.2013.20 · Cited by 385
GUI ripping: reverse engineering of graphical user interfaces for testing
Memon, A., Banerjee, I., & Nagarajan, A. (2004) · 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings. · doi:10.1109/wcre.2003.1287256 · Cited by 379
Crowdsourcing for climate and atmospheric sciences: current status and future potential
Muller, C. L., Chapman, L., Johnston, S., Kidd, C., Illingworth, S., Foody, G., Overeem, A., & Leigh, R. R. (2015) · International Journal of Climatology · doi:10.1002/joc.4210 · Cited by 372
Software engineering economics
Al Hady, M. M. (2007) · Cybrarians Journal · doi:10.70000/cj.2007.14.426 · Cited by 356
Rules of Crowdsourcing: Models, Issues, and Systems of Control
Saxton, G. D., Oh, O., & Kishore, R. (2013) · Information Systems Management · doi:10.1080/10580530.2013.739883 · Cited by 340
Crowdsourcing for relevance evaluation
Alonso, O., Rose, D. E., & Stewart, B. (2008) · ACM SIGIR Forum · doi:10.1145/1480506.1480508 · Cited by 329
Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing
Hossfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., & Tran-Gia, P. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2291663 · Cited by 283
What would other programmers do: suggesting solutions to error messages
Hartmann, B., MacDougall, D., Brandt, J., & Klemmer, S. R. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753478 · Cited by 280
StackOverflow and GitHub: Associations between Software Development and Crowdsourced Knowledge
Vasilescu, B., Filkov, V., & Serebrenik, A. (2013) · 2013 International Conference on Social Computing · doi:10.1109/socialcom.2013.35 · Cited by 277
Alice in warningland: a large-scale field study of browser security warning effectiveness
Akhawe, D., & Felt, A. P. (2013) · Cited by 260
Implementing the “Wisdom of the Crowd”
Kremer, I., Mansour, Y., & Perry, M. (2014) · Journal of Political Economy · doi:10.1086/676597 · Cited by 259
Mining StackOverflow to turn the IDE into a self-confident programming prompter
Ponzanelli, L., Bavota, G., Di Penta, M., Oliveto, R., & Lanza, M. (2014) · Proceedings of the 11th Working Conference on Mining Software Repositories · doi:10.1145/2597073.2597077 · Cited by 240
App StoreMining and Analysis: MSR for App Stores
Harman, M., Jia, Y., & Zhang, Y. (2012) · Cited by 235
Self-citationOptimizing Existing Software With Genetic Programming
Langdon, W. B., & Harman, M. (2015) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2013.2281544 · Cited by 233
Self-citationLim, S. L., & Finkelstein, A. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.36 · Cited by 220
The impact of social media on software engineering practices and tools
Storey, M.-A., Treude, C., van Deursen, A., & Cheng, L.-T. (2010) · Proceedings of the FSE/SDP workshop on Future of software engineering research · doi:10.1145/1882362.1882435 · Cited by 213
Two's company, three's a crowd: a case study of crowdsourcing software development
Stol, K.-J., & Fitzgerald, B. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568249 · Cited by 203
How social Q&A sites are changing knowledge sharing in open source software communities
Vasilescu, B., Serebrenik, A., Devanbu, P., & Filkov, V. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531659 · Cited by 200
Begel, A., Bosch, J., & Storey, M.-A. (2013) · IEEE Software · doi:10.1109/ms.2013.13 · Cited by 197
Archak, N. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772694 · Cited by 187
Current challenges in automatic software repair
Le Goues, C., Forrest, S., & Weimer, W. (2013) · Software Quality Journal · doi:10.1007/s11219-013-9208-0 · Cited by 182
ProtectMyPrivacy: detecting and mitigating privacy leaks on iOS devices using crowdsourcing
Agarwal, Y., & Hall, M. (2013) · Proceeding of the 11th annual international conference on Mobile systems, applications, and services · doi:10.1145/2462456.2464460 · Cited by 182
Models of motivation in software engineering
Sharp, H., Baddoo, N., Beecham, S., Hall, T., & Robinson, H. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.05.009 · Cited by 177
Using test oracles generated from program documentation
Peters, D. K., & Parnas, D. L. (1998) · IEEE Transactions on Software Engineering · doi:10.1109/32.667877 · Cited by 171
SEEDS: a software engineer's energy-optimization decision support framework
Manotas, I., Pollock, L., & Clause, J. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568297 · Cited by 170
Crowdsourcing for usability testing
Liu, D., Bias, R. G., Lease, M., & Kuipers, R. (2012) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.14504901100 · Cited by 138
Prize-based contests can provide solutions to computational biology problems
Lakhani, K. R., Boudreau, K. J., Loh, P., Bäckström, L., Baldwin, C. Y., Lonstein, E., Lydon, M., MacCormack, A., Arnaout, R., & Guinan, E. C. (2013) · Nature Biotechnology · doi:10.1038/nbt.2495 · Cited by 137
The metropolis model a new logic for development of crowdsourced systems
Kazman, R., & Chen, H. (2009) · Communications of the ACM · doi:10.1145/1538788.1538808 · Cited by 136
Crowdsourcing and Its Application to Transportation Data Collection and Management
Misra, A., Gooze, A., Watkins, K. E., Asad, M., & Dantec, C. A. L. (2014) · Transportation Research Record Journal of the Transportation Research Board · doi:10.3141/2414-01 · Cited by 133
The jabberwocky programming environment for structured social computing
Ahmad, S., Battle, A. J., Malkani, Z., & Kamvar, S. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2047196.2047203 · Cited by 132
Efficient crowdsourcing of unknown experts using bounded multi-armed bandits
Tran-Thanh, L., Stein, S., Rogers, A., & Jennings, N. R. (2014) · Artificial Intelligence · doi:10.1016/j.artint.2014.04.005 · Cited by 132
Real-time collaborative coding in a web IDE
Goldman, M., Little, G., & Miller, R. C. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2047196.2047215 · Cited by 130
Solving the Search for Source Code
Stolee, K. T., Elbaum, S. G., & Dobos, D. (2014) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2581377 · Cited by 125
StakeNet: using social networks to analyse the stakeholders of large-scale software projects
Lim, S. L., Quercia, D., & Finkelstein, A. (2010) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1806799.1806844 · Cited by 123
Social media for software engineering
Begel, A., DeLine, R. A., & Zimmermann, T. (2010) · FSE/SDP Workshop on Future of Software Engineering Research (FoSER) · doi:10.1145/1882362.1882370 · Cited by 120
A Survey of Documentation Practice within Corrective Maintenance
Kajko‐Mattsson, M. (2004) · Empirical Software Engineering · doi:10.1023/b:lida.0000048322.42751.ca · Cited by 120
Zogaj, S., Bretschneider, U., & Leimeister, J. M. (2014) · Journal of Business Economics · doi:10.1007/s11573-014-0721-9 · Cited by 119
AutoMan: a platform for integrating human-based and digital computation
Barowy, D. W., Curtsinger, C., Berger, E. D., & McGregor, A. (2012) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2384616.2384663 · Cited by 118
Developer Recommendation for Crowdsourced Software Development Tasks
Mao, K., Yang, Y., Wang, Q., Jia, Y., & Harman, M. (2015) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose.2015.46 · Cited by 115
Self-citationA human study of patch maintainability
Fry, Z. P., Landau, B., & Weimer, W. R. (2012) · International Symposium on Software Testing and Analysis · doi:10.1145/2338965.2336775 · Cited by 113
Reducing Energy Consumption Using Genetic Improvement
Bruce, B. R., Petke, J., & Harman, M. (2015) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2739480.2754752 · Cited by 113
Self-citationQuadrant of euphoria: a crowdsourcing platform for QoE assessment
Chen, K.-T., Chang, C.-J., Wu, C., et al. (2010) · IEEE Network · doi:10.1109/mnet.2010.5430141 · Cited by 112
Making web applications more energy efficient for OLED smartphones
Li, D., Tran, A. H., & Halfond, W. G. J. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568321 · Cited by 103
Calcite: Completing Code Completion for Constructors Using Crowds
Mooty, M., Faulring, A. R., Stylos, J., & Myers, B. A. (2010) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2010.12 · Cited by 102
Wu, F., Weimer, W. R., Harman, M., Jia, Y., & Krinke, J. (2015) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2739480.2754648 · Cited by 99
Self-citationRepeatable and reliable search system evaluation using crowdsourcing
Blanco, R., Halpin, H., Herzig, D. M., et al. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2010039 · Cited by 98
Using Mobile RE Tools to Give End-Users Their Own Voice
Seyff, N., Gräf, F., & Maiden, N. (2010) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2010.15 · Cited by 94
Flight of the FINCH Through the Java Wilderness
Orlov, M., & Sipper, M. (2011) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2010.2052622 · Cited by 94
CrowdOracles: Can the Crowd Solve the Oracle Problem?
Pastore, F., Mariani, L., & Fraser, G. (2013) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2013.13 · Cited by 93
Components and Functions of Crowdsourcing Systems – A Systematic Literature Review
Hetmank, L. (2013) · Journal of the Association for Information Systems · doi:10.13140/2.1.3836.4166 · Cited by 91
Evolving Readable String Test Inputs Using a Natural Language Model to Reduce Human Oracle Cost
Afshan, S., McMinn, P., & Stevenson, M. (2013) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2013.11 · Cited by 91
Microtask programming: building software with a crowd
LaToza, T. D., Towne, W. B., Adriano, C., & van der Hoek, A. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2642918.2647349 · Cited by 85
Liang, C.-J. M., Lane, N. D., Brouwers, N., Zhang, L., Karlsson, B. F., Liu, H., Liu, Y., Tang, J., Shan, X., Chandra, R., et al. (2014) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/2639108.2639131 · Cited by 84
Seahawk: Stack Overflow in the IDE
Ponzanelli, L., Bacchelli, A., & Lanza, M. (2016) · Cited by 82
Apparition: Crowdsourced User Interfaces that Come to Life as You Sketch Them
Lasecki, W. S., Kim, J., Rafter, N., Sen, O., Bigham, J. P., & Bernstein, M. S. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702565 · Cited by 81
LemonAid: selection-based crowdsourced contextual help for web applications
Chilana, P. K., Ko, A. J., & Wobbrock, J. O. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208620 · Cited by 79
Collaborative Software Development Platforms for Crowdsourcing
Peng, X., Babar, M. A., & Ebert, C. (2014) · IEEE Software · doi:10.1109/ms.2014.31 · Cited by 76
Leveraging Crowd Knowledge for Software Comprehension and Development
Ponzanelli, L., Bacchelli, A., & Lanza, M. (2013) · Euromicro Conference on Software Maintenance and Reengineering (CSMR) · doi:10.1109/csmr.2013.16 · Cited by 75
(2012) · doi:10.5555/2666719.2666725 · Cited by 74
Look before you leap: Legal pitfalls of crowdsourcing
Wolfson, S. M., & Lease, M. (2011) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.2011.14504801135 · Cited by 74
Ebner, W., Leimeister, M., Bretschneider, U., & Krcmar, H. (2008) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2008.233 · Cited by 66
An evaluation framework for software crowdsourcing
Wu, W., Tsai, W., & Li, W. (2013) · Frontiers of Computer Science · doi:10.1007/s11704-013-2320-2 · Cited by 65
Ranking crowd knowledge to assist software development
de Souza, L. B. L., Campos, E. C., & Maia, M. D. A. (2014) · International Conference on Program Comprehension · doi:10.1145/2597008.2597146 · Cited by 63
2010 IEEE International Conference on Systems, Man and Cybernetics
(2010) · doi:10.1109/smc16229.2010 · Cited by 62
BlueFix: Using Crowd-Sourced Feedback to Support Programming Students in Error Diagnosis and Repair
Watson, C., Li, F. W. B., & Godwin, J. L. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33642-3_25 · Cited by 61
Social Adaptation - When Software Gives Users a Voice
Ali, R., Solís, C., Omoronyia, I., Salehie, M., & Nuseibeh, B. (2012) · International Conference on Evaluation of Novel Approaches to Software Engineering · doi:10.5220/0003991900750084 · Cited by 61
Crowdsourcing 2.0: Enhancing execution speed and reliability of web-based QoE testing
Gardlo, B., Egger, S., Seufert, M., & Schatz, R. (2014) · International Conference on Communications · doi:10.1109/icc.2014.6883463 · Cited by 59
Crowd-centric Requirements Engineering
Snijders, R., Dalpiaz, F., Hosseini, M., et al. (2014) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1109/ucc.2014.96 · Cited by 57
Crowdsourcing for information retrieval
Lease, M., & Yılmaz, E. (2012) · ACM SIGIR Forum · doi:10.1145/2093346.2093356 · Cited by 56
CrowdLang: A Programming Language for the Systematic Exploration of Human Computation Systems
Minder, P., & Bernstein, A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35386-4_10 · Cited by 55
A human study of fault localization accuracy
Fry, Z. P., & Weimer, W. R. (2010) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2010.5609691 · Cited by 54
Dolstra, E., Vliegendhart, R., & Pouwelse, J. A. (2013) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2013.44 · Cited by 54
Social sensing: when users become monitors
Ali, R., Solís, C., Salehie, M., Omoronyia, I., Nuseibeh, B., & Maalej, W. (2011) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2025113.2025196 · Cited by 54
The Apple business model: Crowdsourcing mobile applications
Bergvall‐Kåreborn, B., & Howcroft, D. (2013) · Accounting Forum · doi:10.1016/j.accfor.2013.06.001 · Cited by 54
Scaling requirements extraction to the crowd: Experiments with privacy policies
Breaux, T. D., & Schaub, F. (2014) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2014.6912258 · Cited by 51
Crowdsourcing and open source software participation
Olson, D. L., & Rosacker, K. M. (2012) · Service Business · doi:10.1007/s11628-012-0176-4 · Cited by 51
Chatfield, A. T., & Brajawidagda, U. (2014) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2014.281 · Cited by 48
Configuring crowdsourcing for requirements elicitation
Hosseini, M., Shahri, A., Phalp, K. T., Taylor, J., Ali, R., & Dalpiaz, F. (2015) · International Conference on Research Challenges in Information Science (RCIS) · doi:10.1109/rcis.2015.7128873 · Cited by 47
Genetic improvement for adaptive software engineering (keynote)
Harman, M., Jia, Y., Langdon, W. B., Petke, J., Moghadam, I. H., Yoo, S., & Wu, F. (2014) · International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) · doi:10.1145/2593929.2600116 · Cited by 47
Self-citationExample overflow: Using social media for code recommendation
Zagalsky, A., Barzilay, O., & Yehudai, A. (2012) · Cited by 47
Automatically generating realistic test input from web services
Bozkurt, M., & Harman, M. (2011) · doi:10.1109/sose.2011.6139088 · Cited by 47
Self-citationLeveraging Crowdsourcing to Facilitate the Discovery of New Medicines
Norman, T., Bountra, C., Edwards, A. M., Yamamoto, K. R., & Friend, S. H. (2011) · Science Translational Medicine · doi:10.1126/scitranslmed.3002678 · Cited by 46
Code hunt: experience with coding contests at scale
Bishop, J., Horspool, R. N., Xie, T., Tillmann, N., & de Halleux, J. (2015) · doi:10.5555/2819009.2819072 · Cited by 45
Verification games: making verification fun
Dietl, W., Dietzel, S., Ernst, M. D., Mote, N., Walker, B., Cooper, S., Pavlik, T., & Popović, Z. (2012) · Formal Techniques for Java-like Programs · doi:10.1145/2318202.2318210 · Cited by 45
LaToza, T. D., Towne, W. B., van der Hoek, A., & Herbsleb, J. D. (2013) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase.2013.6614737 · Cited by 45
Program Boosting: Program Synthesis via Crowd-Sourcing
Cochran, R. A., D’Antoni, L., Livshits, B., Molnár, D., & Veanes, M. (2014) · ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) · doi:10.1145/2676726.2676973 · Cited by 43
Chen, F., & Kim, S. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2786819 · Cited by 41
Challenges and Experiences in Deploying Enterprise Crowdsourcing Service
Vuković, M., Laredo, J., & Rajagopal, S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13911-6_31 · Cited by 40
Nebeling, M., Speicher, M., & Norrie, M. C. (2013) · ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) · doi:10.1145/2494603.2480304 · Cited by 40
Crowdsourced Exploration of Security Configurations
Ismail, Q., Ahmed, T., Kapadia, A. C., & Reiter, M. K. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702370 · Cited by 40
Puzzle-based automatic testing: bringing humans into the loop by solving puzzles
Chen, N., & Kim, S. (2012) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2351676.2351697 · Cited by 40
Nebeling, M., Speicher, M., & Norrie, M. C. (2013) · ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) · doi:10.1145/2494603.2480303 · Cited by 39
Researching crowdsourcing software development: perspectives and concerns
Stol, K.-J., & Fitzgerald, B. F. (2014) · doi:10.1145/2593728.2593731 · Cited by 39
The design of adaptive acquisition of users feedback: An empirical study
Almaliki, M., Ncube, C., & Ali, R. (2014) · International Conference on Research Challenges in Information Science (RCIS) · doi:10.1109/rcis.2014.6861076 · Cited by 38
(2014) · doi:10.1109/icsme.2014.99 · Cited by 38
More testers – The effect of crowd size and time restriction in software testing
Mäntylä, M., & Itkonen, J. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.12.004 · Cited by 37
Li, K., Xiao, J., Wang, Y., & Wang, Q. (2013) · IEEE Annual Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2013.133 · Cited by 37
Emergent, crowd-scale programming practice in the IDE
Fast, E., Steffee, D., Wang, L. L., Brandt, J., & Bernstein, M. S. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2556998 · Cited by 37
Who asked what: integrating crowdsourced FAQs into API documentation
Chen, C., & Kang Zhang, A. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2591062.2591128 · Cited by 36
Facilitating Crowd Sourced Software Engineering via Stack Overflow
Barzilay, O., Treude, C., & Zagalsky, A. (2013) · Springer eBooks · doi:10.1007/978-1-4614-6596-6_15 · Cited by 35
Social networks and collaborative filtering for large-scale requirements elicitation
Lim, S. L. (2010) · UNSWorks (UNSW Sydney) · doi:10.26190/unsworks/23395 · Cited by 34
SPHERES Zero Robotics software development: Lessons on crowdsourcing and collaborative competition
Nag, S., Heffan, I., Saenz‐Otero, A., & Lydon, M. (2012) · IEEE Aerospace Conference (AERO) · doi:10.1109/aero.2012.6187452 · Cited by 32
Borrowing from the Crowd: A Study of Recombination in Software Design Competitions
LaToza, T. D., Chen, M., Jiang, L., Zhao, M., & van der Hoek, A. (2015) · Cited by 30
Bruch, M., Bodden, E., Monperrus, M., & Mezini, M. (2010) · FSE/SDP Workshop on Future of Software Engineering Research (FoSER) · doi:10.1145/1882362.1882374 · Cited by 30
Determinants of success in crowdsourcing software development
Tajedin, H., & Nevo, D. (2013) · ACM SIGCPR Conference on Computer Personnel Research (SIGCPR) · doi:10.1145/2487294.2487332 · Cited by 29
An overview of industrial software documentation practice
Visconti, M., & Cook, C. (2003) · International Conference of the Chilean Computer Science Society (SCCC) · doi:10.1109/sccc.2002.1173192 · Cited by 27
Crowdsourcing suggestions to programming problems for dynamic web development languages
Mujumdar, D., Kallenbach, M., Liu, B., & Hartmann, B. (2011) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1979742.1979802 · Cited by 27
Identification, Impact, and Refactoring of Smells in Pipe-Like Web Mashups
Stolee, K. T., & Elbaum, S. G. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.42 · Cited by 26
A Game Theoretic Model of Software Crowdsourcing
Hu, Z., & Wu, W. (2014) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose.2014.79 · Cited by 25
Facing up to the inequality of crowdsourced API documentation
Jiau, H. C., & Yang, F.-P. (2012) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/2088883.2088892 · Cited by 24
Building test suites in social coding sites by leveraging drive-by commits
Pham, R., Singer, L., & Schneider, K. A. (2013) · doi:10.5555/2486788.2486964 · Cited by 23
Reducing the barriers to writing verified specifications
Schiller, T. W., & Ernst, M. D. (2012) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2384616.2384624 · Cited by 22
PROTEINCHALLENGE: Crowd sourcing in proteomics analysis and software development
Martin, S. F., Falkenberg, H., Dyrlund, T. F., Khoudoli, G. A., Mageean, C. J., & Linding, R. (2012) · Journal of Proteomics · doi:10.1016/j.jprot.2012.11.014 · Cited by 21
SmartNotes: Application of crowdsourcing to the detection of web threats
Sharifi, M., Fink, E., & Carbonell, J. (2011) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2011.6083845 · Cited by 20
Crowdsourcing software evaluation
Sherief, N. H., Jiang, N., Hosseini, M., Phalp, K. T., & Ali, R. (2014) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/2601248.2601300 · Cited by 20
Crowd-centric requirements engineering: A method based on crowdsourcing and gamification
Snijders, R., Atilla, Ö., Dalpiaz, F., & Brinkkemper, S. (2015) · Utrecht University Repository (Utrecht University) · Cited by 20
Chilana, P. K., Ko, A. J., Wobbrock, J. O., & Grossman, T. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470685 · Cited by 18
CrowdLang - First steps towards programmable human computers for general computation
Minder, P., & Bernstein, A. (2011) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-63241 · Cited by 18
Participation in crowd systems
Varshney, L. R. (2012) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2012.6483327 · Cited by 17
The metropolis model and its implications for the engineering of software ecosystems
Kazman, R., & Chen, H.-M. (2010) · doi:10.1145/1882362.1882402 · Cited by 17
Crowdsourcing user interface adaptations for minimizing the bloat in enterprise applications
Akiki, P. A., Bandara, A. K., & Yu, Y. (2013) · ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) · doi:10.1145/2494603.2480319 · Cited by 16
LaToza, T. D., & van der Hoek, A. (n.d.) · Cited by 15
The Power of the Crowd: Performing Usability Testing Using an On-Demand Workforce
Schneider, C., & Cheung, T. C.-H. (2013) · Springer eBooks · doi:10.1007/978-1-4614-4951-5_44 · Cited by 15
Social networks and crowdsourcing for stakeholder analysis in system of systems projects
Lim, S. L., & Ncube, C. (2013) · International Conference on System of Systems Engineering · doi:10.1109/sysose.2013.6575235 · Cited by 14
Understanding and capturing people's mobile app privacy preferences
Sadeh, N., Hong, J., & Lin, J. (2013) · Cited by 13
Nag, S. (2012) · Cited by 12
Software development with real-time collaborative editing
Miller, R. C., & Goldman, M. (2012) · Cited by 10
Social sensing: when users become monitors
Ali, R., Solís, C., Salehie, M., Omoronyia, I., Nuseibeh, B., Maalej, W., & Universtität, T. (2011) · Cited by 9
Method-call recommendations from implicit developer feedback
Amann, S., Proksch, S., & Mezini, M. (2014) · doi:10.1145/2593728.2593730 · Cited by 8
Constructing adaptive configuration dialogs using crowd data
Hamidi, S., Andritsos, P., & Liaskos, S. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642960 · Cited by 8
Context-Aware and Adaptive Web Interfaces: A Crowdsourcing Approach
Nebeling, M., & Norrie, M. C. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-27997-3_17 · Cited by 7
Bernstein, M. S. (2010) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1866218.1866220 · Cited by 7
Ochara, N. M., Asmelash, D., & Mlay, S. V. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2132040 · Cited by 7
Leveraging the crowd for creating wireframe-based exploration of mobile design pattern gallery
Huang, Y., Wang, C.-I., & Hsu, J. Y.-J. (2013) · doi:10.1145/2451176.2451182 · Cited by 6
Crowdsourcing in the Brazilian IT industry: what we know and what we don't know
Machado, L. S., Pereira, G. B., Prikladnicki, R., Carmel, E., & de Souza, C. R. B. (2014) · doi:10.1145/2666539.2666567 · Cited by 6
Value-Adding Intermediaries in Software Crowdsourcing
Tajedin, H., & Nevo, D. (2014) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2014.181 · Cited by 6
Supporting Complex Work in Crowdsourcing Platforms: A View from Service-Oriented Computing
Xiao, L., & Paik, H.-Y. (2014) · Australasian Software Engineering Conference (ASWEC) · doi:10.1109/aswec.2014.11 · Cited by 6
Software evaluation via users' feedback at runtime
Sherief, N. H. (2014) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/2601248.2613085 · Cited by 5
IDE 2.0: Leveraging the Wisdom of the Software Engineering Crowds
Bruch, M. (2012) · TUbilio (Technical University of Darmstadt) · doi:10.26083/tuprints-00003137 · Cited by 4
Using redundancy to improve security and testing
Xue, H. (2013) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 4
Ghost Map: Proving Software Correctness using Games
Watro, R. J., Moffitt, K., Hussain, T. S., Wyschogrod, D., Ostwald, J., Kong, D., Bowers, C. A., Church, E., Guttman, J. D., & Wang, Q. (2014) · Journal of International Crisis and Risk Communication Research · Cited by 4
Crowd outsourcing for software localization
Gritti, A. (2012) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 3
Information Components as a Basis for Crowdsourced Information System Development
Leone, S. (2011) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-a-006776663 · Cited by 2
Exploiting Crowd Knowledge in the IDE
Ponzanelli, L. (2012) · Cited by 2
Search Interfaces for Integrating Crowdsourced Code Snippets within Development Environments
Wightman, D. P. (2013) · QSpace (Queen's University Library) · Cited by 1
Supporting Users After Software Deployment through Selection-Based Crowdsourced Contextual Help
Chilana, P. K. (2013) · ResearchWorks at the University of Washington (University of Washington) · Cited by 1
Cited by
A Survey of App Store Analysis for Software Engineering
Martin, W., Sarro, F., Jia, Y., Zhang, Y., & Harman, M. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2630689 · Cited by 459
Self-citationThe future of AI-driven software engineering
Terragni, V., Vella, A., Roop, P., & Blincoe, K. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715003 · Cited by 62
A technical survey on statistical modelling and design methods for crowdsourcing quality control
Jin, Y., Carman, M., Zhu, Y., & Xiang, Y. (2020) · Artificial Intelligence · doi:10.1016/j.artint.2020.103351 · Cited by 28
Metadata
- Authors
- Mao, K., Capra, L., Harman, M., & Jia, Y.
- Venue
- Journal of Systems and Software
- Publisher
- Elsevier BV
- Published
- 2017-04
- Volume
- 126
- Pages
- 57-84
- ISSN
- 0164-1212 (ISSN-L)
- OpenAlex
- W2522186013
- Cited by
- 385
- References
- 178
- FWCI
- 130.79
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.