Evaluating the Effects of Architectural Documentation: A Case Study of a Large Scale Open Source Project
Abstract
Sustaining large open source development efforts requires recruiting new participants; however, a lack of architectural documentation might inhibit new participants since large amounts of project knowledge are unavailable to newcomers. We present the results of a multitrait, multimethod analysis of the effects of introducing architectural documentation into a substantial open source project-the Hadoop Distributed File System (HDFS). HDFS had only minimal architectural documentation, and we wanted to discover whether the putative benefits of architectural documentation could be observed over time. To do this, we created and publicized an architecture document and then monitored its usage and effects on the project. The results were somewhat ambiguous: by some measures the architecture documentation appeared to effect the project but not by others. Perhaps of equal importance is our discovery that the project maintained, in its Web-accessible JIRA archive of software issues and fixes, enough architectural discussion to support architectural thinking and reasoning. This “emergent” architecture documentation served an important purpose in recording core project members' architectural concerns and resolutions. However, this emergent architecture documentation did not serve all project members equally well; it appears that those on the periphery of the project-newcomers and adopters-still require explicit architecture documentation, as we will show.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
(1984) · Cited by 81,983 · see citing works
Case Study Research: Design and Methods
Chaiklin, H. (1991) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-199102000-00025 · Cited by 39,631 · see citing works
Situated Learning: Legitimate Peripheral Participation.
Bloch, M. E. F., Lave, J., & Wenger, É. (1994) · Man · doi:10.2307/2804509 · Cited by 39,323 · see citing works
Lave, J., & Wenger, E. (1991) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815355 · Cited by 33,219 · see citing works
Designing and Conducting Mixed Methods Research
Robinson, P. M. (2007) · Australian and New Zealand Journal of Public Health · doi:10.1111/j.1753-6405.2007.00096.x · Cited by 23,297 · see citing works
The Structure of Scientific Revolutions: 50th Anniversary Edition
Kuhn, T. S., & Hacking, I. (2012) · doi:10.7208/chicago/9780226458144.001.0001 · Cited by 21,170 · see citing works
(1963) · Medical Entomology and Zoology · Cited by 15,085 · see citing works
A set of measures of centrality based upon betweenness
Freeman, L. (1977) · Sociometry · doi:10.2307/3033543 · Cited by 10,557 · see citing works
Software Architecture in Practice
Bass, L., Clements, P., & Kazman, R. (1997) · Cited by 5,033 · see citing works
Self-citationThe Hadoop Distributed File System
Shvachko, K. V., Kuang, H., Radia, S., & Chansler, R. J. (2010) · Digest of papers - IEEE Symposium on Mass Storage Systems · doi:10.1109/msst.2010.5496972 · Cited by 4,911 · see citing works
Automatic Time Series Forecasting: The forecast Package for R
Hyndman, R. J., & Khandakar, Y. (2008) · Journal of Statistical Software · doi:10.18637/jss.v027.i03 · Cited by 3,548 · see citing works
Design Rules: The Power of Modularity
Baldwin, C. Y., & Clark, K. B. (2000) · doi:10.7551/mitpress/2366.001.0001 · Cited by 3,504 · see citing works
Ghemawat, S., Gobioff, H., & Leung, S.-T. A. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945450 · Cited by 2,665 · see citing works
The 4+1 View Model of architecture
Kruchten, P. B. (1995) · IEEE Software · doi:10.1109/52.469759 · Cited by 2,165 · see citing works
Forecasting seasonals and trends by exponentially weighted moving averages
Holt, C. C. (2004) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2003.09.015 · Cited by 1,750 · see citing works
Two case studies of open source software development: Apache and Mozilla
Mockus, A., Fielding, R. T., & Herbsleb, J. D. (2002) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/567793.567795 · Cited by 1,685 · see citing works
Writing science: Literacy and discursive power
Atkinson, D. (1994) · English for Specific Purposes · doi:10.1016/0889-4906(94)90008-6 · Cited by 1,597 · see citing works
Documenting software architectures: views and beyond
Clements, P., Garlan, D., Little, R., Nord, R. L., & Stafford, J. A. (2003) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2003.1201264 · Cited by 1,434 · see citing works
Mapping the Dynamics of Science and Technology
Callon, M., Law, J., & Rip, A. (1986) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-07408-2 · Cited by 1,346 · see citing works
Evaluation of unsupervised semantic mapping of natural language with Leximancer concept mapping
Smith, A. E., & Humphreys, M. S. (2006) · Behavior Research Methods · doi:10.3758/bf03192778 · Cited by 1,206 · see citing works
The Conduct of Inquiry: Methodology for Behavioral Science.
Nagel, E., & Kaplan, A. (1965) · American Sociological Review · doi:10.2307/2091353 · Cited by 1,155 · see citing works
Rethinking centrality: Methods and examples
Stephenson, K., & Zelen, M. (1989) · Social Networks · doi:10.1016/0378-8733(89)90016-6 · Cited by 965 · see citing works
Case Study Research in Software Engineering
Runeson, P., Höst, M., Rainer, A., & Regnell, B. (2012) · Wiley eBooks · doi:10.1002/9781118181034 · Cited by 913 · see citing works
Design Rules, Volume 1: The Power of Modularity
Hatch, N. W., Baldwin, C. Y., & Clark, K. B. (2001) · Academy of Management Review · doi:10.2307/259400 · Cited by 839 · see citing works
Case Study Research in Software Engineering: Guidelines and Examples
Runeson, P., Höst, M., Rainer, A., & Regnell, B. (2012) · University of Hertfordshire Research Archive (University of Hertfordshire) · Cited by 827 · see citing works
MacCormack, A., Rusnak, J., & Baldwin, C. Y. (2006) · Management Science · doi:10.1287/mnsc.1060.0552 · Cited by 725 · see citing works
The Conduct of Inquiry: Methodology for Behavioral Science.
Rucker, D., & Kaplan, A. (1965) · Philosophy and Phenomenological Research · doi:10.2307/2105486 · Cited by 557 · see citing works
The social structure of free and open source software development
Crowston, K., & Howison, J. (2005) · First Monday · doi:10.5210/fm.v10i2.1207 · Cited by 474 · see citing works
Free/Libre open-source software development
Crowston, K., Wei, K., Howison, J., & Wiggins, A. (2008) · ACM Computing Surveys · doi:10.1145/2089125.2089127 · Cited by 457 · see citing works
Understanding the requirements for developing open source software systems
Scacchi, W. (2002) · IEE Proceedings - Software · doi:10.1049/ip-sen:20020202 · Cited by 405 · see citing works
The Structure of Scientific Revolutions: 50th Anniversary Edition
Kühn, T., & Hacking, I. (2012) · Cited by 342 · see citing works
Semistructured messages are surprisingly useful for computer-supported coordination
Malone, T. W., Grant, K. R., Lai, K., Rao, R., & Rosenblitt, D. (1987) · ACM Transactions on Information Systems · doi:10.1145/27636.27637 · Cited by 320 · see citing works
Information systems success in free and open source software development: theory and measures
Crowston, K., Howison, J., & Annabi, H. (2006) · Software Process Improvement and Practice · doi:10.1002/spip.259 · Cited by 301 · see citing works
What Industry Needs from Architectural Languages: A Survey
Malavolta, I., Lago, P., Muccini, H., Pelliccione, P., & Tang, A. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.74 · Cited by 295 · see citing works
Programming as theory building
Naur, P. (1985) · Microprocessing and Microprogramming · doi:10.1016/0165-6074(85)90032-8 · Cited by 252 · see citing works
Playing Detective: Reconstructing Software Architecture from Available Evidence
Kazman, R., & Carrière, S. J. (1999) · Automated Software Engineering · doi:10.1023/a:1008781513258 · Cited by 226 · see citing works
Self-citationDeterminants of open source software project success: A longitudinal study
Subramaniam, C., Sen, R., & Nelson, M. L. (2008) · Decision Support Systems · doi:10.1016/j.dss.2008.10.005 · Cited by 225 · see citing works
Measuring open source software success
Lee, S. T., Kim, H., & Gupta, S. (2007) · Omega · doi:10.1016/j.omega.2007.05.005 · Cited by 183 · see citing works
Historical scientometrics? Mapping over 70 years of biological safety research with coword analysis
Cambrosio, A., Limoges, C., Courtial, J., & Laville, F. (1993) · Scientometrics · doi:10.1007/bf02016546 · Cited by 161 · see citing works
Communication in open source software development mailing lists
Guzzi, A., Bacchelli, A., Lanza, M., Pinzger, M., & Arie Van Deursen, A. (2013) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624039 · Cited by 140 · see citing works
The metropolis model a new logic for development of crowdsourced systems
Kazman, R., & Chen, H. (2009) · Communications of the ACM · doi:10.1145/1538788.1538808 · Cited by 133 · see citing works
Self-citationSoftware Architecture Knowledge Management
Babar, M. A., Dingsøyr, T., Lago, P., & van Vliet, H. (2009) · Springer eBooks · doi:10.1007/978-3-642-02374-3 · Cited by 129 · see citing works
Detecting software modularity violations
Wong, S., Cai, Y., Kim, M., et al. (2011) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/1985793.1985850 · Cited by 126 · see citing works
Software architecture knowledge management theory and practice
Babar, M. A. (2009) · Cited by 120 · see citing works
Categorizing bugs with social networks: A case study on four open source software communities
Zanetti, M. S., Scholtes, I., Tessone, C. J., & Schweitzer, F. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606653 · Cited by 116 · see citing works
Sack, W., Détienne, F., Ducheneaut, N., Burkhardt, J.-M., Mahendran, D., & Barcellini, F. (2006) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-006-9020-5 · Cited by 111 · see citing works
Knowledge-based approaches in software documentation: A systematic literature review
Ding, W., Liang, P., Tang, A., & van Vliet, H. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.01.008 · Cited by 107 · see citing works
Social Network Structures in Open Source Software Development Teams
Long, Y., & Keng, S.-A. (2007) · Journal of Database Management · doi:10.4018/jdm.2007040102 · Cited by 106 · see citing works
Design rule spaces: a new form of architecture insight
Xiao, L., Cai, Y., & Kazman, R. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2568225.2568241 · Cited by 99 · see citing works
Self-citationMining Task-Based Social Networks to Explore Collaboration in Software Teams
Wolf, T., Schröter, A., Damian, D., et al. (2008) · IEEE Software · doi:10.1109/ms.2009.16 · Cited by 96 · see citing works
OpenBQR: a framework for the assessment of OSS
Taibi, D., Lavazza, L., & Morasca, S. (2007) · IFIP International Federation for Information Processing/IFIP · doi:10.1007/978-0-387-72486-7_14 · Cited by 78 · see citing works
Explaining leadership in virtual teams: The case of open source software
Giuri, P., Rullani, F., & Torrisi, S. (2008) · Information Economics and Policy · doi:10.1016/j.infoecopol.2008.06.002 · Cited by 78 · see citing works
Method for Qualification and Selection of Open Source Software
Semeteys, R. (2008) · Cited by 76 · see citing works
English, R. C., & Schweik, C. M. (2007) · doi:10.1109/floss.2007.9 · Cited by 71 · see citing works
Understanding Requirements for Open Source Software
Scacchi, W. (2009) · Lecture notes in business information processing · doi:10.1007/978-3-540-92966-6_27 · Cited by 63 · see citing works
Ehrlich, K., & Cataldo, M. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145345 · Cited by 62 · see citing works
The Form of Information in Science: Analysis of an Immunology Sublanguage
Ryckman, T. C., & Gottfried, M. (1988) · Medical Entomology and Zoology · Cited by 61 · see citing works
How Do Open Source Communities Document Software Architecture: An Exploratory Survey
Ding, W., Liang, P., Tang, A., Vliet, H. V., & Shahin, M. (2014) · IEEE International Conference on Engineering of Complex Computer Systems (ICECCS) · doi:10.1109/iceccs.2014.26 · Cited by 56 · see citing works
Software architecture awareness in long-term software product evolution
Unphon, H., & Dittrich, Y. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.06.043 · Cited by 48 · see citing works
How do open source communities blog?
Pagano, D., & Maalej, W. (2012) · Empirical Software Engineering · doi:10.1007/s10664-012-9211-2 · Cited by 47 · see citing works
Transition of governance in a mature open software source community: Evidence from the Debian case
Sadowski, B. M., Sadowski-Rasters, G., & Duysters, G. (2008) · Information Economics and Policy · doi:10.1016/j.infoecopol.2008.05.001 · Cited by 44 · see citing works
Ongoing software development without classical requirements
Alspaugh, T. A., & Scacchi, W. (2013) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2013.6636716 · Cited by 35 · see citing works
Software Architecture Documentation for Developers: A Survey
Rost, D., Naab, M., Lima, C., & Chávez, C. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39031-9_7 · Cited by 33 · see citing works
Microblogging in Open Source Software Development: The Case of Drupal and Twitter
Wang, X., Kuzmickaja, I., Stol, K.-J., et al. (2013) · IEEE Software · doi:10.1109/ms.2013.98 · Cited by 31 · see citing works
The use of empirical methods in Open Source Software research: Facts, trends and future directions
Stol, K.-J., Babar, M. A., Russo, B., & Fitzgerald, B. F. (2009) · ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development (FLOSS) · doi:10.1109/floss.2009.5071355 · Cited by 30 · see citing works
Making Open Source Ready for the Enterprise: The Open Source Maturity Model
(2008) · Cited by 26 · see citing works
Making the Right Decision: Supporting Architects with Design Decision Data
van der Ven, J. S., & Bosch, J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39031-9_15 · Cited by 25 · see citing works
Reclassifying Success and Tragedy in FLOSS Projects
Wiggins, A., & Crowston, K. (2010) · IFIP International Federation for Information Processing/IFIP · doi:10.1007/978-3-642-13244-5_23 · Cited by 21 · see citing works
On the necessity of term dependence in a query space for weighted retrieval
Bollmann-Sdorra, P., & Raghavan, V. V. (1998) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(1998)49:13 · Cited by 20 · see citing works
He, P., Li, B., & Huang, Y. (2012) · International Conference on Cloud and Green Computing · doi:10.1109/cgc.2012.50 · Cited by 17 · see citing works
Big Data and the Collective Turn in Biomedicine. How Should We Analyze Post-genomic Practices?
Cambrosio, A., Bourret, P., Rabeharisoa, V., et al. (2023) · Tecnoscienza · doi:10.6092/issn.2038-3460/17168 · Cited by 14 · see citing works
Zanetti, M. S. (2012) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2012.6227030 · Cited by 12 · see citing works
Developing Architectural Documentation for the Hadoop Distributed File System
Bass, L., Kazman, R., & Özkaya, İ. (2011) · IFIP International Federation for Information Processing/IFIP · doi:10.1007/978-3-642-24418-6_4 · Cited by 10 · see citing works
Self-citationCategorizing Bugs with Social Networks: A Case Study on Four Open Source Software Communities.
Scholtes, I., Zanetti, M. S., Tessone, C. J., et al. (2014) · Cited by 10 · see citing works
Assessing the Open Source Development Processes Using OMM
Petrinja, E., & Succi, G. (2012) · Advances in Software Engineering · doi:10.1155/2012/235392 · Cited by 8 · see citing works
Boundary-Spanning Documents in Online FLOSS Communities: Does One Size Fit All?
Østerlund, C. S., & Crowston, K. (2013) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2013.119 · Cited by 6 · see citing works
Monarch, I., Goldenson, D. R., & Osiecki, L. T. (2008) · doi:10.21236/ada488178 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Design Rule Spaces: A New Model for Representing and Analyzing Software Architecture
Cai, Y., Xiao, L., Kazman, R., Mo, R., & Feng, Q. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2797899 · Cited by 39 · see citing works
Self-citationSoftware Architecture in Practice: Challenges and Opportunities
Wan, Z., Zhang, Y., Xia, X., Jiang, Y., & Lo, D. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616367 · Cited by 30 · see citing works
“The benefits of software architecture documentation have been widely investigated in previous studies, e.g., serving as an educational tool to introduce new team members to a software project [20] and reduce entry barriers for new contributors in OSS development [42].”
MethodThe sustainability of open source commons
Curto-Millet, D., & Corsín Jiménez, A. (2023) · European Journal of Information Systems · doi:10.1080/0960085x.2022.2046516 · Cited by 16 · see citing works
Muszynski, M., Lugtigheid, S., Castor, F., & Brinkkemper, S. (2022) · 2022 IEEE 19th International Conference on Software Architecture (ICSA) · doi:10.1109/icsa53651.2022.00013 · Cited by 10 · see citing works
- Authors
- Kazman, R., Goldenson, D., Monarch, I., Nichols, W., & Valetto, G.
- Affiliations
- Software Engineering Institute (US)
- Fondazione Bruno Kessler (IT)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2016-03-01
- Volume
- 42
- Issue
- 3
- Pages
- 220-260
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2296438858
- Semantic Scholar
- 270b5ad26cbb91f665991008a435aad194f5c51c
- Cited by
- 31
- References
- 78
- Citations by year
- 2025: 3 · 2024: 3 · 2023: 4 · 2022: 3 · 2021: 5 · 2020: 2 · 2019: 3 · 2018: 2 · 2017: 4 · 2016: 2
- FWCI
- 7.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Funders
- Department of Defense (FA8721-05-C-0003)
- Carnegie Mellon University
- Software Engineering Institute
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- architectural documentation, open source software, HDFS, newcomer onboarding, software architecture, large-scale software projects, software maintainability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
