Human Versus Artificial Intelligence: A Data-Driven Approach to Real-Time Process Management During Complex Engineering Design
The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Managing the design process of teams has been shown to considerably improve problem-solving behaviors and resulting final outcomes. Automating this activity presents significant opportunities in delivering interventions that dynamically adapt to the state of a team in order to reap the most impact. In this work, an artificial intelligence (AI) agent is created to manage the design process of engineering teams in real time, tracking features of teams’ actions and communications during a complex design and path-planning task in multidisciplinary teams. Teams are also placed under the guidance of human process managers for comparison. Regarding outcomes, teams perform equally as well under both types of management, with trends toward even superior performance from the AI-managed teams. The managers’ intervention strategies and team perceptions of those strategies are also explored, illuminating some intriguing similarities. Both the AI and human process managers focus largely on communication-based interventions, though differences start to emerge in the distribution of interventions across team roles. Furthermore, team members perceive the interventions from both the AI and human manager as equally relevant and helpful, and believe the AI agent to be just as sensitive to the needs of the team. Thus, the overall results show that the AI manager agent introduced in this work is able to match the capabilities of humans, showing potential in automating the management of a complex design process.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This design methodology is applicable to scenarios such as collaborative manufacturing (determining what production metrics robots share) [34], autonomous vehicle coordination (controlling trajectory data broadcast), and distributed energy management (modulating demand forecasts between grid resources).”
From Adaptive Information Modulation: Designing Governance Mechanisms for Multi-Agent Artificial Intelligence Systems (2026) · cites this paper as [34]
Citations
Cites
An introduction to latent semantic analysis
Landauer, T. K., Foltz, P. W., & Laham, D. (1998) · Discourse Processes · doi:10.1080/01638539809545028 · Cited by 4,880
The Effects of Team Diversity on Team Outcomes: A Meta-Analytic Review of Team Demography
Horwitz, S. K., & Horwitz, I. B. (2007) · Journal of Management · doi:10.1177/0149206307308587 · Cited by 1,513
Team Diversity and Information Use
Dahlin, K. B., Weingart, L. R., & Hinds, P. J. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.19573112 · Cited by 784
Branquinho, L. C., Carrião, M. S., Costa, A. S., Zufelato, N., Sousa, M. H., Miotto, R., Ivkov, R., & Bakuzis, A. F. (2014) · Scientific Reports · doi:10.1038/srep03637 · Cited by 491
Gardner, H. K. (2012) · Administrative Science Quarterly · doi:10.1177/0001839212446454 · Cited by 222
Collaborative Creativity—Group Creativity and Team Innovation
Paulus, P. B., Dzindolet, M., & Kohn, N. W. (2012) · Handbook of Organizational Creativity · doi:10.1016/b978-0-12-374714-3.00014-8 · Cited by 189
A Document Analysis Method for Characterizing Design Team Performance
Dong, A., Hill, A. W., & Agogino, A. M. (2004) · Journal of Mechanical Design · doi:10.1115/1.1711818 · Cited by 187
Relationships between Project Complexity and Communication
Senescu, R. R., Aranda‐Mena, G., & Haymaker, J. R. (2012) · Journal of Management in Engineering · doi:10.1061/(asce)me.1943-5479.0000121 · Cited by 139
The Strength of the Strongest Ties in Collaborative Problem Solving
de Montjoye, Y.-A., Stopczynski, A., Shmueli, E., Pentland, A., & Lehmann, S. (2014) · Scientific Reports · doi:10.1038/srep05277 · Cited by 128
A cautionary tale about the impact of AI on human design teams
Zhang, G., Raina, A., Cagan, J., & McComb, C. (2021) · Design Studies · doi:10.1016/j.destud.2021.100990 · Cited by 117
Self-citationEnabling Effective Engineering Teams: A Program for Teaching Interaction Skills*
Seat, E., & Lord, S. M. (1999) · Journal of Engineering Education · doi:10.1002/j.2168-9830.1999.tb00463.x · Cited by 108
Williams, G., Meisel, N. A., Simpson, T. W., & McComb, C. (2019) · Journal of Mechanical Design · doi:10.1115/1.4044199 · Cited by 93
Self-citationDesign Team Communication and Design Task Complexity: The Preference for Dialogues
den Otter, A., & Emmitt, S. (2008) · Architectural Engineering and Design Management · doi:10.3763/aedm.2008.0072 · Cited by 90
Tribelsky, E., & Sacks, R. (2011) · Architectural Engineering and Design Management · doi:10.1080/17452007.2011.582332 · Cited by 89
Machine Learning-Based Design Concept Evaluation
Camburn, B., He, Y., Raviselvam, S., Luo, J., & Wood, K. (2020) · Journal of Mechanical Design · doi:10.1115/1.4045126 · Cited by 75
McComb, C., Cagan, J., & Kotovsky, K. (2017) · Journal of Mechanical Design · doi:10.1115/1.4035793 · Cited by 69
Self-citationArtificial intelligence and expert systems applications in new product development—a survey
Rao, S. S., Nahm, A. Y., Shi, Z., Deng, X., & Syamil, A. (1999) · Journal of Intelligent Manufacturing · doi:10.1023/a:1008943723141 · Cited by 65
Goucher-Lambert, K., Gyory, J. T., Kotovsky, K., & Cagan, J. (2020) · Journal of Mechanical Design · doi:10.1115/1.4046077 · Cited by 54
Self-citationTraining to Reduce the Disruptive Effects of Interruptions
Hess, S. M., & Detweiler, M. C. (1994) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129403801806 · Cited by 51
Mitigating disruptive effects of interruptions through training: What needs to be practiced?
Cades, D. M., Boehm-Davis, D. A., Trafton, J. G., & Monk, C. A. (2011) · Journal of Experimental Psychology: Applied · doi:10.1037/a0023497 · Cited by 50
Enabling effective engineering teams: a program for teaching interaction skills
Seat, E., & Lord, S. M. (2002) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.1998.736842 · Cited by 45
Handbook of Organizational Creativity
Paulus, P. B., Dzindolet, M. T., & Kohn, N. W. (2012) · Cited by 40
Computer-aided mind map generation via crowdsourcing and machine learning
Camburn, B., Arlitt, R. M., Anderson, D. S., Sanaei, R., Raviselvam, S., Jensen, D., & Wood, K. L. (2020) · Research in Engineering Design · doi:10.1007/s00163-020-00341-w · Cited by 39
Song, B., Zurita, N. F. S., Zhang, G., Stump, G. M., Balon, C., Miller, S. W., Yukish, M. A., Cagan, J., & McComb, C. (2020) · Proceedings of the Design Society DESIGN Conference · doi:10.1017/dsd.2020.68 · Cited by 39
Self-citationTransferring Design Strategies From Human to Computer and Across Design Problems
Raina, A., Cagan, J., & McComb, C. (2019) · Journal of Mechanical Design · doi:10.1115/1.4044258 · Cited by 37
Self-citationGyory, J. T., Cagan, J., & Kotovsky, K. (2018) · Research in Engineering Design · doi:10.1007/s00163-018-00303-3 · Cited by 31
Self-citationModelling and Management of Engineering Processes
Robin, V., Merlo, C., Pol, G., & Girard, P. (2010) · Springer eBooks · doi:10.1007/978-1-84996-199-8 · Cited by 27
Design implications for Designing with a Collaborative AI
Koch, J. (2017) · National Conference on Artificial Intelligence · Cited by 24
Sio, U. N., Kotovsky, K., & Cagan, J. (2016) · Memory & Cognition · doi:10.3758/s13421-016-0684-x · Cited by 21
Self-citationDering, M. L., Tucker, C. S., & Kumara, S. R. T. (2017) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4037434 · Cited by 21
The Virtual Design Team: Modeling organizational behavior of concurrent design teams
Jin, Y., Levitt, R. E., Christiansen, T. R., & Kunz, J. (1995) · Artificial intelligence for engineering design analysis and manufacturing · doi:10.1017/s0890060400002183 · Cited by 19
Analyzing the Characteristics of Cognitive-Assistant-Facilitated Ideation Groups
Maier, T., Zurita, N. F. S., Starkey, E. M., Spillane, D. R., Menold, J., & McComb, C. (2020) · ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (IDETC/CIE) · doi:10.1115/detc2020-22555 · Cited by 16
Self-citationA Survey: Towards Understanding Emergent Behavior in Complex Engineered Systems
Zurita, N. F. S., & Tumer, I. Y. (2017) · ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (IDETC/CIE) · doi:10.1115/detc2017-67453 · Cited by 14
Data on the design and operation of drones by both individuals and teams
Zhang, G., Zurita, N. F. S., Stump, G. M., Song, B., Cagan, J., & McComb, C. (2021) · Data in Brief · doi:10.1016/j.dib.2021.107008 · Cited by 12
Self-citationA Query Conundrum: The Mental Challenges of Using a Cognitive Assistant
Maier, T., Abdullah, S., McComb, C., & Menold, J. (2021) · SN Computer Science · doi:10.1007/s42979-021-00621-9 · Cited by 9
Self-citationSpecifications for a social and technical environment for improving design process communication
Dikbas, A., Ergen, E., & Giritli, H. (2009) · CRC Press eBooks · doi:10.1201/9781482266665-36 · Cited by 8
Karau, S. J. (2020) · doi:10.1016/c2016-0-01826-0 · Cited by 8
How Do Interruptions During Designing Affect Design Cognition?
Gero, J. S., Jiang, H., Dobolyi, K., Bellows, B. G., & Smythwood, M. (2015) · Springer eBooks · doi:10.1007/978-3-319-14956-1_7 · Cited by 7
Gyory, J. T., Kotovsky, K., & Cagan, J. (2021) · Journal of Mechanical Design · doi:10.1115/1.4050748 · Cited by 6
Self-citationMcComb, C., Kotovsky, K., & Cagan, J. (2018) · doi:10.31224/osf.io/d4peq · Cited by 6
Self-citationModeling Effects of Organizational Structure and Communication Tools on Design Team Productivity
Kunz, J., Cohen, G. P., & Levitt, R. E. (1993) · Cited by 3
Anachronistic intervention: Performative collaborative design in the wild
Lock, S. (2013) · CoDesign · doi:10.1080/15710882.2012.755550 · Cited by 2
Research works printed but not linked: 1
- Cohen, G. P. , 1993, “The Virtual Design Team: An Information-Processing Model of Design Team Management,” Doctoral dissertation, Stanford University.
Cited by
Who is better in project planning?Generative artificial intelligence or project managers?
Barcauí, A., & Monat, A. (2023) · Project Leadership and Society · doi:10.1016/j.plas.2023.100101 · Cited by 245
Zhang, G., Chong, L., Kotovsky, K., & Cagan, J. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107536 · Cited by 147
Self-citationBartlett, K. A., & Camba, J. D. (2024) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2024.02.006 · Cited by 107
“…collaboration has been investigated in various stages of the design process, including early ideation and concept evaluation [14], later-stage ideation [15], management of the design team [16], aiding teams in design problem-solving [17], and aiding teams in the design of complex systems [18].”
BackgroundBiologically Inspired Design Concept Generation Using Generative Pre-Trained Transformers
Zhu, Q., Zhang, X., & Luo, J. (2023) · Journal of Mechanical Design · doi:10.1115/1.4056598 · Cited by 83
Opportunities for large language models and discourse in engineering design
Göpfert, J., Weinand, J. M., Kuckertz, P., & Stolten, D. (2024) · Energy and AI · doi:10.1016/j.egyai.2024.100383 · Cited by 53
“Gyory et al. [33, 34] analyze real time data of design teams to suggest measures from a predefined list if the communication or action frequency appears to be too low.”
Background
Metadata
- Authors
- Gyory, J. T., Soria Zurita, N. F., Martin, J., Balon, C., McComb, C., Kotovsky, K., & Cagan, J.
- Affiliations
- Carnegie Mellon University (US)
- Pennsylvania State University School of Engineering Design, Technology, and Professional Programs, , University Park, State College, PA 16802 ;
- Applied Research Laboratory at Penn State (US)
- Venue
- Journal of Mechanical Design
- Publisher
- ASME International
- Published
- 2021-10-14
- Volume
- 144
- Issue
- 2
- ISSN
- 0161-8458 (ISSN-L)
- 1050-0472
- 1528-9001
- OpenAlex
- W3200185945
- Cited by
- 56
- References
- 42
- Citations by year
- 2026: 17 · 2025: 13 · 2024: 9 · 2023: 10 · 2022: 7
- FWCI
- 6.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Mechanical Design (publishedVersion)
- Funders
- Air Force Office of Scientific Research (FA9550-18-1-0088)
- Defense Advanced Research Projects Agency (N66001-17-1-4064)
- Topics
- Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
- Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- AI agents, design process management, engineering teams, multidisciplinary team, team communication, adaptive interventions, team performance, team member perceptions, team roles
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.