Automatic coding of dialogue acts in collaboration protocols
Open accessAbstract
Although protocol analysis can be an important tool for researchers to investigate the process of collaboration and communication, the use of this method of analysis can be time consuming. Hence, an automatic coding procedure for coding dialogue acts was developed. This procedure helps to determine the communicative function of messages in online discussions by recognizing discourse markers and cue phrases in the utterances. Five main communicative functions are distinguished: argumentative, responsive, informative, elicitative, and imperative. A total of 29 different dialogue acts are specified and recognized automatically in collaboration protocols. The reliability of the automatic coding procedure was determined by comparing automatically coded dialogue acts to hand-coded dialogue acts by a human rater. The validity of the automatic coding procedure was examined using three different types of analyses. First, an examination of group differences was used (dialogue acts used by female versus male students). Ideally, the coding procedure should be able to distinguish between groups who are likely to communicate differently. Second, to examine the validity of the automatic coding procedure through examination of experimental intervention, the results of the automatic coding procedure of students, with access to a tool that visualizes the degree of participation of each student, were compared to students who did not have access to this tool. Finally, the validity of the automatic coding procedure of dialogue acts was examined using correlation analyses. Results of the automatic coding procedure of dialogue acts of utterances (form) were related to results of a manual coding procedure of the collaborative activities to which the utterances refer (content). The analyses presented in this paper indicate promising results concerning the reliability and validity of the automatic coding procedure for dialogue acts. However, limitations of the procedure were also found and discussed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent developments to automate the coding of the collaborative process may extensively decrease the time needed to code a large number of protocols.
“although recent developments to automate the coding of the collaborative process may extensively decrease the time needed to code a large number of protocols (Erkens and Janssen 2008;Rosé et al. 2008).”
From Making the Black Box of Collaborative Learning Transparent: Combining Process-Oriented and Cognitive Load Approaches (2010) · cites this paper as Erkens and Janssen 2008
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,305 · see citing works
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448 · see citing works
International Encyclopedia of the Social Sciences.
Duncan, O. D., & Sills, D. L. (1968) · American Sociological Review · doi:10.2307/2092889 · Cited by 3,486 · see citing works
Schiffrin, D. (1987) · Cambridge University Press eBooks · doi:10.1017/cbo9780511611841 · Cited by 2,816 · see citing works
Kreijns, K., Kirschner, P. A., & Jochems, W. M. G. (2003) · Computers in Human Behavior · doi:10.1016/s0747-5632(02)00057-2 · Cited by 1,633 · see citing works
Fraser, B. (1999) · Journal of Pragmatics · doi:10.1016/s0378-2166(98)00101-5 · Cited by 1,302 · see citing works
Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
Stolcke, A., Ries, K., Coccaro, N., Shriberg, E. E., Bates, R. A., Jurafsky, D. S., Taylor, P., Martin, R., Ess-Dykema, C. V., & Meteer, M. W. (2000) · Computational Linguistics · doi:10.1162/089120100561737 · Cited by 1,112 · see citing works
Time, Interaction, and Performance (TIP)
McGrath, J. E. (1991) · Small Group Research · doi:10.1177/1046496491222001 · Cited by 1,020 · see citing works
Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review
Wever, B. D., Schellens, T., Valcke, M., & Keer, H. V. (2005) · Computers & Education · doi:10.1016/j.compedu.2005.04.005 · Cited by 940 · see citing works
Content analysis of online discussion in an applied educational psychology course
Hara, N., Bonk, C. J., & Angeli, C. (2000) · Instructional Science · doi:10.1023/a:1003764722829 · Cited by 912 · see citing works
Weinberger, A., & Fischer, F. (2005) · Computers & Education · doi:10.1016/j.compedu.2005.04.003 · Cited by 902 · see citing works
Question Asking During Tutoring
Graesser, A. C., & Person, N. K. (1994) · American Educational Research Journal · doi:10.3102/00028312031001104 · Cited by 900 · see citing works
Rethinking media richness: towards a theory of media synchronicity
Dennis, A. R., & Valacich, J. S. (2003) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.1999.772701 · Cited by 820 · see citing works
Methodological Issues in the Content Analysis of Computer Conference Transcripts
Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2000) · AUSpace (Athabasca University) · Cited by 722 · see citing works
Linguistic Inquiry and Word Count (LIWC2007)
Pennebaker, J., Booth, R. J., & Francis, M. T. (2007) · ResearchSpace (University of Auckland) · Cited by 604 · see citing works
Content analysis: What are they talking about?
Strijbos, J.-W., Martens, R. L., Prins, F. J., & Jochems, W. M. G. (2005) · Computers & Education · doi:10.1016/j.compedu.2005.04.002 · Cited by 545 · see citing works
Collaborative learning tasks and the elaboration of conceptual knowledge
van Boxtel, C., van der Linden, J. L., & Kanselaar, G. (2000) · Learning and Instruction · doi:10.1016/s0959-4752(00)00002-5 · Cited by 468 · see citing works
Validity in quantitative content analysis
Rourke, L., & Anderson, T. D. (2004) · Educational Technology Research and Development · doi:10.1007/bf02504769 · Cited by 452 · see citing works
Communication and learning in small groups
Barnes, D., & Todd, F. (1977) · Saber7 UCAB · Cited by 443 · see citing works
Kramsch, C., Coulthard, M., & Montgomery, M. (1983) · Modern Language Journal · doi:10.2307/327114 · Cited by 394 · see citing works
Janssen, J., Erkens, G., Kanselaar, G., & Jaspers, J. (2007) · Computers & Education · doi:10.1016/j.compedu.2006.01.004 · Cited by 352 · see citing works
Self-citationRosé, C., Wang, Y.-C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-007-9034-0 · Cited by 351 · see citing works
Leaper, C., & Smith, T. E. (2004) · Developmental Psychology · doi:10.1037/0012-1649.40.6.993 · Cited by 255 · see citing works
Getting computers to talk like you and me
Reichman, R. (1985) · Cited by 219 · see citing works
International perspectives on the design of technology-supported learning environments
MacKnight, C. B. (1996) · Journal of Computing in Higher Education · doi:10.1007/bf02942400 · Cited by 197 · see citing works
Evaluation of computerized text analysis in an Internet breast cancer support group
Alpers, G. W., Winzelberg, A. J., Classen, C. C., Roberts, H., Dev, P., Koopman, C., & Taylor, C. B. (2004) · Computers in Human Behavior · doi:10.1016/j.chb.2004.02.008 · Cited by 182 · see citing works
Coordination processes in computer supported collaborative writing
Erkens, G., Jaspers, J., Prangsma, M. E., & Kanselaar, G. (2004) · Computers in Human Behavior · doi:10.1016/j.chb.2004.10.038 · Cited by 163 · see citing works
Self-citationInternational Perspectives on the Design of Technology-supported Learning Environments
(2012) · Routledge eBooks · doi:10.4324/9780203053386 · Cited by 160 · see citing works
Regulative support for collaborative scientific inquiry learning
Manlove, S., Lazonder, A. W., & de Jong, T. (2006) · Journal of Computer Assisted Learning · doi:10.1111/j.1365-2729.2006.00162.x · Cited by 142 · see citing works
Creß, U. (2008) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-007-9032-2 · Cited by 136 · see citing works
Effects of representational guidance on domain specific reasoning in CSCL
van Drie, J. P., van Boxtel, C., Jaspers, J., & Kanselaar, G. (2004) · Computers in Human Behavior · doi:10.1016/j.chb.2004.10.024 · Cited by 130 · see citing works
Wever, B. D., Keer, H. V., Schellens, T., & Valcke, M. (2007) · Learning and Instruction · doi:10.1016/j.learninstruc.2007.04.001 · Cited by 129 · see citing works
Louwerse, M. M., & Mitchell, H. H. (2003) · Discourse Processes · doi:10.1207/s15326950dp3503_1 · Cited by 112 · see citing works
Miller, L. A. (1980) · Computational Linguistics · Cited by 108 · see citing works
Jeong, A. C. (2005) · Distance Education · doi:10.1080/01587910500291470 · Cited by 100 · see citing works
The Process of Question Answering -- A Computer Simulation of Cognition
Colclough, M., & Lehnert, W. G. (1979) · The Journal of the Operational Research Society · doi:10.2307/3009325 · Cited by 98 · see citing works
Visualization of agreement and discussion processes during computer-supported collaborative learning
Janssen, J. J. H. M., Erkens, G., & Kanselaar, G. (2006) · Computers in Human Behavior · doi:10.1016/j.chb.2006.10.005 · Cited by 98 · see citing works
Self-citationSupporting CSCL with automatic corpus analysis technology
Dönmez, P., Rosé, C. P., Stegmann, K., Weinberger, A., & Fischer, F. (2005) · Computer-supported collaborative learning/The Computer-Supported Collaborative Learning Conference · doi:10.3115/1149293.1149310 · Cited by 92 · see citing works
Computational Modeling and Analysis of Knowledge Sharing in Collaborative Distance Learning
Soller, A. (2004) · User Modeling and User-Adapted Interaction · doi:10.1023/b:user.0000043436.49168.3b · Cited by 84 · see citing works
Using Dialogue Features to Predict Trouble During Collaborative Learning
Goodman, B. A., Linton, F. N., Gaimari, R. D., Hitzeman, J. M., Ross, H. J., & Zarrella, G. (2005) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-004-5269-x · Cited by 55 · see citing works
Acquiring the meaning of discourse markers
Hutchinson, B. (2004) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1218955.1219042 · Cited by 34 · see citing works
Identifying Discourse Markers in Spoken DialogPreprint
Heeman, P. A., Byron, D. K., & Allen, J. F. (1998) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9801002 · Cited by 32 · see citing works
The Process of Question Answering — A Computer Simulation of Cognition
Colclough, M. (1979) · Journal of the Operational Research Society · doi:10.1057/jors.1979.185 · Cited by 31 · see citing works
Automatically Selecting Useful Phrases for Dialogue Act TaggingPreprint
Samuel, K., Carberry, S., & Vijay‐Shanker, K. (1999) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/9906016 · Cited by 27 · see citing works
Small-group Interaction and Gender
Ridgeway, C. L. (2001) · Elsevier eBooks · doi:10.1016/b0-08-043076-7/03999-1 · Cited by 23 · see citing works
Towards Automatic Identification of Discourse Markers in Dialogs: the Case of 'Like'
Zufferey, S., & Popescu-Belis, A. (n.d.) · Archive ouverte UNIGE (University of Geneva) · doi:10.7892/boris.78686 · Cited by 18 · see citing works
Visualizing participation to facilitate argumentation
Janssen, J. J. H. M., Erkens, G., Jaspers, J., & Kanselaar, G. (2006) · Utrecht University Repository (Utrecht University) · Cited by 16 · see citing works
Self-citationLaw, N. W. Y., Yuen, J., Huang, R., et al. (2007) · Computer-supported collaborative learning/The Computer-Supported Collaborative Learning Conference · doi:10.3115/1599600.1599678 · Cited by 12 · see citing works
Interactivity in Cooperative Problem Solving with Computers
Kanselaar, G., & Erkens, G. (1996) · Utrecht University Repository (Utrecht University) · Cited by 11 · see citing works
Self-citationvan der Meij, H. (2007) · Sex Roles · doi:10.1007/s11199-007-9270-9 · Cited by 10 · see citing works
A cooperative system for collaborative problem solving
Kanselaar, G., & Erkens, G. (1995) · doi:10.3115/222020.222169 · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Chinn, C. A. (2006). Assessing the quality of collaborative argumentation. In S. A. Barab, K. E. Hay, & D. T. Hickey (Eds.), Proceedings of the 7th International Conference of the Learning Sciences (ICLS) (Vol. 2, pp. 1063-1064).
Cited by
Sorted by citations
Noroozi, O., Weinberger, A., Biemans, H. J. A., Mulder, M., & Chizari, M. (2012) · Educational Research Review · doi:10.1016/j.edurev.2011.11.006 · Cited by 310 · see citing works
Janssen, J., Kirschner, F., Erkens, G., Kirschner, P. A., & Paas, F. (2010) · Educational Psychology Review · doi:10.1007/s10648-010-9131-x · Cited by 180 · see citing works
Self-citationTowards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs
Adamson, D., Dyke, G., Jang, H., & Rosé, C. P. (2014) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-013-0012-6 · Cited by 156 · see citing works
Noroozi, O., Hatami, J., Bayat, A., van Ginkel, S., Biemans, H. J. A., & Mulder, M. (2020) · Interactive Learning Environments · doi:10.1080/10494820.2018.1543200 · Cited by 107 · see citing works
The intersection of epistemic beliefs and gender in argumentation performance
Banihashem, S. K., Noroozi, O., Biemans, H. J. A., & Tassone, V. C. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2198995 · Cited by 29 · see citing works
- Authors
- Erkens, G., & Janssen, J.
- Affiliations
- Utrecht University (NL)
- Corresponding author
- Erkens, Gijsbert
- ORCID
- Janssen, Jeroen
- Venue
- International Journal of Computer-Supported Collaborative Learning
- Published
- 2008-09-20
- Volume
- 3
- Issue
- 4
- Pages
- 447-470
- ISSN
- 1556-1607 (ISSN-L)
- 1556-1615
- OpenAlex
- W2082484572
- Cited by
- 103
- References
- 51
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 3 · 2023: 4 · 2022: 4 · 2021: 5 · 2020: 3 · 2019: 8 · 2018: 6 · 2017: 5 · 2016: 12 · 2015: 8 · 2014: 6 · 2013: 6 · 2012: 5
- FWCI
- 12.69
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Journal of Computer-Supported Collaborative Learning (publishedVersion, cc-by-nc)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- dialogue acts, automated coding, collaboration protocol, protocol analysis, discourse markers, cue phrases, communicative functions, online discourse, coding reliability, coding validity, gender differences in communication, computer-supported collaborative learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
