The DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
Integrating Machine Learning Data with Symbolic Knowledge from Collaboration Practices of Curators to Improve Conversational Systems
Abstract
This paper describes how machine learning training data and symbolic knowledge from curators of conversational systems can be used together to improve the accuracy of those systems and to enable better curatorial tools. This is done in the context of a real-world practice of curators of conversational systems who often embed taxonomically-structured meta-knowledge into their documentation. The paper provides evidence that the practice is quite common among curators, that is used as part of their collaborative practices, and that the embedded knowledge can be mined by algorithms. Further, this meta-knowledge can be integrated, using neuro-symbolic algorithms, to the machine learning-based conversational system, to improve its run-time accuracy and to enable tools to support curatorial tasks. Those results point towards new ways of designing development tools which explore an integrated use of code and documentation by machines.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“For the input sentence embeddings, we used the Universal Sentence Encoder (USE) [11] which is a general-purpose language model based on a Transformer neural network [48] trained on varied sources of data, such as Wikipedia, web news, web question-answer pages, and discussion forums.”
MethodDeepWalk: online learning of social representations
Perozzi, B., Al-Rfou, R., & Skiena, S. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623732 · Cited by 10,123 · cited 2×
“The nameId embeddings were then computed with the DeepWalk algorithm [39].”
MethodSurvey of Clustering Algorithms
Xu, R., & WunschII, D. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2005.845141 · Cited by 6,197 · cited 1×
“utterances which are lexically or syntactically similar (using algorithms such as [55]), followed by a manual labeling procedure for the nameId.”
MethodSpeech and language processing
(2010) · Day-to-Day Dyslexia in the Classroom · doi:10.4324/9780203461891_chapter_3 · Cited by 3,544
LSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963 · cited 1×
“The algorithm is based on an adaptation of the USE+T method of the previous section, named here LSTM+T, where the USE sentence embeddings are replaced by a Long Shortterm Memory (LSTM) neural network [43] trained from scratch, which in our experiments performed better than USE for this task.”
MethodUniversal Sentence Encoder for English
Cer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2029 · Cited by 1,277 · cited 18×
“However, the new possible intents generated in the second step can be mapped into intent nameId embeddings using the DeepWalk algorithm [39] as in the USE+T algorithm of the previous section.”
BackgroundMethodInfluentialNonparametric statistics for non-statisticians: a step-by-step approach
(2010) · Choice Reviews Online · doi:10.5860/choice.47-3859 · Cited by 1,021
Nonparametric Statistics for Non‐Statisticians: A Step‐by‐Step Approach
Corder, G. W., & Foreman, D. I. (2009) · Wiley eBooks · doi:10.1002/9781118165881 · Cited by 954
Foundations of statistical natural language processing
Weikum, G. (2002) · ACM SIGMOD Record · doi:10.1145/601858.601867 · Cited by 615
A Survey of Zero-Shot Learning: Settings, Methods, and Applications
Wang, W., Zheng, V. W., Yu, H., & Miao, C. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3293318 · Cited by 528 · cited 3×
“We present an algorithm combining ideas from zero-shot learning [51] and graph expansion [1, 31] which not only fnds groups of utterances in the chat-logs corresponding to intents which are missing from the workspace but also labels them following the structure of the taxonomy.”
Method
Research works printed but not linked: 11
- 10.5555/324058.324059 · doi:10.5555/324058.324059
- Tarek R. Besold, Artur d’Avila Garcez, Sebastian Bader, Howard Bowman, Pedro Domingos, Pascal Hitzler, Kai-Uwe Kuehnberger, Luis C. Lamb, Daniel Lowd, Priscila Machado Vieira Lima, Leo de Penning, Gadi Pinkas, Hoifung Poon, and Gerson Zaverucha. 2017. Neural-Symbolic Learning and Reasoning: A Survey and Interpretation. arxiv:1711.03902 [cs.AI] · arXiv
- Harry Bunt. 2009. The DIT++ taxonomy for functional dialogue markup. In Proc. of the AAMAS 2009 Workshop, Towards a Standard Markup Language for Embodied Dialogue Acts, Dirk Heylen, Catherine Pelachaud, Roberta Catizone, and David Traum (Eds.). Hungarian John von Neumann Society, Budapest, Hungary, 13–24.
- Harry Bunt and Yann Girad. 2005. Designing an Open, Multidimensional Dialogue Act Taxonomy. In Proc. of the Ninth International Workshop on the Semantics and Pragmatics of Dialogue (DIALOR’05). LORIA-INRIA, Nancy, France, 13–24.
- Ole-Johan Dahl, Edsger Wybe Dijkstra, and Charles Antony Richard Hoare. 1972. Structured Programming. Academic Press Ltd., GBR.
- Lisa Ehrlinger and Wolfram Wöß. 2016. Towards a Definition of Knowledge Graphs. In Proc. of SEMANTICS’16 Posters and Demos Track. CEUR-WS, 1–4. http://ceur-ws.org
- Drew Hudson and Christopher D Manning. 2019. Learning by Abstraction: The Neural State Machine. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc., Redhook, New York, USA, 5903–5916.
- 10.3233/978-1-61499-649-1-87 · DOI · Paper page
- Emilio Parisotto Abdel-rahman Mohamed Rishabh Singh Lihong Li Dengyong Zhou and Pushmeet Kohli. 2016. Neuro-Symbolic Program Synthesis. arxiv:1611.01855 [cs.LG] · arXiv
- Katrin Weller. 2007. Folksonomies and Ontologies: Two New Players in Indexing and Knowledge Representation. In Proc. of the 2007 Online Information Conference. Incisive Media, London, UK, 108–115.
- Ziwei Zhang Peng Cui and Wenwu Zhu. 2018. Deep Learning on Graphs: A Survey. arxiv:1812.04202 [cs.CL] · arXiv
Cited by
A Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86
“C l a r i t y o f t h e M L b u i l d i n g p r o c e s s c a n b e e l u c i d a t e d u s i n g m o d e l r e p r o d u c i b i l i t y c h e c k l i s t s [ 35 , 55 ] a n d b y e m b e dd i n g s t r u c t u r e d m e t a - k n o w l e d g e i n t o t h e d o c u m e n t a t i o n [ 56 ] .”
BackgroundResultHeigl, R. M. (2025) · Management Review Quarterly · doi:10.1007/s11301-025-00494-9 · Cited by 118
Making Conversational Robots Learn from Children in a Collaborative Museum Exhibition
Candello, H. C. D. S. P., Pichiliani, M. C., & Pinhanez, C. S. (2021) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3462204.3482888 · Cited by 14
“It portrays, in the universe of a children’s exhibition, the actual collaborative process of chatbot development as it is done by professionals [12].”
BackgroundSelf-citationTeaching Conversational Robots in a Museum Exhibition with Interactive Surfaces
Candello, H. C. D. S. P., Pichiliani, M. C., & Pinhanez, C. S. (2021) · ACM International Conference on Interactive Surfaces and Spaces (ISS) · doi:10.1145/3447932.3490680 · Cited by 10
“It portrays, in the universe of a children’s exhibition, the actual collaborative process of chatbot development as it is done by professionals [12].”
BackgroundSelf-citationUsing Meta-Knowledge Mined from Identifiers to Improve Intent Recognition in Conversational Systems
Pinhanez, C. S., Cavalin, P., Ribeiro, V. H. A., Appel, A. P., Candello, H. C. D. S. P., Nogima, J., Pichiliani, M. C., Guerra, M. A., de Bayser, M. G., Malfatti, G. L., et al. (2021) · doi:10.18653/v1/2021.acl-long.545 · Cited by 7
Self-citation
Metadata
- Authors
- Pinhanez, C. S., Candello, H., Cavalin, P., Pichiliani, M. C., Appel, A. P., Alves Ribeiro, V. H., Nogima, J., de Bayser, M., Guerra, M., Ferreira, H., & Malfatti, G.
- Affiliations
- IBM Research - Brazil (BR)
- IBM Research Brazil, Brazil
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-13
- OpenAlex
- W3162148497
- Cited by
- 7
- References
- 53
- Citations by year
- 2025: 2 · 2023: 2 · 2021: 3
- FWCI
- 1.04
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- dialogue systems, curators, neuro-symbolic learning, symbolic knowledge, machine learning training data, meta-knowledge, collaborative practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Integrating Machine Learning Data with Symbolic Knowledge from Collaboration Practices of Curators to Improve Conversational Systems. https://app.cochat.ai/papers/10.1145/3411764.3445368
- The paper
- Pinhanez, C. S., Candello, H., Cavalin, P., Pichiliani, M. C., Appel, A. P., Alves Ribeiro, V. H., Nogima, J., de Bayser, M., Guerra, M., Ferreira, H., & Malfatti, G. (2021). Integrating Machine Learning Data with Symbolic Knowledge from Collaboration Practices of Curators to Improve Conversational Systems. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445368
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.