Integrating Machine Learning Data with Symbolic Knowledge from Collaboration Practices of Curators to Improve Conversational Systems
The DOI matches its Crossref record; 52 of the 53 references it cites were found in a registry or an index.
Abstract
This paper describes how machine learning training data and symbolic knowledge from curators of conversational systems can be used together to improve the accuracy of those systems and to enable better curatorial tools. This is done in the context of a real-world practice of curators of conversational systems who often embed taxonomically-structured meta-knowledge into their documentation. The paper provides evidence that the practice is quite common among curators, that is used as part of their collaborative practices, and that the embedded knowledge can be mined by algorithms. Further, this meta-knowledge can be integrated, using neuro-symbolic algorithms, to the machine learning-based conversational system, to improve its run-time accuracy and to enable tools to support curatorial tasks. Those results point towards new ways of designing development tools which explore an integrated use of code and documentation by machines.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“For the input sentence embeddings, we used the Universal Sentence Encoder (USE) [11] which is a general-purpose language model based on a Transformer neural network [48] trained on varied sources of data, such as Wikipedia, web news, web question-answer pages, and discussion forums.”
MethodDeepWalk: online learning of social representations
Perozzi, B., Al-Rfou, R., & Skiena, S. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623732 · Cited by 10,123 · cited 2×
“The nameId embeddings were then computed with the DeepWalk algorithm [39].”
MethodSurvey of Clustering Algorithms
Xu, R., & WunschII, D. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2005.845141 · Cited by 6,197 · cited 1×
“utterances which are lexically or syntactically similar (using algorithms such as [55]), followed by a manual labeling procedure for the nameId.”
MethodSpeech and language processing
(2010) · Day-to-Day Dyslexia in the Classroom · doi:10.4324/9780203461891_chapter_3 · Cited by 3,544
LSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963 · cited 1×
“The algorithm is based on an adaptation of the USE+T method of the previous section, named here LSTM+T, where the USE sentence embeddings are replaced by a Long Shortterm Memory (LSTM) neural network [43] trained from scratch, which in our experiments performed better than USE for this task.”
MethodUniversal Sentence Encoder for English
Cer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2029 · Cited by 1,750 · cited 18×
“However, the new possible intents generated in the second step can be mapped into intent nameId embeddings using the DeepWalk algorithm [39] as in the USE+T algorithm of the previous section.”
BackgroundMethodInfluentialNonparametric statistics for non-statisticians: a step-by-step approach
(2010) · Choice Reviews Online · doi:10.5860/choice.47-3859 · Cited by 1,021
Nonparametric Statistics for Non‐Statisticians: A Step‐by‐Step Approach
Corder, G. W., & Foreman, D. I. (2009) · Wiley eBooks · doi:10.1002/9781118165881 · Cited by 954
Foundations of statistical natural language processing
Weikum, G. (2002) · ACM SIGMOD Record · doi:10.1145/601858.601867 · Cited by 615
A Survey of Zero-Shot Learning: Settings, Methods, and Applications
Wang, W., Zheng, V. W., Yu, H., & Miao, C. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3293318 · Cited by 528 · cited 3×
“We present an algorithm combining ideas from zero-shot learning [51] and graph expansion [1, 31] which not only fnds groups of utterances in the chat-logs corresponding to intents which are missing from the workspace but also labels them following the structure of the taxonomy.”
MethodMao, J., Gan, C., Kohli, P., Tenenbaum, J. B., & Wu, J. (2019) · arXiv · doi:10.48550/arxiv.1904.12584 · Cited by 444 · cited 2×
“The approach belongs to a class of recent algorithms, often referred to as neuro-symbolic algorithms [6], which aim to address typical limitations of modern neural networks and traditional symbolic reasoning by integrating methods of those two approaches [4, 16, 17, 33, 35, 37].”
BackgroundExploration and Explanation in Computational Notebooks
Rule, A., Tabard, A., & Hollan, J. D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173606 · Cited by 335 · cited 1×
“Some recent work [28, 36, 41] has looked into the practices of data science workers but similarly we found no case of mining of comments or markdowns for machine use.”
BackgroundHow Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation
Muller, M., Lange, I., Wang, D., Piorkowski, D., Tsay, J., Liao, Q. V., Dugan, C., & Erickson, T. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300356 · Cited by 261 · cited 1×
“Some recent work [28, 36, 41] has looked into the practices of data science workers but similarly we found no case of mining of comments or markdowns for machine use.”
BackgroundProc IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Czerepinski, P., Tariq, M. F., Bull, D., Canagarajah, C. N., & Nix, A. (2000) · Cited by 250
DeepProbLog: Neural Probabilistic Logic Programming
Manhaeve, R., Dumančić, S., Kimmig, A., Demeester, T., & De Raedt, L. (2018) · arXiv · doi:10.48550/arxiv.1805.10872 · Cited by 239
Extending document management systems with user-specific active properties
Dourish, P., Edwards, W. K., LaMarca, A., Lamping, J., Petersen, K., Salisbury, M., Terry, D. B., & Thornton, J. (2000) · ACM Transactions on Information Systems · doi:10.1145/348751.348758 · Cited by 232 · cited 2×
“taxonomies is a traditional strategy to organize individual items to simplify management and information retrieval [3, 18, 32, 49, 50, 53].”
BackgroundA “pile” metaphor for supporting casual organization of information
Mander, R., Salomon, G., & Wong, Y. Y. (1992) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '92 · doi:10.1145/142750.143055 · Cited by 214 · cited 2×
“taxonomies is a traditional strategy to organize individual items to simplify management and information retrieval [3, 18, 32, 49, 50, 53].”
Background@tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies
Tan, S. H., Marinov, D., Tan, L., & Leavens, G. T. (2012) · 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation · doi:10.1109/icst.2012.106 · Cited by 197 · cited 1×
“…some research looked into inspecting comments to identify possible problems and sometimes suggest modifcations in the source code to developers, such as in [44, 46], but we have not found instances where the computer extracted information from the documentation for its own use as we do here.”
BackgroundDon't take my folders away!: organizing personal information to get ghings done
Jones, W., Phuwanartnurak, A. J., Gill, R., & Bruce, H. (2005) · CHI '05 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1056808.1056952 · Cited by 189 · cited 2×
“More recently, the problem of organizing fle folders in a computer has also been studied in depth, for instance in [23].”
BackgroundMethodGarcez, A. D., Gori, M., Lamb, L. C., Serafini, L., Spranger, M., & Tran, S. N. (2019) · arXiv · doi:10.48550/arxiv.1905.06088 · Cited by 175
A retrospective of knowledge graphs
Yan, J., Wang, C., Cheng, W., Gao, M., & Zhou, A. (2018) · Frontiers of Computer Science · doi:10.1007/s11704-016-5228-9 · Cited by 158 · cited 1×
“[56] presented a summarization of techniques for constructing KGs by reviewing existing systems in the industry and academia, focusing on extraction”
MethodBengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1709.08568 · Cited by 155 · cited 1×
“AI [4, 6, 16, 17, 22, 33, 35, 37] which has been exploring technologies which encompass simultaneously machine learning and symbolic knowledge representation and reasoning.”
BackgroundProc. Advances in Neural Information Processing Systems (NIPS '08)
Ting, J.-A., Kalakrishnan, M., Vijayakumar, S., & Stefan Schaal, A. (2008) · Cited by 143
API documentation from source code comments: a case study of Javadoc
Kramer, D. (1999) · Proceedings of the 17th annual international conference on Computer documentation · doi:10.1145/318372.318577 · Cited by 142 · cited 1×
“In spite of numerous proposals to enhance and facilitate the embedding of documentation in source code [29, 52], including a variety of markup languages [13, 27], almost invariably program compilers simply ignore such documentation.”
BackgroundWhere did you put it? Issues in the design and use of a group memory
Berlin, L. M., Jeffries, R., O’Day, V. L., Paepcke, A., & Wharton, C. (1993) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/169059.169063 · Cited by 132 · cited 1×
“Issues of naming and categorizing shared group knowledge have been studied since the 1980s by the CHI community [2, 5, 7].”
BackgroundThe link-prediction problem for social networks
Liben-NowellDavid, & KleinbergJon (2007) · Journal of the American Society for Information Science and Technology · doi:10.5555/1241540.1241551 · Cited by 125
Learning and remembering command names
Black, J. B., & Moran, T. P. (1982) · ACM Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/800049.801745 · Cited by 97 · cited 1×
“Issues of naming and categorizing shared group knowledge have been studied since the 1980s by the CHI community [2, 5, 7].”
BackgroundaComment: mining annotations from comments and code to detect interrupt related concurrency bugs
Tan, L., Zhou, Y., & Padioleau, Y. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985796 · Cited by 93 · cited 1×
“…some research looked into inspecting comments to identify possible problems and sometimes suggest modifcations in the source code to developers, such as in [44, 46], but we have not found instances where the computer extracted information from the documentation for its own use as we do here.”
BackgroundChen, Y.-N., Hakkani-Tür, D. Z., & He, X. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472838 · Cited by 85 · cited 1×
“USE+C : we used the Convolutional Deep Structured Semantic Model (CDSSM) proposed in [12] which consists of a three-layer convolutional neural network trained for creating embed-dings of nameIds represented as sentences.”
MethodIdyll: A Markup Language for Authoring and Publishing Interactive Articles on the Web
Conlen, M., & Heer, J. (2018) · Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3242587.3242600 · Cited by 72 · cited 1×
“In spite of numerous proposals to enhance and facilitate the embedding of documentation in source code [29, 52], including a variety of markup languages [13, 27], almost invariably program compilers simply ignore such documentation.”
BackgroundOut-of-Domain Detection for Low-Resource Text Classification Tasks
Tan, M., Yu, Y., Wang, H., Wang, D., Potdar, S., Chang, S., & Yu, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1364 · Cited by 72 · cited 1×
“both in-scope and OOS examples; b) False Acceptance Rate (FAR), the rate of OOS examples wrongly considered as in-scope; and c) InScope Error Rate (ISER), the error rate for in-scope only examples with θ set to zero [30, 45].”
BackgroundGries, D. (1978) · Springer eBooks · doi:10.1007/978-1-4612-6315-9_7 · Cited by 59
Classifying code comments in Java software systems
Pascarella, L., Bruntink, M., & Bacchelli, A. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09694-w · Cited by 57 · cited 2×
“A large study of Java repositories [38] proposed a taxonomy of code comments with main categories being purpose, notice, under development, and metadata.”
BackgroundCollaborative Practices with Structured Data: Do Tools Support What Users Need?
Koesten, L. M., Kacprzak, E., Tennison, J., & Simperl, E. P. B. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300330 · Cited by 52 · cited 1×
“Some recent work [28, 36, 41] has looked into the practices of data science workers but similarly we found no case of mining of comments or markdowns for machine use.”
BackgroundMapping Text to Knowledge Graph Entities using Multi-Sense LSTMs
Kartsaklis, D., Pilehvar, M. T., & Collier, N. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1221 · Cited by 51 · cited 2×
“This is, in fact, a characteristic of a new type of neuro-symbolic algorithms such as [10, 25].”
BackgroundMethodOut-of-Domain Utterance Detection Using Classification Confidences of Multiple Topics
Lane, I. R., Kawahara, T., Matsui, T., & Nakamura, S. (2006) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2006.876727 · Cited by 50 · cited 1×
“both in-scope and OOS examples; b) False Acceptance Rate (FAR), the rate of OOS examples wrongly considered as in-scope; and c) InScope Error Rate (ISER), the error rate for in-scope only examples with θ set to zero [30, 45].”
BackgroundExploring knowledge graphs for exploratory search
Sarrafzadeh, B., Vechtomova, O., & Jokic, V. (2014) · Information Interaction in Context Symposium (IIiX) · doi:10.1145/2637002.2637019 · Cited by 34 · cited 1×
“More related to our work, Sarrafzadeh et al. [42] studied how KGs and hierarchical trees support exploratory search.”
MethodNeuro-Symbolic = Neural + Logical + Probabilistic
Raedt, L. D., Manhaeve, R., Dumančić, S., Demeester, T., & Kimmig, A. (2019) · Ghent University Academic Bibliography (Ghent University) · Cited by 29
API documentation and software community values: a survey of open-source API documentation
Watson, R. B., Stamnes, M., Jeannot-Schroeder, J., & Spyridakis, J. H. (2013) · ACM International Conference on Design of Communication (SIGDOC) · doi:10.1145/2507065.2507076 · Cited by 28 · cited 1×
“In spite of numerous proposals to enhance and facilitate the embedding of documentation in source code [29, 52], including a variety of markup languages [13, 27], almost invariably program compilers simply ignore such documentation.”
BackgroundA Survey on Research of Code Comment
Bai, Y., Liping, Z., & Zhao, F. (2019) · International Conference on Management Engineering, Software Engineering and Service Sciences (ICMSS) · doi:10.1145/3312662.3312710 · Cited by 24 · cited 2×
“[57] showed that extracting versioning and authorship information from code comments is possible using semi-automatic data extraction techniques.”
BackgroundProphet -- A Link-Predictor to Learn New Rules on NELL
Appel, A. P., & Junior, E. R. H. (2011) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2011.142 · Cited by 21 · cited 2×
“The algorithm described here can be expanded further by employing known techniques of link prediction in knowledge graphs [1, 31] which can add the most likely missing links between concepts and therefore increase the ability of generating new possible nameIds.”
MethodSelf-citationLearning and remembering interactive commands
Barnard, P. J., Hammond, Ν. G. L., MacLean, A., & Morton, J. (1982) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/800049.801744 · Cited by 20 · cited 1×
“Issues of naming and categorizing shared group knowledge have been studied since the 1980s by the CHI community [2, 5, 7].”
BackgroundCavalin, P., Ribeiro, V. H. A., Appel, A. P., & Pinhanez, C. S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.324 · Cited by 20 · cited 1×
“This is, in fact, a characteristic of a new type of neuro-symbolic algorithms such as [10, 25].”
BackgroundSelf-citationTagging might not be slower than filing in folders
Voit, K., Andrews, K., & Slany, W. (2012) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2212776.2223753 · Cited by 19 · cited 2×
“taxonomies is a traditional strategy to organize individual items to simplify management and information retrieval [3, 18, 32, 49, 50, 53].”
BackgroundUnsupervised learning of an extensive and usable taxonomy for DBpedia
Fossati, M., Kontokostas, D., & Lehmann, J. (2015) · International Conference on Semantic Systems · doi:10.1145/2814864.2814881 · Cited by 17 · cited 1×
“More recent work looked into how the Wikipedia categorization could be used to automatically generate a taxonomy for DBpedia [21].”
BackgroundThornton, K., & McDonald, D. W. (2012) · ACM International Conference on Supporting Group Work (GROUP) · doi:10.1145/2389176.2389210 · Cited by 16 · cited 1×
“Thornton and McDonald [47] explored the emergence of the collaborative tagging process of Wikipedia during the early 2000s.”
BackgroundTAXONOMY AND DESIGN CONSIDERATIONS FOR COMMENTS IN PROGRAMMING LANGUAGES: A QUALITY PERSPECTIVE
Abid, K., Ahmad, F. S., Shafiq, M., & Abid, A. (2014) · Cited by 13
Voida, S., & Greenberg, S. (2009) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1518701.1518959 · Cited by 12 · cited 2×
“taxonomies is a traditional strategy to organize individual items to simplify management and information retrieval [3, 18, 32, 49, 50, 53].”
BackgroundUsing Meta-Knowledge Mined from Identifiers to Improve Intent Recognition in Conversational Systems
Pinhanez, C. S., Cavalin, P., Ribeiro, V. H. A., Appel, A. P., Candello, H. C. D. S. P., Nogima, J., Pichiliani, M. C., Guerra, M. A., de Bayser, M. G., Malfatti, G. L., et al. (2021) · doi:10.18653/v1/2021.acl-long.545 · Cited by 12
Self-citationWeiland, M., & Dachselt, R. (2008) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1358628.1358922 · Cited by 10 · cited 2×
“taxonomies is a traditional strategy to organize individual items to simplify management and information retrieval [3, 18, 32, 49, 50, 53].”
BackgroundUser perceptions of the role and value of tags
Kim, Y.-M., & Rieh, S. Y. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979039 · Cited by 9 · cited 1×
“Kim and Rieh [26] identifed that, in the early days of social tagging, only about 33% of users understood them as categories while more than 64% of the active taggers thought so.”
Background(1999) · doi:10.1002/(sici)1097-4571(1999)50:12
Research works printed but not linked: 11
- 10.5555/324058.324059 · doi:10.5555/324058.324059
- Tarek R. Besold, Artur d’Avila Garcez, Sebastian Bader, Howard Bowman, Pedro Domingos, Pascal Hitzler, Kai-Uwe Kuehnberger, Luis C. Lamb, Daniel Lowd, Priscila Machado Vieira Lima, Leo de Penning, Gadi Pinkas, Hoifung Poon, and Gerson Zaverucha. 2017. Neural-Symbolic Learning and Reasoning: A Survey and Interpretation. arxiv:1711.03902 [cs.AI] · arXiv
- Harry Bunt. 2009. The DIT++ taxonomy for functional dialogue markup. In Proc. of the AAMAS 2009 Workshop, Towards a Standard Markup Language for Embodied Dialogue Acts, Dirk Heylen, Catherine Pelachaud, Roberta Catizone, and David Traum (Eds.). Hungarian John von Neumann Society, Budapest, Hungary, 13–24.
- Harry Bunt and Yann Girad. 2005. Designing an Open, Multidimensional Dialogue Act Taxonomy. In Proc. of the Ninth International Workshop on the Semantics and Pragmatics of Dialogue (DIALOR’05). LORIA-INRIA, Nancy, France, 13–24.
- Ole-Johan Dahl, Edsger Wybe Dijkstra, and Charles Antony Richard Hoare. 1972. Structured Programming. Academic Press Ltd., GBR.
- Lisa Ehrlinger and Wolfram Wöß. 2016. Towards a Definition of Knowledge Graphs. In Proc. of SEMANTICS’16 Posters and Demos Track. CEUR-WS, 1–4. http://ceur-ws.org
- Drew Hudson and Christopher D Manning. 2019. Learning by Abstraction: The Neural State Machine. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc., Redhook, New York, USA, 5903–5916.
- 10.3233/978-1-61499-649-1-87 · DOI · Paper page
- Emilio Parisotto Abdel-rahman Mohamed Rishabh Singh Lihong Li Dengyong Zhou and Pushmeet Kohli. 2016. Neuro-Symbolic Program Synthesis. arxiv:1611.01855 [cs.LG] · arXiv
- Katrin Weller. 2007. Folksonomies and Ontologies: Two New Players in Indexing and Knowledge Representation. In Proc. of the 2007 Online Information Conference. Incisive Media, London, UK, 108–115.
- Ziwei Zhang Peng Cui and Wenwu Zhu. 2018. Deep Learning on Graphs: A Survey. arxiv:1812.04202 [cs.CL] · arXiv
Cited by
A Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86
“C l a r i t y o f t h e M L b u i l d i n g p r o c e s s c a n b e e l u c i d a t e d u s i n g m o d e l r e p r o d u c i b i l i t y c h e c k l i s t s [ 35 , 55 ] a n d b y e m b e dd i n g s t r u c t u r e d m e t a - k n o w l e d g e i n t o t h e d o c u m e n t a t i o n [ 56 ] .”
BackgroundResultHeigl, R. M. (2025) · Management Review Quarterly · doi:10.1007/s11301-025-00494-9 · Cited by 122
Making Conversational Robots Learn from Children in a Collaborative Museum Exhibition
Candello, H. C. D. S. P., Pichiliani, M. C., & Pinhanez, C. S. (2021) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3462204.3482888 · Cited by 14
“It portrays, in the universe of a children’s exhibition, the actual collaborative process of chatbot development as it is done by professionals [12].”
BackgroundSelf-citationTeaching Conversational Robots in a Museum Exhibition with Interactive Surfaces
Candello, H. C. D. S. P., Pichiliani, M. C., & Pinhanez, C. S. (2021) · ACM International Conference on Interactive Surfaces and Spaces (ISS) · doi:10.1145/3447932.3490680 · Cited by 10
“It portrays, in the universe of a children’s exhibition, the actual collaborative process of chatbot development as it is done by professionals [12].”
BackgroundSelf-citationUsing Meta-Knowledge Mined from Identifiers to Improve Intent Recognition in Conversational Systems
Pinhanez, C. S., Cavalin, P., Ribeiro, V. H. A., Appel, A. P., Candello, H. C. D. S. P., Nogima, J., Pichiliani, M. C., Guerra, M. A., de Bayser, M. G., Malfatti, G. L., et al. (2021) · doi:10.18653/v1/2021.acl-long.545 · Cited by 12
Self-citation
Metadata
- Authors
- Pinhanez, C. S., Candello, H., Cavalin, P., Pichiliani, M. C., Appel, A. P., Alves Ribeiro, V. H., Nogima, J., de Bayser, M., Guerra, M., Ferreira, H., & Malfatti, G.
- Affiliations
- IBM Research - Brazil (BR)
- IBM Research Brazil, Brazil
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-13
- OpenAlex
- W3162148497
- Cited by
- 7
- References
- 53
- Citations by year
- 2025: 2 · 2023: 2 · 2021: 3
- FWCI
- 1.04
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- dialogue systems, curators, neuro-symbolic learning, symbolic knowledge, machine learning training data, meta-knowledge, collaborative practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.