The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Evaluating and predicting answer quality in community QA
Abstract
Question answering (QA) helps one go beyond traditional keywords-based querying and retrieve information in more precise form than given by a document or a list of documents. Several community-based QA (CQA) services have emerged allowing information seekers pose their information need as questions and receive answers from their fellow users. A question may receive multiple answers from multiple users and the asker or the community can choose the best answer. While the asker can thus indicate if he was satisfied with the information he received, there is no clear way of evaluating the quality of that information. We present a study to evaluate and predict the quality of an answer in a CQA setting. We chose Yahoo! Answers as such CQA service and selected a small set of questions, each with at least five answers. We asked Amazon Mechanical Turk workers to rate the quality of each answer for a given question based on 13 different criteria. Each answer was rated by five different workers. We then matched their assessments with the actual asker’s rating of a given answer. We show that the quality criteria we used faithfully match with asker’s perception of a quality answer. We furthered our investigation by extracting various features from questions, answers, and the users who posted them, and training a number of classifiers to select the best answer using those features. We demonstrate a high predictability of our trained models along with the relative merits of each of the features for such prediction. These models support our argument that in case of CQA, contextual information such as a user’s profile, can be critical in evaluating and predicting content quality.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sense‐making theory and practice: an overview of user interests in knowledge seeking and use
Dervin, B. (1998) · Journal of Knowledge Management · doi:10.1108/13673279810249369 · Cited by 781
Predictors of answer quality in online Q&A sites
Harper, F. M., Raban, D., Rafaeli, S., & Konstan, J. A. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357191 · Cited by 412
Predicting information seeker satisfaction in community question answering
Liu, Y., Bian, J., & Agichtein, E. (2008) · Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1390334.1390417 · Cited by 254
Evaluation of evaluation in information retrieval
Saracevic, T. (1995) · Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '95 · doi:10.1145/215206.215351 · Cited by 225
Internet-scale collection of human-reviewed data
Su, Q., Pavlov, D., Chow, J.-H., & Baker, W. C. (2007) · International World Wide Web Conference (WWW) · doi:10.1145/1242572.1242604 · Cited by 125
Exploring characteristics and effects of user participation in online social Q&A sites
Shah, C. S., Oh, J. S., & Oh, S. (2008) · First Monday · doi:10.5210/fm.v13i9.2182 · Cited by 92
Self-citationChallenges in cross-cultural/multilingual music information seeking
Lee, J. H., Downie, J. S., & Cunningham, S. J. (2005) · Research Commons (University of Waikato) · doi:10.5072/zenodo.243526 · Cited by 64
A Multi-Dimensional Model for Assessing the Quality of Answers in Social Q&A Sites
Zhu, Z., Bernhard, D., & Gurevych, I. (2009) · TUbilio (Technical University of Darmstadt) · doi:10.26083/tuprints-00001940 · Cited by 61
Cited by
What are developers talking about? An analysis of topics and trends in Stack Overflow
Barua, A., Thomas, S. W., & Hassan, A. E. (2014) · Empirical Software Engineering · doi:10.1007/s10664-012-9231-y · Cited by 612
How do programmers ask and answer questions on the web? (NIER track)
Treude, C., Barzilay, O., & Storey, M.-A. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985907 · Cited by 405
Anderson, A., Huttenlocher, D., Kleinberg, J., & Leskovec, J. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339665 · Cited by 356
Oh, S., & Syn, S. Y. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23320 · Cited by 323
Reputation Transferability in Online Labor Markets
Kokkodis, M., & Ipeirotis, P. G. (2016) · Management Science · doi:10.1287/mnsc.2015.2217 · Cited by 166
“Shah and Pomerantz (2010) use Amazon Mechanical Turk workers to label the quality of the answers, and then they train classifiers that select the highest quality answers.”
Background
Metadata
- Authors
- Shah, C., & Pomerantz, J.
- Venue
- Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
- Publisher
- ACM
- Published
- 2010-07-19
- Pages
- 411-418
- OpenAlex
- W2057415299
- Cited by
- 346
- References
- 8
- FWCI
- 15.29
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.