MRR: an unsupervised algorithm to rank reviews by relevance
The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
The automatic detection of relevant reviews plays a major role in tasks such as opinion summarization, opinion-based recommendation, and opinion retrieval. Supervised approaches for ranking reviews by relevance rely on the existence of a significant, domain-dependent training data set. In this work, we propose MRR (Most Relevant Reviews), a new unsupervised algorithm that identifies relevant revisions based on the concept of graph centrality. The intuition behind MRR is that central reviews highlight aspects of a product that many other reviews frequently mention, with similar opinions, as expressed in terms of ratings. MRR constructs a graph where nodes represent reviews, which are connected by edges when a minimum similarity between a pair of reviews is observed, and then employs PageRank to compute the centrality. The minimum similarity is graph-specific, and takes into account how reviews are written in specific domains. The similarity function does not require extensive pre-processing, thus reducing the computational cost. Using reviews from books and electronics products, our approach has outperformed the two unsupervised baselines and shown a comparable performance with two supervised regression models in a specific setting. MRR has also achieved a significantly superior run-time performance in a comparison with the unsupervised baselines.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
(1999) · Cited by 12,854
Authoritative sources in a hyperlinked environment
Kleinberg, J. M. (1999) · Journal of the ACM · doi:10.1145/324133.324140 · Cited by 8,959 · cited 1×
“Popular algorithms to calculate node centrality are PageRank [12], and HITS [6].”
Background(2010) · Introduction to Combinatorics · doi:10.1201/b12336-12 · Cited by 5,632
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841 · cited 1×
“Since each product that composes the gold standard has a distinct helpfulness score, we used the Normalized Discounted Cumulative Gain (NDCG) [3] measure as the evaluation metric of our ranking task.”
MethodWhat Makes a Helpful Online Review? A Study of Customer Reviews on Amazon.Com1
Mudambi, S. M., & Schuff, D. (2010) · MIS Quarterly · doi:10.2307/20721420 · Cited by 2,490
What makes a helpful online review? a study of customer reviews on amazon.com
Mudambi, S. M., & Schuff, D. (2010) · MIS Quarterly · doi:10.5555/2017447.2017457 · Cited by 1,281
Inferring Networks of Substitutable and Complementary Products
McAuley, J., Pandey, R., & Leskovec, J. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783381 · Cited by 645 · cited 1×
“The dataset comprises reviews of electronic products and books extracted from the Amazon website [9].”
BackgroundAutomatically assessing review helpfulness
Kim, S.-M., Pantel, P., Chklovski, T., & Pennacchiotti, M. (2006) · Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing - EMNLP '06 · doi:10.3115/1610075.1610135 · Cited by 518 · cited 4×
“Previous work on automatic review helpfulness assessment [5, 10] model the relevance as the proportion of users who have given a positive vote about the total of given votes (positive and negative votes).”
BackgroundMethodInfluentialSurvey on mining subjective data on the web
Tsytsarau, M., & Palpanas, T. (2012) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-011-0238-6 · Cited by 443 · cited 1×
“The automatic detection of relevant reviews is an important problem, which plays a major role in tasks such as opinion summarization [7], opinion-based recommendation [1], and opinion-mining [15].”
BackgroundRecommender systems based on user reviews: the state of the art
Chen, L., Chen, G., & Wang, F. (2015) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-015-9155-5 · Cited by 403 · cited 2×
“Let R be a set of reviews, and r ∈ R a tuple ⟨ t , rs ⟩ , where r . t represents the text of the review and r . rs a rating score ∈ [ 1 , 5 ] that the reviewer has assigned to it.”
BackgroundAddressing cold-start problem in recommendation systems
Lam, X. N., Vu, T., Le, T. D., & Duong, A. D. (2008) · Proceedings of the 2nd international conference on Ubiquitous information management and communication · doi:10.1145/1352793.1352837 · Cited by 401 · cited 1×
“However, in addition to being a manual process dependent on motivated consumers, this approach fails to consider the most recent reviews and those with fewer votes, configuring the so-called “cold start problem" [8].”
BackgroundLearning Semantic Representations of Users and Products for Document Level Sentiment Classification
Tang, D., Qin, B., & Liu, T. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1098 · Cited by 365 · cited 1×
“The use of sentiment analysis as the basis for predicting review relevance using supervised learning has been addressed by works such as [11, 13], but the relation between sentiment and relevance has not been proved consistent [2].”
BackgroundWhat Makes a Helpful Review? A Study of Customer Reviews on Amazon.comPreprint
Mudambi, S. M., & Schuff, D. (2010) · SSRN Electronic Journal · Cited by 356
Statistical Normalization and Back Propagationfor Classification
Jayalakshmi, T., & Santhakumaran, A. (2011) · International Journal of Computer Theory and Engineering · doi:10.7763/ijcte.2011.v3.288 · Cited by 298 · cited 1×
“Function sim_star , defined in Equation 2, is based on the euclidean distance normalized by Min-Max scaling[4], which outputs 1 when the ratings scores u . rs and v . rs are identical, and 0 for the maximal distance.”
BackgroundChua, A. Y. K., & Banerjee, S. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.08.057 · Cited by 231 · cited 2×
“in [2] explored other features in addition to”
BackgroundRevRank: A Fully Unsupervised Algorithm for Selecting the Most Helpful Book Reviews
Tsur, O., & Rappoport, A. (2009) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v3i1.13945 · Cited by 134 · cited 8×
“In this sense, unsupervised methods [14, 20] are an attractive alternative to avoid the labor-intense and error-prone task of manual annotation of training datasets.”
BackgroundMethodInfluentialMukherjee, A., & Liu, B. (2012) · Cited by 96
Semantic Analysis and Helpfulness Prediction of Text for Online Product Reviews
Yang, Y., Yan, Y., Qiu, M., & Bao, F. S. (2015) · doi:10.3115/v1/p15-2007 · Cited by 76
Automatically Predicting Peer-Review Helpfulness
Xiong, W., & Litman, D. J. (2011) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 59
Co-Regression for Cross-Language Review Rating Prediction
Wan, X. (2013) · Cited by 38
Modeling the Helpful Opinion Mining of Online Consumer Reviews as a Classification Problem
Zeng, Y.-C., Wu, S.-H., Wu, S.-H., Chen, L.-P., & Chen, G. (2013) · International Joint Conference on Natural Language Processing · Cited by 27
An unsupervised approach to rank product reviews
Wu, J., Xu, B., & Li, S. (2011) · International Conference on Fuzzy Systems and Knowledge Discovery (FSKD) · doi:10.1109/fskd.2011.6019793 · Cited by 15 · cited 7×
“In this sense, unsupervised methods [14, 20] are an attractive alternative to avoid the labor-intense and error-prone task of manual annotation of training datasets.”
BackgroundMethodInfluentialSummarizing User Opinions: A Method for Labeled-data Scarce Product Domains
Kurian, N., & Asokan, S. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.01.062 · Cited by 4 · cited 1×
“The automatic detection of relevant reviews is an important problem, which plays a major role in tasks such as opinion summarization [7], opinion-based recommendation [1], and opinion-mining [15].”
Background
Research works printed but not linked: 3
- 10.5555/1610075.1610135 · doi:10.5555/1610075.1610135
- 10.5555/2390524.2390570 · doi:10.5555/2390524.2390570
- Frank Wilcoxon , SK Katti , and Roberta A Wilcox . 1970. Critical values and probability levels for the Wilcoxon rank sum test and the Wilcoxon signed rank test. Selected tables in mathematical statistics 1 ( 1970 ), 171--259. Frank Wilcoxon, SK Katti, and Roberta A Wilcox. 1970. Critical values and probability levels for the Wilcoxon rank sum test and the Wilcoxon signed rank test. Selected tables in mathematical statistics 1 (1970), 171--259.
Cited by
DDC-Outlier: Preventing Medication Errors Using Unsupervised Learning
dos Santos, H. D. P., Ulbrich, A. H. D. P. S., Woloszyn, V., & Vieira, R. L. R. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2828028 · Cited by 79
“The same idea was successfully employed, for instance, to rank textual information of documents and automatically create textual summaries that contain the most common words based on their centrality on a graph [20], [21].”
MethodSelf-citationWang, R., Chen, Z., Zhang, M. R., Li, Z., Liu, Z., Dang, Z., Yu, C., & Chen, X. '. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445547 · Cited by 42
“The next level lower are reviews that contain the query keywords but do not meet any rules, within which they are ranked by relevance based on the concept of graph centrality following prior work [37].”
BackgroundDistrustRank: Spotting False News Domains
Woloszyn, V., & Nejdl, W. (2018) · ACM Web Science Conference (WebSci) · doi:10.1145/3201064.3201083 · Cited by 20
“However, setting an appropriate threshold is a tricky problem [19].”
BackgroundSelf-citationIs there Gender bias and stereotype in Portuguese Word Embeddings?Preprint
Santana, B. S., Woloszyn, V., & Wives, L. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.04528 · Cited by 16
“On the other side of the spectrum, there are Unsupervised and semi-supervised learning techniques, that are attractive because they do not imply the cost of corpus annotation [16,14,17,15].”
BackgroundSelf-citationBlogSet-BR: A Brazilian Portuguese Blog Corpus
dos Santos, H. D. P., Woloszyn, V., & Vieira, R. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/34xgyquk7mxj · Cited by 14
“Such data sets allow the accomplishment of many different tasks, such as relevance ranking for online documents and several other machine learning tasks(Woloszyn et al., 2016; Woloszyn et al., 2017).”
BackgroundSelf-citation
Metadata
- Authors
- Woloszyn, V., dos Santos, H. D. P., Wives, L. K., & Becker, K.
- Affiliations
- Universidade Federal do Rio Grande do Sul (BR)
- Pontificia Universidade, Catolica do Rio Grande do Sul, Porto Alegre, Rio Grande do Sul, Brazil
- Venue
- Proceedings of the International Conference on Web Intelligence
- Publisher
- ACM
- Published
- 2017-08-23
- Pages
- 877-883
- OpenAlex
- W2744303556
- Cited by
- 9
- References
- 23
- Citations by year
- 2026: 1 · 2021: 3 · 2018: 4
- FWCI
- 0.50
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/WIC/ACM International Conference on Web Intelligence (publishedVersion)
- Funders
- CAPES and CNPq, Brazil (459322/2014-1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- graph centrality, PageRank, opinion retrieval, opinion summarization, product reviews
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.