How Do Users Like This Feature? A Fine Grained Sentiment Analysis of App Reviews
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
App stores allow users to submit feedback for downloaded apps in form of star ratings and text reviews. Recent studies analyzed this feedback and found that it includes information useful for app developers, such as user requirements, ideas for improvements, user sentiments about specific features, and descriptions of experiences with these features. However, for many apps, the amount of reviews is too large to be processed manually and their quality varies largely. The star ratings are given to the whole app and developers do not have a mean to analyze the feedback for the single features. In this paper we propose an automated approach that helps developers filter, aggregate, and analyze user reviews. We use natural language processing techniques to identify fine-grained app features in the reviews. We then extract the user sentiments about the identified features and give them a general score across all reviews. Finally, we use topic modeling techniques to group fine-grained features into more meaningful high-level features. We evaluated our approach with 7 apps from the Apple App Store and Google Play Store and compared its results with a manually, peer-conducted analysis of the reviews. On average, our approach has a precision of 0.59 and a recall of 0.51. The extracted features were coherent and relevant to requirements evolution tasks. Our approach can help app developers to systematically analyze user opinions about single features and filter irrelevant reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 3×
“We group collocations whose pairs of words are synonyms and use Wordnet as a synonym dictionary.”
MethodInfluentialFoundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071802878 · Cited by 9,286 · cited 2×
“The methodological cornerstone for the creation of the truth set was the use of content analysis techniques as described by Neuendorf [20] and by Maalej and Robillard [16].”
MethodFeature-Oriented Domain Analysis (FODA) Feasibility Study
Kang, K. C., Cohen, S. G., Hess, J. A., Novak, W. E., & Peterson, A. S. (1990) · Defense Technical Information Center (DTIC) · doi:10.21236/ada235785 · Cited by 4,282
Natural language processing with Python
Bird, S., Klein, E., & Loper, E. (2009) · CERN Document Server (European Organization for Nuclear Research) · Cited by 3,988
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/jmlr.2003.3.4-5.993 · Cited by 3,024
Extracting product features and opinions from reviews
Popescu, A.-M., & Etzioni, O. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220618 · Cited by 1,698
Mining opinion features in customer reviews
Hu, M., & Liu, B. (2004) · National Conference on Artificial Intelligence · Cited by 1,627
Sentiment strength detection in short informal text
Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., & Kappas, A. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21416 · Cited by 1,449
Cited by
A survey on sentiment analysis challenges
Hussein, D. M. E.-D. M. (2018) · Journal of King Saud University - Engineering Sciences · doi:10.1016/j.jksues.2016.04.002 · Cited by 659
A Survey of App Store Analysis for Software Engineering
Martin, W., Sarro, F., Jia, Y., Zhang, Y., & Harman, M. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2630689 · Cited by 459
“[36], 2014 g PIR 6,938 Guzman and Maalej [88], 2014 a,g RE 7 Khalid et al.”
BackgroundMethodHow can i improve my app? Classifying user reviews for software maintenance and evolution
Panichella, S., Di Sorbo, A., Guzman, E., Visaggio, C. A., Canfora, G., & Gall, H. C. (2015) · 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsm.2015.7332474 · Cited by 455
Self-citationBug report, feature request, or simply praise? On automatically classifying app reviews
Maalej, W., & Nabil, H. (2015) · 2015 IEEE 23rd International Requirements Engineering Conference (RE) · doi:10.1109/re.2015.7320414 · Cited by 453
Self-citationA survey of the use of crowdsourcing in software engineering
Mao, K., Capra, L., Harman, M., & Jia, Y. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2016.09.015 · Cited by 385
A systematic literature review: Opinion mining studies from mobile app store user reviews
Genc-Nayebi, N., & Abran, A. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2016.11.027 · Cited by 358
What would users change in my app? summarizing app reviews for recommending software changes
Di Sorbo, A., Panichella, S., Alexandru, C. V., Shimagaki, J., Visaggio, C. A., Canfora, G., & Gall, H. C. (2016) · Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2950290.2950299 · Cited by 294
On the automatic classification of app reviews
Maalej, W., Kurtanović, Z., Nabil, H., & Stanik, C. (2016) · Requirements Engineering · doi:10.1007/s00766-016-0251-9 · Cited by 280
Self-citationSymeonidis, S., Effrosynidis, D., & Arampatzis, A. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.06.022 · Cited by 229
Shoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102447 · Cited by 171
Automated Checking of Conformance to Requirements Templates Using Natural Language Processing
Arora, C., Sabetzadeh, M., Briand, L., & Zimmer, F. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2428709 · Cited by 168
“Guzman and Maalej [70] employ part-ofspeech tagging, stemming and sentiment analysis for synthesizing user opinions about the requirements of mobile applications.”
MethodEnsemble Methods for App Review Classification: An Approach for Software Evolution (N)
Guzman, E., El-Haliby, M., & Bruegge, B. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.88 · Cited by 143
Studying the dialogue between users and developers of free apps in the Google Play Store
Hassan, S., Tantithamthavorn, C., Bezemer, C.-P., & Hassan, A. E. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9538-9 · Cited by 100
A Survey of Performance Optimization for Mobile Applications
Hort, M., Kechagia, M., Sarro, F., & Harman, M. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3071193 · Cited by 91
“There is a huge number of reviews that are written for applications, where some of them contain relevant information for developers [285].”
BackgroundWhen people and algorithms meet: user-reported problems in intelligent everyday applications
Eiband, M., Völkel, S. T., Buschek, D., Cook, S., & Hussmann, H. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302262 · Cited by 62
“Reviews reflect users’ attitude towards an application and document the experiences they make [21, 32].”
BackgroundThematic Analysis on User Reviews for Depression and Anxiety Chatbot Apps: Machine Learning Approach
Ahmed, A., Aziz, S., Khalifa, M., Shah, U., Hassan, A., Abd-Alrazaq, A., & Househ, M. (2022) · JMIR Formative Research · doi:10.2196/27654 · Cited by 37
Nilashi, M., Abumalloh, R. A., Ahmadi, H., Samad, S., Alrizq, M., Abosaq, H., & Alghamdi, A. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e21828 · Cited by 30
“LDA was used for textual data analysis from the collected online reviews in the Google play store platform [56].”
MethodDu, Y., Choe, S., Vega, J., Liu, Y., & Trujillo, A. (2022) · JMIR Pediatrics and Parenting · doi:10.2196/28661 · Cited by 14
Cloud-Based Sentiment Analysis for Interactive Agents
Keijsers, M., Bartneck, C., & Kazmi, H. S. (2019) · Proceedings of the 7th International Conference on Human-Agent Interaction · doi:10.1145/3349537.3351883 · Cited by 9
Metadata
- Authors
- Guzman, E., & Maalej, W.
- Venue
- 2014 IEEE 22nd International Requirements Engineering Conference (RE)
- Publisher
- IEEE
- Published
- 2014-08
- Pages
- 153-162
- OpenAlex
- W2074975950
- Cited by
- 590
- References
- 26
- FWCI
- 23.72
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.