CoChat
Sign Up
Cited by 3Spam and Phishing DetectionCybercrime and Law Enforcement StudiesAdvanced Malware Detection Techniques

RacketStore

Open access
Hernandez, Nestor, Recabarren, Ruben, Carbunar, Bogdan, Ahmed, Syed Ishtiaque·2021·Proceedings of the 21st ACM Internet Measurement Conference
DOI 10.1145/3487552.3487837
Open this paper in CoChat — free accountOpen PDF

Abstract

Online app search optimization (ASO) platforms that provide bulk installs and fake reviews for paying app developers in order to fraudulently boost their search rank in app stores, were shown to employ diverse and complex strategies that successfully evade state-of-the-art detection methods. In this paper we introduce RacketStore, a platform to collect data from Android devices of participating ASO providers and regular users, on their interactions with apps which they install from the Google Play Store. We present measurements from a study of 943 installs of RacketStore on 803 unique devices controlled by ASO providers and regular users, that consists of 58,362,249 data snapshots collected from these devices, the 12,341 apps installed on them and their 110,511,637 Google Play reviews. We reveal significant differences between ASO providers and regular users in terms of the number and types of user accounts registered on their devices, the number of apps they review, and the intervals between the installation times of apps and their review times. We leverage these insights to introduce features that model the usage of apps and devices, and show that they can train supervised learning algorithms to detect paid app installs and fake reviews with an F1-measure of 99.72% (AUC above 0.99), and detect devices controlled by ASO providers with an F1-measure of 95.29% (AUC = 0.95). We discuss the costs associated with evading detection by our classifiers and also the potential for app stores to use our approach to detect ASO work with privacy.

AI snapshot

Written from the paper’s full text.

TL;DR

RacketStore is a measurement platform that collected Android app and device usage from consenting ASO workers and regular Google Play users. The authors show that ASO workers differ from regular users in registered accounts, reviewed apps, and install-to-review timing, and that these features train classifiers to detect paid app installs/fake reviews and worker-controlled devices with high accuracy.

Key findings

  • The study collected 58,362,249 snapshots from 943 RacketStore installs on 803 unique devices: 580 worker-controlled devices and 223 regular-user devices, covering 12,341 installed apps and 110,511,637 Google Play reviews.
  • Worker devices had far more registered Gmail accounts: average 28.87 per device (M = 21, SD = 29.37), with 13 devices having more than 100 and a maximum of 163; regular devices had a maximum of 10 (M = 2, SD = 1.66).
  • Worker devices posted far more reviews: average 208.91 total reviews per worker device versus 1.91 per regular device; 11 worker devices were each responsible for more than 1,000 reviews, while the maximum for a regular device was 36.
  • Workers reviewed apps sooner after installation: 13,376 of 40,397 worker-account reviews were posted within one day of install, with average waiting time 10.4 days (M = 5.00, SD = 13.72); regular users averaged 85.09 days (M = 21.92, SD = 140.56), and only 4 of 35 regular reviews were posted within one day.
  • Classifiers using app/device usage features detected paid app installs and fake reviews with F1 = 99.72% (AUC above 0.99) and worker-controlled devices with F1 = 95.29% (AUC = 0.95).
  • 69.1% of analyzed worker devices had organic-indicative behaviors, while the remaining devices appeared used exclusively for app promotion; worker-controlled devices had 10,310 Gmail accounts registered, and Google Play was still displaying 217,041 reviews posted from them at the time of writing.

Methods

  • The authors built RacketStore, an Android app plus backend that collected slow snapshots every 2 minutes (accounts, device status, stopped apps) and fast snapshots every 5 seconds (foreground app, screen/battery, install/uninstall events), then aggregated data with Google Play review crawling and VirusTotal analysis.
  • They recruited ASO workers from Facebook groups dedicated to product promotion and regular Android users through Instagram ads, requiring participants to keep RacketStore installed for at least two days; after fingerprinting and coalescing repeat installs, they analyzed 803 unique devices.
  • They extracted app-usage and device-usage features, trained supervised classifiers (including XGB, RF, LR, KNN, LVQ, SVM) with repeated 10-fold cross-validation, and used balanced-dataset experiments with undersampling/oversampling and SMOTE.

Limitations

  • The paper states recruitment bias: workers were recruited only from Facebook groups and those who responded, were English speakers, and consented; regular users came from Instagram ads targeted to similar demographics, and the authors do not claim generalization to all workers or regular users.
  • The paper states the classifier training data were relatively small and biased, which may reduce applicability to other ASO workers and regular users; it also notes possible behavioral influence from participants knowing they were monitored.
  • The paper states classifiers need sensitive device/app data and would likely need to be embedded in pre-installed app-store clients or run locally for privacy; the text also makes evident that ground-truth labels rely on recruitment and co-install heuristics rather than independent verification.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3487552.3487837

Cites

60 of 61 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

9 of 37 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 2 SYSTEM MODEL · p. 3

    Communications between developers, admins and crowdsourced workers often occur through dedicated online boards in Facebook, WhatsApp, and Telegram.

    “Communications between developers, admins and crowdsourced workers often occur through dedicated online boards in e.g., Facebook, WhatsApp, Telegram [67], see Figure 2.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [67] [67] Mizanur Rahman, Nestor Hernandez, Ruben Recabarren, Syed Ishtiaque Ahmed, and Bogdan Carbunar. 2019. The Art and Craft of Fraudulent App Promotion in Google Play. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security . 2437-2454. - source

      Section 5.4 (page unavailable) reports that 10 interview participants used Facebook or WhatsApp for team communications. Section 5.1 (page unavailable) describes job managers who interface with developers and admins who distribute tasks to workers. The cited passages do not establish that developers communicate through those groups, and they do not mention Telegram. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 1 INTRODUCTION · p. 1

    Identifying organic ASO activities is an open problem due to the ability of such workers to evade existing detection solutions.

    “Identifying organic ASO activities is an open problem due to the ability of such workers to evade existing detection solutions, e.g., that leverage lockstep behaviors [32, 43, 51, 72, 73, 78, 85, 86, 89, 92, 93] or review bursts [27, 28, 35, 39, 40, 42, 44, 45, 50-53, 53, 5759, 65, 66, 84, 87, 88, 92].”
    Quote in the paperSubstantial claim1 of 10 sources checked
    • cites [32] [32] Qiang Cao, Xiaowei Yang, Jieqi Yu, and Christopher Palow. 2014. Uncovering Large Groups of Active Malicious Accounts in Online Social Networks. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security (CCS '14) . Association for Computing Machinery, New York, NY, USA, [477-488. https://doi.org/10.1145/2660267.2660269](https://doi.org/10.1145/2660267.2660269) - source
    • cites [43] [43] Nestor Hernandez, Mizanur Rahman, Ruben Recabarren, and Bogdan Carbunar. 2018. Fraud De-Anonymization for Fun and Profit. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security . 115-130. - source

      The Abstract says persistent search-rank fraud shows that current detection and filtering are inefficient. The ‘Fraud detection’ passage identifies bursty activity and lockstep behavior as assumptions used by existing approaches. The paper does not specifically discuss organic ASO activities or establish that their workers evade those approaches. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [51] [51] Huayi Li, Geli Fei, Shuai Wang, Bing Liu, Weixiang Shao, Arjun Mukherjee, and Jidong Shao. 2017. Bimodal Distribution and Co-Bursting in Review Spam Detection. In Proceedings of the 26th International Conference on World Wide Web (WWW '17) . International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1063-1072. https://doi.org/10.1145/3038912.3052582 - source
    • cites [72] [72] Jonghyuk Song, Sangho Lee, and Jong Kim. 2015. CrowdTarget: Target-Based Detection of Crowdturfing in Online Social Networks. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS '15) . Association for Computing Machinery, New York, NY, USA, 793-804. https: //doi.org/10.1145/2810103.2813661 - source
    • cites [73] [73] Gianluca Stringhini, Pierre Mourlanne, Gregoire Jacob, Manuel Egele, Christopher Kruegel, and Giovanni Vigna. 2015. EVILCOHORT: Detecting Communities of Malicious Accounts on Online Services. In 24th USENIX Security Symposium (USENIX Security 15) . USENIX Association, Washington, D.C., 563578. https://www.usenix.org/conference/usenixsecurity15/technical-sessions/ presentation/stringhini -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [78] [78] Tian Tian, Jun Zhu, Fen Xia, Xin Zhuang, and Tong Zhang. 2015. Crowd Fraud Detection in Internet Advertising . International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1100-1110. https: //doi.org/10.1145/2736277.2741136 - source
    • cites [85] [85] Zhen Xie and Sencun Zhu. 2014. GroupTie: Toward Hidden Collusion Group Discovery in App Stores. In Proceedings of the 2014 ACM Conference on Security and Privacy in Wireless and Mobile Networks (WiSec '14) . Association for Computing Machinery, New York, NY, USA, 153-164. https://doi.org/10.1145/2627393. 2627409 - source
    • cites [86] [86] Zhen Xie and Sencun Zhu. 2015. AppWatcher: Unveiling the Underground Market of Trading Mobile App Reviews. In Proceedings of the 8th ACM Conference on Security and Privacy in Wireless and Mobile Networks (WiSec '15) . Association for Computing Machinery, New York, NY, USA, Article 10, 11 pages. https: //doi.org/10.1145/2766498.2766510 - source
    • cites [89] [89] Chang Xu and Jie Zhang. [n. d.]. Combating Product Review Spam Campaigns via Multiple Heterogeneous Pairwise Features. In Proceedings of the 2015 SIAM International Conference on Data Mining . 172-180. https://doi.org/10.1137/1. 9781611974010.20 - source
    • cites [92] [92] Junting Ye, Santhosh Kumar, and Leman Akoglu. 2016. Temporal Opinion Spam Detection by Multivariate Indicative Signals. In Proceedings of the Tenth International Conference on Web and Social Media, Cologne, Germany, May 17-20, 2016. 743-746. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Study design and conductcitation
    § 4.1 Ethical Considerations · p. 5

    The study took utmost care to follow the best ethical practices for conducting sensitive research with vulnerable populations.

    “This is why our study took utmost care to follow the best ethical practices for conducting sensitive research with vulnerable populations [30].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [30] [30] Dearbhail Bracken-Roche, Emily Bell, Mary Ellen Macdonald, and Eric Racine. 2017. The concept of 'vulnerability'in research ethics: an in-depth analysis of policies and guidelines. Health research policy and systems 15, 1 (2017), 8. - source
  • Premisecitation
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 19

    Any work with computers and the Internet is considered prestigious in many communities.

    “while in fact, any work with computers and the Internet is considered prestigious in many communities [62].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [62] [62] Joyojeet Pal, Meera Lakshmanan, and Kentaro Toyama. 2009. 'My child will be respected': Parental perspectives on computers and education in Rural India. Information Systems Frontiers 11, 2 (2009), 129-144. - source

      Section 4.1 (page unavailable) reports that parents associated computers with respect and higher social standing. But section 4.3 says factory-worker parents did not see everyday computer users as elevated in class. The paper therefore does not support the claim that any computer-related work is prestigious, and the cited passages do not establish the same claim for Internet work. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Premisecitation
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 19

    Previous work explains why studying ASO workers does not impact their livelihood.

    “Previous work explains why studying the ASO workers does not impact their livelihood [67].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [67] [67] Mizanur Rahman, Nestor Hernandez, Ruben Recabarren, Syed Ishtiaque Ahmed, and Bogdan Carbunar. 2019. The Art and Craft of Fraudulent App Promotion in Google Play. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security . 2437-2454. - source

      Section 4.3, Ethical Considerations, describes confidentiality and privacy safeguards, and the introduction script in Additional information says the researchers will not block workers’ accounts or remove their reviews. But that script also warns that Google or others may use the published results to detect workers’ activities. The paper therefore does not establish the unqualified claim that studying ASO workers does not affect their livelihood. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 49
  • May combine two claims. 62
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 INTRODUCTION · p. 1

    Rahman et al. [67] have reported a variety of detection-avoidance techniques employed by organizations that specialize in retention installs and fake reviews.

    “Rahman et al. [67] have reported a variety of detectionavoidance techniques employed by organizations that specialize in retention installs and fake reviews.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [67] [67] Mizanur Rahman, Nestor Hernandez, Ruben Recabarren, Syed Ishtiaque Ahmed, and Bogdan Carbunar. 2019. The Art and Craft of Fraudulent App Promotion in Google Play. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security . 2437-2454. - source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1 INTRODUCTION · p. 1

    Farooqi et al. [38] have shown that incentivized app install platforms (IIP) are able to provide thousands of installs that successfully evade Google defenses.

    “In previous work, Farooqi et al. [38] have shown that incentivized app install platforms (IIP) are able to provide thousands of installs that successfully evade Google defenses.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [38] [38] Shehroze Farooqi, Álvaro Feal, Tobias Lauinger, Damon McCoy, Zubair Shafiq, and Narseo Vallina-Rodriguez. 2020. Understanding Incentivized Mobile App Installs on Google Play Store. In Proceedings of the ACM Internet Measurement Conference (IMC '20) . Association for Computing Machinery, New York, NY, USA, 696-709. https://doi.org/10.1145/3419394.3423662 - source

      Section 3.2 (pp. 4–5) reports 1,679 incentivized installs for the authors’ test app and an increase in its public Google Play install count from 0 to 1,000. Section 5.2 (p. 11) says these results suggest Google’s detection and filtering may fail; it does not establish that thousands of installs successfully evaded Google’s defenses. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1 INTRODUCTION · p. 1

    ASO workers lack interest in the apps that they promote.

    “Our work is partially motivated by Farooqi et al. [38]'s finding that ASO workers lack interest in the apps that they promote.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [38] [38] Shehroze Farooqi, Álvaro Feal, Tobias Lauinger, Damon McCoy, Zubair Shafiq, and Narseo Vallina-Rodriguez. 2020. Understanding Incentivized Mobile App Installs on Google Play Store. In Proceedings of the ACM Internet Measurement Conference (IMC '20) . Association for Computing Machinery, New York, NY, USA, 696-709. https://doi.org/10.1145/3419394.3423662 - source

      Section 3.2 (p. 5) says that most users who installed the study’s app were likely not genuinely interested in it and appeared to install it for the payout. That supports the claim’s central point, but the paper hedges its inference and concerns users in incentivized-install campaigns, rather than establishing that ASO workers generally lack interest in the apps they promote. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 INTRODUCTION · p. 1

    Detection-avoidance techniques include crowdsourcing ASO work to organic workers, who use their personal devices to conceal ASO work among everyday activities.

    “Such techniques include crowdsourcing ASO work to organic workers , who use their personal devices to conceal ASO work among everyday activities.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [67] [67] Mizanur Rahman, Nestor Hernandez, Ruben Recabarren, Syed Ishtiaque Ahmed, and Bogdan Carbunar. 2019. The Art and Craft of Fraudulent App Promotion in Google Play. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security . 2437-2454. - source

      Section 5.3 (page unavailable) reports that participants used organic ASO workers’ mobile devices, and §5.5 says some recruited organic workers also wrote genuine reviews. Section 5.8 proposes using organic workers as a possible evasion strategy, but frames it as a conjecture. The paper does not establish that workers used personal devices specifically to conceal ASO work among everyday activities. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § KEYWORDS · p. 2

    The classifiers detect worker-controlled devices with an F1-measure of 95.29% and AUC of 0.95.

    “Further, our classifiers detect worker-controlled devices with an F1-measure of 95.29% and AUC of 0.95.”
    • Also stated in: § ABSTRACT · p. 1 · plain
      “detect devices controlled by ASO providers with an F1-measure of 95.29% (AUC = 0.95).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § KEYWORDS · p. 2

    RacketStore was installed 943 times on 803 unique devices: 580 devices controlled by ASO workers and 223 devices controlled by regular users.

    “In total, RacketStore was installed 943 times on 803 unique devices: 580 devices controlled by ASO workers recruited from Facebook groups that specialize in ASO work, and 223 devices of regular Android users recruited through ads purchased in Instagram.”
    • Also stated in: § 5 DATA · p. 5 · plain
      “After this process, we identified 803 unique devices: 580 devices controlled by ASO workers and 223 devices controlled by regular participants recruited through Instagram ads ( § 4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 7

    Worker-controlled accounts post many more reviews and tend to do it soon after installation.

    “Unlike regular users, worker-controlled accounts post many more reviews and tend to do it soon after installation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 7

    11 worker-controlled devices were each responsible for more than 1,000 total reviews, while the maximum number of total reviews from a regular device is only 36.

    “We found 11 worker-controlled devices each responsible for more than 1,000 total reviews. In contrast, the maximum number of total reviews from a regular device is only 36.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 7

    KS reported significant differences in the distributions of installed apps (p-value = 0.008), but ANOVA did not find a statistically significant difference (p-value = 0.301).

    “KS reported significant differences in the distributions ( 𝑝 -value = 0 . 008), but ANOVA did not find a statistically significant difference ( 𝑝 -value = 0 . 301).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 7

    The average number of apps installed is 65.45 on regular devices and 77.56 on worker devices.

    “We observe an average of 65.45 and 77.56 apps installed on regular and worker devices respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.2 Registered Accounts · p. 6

    Worker devices have an average of 28.87 Gmail accounts registered per device (M = 21, SD = 29.37); 13 worker devices have more than 100 Gmail accounts registered, with a maximum of 163 accounts per device.

    “Worker devices have an average of 28.87 accounts registered per device (M = 21, SD = 29.37); 13 worker devices have more than 100 Gmail accounts registered, with a maximum of 163 accounts per device.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5 DATA · p. 5

    The study collected 592,045 slow snapshots and 57,770,204 fast snapshots.

    “We have then collected a total of 592,045 slow snapshots and 57,770,204 fast snapshots ( § 3).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.2 Registered Accounts · p. 6

    The KS test and parametric and non-parametric ANOVA found statistically significant differences between workers and regular users in Gmail accounts (p-value < 0.05).

    “The KS test, and parametric and non-parametric ANOVA found statistically significant differences between workers and regular users ( 𝑝 -value &lt; 0 . 05).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 7

    On average, a worker device is responsible for a total of 208.91 reviews, while a regular user device has posted an average of 1.91 reviews.

    “On average, a worker device is responsible for a total of 208.91 reviews, while a regular user device has only posted an average of 1.91 reviews.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4 DATA COLLECTION · p. 5

    The participants used IP addresses from Pakistan (420), India (210), Bangladesh (148), USA (10) and other countries from Africa, Asia, South America and Europe (15).

    “The participants used IP addresses from Pakistan (420), India (210), Bangladesh (148), USA (10) and other countries from Africa, Asia, South America and Europe (15).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Participant Engagement · p. 6

    The average number of daily snapshots collected from regular devices is 9,430.71 (M = 3,097.67, SD = 12,789.14, max = 63,452) and from worker devices is 8,208.10 (M = 3,669, SD = 10,303.42, max = 55,281.38).

    “The average number of daily snapshots collected from regular devices is 9,430.71 (M = 3,097.67, SD = 12,789.14, max = 63,452) and from worker devices is 8,208.10 (M = 3,669, SD = 10,303.42, max = 55,281.38).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 7

    On average, worker devices have posted reviews for 40.51 of the currently installed apps, while regular devices did so for an average of 0.7 apps.

    “However, on average, worker devices have posted reviews for 40.51 of the currently installed apps, while regular devices did it for an average of 0.7 apps.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.2 Registered Accounts · p. 6

    On average, regular devices have registered accounts for 6 services (max = 19), mostly for different social networks.

    “On average, regular devices have registered accounts for 6 services (max = 19), mostly for different social networks (Facebook, WhatsApp, Telegram, etc).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 7.2 App Classification · p. 11

    XGB outperforms other algorithms, achieving F1-measure of 99.72% for app usage classifier.

    “Extreme Gradient Boosting (XGB) outperforms the other algorithms, achieving an F1-measure of 99.72%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 9

    Several worker devices have many more apps installed than regular devices, and also have more apps used per day.

    “We observe that several worker devices have many more apps installed than regular devices, and also have more apps used per day.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 7.2 App Classification · p. 11

    The number of accounts that have reviewed the app from the device and the average time between install and review are the most important features for the app classifier.

    “The number of accounts that have reviewed the app from the device and the average time between install and review are most important.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 8.2 Device Classification · p. 11

    Precision is 96.81% and false positive rate is 1.41% for device classifier.

    “The precision is 96.81% and the false positive rate is 1.41%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 8

    Several worker devices have a low daily app churn, while some regular devices have a higher daily app churn, making them harder to distinguish based on this feature alone.

    “We observe however that several worker devices have a low daily app churn, while some regular devices have a higher daily app churn, making them harder to distinguish based on this feature alone.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 7

    There were 25 cases of worker device waiting times longer than 100 days and 4 cases of reviews posted after more than 1,000 days from 2 workers for 2 apps (Facebook and Easypaisa).

    “We have observed 25 cases with waiting times longer than 100 days and 4 cases of reviews posted after more than 1,000 days from 2 workers and for 2 apps (Facebook and Easypaisa).”
    • Also stated in: § 6.3 Installed Apps · p. 8 · plain
      “We have observed 25 cases for worker devices having waiting times longer than 100 days, and 4 cases of reviews posted after more than 1,000 days (from 2 workers for Facebook and 2 for Easypaisa).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.3 Installed Apps · p. 9

    Four other workers said there are permissions they grant grudgingly.

    “Four other workers said that there are permissions that they grant grudgingly”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 7

    For apps that provided an installation time, accounts on regular devices wrote 35 reviews, while accounts on worker devices posted 40,397 reviews.

    “We observe substantial difference in the number of reviews posted from accounts registered on worker vs. regular devices for apps that provided an installation time: accounts on regular devices only wrote 35 reviews, while those on worker devices posted 40,397 reviews.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 8

    Worker devices had an average of 15.94 daily installs (M=6.41, SD=27.37), while regular devices had an average of 3.88 daily installs (M=2.0, SD=7.29).

    “Concretely, worker devices had an average of 15.94 daily installs (M = 6.41, SD = 27.37) while regular devices had an average of 3.88 daily installs (M = 2.0, SD = 7.29).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 9

    While some worker devices host apps with the largest number of dangerous permissions, most installed apps share a similar permission profile across all device types.

    “We found that while some worker devices host apps with the largest number of dangerous permissions, most installed apps share a similar permission profile across all device types.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.4 Investigation of Malware and Attitudes · p. 10

    Five workers reported being concerned about malware; four claimed to use AV apps.

    “Five workers reported being concerned about malware; four claimed to use AV apps.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Findingnone given
    § 6.3 Installed Apps · p. 8

    Another worker claimed limited storage is to blame, saying 'Sometimes the apps get hanged due to a lack of storage'.

    “Another claimed that limited storage is to blame, ' Sometimes the apps get hanged due to a lack of storage '.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.3 Installed Apps · p. 8

    One worker admitted reason #2 applies, saying 'The quality of some apps was bad, I stopped those apps'.

    “One worker however admitted reason #2 applies, i.e., ' The quality of some apps was bad, I stopped those apps '.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 8.2 Device Classification · p. 11

    XGB outperforms other algorithms, achieving F1-measure of 95.29% and AUC of 0.9455 for device classifier.

    “Extreme Gradient Boosting (XGB) outperforms the other algorithms, achieving an F1-measure of 95.29% and AUC of 0.9455.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 2 SYSTEM MODEL · p. 3

    The authors found 11 public and 5 closed Facebook groups that matched their criteria, with 86,718 members in total (Min = 354, Max = 26896, M = 2840.5, SD = 6787.96).

    “Wefound 11 public and 5 closed groups that matched our criteria. We became members of the public groups and sent requests to closed groups which were all accepted. These groups had 86,718 members (Min = 354, Max = 26896, M = 2840.5, SD = 6787.96) in total.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4 DATA COLLECTION · p. 5

    The authors spent a total of $79.23 on Instagram ads.

    “Weposted the ads intermittently between December 17, 2019 and April 15, 2020, spending a total of $79.23.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 8

    The app churn of most regular devices is less than 10 apps per day, while for many worker devices it is above 10 apps per day.

    “The app churn of most regular devices is less than 10 apps per day, while for many worker devices it is above 10 apps per day.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4 DATA COLLECTION · p. 5

    The Instagram ad was shown 136,022 times, reached 61,748 users, and 2,471 users clicked on it and made it to the landing page.

    “According to the Facebook Ads Manager, our ad was shown a total of 136,022 times and reached 61,748 users. 2,471 of these users clicked on the Instagram ad and made it to our landing page.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.4 Investigation of Malware and Attitudes · p. 10

    Two workers were not concerned about malware or privacy leaks and did not have AV apps installed.

    “Two workers were not concerned about malware or privacy leaks and did not have AV apps installed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 8.2 Device Classification · p. 11

    When using no sampling strategy, F-1 increases to 96.86%, at expense of AUC 0.9083.

    “When using no sampling strategy the F-1 increases to 96.86%, at the expense of the AUC (0.9083).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 3 MEASUREMENTS INFRASTRUCTURE · p. 4

    The top 5 most popular Android manufacturers among participant devices were Samsung, Huawei, Oppo, Xiaomi, and Vivo.

    “The top 5 most popular Android manufacturers were Samsung, Huawei, Oppo, Xiaomi, Vivo).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 8

    Regular users wait 85.09 days on average to post a review (M=21.92, SD=140.56, max=606.11), with only 12 users waiting less than 12 days.

    “Regular users also wait for 85.09 days to post a review on average (M=21.92 days, SD=140.56 days, max=606.11 days) with only 12 users waiting less than 12 days to post a review.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.3 Installed Apps · p. 9

    Two regular participants claimed not to grant all requested permissions.

    “Two regular participants claimed not to grant all requested permissions.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 7.1 App Usage Features · p. 10

    XGB performed the best for device classifier.

    “XGB performed the best.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 18

    The PII collected consisted of user accounts registered on participant devices, e-mail address, device IP address, and IDs.

    “It shows that the PII we collected consists of user accounts registered on participant devices, e-mail address, the device IP address and IDs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.3 Installed Apps · p. 9

    One participant claimed selective granting based on client request.

    “However, one participant claimed selective granting, i.e., ' Permissions are given based on the client request. If the client does not ask, we do not give all permissions '.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 8.2 Device Classification · p. 11

    Four features stand out as important for detecting worker-controlled devices: total number of apps reviewed, percent suspicious apps, number of stopped apps, average number of reviews posted per account.

    “Four features stand out, confirming their ability to detect worker-controlled devices, i.e., (1) the total number of apps reviewed from accounts registered on the device, (2) the percent of installed apps that were detected to have been used suspiciously by the classifier of § 7, (3) the number of stopped apps on the device and (4) the average number of reviews posted from an account registered on”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.3 Installed Apps · p. 8

    One worker admitted to have installed apps from third-party stores, saying 'The client gives us a link, we go and install that app'.

    “One worker admitted to have installed apps from third-party stores, i.e., ' The client gives us a link, we go and install that app '.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 8

    Worker devices recorded an average of 7.02 daily uninstalls (M=2.73, SD=15.69) and regular devices had an average of 3.29 daily uninstalls (M=1.8, SD=6.87).

    “worker devices recorded an average of 7.02 daily uninstalls (M = 2.73, SD = 15.69) and regular devices had an average of 3.29 daily uninstalls (M = 1.8, SD = 6.87).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.4 Investigation of Malware and Attitudes · p. 10

    Only 19 devices installed 15 AV apps: 8 worker devices, 7 regular user devices, and 4 unknown.

    “We found only 19 devices that installed 15 AV apps: 8 worker devices, 7 regular user devices, and 4 unknown”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.4 Investigation of Malware and Attitudes · p. 10

    70 unique mobile app identifiers with at least one VT engine flag received at least one review from participants: 64 reviewed by workers, 9 by regular users.

    “We found 70 unique mobile app identifiers with at least one VT engine flag, that received at least one review from our participants: 64 of these apps were reviewed by workers, and 9 apps were reviewed by regular users.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 6.3 Installed Apps · p. 9

    One regular participant claimed to avoid granting location permissions; another was concerned about contacts, images and phone storage permissions.

    “One claimed to avoid granting location permissions, the other was concerned about contacts, images and phone storage permissions.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 4 DATA COLLECTION · p. 5

    RacketStore received 233 installs from regular participants recruited through Instagram.

    “RacketStore received 233 installs from these participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 18

    Participants claimed that they are not afraid of legal aspects of their work.

    “Participants claimed that they are not afraid, for instance, one worker said that ' What we're doing is a legal and right job. So no need to be afraid '.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § KEYWORDS · p. 2

    ASO work continues to be successful and evade app store detection.

    “We found that ASO work continues to be successful and evade app store detection: The worker-controlled devices of participants in our studies had 10,310 Gmail accounts registered on them and at the time of writing, Google Play was still displaying 217,041 reviews posted from them.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § KEYWORDS · p. 2

    Device and app-engagement features can train classifiers to accurately detect not only promotion-dedicated devices but even elusive organic ASO efforts that hide low levels of app promotion activities among regular personal use.

    “This suggests that our device and app-engagement features can train classifiers to accurately detect not only promotion-dedicated devices but even elusive organic ASO efforts that hide low levels of app promotion activities among regular, personal use of devices and of the apps installed therein.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § KEYWORDS · p. 2

    To protect user privacy, the proposed classifiers can execute directly on the user device, e.g., implemented into the Play Store app.

    “We note that to protect user privacy, the proposed classifiers can execute directly on the user device (e.g., implemented into the Play Store app).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § 3 MEASUREMENTS INFRASTRUCTURE · p. 3

    The RacketStore app was approved as compliant by the Play Store.

    “The RacketStore app was approved as compliant by the Play Store.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § 3 MEASUREMENTS INFRASTRUCTURE · p. 4

    RacketStore uses TLS to encrypt all collected data while in-transit.

    “RacketStore uses TLS to encrypt all collected data while in-transit.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1 INTRODUCTION · p. 1

    ASO activities can be illegal in countries like the US, Canada, and Australia, are banned in the EU, violate the terms of service of app stores, and influence users to install and purchase low quality apps and even malware.

    “Such activities can be illegal in countries like the US [1], Canada [14], Australia [77], are banned in the EU [16], violate the terms of service of app stores [2, 11], and influence users to install and purchase low quality apps and even malware [46, 68, 80].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [1] [1] [n. d.]. 15 U.S. Code § 45 - Unfair methods of competition unlawful; prevention by Commission. Legal Information Institute, https://www.law.cornell.edu/uscode/ text/15/45. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [14] [14] [n. d.]. Untrue, misleading or unauthorized use of tests and testimonials. https: //www.competitionbureau.gc.ca/eic/site/cb-bc.nsf/eng/00527.html. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [77] [77] Ben Thorn. 2019. Fake Reviews and the Australian Consumer Law. https: //xuveo.legal/blog/fake-reviews-australian-consumer-law. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3 MEASUREMENTS INFRASTRUCTURE · p. 4

    RacketStore securely stores all participant data on a server accessible from only 4 IP addresses in the authors' campus.

    “We also securely store all participant data on a server that is accessible from only 4 IP addresses in our campus.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § 1 INTRODUCTION · p. 1

    Identifying ASO-promoted apps and the accounts from which they are promoted allows app stores to filter fake reviews and ratings, generate more accurate install count and aggregate rating values, compute more accurate search ranks for apps, and enable users to make better informed app-installation decisions.

    “Identifying ASO-promoted apps and the accounts from which they are promoted allows app stores to filter fake reviews and ratings, generate more accurate install count and aggregate rating values thus compute more accurate search ranks for apps, and enable users to make better informed app-installation decisions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 3 MEASUREMENTS INFRASTRUCTURE · p. 3

    Participants can accept any subset of the requested permissions; if they do not grant a permission, the corresponding data is not collected.

    “Participants can accept any subset of the requested permissions. If they do not grant a permission, we do not collect the corresponding data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § KEYWORDS · p. 2

    RacketStore first discloses the types of data it collects and collects data only after receiving participant consent.

    “RacketStore first discloses the types of data it collects. RacketStore collects the data only after receiving participant consent ( § 4.1 and Appendix D).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4 DATA COLLECTION · p. 5

    IP addresses can only provide an approximate measure of geolocation.

    “Wenote that IP addresses can only provide an approximate measure of geolocation [56].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [56] [56] James A Muir and PC Van Oorschot. 2006. Internet geolocation and evasion. Ottawa: School of Computer Science, Carleton University (2006). -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 3 MEASUREMENTS INFRASTRUCTURE · p. 3

    The passcode is given only after the user has accepted to participate in the study and has agreed to the data collection process.

    “The passcode is given only after the user has accepted to participate in our study and has agreed to the data collection process.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 4.1 Ethical Considerations · p. 5

    The full study procedure was examined and approved by the Institutional Review Board of the authors' university (IRB-19-0392@FIU).

    “The full study procedure was examined and approved by the Institutional Review Board of our university (IRB-19-0392@FIU).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 1 INTRODUCTION · p. 1

    ASO platforms use a variety of techniques, including providing retention installs and posting fake reviews.

    “ASO platforms use a variety of techniques to achieve this [83], including providing retention installs [12] and posting fake reviews [3, 7].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [83] [83] Madeleine Wilson. 2019. How to Improve App Ratings and Reviews. Apptentive, https://www.apptentive.com/blog/2019/08/22/ improve-mobile-app-ratings-and-reviews/. -

      Not checked in the free run.

    • cites [12] [12] [n. d.]. Retention Rate Meaning. https://www.adjust.com/glossary/ retention-rate/. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [3] [3] [n. d.]. Boostyourapps: Buy App Reviews and Get Installs and Rating for Free. https://boostyourapps.org/. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [7] [7] [n. d.]. MobiASO: Mobile App Marketing Agency. https://mobiaso.com/. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data
    § 4 DATA COLLECTION · p. 5

    The RacketStore app did not collect location information from participant devices, so the authors cannot corroborate the IP-based geolocation.

    “The RacketStore app did not collect location information from participant devices thus we cannot corroborate this information.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 6 DEVICE USAGE MEASUREMENTS · p. 6

    The study used the Kolmogorov-Smirnov (KS) test, and non-parametric and parametric ANOVA to compare distributions and check for significant differences in means across groups.

    “We used the Kolmogorov-Smirnov (KS) test to compare the distributions of the features, and non-parametric and parametric ANOVA(Analysis of Variance) to check for significant differences in the means across groups.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 2 SYSTEM MODEL · p. 3

    Some ASO organizations are white hat, providing advice on app naming and keyword optimization, while others provide illegal, banned or discouraged services including retention installs and fake reviews with high ratings.

    “While some ASO organizations are white hat [25], providing advice on e.g., app naming and keyword optimization, others provide illegal, banned or discouraged services that include installing an app on many devices and keeping it installed for prolonged intervals, a.k.a., retention installs , and writing fake reviews with high ratings.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] [25] 2021. What is App Store Optimization? Ultimate Guide to ASO in 2021. App Radar, https://appradar.com/academy/aso-basics/what-is-app-store-optimization-aso. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 4.1 Ethical Considerations · p. 5

    The authors did not ask any participant to write reviews for any app, including for the RacketStore app.

    “We did not ask any participant to write reviews for any app, including for the RacketStore app.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation
    § 3 MEASUREMENTS INFRASTRUCTURE · p. 3

    The RacketStore app asks for explicit consent of the privacy policy and then shows an in-app disclosure of the data being collected.

    “Then, to comply with the Google anti-abuse policy [23], the app asks for explicit consent of our privacy policy (Figure 18(a) in Appendix C) then shows an in-app disclosure of the data being collected (Figure 18(b)).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23] [23] 2020. Privacy, Security, and Deception. Developer Policy Center, https://play. google.com/about/privacy-security-deception/user-data/. -

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 4.1 Ethical Considerations · p. 5

    The consent form explicitly mentions the identity of the researchers, the research objectives, the data to be collected, and the potential impacts on participants, including risks.

    “The consent form explicitly mentions the identity of the researchers, the research objectives, the data that we wanted to collect, and the potential impacts on participants, including risks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 18

    We used GDPR recommended pseudonymisation for data processing and statistics, and other generally accepted good practices for privacy preservation.

    “We used GDPR [63] recommended pseudonymisation for data processing and statistics, and other generally accepted good practices for privacy preservation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [63] [63] TE Parliament. 2016. Regulation (eu) 2016/679 of the european parliament and of the council. Official Journal of the European Union (2016). -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § 6.3 Installed Apps · p. 8

    Differences in app churn between workers and regular participants could also be due to background differences: ASO workers may be more technically skilled due to the nature of their work.

    “We note that the differences in app churn between workers and regular participants could also be due to background differences: ASO workers may be more technically skilled due to the nature of their work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 18

    We only generated aggregated statistics.

    “We also only generated aggregated statistics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 18

    At the completion of the study, we deleted all PII.

    “At the completion of the study, we deleted all PII.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 4.1 Ethical Considerations · p. 5

    Some ASO work is considered unethical according to several ethical frameworks.

    “Some ASO work is considered unethical according to several ethical frameworks, and many ASO workers belong to low-paid vulnerable groups.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 5 DATA · p. 6

    The authors notified study participants about the data collection.

    “We notified study participants about this data collection ( § 4.1).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 3 MEASUREMENTS INFRASTRUCTURE · p. 4

    Upon file reception, the server returns the crypto hash of the received data so the mobile app can validate the transfer with its own hash calculation; if the hashes are equal, the data buffer module deletes the file.

    “To enable resilient communications, upon file reception, the server returns the crypto hash of the received data in order for the mobile app to validate the transfer with its own hash calculation. If the hashes are equal, the data buffer module deletes the file.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 6.4 Investigation of Malware and Attitudes · p. 10

    This suggests potential vulnerabilities and concerns among workers toward keeping apps installed for longer intervals.

    “This suggests potential vulnerabilities and concerns among workers toward keeping apps installed for longer intervals.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 6.3 Installed Apps · p. 9

    The overlap may be due to several worker devices being organic.

    “Nevertheless, we also observe substantial overlap in these features between regular and worker devices, perhaps due to the fact that several of the worker devices are organic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 3 MEASUREMENTS INFRASTRUCTURE · p. 4

    The internal dashboard allows researchers to monitor the data collection process and test and validate the data sent from the app to the server.

    “The internal dashboard allows researchers to monitor the data collection process, and test and validate the data sent from the app to the server.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 6.3 Installed Apps · p. 9

    The higher stopped apps happens due to retention install requirements: workers need to keep promoted apps installed, but want to avoid clutter.

    “We conjecture that this happens due to retention install requirements: workers need to keep promoted apps installed, but want to avoid the clutter.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 18

    No PII of our participants was disclosed outside of the research team.

    “No PII of our participants was disclosed outside of the research team.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 6.3 Installed Apps · p. 8

    The longer waiting time of regular users is consistent with review activity that proceeds from a previous interaction to form a judgment, and inconsistent with paid promotion services.

    “This longer waiting time is consistent with a review activity that proceeds from a previous interaction to form a judgment, and is inconsistent with paid promotion services.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 6.3 Installed Apps · p. 8

    The higher number of stopped apps on worker devices occurs because ASO workers often do not open apps they install to promote, and/or prefer to stop misbehaving apps they need to keep installed for retention.

    “We conjecture that this occurs because ASO workers (1) often do not open the apps that they install in order to promote, and (2) even if they open them and need to keep them installed, e.g., to provide retention installs ( § 2), they prefer to stop apps that misbehave.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 4.1 Ethical Considerations · p. 5

    Each participant needed to explicitly provide consent.

    “Each participant needed to explicitly provide consent.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 18

    The job of our participants was not illegal or unethical according to their own law of the land.

    “Hence, the job of our participants was not illegal or unethical according to their own law of the land.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: ASO work has never been addressed by law enforcing agencies.
    • Rests on an assumption: A similar legal situation is present in many other countries in the Global South including India and Pakistan.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 18

    We deleted all accounts, device IDs and IP addresses after use.

    “We have deleted all accounts, device IDs and IP addresses after use.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 19

    The compensation of the participants was determined according to the fair market rate.

    “The compensation of the participants was determined according to the fair market rate.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 19

    The compensation rate was not so low that participants were exploited, and not so high that they were coerced.

    “We made sure that the rate is not so low that the participants were exploited, and also not so high that they were coerced.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 18

    We deleted participant e-mail addresses after the study.

    “We have deleted participant e-mail addresses after the study.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 19

    The use of deception is not required in studies such as the ones conducted in this work when recruiting participants from countries in the Global South.

    “These arguments further suggest that the use of deception is not required in studies such as the ones we conducted in this work, when recruiting participants from countries in the Global South.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Most local citizens do not understand the technical details of ASO, making it hard for them to judge the work.

  • May combine two claims.

    Interpretationreasoning
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 19

    While in many parts of the West ASO work might occur as a crime, that is not true in countries like Bangladesh, India, Pakistan, or Vietnam.

    “While in many parts of the West, ASO work might occur as a crime, a job that needs to be hidden, that is not true in countries like Bangladesh, India, Pakistan, or Vietnam.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 19

    We disclosed and explained the possibility of impact on profession to participants, and did not hear any concern from any of them.

    “Nonetheless, we disclosed and explained this possibility to our participants, and we did not hear any concern from any of them.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § D ADDITIONAL ETHICAL CONSIDERATIONS · p. 19

    HCI scholars in post-colonial computing argue that many ideas western scholars hold about computer use in non-western contexts are often biased by their own experiences in the West.

    “HCI scholars in post-colonial computing argue that many ideas that western scholars hold around how computers are used in non-western contexts are often biased by their own experiences in the West [47].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [47] [47] Lilly Irani, Janet Vertesi, Paul Dourish, Kavita Philip, and Rebecca E. Grinter. 2010. Postcolonial Computing: A Lens on Design and Development. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10) . 1311-1320. - source
96 of 96

  • The paper relies on this without support.

    Participants recruited from Facebook groups dedicated to product promotion are ASO workers.

    “We recruited ASO workers from Facebook groups that we found to be dedicated to product promotion (see § 2).”
    “We asked them to reply if they were interested and also to answer screening questions, i.e., confirm that they have posted paid reviews, specify how many devices and accounts they have and on how many devices they can install RacketStore.”

  • The paper relies on this without support.

    Participants recruited through Instagram who did not claim to have written paid reviews and did not claim to be administrators are regular users.

    “Of the consenting visitors, we have filtered out those who claimed to have written paid reviews in Google Play (question 1 in the recruitment message, see Appendix B) and who claimed to be administrators (question 6).”

  • The paper relies on this without support.

    Reviews posted from accounts registered on worker-controlled devices are ASO or fake reviews.

    “We also collected 217,041 reviews posted by the 10,310 Gmail addresses registered on the worker-controlled devices of the participants in our studies, with participant consent ( § 4.1).”

  • The paper relies on this without support.

    The recruited sample of ASO workers and regular users is representative of ASO workers and regular Android users generally.

    “This suggests that the constraints associated with ASO work provide opportunities to detect even organic workers.”

  • The paper relies on this without support.

    The Android API only retains the last installation time of an app on a device.

    “However the Android API only retains the last installation time of an app in a device.”

  • The paper relies on this without support.

    Reviews with negative install-to-review times are the result of a past install.

    “We have not considered reviews whose install-to-review times were negative, since they are the result of a past install.”

  • The paper relies on this without support.

    A freshly installed app is in a stopped state until the user opens it for the first time.

    “A freshly installed app is in a stopped state until the user opens it for the first time.”

  • The paper relies on this without support.

    Android devices allow users to stop apps instead of uninstalling them.

    “Android devices also allow users to stop apps, instead of uninstalling them.”

  • The paper relies on this without support.

    A single VirusTotal flag may be a false positive.

    “Since a single VT flag may be a false positive”

  • The paper relies on this without support.

    IP-based geolocation is unreliable.

    “according to the unreliable IP-based geolocation”

  • The paper relies on this without support.

    ASO work has never been addressed by law enforcing agencies.

    “However, ASO work has never been addressed by law enforcing agencies.”

  • The paper relies on this without support.

    A similar legal situation is present in many other countries in the Global South including India and Pakistan.

    “A similar situation is also present in many other countries in the Global South including India and Pakistan.”

  • The paper relies on this without support.

    Most local citizens do not understand the technical details of ASO, making it hard for them to judge the work.

    “Most local citizens do not understand the technical details of ASO, making it hard for them to judge the work”
13 of 13

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reported overall maximum is below the regular-device maximum

    Minor

    The regular-device maximum is reported as 63,452 snapshots per day, but the subsequent overall maximum is only 55,281.38.

  • Maximum worker install-to-review time conflicts with cases over 1,000 days

    Minor

    A reported maximum waiting time of 574 days cannot coexist with four reported waiting times exceeding 1,000 days.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 16

  1. 2005. Directive 2005/29/EC of the European Parliament and of the Council. Official Journal of the European Union https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2005:149:0022:0039:en:PDF.
  2. 2018. Freedom on the Net, Bangladesh. Freedom House, https://freedomhouse.org/report/freedom-net/2018/bangladesh.
  3. 2020. Mobile Application Market Size, and Trends Analysis Report By Store Type (Google Store, Apple Store), By Application (Gaming, Music and Entertainment, Health and Fitness), By Region, And Segment Forecasts, 2020 - 2027. https://www.grandviewresearch.com/industry-analysis/mobile-application-market.
  4. 2021. Mobile App Download and Usage Statistics (2021). BuildFire, https://buildfire.com/app-statistics/.
  5. 2021. What is App Store Optimization? Ultimate Guide to ASO in 2021. App Radar, https://appradar.com/academy/aso-basics/what-is-app-store-optimization-aso.
  6. James A Muir and PC Van Oorschot. 2006. Internet geolocation and evasion. Ottawa: School of Computer Science, Carleton University (2006).
  7. Arjun Mukherjee Vivek Venkataraman Bing Liu and Natalie Glance. 2013. What Yelp Fake Review Filter Might Be Doing? https://www.aaai.org/ocs/index.php/ICWSM/ICWSM13/paper/view/6006
  8. Dario Nisi, Antonio Bianchi, and Yanick Fratantonio. 2019. Exploring Syscall-Based Semantics Reconstruction of Android Applications. In 22nd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2019). USENIX Association, Chaoyang District, Beijing, 517--531. https://www.usenix.org/conference/raid2019/presentation/nisi
  9. TE Parliament. 2016. Regulation (eu) 2016/679 of the european parliament and of the council. Official Journal of the European Union (2016).
  10. Joel Reardon, Álvaro Feal, Primal Wijesekera, Amit Elazari Bar On, Narseo Vallina-Rodriguez, and Serge Egelman. 2019. 50 Ways to Leak Your Data: An Exploration of Apps' Circumvention of the Android Permissions System. In 28th USENIX Security Symposium (USENIX Security 19). USENIX Association, Santa Clara, CA, 603--620. https://www.usenix.org/conference/usenixsecurity19/presentation/reardon
  11. Gianluca Stringhini, Pierre Mourlanne, Gregoire Jacob, Manuel Egele, Christopher Kruegel, and Giovanni Vigna. 2015. EVILCOHORT: Detecting Communities of Malicious Accounts on Online Services. In 24th USENIX Security Symposium (USENIX Security 15). USENIX Association, Washington, D.C., 563--578. https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/stringhini
  12. Kurt Thomas, Damon McCoy, Chris Grier, Alek Kolcz, and Vern Paxson. 2013. Trafficking Fraudulent Accounts: The Role of the Underground Market in Twitter Spam and Abuse. In Proceedings of the 22nd USENIX Conference on Security (SEC'13). USENIX Association, USA, 195--210.
  13. Virus Total. 2012. Virustotal-free online virus, malware and url scanner. Online: https://www.virustotal.com/en (2012).
  14. Xu, Teng and Goossen, Gerard and Cevahir, Huseyin Kerem and Khodeir, Sara and Jin, Yingyezhe and Li, Frank and Shan, Shawn and Patel, Sagar and Freeman, David and Pearce, Paul. 2021. Deep Entity Classification: Abusive Account Detection for Online Social Networks. In Usenix Security Symposium.
  15. Junting Ye, Santhosh Kumar, and Leman Akoglu. 2016. Temporal Opinion Spam Detection by Multivariate Indicative Signals. In Proceedings of the Tenth International Conference on Web and Social Media, Cologne, Germany, May 17-20, 2016. 743--746.
  16. Yajin Zhou, Zhi Wang, Wu Zhou, and Xuxian Jiang. 2012. Hey, you, get off of my market: detecting malicious apps in official and alternative android markets. In Network and Distributed System Security Symposium, NDSS, Vol. 25. 50--52.

Cited by

Sorted by citations

Related

Authors
Hernandez, N., Recabarren, R., Carbunar, B., & Ahmed, S. I.
Affiliations
  • Florida International University (US)
  • FIU
  • University of Toronto (CA)
ORCID
Ahmed, Syed Ishtiaque
Venue
Proceedings of the 21st ACM Internet Measurement Conference
Publisher
ACM
Published
2021-11-02
Pages
639-657
DOI
10.1145/3487552.3487837
OpenAlex
W4288072512
Cited by
3
References
52
Citations by year
2026: 1 · 2024: 1 · 2023: 1
FWCI
0.44
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor, 365 d)
Locations
  • ACM Internet Measurement Conference (IMC) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (CNS-2013671, CNS-2114911)
  • CRDF Global (G-202105-67826)
Topics
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
  • Physical Sciences › Computer Science › Information Systems › Cybercrime and Law Enforcement Studies
  • Physical Sciences › Computer Science › Signal Processing › Advanced Malware Detection Techniques
Keywords
app store optimization, fake reviews, Google Play Store, fraud detection, detection evasion, privacy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review