Crowdsourcing Annotations for Websites' Privacy Policies: Can It Really Work?
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Website privacy policies are often long and difficult to understand. While research shows that Internet users care about their privacy, they do not have time to understand the policies of every website they visit, and most users hardly ever read privacy policies. Several recent efforts aim to crowdsource the interpretation of privacy policies and use the resulting annotations to build more effective user interfaces that provide users with salient policy summaries. However, very little attention has been devoted to studying the accuracy and scalability of crowdsourced privacy policy annotations, the types of questions crowdworkers can effectively answer, and the ways in which their productivity can be enhanced. Prior research indicates that most Internet users often have great difficulty understanding privacy policies, suggesting limits to the effectiveness of crowdsourcing approaches. In this paper, we assess the viability of crowdsourcing privacy policy annotations. Our results suggest that, if carefully deployed, crowdsourcing can indeed result in the generation of non-trivial annotations and can also help identify elements of ambiguity in policies. We further introduce and evaluate a method to improve the annotation process by predicting and highlighting paragraphs relevant to specific data practices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Privacy, Trust, and Self-Disclosure Online
Joinson, A. N., Reips, U.-D., Buchanan, T., & Schofield, C. B. P. (2010) · Human–Computer Interaction · doi:10.1080/07370020903586662 · Cited by 503 · cited 1×
“In addition, we were interested in how clearly these practices were described with respect to particularly sensitive information types [1, 12, 14]: contact information, financial infor-”
Background
Cited by
The Creation and Analysis of a Website Privacy Policy Corpus
Wilson, S., Schaub, F., Dara, A. A., Liu, F., Cherivirala, S., Giovanni Leon, P., Schaarup Andersen, M., Zimmeck, S., Sathyendra, K. M., Russell, N. C., et al. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1126 · Cited by 275
Self-citationPositivity Bias in Customer Satisfaction Ratings
Park, K., Cha, M., & Rhim, E. (2018) · Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW '18 · doi:10.1145/3184558.3186579 · Cited by 19
“Future research may rely on alternative methods like crowdsourcing on a larger dataset [41].”
MethodOn the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices
Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479531 · Cited by 4
“In contrast, enhanced interfaces may facilitate the job of crowd workers and aid these to improved performance [59, 60, 65, 72], as well as adequately limiting the time to judge can accelerate task completion without compromising the quality of the results [42].”
Background
Metadata
- Authors
- Wilson, S., Schaub, F., Ramanath, R., Sadeh, N., Liu, F., Smith, N. A., & Liu, F.
- Venue
- Proceedings of the 25th International Conference on World Wide Web
- Published
- 2016-04-11
- Pages
- 133-143
- OpenAlex
- W2339322788
- Cited by
- 80
- References
- 2
- FWCI
- 27.61
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Crowdsourcing Annotations for Websites' Privacy Policies: Can It Really Work? https://app.cochat.ai/papers/10.1145/2872427.2883035
- The paper
- Wilson, S., Schaub, F., Ramanath, R., Sadeh, N., Liu, F., Smith, N. A., & Liu, F. (2016). Crowdsourcing Annotations for Websites' Privacy Policies: Can It Really Work? Proceedings of the 25th International Conference on World Wide Web. https://doi.org/10.1145/2872427.2883035
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.