CoChat
Sign Up
Cited by 39Hand Gesture Recognition SystemsHearing Impairment and CommunicationHuman Pose and Action Recognition

ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition

Open access
Desai, Aashaka, Berger, Lauren, Minakov, Fyodor, Milano, Nessa, Singh, Chinmay, Pumphrey, Kriston, Ladner, Richard, III, Hal Daumé +3 more·2023·Advances in Neural Information Processing Systems 36
DOI 10.52202/075280-3360
Open this paper in CoChat — free accountOpen PDF

Abstract

Sign languages are used as a primary language by approximately 70 million D/deaf people world-wide. However, most communication technologies operate in spoken and written languages, creating inequities in access. To help tackle this problem, we release ASL Citizen, the first crowdsourced Isolated Sign Language Recognition (ISLR) dataset, collected with consent and containing 83,399 videos for 2,731 distinct signs filmed by 52 signers in a variety of environments. We propose that this dataset be used for sign language dictionary retrieval for American Sign Language (ASL), where a user demonstrates a sign to their webcam to retrieve matching signs from a dictionary. We show that training supervised machine learning classifiers with our dataset advances the state-of-the-art on metrics relevant for dictionary retrieval, achieving 63% accuracy and a recall-at-10 of 91%, evaluated entirely on videos of users who are not present in the training or validation sets. An accessible PDF of this article is available at the following link: https://aashakadesai.github.io/research/ASLCitizen_arxiv_updated.pdf

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/075280-3360

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In disability contexts, key datasets include ORBIT, StammerTalk, ASL Citizen, and the grassroots Disabled And Here stock image collection.

    “In disability contexts, key datasets include ORBIT [80] -videos and images of objects recorded by people who are blind or low-vision; StammerTalk [74] and ASL Citizen [32] for speech and language impairments; and the grassroots Disabled And Here stock image collection celebrating disabled Black, Indigenous, People of Colour (BIPOC) [6].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [32]

    The Introduction identifies ASL Citizen as a crowdsourced dataset of isolated ASL signs, supporting its inclusion as a dataset in a disability-related list. But it describes sign language as central to Deaf culture and identity, not as a dataset for speech or language impairments; that classification is unsupported for ASL Citizen. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • The inherent social nature of data production and the enjoyment from working 'together' towards a shared goal echoes findings from other community-oriented AI dataset specification and curation work.

    “The inherent social nature of data production and the enjoyment from working 'together' towards a shared goal, echoes findings from other community-oriented AI dataset specification and curation work [32, 99, 107].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [32]

    Section 3.1 (Data Collection; page unavailable) describes contributors making videos for a communal dictionary, and Section 6 (Discussion and Conclusion; page unavailable) describes a participatory approach. These support the community-oriented dataset aspect, but the passages do not report that contributors enjoyed working together or experienced data production as inherently social. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Academic scholars and industry corporations have emphasized the pressing need for more disability-centered datasets.

    “Although academic scholars and industry corporations have emphasized the pressing need for more disability-centered datasets [20, 38, 55, 79, 97, 111, 124]”

    From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [38]

    The Introduction says existing sign-language datasets lack scale and diversity and describes calls by disability scholars for collaboration with Deaf communities and attention to real needs. This supports a need for better Deaf-community-centered data, but the paper does not report a call from industry corporations or establish the broader claim about disability-centered datasets generally. The cited passages have section labels but no page numbers. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 39

Research works printed but not linked: 51

  1. IA Adeyanju, OO Bello, and MA Adegboye. Machine learning methods for sign language recognition: A critical review and analysis. Intelligent Systems with Applications, 12:200056, 2021.
  2. Oliver Alonzo, Abraham Glasser, and Matt Huenerfauth. Effect of automatic sign recognition performance on the usability of video-based search interfaces for sign language dictionaries. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility, pages 56-67, 2019.
  3. Vassilis Athitsos, Carol Neidle, Stan Sclaroff, Joan Nash, Alexandra Stefan, Quan Yuan, and Ashwin Thangali. The american sign language lexicon video dataset. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 1-8. IEEE, 2008.
  4. Matyáš Boháček and Marek Hrúz. Sign pose-based transformer for word-level sign language recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 182-191, 2022.
  5. Carl Börstell. Ableist language teching over sign language research. In Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023), pages 1-10, Tórshavn, the Faroe Islands, May 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.resourceful-1.1.
  6. Danielle Bragg, Kyle Rector, and Richard E Ladner. A user-powered american sign language dictionary. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pages 1837-1848, 2015.
  7. Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, et al. Sign language recognition, generation, and translation: An interdisciplinary perspective. In The 21st international ACM SIGACCESS conference on computers and accessibility, pages 16-31, 2019.
  8. Danielle Bragg, Oscar Koller, Naomi Caselli, and William Thies. Exploring collection of sign language datasets: Privacy, participation, and model performance. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, pages 1-14, 2020.
  9. Danielle Bragg, Naomi Caselli, Julie A Hochgesang, Matt Huenerfauth, Leah Katz-Hernandez, Oscar Koller, Raja Kushalnagar, Christian Vogler, and Richard E Ladner. The fate landscape of sign language ai datasets: An interdisciplinary perspective. ACM Transactions on Accessible Computing (TACCESS), 14(2): 1-45, 2021.
  10. Danielle Bragg, Abraham Glasser, Fyodor Minakov, Naomi Caselli, and William Thies. Exploring collection of sign language videos through crowdsourcing. CSCW 2022, 2022.
  11. Diane Brentari and Carol A Padden. Native and foreign vocabulary in american sign language: A lexicon with multiple origins. In Foreign vocabulary in sign languages, pages 87-119. Psychology Press, 2001.
  12. Diane Brentari et al. A prosodic model of sign language phonology. Mit Press, 1998.
  13. Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291-7299, 2017.
  14. Josep Maria Carmona and Joan Climent. A performance evaluation of hmm and dtw for gesture recognition. In Iberoamerican Congress on Pattern Recognition, pages 236-243. Springer, 2012.
  15. Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
  16. Naomi K Caselli, Zed Sevcikova Sehyr, Ariel M Cohen-Goldberg, and Karen Emmorey. Asl-lex: A lexical database of american sign language. Behavior research methods, 49(2):784-801, 2017.
  17. Ming Jin Cheok, Zaid Omar, and Mohamed Hisham Jaward. A review of hand gesture and sign language recognition techniques. International Journal of Machine Learning and Cybernetics, 10(1):131-153, 2019.
  18. Helen Cooper, Brian Holt, and Richard Bowden. Sign language recognition. In Visual analysis of humans, pages 539-562. Springer, 2011.
  19. CVPR 2017 Workshop on Visual Understanding Across Modalities. Charades challenge: Recognize and locate activities taking place in a video, 2017. URL http://vuchallenge.org/charades.html.
  20. Mathieu De Coster, Mieke Van Herreweghe, and Joni Dambre. Sign language recognition with transformer networks. In 12th international conference on language resources and evaluation, pages 6018-6024. European Language Resources Association (ELRA), 2020.
  21. Maartje De Meulder. Is "good enough" good enough? ethical and responsible development of sign language technologies. In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), pages 12-22, 2021.
  22. Adil Er-Rady, Rdouan Faizi, R Oulad Haj Thami, and H Housni. Automatic sign language recognition: A survey. In 2017 International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), pages 1-7. IEEE, 2017.
  23. Jordan Fenlon, Kearsy Cormier, and Adam Schembri. Building bsl signbank: The lemma dilemma revisited. International Journal of Lexicography, 28(2):169-206, 2015.
  24. Susan D Fischer and Harry Van der Hulst. Sign language structures. Deaf studies language and education, pages 319-331, 2003.
  25. Jens Forster, Christian Oberdörfer, Oscar Koller, and Hermann Ney. Modality combination techniques for continuous sign language recognition. In Iberian conference on pattern recognition and image analysis, pages 89-99. Springer, 2013.
  26. Dhananjai Hariharan, Sedeeq Al-khazraji, and Matt Huenerfauth. Evaluation of an english word look-up tool for web-browsing with sign language video for deaf readers. In International Conference on Universal Access in Human-Computer Interaction, pages 205-215. Springer, 2018.
  27. Raychelle Harris, Heidi M Holmes, and Donna M Mertens. Research ethics in sign language communities. Sign Language Studies, 9(2):104-131, 2009.
  28. Saad Hassan, Oliver Alonzo, Abraham Glasser, and Matt Huenerfauth. Effect of sign-recognition perfor- mance on the usability of sign-language dictionary search. ACM Transactions on Accessible Computing (TACCESS), 14(4):1-33, 2021.
  29. Saad Hassan, Akhter Al Amin, Alexis Gordon, Sooyeon Lee, and Matt Huenerfauth. Design and evaluation of hybrid search for american sign language to english dictionaries: Making the most of imperfect sign recognition. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI '22, New York, NY, USA, 2022. Association for Computing Machinery. doi: 10.1145/3491102.3501986. URL https://doi.org/10.1145/3491102.3501986. · DOI · Paper page
  30. Boban Joksimoski, Eftim Zdravevski, Petre Lameski, Ivan Miguel Pires, Francisco José Melero, Tomás Puebla Martinez, Nuno M Garcia, Martin Mihajlov, Ivan Chorbev, and Vladimir Trajkovik. Tech- nological solutions for sign language recognition: a scoping review of research trends, challenges, and opportunities. IEEE Access, 2022.
  31. Hamid Reza Vaezi Joze and Oscar Koller. Ms-asl: A large-scale data set and benchmark for understanding american sign language. arXiv preprint arXiv:1812.01053, 2018. · arXiv
  32. Oscar Koller. Quantitative survey of the state of the art in sign language recognition. arXiv preprint arXiv:2008.09918, 2020. · arXiv
  33. Annelies Kusters, Maartje De Meulder, and Dai O'Brien. Innovations in deaf studies: The role of deaf scholars. Oxford University Press, 2017.
  34. Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1459-1469, 2020.
  35. Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172, 2019. · arXiv
  36. Carolyn McCaskill, Ceil Lucas, Robert Bayley, and Joseph Christopher Hill. The hidden treasure of Black ASL: Its history and structure. Gallaudet University Press Washington, DC, 2011.
  37. Caio DD Monteiro, Christy Maria Mathew, Ricardo Gutierrez-Osuna, and Frank Shipman. Detecting and identifying sign languages through visual features. In 2016 IEEE International Symposium on Multimedia (ISM), pages 287-290. IEEE, 2016.
  38. Eng-Jon Ong and Richard Bowden. A boosted classifier tree for hand shape detection. In Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings., pages 889-894. IEEE, 2004.
  39. G Anantha Rao, K Syamala, PVV Kishore, and ASCS Sastry. Deep convolutional neural networks for sign language recognition. In 2018 Conference on Signal Processing And Communication Engineering Systems (SPACES), pages 194-197. IEEE, 2018.
  40. Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018. · arXiv
  41. Zed Sevcikova Sehyr, Naomi Caselli, Ariel M Cohen-Goldberg, and Karen Emmorey. The asl-lex 2.0 project: A database of lexical and phonological properties for 2,723 signs in american sign language. The Journal of Deaf Studies and Deaf Education, 26(2):263-277, 2021.
  42. Prem Selvaraj, Gokul Nc, Pratyush Kumar, and Mitesh Khapra. Openhands: Making sign language recog- nition accessible with pose-based pretrained models across languages. arXiv preprint arXiv:2110.05877, 2021. · arXiv
  43. MF Tolba and AS Elons. Recent developments in sign language recognition systems. In 2013 8th International Conference on Computer Engineering & Systems (ICCES), pages xxxvi-xlii. IEEE, 2013.
  44. Christian Vogler and Dimitris Metaxas. Adapting hidden markov models for asl recognition by using three-dimensional computer vision methods. In 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, volume 1, pages 156-161. IEEE, 1997.
  45. Ankita Wadhawan and Parteek Kumar. Sign language recognition systems: A decade systematic literature review. Archives of Computational Methods in Engineering, 28(3):785-813, 2021.
  46. Ronnie B Wilbur. Phonological and prosodic layering of nonmanuals in american sign language. In The signs of language revisited, pages 196-220. Psychology Press, 2013.
  47. Chenchen Xu, Dongxu Li, Hongdong Li, Hanna Suominen, and Ben Swift. Automatic gloss dictionary for sign language learners. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics: System Demonstrations, pages 83-92, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-demo.8. URL https://aclanthology.org/2022.acl-demo.8. · DOI
  48. Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In Thirty-second AAAI conference on artificial intelligence, 2018.
  49. Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. Including signed languages in natural language processing. arXiv preprint arXiv:2105.05222, 2021. · arXiv
  50. Morteza Zahedi, Daniel Keysers, Thomas Deselaers, and Hermann Ney. Combination of tangent distance and an image distortion model for appearance-based sign language recognition. In Joint Pattern Recognition Symposium, pages 401-408. Springer, 2005.
  51. Inge Zwitserlood. Classifiers. In Sign language: An international handbook. De Gruyter, 2012.

Authors
Desai, A., Berger, L., Minakov, F., Milano, N., Singh, C., Pumphrey, K., Ladner, R., III, H. D., Lu, A. X., Caselli, N., & Bragg, D.
ORCID
Venue
Advances in Neural Information Processing Systems 36
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2023
Pages
76893-76907
ISSN
1049-5258 (ISSN-L)
DOI
10.52202/075280-3360
OpenAlex
W7133203375
Cited by
39
References
0
Citations by year
2026: 17 · 2025: 10 · 2024: 11 · 2023: 1
FWCI
9.03
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
  • Social Sciences › Psychology › Developmental and Educational Psychology › Hearing Impairment and Communication
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
Keywords
American Sign Language, isolated sign recognition, sign language recognition, sign language transcription

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review