CoChat
Sign Up
Cited by 248Categorization, perception, and languageRobot Manipulation and LearningSpeech and dialogue systems

A Joint Model of Language and Perception for Grounded Attribute Learning

Open access
Matuszek, Cynthia, FitzGerald, Nicholas, Zettlemoyer, Luke, Bo, Liefeng, Fox, Dieter·2012·Maryland Shared Open Access Repository
DOI 10.13016/m2ehu6-1m90
Open this paper in CoChat — free accountOpen PDF

Abstract

As robots become more ubiquitous and capable, it becomes ever more important to enable untrained users to easily interact with them. Recently, this has led to study of the language grounding problem, where the goal is to extract representations of the meanings of natural language tied to perception and actuation in the physical world. In this paper, we present an approach for joint learning of language and perception models for grounded attribute induction. Our perception model includes attribute classifiers, for example to detect object color and shape, and the language model is based on a probabilistic categorial grammar that enables the construction of rich, compositional meaning representations. The approach is evaluated on the task of interpreting sentences that describe sets of objects in a physical workspace. We demonstrate accurate task performance and effective latent-variable concept induction in physical grounded scenes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.13016/m2ehu6-1m90

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LMs are particularly susceptible to hallucination, making it essential to penalize such behavior.

    “LMs are particularly susceptible to hallucination [77,113], making it essential to penalize such behavior.”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [113]

    The Joint Language/Perception Model (p. 4) describes a semantic parser and visual classifiers, while Model Learning (pp. 4–5) describes their training. Searches for hallucination and for penalizing false language-model outputs found no passage addressing either the claimed susceptibility or the need to penalize it. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. Barnard, K., Duygulu, P., Forsyth, D., De Freitas, N., Blei, D.M., and Jordan, M.I. Matching words and pictures. The Journal of Machine Learning Research , 3: 1107-1135, 2003.
  2. Bo, L., Ren, X., and Fox, D. Kernel descriptors for visual recognition. In Neural Information Processing Systems (NIPS) , 2010.
  3. Bo, L., Ren, X., and Fox, D. Depth kernel descriptors for object recognition. In IEEE/RSJ Int'l Conf. on Intelligent Robots and Systems (IROS) , 2011.
  4. Chen, D.L. and Mooney, R.J. Learning to interpret natural language navigation instructions from observations. In Proc. of the 25th AAAI Conf. on Artificial Intelligence (AAAI-2011) , pp. 859-865, August 2011.
  5. Clarke, J., Goldwasser, D., Chang, M., and Roth, D. Driving semantic parsing from the world's response. In Proc. of the Conf. on Computational Natural Language Learning , 2010.
  6. Farhadi, A., Endres, I., Hoiem, D., and Forsyth, D. Describing objects by their attributes. In IEEE Conf. on Computer Vision and Pattern Recognition , 2009.
  7. Goldwasser, D., Reichart, R., Clarke, J., and Roth, D. Confidence driven unsupervised semantic parsing. In Proceedings. of the Association of Computational Linguistics , 2011.
  8. Gorniak, P. and Roy, D. Understanding complex visually referring utterances. In Proc. of the HLT-NAACL 2003 Workshop on Learning Word Meaning from NonLinguistic Data , 2003.
  9. He, Y. and Young, S. Spoken language understanding using the hidden vector state model. Speech Communication , 48(3-4), 2006.
  10. Johnson, A. and Hebert, M. Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence , 21(5), 1999.
  11. Kwiatkowski, T., Zettlemoyer, L.S., Goldwater, S., and Steedman, M. Inducing probabilistic CCG grammars from logical form with higher-order unification. In Proc. of the Conf. on Empirical Methods in Natural Language Processing , 2010.
  12. Kwiatkowski, T., Zettlemoyer, L.S., Goldwater, S., and Steedman, M. Lexical generalization in CCG grammar induction for semantic parsing. In Proc. of the Conf. on Empirical Methods in Natural Language Processing , 2011.
  13. Lee, H., Grosse, R., Ranganath, R., and Ng, A. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proc. of the Int'l Conf. on Machine Learning (ICML) , 2009.
  14. Liang, P., Jordan, M.I., and Klein, D. Learning dependency-based compositional semantics. In Proc. of the Association for Computational Linguistics , 2011.
  15. Matuszek, C., Herbst, E., Zettlemoyer, L., and Fox, D. Learning to parse natural language commands to a robot control system. In Proc. of the 13th Int'l Symposium on Experimental Robotics (ISER) , June 2012.
  16. Mavridis, N. and Roy, D. Grounded situation models for robots: Where words and percepts meet. In IEEE/RSJ Int'l Conf. on Intelligent Robots and Systems , 2006.
  17. Parikh, D. and Grauman, K. Relative attributes. In Int'l Conf. on Computer Vision , 2011.
  18. Quattoni, A., Wang, S., p Morency, L., Collins, M., Darrell, T., and Csail, Mit. Hidden-state conditional random fields. In IEEE Transactions on Pattern Analysis and Machine Intelligence , 2007.
  19. Reckman, H., Orkin, J., and Roy, D. Learning meanings of words and constructions, grounded in a virtual game. In Proc. of the 10th Conf. on Natural Language Processing (KONVENS) , 2010.
  20. Tellex, S., Kollar, T., Dickerson, S., Walter, M.R., Banerjee, A.G., Teller, S., and Roy, N. Understanding natural language commands for robotic navigation and mobile manipulation. In Proc. of the National Conf. on Artificial Intelligence (AAAI) , August 2011.
  21. Wong, Y.W. and Mooney, R.J. Learning synchronous grammars for semantic parsing with lambda calculus. In Proc. of the Ass'n for Computational Linguistics , 2007.
  22. Yang, J., Yu, K., Gong, Y., and Huang, T. Linear spatial pyramid matching using sparse coding for image classification. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2009.
  23. Yu, K. and Zhang, T. Improved local coordinate coding using local tangents. In Proc. of the Int'l Conf. on Machine Learning (ICML) , 2010.
  24. Zelle, J.M. and Mooney, R.J. Learning to parse database queries using inductive logic programming. In Proc. of the National Conf. on Artificial Intelligence , 1996.
  25. Zettlemoyer, L.S. and Collins, M. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proc. of the Conf. on Uncertainty in Artificial Intelligence , 2005.

Cited by

Sorted by citations

Authors
Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D.
Affiliations
University of Washington (US)
ORCID
Venue
Maryland Shared Open Access Repository
Publisher
Maryland Shared Open Access Repository
Published
2012-06-27
Pages
1435-1442
ISSN
2331-8422 (ISSN-L)
DOI
10.13016/m2ehu6-1m90
OpenAlex
W1923162067
Cited by
248
References
2
Citations by year
2025: 1 · 2024: 3 · 2023: 7 · 2022: 4 · 2021: 12 · 2020: 12 · 2019: 21 · 2018: 32 · 2017: 34 · 2016: 26 · 2015: 38 · 2014: 33 · 2013: 23 · 2012: 1
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion)
  • arXiv (Cornell University)
  • Digital Repository at the University of Maryland (University of Maryland College Park)
Funders
  • National Science Foundation (1115966)
  • Army Research Laboratory (W911NF, W911NF-10, W911NF-10-2-0016)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
language grounding, robot interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review