The DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
I2T: Image Parsing to Text Description
Abstract
In this paper, we present an image parsing to text description (I2T) framework that generates text descriptions of image and video content based on image understanding. The proposed I2T framework follows three steps: 1) input images (or video frames) are decomposed into their constituent visual patterns by an image parsing engine, in a spirit similar to parsing sentences in natural language; 2) the image parsing results are converted into semantic representation in the form of Web ontology language (OWL), which enables seamless integration with general knowledge bases; and 3) a text generation engine converts the results from previous steps into semantically meaningful, human readable, and query-able text reports. The centerpiece of the I2T framework is an and-or graph (AoG) visual knowledge representation, which provides a graphical representation serving as prior knowledge for representing diverse visual patterns and provides top-down hypotheses during the image parsing. The AoG embodies vocabularies of visual elements including primitives, parts, objects, scenes as well as a stochastic image grammar that specifies syntactic relations (i.e., compositional) and semantic relations (e.g., categorical, spatial, temporal, and functional) between these visual elements. Therefore, the AoG is a unified model of both categorical and symbolic representations of visual knowledge. The proposed I2T framework has two objectives. First, we use semiautomatic method to parse images from the Internet in order to build an AoG for visual knowledge representation. Our goal is to make the parsing process more and more automatic using the learned AoG model. Second, we use automatic methods to parse image/video in specific domains and generate text reports that are useful for real-world applications. In the case studies at the end of this paper, we demonstrate two automatic I2T systems: a maritime and urban scene video surveillance system and a real-time automatic driving scene understanding system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
A Computational Approach to Edge Detection
Canny, J. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767851 · Cited by 30,004
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472
Normalized cuts and image segmentation
Shi, J., & Malik, J. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.868688 · Cited by 15,502
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Berners-Lee, T., Hendler, J., & Lassila, O. (2001) · Scientific American · doi:10.1038/scientificamerican0501-34 · Cited by 8,651
Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937655 · Cited by 8,216
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916
Toward principles for the design of ontologies used for knowledge sharing?
Gruber, T. R. (1995) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1995.1081 · Cited by 7,753
Active Shape Models-Their Training and Application
Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995) · Computer Vision and Image Understanding · doi:10.1006/cviu.1995.1004 · Cited by 7,190
Shape matching and object recognition using shape contexts
Belongie, S., Malik, J., & Puzicha, J. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.993558 · Cited by 6,306
Content-based image retrieval at the end of the early years
Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., & Jain, R. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895972 · Cited by 6,056
GrabCut: Interactive foreground extraction using iterated graph cuts
Rother, C., Kolmogorov, V., & Blake, A. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015720 · Cited by 5,703
Recognition-by-components: A theory of human image understanding.
Biederman, I. (1987) · Psychological Review · doi:10.1037/0033-295x.94.2.115 · Cited by 5,621
Harnad, S. (1990) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(90)90087-6 · Cited by 4,235 · cited 2×
“Therefore, as suggested by S. Harnad [12], we may further decompose the symbol grounding problem in the visual domain into two levels: • Categorical representations, which are learned and innate feature-detectors that pick out the invariant features of object and event categories from their sensory projections (images…”
MethodLabelMe: A Database and Web-Based Tool for Image Annotation
Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0090-8 · Cited by 4,210
Head-driven phrase structure grammar
Sag, I. A., & Pollard, C. (1994) · Medical Entomology and Zoology · Cited by 3,889
Barthes, R., & Heath, S. C. (1977) · Medical Entomology and Zoology · Cited by 3,644
Speed of processing in the human visual system
Thorpe, S., Fize, D., & Marlot, C. (1996) · Nature · doi:10.1038/381520a0 · Cited by 3,538
Image retrieval: Ideas, influences, and trends of the new age
Datta, R., Joshi, D., Li, J., & Wang, J. Z. (2008) · ACM Computing Surveys · doi:10.1145/1348246.1348248 · Cited by 3,040
One-shot learning of object categories
Fei-Fei, L., Fergus, R., & Perona, P. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.79 · Cited by 3,029
Nonuniversal critical dynamics in Monte Carlo simulations
Swendsen, R. H., & Wang, J.-S. (1987) · Physical Review Letters · doi:10.1103/physrevlett.58.86 · Cited by 2,844
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
(2007) · CaltechAUTHORS (California Institute of Technology) · Cited by 2,274
Pictorial Structures for Object Recognition
Felzenszwalb, P. F., & Huttenlocher, D. P. (2005) · International Journal of Computer Vision · doi:10.1023/b:visi.0000042934.15159.49 · Cited by 2,180
Textons, the elements of texture perception, and their interactions
Julesz, B. (1981) · Nature · doi:10.1038/290091a0 · Cited by 1,703 · cited 1×
“As defined by Julesz in [62], textons are shape structure frequently observed in natural images, such as ‘‘blob,’’ ‘‘bar,’’ ‘‘parallel lines,’’ and ‘‘junctions.”
Advances in Neural Information Processing Systems 19 (NIPS 2006)
Simonotto, E., Whalley, H. C., Lawrie, S. M., Murray, L. M., McGonigle, D. J., & Storkey, A. (2006) · Cited by 1,651
Image Retrieval: Current Techniques, Promising Directions, and Open Issues
Rui, Y., Huang, T. S., & Chang, S.-F. (1999) · Journal of Visual Communication and Image Representation · doi:10.1006/jvci.1999.0413 · Cited by 1,601
Content-based multimedia information retrieval: State of the art and challenges
Lew, M. S., Sebe, N., Djeraba, C., & Jain, R. (2006) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/1126004.1126005 · Cited by 1,528
Representing and Recognizing the Visual Appearance of Materials Using Three-Dimensional Textons
Leung, T., & Malik, J. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011126920638 · Cited by 1,498 · cited 1×
“3 illustrates a rough but intuitive idea about two major tasks of the categorical representation: • Task1: Extract low-level visual features such as coherent regions (segmentation) [13]–[15], salient edges [16], [17], filter-bank responses (textons) [18], [19], SIFT descriptors [20], color histograms Fig.”
BackgroundShotton, J., Winn, J., Rother, C., & Criminisi, A. (2006) · Lecture Notes in Computer Science · doi:10.1007/11744023_1 · Cited by 1,132
Syntactic pattern recognition and applications
Fu, K. (1968) · Medical Entomology and Zoology · Cited by 1,015
Efficient color histogram indexing for quadratic form distance functions
Hafner, J., Sawhney, H. S., Equitz, W., Flickner, M., & Niblack, W. (1995) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.391417 · Cited by 833
Multiscale conditional random fields for image labeling
He, X., Zemel, R. S., & Carreira-Perpinan, M. A. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315232 · Cited by 828
Putting Objects in Perspective
Hoiem, D., Efros, A. A., & Hebert, M. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-008-0137-5 · Cited by 805
Syntactic pattern recognition and applications
Firschein, O. (1983) · Proceedings of the IEEE · doi:10.1109/proc.1983.12755 · Cited by 753
Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling
Zhu, S. C., Wu, Y., & Mumford, D. (1998) · International Journal of Computer Vision · doi:10.1023/a:1007925832420 · Cited by 652
Self-citationAdvances in Neural Information Processing Systems 15 (NIPS 2002)
Williams, C. K. I., & Shawe-Taylor, J. S. (2002) · The MIT Press eBooks · Cited by 620
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244303322533214 · Cited by 615
Visual features of intermediate complexity and their use in classification
Ullman, S., Vidal-Naquet, M., & Sali, E. (2002) · Nature Neuroscience · doi:10.1038/nn870 · Cited by 600
Image segmentation by data-driven markov chain monte carlo
Tu, Z., & Zhu, S.-C. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.1000239 · Cited by 565
Self-citationReal-Time Computerized Annotation of Pictures
Li, J., & Wang, J. Z. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.70847 · Cited by 506
Image Parsing: Unifying Segmentation, Detection, and Recognition
Tu, Z., Chen, X., Yuille, A. L., & Zhu, S.-C. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-6642-x · Cited by 498
Self-citationMultimodal Video Indexing: A Review of the State-of-the-art
Snoek, C. G. M., & Worring, M. (2005) · Multimedia Tools and Applications · doi:10.1023/b:mtap.0000046380.27575.a5 · Cited by 486
Li, L.-J., Socher, R., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206718 · Cited by 484
Fast approximate energy minimization via graph cuts
Boykov, Y., Veksler, O., & Zabih, R. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.791245 · Cited by 470
A Stochastic Grammar of Images
Zhu, S.-C., & Mumford, D. (2006) · now publishers, Inc. eBooks · doi:10.1561/9781601980618 · Cited by 395
Self-citationContextual models for object detection using boosted random fields
Torralba, A. B., Murphy, K. P., & Freeman, W. T. (2004) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 338
A Stochastic Grammar of Images
Zhu, S.-C., & Mumford, D. (2006) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000018 · Cited by 332
Self-citationLearning hierarchical models of scenes, objects, and parts
Sudderth, E. B., Torralba, A., Freeman, W. T., & Willsky, A. S. (2005) · Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 · doi:10.1109/iccv.2005.137 · Cited by 297
Semantic Hierarchies for Visual Object Recognition
Marszalek, M., & Schmid, C. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383272 · Cited by 284
Readings in computer vision: issues, problems, principles, and paradigms
Fischler, M. A., & Firschein, O. (1987) · Morgan Kaufmann Publishers Inc. eBooks · Cited by 245
Unification: a multidisciplinary survey
Knight, K. (1989) · ACM Computing Surveys · doi:10.1145/62029.62030 · Cited by 232
A Component-based Framework for Face Detection and Identification
Heisele, B., Serre, T., & Poggio, T. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-006-0006-z · Cited by 223
Towards automatic discovery of object categories
Weber, M., Welling, M., & Perona, P. (2002) · Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662) · doi:10.1109/cvpr.2000.854754 · Cited by 218
Weakly Supervised Scale-Invariant Learning of Models for Visual Recognition
Fergus, R., Perona, P., & Zisserman, A. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-006-8707-x · Cited by 197
Advances in Neural Information Processing Systems 17 (NIPS 2004)
Allan, M., & Williams, C. K. I. (2005) · Cited by 194
Yao, B., Yang, X., & Zhu, S.-C. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74198-5_14 · Cited by 188
Self-citationGeneralizing Swendsen-Wang to sampling arbitrary posterior probabilities
Barbu, A., & Zhu, S.-C. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.161 · Cited by 181
Self-citationCamera calibration from video of a walking human
Lv, F., Zhao, T., & Nevatia, R. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.178 · Cited by 178
An Ontology for Video Event Representation
Nevatia, R., Hobbs, J. R., & Bolles, B. (2005) · IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvpr.2004.301 · Cited by 145
Composite Templates for Cloth Modeling and Sketching
Chen, H., Xu, Z., Liu, Z. Q., & Zhu, S. C. (2006) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2006.81 · Cited by 138
Self-citationDescribing Visual Scenes using Transformed Dirichlet Processes
Torralba, A. B., Willsky, A. S., Sudderth, E. B., & Freeman, W. T. (2005) · neural information processing systems · Cited by 134
Scene completion using millions of photographs
Hays, J. H., & Efros, A. A. (2008) · Communications of the ACM · doi:10.1145/1400181.1400202 · Cited by 129
Video Event Recognition Using Kernel Methods with Multilevel Temporal Alignment
Xu, D., & Chang, S. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.129 · Cited by 127
Bottom-up/top-down image parsing by attribute graph grammar
Han, F., & Zhu, S. C. (2005) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2005.50 · Cited by 127
Self-citationPrimal sketch: Integrating structure and texture
Guo, C.-E., Zhu, S. C., & Wu, Y. N. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2005.09.004 · Cited by 114
Self-citationGraph partition by Swendsen-Wang cuts
Barbu, & Zhu (2003) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2003.1238362 · Cited by 106
Ontological inference for image and video analysis
Town, C. P. (2006) · Machine Vision and Applications · doi:10.1007/s00138-006-0017-3 · Cited by 86
A Hierarchical and Contextual Model for Aerial Image Parsing
Porway, J., Wang, Q., & Zhu, S. C. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-009-0306-1 · Cited by 74
Self-citationA Numerical Study of the Bottom-Up and Top-Down Inference Processes in And-Or Graphs
Wu, T., & Zhu, S. C. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-010-0346-6 · Cited by 64
Self-citationDeformable Template As Active Basis
Wu, Y., Si, Z., Fleming, C., & Zhu, S. C. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408980 · Cited by 50
Self-citationHierarchical Semantics of Objects (hSOs)
Parikh, D., & Chen, T. (2007) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408960 · Cited by 22
Layered Graph Match with Graph Editing
Lin, L., Zhu, S. C., & Wang, Y. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383190 · Cited by 21
Self-citationMapping Natural Image Patches by Explicit and Implicit Manifolds
Shi, K., & Zhu, S. C. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.382980 · Cited by 15
Self-citation
Cited by
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Metadata
- Authors
- Yao, B. Z., Yang, X., Lin, L., Lee, M. W., & Zhu, S.-C.
- Venue
- Proceedings of the IEEE
- Published
- 2010-08
- Volume
- 98
- Issue
- 8
- Pages
- 1485-1508
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W1987835821
- Cited by
- 310
- References
- 78
- FWCI
- 24.32
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.