Towards total scene understanding: Classification, annotation and segmentation in an automatic framework
The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
Given an image, we propose a hierarchical generative model that classifies the overall scene, recognizes and segments each object component, as well as annotates the image with a list of tags. To our knowledge, this is the first model that performs all three tasks in one coherent framework. For instance, a scene of a `polo game' consists of several visual objects such as `human', `horse', `grass', etc. In addition, it can be further annotated with a list of more abstract (e.g. `dusk') or visually less salient (e.g. `saddle') tags. Our generative model jointly explains images through a visual model and a textual model. Visually relevant objects are represented by regions and patches, while visually irrelevant textual annotations are influenced directly by the overall scene class. We propose a fully automatic learning framework that is able to learn robust scene models from noisy Web data such as images and user tags from Flickr.com. We demonstrate the effectiveness of our framework by automatically classifying, annotating and segmenting images from eight classes depicting sport scenes. In all three tasks, our model significantly outperforms state-of-the-art algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015
Efficient Graph-Based Image Segmentation
Felzenszwalb, P. F., & Huttenlocher, D. P. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000022288.19776.77 · Cited by 6,176
A Bayesian Hierarchical Model for Learning Natural Scene Categories
Li, F.-F., & Perona, P. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.16 · Cited by 3,613
Markov Chain Sampling Methods for Dirichlet Process Mixture Models
Neal, R. M. (2000) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2000.10474879 · Cited by 2,288
Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary
Duygulu, P., Barnard, K., de Freitas, J. F. G., & Forsyth, D. A. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-47979-1_7 · Cited by 1,608
Automatic linguistic indexing of pictures by a statistical modeling approach
Li, J., & Wang, J. Z. (2003) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2003.1227984 · Cited by 1,087
Blei, D. M., & Jordan, M. I. (2003) · Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval · doi:10.1145/860435.860460 · Cited by 1,054
Putting Objects in Perspective
Hoiem, D., Efros, A. A., & Hebert, M. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-008-0137-5 · Cited by 805
What, where and who? Classifying events by scene and object recognition
Li, L.-J., & Fei-Fei, L. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408872 · Cited by 781
Self-citation
Cited by
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Self-citationYFCC100M: the new data in multimedia research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., & Li, L.-J. (2016) · Communications of the ACM · doi:10.1145/2812802 · Cited by 1,541
Self-citationEvery Picture Tells a Story: Generating Sentences from Images
Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_2 · Cited by 1,204
(2011) · Cited by 1,192
Image retrieval using scene graphs
Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298990 · Cited by 1,101
Self-citationDeep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048
Self-citationReconstructing Visual Experiences from Brain Activity Evoked by Natural Movies
Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011) · Current Biology · doi:10.1016/j.cub.2011.08.031 · Cited by 933
“This finding suggests that a hybrid encoding model that combines the structural motion-energy model developed here with a semantic model of the form developed in previous studies [8, 45, 46] could provide even better reconstructions of subjective mental experiences.”
Camouflaged Object Segmentation with Distraction Mining
Mei, H., Ji, G.-P., Wei, Z., Yang, X., Wei, X., & Fan, D.-P. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00866 · Cited by 565
Computer vision techniques for construction safety and health monitoring
Seo, J., Han, S., Lee, S., & Kim, H. (2015) · Advanced Engineering Informatics · doi:10.1016/j.aei.2015.02.001 · Cited by 552
Context tracker: Exploring supporters and distracters in unconstrained environments
Dinh, T. B., Vo, N., & Medioni, G. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995733 · Cited by 510
What is a good evaluation measure for semantic segmentation?
Csurka, G., Larlus, D., & Perronnin, F. (2013) · Procedings of the British Machine Vision Conference 2013 · doi:10.5244/c.27.32 · Cited by 500
Single image depth estimation from predicted semantic labels
Liu, B., Gould, S., & Koller, D. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539823 · Cited by 477
A Guide to Convolutional Neural Networks for Computer Vision
Khan, S. H., Rahmani, H., Shah, S. A. A., & Bennamoun, M. (2018) · Synthesis lectures on computer vision · doi:10.2200/s00822ed1v01y201712cov015 · Cited by 443
Goldszmidt, M. (2011) · Wiley Encyclopedia of Operations Research and Management Science · doi:10.1002/9780470400531.eorms0099 · Cited by 351
AI Benchmark: Running Deep Neural Networks on Android Smartphones
Ignatov, A., Timofte, R., Chou, W., Wang, K., Wu, M., Hartley, T., & Van Gool, L. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11021-5_19 · Cited by 347
A comprehensive survey of loss functions and metrics in deep learning
Terven, J., Cordova-Esparza, D.-M., Romero-González, J.-A., Ramírez-Pedraza, A., & Chávez-Urbiola, E. A. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11198-7 · Cited by 343
Blei, D., Carin, L., & Dunson, D. (2010) · IEEE Signal Processing Magazine · doi:10.1109/msp.2010.938079 · Cited by 335
I2T: Image Parsing to Text Description
Yao, B. Z., Yang, X., Lin, L., Lee, M. W., & Zhu, S.-C. (2010) · Proceedings of the IEEE · doi:10.1109/jproc.2010.2050411 · Cited by 310
What, Where and How Many? Combining Object Detectors and CRFs
Ladický, Ľ., Sturgess, P., Alahari, K., Russell, C., & Torr, P. H. S. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_31 · Cited by 310
Exploiting hierarchical context on a large database of object categories
Choi, M. J., Lim, J. J., Torralba, A., & Willsky, A. S. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540221 · Cited by 304
What helps where – and why? Semantic relatedness for knowledge transfer
Rohrbach, M., Stark, M., Szarvas, G., Gurevych, I., & Schiele, B. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540121 · Cited by 273
Grand Challenges in Immersive Analytics
Ens, B., Bach, B., Cordeil, M., Engelke, U., Serrano, M., Willett, W., Prouzeau, A., Anthes, C., Büschel, W., Dunne, C., et al. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3446866 · Cited by 218
“In addition, automating the behaviour of data visualisations, providing context-aware behaviour, and automating layouts requires scene understanding [1, 83]—the ability to understand objects in the context of their surroundings, including dynamic behaviours and functional relationships.”
SuRVoS: Super-Region Volume Segmentation workbench
Luengo, I., Darrow, M. C., Spink, M. C., Sun, Y., Dai, W., He, C. Y., Chiu, W., Pridmore, T., Ashton, A. W., Duke, E. M. H., et al. (2017) · Journal of Structural Biology · doi:10.1016/j.jsb.2017.02.007 · Cited by 89
Chen, S., Tian, Y., Liu, Q., & Metaxas, D. N. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.014 · Cited by 72
Sung, J., Jin, S. H., & Saxena, A. (2018) · Springer Proceedings in Advanced Robotics · doi:10.1007/978-3-319-60916-4_40 · Cited by 52
“Thus, rather than relying on scene understanding techniques [7, 31, 15], we”
BackgroundMix-and-Match Tuning for Self-Supervised Semantic Segmentation
Zhan, X., Liu, Z., Luo, P., Tang, X., & Loy, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12331 · Cited by 34
The fundamentals of eye tracking part 6: working with areas of interest
Hooge, I. T. C., Nyström, M., Niehorster, D. C., Andersson, R., Foulsham, T., Nuthmann, A., & Hessels, R. S. (2026) · Behavior Research Methods · doi:10.3758/s13428-025-02937-3 · Cited by 22
Metadata
- Authors
- Li, L.-J., Socher, R., & Fei-Fei, L.
- Venue
- 2009 IEEE Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2009-06
- Pages
- 2036-2043
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2106624428
- Cited by
- 484
- References
- 34
- FWCI
- 43.34
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.