The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval
Abstract
The problem of cross-modal retrieval from multimedia repositories is considered. This problem addresses the design of retrieval systems that support queries across content modalities, for example, using an image to search for texts. A mathematical formulation is proposed, equating the design of cross-modal retrieval systems to that of isomorphic feature spaces for different content modalities. Two hypotheses are then investigated regarding the fundamental attributes of these spaces. The first is that low-level cross-modal correlations should be accounted for. The second is that the space should enable semantic abstraction. Three new solutions to the cross-modal retrieval problem are then derived from these hypotheses: correlation matching (CM), an unsupervised method which models cross-modal correlations, semantic matching (SM), a supervised technique that relies on semantic representation, and semantic correlation matching (SCM), which combines both. An extensive evaluation of retrieval performance is conducted to test the validity of the hypotheses. All approaches are shown successful for text retrieval in response to image queries and vice versa. It is concluded that both hypotheses hold, in a complementary form, although evidence in favor of the abstraction hypothesis is stronger than that for correlation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Jolliffe, I. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa501 · Cited by 14,310
Indexing by latent semantic analysis
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6<391::aid-asi1>3.0.co;2-9 · Cited by 12,751
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Introduction to Modern Information Retrieval
Salton, G., & McGill, M. J. (1983) · Virtual Defense Library (Ministerio de Defensa) · Cited by 9,680
Kernel Methods for Pattern Analysis
Shawe-Taylor, J., & Cristianini, N. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809682 · Cited by 6,587
Content-based image retrieval at the end of the early years
Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., & Jain, R. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895972 · Cited by 6,056
Introduction to modern information retrieval
Dillon, M. (1983) · Information Processing & Management · doi:10.1016/0306-4573(83)90062-6 · Cited by 5,909
Swain, M. J., & Ballard, D. H. (1991) · International Journal of Computer Vision · doi:10.1007/bf00130487 · Cited by 5,623
Canonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmak, S., & Shawe-Taylor, J. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,202
Image retrieval: Ideas, influences, and trends of the new age
Datta, R., Joshi, D., Li, J., & Wang, J. Z. (2008) · ACM Computing Surveys · doi:10.1145/1348246.1348248 · Cited by 3,040
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/jmlr.2003.3.4-5.993 · Cited by 3,024
Musical genre classification of audio signals
Tzanetakis, G., & Cook, P. (2002) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2002.800560 · Cited by 2,798
The SMART Retrieval System—Experiments in Automatic Document Processing
Salton, G. (1971) · CERN Document Server (European Organization for Nuclear Research) · Cited by 2,252
Relations Between Two Sets of Variates
Hotelling, H. (1936) · Biometrika · doi:10.2307/2333955 · Cited by 1,645
A new approach to cross-modal multimedia retrieval
Rasiwasia, N., Costa Pereira, J., Coviello, E., Doyle, G., Lanckriet, G. R. G., Levy, R., & Vasconcelos, N. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1873987 · Cited by 1,334
Self-citationEvaluation campaigns and TRECVid
Smeaton, A. F., Over, P., & Kraaij, W. (2006) · Proceedings of the 8th ACM international workshop on Multimedia information retrieval · doi:10.1145/1178677.1178722 · Cited by 1,224
Automatic image annotation and retrieval using cross-media relevance models
Jeon, J., Lavrenko, V., & Manmatha, R. (2003) · Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval · doi:10.1145/860435.860459 · Cited by 1,139
Supervised Learning of Semantic Classes for Image Annotation and Retrieval
Carneiro, G., Chan, A. B., Moreno, P. J., & Vasconcelos, N. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.61 · Cited by 859
Self-citationA Model for Learning the Semantics of Pictures
Lavrenko, V. P., Manmatha, R., & Jeon, J. (2003) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 680
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244303322533214 · Cited by 615
Collecting Image Annotations Using Amazon's Mechanical Turk
Rashtchian, C., Young, P., Hodosh, M., & Hockenmaier, J. (2010) · Cited by 541
Learning the semantics of words and pictures
Barnard, K., & Forsyth, D. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937654 · Cited by 524
Multimodal Video Indexing: A Review of the State-of-the-art
Snoek, C. G. M., & Worring, M. (2005) · Multimedia Tools and Applications · doi:10.1023/b:mtap.0000046380.27575.a5 · Cited by 486
Semantic Annotation and Retrieval of Music and Sound Effects
Turnbull, D., Barrington, L., Torres, D., & Lanckriet, G. (2008) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2007.913750 · Cited by 483
Self-citationBarnard, K., Duygulu, P., Forsyth, D. A., de Freitas, N., Blei, D. M., Jordan, M. I., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2003) · Cited by 459
Automatic image annotation and retrieval using cross-media relevance models
Jeon, J., Lavrenko, V. P., & Manmatha, R. (2003) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/860458.860459 · Cited by 417
Text Information Retrieval Systems
Meadow, C. T., Kraft, D. H., & Boyce, B. R. (1992) · Cited by 339
LIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328
A music similarity function based on signal analysis
Logan, B., & Salomon, A. (2001) · IEEE International Conference on Multimedia and Expo, 2001. ICME 2001. · doi:10.1109/icme.2001.1237829 · Cited by 308
WebSeer: An Image Search Engine for the World Wide Web
Frankel, C., Swain, M. J., & Athitsos, V. (1996) · Cited by 303
Multimedia content processing through cross-modal association
Li, D., Dimitrova, N., Li, M., & Sethi, I. K. (2003) · Proceedings of the eleventh ACM international conference on Multimedia · doi:10.1145/957013.957143 · Cited by 265
Bridging the Gap: Query by Semantic Example
Rasiwasia, N., Moreno, P. J., & Vasconcelos, N. (2007) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2007.900138 · Cited by 247
Self-citationInferring a Semantic Representation of Text via Cross-Language Correlation Analysis
Vinokourov, A., Cristianini, N., & Shawe-Taylor, J. S. (2002) · Cited by 225
Yang, Y., Zhuang, Y.-T., Wu, F., & Pan, Y.-H. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.917359 · Cited by 213
Automatic Generation of Social Tags for Music Recommendation
Eck, D., Lamere, P., Bertin-Mahieux, T., & Green, S. (2007) · Cited by 184
Modeling Semantic Aspects for Cross-Media Image Indexing
Monay, F., & Gatica-Perez, D. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1097 · Cited by 183
Ranking with local regression and global alignment for cross media retrieval
Yang, Y., Xu, D., Nie, F., Luo, J., & Zhuang, Y. (2009) · Proceedings of the 17th ACM international conference on Multimedia · doi:10.1145/1631272.1631298 · Cited by 179
Learning Joint Statistical Models for Audio-Visual Fusion and Segregation
Fisher, J. S., Darrell, T., Freeman, W. T., & Viola, P. A. (2000) · Cited by 176
Mining Semantic Correlation of Heterogeneous Multimedia Data for Cross-Media Retrieval
Zhuang, Y., Yang, Y., & Wu, F. Q. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2007.911822 · Cited by 149
Unifying Textual and Visual Cues for Content-Based Image Retrieval on the World Wide Web
Sclaroff, S., Cascia, M. L., Sethi, S., & Taycher, L. (1999) · Computer Vision and Image Understanding · doi:10.1006/cviu.1999.0765 · Cited by 132
Building text features for object image classification
Wang, G., Hoiem, D., & Forsyth, D. A. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206816 · Cited by 123
Bayesian network model for semi-structured document classification
Denoyer, L., & Gallinari, P. (2004) · Information Processing & Management · doi:10.1016/j.ipm.2004.04.009 · Cited by 113
Towards semantic knowledge propagation from text corpus to web images
Qi, G.-J., Aggarwal, C. C., & Huang, T. S. (2011) · International World Wide Web Conference (WWW) · doi:10.1145/1963405.1963449 · Cited by 111
Multimedia content processing through cross-modal association
Li, D., Dimitrova, N., Li, M., & Sethi, I. K. (2003) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/957142.957143 · Cited by 106
Slaney, M. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5745561 · Cited by 100
Accounting for burstiness in topic models
Doyle, G., & Elkan, C. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553410 · Cited by 99
Self-citationImage retrieval: content versus context
Westerveld, T. (2000) · University of Twente Research Information · Cited by 93
Latent semantic fusion model for image retrieval and annotation
Pham, T.-T., Maillot, N. E., Lim, J., & Chevallet, J. (2007) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1321440.1321503 · Cited by 89
Cross-modal correlation learning for clustering on image-audio dataset
Zhang, H., Zhuang, Y., & Wu, F. Q. (2007) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1291233.1291290 · Cited by 76
Multiclass Boosting: Theory and Algorithms
Saberian, M. J., & Vasconcelos, N. M. (n.d.) · Cited by 76
Self-citationMinimum Probability of Error Image Retrieval
Vasconcelos, N. M. (2004) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2004.831125 · Cited by 69
Self-citationStatistical multimodal integration for audio-visual speech processing
Nakamura, S. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1021886 · Cited by 42
Vinokourov, A., Hardoon, D. R., & Shawe-Taylor, J. S. (2003) · ePrints Soton (University of Southampton) · Cited by 32
Maximum Covariance Unfolding : Manifold Learning for Bimodal Data
Mahadevan, V., Wong, C. W., Pereira, J. C., et al. (2011) · Cited by 26
Overview of the WikipediaMM Task at ImageCLEF 2008
Tsikrika, T., & Kludas, J. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04447-2_66 · Cited by 26
TVGraz: Multi-Modal Learning of Object Categories by Combining Textual and Visual Features
Khan, I., Saffari, A., & Bischof, H. (2010) · Cited by 24
On the regularization of image semantics by modal expansion
Pereira, J. C., & Vasconcelos, N. M. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248041 · Cited by 19
Self-citationManifold Learning Based Cross-media Retrieval: A Solution to Media Object Complementary Nature
Zhuang, Y., Yang, Y., Wu, F. Q., & Pan, Y. (2007) · The Journal of VLSI Signal Processing Systems for Signal Image and Video Technology · doi:10.1007/s11265-006-0020-y · Cited by 16
Knowledge Discovery from Community-Contributed Multimedia
Mei, T., Hsu, W. H., & Luo, J. (2010) · IEEE Multimedia · doi:10.1109/mmul.2010.82 · Cited by 15
Knowledge discovery over community-sharing media: From signal to intelligence
Hsu, W. H., Mei, T., & Yan, R. (2009) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2009.5202775 · Cited by 5
Cited by
Adversarial Cross-Modal Retrieval
Wang, B., Yang, Y., Xu, X., Hanjalic, A., & Shen, H. T. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123326 · Cited by 779
Semantics-preserving hashing for cross-view retrieval
Lin, Z., Ding, G., Hu, M., & Wang, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299011 · Cited by 542
Deep Supervised Cross-Modal Retrieval
Zhen, L., Hu, P., Wang, X., & Peng, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01064 · Cited by 478
Cross-Modal Retrieval With CNN Visual Features: A New Baseline
Wei, Y., Zhao, Y., Lu, C., Wei, S., Liu, L., Zhu, Z., & Yan, S. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2519449 · Cited by 370
Deep Joint-Semantics Reconstructing Hashing for Large-Scale Unsupervised Cross-Modal Retrieval
Su, S., Zhong, Z., & Zhang, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00312 · Cited by 342
Metadata
- Authors
- Costa Pereira, J., Coviello, E., Doyle, G., Rasiwasia, N., Lanckriet, G. R. G., Levy, R., & Vasconcelos, N.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2014-03
- Volume
- 36
- Issue
- 3
- Pages
- 521-535
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W2138118304
- Cited by
- 435
- References
- 62
- FWCI
- 35.31
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.