Multimodal Recurrent Neural Networks With Information Transfer Layers for Indoor Scene Labeling
Open accessAbstract
This paper proposes a new method called multimodal recurrent neural networks (RNNs) for RGB-D scene semantic segmentation. It is optimized to classify image pixels given two input sources: RGB color channels and depth maps. It simultaneously performs training of two RNNs that are crossly connected through information transfer layers, which are learnt to adaptively extract relevant cross-modality features. Each RNN model learns its representations from its own previous hidden states and transferred patterns from the other RNNs previous hidden states; thus, both model-specific and cross-modality features are retained. We exploit the structure of quad-directional 2D-RNNs to model the short- and long-range contextual information in the 2D input image. We carefully designed various baselines to efficiently examine our proposed model structure. We test our multimodal RNNs method on popular RGB-D benchmarks and show how it outperforms previous methods significantly and achieves competitive results with other state-of-the-art works.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recurrent neural networks have proved useful in multimodal problems requiring modeling long- and short-range dependencies across the input sequence, such as semantic segmentation and image captioning.
“In addition to their use for unimodal tasks, RNNs have proved useful in many multimodal problems that require modeling long- and short-range dependencies across the input sequence, such as semantic segmentation [219] and image captioning [220].”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [219]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · see citing works
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · see citing works
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,882 · see citing works
RELATIONS BETWEEN TWO SETS OF VARIATES
Hotelling, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 5,429 · see citing works
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310 · see citing works
Eigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862 · see citing works
Deep learning for detecting robotic grasps
Lenz, I., Lee, H., & Saxena, A. (2013) · Int. J. Robotics Res. · doi:10.1177/0278364914549607 · Cited by 1,820 · see citing works
Fu, J., Zheng, H., & Mei, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.476 · Cited by 1,459 · see citing works
Multimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · doi:10.5555/2627435.2697059 · Cited by 1,070 · see citing works
A Survey on Multi-view LearningPreprint
Xu, C., Tao, D., & Xu, C. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.5634 · Cited by 983 · see citing works
Perceptual Generative Adversarial Networks for Small Object Detection
Li, J., Liang, X., Wei, Y., Xu, T., Feng, J., & Yan, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.211 · Cited by 953 · cited 1× · see citing works
“formation for each pixel within each modality [1], [2], [14], [40].”
BackgroundOffline Arabic Handwriting Recognition with Multidimensional Recurrent Neural Networks
Graves, A. (2012) · Springer eBooks · doi:10.1007/978-1-4471-4072-6_12 · Cited by 685 · see citing works
Detecting activities of daily living in first-person camera views
Pirsiavash, H., & Ramanan, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248010 · Cited by 666 · see citing works
Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images
Gupta, S., Arbeláez, P., & Malik, J. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.79 · Cited by 601 · see citing works
Depth and Appearance for Mobile Scene Analysis
Ess, A., Leibe, B., & Van Gool, L. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4409092 · Cited by 540 · see citing works
Cross Modal Distillation for Supervision Transfer
Gupta, S., Hoffman, J., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.309 · Cited by 530 · cited 1× · see citing works
“[52], where they propose to transfer supervision between images from different modalities.”
BackgroundMulti-cue Pedestrian Detection and Tracking from a Moving Vehicle
Gavrila, D. M., & Munder, S. (2006) · International Journal of Computer Vision · doi:10.1007/s11263-006-9038-7 · Cited by 524 · see citing works
Indoor scene segmentation using a structured light sensor
Silberman, N., & Fergus, R. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130298 · Cited by 485 · see citing works
Advances in optimizing recurrent networks
Bengio, Y., Boulanger-Lewandowski, N., & Pascanu, R. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639349 · Cited by 478 · see citing works
Indoor Semantic Segmentation using depth information
Couprie, C., Farabet, C., Najman, L. A., & LeCun, Y. (2013) · HAL · doi:10.48550/arxiv.1301.3572 · Cited by 451 · see citing works
Robust Face Recognition via Multimodal Deep Face Representation
Ding, C., & Tao, D. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2477042 · Cited by 449 · see citing works
Scene labeling with LSTM recurrent neural networks
Byeon, W., Breuel, T. M., Raue, F., & Liwicki, M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298977 · Cited by 384 · see citing works
Opinion Mining with Deep Recurrent Neural Networks
Irsoy, O., & Cardie, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1080 · Cited by 384 · cited 2× · see citing works
“, speech recognition and natural language processing [17], [18], [20], [31].”
BackgroundMulti-dimensional Recurrent Neural Networks
Graves, A., Fernández, S., & Schmidhuber, J. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74690-4_56 · Cited by 303 · see citing works
Dynamic 3D Scene Analysis from a Moving Vehicle
Leibe, B., Cornelis, N., Cornelis, K., & Gool, L. V. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383146 · Cited by 301 · see citing works
Dense 3D semantic mapping of indoor scenes from RGB-D images
Hermans, A., Floros, G., & Leibe, B. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907236 · Cited by 258 · see citing works
Gupta, S., Arbeláez, P., Girshick, R., & Malik, J. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0777-6 · Cited by 248 · see citing works
Deep Head Pose: Gaze-Direction Estimation in Multimodal Video
Mukherjee, S. S., & Robertson, N. M. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2482819 · Cited by 183 · see citing works
Convolutional recurrent neural networks: Learning spatial dependencies for image representation
Zuo, Z. C., Shuai, B., Wang, G., Liu, X., Wang, X., Wang, B., & Chen, Y. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301268 · Cited by 181 · see citing works
Self-citationReal-Time Dense Stereo for Intelligent Vehicles
vanderMark, W., & Gavrila, D. M. (2006) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2006.869625 · Cited by 172 · see citing works
Tracking via Robust Multi-task Multi-view Joint Sparse Representation
Hong, Z., Mei, X., Prokhorov, D. V., & Tao, D. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.86 · Cited by 155 · see citing works
Large-Margin Multi-Modal Deep Learning for RGB-D Object Recognition
Wang, A., Lu, J., Cai, J., Cham, T.-J., & Wang, G. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2476655 · Cited by 153 · see citing works
Self-citationMultiK-MHKS: A Novel Multiple Kernel Learning Algorithm
Wang, Z., Chen, S., & Sun, T. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.70786 · Cited by 129 · see citing works
Learning Compact Hash Codes for Multimodal Representations Using Orthogonal Deep Structure
Wang, D., Cui, P., Ou, M., & Zhu, W. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2455415 · Cited by 98 · see citing works
Learning depth-sensitive conditional random fields for semantic segmentation of RGB-D images
Muller, A. C., & Behnke, S. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907778 · Cited by 76 · see citing works
Dense real-time mapping of object-class semantics from RGB-D video
Stückler, J., Waldvogel, B., Schulz, H., & Behnke, S. (2013) · Journal of Real-Time Image Processing · doi:10.1007/s11554-013-0379-5 · Cited by 74 · see citing works
Accurate Estimation of Human Body Orientation From RGB-D Sensors
Liu, W., Zhang, Y., Tang, S., et al. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2272636 · Cited by 69 · see citing works
A Multimodal and Multilevel Ranking Scheme for Large-Scale Video Retrieval
Hoi, S. C. H., & Lyu, M. R.-T. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.921735 · Cited by 53 · see citing works
Quaddirectional 2D-Recurrent Neural Networks For Image Labeling
Shuai, B., Zuo, Z. C., & Wang, G. (2015) · IEEE Signal Processing Letters · doi:10.1109/lsp.2015.2441781 · Cited by 39 · see citing works
Self-citationUnsupervised multimodal feature learning for semantic image segmentation
Pei, D., Liu, H., Liu, Y., et al. (2013) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2013.6706748 · Cited by 33 · see citing works
Semantic parsing for priming object detection in indoors RGB-D scenes
Cadena, C., & Košecká, J. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364914549488 · Cited by 32 · see citing works
Episodic CAMN: Contextual Attention-Based Memory Networks with Iterative Feedback for Scene Labeling
Abdulnabi, A. H., Shuai, B., WINKLER, S. S., & Wang, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.665 · Cited by 16 · see citing works
Self-citationPedestrian recognition using combined low-resolution depth and intensity images
Rapus, M., Munder, S., Baratoff, G., et al. (2008) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2008.4621195 · Cited by 14 · see citing works
Abdulnabi, A. H., WINKLER, S. S., & Wang, G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05537 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 21
- [4][4] G. Andrew, R. Arora, J. Bilmes, and K. Livescu. Deep canonical correlation analysis. In ICML , 2013. 2, 3 -
- [9][9] N. Dalal, B. Triggs, and C. Schmid. Human detection using oriented histograms of flow and appearance. In ECCV , 2006. 3 -
- [14][14] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Scene parsing with multiscale feature learning, purity trees, and optimal covers. In ICML , 2012. 2, 8 -
- [17][17] A. Graves, S. Fernandez, and J. Schmidhuber. Bidirectional LSTM networks for improved phoneme classification and recognition. In ICANN , pages 799-804, 2005. 3 -
- [24][24] J. He and R. Lawrence. A graph-based framework for multi-task multiview learning. In ICML , pages 25-32, 2011. 3 -
- [25][25] X. He, R. Zemel, and M. Carreira-Perpindn. Multiscale conditional random fields for image labeling. In CVPR , 2004. 2 -
- [27][27] S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In A Field Guide to Dynamical Recurrent Neural Networks, IEEE Press , 2001. 3 -
- [35][35] S. H. Khan, M. Bennamoun, F. Sohel, and R. Togneri. Geometry driven semantic labeling of indoor scenes. In ECCV , 2014. 9, 13 -
- [36][36] S. Kim and E. P. Xing. Tree-guided group lasso for multi-task regression with structured sparsity. In ICML , 2010. 3 -
- [37][37] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS , pages 1097-1105, 2012. 2 -
- [38][38] H. Lee, C. Ekanadham, and A. Y. NG. Sparse deep belief net model for visual area v2. In NIPS , 2007. 2, 3 -
- [45][45] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. Multimodal deep learning. In ICML , 2011. 2, 3 -
- [48][48] P. H. O. Pinheiro and R. Collobert. Recurrent convolutional neural networks for scene labeling. In ICML , 2014. 2 -
- [51][51] X. Ren, L. Bo, and D. Fox. Rgb-(d) scene labeling: Features and algorithms. In CVPR , 2012. 2, 9 -
- [52][52] M. Rohrbach, M. Enzweiler, and D. M. Gavrila. High-Level fusion of depth and intensity for pedestrian classification. In DAGM 2009: Pattern Recognition , pages 101-110, 2009. 3 -
- [56][56] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. In ECCV , 2012. 1, 2, 7, 9, 13 -
- [58][58] K. Sohn, W. Shang, and H. Lee. Improved multimodal deep learning with variation of information. In NIPS , 2014. 2, 3 -
- [64][64] A. Wang, J. Lu, G. Wang, J. Cai, and T. Cham. Multi-modal unsupervised feature learning for rgb-d scene labeling. In ECCV , 2014. 2, 9, 13 -
- [66][66] W. Wang and Z. hua Zhou. A new analysis of co-training. In ICML , 2010. 2, 3 -
- [70][70] J. G. Yong Jae Lee and K. Grauman. Discovering important people and objects for egocentric video summarization. In CVPR , 2012. 1 -
- [71][71] F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR , pages 1-9, 2016. 10, 11 -
Cited by
Sorted by citations
A survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Deep learning based multimodal biomedical data fusion: an overview and comparative review
Duan, J., Xiong, J., Li, Y., & Ding, W. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102536 · Cited by 284 · see citing works
- Authors
- Abdulnabi, A. H., Shuai, B., Zuo, Z., Chau, L.-P., & Wang, G.
- Affiliations
- Nanyang Technological University (SG)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2018-07
- Volume
- 20
- Issue
- 7
- Pages
- 1656-1671
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2768799425
- Cited by
- 33
- References
- 48
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 5 · 2023: 2 · 2022: 5 · 2021: 5 · 2020: 7 · 2019: 1 · 2018: 3 · 2017: 1
- FWCI
- 1.36
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Singapore Ministry of Education Tier 2 (ARC28/14)
- Singapore Agency for Science, Technology and Research (A*STAR)
- Science and Engineering Research Council (PSF1321202099)
- ADSC from A*STAR
- National Research Foundation, Singapore
- Interactive & Digital Media Strategic Research Programme
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Media Technology › Remote-Sensing Image Classification
- Keywords
- depth images, context modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
