The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images
Open accessAbstract
In this paper, we introduce Recipe1M+, a new large-scale, structured corpus of over one million cooking recipes and 13 million food images. As the largest publicly available collection of recipe data, Recipe1M+ affords the ability to train high-capacity models on aligned, multi-modal data. Using these data, we train a neural network to learn a joint embedding of recipes and images that yields impressive results on an image-recipe retrieval task. Moreover, we demonstrate that regularization via the addition of a high-level classification objective both improves retrieval performance to rival that of humans and enables semantic vector arithmetic. We postulate that these embeddings will provide a basis for further exploration of the Recipe1M+ dataset and food and cooking in general. Code, data and models are publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2008) · Journal of Machine Learning Research · Cited by 33,051
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042
Self-citationDeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534
Food-101 – Mining Discriminative Components with Random Forests
Bossard, L., Guillaumin, M., & Van Gool, L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10599-4_29 · Cited by 2,072
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293
Self-citationUSDA National Nutrient Database for Standard Reference, Release 22
(2018) · Cited by 1,285
Object Detectors Emerge in Deep Scene CNNsPreprint
Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., & Torralba, A. B. (2015) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 523
Self-citationIm2Calories: Towards an Automated Mobile Vision Food Diary
Myers, A., Johnston, N., Rathod, V., Korattikara, A., Gorban, A., Silberman, N., Guadarrama, S., Papandreou, G., Huang, J., & Murphy, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.146 · Cited by 485
Learning Cross-Modal Embeddings for Cooking Recipes and Food Images
Salvador, A., Hynes, N., Aytar, Y., Marin, J., Ofli, F., Weber, I., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.327 · Cited by 404
Self-citationDeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment
Liu, C., Cao, Y., Luo, Y., Chen, G., Vokkarane, V., & Ma, Y. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-39601-9_4 · Cited by 289
FoodCam: A real-time food recognition system on a smartphone
Kawano, Y., & Yanai, K. (2015) · Multimedia Tools and Applications · doi:10.1007/s11042-014-2000-8 · Cited by 191
Learning Aligned Cross-Modal Representations from Weakly Aligned Data
Castrejon, L., Aytar, Y., Vondrick, C., Pirsiavash, H., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.321 · Cited by 188 · cited 1×
“Previous work on cross-modality training [31], [32] suggests training models for different modalities separately and fine tuning them jointly afterwards to allow alignment.”
BackgroundSelf-citationRecipe recognition with large multimodal food dataset
Wang, X., Kumar, D., Thome, N., Cord, M., & Precioso, F. (2015) · 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew.2015.7169757 · Cited by 179
Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings
Carvalho, M., Cadène, R., Picard, D., Soulier, L., Thome, N., & Cord, M. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210036 · Cited by 171
Aytar, Y., Castrejón, L., Vondrick, C., Pirsiavash, H., & Torralba, A. B. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2753232 · Cited by 126
Self-citationFetishizing Food in Digital Age: #foodporn Around the World
Mejova, Y., Abbar, S., & Haddadi, H. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v10i1.14710 · Cited by 111
Deep Understanding of Cooking Procedure for Cross-modal Recipe Retrieval
Chen, J., Ngo, C., Feng, F., & Chua, T. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240627 · Cited by 111
Cross-modal Recipe Retrieval with Rich Food Attributes
Chen, J., Ngo, C., & Chua, T. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123428 · Cited by 106
Geolocalized Modeling for Dish Recognition
Xu, R., Herranz, L., Jiang, S., Wang, S., Song, X., & Jain, R. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2438717 · Cited by 91
Social Media Image Analysis for Public Health
Garimella, V. R. K., Alfayad, A., & Weber, I. (2015) · International Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858234 · Cited by 87
Self-citationIs Saki #delicious? The Food Perception Gap on Instagram and Its Relation to Health
Ofli, F., Aytar, Y., Weber, I. G., al Hammouri, R., & Torralba, A. B. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3038912.3052663 · Cited by 69
Self-citationChang, M., Guillain, L. V., Jung, H., Hare, V. M., Kim, J., & Agrawala, M. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174025 · Cited by 52
A Delicious Recipe Analysis Framework for Exploring Multi-Modal Recipes with Various Attributes
Min, W., Jiang, S., Wang, S., Sang, J., & Mei, S. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123272 · Cited by 51
Cited by
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Large Scale Visual Food Recognition
Min, W., Wang, Z., Liu, Y., Luo, M., Kang, L., Wei, X., Wei, X., & Jiang, S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3237871 · Cited by 253
Dugan, L., Ippolito, D., Kirubarajan, A., Shi, S., & Callison-Burch, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26501 · Cited by 54
Zhang, Z., Wang, Z., Ma, T., Taneja, V. S., Nelson, S., Le, N. H. L., Murugesan, K., Ju, M., Chawla, N. V., Zhang, C., et al. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 · doi:10.1145/3690624.3709382 · Cited by 16
Metadata
- Authors
- Marin, J., Biswas, A., Ofli, F., Hynes, N., Salvador, A., Aytar, Y., Weber, I., & Torralba, A.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2021-01-01
- Volume
- 43
- Issue
- 1
- Pages
- 187-203
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2897709118
- Cited by
- 321
- References
- 33
- FWCI
- 15.98
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.