Dehghani, Mostafa

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv

Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

Cited by 47,472

ViViT: A Video Vision Transformer

Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV)

Human Pose and Action RecognitionAdvanced Neural Network Applications

Cited by 2,767

Scaling Instruction-Finetuned Language Models

Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv

Topic ModelingDomain Adaptation and Few-Shot Learning

Cited by 1,441

Efficient transformers: A survey

Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys

Topic ModelingAdvanced Neural Network Applications

Cited by 1,267

Efficient Transformers: A Survey

Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2020) · arXiv

Topic ModelingAdvanced Neural Network Applications

Cited by 1,267

Gemini: A Family of Highly Capable Multimodal Models

Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv

Multimodal Machine Learning ApplicationsTopic Modeling

Cited by 990

PaLM 2 Technical Report

Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 834

Universal Transformers

Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., & Kaiser, Ł. (2018) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 467

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv

Multimodal Machine Learning ApplicationsTopic Modeling

Cited by 306

Long Range Arena: A Benchmark for Efficient Transformers

Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., & Metzler, D. (2020) · arXiv

Topic ModelingAdvanced Neural Network Applications

Cited by 241

Transformer Memory as a Differentiable Search Index

Tay, Y., Tran, V., Dehghani, M., Ni, J., Bahri, D., Mehta, H., Qin, Z., Hui, K., Zhao, Z., Gupta, J., et al. (2022) · Advances in Neural Information Processing Systems 35

Topic ModelingInformation Retrieval and Search Behavior

Cited by 128

Scaling vision transformers to 22 billion parameters

Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. (2023) · arXiv

Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

Cited by 112

Confident adaptive language modeling

Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V., Tay, Y., & Metzler, D. (2022) · Advances in Neural Information Processing Systems 35

Natural Language Processing TechniquesSpeech Recognition and Synthesis

Cited by 91

Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. (2025) · arXiv

Multimodal Machine Learning ApplicationsTopic Modeling

Cited by 69

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., & Metzler, D. (2021) · arXiv

Topic ModelingNatural Language Processing Techniques

Cited by 63

The Benchmark Lottery

Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., & Vinyals, O. (2021) · arXiv

Machine Learning and Data ClassificationMachine Learning and Algorithms

Cited by 48

Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution

Dehghani, M., Mustafa, B., Djolonga, J., Heek, J., Minderer, M., Caron, M., Steiner, A., Puigcerver, J., Geirhos, R., Alabdulmohsin, I., et al. (2023) · arXiv

Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

Cited by 45

Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023

Topic ModelingNatural Language Processing Techniques

Cited by 34

Transferring Inductive Biases through Knowledge Distillation

Abnar, S., Dehghani, M., & Zuidema, W. (2020) · arXiv

Domain Adaptation and Few-Shot LearningAdvanced Neural Network Applications

Cited by 30

DSI++: Updating Transformer Memory with New Documents

Mehta, S., Gupta, J., Tay, Y., Dehghani, M., Tran, V., Rao, J., Najork, M., Strubell, E., & Metzler, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

Topic ModelingInformation Retrieval and Search Behavior

Cited by 21

Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., & Metzler, D. (2022) · arXiv

Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

Cited by 11