The DOI matches its DataCite record; 89 of the 89 references it cites were found in a registry or an index.
Muse: Text-To-Image Generation via Masked Generative Transformers
Open accessAbstract
We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [163]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“For the base model, the decoder consists of a stack of Transformer layers which are finally mapped to patch space.”
MethodInfluentialU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 1×
“Furthermore, we believe that the use of token-based Trans-formers as a unifying architecture leads to better text-image alignment, compared to diffusion models where the image space is a convolution U-Net (Ronneberger et al., 2015).”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 4×
“Our 3B parameter model achieves a CLIP score of 0.32 and an FID score of 7.88 on the COCO (Lin et al., 2014) zero-shot validation benchmark, which compare favorably with that of other large-scale text-to-image models (see Table 2).”
MethodInfluentialSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 3×
“In Table 3, we also ablated multi-axis Trans-formers (Zhao et al., 2021) against SWIN (Liu et al., 2021), finding that the former performed better.”
MethodInfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“LLMs (including T5XXL, BERT (Devlin et al., 2018), and GPT (Brown et al., 2020; Radford et al., 2019)) have been shown to learn powerful embeddings, enabling few-shot transfer learning.”
BackgroundHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 13×
“…Raffel et al., 2020) and vision tasks (He et al., 2022; Chang et al., 2022); new families of generative models such as diffusion (Ho et al., 2020; Rombach et al., 2022; Saharia et al., 2022) and masking-based generation (Chang et al., 2022); and finally, the availability of large scale image-text…”
BackgroundMethodInfluentialGenerative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 13×
“Our 3B parameter model achieves a CLIP score of 0.32 and an FID score of 7.88 on the COCO (Lin et al., 2014) zero-shot validation benchmark, which compare favorably with that of other large-scale text-to-image models (see Table 2).”
BackgroundMethodInfluential
Research works printed but not linked: 5
- Buolamwini, J. and Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency , pp. 77-91. PMLR, 2018.
- Dulhanty, C. Issues in computer vision data collection: Bias, consent, and label taxonomy. Master's thesis, University of Waterloo, 2020.
- Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.
- Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. In ICLR , 2022.
Cited by
Sorted by citations
Items 101–121 of 121
Riverine flood hazard map prediction by neural networks
Yin, Z., & León, A. S. (2024) · HydroResearch · doi:10.1016/j.hydres.2024.10.003 · Cited by 3
D$$^2$$Styler: Advancing Arbitrary Style Transfer with Discrete Diffusion Methods
Susladkar, O., Deshmukh, G., Mittal, S., & Shastri, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78172-8_5 · Cited by 4
Mapache: Masked Parallel Transformer for Advanced Speech Editing and Synthesis
Cámbara, G., Tobing, P. L., Babianski, M., Vipperla, R., Wang, D., Shmelkin, R., Coccia, G., Angelini, O., Joly, A., Łajszczak, M., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448121 · Cited by 2
“The way we automatically mask tokens in training data is identical to Muse [18], as we use a Co-sine schedule to sample a different masking percentage of the total sequence r ∈ [0 , 100]% for each batch.”
BackgroundMethod2S-ODIS: Two-Stage Omni-Directional Image Synthesis by Geometric Distortion Correction
Nakata, A., & Yamanaka, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73397-0_20 · Cited by 2
Temporal Adaptive Attention Map Guidance for Text-to-Image Diffusion Models
Jung, S.-H., & Heo, Y. S. (2025) · Electronics · doi:10.3390/electronics14030412 · Cited by 2
Painter: Teaching Auto-regressive Language Models to Draw Sketches
Pourreza, R., Bhattacharyya, A., Panchal, S., Lee, M., Madan, P., & Memisevic, R. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00038 · Cited by 4
“Diffusion models and their variants [14, 25, 26, 5] are now state-of-the-art in this domain as they are very creative and generate mesmerizing results.”
BackgroundMethodArtQuest: Countering Hidden Language Biases in ArtVQA
Bleidt, T., Eslami, S., & de Melo, G. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00716 · Cited by 4
“In the art domain, VLP has enabled recent advances in artistic image generation from text prompts [5, 30, 31].”
BackgroundSemantically complex audio to video generation with audio source separation
Kim, S., Jeong, J., In, S., Lee, S. H., Kim, S., Kim, S., Baek, W., Yoon, S. H., Culurciello, E., & Kim, S. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110457 · Cited by 1
Jardim, R. R.-A. J. (2023) · Springer eBooks · doi:10.1007/978-3-031-37345-9_1 · Cited by 1
Applications of transformers in computer vision
Jin, Y., & Ma, R. (2023) · Applied and Computational Engineering · doi:10.54254/2755-2721/16/20230898 · Cited by 1
Stave, D. Å., Korneliussen, H., Hjellup, H. N., & Shrestha, R. (2023) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc57904.2023.10303076 · Cited by 1
“Other recent advancements within the field of image generation models include the DALL-E 2 model, which is based on diffusion techniques [29], and a text-to-image transformer model proposed by providing to be more efficient than current diffusion or autoregressive models [30].”
MethodTEXT TO IMAGE GENERATOR USING AI
Athawale, D. (2024) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem31493 · Cited by 1
MarkovGen: Structured Prediction for Efficient Text-to-Image Generation
Jayasumana, S., Gläsner, D., Ramalingam, S., Veit, A., Chakrabarti, A., & Kumar, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00890 · Cited by 1
“We now demonstrate the benefits of the proposed MRF model by using it to speed up the state-of-the-art Muse image generation model [2].”
InfluentialMultiGen: Zero-Shot Image Generation from Multi-modal Prompts
Wu, Z.-F., Huang, L., Wang, W., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73242-3_17 · Cited by 1
Accelerating Image Generation with Sub-path Linear Approximation Model
Xu, C., Song, T., Feng, W., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73668-1_19 · Cited by 1
Wan, Y., Xiao, L., Wu, X., Yang, J., & He, L. (2024) · Symmetry · doi:10.3390/sym16121608 · Cited by 4
Large Vison-Language Foundation Model in Baidu AIGC Image Advertising
Jin, Z., Tao, W., Li, Y., Yang, Y., Han, C., Li, S., & Liu, L. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709401 · Cited by 1
“The dominant architecture now is the diffusion model, with some pilot studies incorporating Transformer in image generation [43–45].”
Self-citationKasnesis, P., Toumanidis, L., Pagliari, D. J., & Burrello, A. (2023) · doi:10.36227/techrxiv.24504040
Optimizing codebook training through control chart analysis
Wang, K., Shi, Q., Li, X., Wu, E., & Li, Z. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01555-x
Enhancing Gameplay Experience Through Artificial Life
Liu, X., & Zhao, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92578-8_13
Zhang, L., Shi, Y., & Cui, M. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05223-x · Cited by 1
“Tam-pering is an inner-modal editing technique and requires only instructions to modify the existing content (Chang et al. 2023).”
Metadata
- Authors
- Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., Li, Y., & Krishnan, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2301.00704
- OpenAlex
- W4313484371
- Cited by
- 157
- References
- 74
- Citations by year
- 2025: 25 · 2024: 92 · 2023: 40
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- text-to-image generation, image inpainting, parallel decoding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
