Text-to-Image Generative Modeling Without Language Pretraining
Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas, Ehsan Adeli, Jiajun Wu, Juan Carlos Niebles, Li Fei-Fei, Michael Poli
Abstract
Modern text-to-image (T2I) systems use internet-scale language pretraining for text-conditioned image generation. We ask whether language pretraining is necessary for strong T2I generation. We introduce Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100M image-text pairs. Halcyon-0.52B outperforms LlamaGen-XL/8 (48.0 vs. 63.3 gFD) with 18.2× less pretraining compute. Further, Halcyon-1.6B outperforms JiT-T2I-XXL/8++ (35.3 vs. 41.0 gFD) with 275× less pretraining compute. Controlled experiments show that text-only pretraining does not improve T2I generation. Halcyon also learns image captioning by replacing caption dropout with I2T supervision, maintaining competitive T2I generation quality under the same pretraining budget. We show that strong T2I systems can be trained from scratch without internet-scale language pretraining. Code and models are available at https://cs.stanford.edu/~keshik/halcyon
Create a lesson
Related papers
WeAct: Dynamic Weight Switching as Agent Actions
Rui Shi, Xunjian Yin, Jiacheng Sang et al.
Kronecker Delta Rule for Long Context Linear Attention
Anamaria-Roberta Hartl, Levente Zólyomi, Michael List et al.
Next Best Action: Causal Constrained Offline RL for Business Action Targeting
Surya Ramachandiran, Yiming Li, Juan Mancilla-Caceres et al.
Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
Zhuofan Josh Ying, Guanglong Sun, Nolan Ho et al.
Learning to Refer: Inference-Time Collaboration in Decentralized Federated Learning
Yuheng Ma, Chendi Wang, Qiong Zhang
Two hidden layer suffice: the caterpillar net
Jakob Paul Zimmermann