DINOspec: Efficient Multimodal Alignment of Vision and Spectral Foundation Models for Astronomy

Abstract

Astronomical observations provide multimodal views of physical systems, with images and spectra capturing complementary properties of celestial objects. Scientific foundation models can learn powerful representations from these observations, but representations learned by separate models remain difficult to combine. We investigate whether physical representations learned by separate vision and spectral models can be aligned without retraining their encoders. We introduce DINOspec, a multimodal framework that aligns a frozen DINOv3 image encoder with a pre-trained AION-1 spectral tokenizer using lightweight adapters and contrastive learning on 20,472 paired images and spectra of astronomical objects. DINOspec improves galaxy morphology classification (F1: 0.72$\rightarrow$0.78) and spectral classification (F1: 0.70$\rightarrow$0.74) while training at most 21M parameters. Improvements depend on the downstream task, revealing asymmetric transfer between independently learned representations, while spectroscopic redshift prediction remains unchanged ($R^2\approx0.9$). These results demonstrate that scientific foundation models can be composed through lightweight representation alignment.