COOL, a Context Outlooker, and its Application to Question Answering and other Natural Language Processing Tasks
Fangyi Zhu, See-Kiong Ng, Stéphane Bressan
Abstract
Vision outlooker improves the performance of vision transformers, which implements a self-attention mechanism by adding an outlook attention, a form of local attention. In natural language processing, as has been the case in computer vision and other domains, transformer-based models constitute the state-of-the-art for most processing tasks. In this domain, too, many authors have argued and demonstrated the importance of local context. We present an outlook attention mechanism, COOL, for natural language processing. COOL, added on top of the self-attention layers of a transformer-based model, encodes local syntactic context considering word proximity and more pair-wise constraints than dynamic convolution used by existing approaches. A comparative empirical performance evaluation of an implementation of COOL with different transformer-based models confirms the opportunity for improvement over a baseline using the original models alone for various natural language processing tasks, including question answering. The proposed approach achieves competitive performance with existing state-of-the-art methods on some tasks.
Create a lesson
Related papers
Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
Evan Li, Krishna Deshpande, Wen Cheng et al.
Inherited Weights: Measuring the Tokenization Premium and the Sovereignty Cost of Language Models for the Global South
Paulo Camara
BioMem: Agentic Memory for Biomedical LLM Agents
Mingchen Li, Yongkang Xiao, Rumeng Li et al.
Mapping Instructors' Attitudes Towards Cheating with AI: An Exploratory Analysis Using Forum Analysis
Iman Mohammadi, Ali Keramati, Sina Rismanchian
Understanding Students' Rationales for AI-Mediated Cheating: An Instructor Perspective from Online Forums Analysis
Iman Mohammadi, Sina Rismanchian, Ali Keramati et al.
Measuring Definition Sensitivity in LLM Annotation of Social Science Concepts
Bogdan Mamaev, Qizhou Wang, Christopher Leckie et al.