Exploring the Evidence-Based SE Beliefs of Generative AI Tools

Abstract

Background: Recent innovations in generative artificial intelligence (AI) have transformed how programmers develop and maintain software. The advanced capabilities of generative AI tools in supporting development tasks have led to a rise in their adoption within software engineering (SE) workflows. However, little is known about how AI tools perceive evidence-based practices supported by empirical SE research. Aim: To this end, we explore the "beliefs" of generative AI tools increasingly used to support software development in practice. Method: We conduct a preliminary evaluation conceptually replicating prior work to investigate 17 evidence-based claims across five generative AI tools. Results: Our findings demonstrate generative AI tools have ambiguous beliefs regarding research claims and lack credible evidence to support responses. Conclusions: Based on our results, we provide implications for practitioners integrating generative AI-based systems into development contexts and shed light on future research directions to enhance the reliability and trustworthiness of generative AI -- aiming to increase awareness and adoption of evidence-based SE research findings in practice.