Evaluating the Performance of LLMs in Drafting NIH Data Management Plans
Nahid Zeinali, James ONeil, Xuebin Dong, Rebecca Hofstein Grady, Maria Praetzellis, Brian Riley, John Chodacki, Bhavesh Patel
Abstract
Sharing research data is essential to promote transparency, enable reproducibility, and fuel scientific discoveries. To ensure consistent sharing, the National Institutes of Health (NIH) and other funding agencies require applicants to submit data management plans (DMPs) with their proposals. They aim to encourage researchers to consider their data management and sharing strategies early on. However, researchers often face challenges when developing DMPs due to a lack of knowledge about data-sharing practices. Here, we evaluated the performance of Large Language Models (LLMs) in drafting DMPs compliant with the 2023 NIH guidelines. The goal was to investigate whether LLM-supported DMP creation could reduce the burden on researchers and librarians. We compared Llama and GPT performance against NIH-provided DMP samples. We also asked domain experts to evaluate LLM-generated DMPs. Overall, we found that LLMs can accelerate the creation of initial drafts but still require expert review before submission to ensure clarity and completeness. Moreover, open-source LLMs may not be as ready off-the-shelf as commercial ones for drafting funder-compliant DMPs.
Create a lesson
Related papers
Do Network Mechanisms in Science Vary Across Research Topics? Contextual Relational Hyperevent Models for Large-Scale Bibliometric Data
Jürgen Lerner, Mark Wittek, Raphael Heiberger
Extracting Value from Grey Literature: processes and technologies for aggregating and analyzing the hidden big data treasure of organizations
G. Motta, R. Puccinelli, L. Reggiani et al.
Uncontrolled source titles: When Nature, Science, and PNAS meet Scopus
Weishu Liu, Ruifeng Zhang, Haifeng Wang
Theses by publication in Spain: output, authorship and integrity concerns
Álvaro Cabezas-Clavijo, Nazaret Barriga, Rafael Repiso
Where Do Highly Cited Scientists Stand? A Multi-Dimensional Analysis of Positional Distribution in the Top 2% Ranking
Yuehai Li, Zuguang Gu
Assessing the Reliability of Longitudinal Bibliometric Rankings under Incomplete Citation Histories
João Paulo Papa