Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation
Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Damir Juric, Jack Flann, Ehud Reiter, Anya Belz, Aleksandar Savkov
Abstract
In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to understand the impact they may have on both the clinician using them and the patient's clinical safety. To address this we present an extensive human evaluation study of consultation notes where 5 clinicians (i) listen to 57 mock consultations, (ii) write their own notes, (iii) post-edit a number of automatically generated notes, and (iv) extract all the errors, both quantitative and qualitative. We then carry out a correlation study with 18 automatic quality metrics and the human judgements. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore. All our findings and annotations are open-sourced.
Create a lesson
Related papers
Mechanistic Interpretability Needs a Logic of Inference
Zhimin Hu, Chen Jiang, Hanbo Xie et al.
Pterodactyl: Pushing the Limits of Efficient Automated Translation Evaluation
Evelyn Y. Garland, Carola F. Berger
When does text encoding matter for language modeling?
Eugene Jang, Catherine Arnett, Terra Blevins
Blurring the Block Boundaries: Overlapping Block Denoising for Efficient Diffusion Language Models
Amir Mann, Jonathan Fhima, Shaked Brody et al.
ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces
Jiahang Lin, Wanying Ren, Wanghan Xu et al.
Point-in-Time Encoders for Earnings Calls
Alessio Brini