https://shin13.github.io/tags/quest/Recent content in QUEST on Shin LiHugoen-USShin LiWed, 06 May 2026 04:22:36 +0800https://shin13.github.io/notes/framework-for-human-evaluation-of-large-language-models-in-healthcare-derived-from-literature-review/Fri, 14 Nov 2025 14:47:00 +0800https://shin13.github.io/notes/framework-for-human-evaluation-of-large-language-models-in-healthcare-derived-from-literature-review/<h2 id="paper-overview">Paper Overview</h2> <p><strong>Title:</strong> A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review</p> <p><strong>Authors:</strong> Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V. Stolyar, Katelyn Polanska, Karleigh R. McCarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, Sunyang Fu, Piyush Mathur, Giovanni E. Cacciamani, Cong Sun, Yifan Peng, and Yanshan Wang</p> <p><strong>Journal/Conference:</strong> npj Digital Medicine</p> <p><strong>Year:</strong> 2024</p> <p><strong>DOI/Link:</strong> <a href="https://doi.org/10.1038/s41746-024-01258-7">https://doi.org/10.1038/s41746-024-01258-7</a></p> <p>This scoping review analyzes 142 studies of human evaluation for healthcare LLMs and argues that current practice is inconsistent, under-specified, and often too weak for high-risk clinical use cases.</p> <h2 id="selected-figures">Selected Figures</h2> <h3 id="figure-1-healthcare-applications-of-llms">Figure 1. Healthcare applications of LLMs</h3> <p><img alt="Fig. 1: Healthcare applications of LLMs." loading="lazy" src="https://media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41746-024-01258-7/MediaObjects/41746_2024_1258_Fig1_HTML.png"></p> <p>This figure shows where human evaluation has been used most often: clinical decision support, medical education, patient education, and question answering.</p> <h3 id="figure-7-quest-human-evaluation-framework">Figure 7. QUEST human evaluation framework</h3> <p><img alt="Fig. 7: The proposed QUEST human evaluation framework, delineating the multi-stage process for evaluating healthcare-related LLMs." loading="lazy" src="https://media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41746-024-01258-7/MediaObjects/41746_2024_1258_Fig7_HTML.png"></p> <p>This is the most important figure in the paper because it turns the review findings into a practical evaluation workflow.</p> <h3 id="figure-9-prisma-flow-diagram">Figure 9. PRISMA flow diagram</h3> <p><img alt="Fig. 9: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram of the article screening and identification process." loading="lazy" src="https://media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41746-024-01258-7/MediaObjects/41746_2024_1258_Fig9_HTML.png"></p> <p>This figure summarizes the literature search and screening process behind the 142 included studies.</p>