Exploring student perspectives and factors that may influence performance in multiple-choice physiology assessments
OA Version
Citation
Abstract
Graduate and professional level students are frequently evaluated through multiple-choice question (MCQ) examinations. While MCQs provide efficiency, they may present challenges for test-takers. Item writing flaws in MCQs can introduce irrelevant difficulty, or advantages for students with strong test-taking strategies, potentially compromising the fairness and effectiveness of the assessment. Through this mixed methods study, we explored the student perspective of flawed MCQs and investigated the influence item writing flaws have on test performance. A total of thirty-one (31) first-year dental students from Boston University Henry M. Goldman School of Dental Medicine and graduate students in the medical sciences program at Boston University Chobanian & Avedisian School of Medicine completed a 20-question physiology assessment and demographic survey. An evaluation instrument based on the National Board of Medical Examiners (NBME) item writing guide was employed to identify the item writing flaws in each question. Quantitative analysis included: independent two-sample t-tests, Spearman's rank correlation, and Mann-Whitney U test, which were used to evaluate demographic and item analysis factors. Of the students who participated in the assessment and survey, thirteen (13) students participated in semi structured focus group discussions. An inductive thematic analysis was conducted to identify the themes that emerged from the data. Based on the evaluation instrument, we found 75% of the physiology assessment contained at least one item flaw. No significant differences in test performance within demographic groups and no significant relationship between the number of flaws and item difficulty were observed. However, a moderately positive correlation between the number of flaws and point biserial correlation was observed. We also identified differences in point biserial correlation with items containing unclear stems and lead-ins. Through the focus group discussions, participants identified the role of the instructor in facilitating success, discussed their own test-taking strategies and study habits that help them to navigate item flaws, and recognized the outside factors that contribute to student performance. This study highlights the significant role item writing flaws in MCQs can play in influencing test outcomes. Assessments play an important role in decision-making processes and in shaping healthcare students’ career trajectories. Ensuring that assessments are equitable may help diversify the healthcare workforce, which is increasingly needed to match the growing diversity of the U.S. population. These findings highlight the importance of question design in assessments.
Description
2025