Introduction

Mock oral examinations (MOEs) are administered by expert clinicians to prepare residents for the American Board of Surgery Certifying Examination (ABSCE), a summative assessment of safe clinical decision-making.¹ Resident participation in MOEs predicts first-time passage rates on the ABSCE^2-5 and informs program efforts to refine educational curricula.^6-8 In response to the COVID-19 pandemic, several programs successfully adapted MOEs to virtual platforms and demonstrated resident satisfaction and feasibility.^9-11 Nevertheless, MOE administration remains logistically challenging, limiting utility for resident education and assessment.

During conventional MOEs (C-MOE), two faculty examiners present case prompts to the examinee and score responses in real-time. Consequently, C-MOE administration requires numerous faculty examiners to be available simultaneously, which can be especially challenging if recruited from multiple institutions in an effort to better simulate ABSCE testing conditions. Thus, C-MOE implementation can be profoundly hindered by limited availability and time constraints among faculty and residents.^7,10,12,13 This study explores the implementation of a novel self-directed MOE (SD-MOE) format, in which examinees deliver video-recorded verbal responses to written prompts in the absence of a faculty examiner, as a potential supplemental tool given these obstacles. The SD-MOE offers several potential logistical advantages. Most notably, the use of video-recorded submissions would obviate the need for faculty examiners to personally administer the SD-MOE, instead allowing them to later view and score recordings at a time of their convenience. Although prior investigations of live versus video-based assessment have demonstrated agreement in scoring between methods, such studies have focused on assessment of technical skills rather than clinical decision-making.^14-16

The primary objective of this study was to evaluate the feasibility of the SD-MOE format as a supplemental tool during a multi-institutional MOE. As an exploratory aim, this study compared performance metrics between formats, acknowledging that removal of the live probing that occurs naturally during examiner-examinee interactions could meaningfully alter the construct being assessed. The hypothesis of this study is that the SD-MOE can be administered with minimal logistical challenges while demonstrating favorable participant satisfaction ratings.

Methods

A multi-institutional virtual MOE was administered for surgical trainees from five regional general surgery residency programs. Eligible examinees included all general surgery residents of post-graduate year (PGY) levels 3, 4, and 5 from these institutions. All examinees participated in the C-MOE and were presented the option to additionally complete the SD-MOE. Surveys were completed by faculty examiners and resident examinees at the conclusion of the entire MOE program. All case topics for the C-MOE and SD-MOE were derived from the Surgical Council on Resident Education (SCORE) curriculum and selected through consensus by the MOE organizational committee faculty. Case scenarios, prompts, and grading criteria for the C-MOE and SD-MOE were developed by expert surgeons on the organizational committee and reviewed for content accuracy and flow by authors FC, JXW, and JPW. Approval was obtained from the University of California Los Angeles Institutional Review Board (IRB #21-000707). The procedures used in this study adhere to the tenets of the Declaration of Helsinki. This study is reported in accordance with the Guideline for Reporting Evidence-based Practice Educational interventions and Teaching (GREET) 2016 checklist.

Results

A total of 55 examinees participated in the C-MOE, of which 41 (75%) also completed the SD-MOE. There was no significant difference in PGY-level between examinees who participated in the SD-MOE and those who participated only in the C-MOE (Table 1). A total of 62 faculty examiners participated in the C-MOE and 25 faculty examiners participated in the SD-MOE. Of the 25 faculty members participating in the SD-MOE, 19 (76%) also served as examiners in the C-MOE.

Of the 41 examinees who completed the post-examination survey (75% response rate), 32 (78%) participated in both the C-MOE and SD-MOE while nine (22%) participated only in the C-MOE (Table 2). There was no difference in the overall frequency of technical issues between the SD-MOE and C-MOE; six examinees reported minor technical issues with no significant impact in the SD-MOE, compared with one examinee who reported minor technical issues with no significant impact in the C-MOE. No examinees reported major technical issues in either examination format. There was no difference between formats in resident satisfaction, ability to concentrate on prompts, or perceived appropriateness of case content. Of the 32 examinees who completed the SD-MOE, 18 believed scores being derived from video-recorded responses contributed to a more efficient (56%) and fair (56%) examination. Open-text feedback from examinees communicated a desire for real-time, post-case feedback and noted that SD-MOE case flow ambiguities were difficult to navigate without the ability to ask for clarification.

Among faculty examiners, 56 of 62 (90%) C-MOE participants completed the post-examination survey and 20 of 25 (80%) SD-MOE participants completed the post-scoring survey. On average, faculty examiners required 18.5 minutes to fully score each SD-MOE video recording (Table 3). A significantly lower proportion of faculty examiners reported overall satisfaction with the SD-MOE compared with the C-MOE (65% vs. 93%, P<0.01). There was no significant difference between formats in satisfaction with case content or ease of format among faculty examiners. The SD-MOE and C-MOE were perceived as equally valid by 50% of SD-MOE faculty examiners. Open-text feedback from examiners reflected an appreciation for logistical flexibility but noted technological difficulties and the inability to probe examinees as limitations.

There was no significant difference in proportion of questions passed, mean case score, or pass rates on the C-MOE between examinees who participated in both the C-MOE and SD-MOE compared with those who participated only in the C-MOE (Table 1). PGY-level was significantly associated with mean case score (PGY-3: 2.45; PGY-4: 2.58; PGY-5: 2.70; P=0.01) and proportion of questions passed (PGY-3: 81.7%; PGY-4: 87.3%; PGY-5: 90.1%; P<0.01) in the C-MOE, but not the SD-MOE (Table 4). PGY-level was not associated with overall exam pass rate in either format. Sensitivity analyses demonstrated that there remained no significant association between PGY-level and overall exam pass rate for either format at alternative mean case score thresholds of ≥2.0 and ≥2.5. There was no significant correlation between proportion of questions passed in each format (r=0.19, P=0.24) (Figure 1). A moderate positive correlation of mean case scores was identified between formats (r=0.34, P=0.03) (Figure 2). Interrater reliability was low in both formats, as reflected by fair agreement in the SD-MOE (weighted κ=0.32, 95% CI [0.19, 0.46]) and modest-to-fair agreement in the C-MOE (weighted κ=0.25, 95% CI [0.16, 0.35]). The confidence intervals were noted to overlap, but no formal statistical comparison of kappa values between formats was performed.

Discussion

In this feasibility study of a novel self-directed MOE format for use as a supplemental tool in preparation for the ABSCE, the SD-MOE was successfully implemented during a multi-institutional MOE program without the need for faculty examiners to be present for its administration (Figure 3). There were no significant differences in self-reported technical issues between the SD-MOE and C-MOE. Among faculty examiners, satisfaction ratings were lower for the SD-MOE compared with the C-MOE. PGY-level was a significant correlate of performance in the C-MOE, but not in the SD-MOE.

Figure 1.

Correlation of Proportion of Questions Passed Across Examination Formats.

Legend: This figure depicts the correlation between the proportion of questions passed on the C-MOE and SD-MOE. Each data point represents one examinee’s performance on each examination. Abbreviations: C-MOE: conventional mock oral examination; SD-MOE: self-directed mock oral examination

Figure 2.

Correlation of Mean Case Score Achieved Across Examination Formats.

Legend: This figure depicts the correlation between the mean case score on the C-MOE and SD-MOE. Each data point represents one examinee’s performance on each examination. Abbreviations: C-MOE: conventional mock oral examination; SD-MOE: self-directed mock oral examination.

Among examinees, there was no significant difference between the SD-MOE and C-MOE with respect to overall satisfaction, ability to concentrate on prompts, and perceptions of case content appropriateness. In contrast, faculty examiner satisfaction ratings were significantly lower for the SD-MOE relative to the C-MOE. Open-text faculty survey responses indicated that some examiners were dissatisfied by the lack of bidirectional interaction, precluding their ability to probe for reasoning to better assess clinical judgment. Other examiners described difficulty navigating the video recording and scoring interface simultaneously, although some respondents appreciated the ability to replay responses. Additional sources of examiner dissatisfaction with the SD-MOE format were not clearly elucidated in this study, and potential explanations are consequently speculative. While there was no significant difference in reported ease of format between modalities, it is possible faculty examiners were dissatisfied by the asynchronous structure of the SD-MOE, which required setting aside a separate time to view and grade videos. Addressing potential sources of dissatisfaction would first require more detailed investigation into examiner perspectives, but strategies might entail both logistical and technical streamlining of the SD-MOE scoring process; for example, ensuring protected academic time to complete asynchronous grading and integrating SD-MOE video recordings and score input into a single webpage may help to improve the examiner experience.

In the C-MOE, PGY-level was correlated with both mean case score and percentage of questions passed but not with overall exam pass rates. In contrast, PGY-level was not significantly associated with any of these performance metrics in the SD-MOE. These divergent patterns are likely driven in part by the SD-MOE featuring fewer cases, thus contributing to lower statistical power. Additionally, they may also reflect that the SD-MOE and C-MOE potentially constitute different assessment constructs, not simply different delivery modalities. Navigating the unfamiliar assessment structure of the SD-MOE moreover may have introduced variability in performance and scoring among examinees and examiners, respectively. Additionally, examinees may have perceived the SD-MOE as having lower stakes and educational value in its profound departure from the standard oral examination structure, given that similarity to the ABSCE is a well-recognized benefit of C-MOEs.^17,18 It is possible that such perceptions may have attenuated examinees’ level of engagement and effort on the SD-MOE.

Figure 3.

Utility and Feasibility of the Self-Directed Mock Oral Examination

The SD-MOE presents a theoretical opportunity to decrease bias by minimizing unintentional coaching from examiners through standardized prompt presentation; however, eliminating two-way communication also removes the ability to probe for understanding of key concepts and prompt examinees to demonstrate complex judgment, two factors that are key to oral assessments.^19,20 Given that interrater reliability has been described as poor even when experienced examiners grade in isolation, the lack of interaction between faculty examiners may have also negatively impacted interrater reliability in the SD-MOE.²¹ Although beyond the scope of the present study, the inherent contrast between written SD-MOE prompts compared with verbal C-MOE prompts may have had varying impacts on performance across certain subgroups of examinees. For example, learners with dyslexia or attention deficit hyperactivity disorder and those who are non-native English speakers may favor and perform better when case information is presented verbally, rather than in written form.^22-24

There are multiple limitations to this feasibility study and the SD-MOE format. While participation in the SD-MOE was high at a rate of 75%, the opt-in nature of this study raises concern for potential selection bias. However, no significant differences were observed between examinees who participated in the SD-MOE compared with those who opted out with respect to PGY-level or C-MOE performance, as defined by mean case score, questions passed, and overall pass rate. Although the majority of SD-MOE faculty examiners participated in the C-MOE, non-identical groups of faculty examiners participating in the two MOE formats may have led to biases in scoring. In addition, a central limitation of this study is that there were fewer cases in the single SD-MOE session than each C-MOE breakout room, which likely led to compressed score ranges, attenuated correlations, and obscured PGY-level gradients. The small number of observations per SD-MOE examiner dyad also contributed to unstable estimates, which precluded reliable comparison of dyad-level kappa values between formats. Potential order effects on performance could not be investigated because the sequence in which individual examinees completed the SD-MOE relative to C-MOE cases was not recorded. While presenting identical cases in the SD-MOE and C-MOE format may have created opportunities for more robust direct comparison, this was not pursued as part of this feasibility study after considering the logistical limitations with respect to ensuring a standardized C-MOE experience for all examinees irrespective of their participation in the SD-MOE. Uneven survey response denominators between SD-MOE examinees and C-MOE-only examinees as well as between SD-MOE examiners and C-MOE examiners raise concern for potential non-response bias, limiting the reliability of between-format survey comparisons. Furthermore, survey items were not formally piloted before administration. These limitations thus create fragile survey comparisons and must be considered when interpreting results.

While the SD-MOE did not require faculty examiners for its administration, the examination process was not entirely autonomous, as non-faculty proctors were utilized in each SD-MOE session to facilitate video recording and provide logistical support as needed. Further optimizing SD-MOE administration such that support staff are not required would help to make this tool more accessible for prospective users at resource-limited institutions. The SD-MOE may also be further improved by including more dynamic functionality with standardized follow-ups to mimic examiner probing and adaptive branching logic based on examinee responses. With such improvements, residency programs could consider deploying the SD-MOE as a supplemental self-study resource in oral exam curricula and as a means of identifying residents who may benefit most from additional coaching while preparing for the C-MOE and ABSCE. Meaningful future research directions include a study with matched SD-MOE and C-MOE case counts and sets, integrating artificial intelligence-enhanced dynamic functionality, and evaluating the correlation of SD-MOE performance with ABSCE first-time pass rates.

This feasibility study demonstrates the successful implementation of the SD-MOE with minimal logistical challenges during a multi-institutional MOE. Administered in the absence of faculty examiners, the SD-MOE may offer valuable opportunities particularly in resource-limited environments for independent study, self-critique, and use of video recordings to educate other trainees.^3,25,26 Without criterion validity testing against performance on the ABSCE, however, the SD-MOE cannot be proposed as a substitute for the multi-institutional C-MOE and should be considered exclusively as a supplemental educational tool. Thus, the C-MOE should remain the preferred format for ABSCE preparation when possible.

Acknowledgments

None.

Conflict of Interest Statement & Funding

The Authors have no funding, financial relationships or conflicts of interest to disclose.

Author Contributions

Conceptualization: TP, IAH, FC. Data Curation: TP, IAH. Formal Analysis: MN. Investigation: TP, IAH, MF, JXW, JPW, AT, FC. Methodology: TP, IAH, MN. Project Administration: TP. Supervision: FC. Writing - Original Draft: TP, IAH, MF. Writing - Review Editing: TP, MN, MF, JXW, JPW, AT, FC.

References

1. Higgins RM, Deal RA, Rinewalt D, Kuchta K, Luu MB, Statter MB, et al. The utility of mock oral examinations in preparation for the American Board of Surgery certifying examination. Am J Surg. 2016;211(2):416–20.

2. Fingeret AL, Arnell T, McNelis J, Statter M, Dresner L, Widmann W. Sequential participation in a multi-institutional mock oral examination is associated with improved American Board of Surgery certifying examination first-time pass rate. J Surg Educ. 2016;73(6):e95–103.

3. Guzman E, Babakhani A, Maker VK. Improving outcomes on the ABS certifying examination: can monthly mock orals do it? J Surg Educ. 2008;65(6):441–4.

4. Nguyen J, Liu A, McKenney M, Elkbuli A. Predictive factors of first time pass rate on the American Board of Surgery certification in general surgery exams: a systematic review. J Surg Educ. 2021;78(4):1094–104.

5. Maker VK, Zahedi MM, Villines D, Maker AV. Can we predict which residents are going to pass/fail the oral boards? J Surg Educ. 2012;69(6):705–13.

6. Meyerson SL, Lipnick S, Hollinger E. The usage of mock oral examinations for program improvement. J Surg Educ. 2017;74(6):946–51.

7. Kimbrough MK, Thrush CR, Smeds MR, Cobos RJ, Harris TJ, Bentley FR. National landscape of general surgery mock oral examination practices: survey of residency program directors. J Surg Educ. 2018;75(6):e54–60.

8. Strickland C, Jensen A, McArthur T. Does the oral 'mock board' examination still have a role as a training tool? Acad Radiol. 2017;24(11):1463–7.

9. Huang IA, Lu Y, Wagner JP, Sandhu G, Chen DC, Jarman BT, et al. Multi-institutional virtual mock oral examinations for general surgery residents in the era of COVID-19. Am J Surg. 2021;221(2):429–30.

10. Shebrain S, Nava K, Munene G, Shattuck C, Collins J, Sawyer R. Virtual surgery oral board examinations in the era of COVID-19 pandemic. J Surg Educ. 2021;78(2):365–8.

11. Zemela MS, Malgor RD, Smith BK, Smeds MR. Feasibility and acceptability of virtual mock oral examinations for senior vascular surgery trainees and implications for the certifying exam. Ann Vasc Surg. 2021;76:14–9.

12. Smeds MR, Thrush C, Kimbrough MK, Moursi MM. Use of mock oral examinations in vascular surgery training programs: a nationwide survey. J Surg Res. 2018;232:94–8.

13. Peters GW, Decker RH, Park HS, Yu JB, Evans SB. Resident attitudes and benefits of mock oral board examinations in radiation oncology. BMC Med Educ. 2020;20(1):203.

14. House JB, Dooley-Hash S, Kowalenko T, Fales W, Jouriles N, Amini R, et al. Prospective comparison of live evaluation and video review in the evaluation of operator performance in a pediatric emergency airway simulation. J Grad Med Educ. 2012;4(3):312–6.

15. Patrawalla P, Eisen LA, Shiloh A, Ahmed A, Koenig SJ, Narasimhan M, et al. Development and validation of an assessment tool for competency in critical care ultrasound. J Grad Med Educ. 2015;7(4):567–73.

16. Scaffidi MA, Grover SC, Carnahan H, Khan R, Brar S, Jayaprakash A, et al. A prospective comparison of live and video-based assessments of colonoscopy performance. Gastrointest Endosc. 2018;87(3):766–75.

17. Lu Y, Miranda R, Quach C, Huang IA, Wagner JP, Chen DC, et al. Standardized multi-institutional mock oral examination: a feasible and valuable educational experience for general surgery residents. J Surg Educ. 2020;77(6):1568–76.

18. Subhas G, Yoo S, Chang YJ, Mittal VK, Healy M, Marcus C, et al. Benefits of mock oral examinations in a multi-institutional consortium for board certification in general surgery training. Am Surg. 2009;75(9):817–21.

19. Theobold AS. Oral exams: a more meaningful assessment of students' understanding. J Stat Data Sci Educ. 2021;29(2):156–9.

20. Huxham M, Campbell F, Westwood J. Oral versus written assessments: a test of student performance and attitudes. Assess Eval High Educ. 2012;37(1):125–36.

21. Jacobsohn E, Klock PA, Avidan M; Oral Examinations Group. Poor inter-rater reliability on mock anesthesia oral examinations. Can J Anaesth. 2006;53(7):659–68.

22. Waterfield J, West B. Inclusive Assessment in Higher Education: A Resource for Change. Plymouth: University of Plymouth; 2006.

23. Ramella D. Oral exams: a deeply neglected tool for formative assessment in chemistry. In: Blaser M, Clark T, Lamont L, Stewart JJ, editors. ACS Symposium Series. Vol 1340. Washington, DC: American Chemical Society; 2019. p. 79–89.

24. Weston TJ. The validity of oral accommodation in testing [dissertation]. Boulder (CO): University of Colorado; 2002.

25. Kozol R, Giles M, Voytovich A. The value of videotape in mock oral board examinations. Curr Surg. 2004;61(5):511–4.

26. Ruiz TL, Sellers B, Devarakonda A, Wehrle CJ, Arora TK. A novel mock oral curriculum for senior surgery residents: results of a pilot study. J Surg Res. 2022;277:92–9.