USMLE Step 1 and COMLEX Level 1 Scores.
Legend: USMLE Step 1 scores (a) and COMLEX Level 1 scores (b) of the Non-Anki cohort compared to the Anki cohort.
Spencer W. Cooper1, Nicole Twardowski1, Michael Vogel1, Daniel Perling1, Rebecca Ryznar2
doi: http://dx.doi.org/10.5195/ijms.2023.1549
Volume 11, Number 4: 271-275
Received 25 05 2022; Rev-request 21 06 2022; Rev-request 24 07 2022; Rev-request 25 11 2022; Rev-request 19 10 2023; Rev-recd 30 06 2022; Rev-recd 10 09 2022; Rev-recd 27 01 2023; Rev-recd 11 11 2023; Accepted 21 12 2023
ABSTRACT
Background:Spaced repetition learning is a method of learning that relies on a fixed formula for spacing out flashcards to ensure long term retention of a topic. Anki is one such application available online that utilizes spaced repetition learning. This retrospective cohort study analyzed incorporating Anki with medical school curriculum and its effect on board examination scores. The hypothesis is that students who engaged in spaced repetition learning through Anki scored higher on licensing board exams and achieved higher GPAs than students who did not engage with this method.
Methods:The hypothesis was tested through comparison of USMLE Step 1 scores, COMLEX Level 1 scores, and GPA. The samples were selected from the Rocky Vista University Class of 2023 and split between 35 students in the Anki Cohort and 268 students in the non-Anki cohort.
Results:1 (2.8%) student failed USMLE Step 1 in the Anki cohort compared to 28 (10.94%) students who failed Step 1 in the non-Anki cohort (p > 0.05) The Anki cohort achieved a higher mean USMLE Step 1 (223.71 versus 222.58) and COMLEX Level 1 (569.51 versus 559.99). The non-Anki cohort had a higher GPA (85.60) than the Anki cohort (83.82) (p < 0.01).
Conclusion:This is the first study to evaluate for a correlation between COMLEX level 1 scores with use of spaced repetition learning. Although this study was unable to find any significant correlation between Anki use and licensing exam scores, further investigations with better control and sample size are needed.
Medical school is a unique challenge due to the growing amount of basic science and clinical knowledge that students are responsible for retaining. In the United States, medical knowledge is assessed with the use of standardized multiple-choice examinations in both osteopathic and allopathic medical schools. Allopathic students are required to take the United States Medical Licensure Examination (USMLE) Step exams 1 and 2 clinical knowledge (CK), while osteopathic students are required to take the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) Level exams 1 and 2 CK. Passing either set of exams is required for medical licensure. In 2022, USMLE Step 1 and COMLEX-USA Level 1 transitioned to a pass-fail model from a numerical score. Gauer et al. found a statistically significant difference between numerical USMLE Step 1 and USMLE Step 2 CK and specialties in the residency match, with Dermatology having the highest mean score and Family Medicine having the lowest mean score.1 In the past, the numerical USMLE Step 1 score was weighted more heavily in garnering residency positions in competitive specialties.1 With the transition to pass-fail it is thought that residencies will ultimately look to USMLE Step 2 CK instead. Guiot et al. demonstrated that USMLE Step 1 score is a predictor of USMLE Step 2 performance.2
Enhancing medical student performance on medical school board exams by increasing retention of basic science medical knowledge is a challenge that medical educators still face.3 One such way to improve retention of medical knowledge is using retrieval practice also known as testing effect.4 Retrieval practice is the idea that testing one's knowledge will lead to gains in memory retention. This method of knowledge retention is often paired with another modality known as spaced repetition. Spaced repetition stems from the idea of spacing effect. The spacing effect is the idea that educational encounters spaced and repeated over time, results in improved and efficient knowledge retention.5,6 Kerfoot et al. demonstrated that spaced repetition in medical students significantly improves long-term memory of medical knowledge.7 This idea of spaced repetition was further demonstrated by Karpicke et al. to have powerful effects on retention regardless of the schedule of repeated tests.8
One way to implement spaced repetition with retrieval practice into study routine is to use flash cards. Senzaki et al. demonstrated that flashcards can be an effective way to retain long-term knowledge as long as they are utilized with the intent to practice retrieval of knowledge with emphasis on retention, comprehension, and application of knowledge.9 For medical students, a popular way to implement flash cards into their routine is with online applications such as Anki (https://apps.ankiweb.net/). Anki is a flashcard application that can be downloaded on to computers, phones, and tablets. It allows students to create their own flashcards quickly and download flashcards that other students have created. There is a growing number of free flash card decks related to medical school board exams including both the USMLE Step 1 and Step 2CK as well as COMLEX-USA Level 1 and Level 2CK.
The purpose of this study is to analyze the effect of correlating outside spaced repetition learning through Anki synchronized with medical school curriculum on board examination scores. We hypothesized that the cohort of students using Anki for lecture and board studying would ultimately perform better and achiever higher marks on USMLE Step 1, COMLEX-USA Level 1, and grade point average (GPA).
This retrospective cohort study was performed using institutional data from Rocky Vista University College of Osteopathic Medicine (RVUCOM). The Institutional Review Board for RVUCOM approved the study protocol (IRB #2021-0047). The data was de-identified before being provided to the researchers through the office of the registrar at Rocky Vista University.
The population selected to analyze the effect of spaced repetition learning consisted of the Rocky Vista College of Osteopathic Medicine Class of 2023 and were split into two cohorts. The Anki cohort consisted of a collaborative group of students who separated flashcards that correlate with the curriculum and studied for both licensing board exams and exams based on lecture materials through the Anki platform. A different member of the group was assigned to each lecture given in the Rocky Vista curriculum and tagged the relevant Anki cards from that lecture. The second cohort consisted of the remaining members of the Class of 2023. The students in the Anki cohort were committed to keeping up with the responsibility of separating flashcards and therefore Anki was incorporated into their study plans. The non-Anki cohort used a variety of methods to study for both board and class exams. The two cohorts took the exams at the same time of year (May to August 2021) reducing any potential bias based on variations in the annual test forms and board preparation materials.
The inclusion criteria for the Anki cohort included students who participated in separating flashcards from the beginning of the second year (July 20, 2020) through an end date of March 5, 2021. This was chosen as an end date to include students who participated during a majority of the second-year curriculum. The exclusion criteria included any student under the age of 18 at the time of enrollment in medical school to meet standard IRB guidelines for research on adults. The final sample selected for the Anki cohort included 35 students (27 males and 8 females). The inclusion criteria for the non-Anki cohort included students aged 18 and older who did not participate the flashcard collaboration. The exclusion criteria again included any student under the age of 18. There were 268 students who matched the inclusion criteria for the non-Anki cohort, however there were 20 students in the non-Anki cohort who did not have a score available for the USMLE Step 1. Therefore, there were 248 students in the non-Anki cohort for the USMLE Step 1 comparison (136 males and 112 females), and there were 268 students in the non-Anki cohort for the COMLEX Level 1 comparison (144 males and 124 females).
To compare performance on USMLE Step 1 and COMLEX Level 1, the primary outcomes investigated were score attained and pass or fail on first attempt. For the Class of 2023, the passing scores were 194 or above on USMLE Step 1 or a score of 400 or above on COMLEX Level 1. The raw board scores of the Anki and non-Anki cohorts were compared through two sample unequal variance t-tests. The number of failures per cohort were analyzed through direct comparison with a chi-square test.
For performance in the school curriculum, the grade point average (GPA) and number of courses failed were the primary outcomes examined. GPA was compared through a two-sample unequal variance t-test and the number of courses failed were compared with a chi-square test. GPA was calculated from the scores of individual courses in the cohorts and were cumulative through the end of the second year of the curriculum. The total amount of courses failed were calculated for each cohort and included courses that were withdrawn from or dropped.
For the fail rate of USMLE Step 1 in the Anki cohort, 34 subjects (97.1%) passed the exam and 1 (2.9%) student failed. In the non-Anki cohort, 228 students (91.94%) passed the exam and 20 students (8.06%) failed (Table 1). This difference was not significant (p > 0.05). Both sample sizes were sufficiently large to assume normality within the data, and this assumption allowed mean scores to be compared directly. When comparing numerical USMLE Step 1 scores, the mean of the non-Anki cohort was 222.58 and the mean of the Anki cohort was 223.71 (Figure 1). However, the difference was not significant (p <0.05, 95% CI - 7.27, 5.01).
Table 1Comparison of USMLE Step 1 Examination Passing Rate.
| Cohort | Passed (%) | Failed | P-Value |
|---|---|---|---|
| Anki Cohort | 34 (97.1%) | 1 (2.9%) | p > 0.05 |
| Non-Anki Cohort | 228 (91.9%) | 20 (8.1%) |
USMLE Step 1 and COMLEX Level 1 Scores.
When comparing COMLEX scores, the mean score of the non-Anki cohort was 559.99, and the mean of the Anki cohort was 569.51 (Figure 1) (p > 0.05, 95% CI -34.78, 15.73). No subjects within either cohort failed COMLEX, so there is no comparison in that category for COMLEX as they were equal.
When comparing the course results, the cumulative number of passed courses in the non-Anki cohort was 8567 and the number of failed courses was 113. In the Anki cohort, there were 1109 courses passed and 11 failed. When comparing failed courses, the percent of failed courses in the Anki cohort was 1.3% and the percent of failed courses in the non-Anki cohort was 0.99% (p > 0.05, 95% CI 0.66, 2.25).
In terms of the GPA of subjects in the non-Anki vs the Anki cohort, the non-Anki cohort had a higher GPA at 85.60 than the Anki cohort at 83.82 (Figure 2). The grade difference was statistically significant (p < 0.01, 95% CI -3.10, -0.45).
Figure 2Career GPA.
Several studies have demonstrated the benefit of using spaced repetition for improved board score performance. The Anki cohort in this study committed to using spaced repetition flashcards that associated with what was being taught in the school curriculum. The students in the Anki cohort used the same uniform set of flashcards to study for each exam and their board exams. In dividing the work of finding relevant Anki flashcards between the 35 participants in the Anki cohort, there may have been a theoretical benefit of more free time to spend studying leading to potentially higher board scores. The method presented is novel to board exam studying in connecting outside resources with what is taught in the curriculum. The results of the study could not confirm a benefit of using Anki spaced repetition to pass the USMLE examination. However, the results suggest a benefit of increased board scores that may be confirmed in a follow-up study. Students who fail the USMLE exam on their first attempt may have difficulty matching into residency programs. One study found that 5/64 students who failed USMLE Step 1 on their first attempt failed to match into a residency program compared to all students in the pass group who matched for residency.12
This is the first study to evaluate for a correlation between the osteopathic medical school (COMLEX) board examination scores with use of Spaced Repetition memory aids such as Anki. This study was unable to confirm the benefit of Anki in improving COMLEX board scores, nevertheless the board scores of those in the Anki cohort were higher on average compared to the non-Anki cohort (569.51 versus 559.99). Further analysis will be required to verify these findings as there are several limitations to this study that are discussed later.
An interesting result found was that those in the Anki cohort had a lower GPA compared to the non-Anki cohort. This may be due to the spaced repetition goal of the Anki program. Using Anki to study requires students to be reviewing information on a variety of subjects, while the classroom examinations were systems focused. Therefore, the students in the Anki cohort were spending time learning/reviewing information and topics every day that were not related to upcoming material tested on the systems-based examinations. If course tests and quizzes are given at relatively shorter intervals there is less time for spaced repetition learning to occur. In recent years, more literature has come out in search of correlating flashcard use with academic performance in medical school. The findings are consistent with a study conducted by Sun et al. which showed that first year medical students enjoyed using flashcards to study, but did not show an association with higher class exam scores.10
There were several limitations in conducting this study that may have led to a lack of statistical significance comparing the USMLE or COMLEX examination scores. First, this study would have been better performed as a prospective cohort study instead of a retrospective cohort study. This would have allowed for stratifying the students into cohorts that definitively differentiate the study materials used by subjects in both groups. The study described in this paper relied on comparing a cohort where Anki use was emphasized to a different cohort that had no data on study materials used. Second, the sample size of the students who were in the Anki cohort was much smaller than the non-Anki cohort which could have influenced the power of the results. Third, there was no way to measure daily compliance to the Anki flashcards, and the settings of each Anki program were up to the individual user with no standardized format. Finally, there was no way to control for extracurricular activities and if there was any influence on course or licensing exam performance.
In the future, subsequent studies designed as a prospective cohort study or randomized control trial may benefit from improved stratification of those who used spaced repetition software, such as Anki, who were not in the Anki cohort to determine whether their board scores, GPA, or pass rates differ from those in the Anki cohort. As Step 1 and Level 1 move to pass fail, the effect could be targeted for the subsequent licensing exams that are still scored: Step 2, Step 3, Level 2, or Level 3.
The results indicated a higher pass rate, but showed lower GPA compared to those who were not in the Anki cohort. The difference in GPA could be partially explained by the wide variety of flashcards reviewed each day compared to the more focused nature of examinations that the GPA relied on. There was a difference suggested in USMLE Step 1 or COMLEX Level 1 scores, although this difference was not statistically significant. Future studies with more robust controls will be able to further define improvements in exam scores.
The Effect of Spaced Repetition Learning Through Anki on Medical Board Exam Performance
Medical school is a unique challenge due to the growing amount of basic science and clinical knowledge that students are responsible for retaining. Spaced repetition learning is a method of learning that relies on a fixed formula for spacing out flashcards to ensure long term retention of a topic. Anki is one such application available online that utilizes spaced repetition learning. This retrospective cohort study analyzed incorporating Anki with medical school curriculum and its effect on board examination scores. The hypothesis is that students who engaged in spaced repetition learning through Anki scored higher on licensing board exams and in their overall career GPA than students who did not engage with this method. The hypothesis was tested through comparison of USMLE Step 1 scores, COMLEX Level 1 scores, and GPA. The samples were selected from the Rocky Vista University Class of 2023 and split between 35 students in the Anki cohort 268 students in the non-Anki cohort. 1 (2.8%) student failed USMLE Step 1 in the Anki cohort compared to 28 (10.94%) students who failed Step 1 in the non-Anki cohort (P-value 0.13) The Anki cohort scored higher on both USMLE Step 1 (223.71 versus 222.58) and COMLEX Level 1 (569.51 versus 559.99) (p>0.05). The non-Anki cohort had a higher GPA (85.60) than the Anki cohort (83.82) (p<0.05). The difference in GPA could be partially explained by the wide variety of flashcards reviewed each day. This study was unable to find a benefit of using spaced repetition to pass the USMLE examination. This is the first study to evaluate for a correlation between COMLEX level 1 scores with use of spaced repetition learning. Additionally, the study pointed towards an improvement in USMLE Step 1 and COMLEX Level 1 scores that can be investigated in the future.
The authors thank the Rocky Vista Office of the Registrar members David Paltza and Keziah Garcia for their support of this research and providing the data needed.
The Authors have no funding, financial relationships or conflicts of interest to disclose.
Conceptualization: SWC, NT, MV, DP, RR. Data Curation: SWC, NT. Formal Analysis: SWC. Investigation: SWC, NT. Methodology: SWC, NT, MV, DP, RR. Supervision: RR. Writing - Original Draft: SWC, NT, MV, DP. Writing - Review Editing: RR.
1. Gauer, J. L., & Jackson, J. B. The association of USMLE Step 1 and Step 2 CK scores with residency match specialty and location. Med Educ Online. 2017;22(1), 1358579.
2. Guiot HM, Franqui-Rivera H. Predicting performance on the United States Medical Licensing Examination Step 1 and Step 2 Clinical Knowledge using results from previous examinations. Adv Med Educ Pract. 2018;14(9):943–9.
3. Schneid, S. D., Pashler, H., & Armour, C. How much basic science content do second-year medical students remember from their first year?. Med Teach. 2019; 41(2), 231–3.
4. Roediger HL III, Karpicke JD. The power of testing memory: basic research and implications for educational practice. Perspect Psychol Sci. 2006;1(3):181–210.
5. Glenberg AM, Lehmann TS. Spacing repetitions over 1 week. Mem Cogn 1980;8:528–38.
6. Toppino TC, Kasserman JE, Mracek WA. The effect of spacing repetitions on the recognition memory of young children and adults. J Exp Child Psychol 1991;51:123–38.
7. Kerfoot BP, DeWolf WC, Masser BA, Church PA, Federman DD. Spaced education improves the retention of clinical knowledge by medical students: a randomised controlled trial. Med Educ. 2007;41(1):23–31.
8. Karpicke JD, Bauernschmidt A. Spaced retrieval: absolute spacing enhances learning regardless of relative spacing. J Exp Psychol Learn Mem Cogn. 2011;37(5):1250–7.
9. Senzaki S, Hackathorn J, Appleby DC, Gurung RAR. Reinventing flashcards to increase student learning. Psychol Learn Teach. 2017;16(3):353–68.
10. Sun M, Tsai S, Engle DL, Holmer S. Spaced repetition flashcards for teaching medical students psychiatry. Med Sci Educ. 2021;31:1125–31.
11. Lu, M., Farhat, J. H., & Beck Dallaghan, G. L. Enhanced Learning and Retention of Medical Knowledge Using the Mobile Flash card Application Anki. Medical science educator. 2021;31(6), 1975–81.
12. Biskobing DM, Lawson SR, Messmer JM, Hoban JD. Study of Selected Outcomes of Medical Students Who Fail USMLE Step 1. Med Educ Online. 2006;11(1):4589.
13. AACOM Reports on Student Enrollment. Trends in osteopathic medical school applicants, applications, enrollment, and graduates. Available: https://www.aacom.org/reports-programs-initiatives/aacom-reports/student-enrollment. Last updated April 28, 2020.
Spencer W. Cooper, 1 Third year Medical Student, Rocky Vista University College of Osteopathic Medicine, Parker, CO, USA.
Nicole Twardowski, 1 Third year Medical Student, Rocky Vista University College of Osteopathic Medicine, Parker, CO, USA.
Michael Vogel, 1 Third year Medical Student, Rocky Vista University College of Osteopathic Medicine, Parker, CO, USA.
Daniel Perling, 1 Third year Medical Student, Rocky Vista University College of Osteopathic Medicine, Parker, CO, USA.
Rebecca Ryznar, 2 PhD, Rocky Vista University College of Osteopathic Medicine, Parker, CO, USA.
About the Author: Spencer Cooper is currently a third year medical student of Rocky Vista University College of Osteopathic Medicine, Parker, CO, USA of a four year program. He is a member of the Physician-Scientist extracurricular track focusing on providing basic training in the knowledge and skills necessary for success as an academic physician.
Correspondence: Spencer Cooper. Address: 8401 S Chambers Rd, Parker, CO 80134, United States. Email: spencer.cooper@rvu.edu
Editor: Francisco J. Bonilla-Escobar; Student Editors: Richard Christian Suteja, Praise Senyuy Wah, Andrew Elton; Layout Editor: Ana M. Morales; Process: Peer-reviewed
Cite as Cooper SW, Twardowski N, Vogel M, Perling D, Ryznar R. The Effect of Spaced Repetition Learning Through Anki on Medical Board Exam Performance. Int J Med Stud. 2023 Oct-Dec;11(4):271-5.
Copyright © 2023 Spencer W. Cooper, Nicole Twardowski, Michael Vogel, Daniel Perling, Rebecca Ryznar
This work is licensed under a Creative Commons Attribution 4.0 International License.
International Journal of Medical Students, VOLUME 11, NUMBER 4, December 2023