IJMS Title : Reliability Generalization of the Medical Student Stressor Questionnaire

Background: Medical education is known to be stressful. Thus, medical schools have begun amending curricula to incorporate holistic wellness and stress reduction. Assessing medical student stressors is key to curricula development as well as the selection of appropriate reliable measures. This study investigated reliability reporting for studies using the Medical Student Stressor Questionnaire (MSSQ), as no study of this kind currently exists via employing Reliability Generalization (RG).
Methods: A meta-analytic method, RG, was used to analyze the reliability reporting practices and reliability coefficients, in the form of Cronbach’s alpha coefficient, for the MSSQ. While a total of 18 studies were initially isolated related to the MSSQ, only those studies reporting reliability based on their sample (n = 8) were included in the final analysis. Blind coding was utilized and percent agreement among raters was excellent (95.18%).
Results: Reliability estimates reported for the total scale fell within the excellent range (Range alpha coefficient (?) = 0.800 – 0.970; Mean alpha coefficient (M?) = 0.933, Standard Deviation alpha coefficient (SD?) = 0.050). A larger percent of males was negatively correlated to academic stressors while the number of females in studies was negatively correlated with social, drive, group activities and inter/intrapersonal aspects of medical student stressors.
Conclusions: Outcomes provide useful suggestions, implications, and future recommendations regarding the use and application of the MSSQ. Medical student stress is essential to assess via measures which demonstrate robust reliability. Insights into sources of stress can offer important feedback to making specific changes to medical school curricula.

suggests medical education can taint humanism, decrease empathy, and increase rates of depression 4 and suicidal ideation. 1 With approximately 82% of medical students having some degree of distress, 5 prudent assessment of stress and follow-up is necessary to avoid the challenges of the omnipresent 6 hierarchical system of medical training. 2 Unfortunately, there is a 5.7% attrition rate in medicine, with 7 mental stressors being a significant factor. 3 The potential etiologies of stress placed on medical students 8 is innumerable and multifactorial. Stress associated with medical education can have negative effects on 9 patient care and lead to physician burnout 4 therefore, reliably assessing medical student stress can 10 reduce this negative impact and improve later clinical experiences. Research shows that demonstrating 11 attempts to reduce stress and subsequent implementation of curriculum changes can improve the well-12 being of medical students. 5

14
Given that first and second year medical school is physically and demanding, some programs in 15 the United States, such as Case Western Reserve University, developed a wellness elective for their 16 medical students and were subsequently qualitatively evaluated on their stress. 1 This wellness elective, 17 presented by physician mentors in a one-hour lecture format over six weeks, focused on topics related to 18 health and wellness outlining the stressors in medical practice and the importance of self-care. Results

19
demonstrated that medical students struggled to prioritize their own well-being with the stress of medical 20 school. The authors proposed that future studies should explore medical students' perceptions of stress 21 in an effort to promote future wellness. Based on these findings, it is clearly important to assess medical student stress in a reliable and quantitative way in order to localize and implement stress reduction  Related Stressors (GARS). The MSSQ does not appear to yield an overall score of stress, but rather 7 evaluates stress in the context of each stressor group. Stressor scores are ranged from 0 to 4, where 0 -8 1.00 = "cause mild stress," 1.01 -2.00 = "cause moderate stress," 2.01 -3.00 = "cause high stress," and 9 3.01 -4.00 = "cause severe stress". 7

11
The psychometrics of the MSSQ have been assessed among diverse medical students over the 12 past 10 years. The initial alpha coefficients established by Yusoff

21
The MSSQ was developed to help assess medical student stress. Medical students are diverse in 22 age, race, socioeconomic status, race/ethnicity, among many other demographic characteristics. These 23 demographic characteristics, also known as sample characteristics, impact reliability and therefore utility 24 of a measure like the MSSQ. Reliability is heavily influenced by the characteristics of the individuals 25 completing a measure; thus, it is essential for researchers to accurately report reliability coefficients for 26 their study samples. [16][17][18] Currently, there exists a gap in the research for the MSSQ in that there is no 27 study outlining the overall reliability of the measure and the sources of variance. Unfortunately, this critical 28 step is often overlooked as many researchers erroneously induct reliability. Reliability induction is the 29 process of inferring reliability of scores from previous studies. We commonly see researchers stating that 30 a measure is "reliable" or they may cite the initial reliability coefficients found for the measure. This 31 creates an inadequate gauge of a measure's actual reliability, knowing that reliability is unique for each 32 and every sample evaluated. This is problematic as reliability is not a product of a measure, but rather 33 dependent on the individuals examined in a given sample. It is also concerning as it develops poor 34 reliability reporting and limits the widespread utilization of a measure, like the MSSQ. Reliability IJMS analysis is to provide a current summary of reliability estimates, highlight patterns within the sample 1 characteristics that may influence reliability, and speak to reliability reporting patterns of the MSSQ 2 gathered from primary research.

17
An extensive literature search using the terms "Medical Student Stressor Questionnaire" or study. An initial total of 101 articles were reviewed, and of these, 18 were directly related to the MSSQ and were included in this RG (see Figure 1). These 18 articles were assessed by an initial coder and then 23 sorted into one of four categories: (a) articles that used the MSSQ but failed to mention reliability in any 24 form, (b) studies that indicated the instrument was reliable and/or no mention of reliability from the 25 authors' data or from a previous source that used the MSSQ, (c) articles that only presented reliability 26 coefficients from previous studies, and (d) studies that reported reliability based upon their current study 27 data. Only articles within the final category were included in the analysis for the purposes of this RG.

29
A coding sheet was developed in order to gather uniform data across the articles to be analyzed.

30
These articles were initially coded by one researcher and then blind-coded by another researcher to 31 confirm accuracy. Discrepancies were investigated and resolved among coders, as the overall percent 32 agreement among coders was 95.18%, demonstrating excellent interrater reliability. Continuous variables 33 coded included publication year, total reliability score, subscale reliability scores, sample size, and year of IJMS resolved, data was entered into Microsoft Excel and then exported to Statistical Package for Social 1 Sciences (SPSS) for statistical analyses. 14 15

16
The results outlined within this section begin by examining the overall sample size and reliability and present the mean alpha coefficients. Third, results based on correlations conducted for subscales,

19
and demographic variables with alpha coefficients are presented. Finally, variables that could not be 20 assessed are outlined.

22
Data collected for this study represented a total sample of 2,542 participants. In order to 23 determine which of the four categories publications fell within, analysis of reliability reporting practices 24 was conducted. Of the articles reviewed, 44% (N = 10 alpha coefficients; 8 studies 7,22-27 ) of the studies did 25 report a Cronbach's 18 alpha reliability coefficient for their sample. One study reported a total of three alpha 26 coefficients, whereas the remainder of studies reported one alpha coefficient. Another 11%, (n = 2) of the 27 studies reported reliability based on previous sources. Additionally, 6%, (n = 1) of studies stated "it is 28 reliable." Overall, 39%, (n = 7) of studies completely failed to mention reliability at all (see Figure 1). Fifty-29 six percent (56%) of studies failed to report reliability coefficients for their samples. The MSSQ was used 30 predominantly in studies within Malaysia (N = 9 alpha coefficients, 90%) and one study reporting reliability 31 was conducted with a sample from Aruba (N = 1; 10%). None of the studies used the MSSQ in the US.

33
A total of eight studies, providing 10 alpha coefficients, were included for analysis. All the 34 publications reporting alpha coefficients for their study sample were peer-reviewed journal articles IJMS coefficients for subscales of the MSSQ where subscales yielded low alpha coefficients (< 0.70) based on 1 study samples suggesting caution for use and interpretation of scale outcomes (see Table 2). One

19
Publication year, race/ethnicity, and year of study did not demonstrate any impact on reported 20 reliability estimates in published studies reviewed due to lack of variability in data collected (see Table 3).

21
Some sample and measurement variables were not able to be assessed due to lack of reporting which 22 included age of study participants, gender diversity (other than male or female), marital status, religion, 23 income, sampling procedure, total scale mean, and total scale standard deviation.

26
The MSSQ was initially validated and subsequently utilized in European communities as a way to  MSSQ total reliability score analysis found a significant positive correlation between percent of 8 males and total MSSQ reliability. While small in sample size, overall, these outcomes indicate that items 9 on the MSSQ may resonate more consistently with men's experience of medical stressors. Additional 10 subscale analyses found significant negative correlations between percent of males in the study and 11 reported ARS reliability coefficients. Therefore, items on the ARS subscale were less consistent with men.
Items on the ARS relate to tests, heavy workload, falling behind, receiving poor marks, needing to do 13 well, and difficulty answering questions from teachers were not consistently reflective of men's stress 14 factors related to medical education. Related to these outcomes, recent research in medical education 15 that relates to gender specific perception and attitudes toward the burdens of everyday student life 16 indicated that more male students were convinced they were superior to the other sex in handling 17 academic performance pressure. 28 Therefore, this tendency among males may help contextualize 18 inconsistent responding to items on this subscale because males may be less likely to reliably endorse 19 issues with academic performance. In addition, four subscale analyses revealed a significant negative 20 relationship between reported reliability estimates and the number of females within the studies.

22
Outcomes of this study suggest that there are gender differences for some MSSQ subscales.

23
Women's experiences in medical school were not consistently reflected by items on the IRS subscale 24 which relate to conflict with others or poor motivation to learn, nor items on the SRS subscale which are 25 associated with stress in conveying medical information to patients or answering patient questions.

26
Further, the DRS subscale (family responsibility, unwillingness to study medicine, and a parental desire to 27 study medicine) were not consistently reflective of women's drive to pursue medical education. Finally, 28 the GARS subscale assesses perceived pressure to do well by others or feelings of incompetence and 29 was not reliably reflective of stressors experienced by women in medical education. It is possible that like 30 men, women pursue medical education driven by a desire to be a helping professional and as a passion 31 for reducing inequities in health systems which are not currently reflected as items on the MSSQ.

32
Consequently, items on these subscales need further revision to more adequately assess and reflect One of the largest limitations of this reliability generalization meta-analysis was the lack of 3 reliability estimates reported within published studies utilizing the MSSQ. Over half of the studies 4 reviewed did not report reliability; they either inferred or inducted reliability or made no mention of 5 reliability at all. The RG method relies on available literature to report reliability estimates for their sample 6 and details related to study and measurement characteristics. Therefore, studies that lack such 7 information for their sample cannot be utilized. Consequently, this is a limitation to the current data 8 presented within this study, implicating that these results have limited generalizability, and thus, the 9 results should be interpreted with caution. Reliability reporting standards are not currently being upheld.  Overall, reliability estimates reported for studies included in this RG analysis were similar to those 6 initially established by Yusoff et al. 7 for the MSSQ. Total score reliability for the MSSQ falls within the 7 excellent range while subscales showed some variability. Unfortunately, many studies did not report 8 reliability, limiting the number of studies that could be included in this analysis. This is similar to previous 9 RG study findings and indicates reliability reporting practices remain low in published literature. Reliability 10 needs to be assessed as it is an integral first step towards inferring measurement validity. As such, 11 researchers are encouraged to report the reliability estimates for their study samples in order to uphold 12 reliability reporting standards. Assessing medical student stress reliably and accurately is essential to 13 designing interventions and reducing the potential resultant negative impacts that are currently present 14 within the medical education system. It is essential that more researchers use the MSSQ and report their Stressor Questionnaire (MSSQ) from a Sri Lankan medical faculty. J Taibah Univ Med Sci. 2020 Aug