The effects of "score creep". Trends in residency selection criteria

Every other year the National Resident Matching Program (NRMP) publishes a document called Charting Outcomes in the Match. This is accessible through http://www.nrmp.org/data/index.html. The document analyzes many datapoints of interest to medical students, medical schools, and residency programs. For example, many senior medical students look at the average step 1 and step 2 scores, number of research experiences, and number of programs needed to rank to feel comfortable matching. These indicators are all reported by individual specialty. 

A discussion on SDN sparked an interesting debate and analysis about the results of score creep on MD/PhDs. 

http://forums.studentdoctor.net/showthread.php?t=855953

As I have written previously, clinical indicators of performance such as step 1 and step 2 scores are those most important for obtaining a residency position. Score creep is the effect that many MD/PhDs observe during their 7-9 year MD/PhD programs, where residencies seem to get more competitive each year through the long program.

To evaluate whether this "score creep" truly exists, I performed an analysis of markers of competition within Charting Outcomes in the Match over the time period since the beginning of Charting Outcomes analysis of 2005 residency match data to the most recent publication in 2011. I found dramatic evidence for this effect, as shown below.


After discussion with one of several program faculty who posts frequently on SDN, we decided to write a manuscript and publish this data to demonstrate two things. First, we wanted to show that residencies are overall becoming more competitive over time. This is likely due to expansion of medical schools and class sizes, especially osteopathic, with a relatively unchanged number of residency positions. I also personally believe that almost everyone believes step 1 is the single most important factor in residency selection. Thus, Step 1 specific preparation seems to increase every year--with regards to increased enrollment in question banks and other formal review courses, amount of time allotted by the medical schools to allow students to take off to study for the exam, and cirricula revisions to focus more on Step 1 material. I think this creates an artificial distraction from the true goals of medical education for the single purpose of creating a benchmark of competition that most would argue has little reflection of a medical student's future potential as a physician.


Second, our editorial position is that the "score creep" is a problem particularly for MD/PhDs. That is, MD/PhDs take the step 1 exam after second year yet compete with medical students for residency after a four year delay (the PhD). Thus, the MD/PhD student's score may not be as impressive when they graduate. Further, advising at the time of taking the step 1 may not be current by the time of graduation. For example, our MD/PhD program director frequently told us that step 1 score was unimportant as long as we passed. As I have written about previously, this is completely untrue. But this probably was reasonably true until about 10 years ago.

I submitted the following manuscript to three journals and had little luck with it. The first journal returned it without review. The second diplomatically declined to publish it as it was not felt to be relevant to residency programs. The third journal took six months to send back a review that was so off-topic I think they may have sent me the review to someone else's manuscript. Still, I think this data is important and relevant to the pre-medical community. It is self-published below.


There are a few benefits to self-publication. First, I can put all the figures in color. Second, I made a supplemental section with additional figures to show all of the data from Charting Outcomes for matched US seniors. See the very bottom for the supplement.

Trends in residency selection criteria for US allopathic medical students between 2005 - 2011

By: Neuronix (neuronix@neuronix.org) and Fencer of the Student Doctor Network Physician Scientist Forum

Abstract

Background

From 2005 to 2011, the number of National Resident Match Program (NRMP) main match total applicants has climbed 18.4% to its highest point despite only an 8.9% growth in the number of residency positions. Given this disproportionate increase in match applicants, the purpose of this manuscript is to examine objective measures of competition for specialties among matched US allopathic medical school seniors during this period.

Results

Beginning in 2005, the biannual NRMP editions of “Charting Outcomes in the Match” have provided objective application parameters for the 2005, 2007, 2009, and 2011 NRMP main residency matches. The authors compiled and analyzed data for US allopathic seniors over this time to elucidate trends in residency competitiveness.

From the 2007 to the 2011 match, United States Medical Licensing Examination Step 1 and 2 scores increased an average of 5.1 and 9.3 points respectively across 18 specialties. The number of contiguous ranks increased by 1.2 programs; the number of research publications, abstracts, and presentations also increased by 1.2. There is a consistent upward trend from 2005 to 2011 in these parameters for most specialties. Changes across 11 objective parameters are also shown for all 18 specialties.

Conclusions

NRMP residency match competition as measured by several objective parameters has increased over the 2005 to 2011 time period. Advisors and students should prepare for future residency applications with consideration of these trends.

Background

Established in 1952, the National Resident Matching Program (NRMP) has become the primary matching service used by senior medical students to apply to twenty distinct medical specialties in 2011. Numerous surveys have emphasized the importance of factors such as United States Medical Licensing Examination (USMLE) scores and clinical grades in the selection of medical students by residency programs within the NRMP match [1-3]. However, little objective data existed before 2006 to provide insight on how to best prepare for residency applications. This gap was resolved by the publication of “Charting Outcomes in the Match” by the NRMP, which examines various parameters used in the selection of residents and reflects competitiveness among specialties. The NRMP data has been published online in four editions: 2006 (based on 2005 match data), 2007, 2009, and 2011, and is presented with slight variations between editions [4-7].

The period from 2005 to 2011 has been marked by a steady rise in the number of residency hopefuls. These applicants are comprised of seniors of US allopathic medical schools (US seniors) as well as former graduates of US allopathic medical schools, osteopathic seniors and graduates, US citizen international medical seniors and graduates, and non-US citizen foreign medical seniors and graduates. Since 2005, the total number of NRMP applicants has steadily grown 18.4% from 31,862 to 37,735--the highest number of applicants for any historical year [8,9]. Yet the number of residency positions during that same period only rose 8.9% from 24,012 to 26,158. The growth in applicants includes US seniors, increasing 10.4% from 15,308 to 16,893, and osteopathic seniors and graduates, rising 53.8% from 2,043 to 3,142. This discrepancy between applicants and residency positions is expected to worsen, as medical school graduates are projected to increase while funding for residency programs is in jeopardy [10]. Given these trends, it is suspected that competition for residency positions is increasing.

Patterns of change in residency candidate selection are importan0074 to medical students, faculty advisors of medical students, and residency program directors. Anecdotal experience indicates that some medical schools have increased the amount of allowed time for students to study for both the Step 1 and Step 2 [11], perhaps in response to internal data about the competitiveness of graduates. Anticipating future residency applicant needs will allow for medical school administrators and faculty to plan for curriculum modifications. In addition, the changes in competitiveness are of particular importance for medical students taking gaps in medical school to pursue other training activities such as a dual degree in science (i.e.: PhD). Personal communications from MD/PhD program faculty and students suggest increases in competition for residency positions and increased difficulty for matching for MD/PhD students due to increasing standards for Step 1 scores by residency programs. For example, a dual degree student would have completed their Step 1 examination in 2005, and then competed for residency positions in the 2011 match in a typical eight year MD/PhD program. This is compared to MD only peers who would complete the Step 1 examination in 2005 and match in 2007. Tracking of current trends may allow for MD/PhD students to speculate upon levels of competition for future residency applications, and adjust their actions accordingly to remain competitive for academic residency programs.

Methods

Data is compiled from the publicly available (from the NRMP website: www.nrmp.org) NRMP Charting Outcomes in the Match 2006 (2005 data), 2007, 2009, and 2011 into a Microsoft Excel 2010 spreadsheet. The 2005 data is extrapolated from figure 2, 4, 6, and 8 with conversion of plotted values into numeric values by electronic ruler based measurements using Adobe Photoshop CS5. Some variations in data reporting are preserved in the current article. Specifically, 2005 Step 1 score and 2007 number of contiguous ranks data is only reported as a median score, while values in other years are reported as means. Graph data reflects that some specialties are included and excluded in different editions.

From the 2007 and 2011 tables for each individual specialty for seniors of U.S. allopathic medical schools the following parameters are subtracted to create a summary table: number of contiguous ranks, research experiences, Step 1 and Step 2 scores, number of abstracts, presentations, and publications, percentage of AOA members, percentage from a top 40 US medical school ranked by NIH funding, percentage with a PhD, and percentage with another graduate degree. The number of contiguous ranks is a count of the number of sequential ranked programs in the applicant’s preferred specialty. In addition, overall numbers of MD/PhDs matched and unmatched have been calculated from the overall tables. The average numbers of work and volunteer experiences are not included as those values were only reported in 2009 and 2011.

Results

The absolute changes in parameters for US seniors from Charting Outcomes 2007 to 2011 are summarized in Table 1. The 2007 data is used for this analysis as the 2005 data includes many fewer individual parameters for matched US seniors. Over the 2007 to 2011 time period, for 18 medical specialties mean Step 1 score increased 5.1 points, mean Step 2 increased 9.3 points, the number of contiguous ranks increased by 1.2 programs, and the mean number of abstracts, presentations, and publications increased by 1.2 per student. Analysis of the six specialties with a Step 1 average greater than 230 in 2011 does not reveal distinctly increased rates of change for the most competitive specialties. Instead, all specialties increased in value in almost all of the Charting Outcomes indicators of competitiveness. The largest Step 1 and Step 2 score increases are seen in Internal Medicine-Pediatrics combined programs and Anesthesiology.

Table 1 Absolute average changes for US Seniors by specialty between the 2007 and 2011 match.

Step 1 – mean USMLE Step 1 score. Step 2 – mean USMLE Step 2 score. Ranks – number of contiguous ranks (2007 median, 2011 mean). Specialties – mean number of distinct specialties ranked. Articles – Mean number of abstracts, presentations, and publications. Research – mean number of research experiences. AOA – Percentage who are AOA members. 40 Med – Percentage who graduated from a top 40 NIH research medical school. PhD – Percentage who have a PhD degree. Grad Degree – Percentage who have another graduate degree. Matched – Percentage of US Seniors who matched to their preferred specialty.

(*) Specialties with a 2011 Step 1 average of 231 or above.

Charting Outcomes Step 1 averages for each specialty from 2005 to 2011 are shown in Figure 1. Almost all specialties have a clear trend of increasing Step 1 score of matched applicants. 

Figure 1

Figure 1 USMLE Step 1 examination score averages for matched US seniors by specialty. Data for all specialties was not reported for all years. The 2005 Step 1 scores were reported as median scores, while data was reported as means for other years.

The Step 2 averages are shown in Figure 2. The 2005 Charting Outcomes did not include Step 2 average data and is omitted from the graph. Every specialty increased its mean Step 2 score from 2007 to 2009 and 2009 to 2011 for matched US seniors. 

Figure 2

Figure 2 USMLE Step 2 examination score averages for matched US seniors by specialty. Step 2 scores were not reported in the 2005 data set. Data for neurosurgery was not reported in 2007.

Figure 3 indicates the nearly consistent upward trend in the number of contiguous ranks across specialties from 2005 to 2011 for matched US seniors. 

Figure 3

Figure 3 Average number of contiguous ranks for matched US seniors by specialty. Data was reported as median number in 2007, and was not reported for all specialties in all years.


Figure 4 demonstrates an increased amount of research activity for residency applicants from 2007 to 2011 in every specialty. This could be in part explained by a steady increase of matching US Seniors with a PhD within the match over this period. In 2007, 2009, and 2011 there were 531, 573, and 626 matched US senior MD/PhDs respectively. However, the number of MD/PhDs who failed to match also grew during this period from 5.7% (32 unmatched of 531) in 2007 to 8.2% (51 of 573) in 2009 and to 6.8% (46 of 626) in 2011.

Figure 4

Figure 4 Mean number of abstracts, presentations, and posters for matched US seniors by specialty. This data was not presented in 2005, and is not available for neurosurgery in 2007.

Discussion

This analysis of trends in the NRMP Charting Outcomes in the Match data from 2005 to 2011 indicates increasing competitiveness among US seniors seeking residency positions as exemplified by rising USMLE Step 1 scores, Step 2 scores, number of contiguous ranks, and number of abstracts, presentations, and publications in almost every specialty that uses the NRMP match. While this seems concerning for US seniors, the overall match rate of 94.1% for US seniors has remained nearly constant during this interval. Instead, other match applicants have experienced more difficulties, as non-US citizen and US citizen international medical graduates suffered a decline in match rate from 55.6% to 40.9% and 54.7% to 50.0% during this period, respectively. Nevertheless, the growth of US allopathic and osteopathic graduates outpaces the formation of new residency positions. In 2015 or sooner, it is projected that there will be more US medical school graduates than residency positions leading to board certification¹⁰. Thus, the trends shown here can be expected to continue as competition further increases.

The increase in Step 1 and Step 2 scores for US seniors is likely due to a number of factors. The most obvious is increased numbers of match applicants in the setting of residency program emphasis on Step scores for resident selection. While there are no national statistics on the number of residency applicants who use commercial USMLE preparation services, several commercial question banks are currently very popular [12]. These question banks attempt to mimic the format and question content of the actual USMLE examinations. In addition, many medical schools have increased the amount of preparation time allowed specifically preparing for the Step 1 and Step 2. In particular, Step 2 seems to be gaining importance rapidly in residency applications. Charting outcomes indicates that the number of US seniors releasing a Step 2 score to the match from 2007 to 2011 has increased from 69% to 83% of applicants. While most programs currently do not require a Step 2 score for ranking, the University of California at San Francisco began requiring Step 2 scores to rank applicants in any specialty beginning for 2010 match applicants.

The number of contiguous ranks is a count of the number of sequential ranked programs in the applicant’s preferred specialty. The simplest explanation for an increase in this indicator is that applicants are interviewing at an average of 1.2 more programs per specialty as a result of applying to and interviewing with more programs. In theory, changes could also be explained by applicants who are currently ranking programs for which they have not interviewed. This is unlikely as applicants are routinely told that programs typically do not rank applicants they have not interviewed. Another potential interpretation is that the increase in the number of contiguous ranks reflects a change in preference by applicants ranking more broadly than in prior years. The net result for residency programs has been an increase in the overall rate of filled residency positions during this period of time from 92.5% in 2005 to 95.6% in 2011.

An increase in abstracts, posters, and presentations by applicants would seem to indicate an increased emphasis on research by medical students. However, residency programs should be cautioned when interpreting quantity of research output, as it is clear that some applicants misrepresent their research experiences [13-17]. This might reflect cumulative individual responses to increase the competitiveness of the applicant in the setting of overall increased levels of competition for residency positions. 

While Charting Outcomes data does not separate dual degree program students and those who obtained their PhD before medical school, the increase in failure to match from 5.7% in 2007 to 8.2% for match applicants with a PhD in 2009 was nevertheless concerning. The typical MD/PhD program requires an average of four years of additional training with most students taking the USMLE Step 1 examination after completing their first two years of medical school. MD/PhD students then compete for entry into residency four years after their MD-only peers. Residency programs compare Step 1 scores from MD and MD/PhD seniors at the same time, despite the demonstrated specialty average increase in scores during the time the MD/PhD student is completing his or her PhD. A potential improvement to the USMLE score reporting format is to provide “true” percentiles for the Step 1 scores, as those numbers would more likely describe the meaning of a particular score in relation to the cohort of students who took that particular examination.

It is hoped that residency program directors would understand that the MD/PhD student trajectory is towards a highly academic career, and thus may not have prepared as heavily for the USMLE examinations. A traditional model may have been for MD/PhD students to focus upon medical school competency, not excellence, while using extra study time to prepare additional research training and experience. However, anecdotal reports indicate that many academic residency programs still expect an above average Step 1 and Step 2 score from their MD/PhD applicants. In addition, the academic potential of the MD/PhD might be difficult to compare to medical students who are shown to have an increasing number of abstracts, presentations, and publications. To combat increasing difficulties in matching, advisors must be critical of an applicant’s chances of success and provide appropriate guidance. For example, diagnostic radiology is the fourth most popular specialty choice for Medical Scientist Training Program graduates [18]. The average USMLE Step 1 in 2011 for matching graduates in this specialty was 240--at least one standard deviation above the national mean [19]. It is unclear how much PhD-level research can compensate for a national average exam score.

Further, dual degree programs are under pressure to graduate students quickly in the face of increasing graduation times [20]. As a result, curriculum modifications designed to decrease graduation times may provide some disadvantages for MD/PhD students. Some MD/PhD programs do not require the completion of all core clinical clerkships before residency applications are completed due to compression of the third and fourth years of medical school after finishing the PhD program. This would put those students at a significant disadvantage for programs that utilize USMLE Step 2 for considering or ranking candidates, as the Step 2 tests primarily on core clerkship material. Thus, sufficient time and emphasis needs to be provided to MD/PhD students in preparation for the USMLE Step examinations.

                                   

Conclusions

This study measures competition for US allopathic medical student seniors in the NRMP main residency match from 2005 to 2011 based on the NRMP Charting Outcomes in the Match and Results and Data publications. This analysis is of importance for medical students, their advisors, and others who track national trends in the national residency matching programs. Given expectations of future increases in competition, it may be necessary for medical school educators to examine residency applicant credentials more closely, consider longer dedicated study periods and curricula focused for USMLE Step 1 and Step 2 examinations, increase flexible time for a growing number of residency interviews, and provide resources for research experiences. 

References

1. Green M, Jones P, Thomas JX, Jr: Selection criteria for residency: results of a national program directors survey. Acad Med 2009, 84:362-367.
2. Makdisi G, Takeuchi T, Rodriguez J, Rucinski J, Wise L: How we select our residents--a survey of selection criteria in general surgery residents. J Surg Educ 2011, 68:67-72.
3. Otero HJ, Erturk SM, Ondategui-Parra S, Ros PR: Key criteria for selection of radiology residents: results of a national survey. Acad Radiol 2006, 13:1155-1164.
4. Charting Outcomes in the Match. Characteristics of applicants who matched to their preferred specialty in the 2005 NRMP Main Residency Match. National Resident Match Program Web site. July 2006. Accessed September 19, 2011. [http://www.nrmp.org/data/matchoutcomes2006.pdf]
5. Charting Outcomes in the Match. Characteristics of applicants who matched to their preferred specialty in the 2007 NRMP Main Residency Match. 2^nd Edition. National Resident Match Program Web site. August 2007. Accessed September 19, 2011. [http://www.nrmp.org/data/chartingoutcomes2007.pdf]
6. Charting Outcomes in the Match. Characteristics of applicants who matched to their preferred specialty in the 2009 Main Residency Match. 3^nd Edition. National Resident Match Program Web site. August 2009. Accessed September 19, 2011. [http://www.nrmp.org/data/chartingoutcomes2009v3.pdf]
7. Charting Outcomes in the Match. Characteristics of applicants who matched to their preferred specialty in the 2011 Main Residency Match. 4^th Edition. National Resident Match Program Web site. August 2011. Accessed September 19, 2011. [http://www.nrmp.org/data/chartingoutcomes2011.pdf]
8. Results and Data: 2005 Match. National Resident Match Program Web site. April 2005. Accessed September 19, 2011. [http://www.nrmp.org/data/resultsanddata2005.pdf]
9. Results and Data: 2011 Main Residency Match. National Resident Match Program Web site. April 2011. Accessed September 19, 2011. [http://www.nrmp.org/data/resultsanddata2011.pdf]
10. Iglehart JK: The Uncertain Future of Medicare and Graduate Medical Education. N Engl J Med. Sep 7 2011. Epub ahead of print. [http://www.nejm.org/doi/full/10.1056/NEJMhpr1107519]
11. Wilkerson L, Wimmers P, Doyle LH, Uijtdehaage S: Two perspectives on the effects of a curriculum change: student experience and the United States Medical Licensing Examination, Step 1. Acad Med 2007, 82:S117-S120.
12. Cooper ET: Essential Resources for the Boards: USMLE Step 1. March 2008. Accessed September 24, 2011. [http://www.medscape.com/viewarticle/570444]
13. Gurudevan SV, Mower WR: Misrepresentation of research publications among emergency medicine residency applicants. Ann Emerg Med 1996, 27:327-330.
14. Baker DR, Jackson VP: Misrepresentation of publications by radiology residency applicants. Acad Radiol 2000, 7:727-729.
15. Konstantakos EK, Laughlin RT, Markert RJ, Crosby LA: Follow-up on misrepresentation of research activity by orthopaedic residency applicants: has anything changed? J Bone Joint Surg Am 2007, 89:2084-2088.
16. Yang GY, Schoenwetter MF, Wagner TD, Donohue KA, Kuettel MR: Misrepresentation of publications among radiation oncology residency applicants. J Am Coll Radiol 2006, 3:259-264.
17. Kuo PC, Schroeder RA, Shah A, Shah J, Jacobs DO, Pietrobon R: "Ghost" publications among applicants to a general surgery residency program. J Am Coll Surg 2008, 207:485-489.
18. Paik JC, Howard G, Lorenz RG: Postgraduate choices of graduates from medical scientist training programs, 2004-2008. JAMA 2009, 302:1271-1273.
19. United States Medical Licensing Examination Questions About Scores. USMLE Web site. Accessed September 19, 2011. [http://www.usmle.org/general_information/FAQs/FAQs-Scores.html]
20. MSTP study. The careers and professional activities of graduates of the NIGMS Medical Scientist Training Program. National Institutes of Health Web site. September 1998. Accessed September 19, 2011. [http://publications.nigms.nih.gov/reports/mstpstudy/]

Appendix

Figure 5

Figure 5 Percentage of US Seniors matching into their first choice specialty by year

Figure 6

Figure 6 Percentage of US Seniors matching with AOA to their first choice specialties by year

Figure 7

Figure 7 Percentage of US Seniors matching from a top-40 US News research ranking medical school to their first choice specialties by year

Figure 8

Figure 8 Percentage of US Seniors matching with a PhD to their first choice specialties by year