FULL TEXT IS BELOW
In a first-of-its-kind study, researchers used citation analysis to determine the studies most influential on the field of ACL research based on factors such as how frequently the studies were cited.
The researchers performed a systematic inquiry of the ISI Web of Science (Thomson Reuters) in search of any articles pertaining to the ACL, selecting the 50 most-cited articles for inclusion in their evaluation. For each article, the researchers recorded the number of citations, citation density, journal, year of publication, country origin and language. The articles were further characterized by type, subtype and level of evidence.
Results showed the 50 most-cited ACL research articles had a mean of 326 citations and mean citation density of 18.2 citations per year. Articles appeared in one of 11 journals, most frequently published in The American Journal of Sports Medicine and The Journal of Bone and Joint Surgery American, according to the researchers.
The researchers concluded that the list of these top-cited articles may assist fellowship and residency programs when trainee reading curriculums are being compiled. ‒ by Monica Jaramillo
Disclosure: The researchers report no relevant financial disclosures.
The number of times an article has been cited in the peer-reviewed literature is indicative of its impact on its respective medical specialty. No study has used citation analysis to determine the most influential studies pertaining to the anterior cruciate ligament (ACL). The primary aims of this study were to identify the classic works in ACL research using citation analysis and to characterize these articles to determine which types of studies have had the most influence on the field. A systematic query of ISI Web of Science (Thomson Reuters, Philadelphia, Pennsylvania) was performed for articles pertaining to the ACL, and the 50 most-cited articles were selected for evaluation. The following characteristics were determined for each article: number of citations, citation density, journal, publication year, country of origin, language, article type, article subtype, and level of evidence. The number of citations ranged from 219 to 1073 (mean, 326), and the citation densities ranged from 4.9 to 55.6 citations per year (mean, 18.2). All articles were published in 1 of 11 journals, with the most being published in The American Journal of Sports Medicine (46%) and The Journal of Bone and Joint Surgery American (30%). The most common decades of publication were the 1990s (34%), 1980s (28%), and 2000s (26%). The majority (68%) of articles originated from the United States, and all were written in English. By article type, 42% were basic science, and 58% were clinical. Of the clinical articles, 3% were Level I, 17% were Level II, 28% were Level III, and 52% were Level IV. The articles were heterogeneous with regard to article type, article subtype, and level of evidence and tended to have the following characteristics: high-impact journal of publication, recent publication year, US origin, English language, and low level of evidence. These works represent some of the most popular scientific contributions to ACL research. This list may aid residency and fellowship programs in the compilation of articles for trainee reading curriculums. [Orthopedics. 2015; 38(4):e297–e304.]
The authors are from the Department of Orthopaedic Surgery (PBV), University of Pennsylvania, Philadelphia; Jefferson Medical College (GR), Philadelphia; and The Rothman Institute (KBF), Department of Orthopaedic Surgery, Thomas Jefferson University, Bryn Mawr, Pennsylvania; and The Rothman Institute (FPT), Department of Orthopaedic Surgery, Thomas Jefferson University, Egg Harbor Township, New Jersey.
The authors have no relevant financial relationships to disclose.
Correspondence should be addressed to: Kevin B. Freedman, MD, MSCE, The Rothman Institute, Medical Arts Pavilion, 825 Old Lancaster Rd, Ste 200, Bryn Mawr, PA 19010 ( firstname.lastname@example.org).
The number of times an article has been cited in the peer-reviewed literature (citation number) and the frequency of these citations (citation density) are markers for the article’s impact on its respective specialty.1 Although citation number and density do not directly assess the quality of a study, they do shed light on its readership and its potential to inspire change in practice. As such, several authors have used citation analysis to identify classic influential works in a given area of medical expertise.2–13
With regard to the orthopedic literature, Lefaivre et al9 identified the 100 most-cited articles in orthopedic surgery using citation analysis. The classic works on this list spanned all orthopedic subspecialties.9 Other authors have used similar techniques to identify the most cited works in orthopedic subspecialties, including shoulder surgery,14 pediatric orthopedic surgery,15 and fracture surgery.16 However, to the authors’ knowledge, no authors have published on the most cited works pertaining to a particular clinical problem in orthopedics. Specifically, no study has used citation analysis to determine the most influential studies related to the anterior cruciate ligament (ACL).
The purposes of this study were to identify the 50 most-cited articles that pertain to the ACL and to characterize these works to determine which types of articles have had the most influence on the field.
Materials and Methods
The authors performed a systematic query of the ISI Web of Science (v5.11; Thomson Reuters, Philadelphia, Pennsylvania) using the Advanced Search tool on April 12, 2013. They searched for all published articles with a topic listed as anterior cruciate ligament, ACL, or knee ligament. The retrieved articles were then sorted in order of the number of citations, and the 50 most-cited articles were selected for analysis. The full texts of each of these articles were individually scrutinized by 2 authors to ensure that they contained material related to the ACL. After meticulous evaluation, all 50 articles were deemed to appropriately pertain to the ACL. The authors’ query technique was identical to that used in previous analyses of orthopedic literature,9,14–16 with the exception that the authors expanded their search to include all journals, not just those listed under the Web of Science Subject Category of “Orthopedics.” The following characteristics were then recorded for each of the 50 most-cited articles: citation number, citation density, journal, publication year, country of origin, and language. In addition, 2 independent evaluators characterized each article by type (basic science vs clinical), subtype (basic science subtypes: animal studies, anatomic studies, biomechanical studies; clinical subtypes: review articles, technique articles, case series, case-control studies, prospective cohort studies, randomized controlled trials, and meta-analyses), and level of evidence for clinical articles. Level of evidence was determined using the classification system jointly developed by The Journal of Bone and Joint Surgery and the American Academy of Orthopaedic Surgeons.17 Interobserver agreement for all subjective variables was excellent: 100% for article type, 100% for article subtype, and 98% for level of evidence. All discrepancies between authors were resolved by consensus. The results were then analyzed to determine which characteristics were most common among the 50 most-cited articles.
The 50 most-cited articles pertaining to the ACL had citation numbers ranging from 219 to 1073 citations (mean, 326) and citation densities ranging from 4.9 to 55.6 citations per year (mean, 18.2) (Table 1). The 2 articles with the greatest number of citations were “Rating systems in the evaluation of knee ligament injuries” by Tegner et al18 (1073 citations) and “Evaluation of knee ligament surgery results with special emphasis on use of a scoring scale” by Lysholm et al19 (943 citations). Interestingly, these were the only 2 articles that introduced instruments for outcome evaluation. The article with the greatest citation density was “Biomechanical measures of neuromuscular control and valgus loading of the knee predict ACL injury risk in female athletes” by Hewett et al20 (55.6 citations per year).
Top 50 Most Cited Articles in Anterior Cruciate Ligament Research
All of the top 50 articles were published in 1 of 11 journals (Table 2), with the most being published in The American Journal of Sports Medicine (AJSM) (46%) and The Journal of Bone and Joint Surgery American (JBJS-A) (30%). The publication years of the 50 most-cited articles spanned from 1963 to 2007. The most common decades of publication were the 1990s (34%), 1980s (28%), and 2000s (26%) (Figure 1). Of the 50 most-cited articles, those published in the 1980s had the greatest mean number of citations (421) by decade (Figure 2). On the other hand, mean citation density increased with each successive decade (Figure 3), with the top 50 articles published in the 2000s having the greatest mean citation density (30.7).
Number of Top 50 Anterior Cruciate Ligament Articles by Source Journal
Number of top 50 anterior cruciate ligament articles by decade of publication.
Mean number of citations of top 50 anterior cruciate ligament articles by decade of publication.
Mean citation density of top 50 anterior cruciate ligament articles by decade of publication.
The 50 most-cited articles originated from 10 countries (Table 3). The majority (68%) originated from the United States, and all of the top 50 articles were written in the English language. The only individual with lead authorship for 3 different articles on the top 50 list was FR Noyes, whereas DM Daniel, JC Hughston, LS Lohmander, KL Markolf, and KD Shelbourne each had lead authorship on 2 of the articles on the list.
Number of Top 50 Anterior Cruciate Ligament Articles by Country of Origin
By article type, 42% were basic science and 58% were clinical. There was a wide variety of article subtypes (Table 4), but the most common was basic science biomechanical studies (32%). Of the 29 clinical articles, more than half (52%) had Level IV evidence, while the remaining had Level I (3%), Level II (17%), or Level III (28%) evidence (Table 5).
Number of Top 50 Anterior Cruciate Ligament Articles by Article Subtype
Number of Top 50 Anterior Cruciate Ligament Articles by Level of Evidence
The number of times an article has been cited in the peer-reviewed literature is indicative of its impact on its respective specialty. The current study identifies the 50 most-cited articles pertaining to the ACL. This list represents some of the most popular works published on the subject. Although citation number and density are important markers for an article’s impact on a medical specialty, they are not the only such measures. As such, the authors’ list of the 50 most-cited articles pertaining to the ACL is not exclusive—there are certainly several articles not on the list that have had a significant impact on the field. Nevertheless, the list does successfully identify many of the classic studies that have advanced the understanding of the anatomy and biomechanics of the ACL, as well as the diagnosis, management, and prognosis of ACL injuries.
After identifying the 50 most-cited articles pertaining to the ACL, the authors analyzed the characteristics of these studies. They found that the 2 most cited articles18,19 were the only 2 articles that introduced instruments for outcome evaluation. Outcome evaluation tools are extremely useful in providing objective measurements to assess the efficacy of a given intervention. A study that introduces and validates an effective outcome evaluation instrument is likely to be cited repeatedly—whenever that particular instrument is used in another study. Lefaivre et al9 found that the most cited article in orthopedic surgery introduced Harris Hip Scores,21 and Namdari et al14 found that the most cited article in shoulder surgery introduced Constant scores.22 It is consistent with these findings that the outcome evaluation tools introduced by Tegner et al18 and Lysholm et al19 have been cited so frequently in the ACL literature.
The majority of the top 50 most-cited articles list were published in AJSM (46%) or JBJS-A (30%). Indeed, articles published in these 2 journals tend to have a high impact. The annual Thomson Reuters Journal Citation Reports rank journals according to impact factor, which is a measure of the frequency with which the average article in a particular journal has been cited in a certain period of time.23 According to the 2012 version of this report, of the 65 journals in the subject category of Orthopedics, AJSM was ranked first (impact factor: 4.439) and JBJS-A was ranked third (impact factor: 3.234).23Consistent with these findings, Siebelt et al24 found that AJSM was the specialized orthopedic journal with the greatest impact factor, whereas JBJS-A was the general orthopedic journal with the greatest impact factor.
However, it is important to keep in mind that citation number and density for articles, as well as impact factor for journals, can be affected by self-citation rates.25 Hakkalamani et al26 found that, of the 7 general orthopedic journals that they examined, JBJS-A had the greatest self-citation rate, with more than 40% of JBJS-A references being made to other JBJS-A articles. The most common decades of publication for the 50 most-cited articles were the 1990s (34%), 1980s (28%), and 2000s (26%). Therefore, although older articles pertaining to the ACL have had more time to accumulate citations, it seems that more recent articles on the subject have generally had a greater impact. Furthermore, the current authors found that mean citation density increased with each successive decade, with the articles published in the 2000s having the greatest mean citation density (30.7). These findings are in stark contrast to the study by Lefaivre et al,9 which identified the 100 most-cited articles in orthopedic surgery. They found that the 1980s had the greatest number of top 100 articles, with more than twice as many articles as any other decade.9 They also found that mean citation density was greatest in articles published in the 1940s, 1960s, and 1970s.9 These differences reflect the fact that the community’s understanding of the anatomy and biomechanics of the ACL and of the diagnosis, management, and prognosis of ACL injuries is relatively recent. Indeed, available knowledge on the subject has evolved tremendously over the past few decades with the advent of novel techniques and technologies.
The 50 most-cited ACL articles originated from 10 different countries, with 68% originating from the United States. The preponderance of classic articles from the United States has also been demonstrated in the literature of anesthesia,8 critical care,7 emergency medicine,13 plastic surgery,12 general surgery,6 and orthopedic surgery.9 Potential explanations for this finding may be that (1) more studies in general originate from the United States than from any other nation, (2) studies from the United States are more likely to have a greater impact, or (3) authors from the United States have an advantage in publishing in high-impact journals. This advantage may be due to inherent language barriers encountered by foreign authors who are attempting to publish their work in high-impact journals, which tend to be published in the English language. In this regard, English-speaking authors may possess a substantial advantage over their counterparts. The current study found that all of the top 50 articles were written in the English language. However, it should be noted that ISI Web of Science only includes a limited number of journals in languages other than English. Therefore, this investigation may have excluded influential articles published in foreign languages, as well as citations from articles published in foreign languages.
The authors found a fairly even split between basic science (42%) and clinical (58%) articles within the 50 most-cited ACL articles. Lefaivre et al9 demonstrated a less even breakdown of the top 100 articles in orthopedic surgery (24% basic science vs 76% clinical). In addition, studies analyzing the classic articles in other fields also found a dominance of clinical studies over basic science studies.6,8,12,13 Therefore, it seems that basic science ACL studies are more likely to be classic articles than basic science studies in other medical fields. In support of this notion, the current authors found that the most common article subtype among the 50 most-cited ACL articles was not a clinical subtype, but rather basic science biomechanical studies (32%).
When looking at just the clinical articles in the top 50 list, 52% had Level IV evidence, whereas the remaining had Level I (3%), Level II (17%), or Level III (28%) evidence. Samuelsson et al27 examined the level of evidence of all therapeutic studies written in English published from January 1995 to August 2011 that reported on isolated primary ACL reconstruction with clinical outcome measurements. Consistent with the results of the current study, Samuelsson et al27 found that the most common level of evidence encountered in their search was Level IV (33.7%). Furthermore, the orthopedic citation analyses by Lefaivre et al,9 Baldwin et al,15 Baldwin and Namdari,16 and Namdari et al14 also demonstrated a preponderance of Level IV studies among the classic articles identified within all of orthopedics and various orthopedic subspecialties. Because the concept of evidence-based medicine is relatively new, it is possible that higher level studies will infiltrate the top 50 list over time. In support of this notion, Samuelsson et al27 demonstrated a significant trend toward higher mean level of evidence over time. In addition, the current study found that higher level studies on the top 50 most-cited list tended to be published more recently. It will be interesting to study the changes in level of evidence frequencies that occur as ACL research matures. Nevertheless, there will likely always be a role for classic ACL studies, even those that are based on Level IV evidence.
This study has several weaknesses. First, the authors arbitrarily selected to identify the top 50 most-cited articles pertaining to the ACL. There are certainly classic and influential articles that may not fall in the top 50. They selected 50 because this is a reasonable number of articles to include in a resident or fellow reading curriculum and because previous analyses of orthopedic subspecialty literature used the same number.14,15 Second, citation number and density may be affected by confounding factors, such as author self-citation, journal self-citation, exposure time, and journal impact factor. The authors encourage further study of these factors and how they affect frequency of citation. Third, they only included citations from published articles; they did not include citations from lectures, presentations, textbooks, or other non-peer-reviewed literature. As such, the authors may have missed out on certain classic articles or teaching references that tend to be more frequently cited in these venues. Fourth, as proposed in the article by Lefaivre et al,9 some articles may be subject to a “snowball effect” of citation, where they are repeatedly cited because of previous citations rather than for their quality or content. Others may be cited simply because they include well-known authors or authors who sit on editorial boards or serve as peer reviewers. In addition, others may be cited frequently because they address timely and popular topics. These biases may result in inflated citation rates that do not truly reflect influence on the field. Fifth, the study does not address conflicts of interests, which can potentially introduce bias, and the influence of conflicts of interests on citation rates. Finally, citation number and density are only 2 measures of the impact of an article. Therefore, the authors’ list of the top 50 most-cited ACL articles is not exclusive.
This study identifies the 50 most-cited articles pertaining to the ACL: classic works that have greatly influenced the community’s understanding of the subject. This list may aid residency and fellowship programs in the compilation of articles for trainee reading curriculums. In addition, this investigation identifies characteristics that are common and varied among these top 50 most-cited articles, thus providing insight into what factors make an article on the ACL influential.
- Adam D. The counting house. Nature. 2002; 415:726–729. doi:10.1038/415726a [CrossRef]
- Fenton JE, Roy D, Hughes JP, Jones AS. A century of citation classics in otolaryngology-head and neck surgery journals. J Laryngol Otol. 2002; 116:494–498. doi:10.1258/002221502760132557 [CrossRef]
- Ollerton JE, Sugrue M. Citation classics in trauma. J Trauma. 2005; 58:364–369. doi:10.1097/01.TA.0000141882.31281.66 [CrossRef]
- Brandt JS, Downing AC, Howard DL, Kofinas JD, Chasen ST. Citation classics in obstetrics and gynecology: the 100 most frequently cited journal articles in the last 50 years. Am J Obstet Gynecol. 2010; 203:355. e1–e7.
- Dubin D, Häfner AW, Arndt KA. Citation classics in clinical dermatologic journals: citation analysis, biomedical journals, and landmark articles, 1945–1990. Arch Dermatol. 1993; 129:1121–1129. doi:10.1001/archderm.1993.01680300049007[CrossRef]
- Paladugu R, Schein M, Gardezi S, Wise L. One hundred citation classics in general surgical journals. World J Surg. 2002; 26:1099–1105. doi:10.1007/s00268-002-6376-7 [CrossRef]
- Baltussen A, Kindler CH. Citation classics in critical care medicine. Intensive Care Med. 2004; 30:902–910. doi:10.1007/s00134-004-2195-7 [CrossRef]
- Baltussen A, Kindler CH. Citation classics in anesthetic journals. Anesth Analg. 2004; 98:443–451. doi:10.1213/01.ANE.0000096185.13474.0A [CrossRef]
- Lefaivre K, O’Brien P. 100 most cited articles in orthopaedic surgery. Clin Orthop Relat Res. 2011; 469(5):1487–1497. doi:10.1007/s11999-010-1604-1 [CrossRef]
- Ohba N, Nakao K, Isashiki Y, Ohba A. The most frequently cited articles in ophthalmology journals. Arch Ophthalmol. 2007; 125:952–960. doi:10.1001/archopht.125.7.952 [CrossRef]
- Ponce FA, Lozano AM. Highly cited works in neurosurgery: Part I. The 100 top-cited papers in neurosurgical journals. J Neurosurg. 2010; 112:223–232. doi:10.3171/2009.12.JNS091599 [CrossRef]
- Loonen MPJ, Hage JJ, Kon M. Plastic surgery classics: characteristics of 50 top-cited articles in four plastic surgery journals since 1946. Plast Reconstr Surg. 2008; 121:320e–327e. doi:10.1097/PRS.0b013e31816b13a9 [CrossRef]
- Tsai Y-L, Lee C-C, Chen S-C, Yen Z-S. Top-cited articles in emergency medicine. Am J Emerg Med. 2006; 24:647–654. doi:10.1016/j.ajem.2006.01.001 [CrossRef]
- Namdari S, Baldwin K, Kovatch K. Fifty most cited articles in orthopedic shoulder surgery. J Shoulder Elbow Surg. 2012; 21(12):1796–1802. doi:10.1016/j.jse.2011.11.040 [CrossRef]
- Baldwin KD, Kovatch K, Namdari S, Sankar W, Flynn JM, Dormans JP. The 50 most cited articles in pediatric orthopedic surgery. J Pediatr Orthop B. 2012; 21(5):463–468. doi:10.1097/BPB.0b013e328354b0cf [CrossRef]
- Baldwin K, Namdari S. 100 most cited articles in fracture surgery. Am J Orthop. 2013; 42(12):547–552.
- Wright J, Swiontkowski M, Heckman J. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003; 85:1–3.
- Tegner Y, Lysholm J. Rating systems in the evaluation of knee ligament injuries. Clin Orthop Relat Res. 1985; (198):43–49.
- Lysholm J, Gillquist J. Evaluation of knee ligament surgery results with special emphasis on use of a scoring scale. Am J Orthop. 1982; 10(3):150–154.
- Hewett TE, Myer GD, Ford KR, et al. Biomechanical measures of neuromuscular control and valgus loading of the knee predict anterior cruciate ligament injury risk in female athletes: a prospective study. Am J Sports Med. 2005; 33:492–501. doi:10.1177/0363546504269591 [CrossRef]
- Harris WH. Traumatic arthritis of the hip after dislocation and acetabular fractures: treatment by mold arthroplasty: an end-result study using a new method of result evaluation. J Bone Joint Surg Am. 1969; 51:737–755.
- Constant CR, Murley AH. A clinical method of functional assessment of the shoulder. Clin Orthop Relat Res. 1987; (214):160–164.
- Thomson Reuters 2012 Journal Citation Reports. http://www.webofknowledge.com. Accessed March 16, 2014.
- Siebelt M, Siebelt T, Pilot P, Bloem RM, Bhandari M, Poolman RW. Citation analysis of orthopaedic literature; 18 major orthopaedic journals compared for impact factor and SCImago. BMC Musculoskelet Disord. 2010; 11:4. doi:10.1186/1471-2474-11-4 [CrossRef]
- Garfield E. How can impact factors be improved?BMJ. 1996; 313:411–413. doi:10.1136/bmj.313.7054.411 [CrossRef]
- Hakkalamani S, Rawal A, Hennessy MS, Parkinson RW. The impact factor of seven orthopaedic journals: factors influencing it. J Bone Joint Surg Br. 2006; 88(2):159–162. doi:10.1302/0301-620X.88B2.16983 [CrossRef]
- Samuelsson K, Desai N, McNair E, et al. Level of evidence in anterior cruciate ligament reconstruction research: a systematic review. Am J Sports Med. 2013; 41:924–934. doi:10.1177/0363546512460647 [CrossRef]