(P-348) Study Validity in Sports Medicine: A Quantitative Assessment of Orthopedic Research Based on Publicly-obtained Data

Background: Too-often, researchers leverage internet sources to publish publicly-obtained data (POD) studies for orthopedic injuries in elite athletes, without correctly describing or accounting for limitations in these data. In studies of National Football League (NFL) players, this has resulted in biased conclusions.

Objectives: This purpose of this study was to provide a comprehensive review of POD studies regarding musculoskeletal injuries in NFL athletes with a focus on anterior cruciate ligament (ACL) injuries and to quantify under-ascertainment, methodologic errors and bias.

Methods: A systematic review was conducted to identify all published studies utilizing publicly-obtained ACL injury data in NFL athletes from 2000-2019. Two independent reviewers reviewed and included all published original, English-language research studies; abstracts and case reports were excluded. Data regarding player demographics were extracted from each publication. Results were compared to prospectively collected data reported by the teams’ medical staff to the NFL electronic health record database. An ACL “capture rate” for each article was calculated (number of ACL injuries obtained via POD/medically-reported ACL injuries).

Results: Forty-two studies met the definition of a POD study: 28 overall and 14 ACL-specific; 35 (83%) of the 42 studies were published since 2015. POD studies captured an average of 66% (range, 31%-90%) of ACL injuries reported by the teams’ medical staff. This inability to capture all injuries varied by position, with 86% capture of ACL injuries in skill athletes, 72% in mid-skill athletes, and 61% in linemen. POD studies captured 35% of injuries occurring during special teams play.

Conclusions: Studies trusting publicly-obtained injury data in NFL players has rapidly increased since 2000. There is significant heterogeneity in the degree to which POD studies correctly identify and completely ascertain ACL injuries from public reports. Many studies are biased toward more severe injuries, thus misrepresenting the impact and overall injury burden. Sports medicine research relying solely on publicly-obtained sources should be interpreted with an understanding of inherent limitations and biases. These studies have the potential to under-report the true incidence of injuries, with a bias toward capturing injuries in higher profile athletes, while failing to identify injuries in less popular players.