(P-505) Automating Individual Case Safety Report Identification Within Scientific Literature Using Natural Language Processing

Background: Detecting Individual Case Safety Reports (ICSRs) in scientific literature is a core Pharmacovigilance responsibility as part of a Market Authorization Holder’s obligations. Scientific literature is screened continuously for candidate cases, ‘artifacts’, that are reviewed manually to determine if sufficient information exists to consider it as containing a valid ICSR.

Objectives: Currently, artifacts are manually reviewed in the order they are retrieved by searches/alerts. The objective was to assess whether Natural Language Processing (NLP) could be utilized to prioritize the artifacts for review, therefore expediting submission of relevant articles to case processing.

Methods: 449,822 previously manually annotated scientific literature artifacts were divided into training (416,822 artifacts) and test (33,000 artifacts) sets. The training data was balanced to avoid bias, with the remaining sets representative of real-world data (i.e. low occurrence of ICSRs). The test set contained 518 artifacts identified as containing relevant ICSRs (1.57%).

After NLP algorithm model training, the test dataset was processed to produce a relevance score (correlating to the probability of an artifact containing an ICSR) used to rank the artifacts in order for review. The relevance scores were assessed against the literature experts’ manual annotations. Analysis was then made of the performance difference of the NLP prioritized review compared to the existing approach.

Results: After NLP algorithm prioritization of the artifacts, 33% of the dataset contained 99% (513 positives) of the ICSR artifacts and 65% of the dataset contained 100% (518 positives) of the ICSR artifacts. An artifact time in queue metric for the actual ICSRs was calculated as a mean of 1.2 hours for the test dataset sorted on relevance score compared to 4.5 hours on average for the current process random order distribution of ICSRs in artifact review (based on mean rate of review of 1750 average artifacts across an 8-hour workday).

Conclusions: NLP was able to prioritize candidate scientific literature articles for manual review and show promise of tangible benefits in routine production-use of NLP for identifying ICSRs. Artifacts identified as low probability of containing an ICSR (35%) suggest a further potential efficiency gain, enabling over time the gradual reduction of 100% human review.