(P-345) A Comparison of Data Mining Algorithms Using Different Drug Classifications for Signal Identification

Background: Drugs can be classified at different levels of granularity for data mining spontaneous reporting systems like the FDA Adverse Event Reporting System (FAERS). Product active ingredient (PAI) is the most granular classification, in which the ingredient base, salt forms, and prodrugs are considered unique products. In contrast, product active moiety (PAM) considers these related PAIs as one product (e.g., phenytoin, phenytoin sodium, and fosphenytoin are all combined as ‘phenytoin’). While there may be practical reasons to data mine at different drug classifications, it is unclear how classification differences may impact performance of data mining algorithms (DMAs).

Objectives: To compare the performance of a PAI-based versus PAM-based DMA in FAERS.

Methods: Three published reference sets of known adverse drug reactions (ADRs) and negative controls were combined, deduplicated, and standardized to create a larger reference set. FAERS data through December 31, 2020 was used for the analysis. Empirical Bayes geometric mean scores and their 90% confidence intervals (EB05 and EB95) were calculated for each drug–event pair in FAERS using a PAI and PAM-based DMA. Pairs with an EB05 ≥ 2 were categorized as positive, whereas pairs with an EB05 < 2 were categorized as negative. The classifications were then compared to the reference set to determine the sensitivity, specificity, precision, F1, and accuracy of the PAI and PAM-based DMA. Additionally, the alert volume (i.e., pairs with an EB05 ≥ 2) was compared between the two algorithms for the reference set and FAERS overall.

Results: The reference set created had a total of 609 unique PAM-event pairs and 1128 PAI-event pairs. The PAM-based DMA performed better than the PAI-based DMA: sensitivity (0.41 vs 0.27), specificity (0.98 vs 0.98), precision (0.95 vs 0.91), F1 (0.57 vs 0.42), and accuracy (0.74 vs 0.66). When considering whether any relevant PAI component produced an alert for a known ADR, the alerts identified between the algorithms did not completely overlap. Several known ADRs were identified by each DMA that were not identified by the other. The alert volume was higher for the PAI-based analysis for the reference set (150 vs 115 alerts, 30% higher) and overall database (347,264 vs 303,152, 15% higher).

Conclusions: While the analysis was limited by a reference set reflecting known risks and presumed negatives, a PAM-based DMA performed better than a PAI-based DMA and was associated with a lower alert burden. Nonetheless, the PAI-based DMA identified known ADRs not identified by the PAM-based algorithm, suggesting it may be preferred in select scenarios.