(P-400) Imbalanced Data Bias in Logistic Regression to Estimate Treatment Effects: Results from a Virtual Patient Cohort Simulation

Background: Logistic regression is one of the most frequently utilized methods for evaluating treatment effects (TE) on binary health outcomes such as adverse drug events in pharmacoepidemiology. Predictive analytics studies have shown that imbalanced data (outcome events with rare or infrequent occurrence) may affect model prediction accuracy. However, it is unclear whether or to what extent the imbalanced data problem may affect logistic regression models in estimating the true treatment effect (TTE).

Objectives: Using a virtual cohort simulation, we assessed the bias in TE estimated from logistic regression models.

Methods: We simulated virtual patient cohorts with varying outcome event prevalence (1%, 10%, 30%, and 50%), different cohort sample sizes (5,000, 10,000, 30,000, 50,000, and 100,000); effect size of TTEs (for β: small – 0.062, medium – 0.622, and large = 1.245; for odds ratio: 1.06, 1.86, and 3.47, respectively) as well as 40 covariates with their covariance derived from the real patient cohort. In each combination of event prevalence, sample size, and size of TTE, 500 datasets were simulated. We assessed the bias in the logistic regression models by the difference in mean (SD), coverage percentage within 95% confidence intervals (CIs), and median of 95% confidence limit differences (CLDs). The CLD is the size of the 95% CI (upper minus lower 95% confidence limits).

Results: The smaller the sample size and frequency of outcome events, the higher bias in the estimated TE compared to TTE. When the TTE was 0.622 (medium TTE), the mean of estimated TE with 100,000 patients and 50% outcome rate was 0.616 (SD 0.014), but the mean with 5,000 patients and a 1% outcome incidence was 0.846 (0.497). Likewise, the difference between the estimated TE and the TTE decreased when the size of the TTE increased and vice versa. The coverage percentage and CLD was 98.0% and 0.07 for 100,000 patients with 50% outcome rate, 91.2% and 1.77 for 5000 patients and 1% outcome rate. Increasing sample size, outcome rate, and effect size also increased coverage percentage and decreased CLD.

Conclusions: When sample sizes and/or TEs is small, or there is imbalanced data (rare outcome events, which is a common scenario when evaluating safety outcomes in pharmacoepidemiology), the TTE estimated by logistic regression may be markedly biased. Application of remedial measures that address issues of imbalanced data and/or small sample size might be considered to obtain valid treatment effect estimates with logistic regression in these scenarios.