(P-424) Challenges in Studying COVID-19 Treatments in Real-world Hospital Datasets: A Case Study of Tocilizumab

Background: There is an urgent need to find effective treatments of COVID-19. Although randomised clinical trials are viewed as the ‘gold standard’, they can take many years to complete and are expensive to run. Routinely collected real-world data offer a quick and cost-effective way to study potential interventions. However, real-world databases also come with many challenges and the reliability of results can therefore come into question.

Objectives: To explore the challenges posed, and the methods used to address them, in real-world studies investigating COVID-19 treatments.

Methods: We highlight the key problems encountered when analysing real-world hospital databases, covering issues including missing data, confounding, selection bias and selecting appropriate endpoints. We attempt to tackle these difficulties using statistical methods such as time-updated analyses and imputation techniques and emphasise that they are often not sufficient to resolve the challenges posed. We illustrate this with a case study that investigated tocilizumab using a real-world database, Covid Data Save Lives. This was shared as an open data set and included electronic health records from 2547 COVID-19 patients hospitalized in various Hospitals within Grupo HM Hospitales in Spain. The primary outcome was all-cause in-hospital mortality at 28 days.

Results: Our findings suggested that when limitations of the data were ignored in analyses there was a detrimental effect of tocilizumab on 28-day mortality (HR 1.66, 95% CI 1.25 to 2.22, P=0.001) compared to standard care. However, after additional efforts were made to handle these issues, no significant effect was found (HR 1.17, 95% CI 0.86 to 1.59, P=0.31). The differences in these estimates highlight the impact that confounding and other sources of bias can have in an observational setting.

Conclusions: When analysing real-world datasets, limitations should be acknowledged and addressed through either design or analytical means. However, despite our best efforts, some issues may persist and therefore results should always be interpreted with caution.