OP-8A - Impact of Fda’s Regulatory Actions on the Quality of REMS Survey Methodology Protocols (2007 – 2020)

Background: Measuring patients’ and healthcare providers’ understanding of serious risk messages using knowledge surveys is a widely used method to evaluate the effectiveness of risk evaluation and mitigation strategies (REMS). To develop best practices for assessing knowledge, FDA held a workshop and issued a draft guidance on survey methodologies.

Objectives: To evaluate the impact of FDA’s regulatory actions [the 2012 Public Workshop on “REMS Assessments: Social Science” and the 2019 draft guidance “REMS Survey Methodologies”] on the quality of survey methodology protocols submitted to the agency over time since REMS legislation (2007).

Methods: A content analysis of REMS survey methodology protocols submitted to FDA for single product REMS with Elements to Assure Safe Use (ETASU) requiring knowledge assessment using a survey, was conducted (N=40 programs, 78 protocols). Each protocol was scored against 52 core scientific elements described in the draft guidance grouped into six domains: study background (n=5), study design (n=18), survey instrument (n=9), participant recruitment (n=7), survey administration (n=9), statistical analysis plan (n=4). Two reviewers adjudicated 10% randomly selected protocols against the core elements (average IRR = 91%). Mean scores were segmented by three different time periods reflecting regulatory actions: (1) 09/2007-10/2012; (2) 11/2012-02/2019; (3) 03/2019 -12/2020; and compared using ANOVA adjusting for multiple comparisons (Tukeys HSD).

Results: Overall, mean scores improved significantly over time (p < 0.0001): 28.03 (SD= 5.12 period 1), 31.68 (SD=3.72 period 2), 39.00 (SD=2.56 period 3). Survey administration and statistical analysis plan saw the greatest improvement after the 2012 Workshop (p < 0.0001, p=0.0164, respectively) whilst study design, survey administration and statistical analysis plan saw the greatest improvement after draft guidance published (p=0.0042, p=0.0003, p=0.0017 respectively). Among the domains study design, survey instrument, participant recruitment, survey administration and statistical analysis plan improved significantly over time (p=0.0041, p=0.0177, p=0.0026, p<0.0001 and p<0.0001 respectively). Despite improvement over time, the maximum score a protocol received was 42 out of 52. Core elements that did not improve include knowledge threshold justification, sampling frame justification and sample size justification based on sampling design.

Conclusions: The scientific rigor of survey protocols submitted to FDA improved over time. Evidence suggests the public workshop and draft guidance were associated with a positive impact, but scientific quality gaps remain.