The adaptation of the desirability of outcome ranking for interventional clinical trials in epilepsy: A novel consumer‐led outcome measure

Abstract Interventional clinical trials in epilepsy are typically designed and powered to detect a change in seizure frequency as the primary endpoint, with little consideration given to other benefits or harms of the therapy, or impacts on common epilepsy comorbidities. Desirability of outcome ranking (DOOR) is a novel methodology for evaluating benefits and harms associated with introduction of a new treatment. Multiple outcomes are combined and the resulting combinations are ranked according to their desirability. Herein we describe the adaptation of DOOR for use in therapy trials in epilepsy. Consumers with epilepsy were presented with a selection of measures typically included in epilepsy trials and asked to rank their importance in terms of a desirable outcome and to identify interactions between different seizure control levels and other measures. Seizure control, adverse events, and psychiatric comorbidities were identified as most important, and combinations of these outcomes were ranked to form epilepsy‐DOOR. A separate consumer discussion group verified the appropriateness and accuracy of the ranking. The resultant epilepsy‐DOOR includes 60 possible outcomes, representing high granularity for the assessment of future interventions. It demonstrates the importance of consumer involvement in trial design and presents an alternative to seizure frequency for evaluating new treatments for epilepsy.


| INTRODUCTION
Clinical trials of new therapies for drug-resistant epilepsy are typically powered to detect reductions in seizure frequency (most commonly >50% reduction) as the primary endpoint.However, additional factors, including adverse events (AEs), cognitive deficits, psychiatric comorbidities, quality of life and cost, are important to people with long-standing epilepsy when considering the desirability of their epilepsy treatment. 1,2To date almost all clinical therapeutic trials in epilepsy have been of symptomatic antiseizure medications (ASMs), but the development of tions are ranked according to their desirability.Herein we describe the adaptation of DOOR for use in therapy trials in epilepsy.Consumers with epilepsy were presented with a selection of measures typically included in epilepsy trials and asked to rank their importance in terms of a desirable outcome and to identify interactions between different seizure control levels and other measures.Seizure control, adverse events, and psychiatric comorbidities were identified as most important, and combinations of these outcomes were ranked to form epilepsy-DOOR.A separate consumer discussion group verified the appropriateness and accuracy of the ranking.The resultant epilepsy-DOOR includes 60 possible outcomes, representing high granularity for the assessment of future interventions.
It demonstrates the importance of consumer involvement in trial design and presents an alternative to seizure frequency for evaluating new treatments for epilepsy.

K E Y W O R D S
consumer design, epilepsy, interventional clinical trial, trial outcomes disease-modifying treatments has become a major focus of the translational epilepsy research effort, 3 and this is likely to lead to future trials of disease-modifying treatments.Having a broader assessment of the burden of epilepsy as a clinical trial endpoint is likely to be of particular relevance for future disease-modifying treatment trials.
To better capture the multifaceted nature of epilepsy as a primary endpoint for clinical trials of treatment we sought to develop an "epilepsy-DOOR", an adaptation of The Desirability Of Outcome Ranking (DOOR) 4 method.DOOR ranks the outcomes for participants according to the desirability of their overall outcome that may include a combination of benefits and harms.DOOR was initially developed for application to studies of inpatient antibiotic use, and the authors provide a comprehensive explanation of the principles and considerations when adapting DOOR for other uses. 4ne advantage of DOOR over traditional approaches to measuring trial outcomes is that the overall outcome is constructed based on benefits and harms, synthesizing multiple measures for each individual patient.The possible outcomes are used to analyze the patient, rather than patients used to measure the outcome. 5The number and definition of categories is tailored to the disease of interest, with no set number of levels for each category.During analyses, the distributions of DOORs are compared between treatment arms.
The most important factor in the design of the DOOR is to ensure consensus of outcomes.Thus, we performed a two-stage preference elicitation process with consumers to adapt the DOOR approach for use in epilepsy trials.

| Development of outcomes
7][8][9] For each of these measures levels of outcome were defined by the authors based on their clinical experience (Table 1).Seven consumers with long-standing drug-resistant focal epilepsy (n = 5 female, Figure 1) were surveyed to identify and prioritize the top three outcomes for inclusion into the epilepsy-DOOR.The consumers were asked to rank outcomes in order of importance, with additional questions on the interactions between outcomes to further define and order the DOOR.The first iteration of the epilepsy-DOOR was constructed based on the responses to the survey.

| Consumer-led outcome review
The first iteration of the epilepsy-DOOR was reviewed and refined by a separate group of eight consumers (6 people with epilepsy, one consumer representative, and one carer of a person with epilepsy, female n = 6).In accordance with the Cancer Australia Consumer Involvement Toolkit and the Australian Clinical Trials Alliance (ACTA) Consumer Involvement and Engagement Toolkit, 10 consumers were sought through expressions of interest on epilepsy consumer groups (Epilepsy Foundation of Victoria, Epilepsy Action Australia, Australian Women with Epilepsy) social media channels.Respondents to the expression of interest were contacted to further explain the purpose of the consumer group and to affirm participation.The consumer group discussion occurred over Zoom to facilitate participation from people across Australia.As per the ACTA Consumer Involvement and Engagement Toolkit ethical approval was not required.Except that described above, no demographic or clinical information was collected to prevent potential identification of consumers.
The group discussion followed the outline provided by the ACTA toolkit and included a brief outline of the proposed clinical trial and a description of the surveys conducted previously.The main discussion centered around the epilepsy-DOOR as constructed from the surveys.The discussion was an open forum with a free-flowing discussion between the consumers, considering the measures included in the DOOR, suggesting alternatives, and reordering the ranking.

| Category definitions
The consumer surveys resulted in prioritization of a reduction in seizure frequency, followed by AEs, psychiatric symptoms, ASM burden, and cognitive symptoms.The top three were subsequently entered into the epilepsy-DOOR.
Outcomes are measured longitudinally comparing baseline measures to the follow-up measure(s) as available.Change in seizure frequency is defined as >90% reduction, >50% reduction, >25% reduction, or <25% reduction as recorded in participant seizure diaries.AEs are defined according to the CTCAE gradings, with AEs of any severity lasting <72 h considered mild temporary for the purposes of the DOOR.Change in quality of life scores are measured using QOLIE-31, with a change of ±5.19 or more being an improvement/worsening of quality of life. 11Neuropsychiatric symptoms are measured using the NDDI-E and GAD-7, with changes of ≥6 and ≥4, respectively, indicative of change in symptom severity.Change in cognitive performance is defined as ±3 points on the verbal paired associates delay (A7). 12Increase in ASM burden is defined as introduction of a new ASM or a ≥25% increase in dose of a current ASM.

| Category inclusion and prioritization
The consumer prioritization of levels of outcomes, resulting in a ranking of 60 possible individual ranks: the full factorial combination of four levels on seizure frequency, five levels on AEs, and three levels on neuropsychiatric symptoms.The first iteration of the DOOR-epilepsy is found in Table 2. Briefly, the DOOR can be split into four major strata: the bottom stratum, with the worst outcomes (49-60) characterized by severe ongoing AEs, irrespective of effects on other measures, the third stratum represented by minimal (<25%) reduction in seizure frequency (37-48); the second stratum (28-36), again represented by moderate ongoing AEs, and the first stratum (1-27), which is not represented by a singular factor.
The consensus from the consumer group discussion was that quality of life (which was not included as an outcome measure in the original questionnaires) was more important than neuropsychiatric symptoms and thus a quality of life questionnaire (QOLIE-31) was included as the third outcome in the epilepsy-DOOR.The consumer discussion did not result in any changes to the bottom 3 strata of rankings; however, numerous minor changes were made to the first stratum as shown in Table 2.

| Application/implementation of epilepsy-DOOR to compare a random pair of participants
To enable the analysis within the win odds framework (see below for details) it is necessary to compare all participants as randomly selected pairs allocated to different study arms.The following example demonstrates the application of epilepsy-DOOR in a hypothetical clinical trial.At baseline, Patient A has 1 seizure per day and scores 72 on the QOLIE-31.They are randomized to the active treatment arm and commence the study drug.They experienced mild nausea and dizziness in the first 2 days, but this was resolved by day 3 (mild temporary AE).They report no other AEs.At the end of the study, their seizures have reduced to 1 every 3 days (>50% reduction), and their QOLIE-31 score is 78 (+6 points; improvement).As defined in epilepsy-DOOR tool, Patient A's epilepsy-DOOR is 12. Patient B is randomized to the placebo arm.They have 5 seizures per month and a QOLIE-31 score of 79 at baseline.They do not report any AEs during the study (nil AEs).In the end, their seizures have reduced to 3 per month (>25% reduction) and their QOLIE-31 score is 80 (+1 point; no change), thus their epilepsy-DOOR is 17.Comparing these two patients would result in a "win" for the treatment arm and a "loss" for the placebo arm as Patient A's rank is better than Patient B's.Patient C is randomized to the treatment arm.At baseline, they have 4 seizures per week and a QOLIE-31 score of 65.They experience intermittent moderate dizziness and somnolence while taking the study medication (moderate AE).At the end of the study, their seizures are reduced to 1 per week (>50% reduction), and their QOLIE-31 score is 67 (+2 points; no change).Patient C's epilepsy-DOOR is 32.The comparison between Patient B and Patient C will result in a "loss" for the treatment arm and a "win" for the placebo arm.Patient D is randomized to the placebo arm.They have 1 seizure per week and a QOLIE-31 score of 68 at baseline.They do not report any AEs (nil AEs).At the end of the study, their seizure frequency remains 1 per week (<25% reduction) and their QOLIE-31 score is 75 (+7 point; improvement), resulting in an epilepsy-DOOR of 37. Patient C is ranked higher than Patient D, resulting in a "win" for the treatment arm and a "loss" for the placebo arm.
Using the examples described above, if Patients A and B and Patients C and D were compared, Patients A and C would both "win," indicating superiority (or greater benefit than harm) of the treatment over placebo.However, if Patients A and D, and Patients B and C were compared, Patients A and B would "win" demonstrating equal wins between the two groups and, therefore, no benefit of treatment.

| Utilization of epilepsy-DOOR as a study outcome
During the trial outcome analyses, the DOOR distributions are compared between trial arms.Confidence intervals are used to estimate the treatment effect that tallies all the recorded "wins" and "losses" from all possible treatment/placebo patient pairs into the probability (or the odds) that a randomly selected participant in arm A will have a better DOOR than a randomly selected participant in arm B. In case "ties" are recorded, their number is split The full epilepsy-DOOR prior to and following the consumer discussion forum.Sample size calculations can be completed based on the Win Odds, using estimated proportions of "wins," "ties," and "losses" between groups, see Johns et al. 14 for further details and a downloadable tool on calculating sample sizes from Win Odds.

| DISCUSSION
Here we report a novel consumer-codesigned trial outcome measure, epilepsy-DOOR, designed by adapting the previously reported DOOR approach for use in adult drugresistant epilepsy clinical trials.Epilepsy-DOOR combines three common epilepsy trial outcome measures and integrates them into a single outcome measure.This diversifies clinical trial methodologies, extending co-primary endpoints into a multifaceted, but single measure of study outcome.Interventional epilepsy studies are typically powered to detect a >50% reduction in seizure frequency for adjunctive therapies in drug-resistant epilepsy and seizure freedom rates in newly diagnosed epilepsy.Neither of these approaches reflects treatment-emergent AEs or other potential harms that could impact long-term treatment compliance.We developed epilepsy-DOOR for use as the primary outcome in our forthcoming clinical trial of sodium selenate as a disease-modifying treatment for epilepsy (Vivash et al., in press, ACTRN12623000446662).In this trial, we will use 'standard' outcome measures (seizure frequency, quality of life, cognition, neuropsychiatric symptoms) and statistical analysis techniques (LMEMs) as secondary outcomes.By utilizing these established techniques as secondary measures, we will evaluate the epilepsy-DOOR framework against the standard methods.How epilepsy-DOOR is received by the regulatory authorities remains to be seen.However, we believe that it represents a relevant outcome measure for disease-modifying treatment trials, rather than the traditional symptomatic antiseizure medication trials, as it aligns with the increasing understanding that the burden of epilepsy represents more than seizure frequency alone. 1,2,15The FDA patient-reported outcome measures (PROMs) guidance document, 16 provides a framework for development, validation, and use of PROMs as trial endpoints.DOORs could be validated through a similar pathway.Application to real-world data will be the first step in the evaluation and optimization of epilepsy-DOOR as an outcome measure for future pivotal trials.
The epilepsy-DOOR includes measures that have some degree of overlap.The QOLIE-31 includes questions about seizure worry and perception of some common adverse events, which will be influenced by seizure burden and presence of adverse events, respectively.While these will be somewhat interdependent, there are subtle differences in the capture and characterization of these measures (presence vs perception of adverse events, quantification vs perception of seizures) and the time frame for information (52 weeks for the adverse events, previous 4 weeks for QOLIE-31).The QOLIE-31 is also heavily impacted by mood, therefore, when using the epilepsy-DOOR it will be necessary to ensure maintenance of mood stabilizers to demonstrate the impacts on the QOLIE-31 (and, therefore, epilepsy-DOOR) are driven by changes to someone's epilepsy and downstream impacts on mood rather than changes in mood symptoms driven by response to mood stabilizers independent of epilepsy treatment.
While the epilepsy-DOOR measure reported here is only applicable to interventional studies in longstanding drug-resistant focal epilepsy in adults with relatively mild comorbidities, the consumer population with whom it was developed, we have demonstrated the adaptation of the DOOR framework to a chronic neurological disease.Similar consumer-driven multioutcome measures could be tailored to pediatric populations, or other developmental and epileptic encephalopathies as well as other diseases with relative ease.Evans et al. 4 provide key considerations for adaptation and a use case example.

T A B L E 1
Potential outcomes for inclusion in DOOR-epilepsy.Note: Asterisks (*) indicate those included in the final epilepsy-DOOR.

F I G U R E 1
Schematic flow chart of the development of the epilepsy-DOOR.The initial questionnaire was designed by experienced clinicians based on their clinical and research experience.The questionnaire was administered to people with epilepsy (n = 7), and the first version of the epilepsy-DOOR constructed.This was then reviewed by the consumer focus group (n = 8) who revised and optimized the outcome selection and order of the rank.