Impact of schizophrenia relapse definition on the comparative effectiveness of oral versus injectable antipsychotics: A systematic review and meta‐analysis of observational studies

Abstract Although relapse is an important outcome to measure the effectiveness of schizophrenia treatment, no standard definition exists. This review aimed at identifying definitions and measurements of schizophrenia relapse in observational studies of long‐acting injectables (LAIs) versus oral antipsychotics (OAPs) and at determining their impact on heterogeneity of comparative effectiveness estimates. A systematic review was conducted using MEDLINE and Embase (01 January 2010–11 November 2019 [date last searched]). Pragmatic searches of gray literature and snowballing were also conducted. Search outputs were screened independently by two assessors at first stage, and full‐text of potentially eligible sources at second stage. For each retained source, definition and measurement of relapse, study methods, and comparative effectiveness estimates were extracted. Heterogeneity of estimates was assessed using I 2 statistic with a threshold of 50% for substantial heterogeneity. Literature search yielded 543 sources and pragmatic searches, 21, of which 35 were eligible. Twelve definitions of relapse were found based on hospitalization/emergency department (ED) data (28 studies) or clinical assessment (5 studies). No definition was provided in five studies. According to quantitative analyses, in studies defining relapse as schizophrenia‐related hospitalization and/or ED visits over 1‐year follow‐up, LAIs were significantly more effective than OAPs. For studies measuring relapse based on all‐cause hospitalization, heterogeneity was too high for pooling; yet this definition is the most frequently found in pooled estimates published in the literature. Schizophrenia relapse definitions led to substantial heterogeneity of comparative effectiveness estimates of LAIs versus OAPs. Creating study subgroups based on relapse definition effectively reduces statistical heterogeneity.


| INTRODUC TI ON
Schizophrenia is a chronic disabling disorder that affects 1% of the world population. 1 Most patients with schizophrenia experience multiple relapses, characterized by worsening of psychotic symptoms leading to progressive cognitive deterioration, impaired functioning, hospitalizations, greater risk of suicide, and reduced quality of life. [1][2][3][4] Treatment nonadherence to oral antipsychotics (OAPs), ranging from 40% to 50% and even as high as 89%, 5,6 is a known barrier to treatment success in this population 7 and a major contributor to relapse. 8 For patients who are not adherent to OAP treatment, clinical guidelines recommend long-acting injectables (LAIs). 5 Although it is recognized that LAIs are associated with more favorable treatment adherence than OAPs, 7,8 observed benefits on clinical outcomes, such as relapse, are conflicting. 9 According to a systematic review of 87 observational studies, 1 schizophrenia relapse is a complex condition for which a patient may relapse without the need for hospitalization such as experiencing a moderate symptom exacerbation; therefore, there are currently no established criteria used to define schizophrenia relapse.
It is hypothesized that the definition of relapse is a source of heterogeneity across comparative effectiveness studies of LAIs versus OAPs, thereby hampering the pooling of estimates through metaanalysis. Furthermore, the handling of death in studies (i.e., exclusion criterion, censoring criterion, or not considered) may further contribute to the heterogeneity as relapse may be associated with suicide-related mortality in this population. 10 This study aimed at identifying the definitions and measurements of schizophrenia relapse used in observational studies of LAIs versus OAPs, and at determining their impact on the heterogeneity of comparative effectiveness estimates.

| Study design
A systematic review was conducted and reported according to the preferred reporting items for systematic review and meta-analysis (PRISMA) statement. 11 Protocol was registered in PROSPERO under CRD42020162054.

| Information sources
The literature search was conducted in Ovid MEDLINE and Embase using free-text keywords and thesaurus terms (i.e., MeSH and Emtree terms, respectively for MEDLINE and Embase). Search period covered 01 January 2020 until 11 November 2019, date last searched. Search of the databases was in English, but outputs published in English, French, or Spanish were considered.
Pragmatic searches involved web searches using Google (www. google.com/) and Google Scholar (www.google.com/schol ar/) search engines as well as the OpenGrey database. In addition, websites of relevant learned societies were searched in December 2019 using the keywords "schizophrenia relapse." The lists of references of retained sources and reviews were also hand-searched for additional sources (snowballing).

| Study selection
Search outputs were uploaded into EndNote X9 and duplicates were deleted. In phase 1, sources were screened independently by two reviewers (with conflicts resolved by a third), followed by an in-depth review of retained sources to confirm eligibility (phase 2). Study selection is presented in a PRISMA flowchart ( Figure 1). 11

| Eligibility criteria
The PICOTS criteria are as follows:

| Extracted data
Data extraction was conducted by a single reviewer, and the data extraction form was first piloted by two independent assessors (with conflicts resolved by a third). In addition to the outcomes of interest, data extracted included study design, data source, sample size, follow-up duration, patient characteristics (age and sex distribution, ethnicity, comorbidities), number of patients treated with either OAPs or LAIs, and treatment names. No attempt was made to contact the study authors for additional data. The review was K E Y W O R D S comparative effectiveness, long-acting injectable antipsychotics, meta-analysis, oral antipsychotics, schizophrenia relapse, systematic review conducted according to methods proposed by the Cochrane group 12 and the Institute of Medicine (IOM) of the National Academy of Medicine (NAM). 13

| Synthesis of evidence (including meta-analysis)
The subset of studies that reported comparative effectiveness estimates of LAIs versus OAPs were retained for the assessment of heterogeneity. First, a qualitative assessment of methodological and clinical heterogeneity was undertaken based on study design (risk estimate and follow-up duration) and study populations (age and sex distributions). If two or more studies were clinically and methodologically homogeneous, statistical heterogeneity of estimates was quantified using I 2 statistic, which is the percentage of variation across studies due to heterogeneity rather than chance. When overall heterogeneity exceeded 50%, 14 studies were stratified by relapse definition. An I 2 was calculated for each subgroup, and if within the acceptable range, comparative effectiveness estimates were pooled using the inverse variance method with a random effects model. 15 Pooled estimates and corresponding 95% CIs were obtained using the Review Manager Software planned. However, this was deemed unfeasible as the number of studies retained in each subgroup was less than 10, which is the minimum required. 16 A methodological quality assessment of individual studies was not undertaken.

| Search results
The literature search yielded 543 sources, of which 36 were retained for in-depth review. There were 22 sources that were further ex-
Depending on the study, the same definition was based on different data sources and measurements. For example, hospitalization was either measured using administrative claims, 18,21,24,[27][28][29] electronic medical records (EMRs), 7 medical charts, 17,19,[22][23][24][25][26]30 or reports from mental health centers and hospitals from integrated healthcare systems such as the Veterans Affairs (VA) in the United States (US). 20 Schizophrenia-related hospitalization was measured using administrative claims 38,39 based on diagnostic codes ICD-9-CM 295.0-295.9, 37 national discharge registers, 42,43 and EMRs. 40,41 Likewise, psychiatricrelated hospitalizations were identified through EMRs, 35 national patient registers using ICD-10 codes F00-F99 and F20-F29, 31,32 or medical chart reviews from psychiatric hospitals and mental health centers. 20,33,34,36,44 Another example is seen with the increase in CGI-S scores for which thresholds used to define relapse differed from one study to the other. 17,44 These findings highlight the need to consider both definition of relapse as well as method of measurement.

| Handling of death
Death handling was only documented in 6 (17.1%) of the 35 studies, 17,21,31,32,42,43 and was considered as a study outcome in three studies. 32,42,43 The first was a nationwide prospective cohort study conducted in Sweden between 2006 and 2013. Death, distinguished from suicide attempts, was assessed as part of a composite outcome of treatment failure. 32 In two studies from Finland (one retrospective cohort and one prospective cohort), the risk of all-cause death was a main outcome of interest. 42,43 Of note, in all three studies, death was not an outcome associated with schizophrenia relapse. In other cases, death was considered as a censoring criterion (2 studies) 17,31 or an exclusion criterion in one retrospective cohort study. 21

| Handling of loss to follow-up
The handling of loss to follow-up was specified in a minority of publications (10 studies): half used loss to follow-up as censoring criterion, 17,20,25,33,43 while the other half excluded patients lost to follow-up. 7,18,24,29,44 The quantitative impact of handling of loss to follow-up on the comparative effectiveness estimates could not be assessed in this review, owing to the small number of studies.
However, it is a known contributor to heterogeneity across observational studies. The I 2 statistic of HR estimates was 91%, exceeding the acceptable threshold for pooling (forest plot depicted in Figure S1). In the presence of such considerable heterogeneity, subgroups were further investigated based on relapse definition. 39,40 ) Owing to the lack of heterogeneity (I 2 = 0%; p = .39), estimates were pooled resulting in a HR of 0.68 (95% CI, 0.66-0.70) ( Figure S2). Both studies focused on second-generation LAIs, specifically, paliperidone palmitate. However, it is important to note the weight attributed to each study, which might also explain the absence of heterogeneity. Evidently, the estimate obtained from the larger study (n = 45 625) 39 dominates the findings from the other study (n = 218). 40 3.5.4 | Hospitalization as an indicator of schizophrenia relapse (n = 2 studies 23,28 ) Substantial heterogeneity was observed (I 2 = 65%; p = .09), hence pooling was not recommended ( Figure S3). Of note, hospitalization was measured differently from one study to another. In the retro- In addition, comparisons were made using overall LAIs, rather than focusing on a specific type.  The heterogeneity of estimates originating from three studies that followed patients over a 2-year period was acceptable (I 2 = 50%; p = .14). Consequently, the pooled HR estimate was 0.71 (95% CI, 0.43-1.16) ( Figure S4). Although 50% is an acceptable threshold for heterogeneity, Cochrane guidelines indicate that an I 2 between 30% and 60% may nonetheless represent moderate heterogeneity. 52 Therefore, a subgroup analysis was conducted in an attempt to further reduce heterogeneity. In two studies that defined relapse as schizophrenia-related hospitalization, 41,42 heterogeneity decreased to 42% ( Figure S5). The corresponding pooled estimate was not statistically significant (HR = 0.80; 95% CI, 0.56-1.14). In this analysis, more weight was attributed to the larger study (n = 3828) 41 than the other study (n = 2588). 42 3.5.6 | Studies reporting RR estimates (n = 2 studies 18,43 ) Substantial heterogeneity (I 2 = 83%; p = .01) was observed ( Figure S6).

| Schizophrenia-related hospitalization and/or ED visits (n = 2 studies
Both studies, equally weighted, used different definitions of schizophrenia relapse. In a mirror-image study, administrative claims were utilized to identify cases of relapse defined as hospitalization and/ or ED visit. 18 In a prospective cohort study from Finland, data from a national discharge register were utilized to identify schizophreniarelated hospitalization as an indicator of relapse. 43 3.5.7 | Studies reporting ORs (n = 2 studies 33,37 ) The heterogeneity was substantial (I 2 = 66%; p = .09) ( Figure S7).
Each study defined and measured relapse differently. The first study defined relapse as schizophrenia-related hospitalization according to ICD-9-CM codes 295.xx. 37 The other study used medical charts to identify psychiatric-related hospitalizations. 33

| DISCUSS ION
This systematic review confirms that there is no standard method to define or measure schizophrenia relapse in observational studies, which is a major contributor to heterogeneity in estimates of to define relapse as it is a simple measure that provides tangible data. 1 However, it is important to consider that the threshold for hospitalization may vary and reflect the healthcare system in specific countries or during specific time periods. 53 In addition, not all clinically significant exacerbations of symptoms or relapses will result in hospitalization, and therefore this defining criterion might provide a limited view of potential differences in outcomes associated with the use of LAIs compared to OAPs. 37  OAPs restricts conclusions that may be drawn from meta-analyses.
Specifically, a meta-regression to evaluate the concurrent effect of multiple potential sources of heterogeneity could not be performed, as a minimum of 10 studies is required. In addition, while the I 2 statistic is independent of the number of studies, it is found to be imprecise in small meta-analyses, 55 further complicating the assessment of heterogeneity. The weight attributed to each study also impacts pooled estimates. Studies with disproportionally larger sample sizes tend to have more precise results due to a smaller standard error and dominate the findings from smaller studies which may contribute to a lack of heterogeneity observed and is often a challenge found in pooling observational studies. 52 The limited number of studies retained for the meta-analysis also rendered the assessment of publication bias unfeasible, as statistical power of Egger's test may be too low to distinguish between chance asymmetry and real asymmetry when the number of studies is less than 10. 56 Thus, a conclusion on whether the definition and measure of relapse impacted the heterogeneity cannot be drawn in this instance.
This study, however, presents several strengths. It offers a com- Findings from this review suggest that the variation in relapse definitions is a main source of heterogeneity in comparative effectiveness studies of schizophrenia treatment, which hampers the synthesis of evidence on treatment effectiveness. This calls for efforts to attempt to standardize the case ascertainment of schizophrenia relapse in observational studies.

ACK N OWLED G EM ENTS
TC received a studentship from Mitacs through the Mitacs Accelerate program.

D I SCLOS U R E
All authors are employees of YolaRx Consultants that provides consultancy services to pharmaceutical companies, some of which are drug manufacturers of schizophrenia treatment, but not related to the work presented in this paper. No other conflicts of interest to declare.

AUTH O R CO NTR I B UTI O N S
TC developed the protocol and conducted all activities related to the systematic review (searches, data extraction, meta-analysis) as well as manuscript preparation. GC was the second independent reviewer for some of the searches and data extraction, and contributed to the meta-analysis. JFN was responsible for the assessment of clinical and statistical heterogeneity. YM supervised and provided oversight for all components of the project, including the manuscript. All authors reviewed and approved the final version of the manuscript.

E TH I C S A PPROVA L S TATE M E NT
As this was a systematic review, no ethics approval was sought.

DATA AVA I L A B I L I T Y S TAT E M E N T
Data sharing is not applicable to this article as no new data were created or analyzed in this study.