Criteria for considering studies for this review
Types of studies
Clinical drug trials that include a comparison of two or more methods to elicit participant-reported AEs
Research studies that have been performed outside the context of a clinical drug trial to compare two or more methods to elicit participant-reported AEs, and which could be used in clinical trials (evidenced by reference to such applicability)
Types of data
AEs elicited from participants taking part in any such clinical trial. For the purposes of this review, AEs are defined as those outcomes that were pre-specified as potential AEs to be investigated in the trial (including expected or unexpected AEs, the latter which will not be known, but are intended to be detected during the trial/study), recognizing that the trial itself might reveal that these are not actually increased in the intervention group compared with the control group. Concomitant medication and medical history data will also be included in this review, should eligible studies also include a comparison of methods used to elicit those.
Types of methods
Any combination of elicitation methods within- or between-participants. This may include, but is not limited to, unstructured or structured enquiries, checklists or questionnaires (e.g. by body system, symptom etc), diaries and memory aids.
Types of outcome measures
The effect measure (or number, proportion) and/or nature (e.g. characteristics, severity, causality assessment) of AEs identified by the method of elicitation, as defined by the original authors
If relevant, the effect measure (or number, proportion) and/or nature (e.g. medication class) of concomitant medications and/or medical histories identified by the method of elicitation, as defined by the original authors
If relevant, summary results of qualitative methods used
If relevant, results of inherent elicitation method validation studies.
Search methods for identification of studies
There will be no date, sample size or language restrictions in the searches. However, it is likely that only English reports will be included fully in the review, because of resource constraints as regards translation.
Electronic search strategies will be developed in consultation with an experienced information specialist for relevant bibliographic databases. These are likely to include MEDLINE, Popline, The Cochrane Library (Cochrane Central Register of Controlled Trials and Cochrane Methodology Register), CINAHL, CAB Abstracts, BIOSIS, JSTOR, and SCISEARCH. A list of databases and search strategies will be finalized prior to starting the search, with subsequent iterations fully documented.
Searching other resources
The electronic searches will be supplemented by a review author checking the following: reference lists of included reports (and relevant reports known to the authors who are familiar with the research area) (Horsley 2011), handsearching of relevant topic-area conference abstracts (e.g. International Conference on Pharmacoepidemiology and Therapeutic Risk Management, International Society of Pharmacovigilance annual meeting) (Scherer 2007), and searching online libraries of theses/dissertations. We will ask known content experts for information about potentially eligible studies that we may have missed.
Data collection and analysis
Selection of studies
The first author will examine titles and, where available, abstracts of identified citations in order to remove obviously irrelevant reports (e.g. non-human studies). Thereafter, two authors will independently review the remaining titles and abstracts for eligibility according to the Criteria for considering studies for this review in this protocol. The full text for all reports that appear relevant will be obtained, as well as those for which the title and abstract is insufficient to determine eligibility. Reports from the same piece of research will be linked together. Two review authors will determine final eligibility independently, with disagreements resolved by discussion (involving a third person with relevant experience if necessary). Review authors will not be blinded to any information, and all documents relating to this search and selection process, including the primary reason for non-inclusion, will be recorded.
Data extraction and management
One review author will extract data onto a data extraction form according to a pre-specified list, with a second review author checking 100% of fields. Disagreements will be resolved by consensus, with, if necessary, a third person with relevant experience consulted to resolve disagreements. The list will be pre-tested in a minimum of two reports and modified accordingly. It is likely to include the following.
Citation, author, country and contact details
Source (journal or other)
Study design and methods:
Country/countries where study was conducted
Setting (hospital, clinic etc)
Design (description of trial arm(s), disease or indication [including whether acute or chronic], sampling strategy, intervention [if relevant], assessment(s) schedule, duration of follow-up)
Elicitation techniques’ properties (including, if available, descriptions of their development, components and application methods. Also training/experience of those who elicited information from participants, the detail of how AEs were described and by whom, whether verbatim reports of participants were captured, the language or dialect used in conversations. In addition how reports were analysed, verified and recorded)
References to animal or human toxicology, pharmacovigilance databases, participants or patient/consumer experiences (including explanations for differential reportings, such as qualitative results, and underlying conceptual theories or orientations)
Validation technique method details, if relevant
Outcomes and results
The relative effect estimates derived from one method of ascertainment versus the other
The number/proportion and/or nature of AEs as defined by the authors of the original study
If relevant, the relative effect estimates/number/proportion and/or nature of concomitant medications and/or medical histories
If relevant, summary results of qualitative methods used
if relevant, statistical test results (including those from validation studies)
Assessment of risk of bias in included studies
It is anticipated that the comparison of elicitation methods will largely be within-participants and not between-participants. Furthermore, it is possible that comparisons within-participants may involve the addition of one elicitation method to another (i.e. cumulative, rather than direct, comparison). Such heterogeneity in study designs will limit our ability to assess methodological quality and risk of bias using currently accepted methods. However, for reports that do compare outcomes between-participants, the risk of bias will be independently assessed by two review authors, according to The Cochrane Collaboration’s 'Risk of bias' tool, as far as is feasible in terms of the actual study design encountered (Cochrane 2011). For reports that cannot be assessed in this way, including those involving a within-participant comparison, the review authors will independently critically evaluate each study in terms of the potential impact of the study’s design and conduct on its findings. These general observations, which will be made within the framework of potential selection, performance, detection, attrition, reporting and other biases, will be discussed and reported in the full review.
It is acknowledged that Risk of bias' assessment is dependent on the completeness and quality of the original study report and an attempt will be made to contact study authors to retrieve protocols or specific relevant missing information (Young 2011). Open questions will be used in these communications to minimize bias in reporting. Reports will not be excluded from this review on the basis of quality.
Measures of the effect of the methods
As mentioned above, the comparison of elicitation methods may be within-participants or between-participants, with the latter involving a cumulative rather than direct comparison. Therefore, the details of all measures of effects of methods studied will only be known once the review is ongoing. Effect measures from different methods can be compared by taking their ratio, or overlap in 95% confidence interval (Golder 2011).
Unit of analysis issues
The details of units of analysis will only be known once the review is ongoing.
Dealing with missing data
We will seek to minimize the amount of missing data through contact with authors as mentioned above (Young 2011). Thereafter, we will report any assumptions made about missing data, any statistical methods used to impute them and the potential impact of these methods on the findings of the review. Sensitivity analyses may be applied where assumptions have been made.
Assessment of heterogeneity
If pooled estimates are calculated, heterogeneity will be assessed by the Q test and I2 statistic (Higgins 2002).
Assessment of reporting biases
If a meta-analysis is conducted, reporting bias will be explored using a funnel plot, whereby the studies’ measures of effects will be plotted against a measure of precision (Sterne 2001). We do not, however, anticipate sufficient eligible studies to explore asymmetry of the plot statistically.
It is anticipated that meta-analysis is unlikely to be feasible due to the heterogeneity of included studies, unless similar studies are identified. Therefore, results are likely to be described in narrative form. However, if meta-analysis is appropriate for any outcomes, overall pooled estimates will be calculated using random-effects models as significant heterogeneity is anticipated. The review report will provide a summary description of each included study’s methods, results, strengths and limitations.
Subgroup analysis and investigation of heterogeneity
If necessary, we will conduct subgroup analyses by study or elicitation method.
If necessary, the potential impact of risk of bias and assumptions about missing data will be investigated by sensitivity analyses.