A review of studies evaluating the effectiveness of risk minimisation measures in Europe using the European Union electronic Register of Post‐Authorization Studies

Abstract Purpose An important element of risk management is the planning and implementation of risk minimisation measures (RMMs) and the evaluation of their effectiveness by process or outcome indicators. The aim of this review is to summarize the characteristics of risk minimisation (RM) effectiveness studies in Europe and provide an overview of RMMs and their effectiveness. Methods This was a qualitative review of RM effectiveness studies in the European Union electronic Register of Post‐Authorization Studies (EU PAS Register); data extracted included study design, population, sample size, data sources, drug information, RMMs, study period, indicators, and their reported effectiveness. Results Of the 872 records in the EU PAS Register, 19 studies evaluating the effectiveness of RMMs were included. Eleven were cross‐sectional surveys and 8 used secondary data sources. Eighty‐nine percent (17/19) evaluated additional RMMs (used when routine RMMs are considered insufficient), and 36% (7/19) evaluated changes in routine RMMs (applicable to all medicinal products). A total of 42 effectiveness indicators were identified: 18 process and 24 outcomes. Half of the indicators (21/42) were successful; 2% (1/42) indicators were partially successful; 17% (7/42) indicators were inconclusive. Effectiveness of the remaining 31% (13/42) indicators could not be determined owing to limited information. The United Kingdom was the most frequent country for the conduct of RM effectiveness studies. Conclusions Most of the included studies evaluated additional RMMs. Half of the effectiveness indicators (process and/or outcome) were reported as successful. This review provides evidence to support the development of future guidance on the effectiveness of RM in Europe.

According to GVP module XVI, "RMMs are interventions intended to prevent or reduce the occurrence of adverse drug reactions associated with the exposure to a medicine or to reduce their severity or impact on the patient, should adverse reactions occur". 4 Marketing authorization holders are required to monitor the outcome of RMMs, which are included in the risk management plan (RMP) or as a condition of market authorization. The RMMs may be classified as routine (rRMMs) or additional risk minimisation measures (aRMMs). The rRMMs are applicable to all medicinal products, and the majority of safety concerns are adequately addressed by rRMMs. When rRMMs may not be sufficient, aRMMs may be required to manage and mitigate the risk(s) that supplement rRMMs. 4 Effectiveness of RMMs is mainly evaluated for aRMMs but also sometimes for rRMMs. The effectiveness of RMMs can be evaluated by process and/or outcome indicators. Process indicators measure the extent to which a programme was implemented, whether the execution was as planned, and the impact on knowledge and behaviour of the target population. Outcome indicators provide an overall measure of the level of risk control achieved by RMM, for example, measuring rates of an adverse drug reaction or other safety-related outcome. 4 Evaluation of effectiveness of RMMs is important to manage the benefit-risk balance of a medicinal product. Effectiveness of RMMs can be evaluated by using cross-sectional survey studies and studies using secondary data sources.  [5][6][7] The EU PAS Register includes study documents such as study protocol and report of the registered studies (based on status-planned, ongoing, or completed), which provides a unique opportunity to examine study details. According to the GVP VIII, marketing authorization holders are legally required to register non-interventional post-authorisation safety studies (PASS) imposed as an obligation (ie categories 1 and 2).
It is also recommended to register all category 3 (required in the RMP) non-interventional PASS and any other PASS to support transparency and facilitate exchange of information between different stakeholders. 8 Therefore, the EU PAS Register is a valuable resource for PASS, including those evaluating the effectiveness of RMMs 9 and those mandated by the European Medicines Agency. A review conducted by Gridchyna et al 10 using MEDLINE and Embase included published studies evaluating the effectiveness of RMMs worldwide up to 2013, which was before GVP XVI was adopted. However, a comprehensive review of studies assessing the effectiveness of RMMs in European countries using EU PAS register is lacking including the studies initiated after GVP XVI. Therefore, the aim of this review is to describe and summarize PASS evaluating the effectiveness of RMMs in Europe and provide an overview of the RMMs and their effectiveness.

| METHODS
This study was a qualitative review of RMM effectiveness studies using the EU PAS Register. All studies registered in the EU PAS Register from 2010 to 30 August 2016 were screened. Study titles were screened to identify those assessing the effectiveness of RMMs (hereinafter "RM effectiveness studies") using keywords such as "minimisation," "survey," "effectiveness," "drug utilization," "behaviour," "knowledge," "materials," and "physician." Studies were reviewed using information provided in the register. If there was any ambiguity of eligibility, the study protocol and/or report was reviewed. Studies were included if they evaluated the effectiveness of RMM(s), were conducted in at least one European country, and a report or executive summary was available in the EU PAS Register. The screening process and data

KEY POINTS
• The EU PAS Register is a valuable resource to identify post authorization studies evaluating the effectiveness of risk minimisation measures in Europe, for which study protocols and reports are available.
• This review summarizes the different routine and additional risk minimisation measures assessed in risk minimisation effectiveness studies.
• This review provides an overview of the different process and outcome indicators used to assess the effectiveness of risk minimisation measures.
• The majority of included studies did not pre-specify a threshold for success.
• This review provides evidence to support further development of the guidance. extraction were conducted by 2 independent reviewers (P. V. and E. A.), and discrepancies were discussed and resolved. For the purpose of data extraction, the final version of study reports was used. When the study report was absent, the executive summary was used. All references were managed by using EndNote X7 (Thomson Reuters, USA). Table 1 summarizes the variables abstracted from the included studies and their corresponding operational definitions.

Indicators
Process indicators: implementation and receipt of the RMM (eg educational materials reaching the target group and change in knowledge), understanding and awareness of HCPs or patients (eg knowledge gained by physicians about the importance of metabolic monitoring), and behavioural change (eg actual proportions of testing conducted by physicians). Outcome indicators: rates of an adverse drug reaction or other safety-related outcome; (eg reduction in the incidence of risk under consideration after the implementation of RMM). Note: One or more indicators could be assessed to evaluate the effectiveness of a single RMM. 4

Implementation date
Date of implementation of the RMM

Study period
The period of data collection (for surveys); the period for which the data were analyzed (for studies using secondary data sources).

Reported effectiveness (for each individual indicator)
Successful: the indicator assessing RMM achieved a pre-specified threshold, the study concluded that the RMM was successful, no further RMM was required, RMM was sufficient, or used similar terms. Inconclusive: the indicator assessing RMM did not achieve a pre-specified threshold, the study concluded that the results provided insufficient evidence, further analysis was required, or similar terms. Note: these categories were ascertained solely based on the study results and conclusions in the study report.
Abbreviations: aRMM, additional risk minimisation measure; DHPC, Dear Healthcare Professional Communications; GP, general practitioner; GVP, good pharmacovigilance practices; HCP, health care professional; PPPs, pregnancy prevention programmes; rRMM, routine risk minimisation measure; RMM, risk minimisation measure; SmPC summary of product characteristics. effective. The remaining 6 indicators were inconclusive, of which 1 indicator was understanding and knowledge of the physician's guide, 13 1 was receipt of educational materials by HCPs, 16 and the other 4 indicators were awareness, use, knowledge, and behaviour of using the checklist, Q&A brochure, and patient alert card. 21 All countries planned for the survey eventually participated in the study except for 1 study where the survey was not initiated in one country (Sweden). 13 Two studies planned to include both HCPs and patients; however, one study could not recruit patients owing to confidentiality regulations. 13 Table 2 summarizes the characteristics of the cross-sectional surveys evaluating the effectiveness of RMMs and their reported effectiveness.  Table 3 summarizes the characteristics of RM effectiveness studies using secondary data sources, and Figure S2 shows the countries where these studies were conducted. Seven were retrospective cohort studies, 11,18-20,26-28 and 1 was cross sectional. 17 All 7 retrospective studies were drug utilization studies of which 3 had a pre-post design. 11,19,26 The total sample size in the 7 retrospective cohort studies ranged from 687 to 34 975. 11,[18][19][20][26][27][28] One cross-sectional study used electronic medical records from the IMS® Disease Analyzer and included 294 subjects. 17 All 8 studies evaluated aRMMs that included educational materials, a pregnancy prevention programme, DHPCs; 5 studies also evaluated changes in rRMMs (SmPC/label changes). 11,20,26-28 These RMMs were evaluated by using outcome indicators. Of the total of 23 indicators, success was achieved with 8 indicators, 11,27,28 1 was inconclusive, 18 1 study with 1 indicator achieved success in 1 country but was inconclusive in the other included country (partially successful), 17 and success could not be determined due to limited reported information with remaining 13 indicators. 19,20,26,27 In general, the indicators examined changes in incidence of the risk under evaluation (n = 7), monitoring parameters (n = 7), drug use patterns (n = 5), pregnancy prevention (n = 1), and prescribing in patients with contraindications (n = 3). Three studies with 9 indicators used pre-post design to evaluate effectiveness. 11,19,26 Two studies were unable to use the planned data sources due to various challenges encountered. 11,18 These include medical records from Spain that required informed consent from all patients (deceased and living), medical records from Italy where physicians declined to participate owing to informed consent form requirements or lack of staff, Secure Anonymised Information Databank (SAIL) from the United Kingdom where ethics approval could not be obtained, General Practice Research Database where the study had to be carried out as feasibility study because Independent Scientific Advisory Committee claimed the study was not feasible, Tuscany regional database in Italy where the data could not be obtained in time, and Statistik Denmark from Denmark where it was identified that the required information for the study was limited.    and the data collection method. Two in every 3 surveys in the review used online questionnaires, providing real-time data efficiently and allowing the implementation of controls and data checks to enhance data quality (eg questions provided in a sequence, skipping patterns, and restricting changes). Also, survey studies are prone to recall, selfreport, and non-response biases if participants who participate differ from those who do not, resulting in a non-representative sample.

| Studies using secondary data sources
However, outcomes for which studies using secondary data might not be feasible, survey studies can provide valuable insights. Few survey studies had disproportionate participation across countries, which may be a reflection of the usage of the product or difficulties identifying prescribers in these countries that could affect the generalizability of the results. 12,21,24 Some studies encountered operational challenges such as access, approval, feasibility, and resources, which hindered their participation. 11,18 Sample size or the number of participants in survey studies can affect the results of the study if targeted number of participants is not achieved. Two studies in the review did not reach the pre-specified sample size 13,21 ; of which one only recruited half of the planned number of participants and was inconclusive. 21 Recruitment might be challenging in surveys and may limit generalizability (eg 1 survey study planned in patients and HCPsthe patient survey part was excluded owing to confidentiality regulations). 13 For studies using secondary data, quality of the data sources as well as validation of the outcomes is important to consider. Threshold for success of a RMM should be defined on a case-by-case basis considering the outcome, and a rationale for selection of the threshold should ideally be reported. While there should be strict threshold criteria for adverse events, for example, no pregnancy during exposure to teratogenic drugs, in other cases, the event cannot be mitigated and the RMM will only result in a reduction of the severity or aim to promote better control through monitoring or promoting action. However, few studies in this review reported a threshold for success and results were difficult to interpret for the others, highlighting the need for further guidance.
One potential approach for RM effectiveness could include survey component to evaluate process indicator(s) and secondary use of data to evaluate outcome indicator(s). One such example is included in this review, where physician's self-reported behaviour (process indicator) was assessed through a cross-sectional survey and monitoring of patients by physicians (outcome indicator) was assessed by using secondary data. 16,17 As these were 2 separate studies, no correlation of process and outcome indicators was reported.

| CONCLUSIONS
Most of the included studies evaluated aRMMs, and some also evaluated rRMMs. Cross-sectional surveys were used to assess process  interpretation, and use of data sources are required to conduct RM effectiveness studies with high quality standards.