SEARCH

SEARCH BY CITATION

Keywords:

  • Systemic lupus erythematosus;
  • Clinical trials;
  • Preferences;
  • Qualitative research;
  • Prevention;
  • Atherosclerosis

Abstract

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. PATIENTS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. Acknowledgements
  8. AUTHOR CONTRIBUTIONS
  9. REFERENCES

Objective

A feasibility study for a trial of strategies for the prevention of atherosclerosis in patients with systemic lupus erythematosus (SLE) was stopped because of inadequate recruitment. There is little understanding of the factors influencing patients' decisions about participation in prevention trials. Our goal was to determine factors that patients with SLE consider in deciding about participating in prevention trials, to uncover concerns about SLE trials, and to investigate how study design and purpose affect participation decisions.

Methods

We conducted focus groups with trial participants (n = 13), trial nonparticipants (n = 8), and a group of patients with diabetes (n = 9). We conducted telephone interviews with SLE patients who refused participation in the trial and the focus groups (n = 10). A trained facilitator elicited factors influencing participation decisions. Transcripts were coded and grouped into themes using grounded theory.

Results

Demographic characteristics of the groups were similar. Seven factors emerged as important in decision making: current health status, study design, physician involvement, personal benefit, altruism, time, and incentives. These factors were considered by individuals who elected to participate and those who did not, but weighed differently. Among the trial participants, good health status, encouragement from one's physician, and desires to learn and to contribute stimulated participation. Reasons for nonparticipation included current health status, medication and randomization concerns, and personal factors.

Conclusion

We observed that similar factors were weighed differently by participants and nonparticipants. Our results suggest that strategies such as health education, enlistment of personal physicians, and limitation of time requirements may enhance recruitment of patients with SLE into clinical prevention trials.


INTRODUCTION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. PATIENTS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. Acknowledgements
  8. AUTHOR CONTRIBUTIONS
  9. REFERENCES

Systemic lupus erythematosus (SLE), a multisystem autoimmune inflammatory disease, is a chronic disease and the long-term sequelae of cumulative organ damage and/or its treatment now dominate the clinical picture. Accelerated atherosclerosis in particular is an important cause of premature morbidity and mortality, and the optimal way to prevent it is unknown. Although aggressive treatment of cardiac risk factors in patients with SLE is advocated (1–5), none of the prevention regimens or strategies have been critically evaluated. Clinical trials with the potential for important therapeutic benefit in the prevention of SLE-related atherosclerosis are therefore of paramount importance. A pilot study funded by the National Institutes of Health that intended to test the feasibility of such a prevention trial was terminated early because of insufficient enrollment (6).

To identify the factors and to understand how they influence SLE patient involvement in atherosclerosis prevention studies, we organized focus groups of patients with SLE who had been contacted for potential recruitment to a double-blind randomized controlled trial of strategies to prevent atherosclerosis. Our goals were to determine the reasons that patients with SLE choose to participate or not to participate in prevention clinical trials, to identify concerns about participation in SLE trials and other factors that influence these patients' decisions, and to investigate how aspects of study design and purpose affect participation decisions.

PATIENTS AND METHODS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. PATIENTS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. Acknowledgements
  8. AUTHOR CONTRIBUTIONS
  9. REFERENCES

Patient identification.

All patients with SLE who had been contacted for potential participation in the Prevention of Accelerated Atherosclerosis in SLE Study (PASS) were eligible. We mailed a letter to these patients, followed by a telephone call, inviting their participation in a focus group. PASS was a randomized controlled trial of prevention strategies, including a hydroxymethylglutaryl-coenzyme A reductase inhibitor (statin), low-dose aspirin, and an angiotensin-converting enzyme inhibitor, and individualized patient education. Details of the clinical trial are described elsewhere (6). (Participants were asked to undergo a 90-minute baseline visit with questionnaires and blood sampling, randomization to an unknown daily medication or placebo for up to 3 years, standard or individualized education by a trained nurse, a short visit at 3 weeks, telephone contact and questionnaires every 3 months, and yearly visits thereafter.) For comparison, we recruited all patients with diabetes mellitus from 1 primary care physician's practice (n = 37), with a similar letter and followup telephone call.

For the current study, we conducted 5 focus groups: 2 with patients who had been enrolled in PASS (participants), 2 with patients who had not participated in PASS (nonparticipants), and 1 with patients with diabetes. We conducted telephone interviews with patients who had not participated in the PASS trial and could not attend a focus group. All of the potential participants were followed in an academic health center. All aspects of the study were approved by the Partners HealthCare System Institutional Review Board and informed consent was obtained from all patients.

Focus group methodology.

An independent expert in focus group methodology ran all of the focus groups using a moderator guide developed by the research team (Table 1). An observer/note taker was present for each session. Focus groups were held in the evenings and on weekends and lasted 90 minutes. The goal was to generate revealing comments and insights about the topic in a spontaneous but structured conversation among small groups of strangers. The moderator prompted discussion with open-ended probes. The groups discussed factors that influence their decision regarding participation in clinical studies in general and specific concerns about study participation. Participants were not referred to by name and were not individually called upon for an answer to each question.

Table 1. Focus group questions
For participants
 1. How did you decide to participate in this trial? What factors did you consider?
 2. Did your health influence your decision to participate? If yes, what did you consider?
 3. Did the fact that the trial was aimed at the prevention of future heart attacks and strokes influence your decision?
 4. After agreeing to participate did you feel that you could change your mind? What could have been done differently?
 5. In general, how would you describe your experience in participating in this trial?
 6. Prior to this study have you participated in other medical research?
 7. Why did you decide to participate in this focus group?
 8. In what type of research studies are you interested in participating?
For nonparticipants and refusers (by telephone)
 1. How did you make the decision not to participate in the study? What factors did you consider?
 2. Did your health influence or motivate your decision not to participate? If yes, what did you consider?
 3. Prior to this study have you participated in other medical research?
 4. Why did you decide to participate in this focus group?
For control patients with diabetes
 1. Have you ever been asked to participate in medical research?
 2. How would you go about making a decision whether or not to participate in a medical research prevention study? What factors would you consider?
 3. In general how would you describe your experience with the recruitment process?
 4. After agreeing to participate did you feel that you could change your mind? What could have been done differently?
 5. Why did you decide to participate in this focus group?
 6. In what type of research studies are you interested in participating?

Theoretical saturation, or the point at which no new themes are identified by adding focus groups, was achieved after 2 focus groups. We audiotaped and transcribed verbatim all focus groups. Using grounded theory, in which the underlying concept is inferred and then verified in subsequent dialogue in an iterative fashion (7), we coded the transcripts for themes. One reviewer read all the transcripts for themes, categorized the transcripts, and then sorted them according to concepts that were coded explicitly. A second reviewer then re-read the transcripts and coded them according to the previously identified themes. The first reviewer then reviewed coding to ensure reliability and to check for discrepancies. The first reviewer also reviewed the focus group transcripts to check that the coding made conceptual sense in light of the question that the respondent was answering, and to group the themes to form super-ordinate concepts or categories. For example, themes related to factors that influenced the decision to participate in the study that pertained to illness, health stability, and medication concerns were grouped together. Once these categories were established, we then recorded for each focus group the number of participants by participant group responding according to the major category, and we examined the differences in narrative within and between groups.

At the end of each focus group, we asked participants to complete a questionnaire that elicited age, race and ethnicity, sex, employment status, level of education, and residence. Pearson's chi-square tests and analyses of variance were used to perform quantitative analyses of patient demographics.

RESULTS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. PATIENTS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. Acknowledgements
  8. AUTHOR CONTRIBUTIONS
  9. REFERENCES

Characteristics of the patients.

Of 181 patients originally contacted for the randomized controlled prevention trial, we obtained primary physician consent to recontact 158 for the focus groups. Therefore, 35 PASS participants and 123 PASS nonparticipants were invited to focus groups. Nonparticipants were oversampled because it was expected that this group would be more difficult to recruit, and we were particularly interested in reasons why individuals do not participate.

Twenty-one of the PASS participants expressed interest in the focus groups, and 13 (61.9%) participated in 1 of 2 focus groups (others canceled or did not come). Of the nonparticipants, 22 expressed interest, 15 were scheduled, and 8 of these came. Four of the 8 nonparticipants had declined participation in PASS. The remaining 4 had expressed interest but never participated because the trial had been terminated. Because the true refusers were of most interest and also the most difficult to engage, we attempted to interview by telephone an additional 16 patients and 10 agreed. Nine patients with diabetes mellitus agreed to participation in the focus groups, and we conducted 1 focus group with these patients using similar guiding questions.

Age, sex, educational level, and employment status of the participants in each group are shown in Table 2. A higher proportion of participants were working full time or part time and participants had a slightly higher number of years of education than nonparticipants and those who refused to participate in the focus groups; however, this was not statistically significant. The groups had similar racial compositions, and total household income and the percentage with private medical insurance did not differ between groups. Overall, group participants were typical of patients with SLE seen in the outpatient Arthritis Center at the Brigham and Women's Hospital, and of diabetes patients followed in the practice of 1 primary care physician.

Table 2. Characteristics of the participants*
CharacteristicParticipants (n = 13)Nonparticipants (n = 8)Refusers, telephone (n = 10)Diabetes (n = 9)P
  • *

    Values are the number (percentage) unless otherwise indicated.

  • Analysis of variance.

  • Pearson's chi-square test.

Age, mean ± SD years50.5 ± 10.048.4 ± 9.551.8 ± 11.158.0 ± 7.00.2
Female sex13 (100)6 (75)10 (100)9 (100) 
Disease duration, mean ± SD years17.8 ± 8.312.1 ± 8.214.1 ± 8.95.4 ± 2.40.008
Current medications, mean ± SD6.5 ± 2.94.4 ± 1.85.8 ± 12.08.1 ± 3.10.03
Education, mean ± SD years15.0 ± 3.114.0 ± 2.314.3 ± 3.613.3 ± 1.80.6
Living outside metropolitan Boston area8 (62)4 (50)8 (80)4 (44)0.37
Working full time or part time8 (62)4 (50)4 (40)4 (44)0.7
Disabled4 (31)2 (25)5 (50)2 (22)0.7
Racial/ethnic origins
 White8 (61)6 (75)7 (70)5 (63) 
 African American4 (31)1 (13)2 (20)3 (38) 
 Hispanic1 (8)1 (13)0 (0)0 (0) 
 Native American0 (0)0 (0)1 (10)0 (0) 
Total household income <$30,000/year5 (38)3 (38)4 (40)2 (22)0.3
Private health insurance6 (46)4 (50)4 (40)7 (78)0.6
Medicare or Medicaid4 (31)1 (13)3 (30)0 (0)0.6

Themes.

Discussions about participation in clinical studies centered on the thematic areas described below.

Current health status.

Both participants and nonparticipants thought that their current health, in particular whether their SLE was active or stable, was central to decisions concerning study participation. Eight (62%) participants in the trial believed that having stable SLE or good overall health influenced their decision to participate in PASS. For example, one participant stated, “My lupus had been stable for several years…If I were getting the letter today, I might not make the same decision because things have changed.” Another commented, “I thought, I can do this, because I feel kind of stable, balanced or whatever.” Of those who participated in PASS, 2 individuals who were experiencing stable or good health at the time of the trial indicated that their health had not been a significant factor in their decision. One stated, “I take 10 meds, 10–11 meds every morning. They say, ‘You want to be in a study?’ Sure, what's one more.”

Similar to the patients in the participant group, 7 (88%) of the nonparticipants indicated that health had been central to their decision. There were differences, however, in the role that health had played in this decision. Stable disease and current medical regimen had, in fact, influenced several (n = 4) decisions not to participate: “I thought it would be kind of complicating my own system, taking new medications…I don't want to get my system off kilter, so that's the only reason why.” Three nonparticipants preferred not to participate in medical research when they were feeling good because this might make them think about being ill: “It's like I'm feeling good right now. Don't bother me with all that now. I am living my life.”

In telephone interviews with nonparticipants in PASS, it was clear that health had been an influential factor as well. These individuals chose not to participate because they were concerned about their health status (n = 4): “Maybe if I was on the decline or my arthritis was worse and I was all out of options, but fortunately with the medication that I'm taking I'm pretty stable and my quality of life is very good.” Others were worried about how medication would impact their system (n = 4): “I just am screwing up my system already enough with the medications, without…extra additional stuff.”

Patients in the diabetes mellitus group also thought their health would influence their decision to participate in a prevention trial. Five of the patients with diabetes cited unstable or poor health as a reason to participate in a prevention clinical trial. For these patients participation in the prevention trial would be embraced because other medical options or treatments had been ineffective: “I think if it were, like, any type of insulin that would allow me to take less shots a day or I'm on oral medication, then I would be inclined to do it.” Two patients cited poor health as a reason not to participate in such a trial: “You don't know which of the many illnesses you have that this could affect.” Two patients stated that good health was a reason not to participate in a prevention clinical trial: “I'm happy the way I am right now. I probably wouldn't participate in anything. …”

Study design and purpose.

Both PASS participants and nonparticipants expressed concerns about randomization. Despite their participation in PASS, 6 (46%) of the 13 PASS participants indicated that they were worried about randomization or about unknown medication side effects or drug interactions: “Not knowing exactly what I was taking probably for like the first week I took it, I think, or 2 weeks, I was nervous about it.” Among refusers of the trial, there were comments about loss of control over which medications one was taking, and worries about potential drug interactions, medication side effects, and triggering of lupus flares. Some of this risk aversiveness to trial involvement extended beyond randomization: “I can't play Russian roulette with my health. I have too many people who depend on me.”

Participants in all of the focus groups indicated that they would be more likely to participate if given more information about medications and side effects, and more control over which treatment they would receive. One individual in the diabetes group expressed negative feelings about being in a study that would involve invasive procedures: “I'd want to know how invasive the tests are going to be. I wouldn't mind giving blood because they are taking it anyway for the different tests that I have to have.”

There was little discussion about the preventive aspect of the trial and whether the long-term risk of cardiovascular disease warranted involvement in a randomized trial at this point in patients' lives. Four of 10 refusers who were telephone interviewed stated that they did not perceive themselves to be at an elevated risk of heart attack or stroke, compared with just 1 such comment in the participant and nonparticipant groups and none among the patients with diabetes.

Involvement of personal physician.

The majority (n = 14) of the 21 patients in the SLE focus groups indicated that their personal physician's opinion influenced their decision to participate in PASS. One PASS nonparticipant commented, “You've been with your physician for X amount of years and you definitely don't want to change medicines or change whatever you're doing as far as the disease is concerned without talking to this physician.” Another nonparticipant stated, “I think most people with lupus would follow the rheumatologist to the end of the earth, you know. You really put a lot of stock in them and maybe that had to do with my getting enrolled in the study.”

Four participants also believed that the development of a relationship with a nurse or physician who was knowledgeable about the study and constant supervision were essential to involvement in such a randomized trial: “I also think I would be much more likely to do it if somebody who was really knowledgeable about the study that I could actually develop a relationship with and call up and say that clinically, this is what I'm seeing.” Interestingly, this was not the case for individuals who were interviewed. All interviewed patients (n = 10) indicated that their physician's opinion would not influence their decision, although 5 interviewees stated that they discussed PASS with their physician. It is unclear whether differences between interviewed patients and focus group patients were due to method variance, given that there was much discussion about the importance of personal physicians in the focus groups. All of the patients in the diabetes focus group (n = 9) indicated that they would consult their physician before participating in a study.

Personal benefit.

The potential for personal benefit in improving one's health, learning more about the disease and its consequences, or preventing long-term sequelae was mentioned by 7 participants and 2 nonparticipants as an influential factor. Two participants had a family history of heart disease and were therefore motivated to prevent heart disease. Thus, 29% of lupus focus group patients identified a potential personal benefit as a reason to participate. The following quotes illustrate this sentiment: “I want to find out whether I'll be in that situation, whether my heart or my arteries will be severely affected” and “I was hoping that maybe, you know, if the medication they were giving us was gonna help lower the possibility of heart attack or stroke, then that'd be great.”

In examining the responses of nonparticipants, personal gain was not a major issue. For them, the concern was that they could potentially “lose something as a result of participating” (e.g., time, current health status, convenience). Seven (78%) of the 9 patients with diabetes mentioned personal benefit as a reason to participate in a clinical trial. As one diabetes patient stated, “I just participated in a research study. And the reason I did it was for a pain relief patch.”

Altruism.

The desire to help others by being a part of medical research was frequently cited by PASS participants as an important motivation. Eleven of the 13 PASS participants responded to this question. Six (64%) of the 11 participants explicitly stated that altruism had been important in influencing their participation. As one PASS participant said, “I just feel as though it's my responsibility because I have SLE to help others as well.”

For patients who declined to participate in PASS, helping others was not mentioned as a factor that was considered or that influenced their decision. Interestingly, in discussing factors that would influence their decision to participate in a trial, only 2 of the patients in the diabetes group mentioned helping others as a reason to participate. It is also important to note that across all focus groups, with the exception of 2 patients, those who indicated altruistic reasons also indicated other factors that influenced their decision to participate in research. Altruism was an important factor for some, but was not the only factor.

Time commitment.

The time required for the study and the potential inconvenience of participation were much discussed by study refusers, who generally thought that time was a major consideration in decision making. Six of 10 agreed that involvement in such a trial created too big a burden in terms of time and inconvenience. The following statement captures this issue: “I work part time … phone studies would have been better for me because it would be any day I would choose.”

Four of the participants believed that although the time commitment had not been burdensome, they would be unlikely to participate in a future trial that was overly time consuming, did not fit into their lifestyle, or continued for a long duration. For example, one focus group participant commented, “I wouldn't want the study to become consuming, like all-consuming where I have to be home at 10 o'clock because I have to take that pill.”

Study marketing and incentives.

For the most part, study participants (9 of 12) and refusers (8 of 10 interviewees) recalled that incentives for being in the study, including monetary reimbursement, knowledge of cholesterol levels and cardiac risk factors, free home blood pressure machines, and access and consultation with a nurse practitioner/health educator, did not factor into their decision about participating in the trial: “I don't think any of those things were really a factor in deciding to do the study, but I agree that some of it was a benefit afterwards. I like having that blood pressure machine.”

DISCUSSION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. PATIENTS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. Acknowledgements
  8. AUTHOR CONTRIBUTIONS
  9. REFERENCES

Controlled trials to inform the optimal prevention of accelerated atherosclerosis in patients with SLE are of paramount importance. Our feasibility study for a large randomized controlled trial of strategies to prevent accelerated atherosclerosis in patients with SLE was stopped early due to inability to enroll sufficient numbers of participants (6). In the current qualitative study, we sought to identify patient perceptions and attitudes concerning randomized trials aimed at the prevention of atherosclerosis. We elicited and thematically categorized factors that influenced decisions about involvement and found similar factors that were weighed differently by participants and nonparticipants. PASS participants considered a number of issues in deciding about participation: good health, encouragement from one's physician, and the desire to contribute stimulated trial involvement. Among those who declined participation, current health status, concern that the medications might exacerbate their disease or cause side effects, and the time required and inconvenience were reasons for refusing. This group did not desire the involvement of their own physicians, and they were not interested in potential personal benefits or altruism. Current health status, good or bad, weighed positively for patients who elected to participate and negatively for those who refused. The diabetes group shared perspectives of both PASS participants and nonparticipants. The issues expressed by this comparison group support the finding that health status, study design, physician involvement, personal benefits, and time issues are important factors in medical research decision making.

Randomization and new unknown medications were especially troublesome to patients with SLE. The unpredictable course of SLE is unsettling and is a source of considerable anxiety (8, 9), and may explain the reticence to “rock the boat.” The idea of adding new and unknown medications to an already complicated medical regimen taken for a disease that may flare precipitously was an issue for the majority of the patients with SLE. The complicated study design of PASS, with the potential to be randomized to 1 or more of several medications, may have contributed to this anxiety. PASS participants noted that involvement of their physicians, trust and communication with the study staff, and a study regimen that was not demanding positively influenced them to participate.

Trials in SLE are difficult to mount and several trials in SLE have had difficulties recruiting participants. For instance, the SMILE (Superiority of Methotrexate in Lupus Erythematosus) trial in Canada had similar challenges, and the researchers outlined their strategies to increase enrollment (10). Little research has been conducted on potential participant attitudes toward prevention trials in SLE, but recruitment has been studied in other diseases. Work in other diseases has found features associated with high recruitment rates: clinical trials should address questions deemed important to patients and clinicians, protocols should be simplified as much as possible, and demands on clinicians and patients should be kept to a minimum (11–15). In human immunodeficiency virus trials, early enrollment within the first few months correlated strongly with longer-term patient enrollment and the trial design and attractiveness influenced accrual rates (16). Our findings underscore the importance of these factors. Indeed, patients may be right that participation has few benefits, and the trial may be poorly designed or conducted so as to hinder the finding of valid results (17).

Limitations of our study require commentary. Focus group remarks may have been influenced by unmeasured factors, such as group dynamics, patient age and sex, and underlying personality. The requirement that the personal physician approve recontacting patients for this qualitative study may have introduced a subtle selection bias, and we were unable to recruit many PASS nonparticipants to the focus groups, potentially allowing for a response bias. For this reason we interviewed true refusers, who had refused both the randomized trial and the focus groups, for their comments, albeit not in an open focus group setting. We also conducted a focus group of control patients with diabetes in an attempt to elicit SLE-specific issues that may impede participation in preventive clinical trials. Patients in these groups did not specifically address the preventive nature of this trial; it is not clear whether their attitudes and preferences regarding a trial designed to prevent a distant and, in many minds perhaps, unlikely event would hold equally for a trial of a treatment for active SLE, where individuals with the disease might assess the risks of an unknown medication differently.

Our study also identified differences in attitudes toward risk that have been studied previously (18, 19) and factors in personal decisions about research involvement in SLE. Trial participants, motivated by altruism and desire to learn and to educate others, accepted the potential risks of an unknown medication and the inconvenience of study visits and phone calls. The nonparticipants, however, were generally risk averse and many stated that they would not have wanted to be involved under almost any circumstance. After making hundreds of telephone calls to recruit patients with SLE to participate in this trial, however, we have learned that the first negative indication, including nonresponse to voicemail messages, should be accepted as a refusal. Trying to convince those at the risk-averse refuser end of the spectrum to join a prevention trial in SLE may be futile, and leads to no shows and dropouts. It may be wiser to target limited resources at patients already open to the idea of involvement. Future prevention trials in SLE should utilize strategies such as education, about both the clinical problem and the importance of trials in guiding therapy, and the involvement of patients' personal physicians in the recruitment process. In addition, we should design trials to be both efficient and effective, limiting demands on this chronically ill and overwhelmed group.

Acknowledgements

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. PATIENTS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. Acknowledgements
  8. AUTHOR CONTRIBUTIONS
  9. REFERENCES

The authors would like to thank Diana Post, MD and Merri Pendergrass, MD for their help in identifying patients for diabetes focus groups.

AUTHOR CONTRIBUTIONS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. PATIENTS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. Acknowledgements
  8. AUTHOR CONTRIBUTIONS
  9. REFERENCES

Dr. Costenbader had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study design. Drs. Costenbader, Brome, Karlson, and Liang, and Ms Blanch.

Acquisition of data. Drs. Brome and Liang, Ms Blanch, and Ms Gall.

Analysis and interpretation of data. Drs. Costenbader, Brome, Karlson, and Liang, Ms Blanch, and Ms Gall.

Manuscript preparation. Drs. Costenbader, Brome, Karlson, and Liang, and Ms Gall.

Statistical analysis. Drs. Costenbader, Brome, and Liang.

REFERENCES

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. PATIENTS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. Acknowledgements
  8. AUTHOR CONTRIBUTIONS
  9. REFERENCES