Assessing the effectiveness of HIV prevention peer education workshops for gay men in community settings
Correspondence to: Benjamin Bavinton, The Kirby Institute, The University of New South Wales, Sydney, NSW 2052; e-mail: email@example.com
Objective : To use existing evaluation data of community-based HIV prevention peer education workshops (PEWs) for gay men to explore the challenges in evaluating such programs in community settings.
Methods : Data came from 33 PEWs conducted with gay and bisexual men. A basic pre/post-test design was used to measure sexual health capacity. The Sexual Health Capacity Scale (SHCS) was anonymously completed before participation and twice afterward, with the men measuring perceptions of themselves before participation and perceptions of themselves after participation. The anonymous nature of the SHCS created problems for matching data so, for the most part, independent samples tests were used for analysis.
Results : Overall, 399 gay and bisexual men participated in PEWs. Participants perceived themselves as having more sexual health capacity after participation than before (p<0.001). Those who had previously been HIV tested before the PEW had higher perceived capacity (p<0.001).
Conclusions and Implications : Participation in the PEWs appeared to increase the perceived sexual health capacity of gay and bisexual male participants. Several limitations in the data arose from issues in the original data collection. A mixture of anonymous and identifiable data-sources meant that data could not always be matched to individuals. Stronger partnerships between HIV researchers and professionals within community organisations could significantly improve evaluation of the effectiveness of HIV peer education.
Globally, peer education is one of the primary community-based behavioural change interventions aiming to prevent HIV transmission,1 including transmission among gay men and other men who have sex with men (MSM).2,3 A meta-analytic review of 33 studies of HIV interventions with gay men and other MSM found that behavioural change interventions with gay and other MSM were effective in decreasing unprotected anal intercourse (UAI) and increasing condom use, and that these effects could last up to 12 months after the interventions.3 Interventions associated with the greatest effects included four major components: a theoretical model of behavioural change; interpersonal skills training; four or more methods of educational delivery (e.g. counselling, group discussions, lectures, live demonstrations and role plays); and more than one educational session and four or more hours of ‘exposure’.3 Even so, peer education programs are often not well evaluated,4 and are hampered by limitations, including small sample sizes, lack of statistical power, lack of controls and lack of experimental or quasi-experimental design.5,6 The extant peer education literature is limited by a lack of detailed description of the programs’ implementation factors such as selection, training and supervision of peer educators (PEs),5 making evaluation and replication difficult.6 Despite this, there is some evidence that group-based peer education can be effective in increasing rates of protected sex,7–10 HIV knowledge5 and HIV testing.11
In Sydney, Australia, peer education most commonly takes the form of group-based, multi-session, volunteer-facilitated workshops implemented by a community-based HIV organisation in the local gay community, ACON (formerly known as the AIDS Council of NSW).12–14 Peer education workshops (PEWs) have been continuously conducted with gay men in Sydney since 1988,14 long before there was evidence of the effectiveness of this approach and often without reference to academic and traditional public health discourse.15 The PEW model that evolved in Australia implicitly contained the four components of the most effective behavioural HIV interventions among gay and other MSM worldwide.3 Despite this, and despite the highly effective community-based response to controlling HIV in Sydney,16 there is very little published in the literature on PEWs in Australia. Most PEW programs in Australia are not well documented or evaluated.
The majority of HIV infections in Australia are through homosexual contact,17 with UAI as the highest risk factor.18 HIV infection rates have been stable among gay men in Sydney, Australia, for more than 10 years.19 Rates of HIV testing are high among Sydney gay men, with approximately two-thirds reporting being tested in the previous year20 and about 8% reportedly being HIV-positive.21 The proportion of Sydney gay men reporting any UAI with casual partners in the previous six months has increased from 29.7% in 2007 to 34.1% in 2010, but consistent condom use with casual partners has nonetheless remained stable from 2004 to 2010 at approximately 50%.22 We focused on the PEWs conducted at ACON to explore: 1) the ‘Australian model’ of peer education in gay communities in terms of implementation and evaluation; and 2) the challenges in scientifically evaluating such programs in community settings. The purpose of this paper is not to evaluate the PEW program at ACON per se, but to use this program and its evaluation as a case study to explore some complex issues involved in evaluating community-based programs.
The PEW program
The PEW program had two stages: the recruitment and training of gay and bisexual male PEs; and the delivery of PEWs to gay and bisexual men. Gay community members were informed of the PE training program via print, online and email advertisements. PEs self-nominated and were selected through an interview process by project staff using predetermined criteria, including identification as gay, bisexual or same-sex attracted, demonstrated understanding of the program objectives, enthusiastic attitude, empathy, and capacity to consider responses to challenging group work scenarios. Successful applicants attended a 30-hour training course including information on HIV, sexual health, alcohol and other drugs, and facilitation skills and practice. Once trained, the PEs facilitated workshops with gay male community members, and were supervised by project staff through communications before and after each workshop session. Each PEW was facilitated by two PEs. Thirty-two PEs were involved in the PEW program during the 2.5 year period of this analysis.
The PEWs consisted of four to six sessions of about 2.5 hours of workshop content, one evening per week. The core modules covering HIV and sexual health information were essentially the same in each PEW, whereas the remaining content varied. Workshops focused on gay life in Sydney, relationships, sexual techniques, using sex-on-premises venues, and anal health and pleasure. Some PEWs were open only to men 29 years and under (youth workshops) while others were open to all gay men aged 18 years or above (all-ages workshops). In each workshop, the topics on HIV prevention and sexual health were presented explicitly in their own modules and interwoven into modules on other topics. Each PEW had a detailed facilitator manual and involved various educational methods including: group discussions, hypothetical scenarios, role-plays, quizzes, didactic presentations, video scenarios, individual reflection, and practical skills-building activities. For this analysis, data came from 33 of the 38 PEWs implemented between January 2008 and August 2010.
Men were informed of PEWs through print, online and email advertising. Importantly, the PEWs were never advertised as ‘HIV prevention workshops’. Instead, the workshops were centred on topics of interest to gay and bisexual men, with HIV and sexual health information presented as simply one aspect of the broader topic. This approach was taken for three primary reasons:
- • The broader topics were used as ‘hooks’ to garner the interest of members of a community that typically had high levels of HIV knowledge, HIV testing and understandings of HIV risk.
- • The lack of a sense of crisis around HIV in the gay community of Sydney meant that gay and bisexual men were unlikely to be motivated to attend HIV prevention-specific workshops.23
- • The topics were explored in their own right as part of a broader gay men's health agenda.24
Men registered their interest with project staff, and were interviewed prior to participation to ensure the workshop was appropriate to their needs. Each PEW was conducted with a closed group of up to 16 men (i.e. no new men could join the group after the first session).
Evaluation of the intervention: measures and procedures
The measures and procedures to evaluate the PEWs were devised by project staff with only limited research experience. First, PEW participants provided demographic information at their intake interview: age, sexual identity, gender identity, cultural/ethnic background and postcode of residence. Second, participant attendance rates were recorded by the facilitators and could be matched to these demographic data. Third, participants reported as part of an anonymous questionnaire whether they had ever had an HIV test. They were asked this before the workshop and at the final session. Finally, the main outcome measure of sexual health capacity was a battery of six items using a Likert scale with response categories ranging from 1 = strongly disagree to 5 = strongly agree, and a maximum score of 30. Although the various PEWs had different health objectives, all had the explicit aim of increasing sexual health knowledge and capacity. These six items were common to all PEWs and were used to construct the Sexual Health Capacity Scale (SHCS). Five items measured self-perceived HIV and sexual health knowledge and one item related to self-efficacy in negotiating condom use (Table 1). Higher scores indicated greater perceived knowledge and confidence in maintaining sexual health. The participants completed this scale anonymously before starting the first session. This data-point was termed Before-Before (BB), indicating that the score represented the participants’ perceptions of themselves before the workshop, collected before the workshop. At the end of the final session, remaining participants completed an anonymous questionnaire that included the SHCS again. At this time, they were asked to rate each item twice: (1) perceptions of themselves before the workshop, collected after the workshop, termed Before-After (BA); and (2) perceptions of themselves after the workshop, collected after the workshop, termed After-After (AA). Cronbach's Alpha reliability statistic for the BB scores of the SHCS was 0.786, indicating a high level of reliability. The reliability of the scale decreased if any of the items were deleted.
Table 1. Independent samples analysis of Sexual Health Capacity Scale before and after participation in a peer education workshop.
|I have a good understanding of how HIV is transmitted.||4.3 (0.67)||4.7 (0.51)||<0.001|
|I have a good understanding of what sexual activities are considered ‘safe sex’ and ‘unsafe sex’.||4.2 (0.71)||4.7 (0.52)||<0.001|
|I would be able to recognise the symptoms of a sexually transmissible infection (STI).||3.4 (1.00)||4.2 (0.72)||<0.001|
|I know where to go to get a full sexual health check-up.||4.3 (1.03)||4.7 (0.59)||<0.001|
|I know how to put a condom on properly.||4.3 (0.80)||4.7 (0.53)||<0.001|
|I feel confident that I can negotiate the use of condoms with sexual partners.||4.2 (0.81)||4.5 (0.70)||<0.001|
| Complete scale || 24.7 (4.3) || 27.4 (2.4) || <0.001 |
The SHCS responses were anonymous so that participants would feel comfortable answering the questionnaire and participating in the PEW, and would be more likely to provide honest answers. It is important to note that due to the anonymous nature of the SHCS, the BB SHCS scores could not be matched to any other data. The BA and AA SHCS scores could be matched to each other, but not to the demographic or attendance data. The before-workshop HIV testing data could be linked to the BB SCHS scores, and the after-workshop HIV testing data could be linked to the BA and AA SHCS scores. This mixture of anonymous and identifiable data had ramifications for the types of analyses that could be conducted.
The key question for the evaluation was whether the workshops were effective at increasing the perceived capacity of gay men to engage in sexually healthy practice, using a pretest-posttest design. However, as the SHCS data were anonymous, the data could not be matched to specific individuals, meaning that paired-samples statistical tests could not be used. Thus, the comparison between the SHCS scores before the workshop and those after the workshop could only be conducted using an independent samples test, treating the BB and AA groups as independent samples drawn from the same underlying population. Additionally, this key question was explored by comparing the BA and AA SHCS scores for a sub-sample of 190 participants. Since the BA and AA scores could be matched, it was possible to conduct a paired-samples test. However, this meant that the ‘before’ scores were the participants’ recollections of themselves before the workshop, rather than actual perceptions of themselves collected before the workshop. The BA scores were collected because it was hypothesised that participation in PEWs might change how the participants viewed their sexual health capacity before participation. For example, learning new information might cause them to realise that they did not have as much capacity as they previously believed.25,26
To explore the meaning of the above two analyses, other tests were conducted. The BB SHCS scores were compared to rates of previous HIV testing in the sample. The sample was divided into two groups: those who indicated they had previously been tested for HIV (n=268); and those who did not indicate they had previously been tested for HIV (n=79). Additionally, it is important to acknowledge that demographic factors such as age may have affected perceived sexual health capacity. Given that it was not possible to match the demographic data to the individual participants’ SHCS scores, it was not possible to control for age. However, the SHCS scores were analysed according to the type of workshop: youth workshops for young gay men aged 29 years and under, and all-ages workshops for gay men 18 years and older. The mean age was 24.8 in the youth workshops and 36.8 in the all-ages workshops. Participation in the two types of workshops was taken as a proxy for age.
Overall, 399 gay and bisexual men participated in PEWs in the study period. They ranged in age from 17 to 75, with a mean of 32.1 (age data were missing for 20 participants). Just over half (53.7%) the participants were of Anglo-Australian background. There were four Aboriginal participants. Most (86.4%) participants identified as gay/homosexual.
The mean BB SHCS score was 24.7 (±3.5, n=347) and the mean AA SHCS score was 27.4 (±2.4, n=266), indicating that participants perceived themselves as having more capacity after the workshop than before (p<0.001; Table 1). The paired-samples analysis on the sub-sample of 190 participants determined that the mean difference between the BA and AA SHCS scores was significantly different (p<0.001), indicating that this sub-sample of participants perceived themselves to have more capacity after the workshop than before, when asked at the end of the PEW (Table 2).
Table 2. Paired-samples analysis of 190 participants on the Sexual Health Capacity Scale: perceptions before and after participation in a peer education workshop, collected after participation.
|I have a good understanding of how HIV is transmitted.||3.9 (0.98)||4.7 (0.52)||<0.001|
|I have a good understanding of what sexual activities are considered “safe sex” and “unsafe sex”.||3.6 (0.92)||4.6 (0.52)||<0.001|
|I would be able to recognise the symptoms of a sexually transmissible infection (STI).||3.5 (0.96)||4.1 (0.71)||<0.001|
|I know where to go to get a full sexual health check-up.||4.1 (1.19)||4.7 (0.58)||<0.001|
|I know how to put a condom on properly.||4.3 (0.78)||4.7 (0.53)||<0.001|
|I feel confident that I can negotiate the use of condoms with sexual partners.||4.0 (0.95)||4.5 (0.70)||<0.001|
| Complete scale || 23.6 (4.2) || 27.3 (2.4) || <0.001 |
The BB SHCS scores were compared between those who had previously been tested for HIV and those that had not. The two groups had significantly different SHCS mean scores: 25.4 for those tested and 22.1 for those not tested (p<0.001), indicating that those who had previously been tested for HIV had higher perceived capacity. The BB and AA SHCS scores of those participating in the youth workshops were compared to those participating in the all-ages workshops (Table 3). Men in the all-ages workshops had significantly higher perceived capacity before the workshop (BB scores; p=0.01) but not after the workshop (AA scores; p=0.69).
Table 3. Independent samples analysis of Sexual Health Capacity Scale between participants in the youth workshops and all-ages workshops.
|Before-Before||24.1 (3.6)||137||25.1 (3.4)||210||0.01|
|After-After||27.4 (2.5)||113||27.3 (2.4)||153||0.69|
The aim of this analysis was to explore the issues and challenges arising from community-based evaluation of HIV prevention PEWs. The findings indicated that men attending PEWs already reported a high degree of perceived capacity beforehand. Nonetheless, the overall results indicated that participation in PEWs increased the self-perceived sexual health capacity of gay and bisexual men of all ages when: (1) comparing the SHCS scores before and after the workshop as independent samples; and (2) comparing the SHCS scores of the sub-sample of 190 men in a paired-samples analysis of their perceptions before and after the workshop, collected after the workshop. The greatest improvement was in the men's perceived capacity to recognise the symptoms of STIs; however, all items individually showed statistically significant improvement.
Examining these data indicated that for many of the items, and for the overall scale, there was a pattern where the participants reported very high levels of sexual health capacity before and after the workshop, but the BA score showed a marked decrease. This pattern has been termed ‘negative response shift’ in other health literature, such as chronic disease self-management, whereby participants recalibrate, re-evaluate or reconceptualise their response to a particular construct.25,26 In this case, response shift was evident in that most participants came to the PEW feeling confident in their sexual health capacity, but participation in the workshop appears to have caused some men to re-evaluate how much they thought they knew or how skilled they thought they were before, resulting in lower agreement with the scale items. After the workshop, they again indicated strong agreement with the scale items. The workshop may have demonstrated to the men that their self-perceptions were perhaps overestimated before the workshop. After participating in several sessions of educational content, they may have been aware that they were less competent than they thought they were. Alternative explanations for this pattern include social desirability bias and/or the fact that the mean BB SHCS scores may have been inflated by the participants who started the workshop but did not finish it. Since we cannot match these data, this cannot be determined. Some men who perceived they had high sexual health capacity may have not completed the workshops. Thus, when it came to the final session and participants were asked to reflect on themselves before the workshop, those remaining might have been the men who actually did have less competence at the beginning.
The youth workshop participants, who had a younger average age, had significantly lower sexual health capacity than the all-ages workshop participants, who were older on average. However, this effect disappeared after the workshop, with the groups from both projects showing the same increased level of sexual health capacity. It is not possible to conclude that age was the underlying factor accounting for the before-workshop difference as there may be other ways that the two groups differed. Also, the age profile may have changed from beginning to end due to workshop drop-outs. It nonetheless suggests a possible relationship to be explored in future studies. Also, those who had been tested had higher sexual health capacity. This could be tentatively interpreted to mean that those who perceived themselves to have greater capacity did look after themselves more (that is, exercise that capacity). Unfortunately, we were not able to test for any more modifying variables because of the limitations of the data collected.
These limitations in the data demonstrate one of the primary difficulties in conducting HIV prevention with gay men in Sydney – a city that has benefited from sustained community-based HIV prevention efforts from the beginning of the epidemic.15 According to these data it is likely that HIV prevention PEWs can significantly increase perceived levels of knowledge around sexual health and skills in condom use, even against a background of such high perceived sexual health capacity. One of the potential strengths of the ‘Australian model’ of peer education may be in the broader gay men's health framework used to promote the PEWs. The PEWs targeted a wider range of issues than just HIV prevention, and so did not necessarily depend upon a “motivated ‘taught’”.23 As noted in previous research,14 young gay men did not participate in the programs to gain information about HIV, but rather to connect with other gay men. The Project “incorporates safe sex, and knowledge and skills for dealing with other (sexual) health issues into the experience of being gay and developing a gay life” (p.3).14 While this approach still relies upon an overarching sense of gay community and identity, it allows for at least one way to motivate a community of gay men who already perceive themselves to have a high degree of sexual health capacity to be exposed to new or reinforcing HIV prevention information. Once exposed, the data from this analysis suggest that some men may revise down their original estimates of their earlier competence.
The major limitations in these analyses were due to issues in the original data collection. Data were collected anonymously, precluding the capacity to match each participant's demographic and attendance information with his before and after SHCS scores. This meant the more appropriate repeated-measures statistical tests could not be used, and the SHCS could not be adjusted for confounding variables such as age or education. Furthermore, as the BA and AA scores were collected only from those participants who stayed in the PEWs to the end, it was not possible to determine whether there were any fundamental differences between those who completed and those who did not. The BA scores were particularly open to bias, given their retrospective nature. Other limitations included: missing data; the use of self-report measures of perceptions only (that is, no measures of actual knowledge or self-reported behaviour); and the lack of a longer-term follow up of participants.
Individuals working within community-based HIV organisations are rarely experts in research methodology and typically work towards pragmatic goals around managing programs. Although evaluation is critical to effective HIV prevention, data collection and scientifically robust data analyses are usually not the highest priorities of workers aiming to achieve immediate programmatic goals on small budgets. Despite the strong extant partnership between community organisations and researchers in the HIV sector in Sydney,27 this project indicates that such partnerships could be utilised more. In Australia, representatives from community organisations are usually consulted in the design and management of large academic HIV research projects, but the opposite situation of professional researchers assisting community organisations to gather and analyse data to scientifically acceptable standards appears to be rare. Despite its imperfections, this project represents a step toward such partnerships, whereby professional researchers engage to a greater degree with the data collection occurring within community HIV prevention settings. While researchers were not involved in the collection of the data used in this analysis, the collaboration in analysis has initiated greater involvement of researchers at the data collection level for future rounds of evaluation.
In this project, a significant improvement would have been the ability to match data from individuals before and after PEWs, and to their demographic information. Such matching could have been achieved by using a unique alphanumeric code for each participant, perhaps generated from appropriate stable information. The issue of participants feeling threatened by a lack of anonymity is real, especially for younger participants. However, this can be mitigated by clear explanations of how the data will be collected, how confidentiality will be protected, and that participation in PEWs will not be dependent on participation in the research.
A further improvement would be the collection of self-reported behavioural data and actual HIV knowledge along with attitudes and perceptions. Knowledge and behavioural indicators could be drawn from comparable tools previously used in the Australian context. The SHCS measure used in this analysis nonetheless generated useful data regarding participants’ perceptions. However, the scale was unable to indicate if knowledge or behaviour changed as a result of participation in PEWs due to the limitations in data collection and recording. Despite these limitations, the scale itself appears to be reliable and at least provides a basis for developing similar tools for future use.
Conducting a follow-up evaluation with PEW participants three or six months after participation would provide evidence as to the durability of any effects. Finally, training workers and volunteers in HIV community organisations in research methods and skills may help to increase the quality of and value placed upon rigorous data collection and management.
Overall, there were a number of significant limitations in the methods and processes used to evaluate this community-based PEW program. Nonetheless, this analysis has revealed some important insights for future work in this area. First, although the concept of response shift has been discussed in various health literature, it has not been adequately explored with regard to HIV prevention in gay men. Despite the limitations of the SHCS, the results indicated that gay men may overestimate their sexual health knowledge and capacity, and this has real implications for the effectiveness of various HIV education modalities. When the men participated in a PEW, they appeared able to revise down their estimate of their pre-workshop perceived capacity, suggesting that group-based educational activities may play an important role in challenging men's overestimations of capacity. Second, the analysis clearly demonstrated that self-perceptions are not adequate for the evaluation of HIV prevention workshops, and that more direct measurement of knowledge and behaviour is recommended. Finally, community workers need support from, and partnerships with, professional or academic researchers to develop stronger evaluation tools and methods. Good evaluation is difficult to achieve in community settings where workers are faced with conflicting priorities. Partnerships with researchers allow for expertise to be shared and novel approaches to evaluation to be developed.