A health facility based case–control study of effectiveness of insecticide treated nets: potential for selection bias due to pre-treatment with chloroquine
Case control studies offer an attractive way to assess the effectiveness of insecticide treated nets (ITN) under programme conditions but have the drawback of being susceptible to bias in the choice of controls. We evaluated the potential for pre-treatment with chloroquine to result in misclassification of cases and controls and affect estimates of ITN effectiveness in case control studies in urban and rural clinics in Eastern Afghanistan. During the one-month study, use of ITN showed no effect against malaria in the urban clinic (adjusted odds ratio OR 1.08; 95% CI 0.73–1.6) and the protective effect seen in the rural clinic was not significant (OR 0.62; 95% CI 0.2–2.4). Levels of pre-treatment were high in both clinics: 24% in urban and 19% in rural clinic attenders. In the urban clinic attenders the level of pre-treatment between bed net users and non-users was not significantly different (OR 1.07, 95% CI 0.70–1.64); therefore the misclassification of cases as controls did not introduce any selection bias. Amongst rural clinic attenders, bed net users were less likely to pre-treat with chloroquine than users (OR 0.33, 95% CI 0.14–0.77); this introduced a selection bias that resulted in an underestimation of the effectiveness of bed nets. Case control studies using health facility data are liable to selection bias especially in areas of high pre-treatment rates with chloroquine. Generalisation of results over a wide geographic region, or between urban and rural settings, may not be appropriate.
Several studies have shown that insecticide treated nets (ITN) reduce malaria mortality and morbidity, and the promotion of ITN within national control programmes is gaining momentum (Lengeler et al. 1996; Lengeler 1998). Most of these studies have been carried out under ideal trial conditions and thus measure efficacy. There is a potential difference between efficacy in randomised controlled trials (RCTs) and effectiveness under programme conditions where health promotion and health service delivery is limited (Lengeler & Snow 1996). The effectiveness of ITN in reducing malaria morbidity has been demonstrated in programme settings, albeit to a lesser degree than in earlier efficacy trials (D'Alessandro et al. 1995; Rowland et al. 1997; Abdulla et al. 2001). The recent development of pyrethroid resistance among mosquitoes in Africa (Chandre et al. 1999), and the potential of ITN programmes for shifting malaria mortality from younger to older children in high endemic areas (Trape & Rogier 1996; Snow et al. 1997) highlight the need for the regular monitoring of the effectiveness of ITN programmes.
Analysis of health service-based passive data in a case–control study design has been used for monitoring the impact of ITNs in both Africa and Asia (D'Alessandro et al. 1997; Rowland et al. 1997). This appears to be an attractive option for the continuous monitoring of the effectiveness of ITN programmes. Rowland et al. (1997, 2002), working with Afghan populations, used a simple design based solely on microscopy, where patients presenting with fever and found to be positive for malaria parasites were classified as cases, and those found to be negative, as controls. The odds of bed net use was then compared between these two groups. The advantage of this approach is that any competent microscopist can gather the necessary data. The limitation of this study design is the potential for misclassification of malaria cases as controls if use of anti-malarial drugs is common prior to attending health facilities.
In this paper we replicate this health facility based case–control study design, and use sensitivity analysis to assess the effect of selection bias introduced by the pre-treatment of fever episodes with chloroquine prior to attendance at health facilities, on the measured effectiveness of bednets. Methods for monitoring the effectiveness of ITNs in areas of unstable transmission with variable levels of chloroquine usage prior to attending health facilities are then discussed.
The study was carried out in July 1999 at two health facilities in Nangahar Province, Eastern Afghanistan—the Malaria Reference Centre (MRC) in Jalalabad city and the Austrian Relief Committee clinic (ARC) in Behsud. Although less than 5 km apart, the clinics differ in their catchment populations. ARC is a rural clinic in a district with a history of ITN intervention, and a catchment population of the surrounding villages. Since 1993 HealthNet International (HNI) has been carrying out intensive promotion and sale of subsidised ITNs in Behsud District in which ARC is situated. MRC is a reference centre for malaria situated in the urban centre and has a much wider catchment area, from several districts within the Province.
Transmission of malaria in the study areas is unstable and peaks twice a year, vivax malaria in mid-summer and falciparum in autumn. Plasmodium vivax accounts for more than 85% of the infections. The main vectors are Anopheles stephensi, A. culicifacies, A. superpictus and A. pulcherrimus.
We collected information on recent medication, treatment-seeking behaviour, use of bed nets, socio-economic and educational status using a structured questionnaire on all patients presenting with a history of fever at the clinics. We also collected a blood smear and a urine sample from all patients enrolled in the study. The blood smears were stained with Giemsa and examined by the MRC and ARC laboratory technicians for malaria parasites. All slides were re-examined by HNI laboratory technicians. The urine samples were tested for the presence of chloroquine using a field modification of the Saker-Solomon test (Mount et al. 1989). Chloroquine was partitioned from buffered urine (pH 8) into a chloroform solution of tetrabromophenolphthalein ethyl ester (TBPEE) giving a resultant purple colour in the organic layer. We examined the reaction by eye to obtain qualitative results where a red to purple colour in the organic layer was classified as positive for chloroquine, and a green to yellow colour as negative.
For the analysis we compared the odds of using a bed net among cases (fever and blood smear positive for malaria) and among controls (fever and blood smear negative for malaria). We used logistic regression models to assess the independent effect of use of bed net on the risk of malaria, and the determinants of using chloroquine prior to attending the clinics taking into account the effects of other predictor variables. As the effect of chloroquine on clearing parasites is independent of net use, it is unlikely that there is any interaction between chloroquine use and net use. However, confounding may be present as net users may be more or less likely to use chloroquine. The logistic regression model used is therefore measuring confounding not effect modification.
We compared the number of possible misclassifications of cases as controls (those with urine positive for chloroquine and negative blood films) amongst bednet users and bednet non-users to assess the potential for selection bias. Sensitivity analysis was then used to estimate the effect of misclassification and any selection bias introduced by this misclassification, on the measure of effectiveness of bednets.
There were only three cases of P. falciparum malaria and they were excluded from the analysis as resistance to chloroquine of P. falciparum has been reported in Afghanistan (Rowland et al. 1997; Shah et al. 1997). In the presence of this resistance the potential of any effect of pre-treatment with chloroquine on measures of the effectiveness of bednets should be considered separately for P. vivax and P. falciparum. In this study cases of malaria refer to P. vivax only.
The distribution of demographic and socio-economic characteristics of the study populations and the slide positive rate for malaria (Table 1) differed between the populations enrolled at the two clinics; data of these two populations were analysed separately. The proportion of <5 year old children was lower and the proportion of adults was higher among MRC attenders than among ARC (15%vs. 25% and 49%vs. 39%, respectively; P < 0.005). The male:female ratio was fairly balanced among MRC attenders (1:0.8) while there was a higher proportion of females among ARC attenders (1:2.6). The literacy level was higher among MRC attenders (28%vs. 10%; P < 0.001), but the socio-economic status was lower than ARC (36%vs. 52% having a bicycle; P < 0.001). Use of a bednet was lower among MRC attenders than ARC attenders (22%vs. 45%; P < 0.001). Among MRC attenders 4% of people said they had a bed net but did not use it, whilst among ARC attenders the people who had bed nets also said that they used them. Use of chloroquine prior to attending clinic was slightly higher in MRC attenders but this difference was not statistically significant. Incidence of malaria among patients with fever attending MRC was significantly higher than those attending ARC (35%vs. 7%; P < 0.001).
Table 1. Distribution of potential predictors of malaria in bednet users and non-users in the two study populations
|Age|| || || || |
| 0–4||27 (16%)||88 (15%)||29 (32%)||23 (20%)|
| 5–9||33 (19%)||103 (17%)||10 (11%)||21 (19%)|
| 10–14||32 (19%)||105 (18%)||15 (16%)||26 (23%)|
| >14||79 (46%)||301 (50%)||38 (41%)||43 (38%)|
|Sex|| || || || |
| Male||106 (62%)||331 (55%)||27 (29%)||30 (27%)|
| Female||65 (38%)||266 (45%)||65 (71%)||83 (73%)|
|Able to read||76 (44%)||143 (24%)*||11 (12%)||9 (8%)|
|Family had a bicycle||96 (58%)||181 (30%)*||65 (71%)||39 (35%)*|
|Urine +ve for chloroquine||41 (24%)||142 (24%)||11 (12%)||29 (26%)*|
|Slide +ve for malaria parasite||62 (36%)||0 (34%)||5 (5%)||9 (8%)|
The distribution of certain characteristics that may be associated with the risk of malaria among bed nets users and non-users was examined (Table 1). There was no association between age and sex of the patient and the use of bed nets in the two study populations. Literacy rate was higher in bed net users than non-users, and for the MRC population the result was significant (P < 0.01). Ownership of a bicycle was higher in bed net users than non-users in both the MRC and ARC population. Use of chloroquine was significantly lower among bed net users than non-users in ARC population (P < 0.01), whereas in the MRC population there was no difference in urine chloroquine positivity between these two groups (24%vs. 24%).
Older individuals were less at risk from malaria in both MRC and ARC populations (Table 2). Sex, literacy and economic status had no significant effect on the risk of malaria. As expected the use of chloroquine had a protective effect on malaria parasitaemia, but the magnitude of the effect differed between MRC and ARC populations (34% protection in MRC versus 100% in ARC). Use of bed nets did not have an effect on malaria in MRC, and the protective effect seen in ARC was not significant statistically.
Table 2. Effect of potential predictor variables on clinical malaria
|Age group|| || || || || || |
| 0–4||62 (22)||58 (11)||1.0||7 (47)||45 (23)||1.0|
| 5–9||73 (26)||71 (13)||0.95 (0.56–1.60)||4 (27)||28 (15)||1.0 (0.24–4.01)|
| 10–14||55 (19)||92 (18)||0.54 (0.32–0.92)*||2 (13)||41 (21)||0.44 (0.08–2.46)|
| >14||93 (33)||304 (58)||0.29 (0.18–0.46)*||2 (13)||80 (41)||0.13 (0.01–1.21)|
|Sex|| || || || || || |
| Male||152 (54)||301 (57)||1.0||6 (40)||53 (27)||1.0|
| Female||132 (46)||224 (43)||1.20 (0.84–1.69)||9 (60)||141 (73)||0.88 (0.25–3.13)|
|Unable to read||211 (75)||363 (71)||1.0||15 (100)||174 (90)||1.0|
|Able to read||72 (25)||153 (29)||1.05 (0.70–1.57)||0 (0)||20 (10)||xxx|
|Has no bicycle||182 (65)||329 (63)||1.0||9 (60)||95 (49)||1.0|
|Has bicycle||100 (35)||191 (37)||0.92 (0.64–1.30)||6 (40)||99 (51)||1.29 (0.34–4.83)|
|Urine −ve for CQ||226 (82)||381 (74)||1.0||14 (100)||153 (79)||1.0|
|Urine +ve for CQ||48 (18)||137 (26)||0.66 (0.45–0.98)*||0 (0)||40 (21)||xxx|
|Did not use bednet||200 (76)||395 (78)||1.0||9 (64)||104 (55)||1.0|
|Used bednet||62 (24)||110 (22)||1.08 (0.73–1.60)||5 (36)||87 (45)||0.62 (0.16–2.41)|
The likelihood of taking chloroquine prior to attending clinics was highest in the age group over 14 years. Among the ARC population, bed net users were significantly less likely to take chloroquine than bed net non-users (adjusted OR 0.33; 95% CI 0.14–0.77). But among the MRC population neither was there any association between bed net use and chloroquine intake (adjusted OR 1.07, 95% CI 0.70–1.64) nor between chloroquine intake and gender, literacy or socio-economic status.
Amongst ARC attenders 40 people had positive urine tests and negative tests for malaria parasites. These are the people who may potentially be cases misclassified as controls. Of the 40, 11 were bednet users and 29 were non-users. It is not possible to be sure how many of these were true malaria cases misclassified as controls and how many were true controls. A sensitivity analysis in Table 3 shows the effect of different levels of misclassification on the association between malaria positivity and using a bed net. As the rate of misclassification increases, so does the protective effect of bed nets. If 75% of the urine positives and malaria negatives were true malaria positives then there is a 56% (P = 0.02) protective effect of using a bed net against malaria.
Table 3. Effect of the level of misclassification due to pre-treatment with chloroquine on the protective effect of bednets
Passive case data from the urban MRC showed no protective effect of using bed nets against vivax malaria while that from the rural ARC showed a protective effect of 38% although the role of chance cannot be ruled out owing to the small sample size and short study period. These findings are consistent with the results obtained during the ITN effectiveness trial of 1995–1997 at ARC Behsud (Rowland et al. 2002), but differ from the 69% protection against vivax (95% CI 81–49%) reported by a clinic-based case–control study in an Afghan refugee camp in Pakistan (Rowland et al. 1997). The introduction of selection bias is a possible reason for this disparity.
The case–control design employed has the potential for misclassification of ‘cases as controls’ and vice versa, and ‘bednet users as non-users’ and vice versa. Selection bias would be introduced if the levels of misclassification of cases and controls vary between bednet users and non-users or if the misclassification of bednet users and non-users varies between cases and controls. The use of chloroquine before attendance at the clinic would result in the misclassification of cases as controls.
Generally pre-treatment with chloroquine before attending health facilities was common in the Afghan population. Amongst ARC attenders pre-treatment with chloroquine was less common in net users than in non-users. This would have led to a selection bias that would cause an underestimation of the protective effect of bed nets. The level of pre-treatment with chloroquine in the Afghan refugees in Pakistan may have been much less because free treatment was available from the camp's clinic, and hence the study observed a better protective effect of bednets. However, although this argument may be valid for the rural ARC population, it cannot explain the lack of protective effect among the MRC population, where there was no difference in levels of pre-treatment with chloroquine between net users and non-users.
Other possible sources of misclassification include misclassification of cases and controls due to inaccurate reading of the slides and misclassification of bednet users and non-users due to the possibly low specificity of the simple question used. All slides were crosschecked and there is no reason why any inaccuracies would vary between bednet users and non-users. The possibility of misclassification from using a simple question was raised by D'Alessandro et al. (1997). However, the same question was used in this study as in the earlier case–control study (Rowland et al. 1997) where bednets were found to be highly protective.
Differences in wealth and education between net users and non-users were other potential sources of selection bias. However, unlike chloroquine pre-treatment, neither affected malaria risk and therefore both could be disregarded in the present context.
Later in the year, during the falciparum transmission season, the clinic-based case–control method showed a clear protective effect of bed nets against falciparum malaria (Rowland et al. 2002). Here, any confounding effect of chloroquine pre-treatment would be less because falciparum is resistant to chloroquine (R1 frequency of approximately 40%) in Afghan populations and many parasitaemias would stay patent after pre-treatment (Rowland et al. 1997; Shah et al. 1997).
The high levels of pre-treatment with chloroquine before attendance at clinics with malaria observed in this study, highlight the danger of using such health facility data to assess the incidence of clinical vivax malaria in the community. The incidence in this community would have been underestimated by 20–25%.
Our study also highlights some caveats in generalising results of health facility based case–control studies. Although the two clinics were just 5 km apart, the user population differed in demographic, socio-economic and treatment-taking characteristics. The difference in age and sex structure is explained by the restrictions placed on women (and their children) travelling to urban centres, even over short distances. There are fewer problems attending rural clinics. Until recently education for women was prohibited, and in the past literacy among women (especially in rural areas) has been very low in comparison to that of men (Dupree 1980; Marsden 1999).
The reasons for bed net users in the Behsud area pre-treating with chloroquine less than bed net non-users are unclear. Surprisingly higher socio-economic status was not a strong predictor of use of chloroquine. Hence although bednet use was associated with socio-economic status, the observed higher pre-treatment levels among bednet nonusers are unlikely to be due to differences in socio-economic status. It is probable that the intensive promotion and health education campaigns on malaria and use of ITNs in Behsud (Rowland et al. 2002) has made the inhabitants of the district more aware of the protective effect of bed nets than is known by the catchment population of the MRC.
The important methodological concern highlighted by this study is the difference in the association between pre-treatment with chloroquine and net use which led to a loss in the power of the study at MRC, but to an underestimation of the protective effect in ARC. The problem of misclassification due to pre-treatment cannot be overcome by excluding chloroquine-positive subjects from the study. In the rural population this would fail to take into account the differential use of chloroquine between net users and non-users and would introduce selection bias.
We assume that people are not more likely to attend health facilities for fever due to malaria than for fever due to other causes. Hence the sample population is unlikely to be dependent upon the exposure (use of bednets) and the selection of cases and controls will not be biased away from the null. The basis for this assumption is twofold: first, the overwhelming majority of fever cases (over 80%) that attend the clinic are slide negative for malaria and due to other causes; second, the odds ratio of the association between malaria and bednet use in the clinic based study was similar to that obtained by cross sectional prevalence surveys in the same communities and such surveys would be less liable to potential bias (Rowland et al. 2000). Both these observations suggest that in this community the clinic-based sample with fever is representative of the population at large. However, this may not be the case in some settings. If malaria was perceived to be more life-threatening than other causes of fever and such cases of malaria could self diagnose their condition more accurately than cases of fever due to other causes, then the cases and controls would be a self-selected subset of the source population and the level of bednet use among cases and controls would not reflect the proportions of bednet use in the community. As the effect of this selection bias depends upon the association between the risk of malaria and bednet use, if this bias occurs it would be impossible to reach any conclusion about the effectiveness of bednets.
We conclude that the health facility-based case–control studies are liable for selection bias particularly in areas where anti-malarial drugs are available from sources other than health facilities. Caution should be taken in generalising the results of such studies to a wider geographic region and different transmission seasons.
HealthNet International's malaria control and research programme is supported by the European Commission (DG1), the United Nations High Commissioner for Refugees, and WHO/UNDP/World Bank Special Programme for Research and Training in Tropical Diseases (project no. 960662). M.R. and D.C. are supported by the UK Department for International Development and the Gates Foundation. However, none of these donors can accept responsibility for any information provided or views expressed.
Dr Daniel Chandramohan, Department of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, UK. E-mail: firstname.lastname@example.org
Tim Freeman, HealthNet International, 11A, Circular Lane, PO Box 889, Peshawar, Pakistan. E-mail: email@example.com
Professor Brian Greenwood, Department of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7Ht, UK. E-mail: firstname.lastname@example.org
Amin Ullah Kamawal, HealthNet International, 11A, Circular Lane, PO Box 889, Peshawar, Pakistan.
Fazle Rahim, HealthNet International, 11A, Circular Lane, PO Box 889, Peshawar, Pakistan. E-mail: email@example.com
Mark Rowland, Department of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, UK. E-mail: firstname.lastname@example.org
Jayne Webster, Department of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine, Keppel Street, London, UK. E-mail: email@example.com (corresponding author).