Validation of the International Study of Asthma and Allergies in Children (ISAAC) and U.K. criteria for atopic eczema in Ethiopian children

Authors

  • A. Haileamlak,

    1. Department of Paediatrics and Child Health, Jimma University, Ethiopia
      Departments of *Respiratory Medicine, †Epidemiology and Public Health, City Hospital, University of Nottingham, U.K.
      ‡Centre of Evidence-Based Dermatology, Queen's Medical Centre, University of Nottingham, Nottingham NG7 2UH, U.K.
    Search for more papers by this author
  • S.A. Lewis,

    1. Department of Paediatrics and Child Health, Jimma University, Ethiopia
      Departments of *Respiratory Medicine, †Epidemiology and Public Health, City Hospital, University of Nottingham, U.K.
      ‡Centre of Evidence-Based Dermatology, Queen's Medical Centre, University of Nottingham, Nottingham NG7 2UH, U.K.
    Search for more papers by this author
  • J. Britton,

    1. Department of Paediatrics and Child Health, Jimma University, Ethiopia
      Departments of *Respiratory Medicine, †Epidemiology and Public Health, City Hospital, University of Nottingham, U.K.
      ‡Centre of Evidence-Based Dermatology, Queen's Medical Centre, University of Nottingham, Nottingham NG7 2UH, U.K.
    Search for more papers by this author
  • A.J. Venn,

    1. Department of Paediatrics and Child Health, Jimma University, Ethiopia
      Departments of *Respiratory Medicine, †Epidemiology and Public Health, City Hospital, University of Nottingham, U.K.
      ‡Centre of Evidence-Based Dermatology, Queen's Medical Centre, University of Nottingham, Nottingham NG7 2UH, U.K.
    Search for more papers by this author
  • D. Woldemariam,

    1. Department of Paediatrics and Child Health, Jimma University, Ethiopia
      Departments of *Respiratory Medicine, †Epidemiology and Public Health, City Hospital, University of Nottingham, U.K.
      ‡Centre of Evidence-Based Dermatology, Queen's Medical Centre, University of Nottingham, Nottingham NG7 2UH, U.K.
    Search for more papers by this author
  • R. Hubbard,

    1. Department of Paediatrics and Child Health, Jimma University, Ethiopia
      Departments of *Respiratory Medicine, †Epidemiology and Public Health, City Hospital, University of Nottingham, U.K.
      ‡Centre of Evidence-Based Dermatology, Queen's Medical Centre, University of Nottingham, Nottingham NG7 2UH, U.K.
    Search for more papers by this author
  • H.C. Williams

    1. Department of Paediatrics and Child Health, Jimma University, Ethiopia
      Departments of *Respiratory Medicine, †Epidemiology and Public Health, City Hospital, University of Nottingham, U.K.
      ‡Centre of Evidence-Based Dermatology, Queen's Medical Centre, University of Nottingham, Nottingham NG7 2UH, U.K.
    Search for more papers by this author

  • Conflicts of interest: The authors have none to declare.

Hywel Williams.
E-mail: hywel.williams@nottingham.ac.uk

Summary

Background  Reliable diagnostic criteria for atopic eczema (AE) are essential in order to make international comparisons and to identify possible disease risk factors. Little is known about the prevalence of atopic eczema and validity of diagnostic criteria for AE in developing countries where English is not the first language.

Objectives  We sought to determine the prevalence of AE in an area of urban and rural Ethiopia, and to compare the predictive values of different questionnaire and examination methods for diagnosing AE in this population.

Methods  We conducted a cross-sectional survey of 7915 children aged 1–5 years living in and around the town of Jimma in southwest Ethiopia. AE prevalence was assessed in two ways: (i) by using the International Study for Asthma and Allergies in Childhood (ISAAC) questionnaire, and (ii) using the U.K. refinement of Hanifin and Rajka's diagnostic criteria. All possible cases identified by screening questions and random samples of controls were then examined by an experienced local paediatrician, who acted as a reference standard to determine the predictive value of the criteria used to diagnose AE.

Results  The overall 1-year period prevalence of AE according to ISAAC and U.K. criteria was 4·4%[95% confidence interval (CI) 3·95–4·85] and 1·8% (95% CI 1·5–2·1), respectively. Corresponding point prevalence estimates (symptoms in the last week) were 1·8% for ISAAC and 1·3% for the U.K. criteria. The positive predictive values of the ISAAC and U.K. criteria questions for AE symptoms still reported to be present (in the last week) at the doctor's examination were 48·8% and 55·5%, respectively. Corresponding negative predictive values were 90·5% and 90·1%, respectively. The sign of visible flexural dermatitis (a component of the U.K. criteria) when used alone had positive and negative predictive values of 57% and 91%, respectively.

Conclusions  Neither the ISAAC nor U.K. criteria performed especially well in predicting cases of AE in this survey. Possible reasons include problems with questionnaire translation, cultural conceptions of terminology, asking parents rather than the child about symptoms, the transient nature of AE signs, and differences in what a doctor perceives to constitute a typical case of AE. The results do not preclude the use of standardized diagnostic criteria alongside a doctor's examination in future surveys of Ethiopian children, and knowledge of the criteria's limited predictive value should help to interpret study findings that have employed such criteria. Consideration should be given to adopting the sign of visible flexural dermatitis as a standard for estimating the point prevalence of AE throughout the world because it is less susceptible to problems with translation and interpretation.

Atopic eczema (AE), also known as atopic dermatitis, is a pruritic chronic skin condition of both children and adults. The prevalence of AE has probably been increasing in many parts of the world over the past 50 years, and AE is now causing a significant adverse impact on the public health.1,2 The International Study of Asthma and Allergies in Childhood (ISAAC) study showed a substantial variation in the prevalence of symptoms of AE around the world.3 In general the geographic pattern of eczema prevalence is concordant with that of other major atopic diseases, asthma and hay fever, with high prevalence of all conditions found in the developed countries such as northern Europe, Japan and Australasia and low prevalences in countries such as China.3 Surveys conducted worldwide revealed a prevalence of 5–23% of AE in different age groups of children.3–7

One possible reason for this wide variation in the observed prevalence of AE symptoms could be variations in the validity of the instruments used for the survey. Reliable diagnostic criteria for AE are essential in order to make international comparisons and to identify possible risk factors. Ten years after Hanifin and Rajka proposed a list of possible diagnostic criteria for the clinical diagnosis of AE,8 questionnaire-based instruments were developed for epidemiological studies of eczema, such as the ISAAC questionnaire (Table 1).9 Just after the ISAAC questionnaire was developed, a U.K. Working Party set about developing a minimum list of reliable discriminators for AE, based on the original Hanifin and Rajka list of features. The U.K. criteria include five questions and one physical sign of flexural dermatitis that nondermatologists can be trained to ascertain (Table 1).10,11

Table 1.  The ISAAC questionnaire designed to detect symptoms of atopic eczema and the U.K. Working Party's diagnostic criteria for atopic eczema. Questions used in the screening cross-sectional survey are shown in bold
The ISAAC questionnaire designed to detect symptoms of atopic eczema
Question 1. ‘Has your child ever had an itchy rash which was coming and going for at least 6 months?’
Question 2. ‘Has your child had this itchy rash at any time in the last 12 months?’
Question 3. ‘Has this itchy rash at any time affected any of the following places: the folds of the elbows, behind the knees, in front of the ankles, under the buttocks, around the neck, around the ears or eyes?’ A positive response could be given to one or more.
Positive response to questions 1–3 is diagnostic of atopic eczema.
Positive response to questions 1 and 2 was defined as possible case for examination.
The U.K. Working Party's diagnostic criteria for atopic eczema
Must have: An itchy skin condition (or parental report of scratching or rubbing in a child) (screening question)
Plus 3 or more of the following:
1. History of involvement of the skin creases such as folds of elbows, behind the knees, front of ankles or around the neck (including cheeks in under 10 years).
2. A personal history of asthma or hay fever (or history of atopic disease in the first degree relative in those under 4 years).
3. A history of a generally dry skin in the last year.
4. Visible flexural dermatitis (or dermatitis involving the cheeks/forehead and outer of the limbs in children under 4).
5. Onset under the age of 2 (not used in less than 4 years old).
Full instructions on how to use the U.K. criteria are available in a manual athttp://www.nottingham.ac.uk/dermatology/eczema/Section6-3Appendix1.html
Positive response to the screening question was defined as possible case for examination.

Comprehensive validation studies for the U.K. Working Party12–17 and the ISAAC diagnostic criteria18,19 performed in many parts of the world have demonstrated their validity, although some have shown low sensitivity.17,19 One study in Germany has suggested that cultural and educational factors could influence the universal application of such questionnaire-based criteria.20 Our prime research goal was to investigate possible risk factors for AE in an urban and rural Ethiopian population.21 Before accepting the use of criteria such as ISAAC and the U.K. criteria as our main methods of case definition, we have carried out a validation study to compare the predictive values of different ways of diagnosing AE in this population, and determined the prevalence of AE in an area of urban and rural Ethiopia according to these validated criteria.

Materials and methods

Study design and population

We conducted a cross-sectional screening survey targeting all children aged 1–5 years living in the town of Jimma and 11 surrounding rural farming associations in Mana and Seqa-chekorsa (Shebe) districts in southwest Ethiopia during the last week of August and first week of September 2003. We used a brief interviewer-administered questionnaire to obtain information from parents or other carers on the occurrence of itchy skin conditions to identify possible cases. The questionnaire comprised the three ISAAC criteria questions and the screening question for the U.K. Working Party criteria (Table 1). We identified as possible cases those having a positive response to the first two ISAAC questions and/or the U.K. diagnostic criteria screening question (Table 1). We also drew a random subsample of all children who participated in the cross-sectional survey, irrespective of their questionnaire response, using random selection in SPSS, and took as controls those from the random subsample who did not qualify as cases. We revisited all possible cases and controls to administer the full ISAAC questionnaire and questions on AE symptoms from the U.K. criteria, translated into Amharic, and trained fieldworkers looked for signs of visible flexural dermatitis according to the training protocol (http://www.nottingham.ac.uk/dermatology/eczema/index.html, accessed 4 November 2004). All possible cases and the random subsample of controls were then examined by a local experienced paediatrician, familiar with AE and blinded to the questionnaire responses, who acted as a reference standard to determine the predictive value of the ISAAC and U.K. Working Party criteria for AE. As several months elapsed in some cases between the cross-sectional survey and the clinical examination, we asked both sets of patients at the clinical examination whether their symptoms had been present in the last week.

Operational definitions

Cases  To fulfil the ISAAC criteria, a case of AE must have positive responses to all three ISAAC eczema questions. To fulfil the U.K. diagnostic criteria, a case of AE must have an ‘itchy skin condition’ plus three of the five subsidiary questions shown in Table 1.

Controls  We selected a random subsample from all those children who took part in the cross-sectional survey, comprising a similar number of children to the potential case group, and took those who were not cases as controls, as shown in the flow diagram in Figure 1.

Figure 1.

Flow diagram showing how study participants were selected.

Data analysis

Data were entered and analysed using SPSS package version 11.0. The prevalence of AE, by age and sex, and the positive and negative predictive values (PPV and NPV) were calculated for each criteria, and for those satisfying the criteria with symptoms that were present in the last week at the doctor's examination, using standard formulae.22 Where components of the criteria were collected at the validation stage, prevalence estimates were weighted to adjust for the response rate at this stage. It was not possible to calculate the specificity of the various criteria used in this paper due to the sampling methods that were needed to increase efficiency for the main case–control risk factor analysis.

Ethics clearance

Ethics permission for the study was obtained from the relevant committees in both Nottingham and Jimma Universities. Consent was obtained from parents/caretakers after informing about the procedures and the purpose of the study.

Results

There were 7915 children aged 1–5 years (mean 3 years) who participated in the survey, 4012 (51%) male and 3903 (49%) female. Reliable census data are not available for the Jimma population under age 5, so the overall response rate is unknown, but anecdotally there were few refusals. We identified 590 possible cases in the survey, and generated a random sample of 554 individuals from all participants (34 of whom had already been identified as possible cases). We succeeded in completing a detailed assessment in 506 (86%) of possible cases and 438 (79%) of the random subsample (Fig. 1). The male to female ratio, age and urban–rural distribution of those participating in the detailed study was similar to that of the initial screening survey.

Prevalence

ISAAC criteria  Three hundred and fifty children met the full ISAAC diagnostic criteria, corresponding to a 1-year period prevalence of AE of 4·4%[95% confidence interval (CI) 3·95–4·85] by the ISAAC criteria. Responses to individual components of the ISAAC questions are shown in Table 2. Figure 2(a) shows that the prevalence of AE as defined by ISAAC questions declined nonsignificantly with increasing age in both sexes but that AE appeared more common in boys than girls during the first, third and fourth years of life.

Table 2.  Prevalence and diagnostic validity of the ISAAC and U.K. diagnostic criteria for atopic eczema against a physician diagnosis, Jimma, Ethiopia, 2004
 Prevalence n (%)PPVNPV
  1. PPV, positive predictive value; NPV, negative predictive value. aObserved numbers are not shown because these data were collected at the validation stage, and prevalence estimates have been weighted to adjust for the response rate at this stage.

Total7915  
ISAAC criteria
1. Itchy rash, coming and going, over 6 months in the past year482 (6·1%)32·9 (137/416)91·5 (386/422)
2. As (1) above, and affecting skin creases (ISAAC complex)350 (4·4%)33·3 (102/306)91·1 (388/426)
As (2) above, and reported to be present in the last week at clinical examination1·8%a48·8 (62/127)90·5 (391/432)
U.K. diagnostic criteria
1. Itchy skin condition in past year506 (6·4%)29·7 (132/445)91·2 (385/422)
As (1) above AND
 Onset before age 2
3·6%a 30·2 (76/251) 90·4 (388/429)
 Involvement of skin creases2·1%a47·3 (69/146)91·0 (392/431)
 History of asthma or rhinitis2·3%a35·0 (57/163)90·3 (391/433)
 Dry skin2·3%a43·4 (69/159)90·5 (391/432)
 Visible flexural dermatitis1·7%a57·0 (69/121)91·0 (393/432)
2. As (1) above plus two of the additional criteria3·6%a41·5 (105/253)91·3 (390/427)
3. As (1) above plus three of the additional criteria1·8%a52·0 (66/127)90·8 (393/433)
As (3) above and reported to be present in the last week at clinical examination1·3%a55·5 (50/90)90·1 (393/436)
Figure 2.

Prevalence of atopic eczema by age and sex, Jimma, Ethiopia, 2004. (a) ISAAC criteria and (b) U.K. Working Party criteria.

U.K. Working Party diagnostic criteria  One hundred and twenty-seven children met the full U.K. Working Party's diagnostic criteria for AE, corresponding to a 1-year period prevalence of AE of 1·8% (95% CI 1·5–2·1). The prevalence of responses to each component of the U.K. criteria are shown in Table 2. The prevalence of AE declined nonsignificantly with increasing age in both sexes as shown in Figure 2(b).

Positive and negative predictive values

ISAAC criteria  Of 306 children who fulfilled the ISAAC criteria for AE symptoms in the last year and who were also available for examination, 102 had AE by clinical examination, PPV 33·3% (95% CI 28·0–38·6). The PPV ranged from 32·9% (95% CI 28·4–37·4) ‘for itchy rash in the last 6 months’ to 48·8% (95% CI 40·1–57·5) for the full ISAAC criteria plus ‘itchy skin rash in the last one week’. A total of 426 controls (non-ISAAC cases) were identified from the random subsample and examined. Of these, 388 were confirmed as not having current AE by clinical examination, NPV 91·1 (95% CI 88·4–93·8). The NPV was above 90% for individual as well composite computation (Table 2).

U.K. Working Party criteria  Of 127 children who met the U.K. Working Party's criteria for AE, 66 (52%) were confirmed by clinical examination as having AE (95% CI 43·3–60·7). The PPV ranged from 29·7% (95% CI 25·4–33·9) ‘for itchy skin condition in the past year’ to 57·0% (95% CI 48·2–65·9) for visible flexural dermatitis. Of the 433 controls (those in the random subsample who did not have AE in the last year according to the U.K. criteria), 393 were confirmed as not having current AE by clinical examination, NPV 90·8 (95% CI 88·0–93·5) (Table 3). The NPV was above 90% for the U.K. Working Party's eczema question for individual as well as for composite computation. The PPV and NPV for ‘itchy skin condition’ in the last week plus three from the remaining five U.K. eczema questions was 55·5 (50 of 90) and 90·1 (393 of 436), respectively (Table 2).

Table 3.  Doctor diagnosis of skin conditions in cases and controls by ISAAC and U.K. Working Party's criteria for atopic eczema. Jimma, Ethiopia, 2004
Doctor diagnosisISAACU.K. Working Party
Case; n (%)Control; n (%)Case; n (%)Control; n (%)
AE102 (33·3)38 (8·9)66 (52·0)40 (9·2)
Scabies 49 (16·0)28 (6·6)23 (18·1)30 (6·9)
Warts  3 (1)  1 (0·2) 0 (0) 1 (0·2)
Molluscum contagiosum  5 (1·6) 13 (3·1) 1 (0·8)14 (3·2)
Fungal infection  0 (0)  0 (0) 0 (0) 0 (0)
Impetigo  6 (1·9)  0 (0) 4 (3·1) 0 (0)
Other skin problem  1 (0·3)  2 (0·4) 0 (0) 2 (0·5)
Healthy skin139 (45·4)344 (80·7)33 (26·0)346 (79·9)
Total306 (100)426 (100)127 (100)433 (100)

Other diagnoses at the clinical examination in the false positives

Forty-nine (16%) and 23 (18%) identified as cases by the ISAAC and U.K. Working Party criteria, respectively, were found to have scabies by clinical diagnosis. Scabies was diagnosed in 28 (6·6%) and 30 (6·9%) of controls for ISAAC and U.K. Working Party criteria, respectively (Table 3).

Discussion

Main findings

This is the first study of its kind to evaluate these questionnaire-based instruments for AE on such large numbers of children living in a developing country. The prevalence of AE was found to be low by both criteria (ISAAC 4·4% and U.K. 1·8%) in this study population living in southwest Ethiopia, and much lower than eczema symptom prevalence estimates in many other parts of the world.1–4 Generally, the ISAAC and U.K. diagnostic criteria did not perform well in relation to examination by an experienced paediatrician in our study. PPVs for symptoms in the last week were about 50% for both instruments, and NPV was about 90%. The one objective sign of visible flexural dermatitis showed the best validity, with a PPV and NPV of 57% and 91%, respectively, in relation to the physician's examination.

There are at least six possible reasons why the various diagnostic criteria might not have performed as well as we had hoped in this Ethiopian study. First, our doctor's examination suggested that some of the reported skin conditions were in fact scabies, which is common in this population. Second, low disease prevalence will affect the PPV as shown in the worked examples illustrated in the U.K. Working Party's online manual (http://www.nottingham.ac.uk/dermatology/eczema/Section5-1.html, accessed 4 November 2004). Even a set of criteria with 80% sensitivity and 97% specificity will yield a PPV of 45% if the real disease prevalence is as low as 3%. Third, it is possible that asking parents for symptoms in their children, as opposed to asking the children who experience the symptoms themselves, might result in some further misclassification. Fourth, it is possible that some validity could have been lost during the translation of the questionnaires used in this study. Although great care was taken to ensure that translation was done by two people fluent in Amharic and English, it is possible that some of the phrases were translated incorrectly. To investigate this possibility, we arranged for an independent back translation to be done. Although the back translation identified a number of minor inconsistencies, these were unlikely to have influenced the criteria validity to the extent found. Fifth, it is possible that cultural perceptions of questions pertaining to AE symptoms may be very different in rural Ethiopia when compared with other Western cultures. For example, even if the translation were perfect, people living in a culture which associated itchy skin conditions with arthropod infestation and uncleanliness may be less likely to answer affirmatively to such questions even when present. Further ethnographical and anthropological studies are needed to clarify such cultural subtleties.

Finally, it is possible that the doctor responsible for ascertaining cases in order to act as a reference standard for the criteria under test might have had a different concept of what constituted a typical case of AE when compared with northern European and U.S. physicians, an idea which is supported by a study from Iran.17 We mitigated against this possibility by selecting an experienced paediatrician (A.H.) who was familiar with AE and its differential diagnosis to act as our reference standard. The same person also undertook a period of study with a dermatology department in the U.K. in order to further familiarize himself with the concept of AE. However, it should be acknowledged that AE may well be composed of a range of specific diseases with different genotypes, and even though such cases may appear similar in countries such as the U.K., they may exhibit different phenotypes when exposed to a very different range of environmental risk factors in rural Ethiopia. If such a situation turns out to be true, then it may be argued that rather than validating criteria such as the U.K. criteria against local doctors in developing countries, they should be used side by side—the U.K. criteria (or just the sign of visible flexural dermatitis that is immune from translational and cultural problems) determining prevalence of the more typical flexural phenotype seen in developed countries, and the local doctor-defined cases denoting what is considered to be the local AE phenotype. Analytical studies that employ both definitions might reveal a different risk factor profile for each definition. From a public health perspective, those cases defined by a local doctor are most likely to represent the main burden of disease requiring access to healthcare services.

Strengths and limitations of this study

Strengths of this study include the representative and large sample of children from both urban and rural locations in Ethiopia, and the high response rate. Field workers were trained in administering the questionnaires and in ascertaining the sign of visible flexural dermatitis according to the recommended manual. The doctor acting as a reference standard was kept blinded from the results of the questionnaire aspect of the study.

Limitations include our inability to ascertain the specificity of the various instruments because of the sampling methods that were chosen for the main case–control study. Another limitation was that it was not always possible to examine children within 1 week of the completion of the questionnaire data, this being the recommended time interval for assessing the symptom-based questionnaires against a point-prevalence based on a single physical examination. For practical reasons, in some cases, a 2-month interval occurred between parents answering the questionnaires and the doctor examining the child, during which time previously active eczema could have disappeared or new eczema could have developed.

One further important consequence of using a definition with a low PPV of about 50% is loss of power (secondary to halving the cases) to undertake associative analyses. A balance has therefore to be struck between validity and obtaining enough cases to adequately examine the relationship between various risk factors in a case–control study.

Conclusions

We have validated the Amharic version of the U.K. Working Party and ISAAC questionnaires for AE. We have shown that the predictive values are less than those ascertained from other validity studies conducted in developing countries and have suggested a number of reasons why this might be so in our study population. We recommend using such questionnaires alongside clinical examination by a local dermatologist where possible for prevalence and analytical studies in developing countries, especially where English is not the first language. Further studies exploring the validation of the same instruments using a cross-sectional design so that specificity can be determined are currently underway in this region of Ethiopia.

Ancillary