Quality of questionnaires for the assessment of otitis media with effusion in children

Audiometric tests provide information about hearing in otitis media with effusion (OME). Questionnaires can supplement this information by supporting clinical history‐taking as well as potentially providing a standardized and comprehensive assessment of the impact of the disease on a child. There are many possible candidate questionnaires. This study aimed to assess the quality and usability of parent / child questionnaires in OME assessment.


| INTRODUCTION
Otitis media with effusion (OME) is the commonest cause of hearing loss among children. Studies on British children have found prevalence as high as 36.6% among 8-month-old children, 1 an annual prevalence of 16% among 5-year-olds, 2 which decreases to 6% in those aged 8 years. 2 These studies have also shown that OME is more common in the winter months. 1 Although the natural history is towards self-resolution, 2 in some cases, it persists beyond a few months and may require active treatment to prevent language, learning and behavioural problems. Treatment options include hearing aids, ventilation tube insertion or autoinflation. 3,4 Decisions about treatment in the United Kingdom are supported by NICE guidelines which recommend that ventilation tube insertion should be considered for OME persisting at least 3 months and with hearing loss of 25 to 30 dB HL or worse in the better ear, or if there is a significant impact on the child's social, educational or developmental outcome. 5 Thus, in addition to objective tests such as the pure-tone audiogram, clinical history through parental report is used in decisionmaking, and questionnaires could offer a formalised means for obtaining this information. Decision-making in OME often does not follow guidelines, 6 and published studies often report a variety of different outcomes 7 ; hence, identification of high-quality tools to capture parental report of the impact of OME is crucial to facilitate good and transparent decision-making both in the clinic and research setting.
In addition to supporting clinical history-taking, questionnaires can be useful tools, as they provide a standardised and comprehensive method of assessing the impact of the hearing loss on the everyday functioning of the child over a period of time. As the incidence and impact of OME fluctuate throughout the year, 8 such report-based measures provide additional benefit by capturing long-term effects.
Many different questionnaires have been developed for assessing children with hearing loss and associated difficulties. Some like the Otitis Media 6 (OM-6) 9 or the Quality of Life in Children's Ear Problems (OMQ- 14) 10 have been specifically designed to support clinical historytaking for children with OME. Others have been developed for hearing screening purposes, or to target problems specific to the auditory periphery or other problems associated with the processing of auditory inputs (ie [central] auditory processing disorder [APD], sometimes referred to as listening difficulties), but as children with hearing loss share symptoms with children with OME, they may still be useful in the context of OME assessment.
The aim of this study was to systematically review questionnaires that are commonly used in the clinic to assess problems with either hearing-or listening-related difficulties in the context of clinical assessment of otitis media with effusion.

| ME TH ODS
This study focused on assessing publicly-available questionnaires that collected carer or child (self) report-based assessment of hearing or listening abilities in children aged below 18 years. These Andresen's framework in the hearing context. We further adapted this framework as described below to achieve a similarly structured assessment of questionnaires in the context of OME.
Bagatto et al applied 13 criteria including conceptual clarity (captures relevant domains ie symptoms of disease together with impacts on social, educational and developmental outcomes, normative data, measurement model (ceiling or floor effects), item or instrument bias, respondent burden (number of items, interpretability of items and usability of response scale), administrative burden (ease of scoring and interpretation), reliability (consistency of results across time and scorers), discriminant validity, convergent validity, responsiveness (ie sensitivity to treatment-related change), alternate or accessible forms (eg paper vs electronic), and adaptations for use in different languages or cultures). While it would have been interesting to assess treatment-related responsiveness, even the questionnaires specifically designed to look at treatment effects (see Children's Outcome Worksheets (COW) and Early listening Function (ELF), Table 1) did not formally assess this, and responsiveness was therefore excluded. We also did not assess whether there were alternative formats or other adaptations as there was little or no information available for any of the questionnaires assessed here.
Once the final framework of eight relevant categories was established, the questionnaires were reviewed and scored with respect to these categories (Tables 1 and 2). The ratings of some characteristics involved a subjective assessment on the part of the authors, while Keypoints • There are many questionnaires available to aid assessment of otitis media with effusion (OME) in children.
• There are considerable gaps in the formal evaluation of available questionnaires.
• Evaluation of Children's Listening and Processing Skills (ECLiPS), LittlEARS Auditory Questionnaire (LittlEARS), and Parents' Evaluation of Aural/Oral Performance in Children (PEACH) satisfy most of the criteria laid out in this study for the evaluation of questionnaires.

| Conceptual clarity
The questionnaire should be developed with the purpose of capturing information regarding the disease's impact not only in relation to the child's perceived level of hearing, but also their speech, language, psychosocial development 5,13 and capacity to function in educational settings. 5 Ratings were based on the authors' evaluation. A rating of excellent "A" was awarded if the questionnaire had a completely relevant purpose, moderate "B" if it captured some issues of relevance, but not all and poor, "C," if it was only tangentially relevant.

| Respondent burden
Reliability of response depends to some extent on respondent burden.
On a simplistic level, burden can be defined according to numbers of items. However, of more significance is the ease with which items can

| Reliability
The questionnaires should provide consistent results. Reliability can be further split into test-retest reliability, inter-rater reliability and internal consistency (Cronbach's alpha). These three measures were available for the Evaluation of Children's Listening and Processing Skills (ECLiPS). Scores for the remaining questionnaires were based on whatever was reported (typically Cronbach's alpha or test-retest reliability). If the necessary data and the statistics assessing test-retest, inter-rater reliability and internal consistency were unavailable, the questionnaire was given a score of "U." Test-retest reliability is how consistent the results are after multiple administrations to the same group of respondents. This is statistically measured using the retest intraclass correlation coefficient (r).
Questionnaires were rated "A" if r was greater than or equal to .75, "B" if it was between .75 and .40, and "C" if it was less than or equal to .40.  Internal consistency refers to how consistently items within the same factor measure the same construct. 12 This can be assessed statistically using Cronbach's alpha, a correlation coefficient. Questionnaires with a Cronbach's alpha greater than or equal to .8 were rated "A," less than .8 but more than .7 "B" and less than or equal to .7 "C." Inter-rater reliability refers to how two different people score the same child. If the intraclass correlation was greater than or equal to .75 questionnaires were rated "A," "B" if it was between .75 and .40, and "C" if it was less than or equal to .40.
Reliability can be considered a precursor to validity, 14 as a test must be reliable to be valid. 15  not exist making it hard to assess criterion validity. However, if criterion validity was assessed, a rating of "A" was given for a correlation coefficient of greater than or equal to .60, a rating of "B" was assigned if it was below .60 but greater than .30, and a rating of "C"

| Validity
if it was less than or equal to .30.

| Normative data
A child's symptoms of disease impact may change with age as their speech, language, cognitive and psychosocial skills develop. As a consequence, any non-physical symptoms of disease should be normed for these changes. Questionnaires with normative data for a sufficiently large number of children for reliable regression analyses with age (ie 20 -30 children per age band), spread across different ages, with normal hearing and those with hearing loss were given a score "A"; "B" if the data were only available for children with normal hearing; "C" if only some preliminary experimental data were available; and "U" if no data were available.

| Item bias
Items within the questionnaire should not show bias towards a particular culture, race or gender. This can be assessed using a variety of techniques including simple regression analyses or more sophisticated modern psychometric techniques such as Rasch analysis.
Questionnaires were rated "A" if there was evidence that bias effects were assessed, with sufficient numbers of participants to be able to do this reliably, and any effects observed were addressed as appropriate. Questionnaires were rated "B" if there was evidence of an attempt to assess bias effects, but there were insufficient numbers of participants to be able to do this reliably. They were rated "C" if there was acknowledgement that bias may affect observations, but no attempt was made to assess this, possibly because of small participant numbers. They were rated "U" if the issue of bias was not considered during development.

| Ceiling/floor effects
To be maximally sensitive to as many individual differences as possible, questionnaires should be minimally susceptible to ceiling or floor effects. This is evident in a marked tendency for response distribu-

| Administrative burden
Administration of questionnaires should have a low administrative burden. That is results should be easily obtained, quantified and interpreted by the clinician. This was based on the subjective assessment of the authors, giving a score of "A" if the questionnaire involved simple addition and outputs were easy to interpret and interpret; "B" if scoring involved some additional manipulations but outputs were still easy to interpret; and "C" if scoring involved considerable manipulation of data and/or interpretation of outputs was not immediately obvious.

| Ethical considerations
Approval of an ethics committee was not required as analyses were based on published anonymised data.

| RESULTS
Fifteen questionnaires were identified and were assessed and rated (Tables 1 and 2). Ratings for each questionnaire according to the eight defined characteristics are summarised in Table 2.

| Conceptual clarity
Reflecting our search criteria, most questionnaires covered domains that were considered of at least adequate relevance to OME. OMQ-14 and OM-6 were specifically designed to assess children with OME and, as such, they captured most aspects of relevance to the disease and were rated "A." The majority of the remaining questionnaires captured some aspects but not all of interest to clinicians assessing children with OME. They were therefore rated "B." Auditory Behaviour in Every- Listening Situations Questionnaire and COW were rated "C" because they focused solely on hearing in a limited range of settings as well as being primarily designed to assess the benefit of amplification on hearing function.

| Respondent burden
The two volunteer parents found the ABEL, ECLiPS, ELF, FAPC, HL- Data for reliability, validity, normative data, item bias and ceiling/ floor effects were unavailable for most questionnaires.

| Reliability
Parents' Evaluation of Aural/Oral Performance of Children and OM-6 had test-retest intraclass correlation coefficients of 0.93 16

| Validity
Evaluation of Children's Listening and Processing Skills had good criterion-related and discriminant validity. Where ECLiPS domains were predicted to measure the same constructs as other questionnaires correlations, r of greater than .50 18 were observed (ie evidence of criterion validity). Likewise, where ECLiPS domains were predicted to measure different constructs to those measured by other questionnaires (ie evidence of discriminant validity), weak or no correlation was observed (r < .35). 18 OM-6 was described as having good construct validity based on 87.5% of hypothesised correlations between items (inter-item), between items and summary scores (item-total) and between summary scores (total-total) within and between the questionnaires. 17 Factor loadings from factor analysis of ABEL also suggested good

| Challenges in assessing OME
Despite clear guidelines from NICE and other bodies setting out criteria for the treatment of patients with OME, 5 clinicians frequently treat patients because of "extenuating circumstances" rather than because of meeting core criteria for ventilation tube insertion. 6 These extenuating circumstances refer to the perceived detrimental effects of recurrent infections as well as impact of OME on quality of life, speech, language psychosocial and educational development. 6 How these extenuating circumstances are established is not clear, and certainly, there is a need for an assessment tool that would allow clinicians to manage the child as a whole, not just on the basis of the pure-tone audiogram. Good quality questionnaires designed to capture these broader issues, in addition to hearing loss, could potentially meet this need.
A further challenge is one of distinguishing hearing-related problems from those linked to listening, attention and other cognitive aspects that influence a child's daily functioning. Do the children having surgery due to extenuating circumstances have problems only and predominantly due to OME? Or do these children have additional underlying language, social or cognitive problems with OME being coincidental? This is at present unknown, but reliably differentiating hearing problems from these other difficulties is clearly important, and questionnaires have a potential role to play in doing this.
Furthermore, studies on OME report a wide range of different outcome measures making comparison between them difficult. 7 The use of a well-developed psychometrically robust questionnaire which is sensitive to all issues relevant to OME would help standardise the measurement of disease impact and treatment outcome, thus aiding effective comparison between studies.

| Questionnaires in OME
Clinicians have access to many different questionnaires to support their assessment and decision-making. It can be difficult to decide which questionnaires to use. Ideally, clinicians need a questionnaire that offers information about the disease of interest that is clinically relevant, and psychometrically robust, while at the same time keeping respondent and administrator burden to a minimum.
As is apparent from Table 2, no single questionnaire met all the criteria established in this study. There were considerable gaps in the formal evaluation of the quality of the available questionnaires.
As noted, when reviewing components of the analysis framework, it may have been useful to measure sensitivity of the questionnaires to treatment-related change, that is, "responsiveness." However, there is a lack of clarity on the definition of "responsiveness," meaning a lack in standardization of methodology for its calculation. 26 Furthermore, although important for interpreting changes in scores subsequent to treatment, 26 the concept provides little insight into the measurement validity of an instrument. It has consequently been argued that "responsiveness" does not need to be separately assessed when validating a questionnaire. 26

| Limitations
Evaluation of Children's Listening and Processing Skills, LittlEARS and PEACH scored better than most of the other questions, in part because the developers included more thorough assessments of the psychometric properties of the questionnaires as part of the development process. However, these tools were not specifically designed to assess OME and may assess issues that are not strictly relevant to OME. As such, they may contain items that collect information that is redundant in the context of OME. Further exacerbating this problem, all 8 assessment criteria in the assessment framework received equal weighting in terms of relative importance. Yet, it could be argued that some components of the framework specifically conceptual clarity and validity are more important than others, such as administrative or respondent burden. The framework applied was useful for supporting a systematic assessment of the questionnaires but future researchers using this framework could consider incorporating some method for weighting the different assessment criteria according to importance relative to purpose.
In terms of item redundancy, LittlEARS was designed to evaluate age-appropriate auditory behaviour, in the pre-verbal developmental phase, while PEACH focuses on the effects of intervention with hearing aids or implants. Both of these questionnaires are narrowly focused on hearing with many items focusing on capturing symptoms of hearing loss in different contexts. This is potentially very GAN ET AL.

| CONCLUSION
We have assessed 15 commonly used questionnaires for possible use in children with OME, using set criteria. There were considerable gaps in the formal evaluation of the available questionnaires. ECLiPS, LittlEARS and PEACH appear to be the most suitable ones as they most fully satisfy the criteria laid out in this study for the assessment of questionnaires. Although initially developed for different purposes, they cover domains, which are also of relevance to OME. Questionnaires in OME could provide a useful adjunct to audiometry, as a means of assessing a child's everyday function, but further research is needed to determine how they fit into the overall assessment of children with OME.