- Top of page
- RésuméAbstractResumenZhaiYaoYo yak
This article presents the results of an experiment designed to compare item-level non-response rates to paper- and web-based versions of a survey questionnaire, focusing on attitudinal variables, that was administered to highly accomplished teachers. A sample of teachers reported their perceptions of professional community; half were assigned to a web-based version of the questionnaire, and the other half to a paper-based version. In both groups, the survey implementation procedures reflected Dillman’s (2007)Tailored Design Method. Item-level nonresponses were compared between groups for overall rates, and differential response rates by demographics, item position, item format (i.e., branching items and respondent-provided text), and item content. Results revealed small differences in item-level nonresponse rates, both overall and in comparisons between demographic groups. In addition, there was no evidence of differential item-level nonresponse by item position and item content. However, item-level nonresponse rates were considerably higher for fill-in-the-blank items on the web-based questionnaire.
Les taux de non-réponse dans un questionnaire d’attitude transmis à des professeurs par courrier et par le web
Cet article présente les résultats d‘une expérience conçue pour comparer les taux d’absence de réponse à chaque question des versions sur papier et sur le web d‘un même questionnaire portant sur les variables d’attitude et distribué auprès de professeurs fort accomplis. Un échantillon de professeurs ont témoigné de leurs perceptions de leur communauté professionnelle. Une moitié fut assignée à une version web du questionnaire, tandis que l‘autre moitié a répondu à une version sur papier. Pour les deux groupes, les procédures d’implantation du sondage reflétaient la méthode du modèle personnalisé de Dillman (2007). Les non-réponses aux questions furent comparées entre les groupes pour obtenir des taux globaux et les taux de réponse différentiels furent comparés selon les données démographiques, la position de la question, le format de la question (par exemple les questions de redirection ou celles demandant au répondant de développer sa réponse) et le contenu de la question. Les résultats révèlent de petites différences dans les taux de non-réponse aux questions, tant en général que dans les comparaisons entre les groupes démographiques. La position et le contenu des questions n‘a pas révélé de taux différentiels d’absence de réponse aux questions. Cependant, les taux de non-réponse aux questions étaient considérablement plus élevés face aux questions à développement du questionnaire web.
Nichtantwortquoten auf Item-Level in einer Einstellungsbefragung (Mail und Web) von Lehrern
Dieser Artikel präsentiert die Ergebnisse eines Experiments, mit Hilfe dessen die Nichtantwortquoten auf Item-Level bei papier- und webbasierten Fragebogenversionen verglichen werden sollte. Der Fragebogen fokussierte auf Einstellungsvariablen und wurde an gut ausgebildete Lehrer zugestellt. Ausgewählte Lehrer teilten ihre Sichtweise auf die berufliche Gemeinschaft mit, wobei jeweils eine Hälfte der webbasierten Gruppe, die andere Hälfte der Papierversion zugeordnet wurde. In beiden Gruppen folgte die Fragebogenprozedur der Tailored Design Method nach Dillman (1980). Die Nichtantwortquoten beider Gruppen wurden bezüglich der Gesamtquote und verschiedenen Antwortraten nach demographischen Variablen, Item-Position, Item-Format (z.B. verzweigte Items und Text des Befragten), und Item-Inhalt verglichen. Die Ergebnisse zeigen geringe Unterschiede der Nichtantwortquoten im Gesamtsample und im Vergleich verschiedener soziodemographischer Gruppen. Es gab keine Belege für unterschiedliche Nichtantwortquoten aufgrund der Position oder des Inhalts des Items. Letztendlich waren die Nichtantwortquoten bei offenen Frage-Items im webbasierten Fragebogen größer.
La Frecuencia del nivel de la No-Respuesta de una Encuesta Actitudinal de Profesores Enviada a través del Correo y la Web
Este ensayo presenta los resultados de un diseño experimental que compara la frecuencia del nivel de la no-respuesta de las versiones en papel y de la web de un cuestionario de encuesta, focalizando en las variables actitudinales, que fueron administradas a profesores altamente consumados. Una muestra de profesores reportó sus percepciones sobre la comunidad profesional; la mitad fueron asignados a una versión web del cuestionario, y la otra mitad a una versión en papel. En ambos grupos, la implementación de los procedimientos de la encuesta reflejó el Modelo de Diseño de Dillman (2007). El nivel de la no-respuesta fue comparado entre los grupos por sus frecuencias generales y por la frecuencia de las respuestas diferenciales basados en datos demográficos, posición del artículo, formato del artículo (a saber, ramificación de los artículos y el texto provisto en la respuesta), y contenido del artículo. Los resultados revelaron diferencias pequeñas en el nivel de frecuencia de la no respuesta, en términos generales y en la comparación de grupos demográficos. Además, no hubo evidencia en el nivel diferencial de los artículos de no-respuesta por la posición del artículo y el contenido del artículo. No obstante, la frecuencia del nivel del artículo de la no respuesta fue considerablemente mayor en los cuestionarios para completar que en los basados en la web.
Increasingly, survey researchers and practitioners are opting to use web-administered questionnaires due to the potential for reduced administration cost, shorter duration of data collection, and automation of data collection, scoring, and reporting. As this shift occurs, however, potential concerns arise about the comparability of results, specifically the reliability or consistency of measures, between administration media. In particular, considerable attention has been directed toward concerns that web-based surveys, although they can be broadcast more widely, may achieve lower unit-level response rates (e.g., higher noncontact and nonreturn rates) (Cook, Heath, & Thompson, 2000; Fricker & Schonlau, 2002; Sheehan & McMillan, 1999). By contrast, few studies have addressed other potential sources of noncomparability between mailed and web-based surveys. The purpose of our study is to go beyond the literature reporting general nonresponse rates and more specifically determine whether respondents who choose to respond to web- or paper-administered surveys differ with respect to the frequency with which they leave items unanswered and the types of items to which responses are not provided.
Prior research indicates that item-level nonresponse rates (i.e., items that are left unanswered on returned questionnaires) are not high, although unanswered items in returned questionnaires may impact item- and scale-level inferences. In one of the more extreme examples, Bosnjak and Tuten (2001) determined that up to 36% of respondents may leave at least one item unanswered. Further, Wolfe (2003) found that, taking into account potential item nonresponses, marginal percentages of estimated drug purchasing rates among teens could range from as low as 1% to as high as 11%. However, it is clear from previous research concerning item-level non-responses that several features of the questionnaire may increase the likelihood of item-level nonresponse. Questions that request personal and sensitive information have higher rates of nonresponse—item-level nonresponse rates of 7% have been observed for questions relating to sexual orientation and income (Gruskin, Geiger, Gordon, & Ackerson, 2001) and ranging from 7% to 14% for questions focusing on illicit drug use and purchases (Kadushin, Reber, Saxe, & Livert, 1998; Wolfe, 2003). Questions that follow branching instructions are also frequently left unanswered (Messmer & Seymour, 1982). Item-level nonresponses also vary across item formats, particularly in web-based questionnaires (Healey, 2007; Smyth, Dillman, Christian, & Stern, 2006). The format and content of the response options provided for a question may also influence item-level nonresponse rates. For example, item nonresponse varies with the number of scale points (Leigh & Martin, 1987), allowance of multiple selections (e.g., to race/ethnicity questions) (Brener, Kann, & McManus, 2003), and provision of “don’t know” options (Johanson, Gips, & Rich, 1993).
Other research indicates that characteristics of the respondent may impact item nonresponse. Most research in this area indicates that those with less education and lower social class tend to produce higher item-level nonresponse (Alvik, Haldorsen, & Lindemann, 2005; Craig & McCann, 1978; Gruskin, Geiger, Gordon, & Ackerson, 2001; Guadagnoli & Cleary, 1992; Kupek, 1998, 1999). Also, older respondents tend to leave more questions unanswered (Colsher & Wallace, 1989; Craig & McCann, 1978; Gruskin, Geiger, Gordon, & Ackerson, 2001; Messmer & Seymour, 1982). Results have been mixed concerning gender effects (Colsher & Wallace, 1989; Guadagnoli & Cleary, 1992; Messmer & Seymour, 1982) and race effects (Gruskin, Geiger, Gordon, & Ackerson, 2001) for item-level nonresponse. Although the literature discusses these characteristics separately, it is likely that these issues are dependent on one another and are not as easily separated in operational settings (i.e., they interact to influence item-level nonresponse).
An important issue that is not clear from prior research is the nature of item-level nonresponse rates in web-based administration of surveys. While several studies exist that compare unit-level response rates between paper- and web-based surveys, few studies exist that directly compare item-level response rates between these two survey media. Research studies report only small differences in overall item-level nonresponse rates between telephone- or paper- and web-based surveys (Bongers & van Oers, 1998; Fricker, Galesic, Tourangeau, & Yan, 2005; Mehta & Sivadas, 1995; Pettit, 2002; Stanton, 1998; Tse, 1998). Detailed studies of differential item nonresponse rates for paper- and web-based surveys with respect to demographic groups, item format, and item content have not been undertaken. Our study provides a direct comparison of the nature of item-level nonresponses for web-based and paper-based versions of a questionnaire.
Although previous research has not focused on the extent to which item-level response rates vary by demographic group, item position (i.e., the serial position of the item), item format (e.g., items requesting respondent-supplied text), and item content (e.g., attitudinal items) on web-based versus paper-based questionnaires, there is reason to be concerned about whether these aspects of the survey context are associated with differential item-level nonresponses. Specifically, parameter estimates may be biased if survey respondents differ in the decisions that they make about responding to survey items between the two media as a function of these contextual features. Given previous evidence of demographic differences in item-level nonresponse (e.g., Gruskin, Geiger, Gordon, & Ackerson, 2001), it appears useful to examine whether these effects differ between media. Similarly, practical, perceptual, and motivational factors may impact how respondents approach the task of answering questions in different survey administration media (e.g., Dillman, 2007), suggesting that item characteristic effects may also be important to examine. For example, prior research already indicates that, within survey administration media, questions elicit different levels of item-level non-response as a function of the sensitivity of the question’s content, whether the item is associated with branching, and the question or response formatting. Hence, an important extension of this research is to determine whether item-level nonresponse rates vary between survey administration media in order to allay potential concerns about such potential bias in the survey results. In our study, we focus on whether item position, format, and content vary between web- and paper-administered questionnaires.
To examine these issues, the present study addresses the following research questions:
Research Question 1 (Overall): Do overall item-level response rates differ between web-based versus paper-based instruments?
Research Question 2 (Demographic): Do item-level response rates vary across demographic groups on web-based versus paper-based instruments?
Research Question 3 (Item Position): Do item-level response rates vary by item position on the questionnaire on web-based versus paper-based instruments?
Research Question 4 (Item format): Do item-level response rates vary by item format on web-based versus paper-based instruments?
Research Question 5 (Item Content): Do item-level response rates vary by item content on web-based versus paper-based instruments?
- Top of page
- RésuméAbstractResumenZhaiYaoYo yak
Our results lead to several conclusions concerning populations of highly competent teachers responding to questionnaires that focus on their beliefs about teaching. First, the results relating to Research Question 1 (Overall) indicate that there are only small differences between the overall rates of item-level nonresponse when questionnaires are administered in a paper- versus a web-based medium to this population—only about 2% of the items were not answered in this study in each medium. These numbers are consistent with previous comparisons of overall item-level nonresponses between these two media (Mehta & Sivadas, 1995; Stanton, 1998; Tse, 1998) and are much lower than surveys focusing on more sensitive topics (Gruskin, Geiger, Gordon, & Ackerson, 2001; Kadushin, Reber, Saxe, & Livert, 1998; Wolfe, 2003). Similarly, our results indicate that a slightly higher percentage of paper respondents may answer all items and that the percentages in our study are about double those observed elsewhere (Bosnjak & Tuten, 2001), perhaps because of a combination of our selective sample of NBCTs and because of our incentive for responding.
Concerning Research Question 2 (Demographics), unlike research results that have focused on other populations, there were no large demographic differences with respect to item-level nonresponse rates for these accomplished teachers. Our study focuses on a fairly well-educated population, so it is not surprising that we observed no differences relating to the level of education of our teachers. Other studies that have observed education differences have tended to focus on populations that were more variable in terms of education level (Alvik, Haldorsen, & Lindemann, 2005; Craig & McCann, 1978; Gruskin, Geiger, Gordon, & Ackerson, 2001; Guadagnoli & Cleary, 1992; Kupek, 1998, 1999). Prior research concerning item-level nonresponse levels by gender has produced mixed results (Colsher & Wallace, 1989; Guadagnoli & Cleary, 1992; Messmer & Seymour, 1982). Our study adds to this body of literature indicating that, for populations of accomplished teachers, administration medium has no meaningfully large differential effects on item-level nonresponse rates for education or gender.
Concerning Research Question 3 (Item Position), the results of our exploratory analyses indicate a slightly higher tendency for web respondents to leave unanswered items toward the end of the questionnaire and to leave a greater proportion of the questionnaire unfinished than was the case for paper respondents. However, noncompletion rates were only slightly greater for web respondents. Prior research has not investigated the potential of differential effects of item position on item nonresponse between web- and paper-based questionnaires.
Concerning Research Question 4 (Item format), routing errors were more common on conditional items among paper respondents. In our study, this outcome may have been due to the positioning of a page break for two conditional items (an important practical design consideration for paper questionnaires), which is consistent with other studies of branching errors on paper-based questionnaires (Messmer & Seymour, 1982). Our results suggest that, because our web-based questionnaire did not utilize features that could have potentially reduced the number of branching errors (e.g., via explicit redirection after responding to a trigger question), the use of a single scrolled page may decrease the rate of item-level nonresponse due to routing errors. We also found no statistically significant differences between paper and web respondents with respect to the likelihood of providing respondent-supplied text for items requesting such optional information.
Results relating to Research Question 5 (Item Content) indicate that web respondents were considerably less likely to respond to numerical fill-in-the-blank items (e.g., items that require the respondent to identify the number of students or teachers in the school belonging to a particular group). We believe this occurred because web respondents chose to leave these items blank when the answer was “zero,” and the fact that the reported averages for these items were always greater for web-respondents supports that notion. An alternative explanation could be that these items place a higher cognitive demand on the respondent, and some feature of the response medium increased the probability that web-based respondents would choose to leave these items unanswered (e.g., the paper version could be carried by the respondent to a physical location to look up the necessary information while those on computer would be more inconvenienced by finding that information). On attitudinal items, item-level nonresponse rates were comparable for items asking for different types of judgments (e.g., agreement with a statement vs. importance of a statement) and for items forcing a choice versus items allowing for “not applicable” or “do not know” responses—a result that is consistent with prior research comparing item-level nonresponses between paper and web media (Smyth, Dillman, Christian, & Stern, 2006).
There are important limitations that should be kept in mind when interpreting these results. First, although we attempted to make the two versions of the questionnaire as comparable as possible, the two versions neither perfectly replicate one another nor are perfect implementations of the questionnaire in their respective media. One potentially important difference relates to the space provided for several of the items: The paper version involved a wider item column than the web version, and thus the items carried over to a second or third line less often for the paper version. This created a difference in item appearance, with the web version formatting creating greater density of text, which may have impacted item nonresponse. For instance, this greater density may have reduced respondent willingness to read and answer each item, leading to the slightly higher item-level nonresponse rate observed for the web survey. On the other hand, this increase in the number of lines per item meant the response options for adjacent items were separated more in the web version, potentially reducing the likelihood of accidentally skipping an item for the web survey (given that it would be easier to detect a missing response). Thus, the effect of this difference is not entirely clear, but it should be considered in interpreting the current results. In addition, conditional items were presented somewhat differently for the two versions. In the paper version, arrows were used to direct respondents to subitems (and in one case no specific guidance was given following a “No” response); in the web version, short written descriptions were used. This difference may have also contributed to the conditional item results, in addition to the page break issue discussed above.
The second issue concerns the appropriate population for generalizing these findings. Our study seeks to generalize its results to a population of accomplished teachers—individuals who are fairly highly educated, experienced in developing assessments, and likely to be computer-savvy. These teachers represent a fairly homogenous group with respect to gender and race. Thus, perhaps it is no surprise that demographic differences between media were relatively minor in our study, although more generally speaking, differences have been found to exist in the demographic characteristics of paper- versus web-based unit-level respondents (Yun & Trumbo, 2000). But given the homogeneous population we were studying, there perhaps were other individual differences variables that would indicate which teachers decided to respond. For instance, perhaps those with more computer experience and access to a computer would be more likely to respond to a web survey request, and those teachers who are more conscientious would be more likely to respond to a mail survey request. Third, the information requested in our questionnaire contained relatively innocuous questions, leaving unaddressed the issue of the degree to which requests for sensitive information may elicit differential item-level nonresponse rates across survey administration media. Fourth, most of the results presented here are exploratory—the comparisons are based on questions from an existing questionnaire rather than on items that were designed to address group differences hypothesized a priori based on sample and item characteristics (in fact, substantive theory on group-by-measure interactions is completely lacking). As a result, the implications that we present here are tentative, but are guided by a useful framework that could be helpful in conducting future research of this nature.
Within these limitations, our results suggest several potential implications for survey researchers who desire to utilize web-based questionnaires that focus on populations and attitudinal measures similar to those examined in this study. First, existing concerns regarding the comparability of item-level response rates between paper- and web-based questionnaires, an important indicator of the reliability of measures made in these two survey media, may be unwarranted. For example, our study suggests that there are only small differences in overall item-level nonresponse rates between web- and paper-based surveys and that these rates are consistent across demographic groups within the relatively educated, professionally accomplished, and computer-savvy population of teachers. Similarly, noncompletion rates may be comparable between these two survey administration media.
However, a second implication may be that the choice of administration medium has a differential impact on respondent motivation. For example, our study suggests that there may be differences in the number of omitted items for surveys that were not fully completed. Not only are web respondents less likely to complete all items than paper respondents, but when web respondents lose motivation, they may be more likely to leave larger portions of the questionnaire blank. In our study, because we utilized a single “scrolled” page displaying the items, the observed nonresponse patterns may have been due to the fact that respondents could not “save” work and return to it at a later time, leading web respondents to be more likely to abandon the questionnaire. However, it is unclear whether presenting the items on separate “pages” allowing for a partial save of the data would improve the situation because respondents could potentially abandon the survey due to uncertainty of how many questions remain to be answered. Future research could modify web-based questionnaires to manipulate some of these factors that might affect respondent motivation. Designers of web-based survey instruments may need to take these issues into account during instrument development.
A third implication of our results concerns the formatting of routing instructions for conditional questions. Web-based questionnaires may make it easier to cue respondents to the proper routing through conditional questions. In this study, this was true even though explicit controls were not programmed into the web-based questionnaire (e.g., “pop-up questions” or using text “graying” to conceal questions that should not be answered). Survey instrument developers may want to take advantage of the higher levels of flexibility concerning coloring and placement of arrows and text on the web screen in order to help respondents more easily visually maneuver through conditional questions in the web medium so that fewer routing errors are made.
The fourth implication of our results concerns fill-in-the-blank questions that request counted estimates. One of the most apparent outcomes of this study is the fact that numerical fill-in-the-blank items produced a considerably higher item-level nonresponse rate on web-based questionnaires than on paper-based questionnaires. We believe that this is because web respondents left those blanks empty when the implied answer was “zero” while paper respondents wrote in the numeral value for zeros. Hence, web formats with numerical fill-in-the-blank may need to provide a direct prompt for the individual to enter zero when that is the intended response to avoid treating these values as item-level nonresponses, thus upwardly biasing the resulting numerical estimates.