SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. E-mail Compared to “Snail” Mail
  5. The Web Compared to… the Ones That Got Away?
  6. Surveying the Web Presence Provider Community
  7. Results
  8. Discussion
  9. Acknowledgments
  10. References

At any given moment there are thousands of surveys and polls being conducted on the web, yet surprisingly little scholarly research is reported about this new technique. After a summary review of the comparative literature on e-mail and “snail mail” and a more extensive review of research involving web-based methods, this article contrasts e-mail and web-based survey techniques used in an ongoing study of the web presence provider industry. Practical issues of web-surveying methods are highlighted, such as programming pitfalls, sample-building, and incentives.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. E-mail Compared to “Snail” Mail
  5. The Web Compared to… the Ones That Got Away?
  6. Surveying the Web Presence Provider Community
  7. Results
  8. Discussion
  9. Acknowledgments
  10. References

The self-administered survey questionnaire has long been an economic alternative to labor-intensive and expensive face-to-face and telephone interviews. With the widespread adoption of e-mail among corporate, scholastic, and government populations, and to a somewhat lesser extent the general computing public, dissemination of survey materials among such populations has never been so easy or cheap. Another alternative gaining in popularity is the fill-in-form survey which uses the hypertext markup language (HTML) and common-gateway-interface (cgi) programming of the World Wide Web to construct, format, and administer questionnaires to web users. This paper contrasts the two electronic survey techniques in light of methodological issues which have arisen during a work-in-progress: an extensive survey of the web “presence” provider industry.

At best guess, thousands of ongoing, informal polls are liberally sprinkled among millions of web pages. There are probably an equal number of marketing surveys, but results of these are closely held as proprietary information generated for paying clients. Little of the methodological information gathered from these surveys finds its way into the scholarly literature, still less into the public domain, although it is easy to spot the tantalizing nuggets of this massive proprietary research effort in press releases and sales literature. Consequently, communication researchers may be ill-informed about the advantages and liabilities of web survey methodology relative to other, more familiar techniques. Thus, the focus here is primarily on this newer method, and that is limited largely to response quantity rather than quality, although many of the researchers cited are analyzing response times and quality.

E-mail Compared to “Snail” Mail

  1. Top of page
  2. Abstract
  3. Introduction
  4. E-mail Compared to “Snail” Mail
  5. The Web Compared to… the Ones That Got Away?
  6. Surveying the Web Presence Provider Community
  7. Results
  8. Discussion
  9. Acknowledgments
  10. References

E-mail is rapidly becoming an indispensable tool for gathering detailed information on selected populations, despite some early anomalies. For example, [Parker (1992)] reported a 68 percent response rate on a survey administered electronically to AT&T employees, contrasted with a 38 percent return for the same survey administered via postal mail. [Schuldt and Totten (1994)] later found almost the exact reverse of the 30-point spread (56.5% mail and 19.3% e-mail) in a national sample of marketing and MIS faculty. Similarly, [Tse et al. (1995)] reported much lower e-mail response rates for Hong Kong University faculty and administrative staff (27% vs. 6%). On the other hand, the early work of [Rosenfeld, Booth-Kewley, and Edwards (1993)] revealed almost no differences in response rates among Navy personnel. More recently, [Mehta and Sivadas (1995)] and [Bachmann, Elfrink, and Vazzana (1996)] indicate that their e-mail and postal mail response rates are comparable, and that the gap is narrowing between e-mail and traditional “snail” mail techniques. [Kittleson's (1995)] work is especially notable in that he administered two different, postcard-length surveys to the same sample via e-mail and postal mail; however, he used only a two-group design (both groups received both surveys delivered the same way), so it is theoretically possible to attribute at least some of the disparity in response rates (e-mail = 28.1%, mail = 76.5%) to the survey content rather than the medium of delivery.

With the caveat that one's target population must be technologically savvy enough to use it, persuasive arguments for using e-mail include extreme cost reduction and quick turnaround time, facilitative interaction between survey authors and respondents, collapsed geographic boundaries, user-convenience, and, arguably, more candid and extensive response quality ([Bachmann et al., 1996]; [Mehta & Sivadas, 1995]; [Kiesler & Sproull, 1986]; [Sproull, 1986]). Aside from potential response-rate disparity, disadvantages of using the newer medium include lack of cosmetic features, such as precise layout and font styles, difficulties in including monetary or other tangible incentives, and a still rather broad and unpredictable range of technological ability and extent of use in otherwise fairly homogenous populations.

The Web Compared to… the Ones That Got Away?

  1. Top of page
  2. Abstract
  3. Introduction
  4. E-mail Compared to “Snail” Mail
  5. The Web Compared to… the Ones That Got Away?
  6. Surveying the Web Presence Provider Community
  7. Results
  8. Discussion
  9. Acknowledgments
  10. References

Relatively little scholarly research has been done on the newer technique of administering surveys via the web using fill-in forms, although anecdotal accounts include astronomical results of “thousands of responses” within a few hours or days ([Gjestland, 1996]; [Palmquist & Stueve, 1996]). Perhaps the most well-known and ubiquitous web survey is that of Georgia Tech's Graphics, Visualization, and Usability Center (GVU), now in its 7th incarnation and still adhering to its initial purpose of identifying Internet user demographics, behaviors, and attitudes towards technology and commercial activity on the Internet ([Kehoe & Pitkow, 1996]; [Pitkow & Recker, 1995]). A pioneer in web-based survey design and implementation, the GVU Web Survey uses self-selection sampling, and relies on repeat participation via e-mail notification to past survey respondents. Endorsed by the web indexing community (e.g., Alta Vista, Excite, Lycos, Yahoo, etc.) and the Internet publishing industry (e.g. newspapers, trade magazines, etc.) the biannual GVU surveys are heavily promoted on the web. And each new version is heralded throughout Usenet and among Internet mailing lists by an extensive volunteer network of users and moderators. Kehoe and Pitkow (1996) report over 55,000 respondents for the first five surveys, and it may be inferred that the rate of new respondents far exceeds the rate of attrition from survey to survey (attrition rate ranged from 4.54% to 12.58% for the 3rd survey). Interestingly enough, considerably more of GVU's focus is turning toward those who do not fill out, complete, or return for new surveys, a laudable effort which should vastly improve our methodological knowledge base.

Other Internet-demographic and trend-oriented researchers have combined data-gathering techniques. FIND/SVP, for example, reportedly deploys focus group, random-digit-dialing (RDD), and on-line questionnaire techniques to gather information on both users and non-users, but no evidence of on-line survey administration is available at their web site. The self-proclaimed, “only statistically defensible” Internet-user survey administered by O'Reilly & Associates and Trish Information Services (1995) sampled in three phases: 1) questionnaires mailed to 2,052 subscribers of (then) O'Reilly's GNN Magazine, 2) telephone interviews of 1,000 U.S. Internet users identified via RDD calls, and 3) telephone interviews of 500 RDD-identified U.S. commercial on-line service users. O'Reilly's on-line summary of the survey does not disclose the response rate of the mail survey or the total number of random calls made in the second and third phases of their study, but according to their sales representative, Sandy Torre, 818 responses were received from the mail-survey, for a response rate of 40 percent. Nielsen Media Research (of TV ratings fame), in conjunction with CommerceNet, a consortium of companies interested in Internet trends and demographics, still relies on its well-traveled RDD methodology to survey (and re-survey) thousands of North American households on media technology use.

Another survey summary describes simultaneous use of e-mail and web (HTML) techniques. On his web page about the Third MIDS Internet Demographic Survey, [Quarterman (1996)] notes that user comments motivated him to provide his questionnaire in HTML form, and that “many of the responses were received through it,” but he does he not state the number of responses attributed to each medium of delivery, merely that “…we checked to see if there were any significant differences in the nature of the results received on the ASCII vs. the HTML questionnaires, and there were not.” The full report, available for purchase from MIDS, likely contains more details.

One of the few publicly available analyses of web versus e-mail techniques is contained in [Patrick, Black, and Whalen's (1995)] demographic and satisfaction survey of 1073 users of a Community FreeNet in Ottawa, Canada. Patrick et al. offered e-mail and HTML alternatives, and 95 percent of the 1,073 participants elected to use the newly installed Lynx browser rather than request an e-mail version of the 74-item survey. Length was indeed an issue, according to Patrick et al., but it is not clear whether length was a factor in more participants' choice of the form-based option. They also report that some users had difficulty with the new web interface, due to unfamiliarity and terminal emulation problems. What impact this had on mode of participation, or participation in general, is unspecified.

Finally, [Comley (1996)] of Simon Godfrey Associates, a market research firm based in the United Kingdom, offers a seminal comparison of multiple surveying methods. SGA used a publisher-provided list of an Internet-related magazine's subscribers and e-mail contacts. The test groups are summarized below:

  • 1
    E-mail, with pre-notification, surveys sent on receipt of consent to participate
  • Response: 45% of those who consented to participate
  • 2
    Postal, paper version, S.A.S.E. included
  • Response: 15.4%
  • 3
    Postal + Web option, same as 2, but cover letter included URL for survey at SGA web site
  • Postal response: 16%
  • Website response: 1%

One could immediately wish that Comley had rounded off the study with two more groups, especially given his rather large sampling frame of over 2,000 individuals. Missing is an e-mail group that received the survey without prior notification or consent, but length was (again) an issue and perhaps this guided the design. Also missing was an e-mail group receiving simply a request to participate and a pointer to the web survey. Comley notes in his discussion that SGA has had considerable success with this method in other projects. Perhaps in the interest of saving the client money, it was felt to be redundant, but the oversight does seem to leave a gaping hole in an otherwise laudable study purporting to examine “how early adopters of new technology were using the Internet.”

Surveying the Web Presence Provider Community

  1. Top of page
  2. Abstract
  3. Introduction
  4. E-mail Compared to “Snail” Mail
  5. The Web Compared to… the Ones That Got Away?
  6. Surveying the Web Presence Provider Community
  7. Results
  8. Discussion
  9. Acknowledgments
  10. References

First, let me define and distinguish some terminology. I define “web presence provider” (WPP) as an entity who (or which) takes an active role in developing, maintaining, or promoting content on the web. This role is distinguished from that of Internet access/service providers (IAP/ISP), i.e., those who merely provide via appropriate hardware, telephony, and server configuration the wherewithal to access the Internet and store one's “presence materiel” there. Presence materiel consists of everything from one's domain name and e-mail address to the actual files and scripts used to array oneself as a virtual entity in cyberspace. Because “Internet Presence Provider” is often used synonymously with IAP and ISP, I chose to identify my units of analysis as “web presence providers” although some of their content and promotion activity spills over into other Internet domains (e.g., Usenet news, e-mail mailing lists, chat servers). In brief, I am interested in the professional “webmaster” industry.

One of the primary ways to gain insight into the business of providing presence is to lookat how WPPs present themselves, with a focus on the range and extent of services offered, and how they see themselves in relation to the more traditional “presence-providing” paradigms. My survey is a broadbrush effort to determine what characterizes the industry and what distinguishes WPPs from one another. Furthermore, in order to gauge the changes which have occurred, those which mark the growth of the industry from spare-time web page building to the presence architects of today, I ask WPPs about changes in their services and staff, and about their professional networking activities and affiliations.

The Questionnaires

Two surveys were constructed, one for independents who perform presence services for organizations and individuals on a contract basis (Survey I), and the second for organizational employees whose web site development and maintenance activities are (increasingly but not always) included in their formal job descriptions (Survey II). These consisted of open-ended, multiple-choice-response, and fill-in-the-blank questions. They differed in length: Survey I consisted of 66 questions, Survey II of 56 questions, but with one exception (Survey II respondents were not asked about problems with individual clients) both questionnaires contained the same open-ended questions, slightly reworded to fit the different contexts. Otherwise, the questionnaires differed substantively in only a few key question areas. Several questions relating to clients and general business operations were omitted from Survey II, and Survey II contained a few questions addressing organizational practices and demographics that were omitted from Survey I.

The point I wish to emphasize here is that these questionnaires are extraordinarily lengthy, even by the broad standards of exploratory research. Furthermore, the open-ended questions invited studied and often lengthy responses despite urgings to be brief. To understand more fully the expository exercise these questions entailed, the open-ended questions from Survey I are presented below:

35. What are the top three problems you encounter in working with your organizational clients to develop a dynamic online presence for the organization? Please be very specific (feel free to vent).

  • 1
    []
  • 2
    []
  • 3
    []
    Others you'd like to add? []

36. What are the top three problems you encounter in working with your individual/family clients to develop a dynamic online presence for the individual/family?

  • 1
    []
  • 2
    []
  • 3
    []
    Others you'd like to add? []

40. List the top 3 to 5 elements of a web site which, in your opinion, reflect its dynamism. Please be very specific, for example say “internal navigation” or “interconnectedness with other sites” rather than just “links.”

  • []

  • []

  • []

  • []

  • []

41. List the top 3 to 5 elements of a web site which, in your opinion, reflect its interactivity. Please be very specific, for example say “visitor's ability to share comments with other visitors” rather than just “feedback.”

  • []

  • []

  • []

  • []

  • []

43. Many writers have said recently, “It used to be enough to just HAVE an online presence. Now, simply being there (on the Internet) isn't enough.” Please complete one of the following statements in a few short phrases or sentences, typing within the brackets. Your answer should reflect whether and how you think the nature of internet presence has changed in the past two years.

  • I agree because []

  • I disagree because []

44. Assume you are an organization looking for a Web Presence Provider. The first thing you'll notice is there are thousands vying for your business. What are the top 5 criteria you would use for evaluating and distinguishing among them? Your answers would probably reflect the major selling points you use to set your WPP business apart from your competitors.

  • 1
    []
  • 2
    []
  • 3
    []
  • 4
    []
  • 5
    []

45. What is your strongest argument for why organizations should(or should not) outsource their presence development and maintenance needs rather than hire their own in-house web masters and teams?

  • Begin here: []

46. Please define or describe “Presence” in a few brief phrases or sentences. You may cut and paste an existing definition or description if it adequately expresses your definition or description, but if you quote someone else I'd appreciate knowing whom you are quoting, if possible.

  • Begin here: []

Each survey was prefaced by an introductory note in which I introduced myself and my academic affiliation, described the purpose of the research, and offered two incentives for completing the survey. The first incentive offered was that completing the survey might be useful as a “means of defining your current professional perspectives and activities.” The second incentive was the opportunity to be listed (with or without a company-logo graphic or banner) on a web page intended to be linked to the executive summary of the research. I promised that the page would be suitably announced and promoted, noting that the site was home to well-known pages regularly visited by two disparate audiences: real estate agents/buyers and genealogy researchers. Respondents were then asked if they wished to be included. At the end of the questionnaire, respondents were also asked a) whether they could be contacted for further information, b) whether they wished to be sent an executive summary, and

  • c) …was it useful to you in clarifying or defining your professional perspectives and operations?

  • () YES!

  • () Well, it wasn't entirely a waste of time.

  • () NO! A total waste of time.

HTML-izing the Survey

Both surveys were reformatted in HTML (a fill-out form) for web distribution. An introductory page was created where respondents could self-select which questionnaire to fill out. This URL was later included in my e-mail signature file, which was selectively appended to my outgoing e-mail messages. A link to the survey site was also included on my web home page. Because of the undue length of the survey, each questionnaire was split into three distinct parts which could be filled out and submitted separately, at the respondent's convenience. A text box was provided at the beginning of parts II and III for respondents to type in identifying information so that separate form submissions could later be concatenated. The typical manner in which such forms work is as follows:

  • Data is entered in each of the form fields, which are uniquely named, much like variables in a statistical data file must have unique names.
  • When the “Submit” button is clicked, the cgi script referenced in the <FORM action=”“> tag is executed on the server and begins to receive and process the data according to its instructions.
  • A typical form-handler script parses the submitted data into an array of keyword-value pairs. This array consists of each field's unique name and the value entered for that field.
  • Once the data is parsed into a readable array, the script calls the server's system mailer and sends the array to a designated e-mail address.
  • Finally, the script sends a default “Thank You” page to the browser, and the respondent continues along his or her way.

Quite literally then, one may simply save the resulting e-mail document as text without the headers, delete the keywords, append subsequent files in the same manner, and thus create a machine readable data file ready for statistical analysis. This easy method obviously streamlines the data collection and entry process, but it does not eliminate the task of careful visual screening and cleaning of the data. One missing value in a data field could throw the entire record off. There are two ways to deal with this potential disaster. The first is to have the form return a default (hence “missing”) value if no value is entered by the respondent. This effectively serves as a place holder for that variable's spot in the data file. The second is to require that all fields be filled out before the form data is processed to completion on the server. This is accomplished by a two-step method of designating fields as “required” in the HTML coding of the form, and then including check-value routines in the form-processing script. Note that this routine does not check for validity of responses, merely their presence. More sophisticated routines are available. For example there are those which check that a zip code, phone number, or credit card number is entered in a recognizable format, and conceivably routines may be written which check for precise values. The latter method could streamline even the initial analytical process by categorizing response data according to key values, but the myriad possibilities of scripting are beyond the scope of this paper.

Disseminating the Survey Via E-Mail and Web

After pretesting with five volunteers (4 independent designers and one organizational webmaster), the e-mail questionnaires were judged ready for distribution. A sample of 300 WPPs, selected from a directory listing of the international Web Consultants Association, was divided into two listings of 150 each using random number start and interval-selection. Group 1 received the e-mailed survey and Group 2 received a call for participation, requesting permission to send the survey. Participants were sent Survey I or II, depending on the consultants's profile.

Launching the web-based survey was not so simple. As mentioned above, the typical web form-handling script includes a call to execute the server system's mailer program and e-mails the form data to an address specified in the script. Such mailers are resident on (i.e., built into) X/UNIX and LINUX systems; however, they are entirely missing on Microsoft operating systems (on which the popular NT web servers are run). Administrators running an NT server must install and configure a separate suite of applications to run e-mail from the server. Packages such as Winmail are readily available and easily configured, but (as in my case) the sys-admin may elect not to use them.

Another alternative presented itself in [Lemay, Murphy, and Smith's (1996)] Creating Commercial Web Pages. In their chapter on web-based surveys and questionnaires, they describe a simple method of receiving form data using “mailto:e-mail@your.address” in the form's ACTION=”“parameter (rather than a call to a cgi script). With this method, the form's field values are simply piped directly from the form and e-mailed to the address specified as a stream of URL-encoded data. Lemay et al. note that the mailto: function “is supported by browsers as an action in the <FORM> tag created at the beginning of any form” (p. 141), but that a decoding utility is needed to translate the URL-encoded data into readable form. They also note that using this method precludes any acknowledgment that the form data has been sent (i.e., no “Thank you” page is generated) so they include a simple javascript which causes a window to pop up when the “SUBMIT” button is clicked. One may alter the javascript to include any appropriate “thank you” text.

After obtaining the appropriate decoding utility, and after testing it with the mailto-formatted questionnaires, I was ready to launch the second sampling phase of my survey of web presence providers. A call-for-participants which included the URL for the survey site was submitted to the following web-content-oriented mailing lists, selected via a topic-search of the LISZT mailing list index:

The exact numbers of subscribers to these lists at the time of exposure to my call for participants is unknown, but where possible, conservative estimates were derived from mailing-list information files and moderators. Of course, there is no way to tell precisely how many subscribers actually received (much less read) my CFP. At any given time, mailing-list subscribers are changing their addresses (rendering their subscriptions invalid), voluntarily not receiving list e-mail, actively filtering their e-mail (either by message size or by header information), or just not reading their e-mail for various reasons. Based on these figures, I can estimate that roughly 8,000 (± 15%) subscribers received my CFP via e-mail list distribution.

Responses started arriving almost immediately, along with some disconcerting information from Microsoft Internet Explorer (MSIE) browser users: the mailto: form parameter is not supported by MSIE browsers (easily the third most popular browser behind Netscape Navigator and AOL's browser). I was faced with an unknown portion of my target population unable to see anything but a blank e-mail window after pressing “SUBMIT.”

Text versions of the questionnaire were immediately uploaded to the survey site, along with a help file explaining the browser incompatibility problem (some Lynx browsers also did not support the mailto: form action) and a statement that I hoped to have the problem resolved quickly. Two people requested that e-mail questionnaires be mailed directly to them.

To locate a script which would save form submissions to a file in my directory I posted an inquiry to ADV-HTML. Within two days of posting my query I had a workable cgi script which performed exactly like a form2mail script, except that instead of e-mailing me the data, it wrote the submitted form data to a unique file which was saved in my directory on the server. This was immediately tested and implemented, and at this time I activated my e-mail signature file with the survey site URL prominently displayed. For example, when I posted the requisite summary to ADV-HTML regarding the results of my scripting query the .sig file was appended.

Although I could have posted another call for participants, noting that the form implementation problem was resolved, I chose a different route in order to re-establish my credibility as participant observer of the industry. For the next several weeks, I sought “windows of opportunity” to respond publicly and privately to issues discussed on each of the mailing lists which had received my initial CFP. The rationale behind this was to get exposure for my survey indirectly via its “plug” in my signature file. I was able to use this tactic on all but the legal-webmasters and MUSWEB-L lists. Furthermore, I expanded my participation to two other mailing lists which catered to the web-developer crowd: http://I-SALES@mmgco.com (a moderated list primarily for Internet marketers, with over 5,000 subscribers) and http://spiderwoman@amazoncity.com (a list supporting women involved in web development, which almost immediately suffered a server crash and was “down” for most of the data gathering period). All private messages to web-developers were logged in order to compare the list of recipients against the list of survey respondents at a later time.

Finally, because I believed that survey length was a critical issue, two follow-up polls were constructed for comparative purposes. The first was a two-question poll administered to three of the mailing-lists on which I was active, and asked about readers' preferred job titles and whether they were independent contractors or organizational employees. Comments were solicited regarding web-development job titles. The results of the poll were immediately tallied and formatted as a web page: “Coming to terms with what we do: Web Mastery and Mystery”. A poll summary and a pointer to the poll's URL were posted to each of the lists on which participants were polled. The second poll was a single question asking about the web consultant's networking behaviors, and was e-mailed to the same list of respondents who received the e-mailed survey and call for participation. Again, appended to every message was my signature file with a pointer to the survey site.

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. E-mail Compared to “Snail” Mail
  5. The Web Compared to… the Ones That Got Away?
  6. Surveying the Web Presence Provider Community
  7. Results
  8. Discussion
  9. Acknowledgments
  10. References

E-mail Results and Observations

The e-mail distribution efforts, summarized by Table 1, were more frustrating than fruitful. Of the 150 e-mailed copies of the survey, 12 “bounced.” In other words, they were returned for failure to find the addressee because the address was either incorrect or no longer in service. In two cases I received autoresponse messages that the addressee was not currently receiving e-mail. From the remaining 136 questionnaires that presumably reached their target I received 11 completed surveys, for an actual response rate of 8 percent. One person promptly accused me of spamming him and copied his message to my ISP. Twelve people simply replied that the survey was “too long.” One individual requested an HTML version, and subsequently responded to the web version of the survey. The 150 calls for participation generated 4 delivery failures and 42 requests for surveys, for an initial response rate of 29.7 percent. Twenty of these surveys were returned, yielding an encouraging 47.6 percent return rate, but a rather disappointing overall response rate of 13.3 percent for Group 2. Still, given the length and laborious nature of the questionnaires, these results compare favorably with some results reported in the literature. In both groups, many people were extremely interested in learning how I acquired their names and e-mail addresses.

Table 1. 
ModeNDelivery FailureSurvey RequestsValid ResponsesResponse Rate
Direct E-mail15014 118%
Call for E-mail Participation1504422013.3%
Follow-up Poll30038 10239%

The follow-up poll was clearly identified as such, and asked a multiple-choice-response question about networking with other web developers. The same address list was used for the poll, unedited in case of simple delivery errors in the first instance of use and because I wanted to compare the intervening attrition. The same 16 addresses were inoperative, along with 22 others, leaving a total possible response pool of 262 (no “away” messages were generated). The poll generated 102 valid responses, for a response rate of 39 percent. Nine of these respondents, presumably noting the survey URL in my signature file, went on to complete questionnaires at the survey site.

It would seem that length is indeed an issue, at least for e-mailed questionnaires. Language was apparently an issue in only one case, and this was made obvious by the note accompanying the survey response. This individual simply left all open-ended questions and fill-in-the-blank questions empty and, because I was concerned that many other questions may have been misunderstood, the response set was eliminated from the data set and analysis. It is not known how many other potential respondents may have failed to participate due to language problems; fifteen percent of the addresses in the e-mail/poll sample were from countries in which the primary language was not English.

Web Survey Results and Observations

My original methodological intent in this project was to contrast e-mail and web survey administration techniques, but the form submission and browser incompatibility problem raised real concerns about whether I could gather enough data to address adequately the project's primary objective of surveying the web presence provider industry. Hence, my concerted efforts to publicize the survey beyond the initial calls for participants and followup polls somewhat muddies the methodological analysis of the web-based phase. Still, the process and results are both illuminating and encouraging in the face of virtually non-existent empirical research in web survey administration.

Given the conventional wisdom that a large if not majority proportion of survey responses are submitted within 24–48 hours of exposure, it is likely that the incompatibility problem had a significant effect on the number of responses received in this initial period; however, it is obviously impossible to measure what did not occur. The initial call produced 99 more-or-less completed surveys (in order to be retained, the first two parts of a survey needed to be complete, with the exception of the open-ended questions). Just as a reminder about the significance of the browser incompatibility problem, these 99 respondents consisted almost exclusively of Netscape Navigator users (two indicated they had successfully used Lynx).

Other, rather minor problems cropped up during the initial phase. It soon became obvious that the javascript pop-up “thank you” message was not working (although it worked fine in testing) and people were not receiving acknowledgment that their data had been submitted. It is not known whether this was because most people had javascript disabled or because of some problem with the javascript I used. In any case, I typically received two or more submissions of the same data, and this redundant information had to be deleted. Ordinarily, this would not have been terribly inconvenient, except that I had included “SUBMIT” buttons at the end of each part of the questionnaire. Thus there was no way of verifying whether information was indeed redundant or instead a submission of more data without visually screening each submission after it had first been “translated” by the URL-decoding utility. Additionally, there were 20 instances of incomplete transmission, which may have resulted from some failure of my server, the respondent's server, or somewhere in between. Luckily several of these people re-submitted data, which allowed me to construct complete response sets. One respondent reported that every other question in Part I of his questionnaire was missing, and that he only discovered this when he scrolled up to review his responses. This remains a mystery.

Once the cgi form-handling script was implemented, 53 more-or-less completed questionnaires were received, indicating a trend of diminishing returns despite my efforts to indirectly publicize the survey further by posting to the lists with the survey's URL in my signature file. Worth mentioning, however, is the observation that the number of responses spiked each time I posted to a list, and the followup poll produced a comparative frenzy of survey responses (approximately 30–35). The followup poll, you will recall, was posted to only 3 of the lists on which I was actively fishing for participants, and after two days, was officially closed with 86 valid responses. The web-related results are summarized in Table 2.

Table 2. 
ModeNull or Incomplete ResponsesValid Responses
Mailto: Action4299
Web cgi Script2053
From E-mail sample Follow-up Poll19
Total63161
Follow-up Poll 86

Did the promised incentives to complete the survey work? The listing on a “thank you” page linked to an executive summary was probably not a great incentive although 149 individuals and organizations wished to be included. Most, if not all, of the (as yet) incomplete survey respondents also desired inclusion, and a substantial portion of finishers (42) declined. More telling was the other incentive – albeit after the fact and therefore questionable as motivation to completion – that the survey may be of some use in clarifying professional perspectives and practices. Of the 143 people who responded to the question of how useful the survey was, only 5 indicated that it was a total waste of time. Sixty-six indicated that the survey was of some use, and 72 respondents gave it an unequivocal thumbs-up.

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. E-mail Compared to “Snail” Mail
  5. The Web Compared to… the Ones That Got Away?
  6. Surveying the Web Presence Provider Community
  7. Results
  8. Discussion
  9. Acknowledgments
  10. References

“If studies based on unrepresentative samples were excluded from social science research, whole sections of library shelves would begin to look like supermarkets in the former Soviet Union.”

–Dana Ward, Professor of Political Studies, Pitzer College

Perhaps the most critical problem with Internet-based research is the practical impossibility of probability sampling, that one can only tentatively generalize to a very specific population, if at all. But as Professor Ward so aptly implies above, we do this all the time. As we acquire more sophisticated means of tracking users and continue to learn more about their demographics, generalizability to well-studied segments of the overall population will become less of an issue. The web is obviously a viable mode of survey administration. Yet the lack of standardization among operating systems, servers, and browsers creates a challenging milieu in which the researcher must be technologically savvy as well as methodologically sound. While I am by no means a “propeller-head” I have worked extensively with and followed developments in Internet technology since before the web went “graphical” in 1993. Still I found it difficult to anticipate every problem.

The primary problem with web-administered surveys is there is no precise and reliable means of determining response rate. One could install a “hit” counter which tracks the number of times a page is called from the server, but these counters do not distinguish repeat from unique visitors, nor do they reflect whether a page is allowed to fully load in the browser. Perhaps the most valid means of determining an approximate visitor count is logfile analysis, whereby server activity logs are examined for complete page loads and calls from unique DNS identifiers (i.e., effectively discerning redundant calls from the same browser). However, even these are not perfect because many Internet users are assigned IP addresses dynamically, which means their DNS identifiers differ from dialup session to dialup session. Certainly, the same person can access a page at different times under two or more completely unique identifiers. My logfiles from the data collection period were unfortunately deleted by my ISP, but it is reasonable to expect that any future web-based research demands corresponding logfile analysis in order to address validity issues.

I am of two minds on the issue of the obtained response rate. Based on the potential 8000 participants from the mailing lists on which my CFP appeared, and the 300 person e-mail sample, one could call 2.3 percent (192) a truly dismal catch compared to the lofty “one-ton wonders” reported here and elsewhere. However, I feel I have more than enough data to take the baseline pulse of the industry, and it should be noted that two-thirds of my respondents represented staffs of two or more people. While only one staff member filled out my survey, all probably subscribe to the same mailing lists.

The new technology offers a spate of new problems layered over the old, not the least of which is sheer competition from marketers, journalists, and other researchers. The increasing size and demographic extent of the Internet population notwithstanding, the ability of people to deal with increasing information overload remains fairly low. While e-mail is a wonderful tool for impromptu polling on timely issues, and an extremely useful tool for building a potential sample, its utility for anything more is increasingly questionable in the age of “infoglut” and pervasive e-mail spamming by unscrupulous marketers. An aura of suspicion often surrounds any stranger-to-stranger communication in cyberspace, even when the declared topic is of mutual interest. Many of my respondents indicated that they did indeed verify my identity and credentials before participating in either the polls or the survey. The true extent of the age of suspicion remains to be seen and its impact on research measured.

Netiquette proscribes e-mailing large surveys unannounced, but this practice, I would argue, is still valid for the purposes of survey design and methodological research. Survey length has been an issue for most of the researchers cited, and while we know there is such a thing as “too long” we do not yet know its dimensions. Is it media-dependent? The comparative use of polls in this study suggest that, for e-mail, length is a powerful factor in likelihood of response. Length may be a different perceptual concept on the web because it is more easily compensated for by design.

Many participants took advantage of the built-in “coffee breaks” offered via the multiple “SUBMIT” opportunities, and I am convinced that this is good practice when constructing longer surveys. Certainly its utility from both participant and researcher perspectives is worth further exploration. My poor adaptation is modeled after the extensive Java applet scripting behind the GVU surveys, a programming marvel which allows for longitudinal analysis over several surveys, and fairly complicated rules-based questioning procedures (e.g., adaptive follow-up and skip questioning). Web survey development software is increasingly available, but varies wildly in terms of price, function, and server compatibility. A thorough and ongoing review of such software is warranted, because obviously not every researcher will have access to custom programming. One could readily predict that such a tool will soon be in every survey researcher's arsenal, along with or even instead of analytical tools like SPSS, because some of the higher-end applications are already incorporating database functions, statistical analysis modules, and dynamic report-generating features.

Inducements and incentives for Internet research participation relative to traditional motivators deserve further scrutiny and systematic review. Although the question did not pertain to my target population, I wonder how great a role technological novelty plays in participation. Are “newbies” more inclined to venture into research spaces and leave their mark? Or is the increasingly steep learning curve too overwhelming until a certain level of sophistication is achieved? Has the Internet population of “old salts” been “over-fished?” And finally, is the “quid pro quo” effect undergoing radical revision as Internet demographics change?

In conclusion, there are other areas of web-survey research which need to be more systematically addressed. In my research, I was able to independently verify each participant's credentials (along with many of their responses), but in a less narrowly-defined population this may be impossible. What prevents “ballot stuffers” from padding the datasets with repeat participation? Is the much-maligned “cookie” the only way to properly hook a web participant? Then there is also the problem of unwanted participants in an open medium. This may be circumvented by employing password protection of a survey site, but to what extent does this make it more difficult getting the “right” participants, especially when privacy, or even anonymity, is an issue? Are people indeed more candid in electronic media? Or is that a myth, a relic of an older, less suspicious (or canny) Internet population? What design and formatting issues obtain and how are they implemented or compensated for? And speaking of compensation, what new possibilities does the Internet hold for researchers needing to offer incentives?

Acknowledgments

  1. Top of page
  2. Abstract
  3. Introduction
  4. E-mail Compared to “Snail” Mail
  5. The Web Compared to… the Ones That Got Away?
  6. Surveying the Web Presence Provider Community
  7. Results
  8. Discussion
  9. Acknowledgments
  10. References

I would like to thank the following people for their prompt and useful responses to last-minute queries: Diane Witmer of Purdue University; Patrick Douglas Crispen, moderator of ADV-HTML; Mike Ledoux, list owner of HWG Business; Donna, list owner of WEB-WOMEN; Sandy Torre of O'Reilly Research; and Dana Ward of Pitzer College.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. E-mail Compared to “Snail” Mail
  5. The Web Compared to… the Ones That Got Away?
  6. Surveying the Web Presence Provider Community
  7. Results
  8. Discussion
  9. Acknowledgments
  10. References
  • Bachmann, D., Elfrink, J., & Vazzana, G. (1996). Tracking the progress of e-mail vs. snail-mail. Marketing Research, 8(2), 3035.
  • Comley, P. (1996). The use of the Internet as a data collection method, [Online] Available: WWW URL http://www.sga.co.uk/esomar.html.
  • FIND/SVP. (1995). The American Internet users survey, [Online] Available: WWW URL http://etrg.findsvp.com/index.html.
  • Gjestland, L. (1996). Net? Not yet. Marketing Research, 8(1), 2629.
  • Kehoe, C. M., & Pitkow, J. E. (1996). Surveying the territory: GVU's five WWW user surveys. The World Wide Web Journal, 1(3). [Also Online]. Available: WWW URL http://www.cc.gatech.edu/gvu/user_surveys/papers/w3j.html.
  • Kiesler, S., & Sproull, L. (1986). Response effects in the electronic survey. Public Opinion Quarterly, 39, 82101.
  • Kittleson, M. J. (1995). An assessment of the response rate via the postal service and e-mail. Health Values, 18(2), 2729.
  • Lemay, L., Murphy, B. K., & Smith, E. T. (1996). Creating commercial web pages Indianapolis , IN : Sams net Publishing.
  • Mehta, R., & Sivadas, E. (1995). Comparing response rates and response content in mail versus electronic mail surveys. Journal of the Market Research Society, 37(4), 429439.
  • Nielsen Media Research/CommerceNet. (1995). The CommerceNet Nielsen Internet Demographics Survey, [Online] Available: WWW URL http://www.nielsenmedia.com/.
  • O'Reilly Research. (1995). Defining the Internet Opportunity, [Online] Available: WWW URL http://www.ora.com/www/research/users/index.html.
  • Palmquist, J., & Stueve, A. (1996). Stay plugged in to new opportunities. Marketing Research, 8(1), 1315.
  • Parker, L. (1992, July). Collecting data the e-mail way. Training and Development, 5254.
  • Patrick, A. S., Black, A., & Whalen, T. E. (1995). Rich, young, male, dissatisfied computer geeks? Demographics and satisfaction from the National Capital FreeNet. In D.Godfrey & M.Levy (Eds.), Proceedings of Telecommunities 95: The International Community Networking Conference (pp. 83107). Victoria, British Columbia, Canada : Telecommunities Canada. [Also Online]. Available: http://debra.dgbt.doc.ca:80/services-research/survey/demographics/vic.html.
  • Pitkow, J. E., & Recker, M. M. (1994). Using the web as a survey tool: Results from the second WWW user survey. Journal of Computer Networks and ISDN Systems, 27 (6). [Also Online]. Available: WWW URL http://www.cc.gatech.edu/gvu/user_surveys/papers/survey_2_paper.html.
  • Quarterman, J. S. (1996). Summary: Third Internet demographic survey (MIDS3), [Online] Available: WWW URL http://www3.mids.org/mn/603/ids3sum.html.
  • Rosenfeld, P., Booth-Kewley, S., & Edwards, J. E. (1993). Computer-administered surveys in organizational settings: alternatives, advantages, and applications. American Behavioral Scientist, 36(4), 485511.
  • Schuldt, B., & Totten, J. (1994). Electronic mail vs. mail survey response rates. Marketing Research, 6, 3639.
  • Sproull, L. S. (1986). Using electronic mail for data collection in organizational research. Academy of Management Journal, 29, 159169.
  • Tse, A. C. B., Tse, K. C., Yin, C. H., Ting, C. B., Yi, K. W., Yee, K. P., & Hong, W. C. (1995). Comparing two methods of sending out questionnaires: E-mail versus mail. Journal of the Market Research Society, 37(4), 441446.