An Evaluation of Internet Versus Paper-based Methods for Public Participation Geographic Information Systems (PPGIS)

Authors


Amy Pocewicz, The Nature Conservancy, Wyoming Chapter, 258 Main St., Suite 200, Lander, WY 82520, USA. E-mail: apocewicz@tnc.org

Abstract

Public participation geographic information systems (PPGIS) are an increasingly important tool for collecting spatial information about the social attributes of place. The availability of Internet-based options for implementing PPGIS presents new opportunities for increased efficiency and new modes of access. Here we used a mixed-mode approach to evaluate paper versus Internet mapping methods for the same PPGIS survey in Wyoming. We compared participant characteristics, mapping participation, and the spatial distribution of mapped attributes between participants who responded to the paper versus Internet option. The response rate for those who completed the paper version of the survey was nearly 2.5 times the response rate of the Internet version. Paper participants also mapped significantly more places than did Internet participants (43 vs. 18). Internet participants tended to be younger, more likely to have a college degree, and had lived in the region for less time than paper participants. For all but one attribute there was no difference in the spatial distribution of places mapped between Internet and paper methods. Using a paper-based PPGIS survey resulted in a higher response rate, reduced participant bias, and greater mapping participation. However, survey mode did not influence the spatial distribution of the PPGIS data.

1 Introduction

Public participation geographic information systems (PPGIS) are an increasingly important tool for collecting spatial information about the social attributes of place. PPGIS refers to the use of GIS technology to support and empower public participation in planning, natural resource management, and policy development (Sieber 2006). The formal definition of PPGIS remains “nebulous” (Tulloch 2007) and “inconsistent across applications” (Schlossberg and Shuford 2005) due, in part, to the social context in which PPGIS methods are applied. PPGIS methods in developing countries are often used to promote the goals of non-governmental organizations, grassroots groups and community-based organizations in opposition to government policies, while in developed countries, PPGIS methods are often used to augment and infuse existing participatory methods with local spatial knowledge. In the latter approach, planners and natural resource managers commonly use spatial data on the ecological, infrastructure, planning, economic and other characteristics of the landscape; however, social data at a similar spatial scale is much rarer. PPGIS is one approach to filling this gap.

PPGIS can be used to answer many potential basic and applied research questions, and implementation is limited mainly by the creativity of the researcher. PPGIS may be implemented through a variety of tools, including stakeholder workshops or individual interviews (Donovan et al. 2009, Raymond et al. 2009), mail surveys with a mapping exercise (Brown 2005), or other methodologies where participants volunteer spatial information about important places (Sieber 2006). The development of Internet-based applications has increased the available tools and methods for soliciting information for PPGIS (e.g. Beverly et al. 2008, Brown and Reed 2009, Simão et al. 2009, Brown and Weber 2011, Brown et al. 2011).

The Internet presents many new opportunities not available through paper-based surveys with a mapping exercise. In addition to providing a new mode of access for participation in PPGIS, Internet-based applications may reduce data collection costs, increase efficiency through reduced time required for data entry, and increase precision in mapping (Couper and Miller 2008, Brown and Reed 2009). However, Internet survey applications may also affect who participates, the quality of participation, and potentially introduce bias into data collection (Olsen 2009). Internet-based surveys have also shown lower response rates than other survey modes (Manfreda et al. 2008). The implications of choosing traditional survey methods or Internet-based applications are not well understood for PPGIS studies specifically.

In this study we evaluated paper versus Internet methods for the same PPGIS survey in Wyoming, using a mixed-mode approach. Our objective was to determine whether one method is preferable in terms of response rate and opportunity for bias and to assess inter-method reliability for producing similar spatial results. We compared overall response rates, participant characteristics, mapping participation, and the spatial distribution of mapped attributes for those who completed the survey using paper versus the Internet. This study is the first to compare mapping participation and the spatial distribution of mapped attributes between Internet and paper PPGIS survey modes.

2 Methods

2.1 Study Design and Data Collection

The PPGIS survey was completed in Albany, Carbon, and Sweetwater counties in Wyoming, USA (Figure 1, 58,962 km2). The area is predominantly rural, with an approximate population of 88,000 (2007, U.S. Census). The natural landscape includes portions of the Green and North Platte rivers and is dominated by shrublands, with forests and grasslands occurring in some areas. The study area is rich in energy resources. Extraction of oil, gas, coal, and the mineral trona (source of sodium carbonate) has traditionally been important to the local economy. In recent years, development of wind energy resources has increased rapidly. Ranching is also an important part of the local economy and cultural identity.

Figure 1.

The survey area in southern Wyoming, displayed in similar format to the paper map received by survey participants

We developed paper and Internet versions of the same PPGIS survey. Initial invitations were for the Internet survey and provided the option to complete a paper survey. Invitation letters were mailed to 2,000 randomly-selected residents in late March 2010. The invitation included the website address for accessing the survey and a unique access code. A postage-paid return postcard was enclosed, providing the options to request a paper version of the survey or to decline participation entirely. Two and four weeks later a reminder postcard was mailed to all invitees that included the Internet address, access code, and contact information to request paper materials. After four weeks we also mailed unsolicited paper surveys to a random sample of 458 non-participants.

The first part of the survey was a mapping exercise. Participants were asked to place map markers for 16 attributes representing important places, development preferences and knowledge of conditions (Table 1). The Internet-based survey used the Google Maps application programming interface and allowed participants to zoom, pan and view the maps in multiple views (e.g. terrain, map, satellite).1 Participants also had the option to view additional map layers showing streams and land tenure. Participants could drag and drop as many map markers as desired for each attribute and could vary the map scale. In the default map view, 1 cm represented 9.5 km on the ground. Point markers were stored in a SQL database, from which they were directly imported into the ArcGIS® software.

Table 1.  Descriptions of the attributes that participants were invited to map
Mapped AttributeDescription
Important places
RecreationIt is important to maintain outdoor recreation opportunities in these places.
Habitat protectionIt is important to protect fish and wildlife habitat in these places.
AgricultureIt is important to maintain working farms and ranches in these places.
Open spaceIt is important to maintain wide open spaces and scenic views in these places.
WaterThese places are important sources of water.
EconomicThese places are important because of the economic opportunities they provide.
Family traditionsThese places are important to the traditions and history of my family.
Special placesUse these stickers to identify places that are special to you.
Development preferences
Wind developmentIf new wind energy farms are built in these counties, these are the places where I would prefer that development to occur.
Residential developmentIf new homes are built in these counties, these are the places where I would prefer that development to occur.
Oil/gas developmentIf new oil and gas wells are drilled in these counties, these are the places where I would prefer that development to occur.
Knowledge of conditions
Abundant wildlifeThere are abundant wildlife populations in these places, such as large herds of antelope and deer and large numbers of birds and fish.
Good water resourceThese streams, rivers, and lakes are in good condition. There is plentiful and good quality water for fish, wildlife, agriculture, and people.
Water shortageThere is not enough water in these places to keep up with demand for its use.
Good land conditionLands in these places are in good condition. For example, there is little soil erosion, plenty of native vegetation, and good resources to support wildlife or livestock.
Poor land conditionLands in these places are in poor condition. For example, there are problems with soil erosion and weeds, and resources are lacking to support wildlife or livestock.

Paper maps displayed terrain, land tenure, major roads, streams and rivers, towns and place names. Participants were provided with a labeled sticker sheet with six stickers available per attribute and six extra stickers to use for any attribute. Sticker locations were hand-digitized into ArcGIS. On the paper maps, 1 cm represented 4 km on the ground, and each attribute sticker corresponded to approximately 2 km.

Participants were also asked to complete a questionnaire that included questions related to demographics, Internet use, and knowledge about and perceived importance of attributes included in the mapping exercise. Participants were asked to rate the importance of each of seven characteristics for the landscape, on a scale of not at all important (0) to extremely important (4): working farms and ranches, recreation opportunities, fish and wildlife habitat, open spaces and scenic views, family traditions and history, economic opportunities, and availability of water. Participants were also asked to describe their knowledge of each of six issues, on a scale of none (0) to excellent (4): residential development, wind development, oil and gas development, fish and wildlife habitat, land or vegetation condition, and water quality and quantity.

To identify reasons for lack of participation and possible response bias, we completed a phone survey of randomly-selected non-respondents. We were able to interview 45 non-respondents to determine general reasons for lack of participation, 34 of whom completed full interviews. We asked if there was any reason they chose not to participate, and if Internet access affected their decision. The interview included demographic questions and a subset of survey questions related to importance and knowledge of mapped attributes.

2.2 Statistical Analysis

To test for differences in characteristics, values, or knowledge between participants and non-participants or between Internet and paper participants, we applied chi-square or Fisher's tests for proportional data and t-tests or Wilcoxon rank sum tests for continuous data (α= 0.05), using SAS 9.2 (SAS Institute, Cary, North Carolina). Fisher's tests were used when any cell value was less than five, and Wilcoxon tests were used when data were not normally distributed. We tested for differences in the proportion of participants mapping each attribute for Internet versus paper surveys using chi-square tests (Bonferonni α= 0.003).

To assess inter-method reliability of the mapping, we tested whether mapped points from Internet versus paper surveys were placed in similar locations using the bivariate K-function in the R (R Development Core Team 2011) spatstat package (Baddeley and Turner 2005). The bivariate K-function uses the expected number of type-2 points (i.e. Internet points) occurring within a given radius of a type-1 point (i.e. paper points) to describe whether the two point patterns are spatially independent or correlated (Lotwick and Silverman 1982). The observed values were tested against an expected completely random spatial distribution using a Monte Carlo approach with 999 iterations to simulate a confidence envelope (α= 0.001) around Besag's – L function values (Besag and Diggle 1977). Besag's-L standardizes the K(r) statistic so that L(r) > 0 indicates spatial clustering, L(r) < 0 indicates dispersion, and L(r) = 0 represents the expected random spatial arrangement (Besag 1977). We specified the search radius in 2-km increments up to one-third of the size of the study area, and applied Ripley's isotropic correction for edge effects (Ripley 1977).

3 Results

3.1 Survey Participation

Approximately 10% of 1,961 deliverable surveys were completed. The mapping activity was completed by 198 participants, 99 via the Internet and 99 via paper. Mapping response rate was 17% for those who received paper surveys and 7% for the Internet survey. The questionnaire was completed by 191 participants, 85 via the Internet and 106 via paper. The response rate was 56% for requested paper surveys and 7% for unsolicited paper surveys sent as the third contact. We received 107 postcards declining participation (∼5%).

Through the non-participant phone survey we learned that most people (64% of 45) did not participate for reasons unrelated to the survey content or mode. For example, they did not remember receiving the survey, thought it was “junk” mail, generally disliked surveys, or did not have time. The other 36% provided reasons related to survey content, including not feeling knowledgeable enough, not understanding the survey, or perceiving a conflict of interest. Five people (11%) reported that lack of Internet access or the inconvenience of visiting a website to complete the survey affected their decision. Participants were skewed towards males and those with a college education, but there was no bias in age. The importance of working farms and ranches, fish and wildlife habitat, and availability of water did not differ between participants and non-participants, but economic opportunities were more important to non-participants. Participants had greater self-described knowledge concerning wind development and fish and wildlife habitat than did non-participants.

3.2 Participant Characteristics

Participants with a 4-year college degree responded more often via the Internet than paper (Table 2). Internet participants were also younger and had lived in the area fewer years than paper participants. Participation mode was not related to gender or residence in a town versus rural area (Table 2). Those who reported using the Internet every day responded more often via the Internet (Table 2). However, only 61% of participants who reported daily Internet use completed the survey online. Nearly all Internet participants (96%) reported that their Internet access was via a high-speed connection. Fewer paper participants (65%) reported that their Internet access was via a high-speed connection. Only 13% of all survey participants reported that they never used the Internet, all of whom completed paper surveys.

Table 2.  Survey participant characteristics, by paper (n = 106) versus Internet (n = 85) survey mode
CharacteristicPaperInternetDifference with mode?
%Female/%Male28/7235/65No (χ2= 0.9, p= 0.36)
Avg. age (SE)56 (1.3)49 (1.4)Yes (t-test, p= 0.001)
% 4-yr college degree3553Yes (χ2= 6.3, p= 0.01)
Avg. years in county (SE)34 (2.2)24 (1.9)Yes (Wilcoxon, p= 0.003)
%Live in a town8180No (χ2= 0.04, p= 0.84)
%Use Internet daily4473Yes (χ2= 34.9, p < 0.0001)

Fish and wildlife habitat was more important to paper than Internet participants (Wilcoxon test, p = 0.02). This was the most important of the values included in the survey, with 96% of paper participants reporting that fish and wildlife habitat was extremely or very important, as compared with 88% of Internet participants. Self-rated importance of the other six attributes did not differ with survey mode. Self-described knowledge related to mapped attributes also did not differ with survey mode.

3.3 Mapping Participation and Spatial Distribution

A total of 6,020 map markers were placed by 198 participants, with an average of 30 markers per participant. Participants who used paper maps placed more markers (4,281, average 43 per participant) than those who used Internet maps (1,739, average 18 per participant) (Wilcoxon test, p = <0.0001). The proportion of participants who placed at least one marker was higher for every attribute for paper, as compared to Internet (p < 0.003; Figure 2). The number of markers placed per attribute was also greater for paper than Internet participants, for all attributes (Figure 3). Three-quarters of Internet participants placed between 4 and 21 map markers, while three-quarters of paper participants placed between 22 and 65 map markers. The proportion of participants placing five or fewer markers was much greater for Internet (27%) than paper participants (4%). Although we did not collect data on time spent on the mapping activity from paper participants, we were able to measure time spent on the mapping for Internet participants. The majority of Internet participants (75%) spent between 2.5 and 13 minutes completing the mapping exercise, with a mean of 9 minutes. The majority of the Internet-mapped points (69%) were placed using the default map scale. Most participants (77%) accessed the website only once. There were 22 people who logged in to the website but did not complete any portion of the survey.

Figure 2.

The proportion of participants completing paper or Internet surveys who placed at least one map marker for each of the mapping attributes. For all attributes, the proportion of participants was significantly higher for the paper version of the survey (chi-square, Bonferonni-corrected α= 0.003)

Figure 3.

The number of points mapped for each attribute, by paper versus Internet participants

Points mapped via Internet and paper modes were placed in similar locations, as illustrated by significant spatial clustering between Internet and paper map points in the bivariate K-function plots (Figure 4). The spatial clustering was also observed visually in maps displaying both Internet and paper points; examples from a subset of attributes with varying levels of clustering are shown in Figure 5. Internet and paper map points were spatially independent for only one attribute, poor land condition, which was indicated by a K-function line that occurred entirely within the random simulation envelope (Figure 4). The distance at which clustering began – the typical distance between Internet and paper points – varied among attributes. Several attributes (recreation, agriculture, water, economic, residential development) showed significant clustering beginning at very short distances, as illustrated by K-function lines that occurred entirely outside the random simulation envelope (Figure 4). Clustering was especially strong for economic and residential development attributes; the K-function lines declined at large distances because few Internet and paper points were located that far apart anywhere across the landscape (see residential development example in Figure 5). For other attributes clustering was significant but distances between points were greater. Many of the attributes (wind development, family traditions, special places, oil/gas development, abundant wildlife, good water resource) showed clustering between Internet and paper points at distances of 2,000 or 2,500 m, close to the 2,000 m spatial resolution associated with marker placement. Clustering between Internet and paper map points was still significant, but weakest for habitat protection (5,000 m), good land condition (5,000 m), open space (6,000 m, points shown in Figure 5), and water shortage (7,500 m) (Figure 4). Weaker clustering between paper and Internet map points was generally consistent with weak within-group or overall clustering; some attributes were placed in more diverse locations across the landscape than others, regardless of survey mode (Figure 5).

Figure 4.

Bivariate K-function plots are shown for each mapped attribute that illustrate the spatial relationship between map points from paper versus Internet surveys. A value of 0 for the L-standardization of the K-function represents the expected random spatial arrangement, and the gray shading represents the simulation envelope. Where the line resides outside and above of the gray shading, there is a significant clustered spatial relationship between the paper and Internet points. Clustering begins at varying spatial extents, as indicated by the distance on the x-axis at which the line is first outside the envelope. The special places attribute is not shown; it had a pattern identical to that of oil/gas development

Figure 5.

Locations of Internet versus paper map points for a subset of survey attributes: recreation, open space, residential development, and wind development. Internet points are displayed transparently over top of paper points to better illustrate overlap between the two point types

4 Discussion

All modes of survey data collection have shown declining response rates over time. However, Internet-based surveys have shown, on average, 11% lower response rates than other survey modes (Manfreda et al. 2008). Internet surveys perform worse than other survey modes, and Internet survey participation is even more disadvantaged when invitation is by postal mail or when recruitment is limited to a single study (Manfreda et al. 2008). The additional burden of a PPGIS mapping activity appears to further depress response rates. Low response rates are a clear indication of potential non-response bias, but appropriately quantifying and correcting for actual non-response bias remains unclear, especially for PPGIS.

The response rate for the paper version of the survey was more than double that for the Internet version. Had we been prepared to mail paper surveys to all non-participants as the third contact, the total response rate would have likely been at least 5% greater, based on observed response to the unsolicited paper surveys. It is likely that the response rate would have increased even further had we completed the entire survey via paper mailings, as found by Smyth et al. (2010). Our response rate was slightly lower than that of similar Internet-based PPGIS mapping surveys, which have averaged 13% across five studies (Beverly et al. 2008, Brown and Reed 2009, Brown et al. 2011). For similar paper-based mapping surveys, response rates have ranged from 15–47%, with an average of 30% across 11 surveys (Brown et al. 2004, Brown 2005, Alessa et al. 2008, Zhu et al. 2010, Clement and Cheng 2010, Nielsen-Pincus 2011, Raymond and Brown 2011).

Convenient Internet access only partially explains the lower Internet response. Only 11% of those who chose not to participate related their decision to Internet access, and only 13% of participants reported that they never use the Internet. This is lower than findings from similar Internet-based PPGIS surveys, where 22–31% of non-participants reported lack of convenient Internet access as a participation barrier (Brown and Reed 2009). It may have simply been more convenient to complete a paper survey that did not require the extra step of visiting a website. Other studies have also demonstrated a preference for paper-based surveys that is independent of Internet access (Schonlau et al. 2003, Millar and Dillman 2011).

Survey mode did result in some participation bias. We observed an age bias that was not present in the overall sample of participants, and Internet participants were even further skewed towards those with college degrees. These same biases were observed when comparing Internet and paper survey modes in rural Idaho and Washington (Smyth et al. 2010). An Internet-only survey may have excluded some older individuals or more individuals without college degrees. Knowledge of the issues included in the survey did not differ between participants who completed the survey using different modes, suggesting that survey mode did not bias participants' ability to map attributes or respond to the questionnaire. Only one of seven values (wildlife) differed in importance to participants between survey modes, suggesting that survey mode did not result in cultural bias. The importance of wildlife habitat was not correlated with age or education, characteristics that differed between paper and Internet participants.

A notable difference between the survey modes was the number of markers placed on the maps. Internet participants placed fewer markers on the maps, and each Internet participant marked fewer attributes overall than did paper participants. Internet participants also probably spent less time on the activity than did paper participants, who mapped nearly 2.5 times as many markers as did the Internet participants. These patterns have not been compared previously, as previous PPGIS surveys have used either Internet or paper methods independently. Across nine similar paper PPGIS surveys, each participant placed an average of 27 map markers (Brown et al. 2004, Brown 2005, Zhu et al. 2010, Nielsen-Pincus 2011, Raymond and Brown 2011), and across five similar Internet PPGIS surveys, each participant placed an average of 29 map markers (Beverly et al. 2008, Brown and Reed 2009, Brown et al. 2011). We are not able to explain the comparatively lower effort expended by Internet mappers in our study, but it is consistent with the fast-paced, multi-tasking mentality generally associated with Internet use. Distractions and decreased concentration associated with Internet use may lead people to be less productive at some tasks, a phenomenon that is receiving growing attention in the popular literature (i.e. Jackson 2008, Carr 2010).

Mapping effort in our study was above average for paper participants and below average for Internet participants. This pattern was partly explained by 27% of the Internet participants, who placed five or fewer markers. Possible explanations for the particularly low response from these participants include difficulties with the website while mapping that led them to give up, or not fully understanding the activity before beginning and deciding after a quick attempt that they were not interested. The paper participants had all the materials in front of them, and thus they may have been able to make a more informed decision about their participation. Another potential explanation is that the finite number of markers per attribute in the paper mode (n = 6) provided clearer survey response expectations than having an unlimited number of markers as in the Internet application. Ambiguity and uncertainty about the adequacy of the response (i.e. how many markers placed is good enough?) may result in participant disengagement with the process. Differences in map scale between the modes may also contributed to this outcome. Most Internet participants viewed a coarser scale map than paper participants, and that could have led to placement of fewer markers perceived to represent larger areas.

For all but one attribute there was no difference in the spatial distribution of places mapped using Internet versus paper methods. For the poor land condition attribute for which Internet and paper points were spatially independent, this pattern was likely explained by a lack of clustering overall due in part to a particularly small number of mapped points (Figure 3). Some attributes were placed in more diverse locations across the landscape than others, regardless of survey mode, but we found no evidence that survey mode systematically influenced where participants placed map markers. Survey mode did not appear to relate to places designated as being important, preferred for development, or to which condition attributes were assigned. A limitation of this analysis is that we only considered point patterns. Conclusions regarding spatial association may differ if density or ‘hotspot’ surfaces derived from points were compared instead. We were unable to compare derived surfaces in this case because the large sample size differences between Internet and paper modes biased the spatial extents and number of derived high-density areas.

Overall we found that using a traditional paper-based PPGIS survey resulted in a higher response rate, reduced participant bias, and greater mapping participation, as compared with an Internet-based survey. The paper survey did not appear to have the potential to exclude those able and interested in participating as the Internet survey did. However, survey mode did not influence where participants placed markers on maps, indicating that the two modes of response produce reliably similar spatial results for the PPGIS attributes. But while the paper and Internet modes produce reliably similar spatial results, the higher response rate and participation effort of the paper-based PPGIS survey suggests that a paper-based PPGIS survey is more effective and representative when surveying the general public via mail.

Acknowledgements

We are grateful to the Wyoming residents who shared their time for this survey. We thank Diana Hulme for assistance with survey design, Jeremy Murray for website work, Dot Newton for data entry and the phone survey of non-participants, and Jeffrey Evans for statistical analysis assistance.

Note

  • 1

    The website was created by the Center for Spatial Information at Central Washington University, and the survey interface can be viewed at http://www.landscapemap2.org/TNC (access code 101-0101).

Ancillary