Measuring Mobile Phone Use: Self-Report Versus Log Data

Authors


Abstract

Approximately 40% of mobile phone use studies published in scholarly communication journals base their findings on self-report data about how frequently respondents use their mobile phones. Using a subset of a larger representative sample we examine the validity of this type of self-report data by comparing it to server log data. The self-report data correlate only moderately with the server log data, indicating low criterion validity. The categorical self-report measure asking respondents to estimate “how often” they use their mobile phones fared better than the continuous self-report measure asking them to estimate their mobile phone activity “yesterday.” A multivariate exploratory analysis further suggests that it may be difficult to identify under- and overreporting using demographic variables alone.

Introduction

Mobile phones are without question the most common form of electronically mediated communication in the world. We have adopted the mobile phone as a safety link, and we have adopted it to facilitate coordination in the family and at work. Textile weavers in Nigeria (Jagun, Heeks, & Whalley, 2008) fishers in India (Jensen, 2007), and lovers in Mozambique (Archambault, 2011) have all adopted the mobile phone for their various purposes. Teens in the US, as well as many other countries, use the mobile phone to send and receive an endless number of texts on all the large and small events of their lives (Lenhart, et al., 2010). Given these diverse uses, it is a fertile area of research. Understanding how we use the mobile phone helps to understand the functioning of society.

The ability to do research on mobile telephony (or for that matter any form of mediated communication) however is based on the ability to use valid data. In many cases research on the mobile phone relies on questionnaires. In the questionnaires, respondents are often asked to estimate the frequency or the duration of their mobile phone use either in summary form or during a specific time period. Alternatively respondents can be asked to use a diary or some other device to quantify their use. These methods are nearly the same methods used to collect data on internet and social media use.

There are clearly problems with these approaches. It is difficult for people to remember or to characterize their use. In addition there may be different motivations to over- or underreport mobile phone use. For people who see use as a marker of their popularity, there will be the impulse to pad the numbers. The opposite is true for people with a more sober relationship to technology and who feel the device is not so central to their lives but who nonetheless have one. We are interested to get a handle on the reliability of this information. In this paper we compare self-report measures of mobile phone voice and SMS (Short Message System; known as “texting” or “text message”) use with server log data to evaluate the criterion validity of self-report measures. We further conduct an exploratory analysis to examine the extent to which demographic characteristics are associated with under- and overreporting in self-report measures. Our analysis has two main implications. First, given that many influential studies of mobile communication rely on self-report measures of mobile phone use, our evaluation of these measures has strong implications for the field of mobile research as well as studies on other forms of mediated communication that rely on similar methods. Second, given that most research involving self-report measures also collects demographic data, our exploratory analysis of the demographic characteristics associated with under- and overreporting of mobile phone usage informs existing and future research about the types of respondents that are likely to contribute to the decreased validity of self-report measures.

Self-Report Measures in Mobile Communication Studies

To confirm that the self-report measures we examine in this paper are indeed commonly used in scholarly communication research we conducted an analysis of journal articles found through the Social Sciences Citation Index. Using the “communication” filter to focus only on publications appearing in communication sources, we searched with the keywords “mobile phone.” We then manually sifted through the results and removed papers that were not based on empirical research, leaving us with 41 journal papers published between 2003 and 2010.

In many cases the authors of these papers only mention the general nature of the questions asked and do not provide their exact wording. However, enough description was provided for us to identify two common types of self-report measures that operationalize intensity of mobile phone use: frequency and duration. Frequency measures typically asked respondents to report on the number of calls or SMS messages sent and received in a single day, or how often they use their mobile phones in more general terms (e.g. “less than once a month,” “one to three times a month,” “once a week,” etc.). Duration questions ask respondents to estimate amount of time that they spend using mobile phones in a single day or over several days. Duration studies often involve the use of time diaries where respondents write down the amount of time that they spend communicating using mobile phones. Sometimes duration studies simply ask respondents to estimate the amount of time that they spend using their mobile phones during a short period of time. Other types of self-report mobile phone use measures include the number of friends and family contacted regularly by mobile phones, the places where mobile phones are used, and self-perception attitudes about how extensively mobile phones are used. Approximately 40% of papers reviewed include at least one frequency of use measure, 27% include at least one duration of use measure, 9% rely on self-report measures other than frequency or duration to gauge level of use, and the remaining papers do not include any quantitative mobile phone use measures. Correlation analysis showed little association between the type of measure used and the number of times that a paper has been cited.

The variety of approaches employed to measure mobile phone use indicates that there is not a single commonly accepted approach to gathering data on this issue, likely because different questionnaires are developed for different purposes. Nevertheless, the fact that approximately 40% of the papers reviewed include at least one self-report measure regarding frequency of mobile phone use indicates that collecting this type of information is applicable to a variety of studies. This is not surprising given that frequency of mobile phone use is indicative of the extent to which mobile phones have become embedded into everyday life. Moreover, as with any study in mediated communication, establishing that a technology is used at least to some extent is necessary before further implications of its use can be investigated. Given the wide applicability and popularity of self-report frequency of use measures we focus specifically on this type of measure in this study.

Examining Self-Report Measures of Mobile Phone Use

Although communication researchers often use self-report measures of mobile phone use, we could identify only two studies that examined the validity of these measures. One is a convenience sample study of 93 volunteers conducted by Parslow, Hepworth, and McKinney (2003). In this study volunteers were asked to report on the number of incoming and outgoing voice calls during the previous day, week, or month, and further estimate the number of minutes spent on the voice calls. Their responses were then compared to log data collected from four different mobile phone operators. Their results showed moderate correlation between the self-report and log data, indicating that self-report measures did not fully represent actual use patterns. There was also evidence of over-reporting, especially among users with relatively low levels of activity. In another study, Cohen, and Lemish (2003) used a quota sample to compare self-report measures with real time data collection using Interactive Voice Response technology over a 5-day period. They found the correlation between the questionnaire data and billing data regarding the frequency of calling to be moderately strong, indicating a reasonable fit between the self-report measures and actual behavior.

While these two studies represent an important first step towards understanding the validity of self-report measures in mobile communication studies, the nonrandom nature of their sampling methods warrants further study of this issue. Moreover, these studies focused only on voice calling activity and did not include measures of SMS messaging.

In related research, social network studies have looked at self-report measures of communication networks. Using four different samples, Bernard and Killworth (1977), conclude that respondents do not know with any accuracy the individuals with whom they communicate. In another paper that examines the results from seven experiments, Bernard, Killworth, and Sailer (1982), argue that self-report of communication behavior bears no useful resemblance to actual communication behavior. Freeman, Romney, and Freeman (1987) also find low levels of accuracy in respondents' ability to identify individuals who coattended colloquium presentations with them, further finding that errors were biased by long-term attendance patterns. Although none of these studies examine self-report in regards to mobile phone use, they do indicate that problems with self-report measures extend beyond mobile phone use as a particular form of communication.

There are three main reasons to question the validity of self-report measures regarding mobile phone use: cognitive burden, social desirability, and conceptual validity. Cognitive burden occurs when respondents attempt to recall behaviors that they do not typically think about or record on a regular basis. To decrease this burden respondents often use heuristic shortcuts that decrease the validity of measures (Sudman, Bradburn, & Schwartz, 1996). Moreover, since mobile phone interaction is potentially regarded as a more general indicator of sociability, interest in increasing desirability may occur and cause respondents to overreport the extent to which they use their mobile phones. Finally, even if self-report measures of mobile phone activity are accurate, there is some reason to question the extent to which they measure the concepts that they are intending to measure. In particular, questions that focus on mobile phone activity in a single day are suspect because it is unclear if the activity occurring in that day is actually representative of more general levels of activity. For individuals with irregular schedules, day-to-day mobile phone activity may fluctuate dramatically.

Method

In this study we draw on a combination of data collected through a nationally representative internet-based survey conducted in Norway during October and November of 2008 and sever log data for the month in which respondents completed the survey. In order to ensure the representativeness of the sample, participants were recruited from an earlier telephone survey and a few questionnaires were given over the phone to older respondents who did not have internet access. Two versions of the mobile use question were included in the survey: a frequency of mobile activity occurring “yesterday” and frequency of mobile phone use occurring over longer periods of time, e.g. on a weekly basis. The survey data that we use in this study is drawn from the second wave of a two-wave study. In total, 1499 respondents agreed to participate in the second wave (65% response rate), and respondents showing inconsistencies between the first and second wave (for example, radically different ages, change in gender, etc. between wave 1 and wave 2) were dropped leaving a total of 1382 respondents. Researchers potentially had access to the log data for 613 of these second-wave respondents because these respondents were customers of the telecommunication company funding the survey. Four hundred twenty-six of those customers granted their informed consent for the researchers to access their log data.

The demographic composition of those granting access to their data is as follows: 50% are female, 58% are married, 23% hold a college degree, 58% are working full time, the mean age is 49 years (SD = 15.00; skewness = −.17), the mean household size is 2.7 individuals (SD - 1.41; skewness - .85). Those granting access to their log data were similar with the other second wave respondents who did not grant access to their log data in terms of their sex, household size, and education. However, compared to the other respondents, those who granted access to their log data were a mean of approximately 8 years older, more likely to be married (58% married versus 42% married), and more likely to be working full time (58% working full time versus 48% working full time). A separate analysis of second-wave respondents indicates that the subset of respondents used in the analysis reported here–i.e. customers who granted access to their log data–differ from other respondents in our representative sample mainly because they are customers of a particular telecommunication company and not because of self-selection in regards to their willingness to share their log data. This bodes well for the generalizability of our results, although we caution that these results are still somewhat limited insofar as they apply most directly to individuals who share the demographic traits of our subsample.

Self-Report Measures

The telephone survey data included two types of self-report measures aimed at measuring intensity of mobile phone use for both voice and SMS messages. The first type of measure is continuous in nature and asked respondents to report the number of times that they used their mobile phone “yesterday” for outgoing voice calls and SMS messages. In one question they were asked: “How many times did you use a mobile phone yesterday to call others, either for work or privately.” In another question respondents were asked, “How many text messages did you send yesterday, either for work or privately?” We refer to these types of continuous self-report questions as “yesterday” measures.

The second type of question is categorical in nature and required respondents to report on how often they use their mobile phones by choosing the most appropriate response from a set list. In one question they were asked, “How often do you use a mobile phone to call others (not texting or SMS)? More than 10 times a day, 5–10 times a day, 2–4 times a day, at least once a day, 3–6 times a week, 1–2 times a week, Less often, Never.” In another question they were asked, “How often do you use the mobile phone to send or receive text messages/SMS?” and were given the same set of categorical response options. We refer to these types of categorical self-report questions as “how often” measures. Note that all self-report measures except for this last categorical measure of SMS messages refer only to outgoing activity and that all questions have been directly translated from Norwegian.

Log Data

Server log data includes all outgoing voice and SMS events captured during the month in which the survey took place. Three types of variables are constructed with this data for both voice and SMS events. The first is simply the number of outgoing events that month. The second is the average number of outgoing events per day as calculated by dividing the total number of events by the number of days in that month. This type of variable is used for comparison with the yesterday measures of daily use. The third is categorical, constructed using the same categories as those used in the how often measures, i.e. More than 10 times a day, 5–10 times a day, 2–4 times a day, at least once a day, 3–6 times a week, 1–2 times a week, Less often, Never. Here a calculation is used to select the most appropriate category from these choices using the log data.

Conceptualization and Operationalization

To allow for a fair comparison between the self-report and log measures it is important to be clear about exactly what is being compared. Our comparison is concerning the various possible operationalizations of a concept, rather than the accuracy of the measures themselves. In other words, we are more interested in the extent to which self-report measures can adequately operationalize a particular concept than the extent to which they directly measure the exact nuances of the behavior about which they are asking. For example, we are more interested to know if asking people to report on the number of people they called yesterday is a good operationalization of mobile phone use, in general, than if respondents are able to recall this behavior on a specific day with a high degree of accuracy.

The broad concept that we are focusing on is intensity of mobile phone use, and the specific dimension of this concept being examined is frequency of mobile phone use. Among the various ways of operationalizing this dimension, monthly server log data represents one of the best possible operationalizations because it is highly accurate. Indeed, it must be accurate because it is the basis of billing. Moreover, since the data is collected for a sustained period of time it is less affected by unusual events. Although it may be the case that monthly server log data collected during the holiday season may be unusually high, the data that we use in this analysis is not collected during this period of time. Moreover, all the respondents included in our analysis indicated that they are the sole owner of their mobile phone and that they do not use anyone else's mobile phone. In addition, the practice of “SIM-switching” that is often seen in developing countries is not common in Norway.1 Thus, for all intents and purposes, the log data represents the total mobile phone use of a single individual.

Given that the server log data represents an ideal way of operationalizing frequency of use, our analysis focuses on comparing it to the two types of operationalizations that are less ideal–namely, the two types of self-report measures discussed above. While these measures are clearly aimed at operationalizing frequency of use, issues of respondent recall, social desirability, and generalizability give reason to doubt their validity. In the analysis that follows we examine the criterion validity of the self-report measures by comparing them to the more ideal sever log measures to address the following research questions.

Research Question 1: How well do the self-report measures compare to sever log data?

Since it is likely that at least some respondents will be more prone to over- or underreport their usage and identifying such respondents may help researchers to develop better measures and decrease their negative impact in the analysis, we further ask:

Research Question 2: What are the demographic characteristics associated with over- and underreporting of mobile phone use?

We focus on demographic characteristics primarily because such characteristics are typically collected in self-report surveys.

Analysis and Results

To address our first research question, we start by comparing our log data with the self-report measures using pairwise correlations. The results show that although the self-report measures are significantly correlated with the log data (p < .001 for all correlations), the correlations are not as strong as could be expected from measures that operationalize the same concept equally well.

The correlations between the yesterday self-report questions and the voice log measures are .55, p < .001, for outgoing voice calls, and .58, p < .001, for outgoing SMS messages. In order to examine the possibility that the presence of outliers adversely affected these correlations, we preformed logarithmic transformations on these variables. Our results show lower correlations with these transformed variables, at .23, p < .001, for outgoing voice calls and .41, p < .001, for outgoing SMS messages.

The correlations between the how often self-report questions and the SMS message log data are .48, p < .001, for voice calls and .35, p < .001, for SMS messages. Correlations with a transformed log variable are .66, p < .001, for voice calls and .74, p < .001, for SMS messages. Given that the transformed variables better controlled for the influence of outliers, these results indicate that the correlations between self-report and log data for the how often measures were stronger than the correlations for the yesterday measures.

In sum, although these correlations may be high for variables that measure two different yet associated things–for example, mobile phone use and life satisfaction–they are not high for variables that are supposed to be measuring exactly the same thing. Overall, correlating self-report measures to log data generally indicates that the self-report measures are only somewhat similar to the superior log measures. The most promising self-report measure was the categorical how often measure of SMS messaging. We note that this finding is unexpected given that this measure focused on both outgoing and incoming messages, while the log data captured only outgoing SMS messages, though there is often symmetry between in and out-going calls. The other self-report measures that show less correlation with the log data asked about outgoing voice and SMS events only, making them theoretically more comparable to the outgoing log data.

Next, we compare the self-report measures directly to the log data to understand exactly how the self-report measures differ from the log data, and to confirm that these differences are indeed significant. For this set of analyses we use log variables that were recoded to make them directly comparable to the self-report measures. We start by describing the difference between yesterday self-report and log measures, as presented in Table 1. We find that respondents report making more voice calls yesterday than the log data shows that they make during an average day. The mean number of reported calls yesterday is 3.88, while the mean number of calls logged on an average day is 40 percent lower than that number (2.38). The median number of calls reported yesterday is 3, while the median number of calls logged on an average day is less than half that number (1.43). We also find a much larger standard deviation for the self-report yesterday measures than the voice log measure (4.8 versus 2.99, respectively), indicating a much greater range of variability in the responses given through this self-report measure than we would expect based on the voice log measure. As with the voice self-report yesterday measures, the SMS message self-report measures were also higher than the log SMS measures. Respondents report a mean of 6.19 SMS messages yesterday, while the mean number of SMS messages logged in an average day is 3.95. The median number of self-report SMS messages is 3.00, while the median number of logged SMS messages is 1.87. The standard deviation for the self-report SMS measure is 13.98, while the standard deviation of the log SMS measure is 8.81.

Table 1. Central tendency comparison of the “yesterday” self-report measure and average daily call log activity
 VoiceSMS
Self-reportLogDifferenceSelf-reportLogDifference
Mean3.882.381.506.193.952.24
Median3.001.431.573.001.871.13
SD4.802.991.8113.988.815.17

A one-tail t test comparing the self-report and log measures of voice calls occurring yesterday confirms that the mean of the self-report measure is significantly higher than the mean of the log measure, t(425) = 7.72, p < .001. A one-tail t test comparing the self-report and log measures for the number of SMS messages sent yesterday also shows that mean of the self-report measure is significantly higher than the mean of the log measure, t(425) = 4.07, p < .001.

We also compared the how often self-report measure to recoded log variables. Table 2 shows the percentage of respondents in each category of these self-report and log measures. The results generally show that a greater percent of respondents in each category of the self-report measures than in each category of the log measures. The main exception is that respondents whose log data show an extremely low amount of usage do not tend to report this low usage in the self-report measures. The log data showed that 17% of the respondents made less than one voice call per week, however only 3% of the respondents self-reported having this low level of calling activity. This finding may point to respondents' reluctance to report low levels of socialization in the questionnaire. The log data also showed that 23% of the respondents sent less than one text message per week, yet only 5% of the respondents self-reported exchanging less than one text message per week.

Table 2. Percent comparison of the “how often” self-report measure and categorized call log activity
 VoiceSMS
Self-reportLogDifferenceSelf-reportLogDifference
11+ times a day9361174
5-10 times a day1510524159
2-4 times a day3228428271
At least once a day1418−31013−3
3-6 times a week1915314104
1-2 times a week89−1861
Less often317−14523−18
Total percent100100 100100 

To confirm that the self-report and log measures for the categorical how often variables differed significantly according to more stringent statistical criteria, we conducted a Chi-square goodness of fit tests using csgof program installed as an add-on command in STATA. Using this program we were able to compare the frequency of respondents in each category of the self-report measures with the frequencies that we would expect based on the log data. The results of this test showed that the frequencies of respondents in the categories of the voice self-report measure differed significantly from the frequencies that could be expected based on the voice log measure, chisq(6) = 121.49, p = 0. Similarly, results showed that the frequencies of respondents in the categories of the SMS self-report measure differed significantly from the frequencies that could be expected based on the SMS log measure, chisq(6) = 124.71, p = 0.

In sum, regarding our first research question, our results show that self-report measures of voice calling and SMS messaging generally do not compare favorably to log measures. This is true for both the self-report measures that require respondents to report on these behaviors yesterday, and for the self-report measures that requires respondents to report how often these behaviors occur by selecting from a number of predetermined categories. In general, respondents are more likely to overreport than to underreport voice calling and SMS messaging.

To address our second research question, we conducted an exploratory multivariate analysis to examine the extent to which demographic characteristics can be used to predict under- and overreporting of voice calling and SMS messaging. We control for whether respondents pay their mobile phone bills completely on their own since this likely affects the self-monitoring behavior that would assist in recall and is also related to factors such as employment status, age, and education.

We first examine under- and overreporting of the yesterday measures. Accordingly, two independent variables are used: one indicating the amount of overreporting that occurs when the self-report measure is greater than log measure of an average day, and the other indicating the opposite. The results of this analysis are presented in Table 3.

Table 3. Regression analysis of overreporting and underreporting with the “yesterday” self-report measure
 VoiceSMS
OverreportingUnderreportingOverreportingUnderreporting
  1. Note: scores are unstandardized regression coefficients.
  2. ap < .05.
  3. bp < .01.
  4. cp < .001.
Pays Cost−.438a−.200−.248−.177
Male.551c.064.290−.507a
Age−.011−.008−.036c−.028b
House size−.027.253c−.033.087
Married−.056−.117−.703b−.308
College degree−.021.114−.389−.392
Working.027.367a−.309.521a
Constant1.426b−.4483.403c1.507a
R2.121.236.250.302
Adjusted R2.095.202.224.273
N243164212174

In regard to voice calling, different demographic traits are associated with overreporting than are associated with underreporting. Being a male is positively associated with overreporting while household size is positively associated with underreporting (p < .001 for both coefficients). Working full or part time is positively associated with underreporting (p < .05). In regard to SMS messaging, age is negatively associated with both overreporting and underreporting (p < .001 for both coefficients), implying that young adults provide less valid reports than those who are older. Being a female and being employed are both positively associated with underreporting of SMS messages (p < .05). Being married is negatively associated with overreporting of SMS messages (p < .01). The adjusted R2 statistics for these models are all relatively low, indicating that demographic traits do not explain a great amount of the variance in under- and overreporting.

We also used a multivariate analysis to explore demographic factors associated with under- and overreporting that occurred with the how often self-report variable, using dichotomous variables to indicate under- and overreporting that occurred when respondents selected a category that differed from the appropriate category indicated by the log data. The results of this analysis are presented in Table 4.

Table 4. Logistic regression analysis of overreporting and underreporting with the “how often” self-report measure
 VoiceSMS
OverreportingUnderreportingOverreportingUnderreporting
  1. Note: scores are unstandardized regression weights.
  2. ap < .05.
  3. **p < .01.
  4. ***p < .001.
Pays Cost−.317.259−.180−.057
Male.490−.704a.535a−.348
Age.018.002.025a.006
House size−.085.142.042−.026
Married−.169−.078−.138−.015
College degree.281−.052.474.086
Working−.096.017.329−.372
Pseudo R2.030.023.035.011
χ215.059.5117.943.81
N248219260214

This multivariate analysis shows that only a few factors are significantly associated with under- and overreporting. In regards to voice calls, only being a male was significantly and negatively associated with underreporting (p < .05). In regard to SMS messaging, only being a male and age were significantly and positively associated with overreporting (p < .05). That is, this analysis indicates that males significantly overreport their voice mobile use. Further, males and also people who are older overreport their use of SMS. As with the multivariate analysis of the yesterday self-report measures, the low Pseudo R2 statistics of these multivariate models indicates that demographic traits do not explain a great amount of the variance in under- and overreporting that occurs with the how often self-report measures.

In sum, our exploratory analysis links a few demographic traits to under- and overreporting, especially with regard to the yesterday self-report measure. Nevertheless, these demographic traits to not explain a large amount of variance in our models, which suggests that it may be difficult to identify and compensate for under- and overreporting using demographic traits alone.

Limitations

This study faces a few limitations. First, given that it takes place in Norway, it is possible that the results may not generalize well to other cultures or contexts. While this problem is common to most studies and there is no obvious reason to think that Norwegians would be more likely to under- or overreport their mobile phone usage than people in other countries, this point is worth bearing in mind. Second, our focus has been on how well these measures operationalize the concept of frequency of mobile phone use, rather than the specific accuracy of these measures in their own right. For example, we are more interested in how well questions about mobile activity that occurred yesterday generally measure frequency of mobile phone use than we are interested in how well they accurately reflect activity that occurred on that specific day. For this reason these results do not apply to studies for which it is important to have an accurate measure of activity occurring on a specific day. This is not to say that these “yesterday” measures are necessary accurate–just that we cannot comment on their accuracy based on these findings. Third, although our data comes from a subset of a nationally representative sample, it is possible that groups who systematically avoid participating in these surveys answer self-report questions differently than those who participate. However, we find that the demographic composition of the subsample is mostly accounted for by the fact that respondents in the subsample were customers of the telecommunications company funding this study and customers of this company tended to have these demographic traits. Another issue is that this study is founded on behavioral data in addition to a few demographic variables. The data that was available did not include eventual motivational questions because the questionnaire was focused on other issues. This does not mean that there is not an interesting link between motivational variables and actual traffic data, only that it was not a part of this study.

Finally, although all respondents who participated in this study report that they are the sole users of the phones for which we collected the log data, it is possible that family members or friends may occasionally borrow their phones thereby decreasing the validity of the log data. We have little reason to believe that this is a common problem in our data, however, such an error may have occurred in some cases.

Discussion and Conclusion

Our results indicate that while self-report indicators aimed at measuring the frequency of mobile phone use are not completely out of line with actual behavior, our analysis shows that these self-report measures suffer from low criterion validity. That is to say, they are generally suboptimal ways of operationalizing this dimension of mobile phone use. These measures only moderately correlate with actual behavior, they vary more widely than actual behavior, and they are prone to overreporting. The categorical how often frequency measure that used predetermined response categories fared better than the continuous yesterday frequency measure that used an open-ended response approach. Perhaps this was because providing predetermined responses constrained respondents who would have more widely overreported, and cued them to an appropriate range of possible responses. The downside of using the categorical measure is that it can be more difficult to incorporate it into certain types of analyses and it provides less information than the continuous measure.

The results of this study have strong implications for the many studies that use these types of measures. As discussed in our analysis of the mobile phone literature, approximately 40 percent of mobile phone use studies published in scholarly communication journals base their findings on self-report frequency measures. Following from the notion of W.I. Thomas, our perception of use may be real in its consequences. That is, if someone thinks that they are a heavy or a light user of mobile communication they will likely arrange other contingent activities based on that perception.

That said, given the moderate correlations between these self-report measures and log data there is good reason to be suspicious of studies that find significant correlations between these self-report measures and other variables. Moreover, the implications for future research are potentially serious when considering access to more accurate trace data–such as mobile phone log or server log data–is generally difficult, time consuming, and expensive to obtain. Researchers might improve self-report measures by asking respondents to access the voice and SMS logs of their mobile phones or find old phone bills during interviews or while completing paper or web questionnaires. We caution that, this approach may result in frustration and additional time to complete surveys as respondents may not easily be able to pull this information from their mobile phones or locate old phone bills. The use of diaries is also a possibility, though this also implies a greater commitment by respondents and more difficulty in gathering data. It is also possible to consider a type of proactive data collection system such as calling people at random times via their mobile phones and asking for the previous mobile-based interaction. Another approach would be to use mobile applications designed to pull log data from mobile phones and send it directly to researchers. For a detailed discussion of the merits of such an approach, see Raento, Oulasvirta, and Eagle (2009).

To inform current and future studies, we ran exploratory multivariate analyses to examine the possible correlation between demographic traits and under- and overreporting. We focused on demographic traits because they are generally present in most surveys and therefore are obvious targets for identifying and compensating for inaccurate self-reporting behavior. While we did find some associations between demographic traits and misreporting on the yesterday measures, we didn't find many associations with the categorical how often measure. Moreover, the low R2 statistics in our models indicate that demographic traits will not go far in helping to identify and compensate for under- and overreporting in both the yesterday and the how often measures.

In conclusion, our results point towards the need to improve self-report measures that operationalize frequency of mobile phone use. Given that these measures are often used in influential mobile phone studies, more resources should be dedicated to understanding the sources of their weakness and examining alternative measures that operationalize the same concept and can be easily deployed into survey questionnaires.

There are always boundaries in terms of funding, time, the constraints of the questionnaire method, etc. As a result we often muck along as best we can and we accept the results of others given the constraints that they work under. This ignores, however the larger question of the validity of the results. In a perfect researcher's world there would be access to valid and reliable data that would not threaten the privacy of the individual. However, we do not live in that world. We have to think about how to get the best data given the instruments and the constraints that we currently have and we have to think about better data collection methods. Finally, we need to think about the way that our results are reported and the caution needed in pushing the meaning of the results beyond what the data can support.

Note

  1. 1

    An individual who practices SIM switching will have SIM cards from several different operators and will switch between these during the day in order to take advantage of daily tariff differences between the operators

Biographies

  • Jeffrey Boase (jeffrey.boase@ryerson.ca) is an assistant professor in the School of Professional Communication at Ryerson University in Toronto, Canada. His research focuses on the relationship between communication technology and social networks. Further information and publications can be found at: http://www.ryerson.ca/∼jboase/

    Address: School of Professional Communication, Ryerson University, 350 Victoria Street, Toronto, Ontario M5B 2K3

  • Rich Ling is a professor at IT University of Copenhagen in Copenhagen Denmark. His work has focused on the social consequences of mobile communication.

    Address: IT University of Rued Langgaards vej 7, 2300 Copenhagen

Ancillary