Contribution from the Field
Effect of handwritten, hand-stamped envelopes on response rate in a followup study of hip replacement patients
Mailed surveys are valuable and efficient research tools, but their validity hinges on an adequate response rate. Low response rates open the possibility of nonresponse bias, which is particularly concerning in longitudinal studies of the elderly because the response rate is already threatened by attrition from death, illness, and cognitive disability.
We recently surveyed patients who had undergone total hip replacement (THR), and only 55% of patients accepted our invitation to participate (1). This experience prompted us to consider methods for improving acceptance rates. Several such methods have been reported, including individually addressed letters and envelopes (2, 3) and incentives. Although material and monetary incentives appear to increase response rates (4, 5), there has been limited study of whether handwritten, hand-stamped envelopes increase participation. Existing studies of this strategy have had conflicting results, are not easily generalizable, and have not considered the cost effectiveness of handwritten, hand-stamped mailings (6, 7). We hypothesized that handwritten, hand-stamped envelopes would improve acceptance rates in a followup survey performed within a longitudinal study of elderly THR recipients.
We designed a randomized trial to test this hypothesis. The principal aim of the trial was to test whether handwritten, hand-stamped envelopes containing the letter of invitation would increase the rate at which respondents accepted our invitation to participate in a survey. The secondary aim was to determine if handwritten, hand-stamped envelopes containing surveys increased the rate of survey completion. In addition, we estimated the cost effectiveness of handwritten, hand-stamped envelopes as compared with standard mailings.
Study design and patient sample.
This randomized trial was fielded in the context of a longitudinal study of 919 Medicare recipients who had primary THR in 1995 and had previously accepted our invitation to participate in a survey in 1998 (1). The survey that forms the basis of this trial was done in 2001, 6 years following THR, and asked primarily about patients' pain, functional status, satisfaction, complications, and general health. The study focused on patients having elective procedures and therefore excluded patients with infection of the hip, metastatic cancer or bone cancer, conversion of hemiarthroplasty (or other hip surgery) to total hip replacement, and hip fracture. We randomized 300 of the participants to receive their letters of invitation to participate in the study with handwritten addresses and postage stamps. We sent the rest of the cohort (n = 619) identical letters of invitation contained in envelopes with computer-printed addresses and institutionally metered postage. The participants within the handwritten group were then further randomized to receive their surveys in handwritten and hand-stamped envelopes versus computer-printed and metered envelopes, should they agree to participate.
We adhered to a survey protocol mandated by the Center for Medicare and Medicaid Services (CMS) for surveys of Medicare beneficiaries. First we sent a letter of introduction on CMS letterhead. A week later, we mailed a letter of invitation from the research team on our own letterhead. Nonresponders were sent up to 2 additional letters of invitation, with a month between each letter. Those subjects who accepted could opt to do the survey by phone or mail. Those opting for mail surveys received up to 2 surveys, a month apart. For both the intervention and the control subjects, we used a common mail-merge program (MS Office; Microsoft, Redmond, WA) to apply the subjects' names and addresses on the letters of introduction. All letters were hand-signed in blue ink. As a gesture of appreciation, we included a book of stamps with the first letter of invitation to both the intervention and the control groups.
We examined the success of randomization in distributing several characteristics between experimental and control groups, including age, sex, education, and residency type (urban or rural). Our primary outcome for the trial was the proportion of patients accepting our invitation to participate in the survey. Secondarily, we compared the proportion completing surveys among patients randomized to handwritten, hand-stamped mailings versus controls. We compared proportions with the chi-square test, using P = 0.05 as the critical value for statistical significance. The trial was designed to have 80% power to detect differences in acceptance rates (the primary outcome) of 5%.
Costs and cost effectiveness.
We estimated the additional costs incurred by the intervention group as compared with the control group. These costs included personnel time spent handwriting and applying stamps to envelopes. We estimated, from our experience, that 300 letters per hour could be assembled using printed address labels as compared with 150 letters per hour using the handwritten, hand-stamped approach. The personnel costs were based on a $15.00 per hour pay rate, corresponding with the wage plus benefits of a research assistant. We did not include costs of other materials (paper, computer use, and maintenance), postage, or institutional costs for delivering mail because these were identical for both groups. We examined the cost effectiveness of the handwritten strategy as compared with the usual strategy. We made these calculations separately for each mailing (e.g., the first, second, and third letters of invitation). We divided the incremental cost per individual by the incremental effectiveness. The latter was defined as the increase in number of participants agreeing to participate in a study in the intervention group compared with the control group.
In the same manner, we also assessed the cost effectiveness of the strategy of sending a second (and third) letter of invitation to patients who did not respond to the first. This analysis was done in the control group.
The Human Investigation Committee of the Brigham and Women's Hospital approved the study. None of the authors had a conflict of interest.
Thirty-six percent of participants were male, 36% were >75 years old, 21% lived in a rural area, and 20% had less than a high school education. The intervention and control groups did not differ with respect to age, sex, rural versus urban residency, or education (P ≥ 0.20 for each of the comparisons).
There was no statistically significant or meaningful difference between the control and experimental groups in the cumulative proportion of subjects who accepted our invitation to participate after 3 letters of invitation (84.2% in the control group versus 83.7% in the experimental group; P = 0.85; Table 1).
Table 1. Acceptance rates stratified by number of letters of invitation sent to intervention and control subjects
Survey completion rates
Survey completion rates were not statistically significantly different between intervention and control subjects. Fully 99.1% of subjects receiving their surveys in handwritten, hand-stamped envelopes completed the surveys, as compared with 96.3% of subjects receiving their surveys in computer-written, metered envelopes (P = 0.23; Table 2).
Table 2. Survey completion rates stratified by number of surveys sent for intervention and control subjects
After a single letter of invitation was sent, the handwritten, hand-stamped strategy cost $30.95 more than the control strategy, and yielded 21 additional acceptances, for an incremental cost effectiveness of $1.47 per extra acceptance. The incremental cost effectiveness of hand addressing and hand stamping a second letter of invitation was $3.03 per extra acceptance. We did not calculate a cost-effectiveness ratio for the third letter of acceptance, nor for the overall strategy based on 3 letters, because there was a lower yield associated with handwriting and hand-stamping the third letter and overall than with the usual computer-addressed, metered letter.
We also estimated the cost effectiveness of sending additional letters of invitation within the control group to subjects who did not respond to the first. The incremental cost of the second letter was $8.80, and each second letter resulted in 77 acceptances, for an incremental cost effectiveness of $0.11 per additional acceptance. Similarly, the incremental cost effectiveness of a third letter of invitation was $0.20 per acceptance.
This randomized controlled trial found no significant difference in survey acceptance rates with handwritten, hand-stamped envelopes compared with standard computer-written, metered envelopes. Furthermore, there was virtually no difference in survey completion rates between those randomized to receive surveys in handwritten, hand-stamped envelopes versus the standard mailing. Of note, this cohort had already agreed to participate in the first wave of the study 3 years prior to the 2001 survey. Thus, the findings apply most directly to longitudinal studies that involve the administration of serial followup surveys.
We also examined the cost effectiveness of the handwritten, hand-stamped technique and of the use of serial letters of invitation for patients who do not initially respond. Our findings suggest that investigators can expect to spend $1.47 per additional acceptance if they handwrite and hand-stamp the first envelope, and $3.03 per additional acceptance if they handwrite and hand-stamp the second. The findings may be useful to investigators who undertake longitudinal survey studies and must decide how to allocate project resources. Our findings also suggest that sending a second letter and even a third represents a modest investment—11 cents per additional acceptance for the second letter and 20 cents per additional acceptance for the third—when computer-written and metered envelopes are used.
Several prior studies have found that characteristics of the envelope containing the letter of invitation affect survey response rates. In one survey of academic internists, a Veterans Administration envelope was associated with a 20% higher response rate than a university envelope (8). Other investigators found that placing a large commemorative stamp on the return envelope boosted response rates by 31% (9).
We found only 2 other studies from the last 10 years that investigated specifically whether handwritten envelopes enhanced response rates. Gerace et al (6) recruited a cohort of women, 50–79 years old, into a clinical trial that involved dietary and medical therapy. Within this study, the investigators performed a trial of the effect of handwritten versus standard computer-addressed envelopes. The response rates were 6.0% and 8.1% for handwritten and computer written, respectively. This study cohort was approached for the first time, whereas ours had participated previously. Also, they were recruiting for a randomized trial, which involved a greater commitment of time and energy than filling out the brief survey required in our study.
Rimm et al (7) sent different types of mailings to nonresponders in the Health Professionals Follow-up Study, a longitudinal study of health professionals. These strategies included certified mail, United Parcel Service, window envelope with typewritten return address, typed address, handwritten address, and a window envelope with computer-printed address. The last was their standard reference mailing. The greatest response came from the group that was sent certified mail (63.2% versus 25.9% for the reference mailing; P < 0.001). Subjects sent handwritten envelopes also had significantly higher response rates than subjects who received computer-printed envelopes (41.6% versus 25.9% in the reference mailing; P < 0.001). This study was comprised only of nonresponders and was restricted to physicians, who typically have low response rates (10).
A recent study examined recruitment rates into an intervention trial of exercise on breast cancer biomarkers (11). Subjects who received invitations in first class mail were slightly more likely to respond that they were interested in the trial than subjects who received bulk mail invitations (8.1% versus 7.0%).
The principal strengths of our study are that the sample is population based and the design is a randomized controlled trial. The results apply most directly to serial surveys conducted in the context of longitudinal studies. For investigators who would like to increase response rates in subsequent waves of projects that include serial surveys, our findings suggest that handwritten, hand-stamped envelopes are unlikely to be useful. We cannot determine from our study whether previous survey experience influences willingness of persons to participate in subsequent surveys.
The events of September 11, 2001 and the subsequent anthrax scare raised serious concerns about the safety of the US mail. These issues may have influenced our study. For the first and the second letter (sent in the summer of 2001), the acceptance rate in the intervention group was 3–4% higher than in the control group. However after the third letter was sent out on September 9, 2001, the acceptance rates for the computer-printed letters were almost twice as high as for the handwritten. We do not know whether this difference was, in part, due to concern about safety of the mail, or whether it would persist today.
In conclusion, our randomized controlled trial found no benefit to the hand-stamped, hand-addressed letter of invitation. The handwritten, hand-stamped surveys were completed more often than those sent in standard envelopes, but the difference in completion rates (3%) was small. The findings will assist in the design and budgeting of serial administrations of longitudinal surveys, particularly those involving the elderly. We encourage other investigators to enlarge the base of evidence supporting survey methodologies.