Success of the new human papillomavirus (HPV) DNA test for low-resource settings (careHPV™ test; QIAGEN Gaithersburg Inc., Gaithersburg, MD) requires good test performance when operated by personnel with limited laboratory experience. We evaluated the transferability, reliability, and accuracy of the careHPV test nested within a cervical screening project in a large Nigerian village. CareHPV testing was performed on screen-positive (n = 345) and screen-negative (n = 42) women attending colposcopy (68.3% of referred). Biopsies of abnormal-appearing areas were processed and read in the U.S. CareHPV specimens taken immediately before colposcopy were processed up to four times (in the field) by two secondary school graduates without laboratory experience, trained for this study. Specifically, QIAGEN Gaithersburg trained a laboratory-inexperienced U.S. researcher, who trained the first local technician who, in turn, trained the second. Residual specimens were sent to the U.S. for MY09/MY11 PCR testing for 13 carcinogenic genotypes (HPV16, 18, 31, 33, 35, 39, 45, 51, 52, 56, 58, 59, 68) plus HPV66 (included in careHPV). Intrarater agreement was 98.8% (κ = 0.97) and 98.9% (κ = 0.97) for Technicians 1 and 2, respectively, while inter-rater agreement was 96.3% (κ = 0.90). Agreement with MY09/MY11 PCR (virologic reference standard) was 89.3% (κ = 0.73) with 74.2% sensitivity and 95.7% specificity. The careHPV test detected 12 (80%) of 15 histologically confirmed cervical intraepithelial neoplasia Grade 2 (CIN2) or worse lesions, with an estimated 83.0% specificity to detect <CIN2. In a challenging low-resource setting with minimal intervention, the careHPV test performed adequately with high specificity but possibly lower sensitivity than HPV DNA tests currently used in controlled situations.
In 2008, over 85% of an estimated 530,000 new cases of cervical cancer were diagnosed in less developed countries.1 Virtually, all cases of cervical cancer worldwide are caused by persistent cervical infection with one or more of approximately a dozen carcinogenic genotypes of human papillomavirus (HPV).2–4 Although effective prophylactic HPV vaccines have been developed, they have relatively high cost and require a three-dose regimen.5 Moreover, they do not treat pre-existing infections and related disease. Fortunately, with adequate screening and treatment, the vast majority of cervical cancer can be prevented during the typical precancerous period lasting 10–15 years or more before invasion.6, 7 Yet the conventional screening model (Pap smear followed by colposcopically directed biopsy) is neither sufficiently developed nor sustainable in most countries with limited resources.8
Screening for the presence of one or more carcinogenic HPV genotypes is appealing due to high sensitivity and negative predictive value as well as superior reliability compared with other screening methods.9, 10 HPV testing is amenable to self-sampling outside the clinic, which allows expanded population coverage.11 Also, whereas cytology has well-documented and logistical limitations, HPV DNA tests continue to improve and are now being developed for low-resource settings.
Successful HPV screening programs in less developed countries require affordable and simple HPV DNA tests that will perform well in remote settings. One such test is the careHPV™ test (QIAGEN Gaithersburg, Inc., Gaithersburg, MD).12–14 Widespread implementation to emerging global markets in less-developed countries will likely require personnel without extensive laboratory background (i.e., training of a trainer) themselves to set-up and train operators of the devices. We sought to assess the practicality of establishing and running the careHPV test in a rural setting using local personnel without laboratory experience and evaluate the reproducibility and accuracy of the careHPV test compared with an established HPV DNA polymerase chain reaction (PCR) test and a histopathologic diagnosis.
Material and Methods
Project Itoju is a population-based HPV prevalence study conducted in Irun, a large rural Nigerian village. The careHPV test was performed in the second phase for women referred to a diagnostic colposcopy examination. Details of the enrollment screening phase are described elsewhere.15 Briefly, we consented and enrolled ∼1,500 women aged 15+ who attended an initial screening visit. At the screening visit, locally trained nurses conducted a cervical exam on nonvirgins, which included visual inspection with acetic acid (VIA) and the collection of a cervical specimen used for liquid-based cytology (LBC) (prepared and read in the United States) and PCR-based HPV testing (also done in the United States). Women with any positive screening result (VIA positive, non-normal LBC or PCR positive for carcinogenic or possibly carcinogenic HPV types) and a random selection of screen-negative controls (VIA negative, normal LBC and PCR negative for carcinogenic or possibly carcinogenic HPV) were referred to colposcopy at a local clinic in Irun established for this study.
At the colposcopy visit, a specimen for careHPV was taken from all women using a digene® cervical brush (QIAGEN Gaithersburg) and placed into 1 ml of careHPV collection medium (CCM). Subsequently, up to four acetowhite areas were biopsied. An endocervical curettage (ECC) specimen was taken if the cervical transformation zone was not visualized by colposcopy and/or the woman was aged 40 or older. Biopsy and ECC specimens were preserved in 10% formalin and later processed and read in the United States. CareHPV specimens were stored at ambient temperature (25–31°C) and tested four times within 2 weeks. Residual specimens were sent frozen to the United States and tested for the presence of any carcinogenic HPV using the same research PCR assay with which enrollment specimens were tested.
CareHPV training began at the QIAGEN Gaithersburg laboratory where a National Cancer Institute (NCI) researcher without laboratory background received training in the careHPV test according to a training protocol developed by Research and Development at QIAGEN Gaithersburg. The training protocol involved the following steps: (i) explanation and overview of careHPV technology, (ii) demonstration of the operation and maintainence of careHPV equipment, (iii) one run by the trainer to demonstrate steps of the assay, (iv) side by side runs by the trainee with trainer observation and assistance until the trainee felt comfortable and (v) an evaluation run with a training panel of targets of known HPV DNA concentrations to demonstrate reproducibility and accuracy. The following month, the NCI scientist spent 3 days training a local technician in a simple office at the Obafemi Awolowo University in Ile-Ife, Nigeria. This first technician successfully passed the training and demonstrated competency by correctly running a training panel unaided four times. The first technician then spent 3 days independently training a second technician who successfully passed by correctly running the same training panel with the test unaided five times. Both technicians had secondary school training and neither had laboratory experience.
The first and second technicians each tested the same specimens in two separate batches. The simple careHPV machine will fail the batch rather than provide results if the internal control results are outside valid range. If a batch failed, all specimens in that batch were tested again and the results of the subsequent batch were considered in the analysis.
We note some exceptions to the standard, simple careHPV test protocol. Before the technicians moved the careHPV system to Irun, 14 specimens were processed at the office in Ile-Ife. CareHPV kits were stored in Ile-Ife, as opposed to the on-site Irun clinic. To avoid early expiration due to the long duration of the project, and lacking the capability for repeated importations, kits were sometimes stored at ∼25°C. This is lower than the ambient temperature that approached 30°C. In several circumstances when the technicians were not available, careHPV could not be performed within 2 weeks after collection, requiring that the specimens be frozen at −20°C before processing (n = 56). The technicians were not completely unsupervised; NCI researchers and QIAGEN Gaithersburg scientists conducted quality control remotely as electronic log files were periodically extracted and sent via internet for review. Midway through the study, the careHPV test controller showed failing results for the first technician. A review of log files indicated poor performance [low positive calibrator mean to negative calibrator mean (S/N) ratios and high coefficients of variation]. The technician admitted losing interest in the study and the second technician alone tested the remaining specimens. Finally, we note that careHPV test kits and CCM were not commercially available but were manufactured by Research and Development at QIAGEN Gaithersburg. In addition, the careHPV equipment system was a pilot model and not the final commercial system.
To measure test reliability, we calculated agreement and kappa (κ) statistics within technicians and between technicians. To calculate agreement between the HPV PCR and the careHPV results, we considered the first technician's first result or, if not available, the second technicians' first result. For the comparisons of the careHPV test to PCR, we considered specimens PCR positive if one or more of the 13 carcinogenic types (HPV16, 18, 31, 33, 35, 39, 45, 51, 52, 56, 58, 59, 68) plus HPV66 (included in careHPV) were detected. Virologic sensitivity and specificity of the careHPV test were calculated using the HPV PCR results as a reference standard.
The careHPV test results were compared with a clinical reference standard of the worst histology from colposcopically guided biopsy and excisional treatment. Sensitivity was calculated among women with a histology result from concurrent biopsy or excisional treatment of cervical intraepithelial neoplasia Grade 2 (CIN2) or worse. Specificity to detect women without confirmed high-grade disease was also calculated. Because the careHPV test was performed at the referral colposcopy visit among a subset of the screening population, our estimate of test specificity was adjusted to apply to all women attending the screening visit in the first phase of the study. Specifically, we extrapolated the age-stratified test performance of careHPV among randomly selected screen-negative controls (women with negative screening results who were randomly selected to invite to colposcopy) to all remaining women with negative screening results in the cohort. Compliance with colposcopy visit among women with negative screening results was assumed to be the same as that observed among randomly selected screen-negative women, after stratifying by age.
In sensitivity analyses, we considered the age-stratified performance of the careHPV test among women within the age range (30–45 years) considered for HPV-based screening programs. Older women are excluded because the treatment available in resource-poor settings, cryotherapy, is not effective in the older population. Analyses were performed using Stata 11.0 software (Stata Corp LP, College Station, TX).
The protocol was reviewed and approved by both the Nigerian and the NCI institutional review boards (NCT 00804466). Appropriate treatment among positive women with diagnosed cervical precancer and cancer is now underway, using a combination of cryotherapy, loop electrosurgical excision procedure (LEEP), and conization for HPV and CIN and more advanced treatments for invasive cancer.
A total of 387 women attended the colposcopy visit [345 of 500 (69%) referred for any positive screening result and 42 of 67 (63%) randomly selected screen-negative controls, p = 0.3] (Fig. 1). Mean and median times between enrollment and colposcopy visit were 10.7 and 11.2 months (range: 5.4–17.4). The first and second technicians processed 47.6 and 98.3% of specimens at least once, respectively. After initial difficulties, most runs of the careHPV test were successful. Specifically, during the first 2 months of running the careHPV test, we experienced complications with the reagents and a fire occurred in the storage room. Therefore, ten of the 20 runs during this time were invalid. Subsequent to a new shipment of reagents, the first technician successfully ran ten of 15 tests with the last four tests failing as an indicator of poor technician performance described earlier. The second technician successfully ran 28 of 29 runs with one incomplete test due to a power failure.
Considering only the results that passed the machine quality control, agreement within technicians was high (98.8% of 81 and 98.9% of 348 for the first and second technicians, respectively, with corresponding κ values of 0.97 for both technicians) (Table 1a). Agreement between technicians was also high at 96.3% (κ = 0.90) (Table 1b). Despite high agreement, the first technician had significantly more positive results than the second, when the two disagreed (6 vs. 1, p < 0.01).
Table 1. Intrarater and inter-rater agreement of the careHPV™ tests in women age 18–85
PCR and careHPV results (result of first test of first technician, else first test of second technician) were available for 299 women (Table 2a). Among all women, agreement was 89.3% with a κ of 0.73. The careHPV test was less likely to test positive than PCR for carcinogenic HPV (25.1 vs. 29.8%, respectively, p < 0.01). Among 23 discordant cases where PCR was positive and the careHPV test was negative, one woman had a histologically confirmed CIN3 lesion (HPV18 positive). Among nine discordant cases that were PCR negative and careHPV positive, one histologically confirmed CIN3 was detected. Considering PCR results as a virologic reference standard, careHPV correctly identified 74.2% of 89 women infected with carcinogenic HPV and 95.7% of 210 women without carcinogenic HPV infections. Agreement varied by age (Table 2b). When considering results for 111 women within the age range typically considered for screen and treat strategies using HPV testing and immediate cryotherapy (30–45 years), agreement was 92.8% (κ = 0.74, p = 0.15).
Table 2. The careHPV test compared with the MY09/MY11 PCR test from the same specimen
Table 3 presents the performance of the careHPV test compared with the histopathology results. Of 15 women with a confirmed CIN2 or worse (six CIN2, eight CIN3 and one cancer), 12 (80%) tested careHPV positive. The three women testing careHPV negative had relative light unit (RLU) values of 0.647 (HPV18, 35 and 53 by PCR), 0.513 (PCR missing) and 0.303 (was also PCR negative). Among 367 women without histologically confirmed CIN2 or worse lesions, careHPV specificity was 78.5%; highest for women aged 30–45 at 85.3% (p = 0.04). When imputing to the entire screening population, the estimated specificity if women without a positive screening result and not randomly selected for colposcopy had been invited (n = 815) was 83.0% and the estimated negative predictive value was 97.9%. For comparison, the estimated HPV PCR test sensitivity and specificity were 84.6% (of 13 cases of CIN2 or worse) and 72.7% (of 286 women with less than CIN2), respectively (data not shown). The imputed specificity of the HPV PCR test was 83.5%.
Table 3. Women's risk of CIN2 or worse by careHPV result
In a validation study with real-world training and testing, the careHPV test had high intrarater and inter-rater agreement. When comparing the careHPV test with HPV DNA testing by PCR, we found that the assays had good agreement, particularly among women within the age range typically considered for screening (30–45 years). However, using the PCR test as a reference standard for careHPV, sensitivity to detect women with carcinogenic HPV was 74.2% (95% CI: 63.8–82.9%) and specificity to detect women without carcinogenic HPV infection was 95.7% (95% CI: 92.0–98.0%).
Despite examining 387 women in the context of colposcopic referral (most after an abnormal screening result), relatively few histologically confirmed cases of high-grade lesions were detected (n = 15) in this rural Nigerian screening population, providing inadequate power to accurately measure careHPV sensitivity. An initial study of the careHPV test among 2,530 women in China reported a sensitivity of 84.3% (95% CI: 75.8–92.8%) to detect 70 women with histologically confirmed CIN2 or worse, similar to our estimation of sensitivity at 80.0% (95% CI: 51.9–95.7%).12
In this study, we estimate that careHPV is 83.0% (95% CI: 82.2–87.2%) specific for identifying women without high-grade disease, similar to that reported by Qiao et al. of 87.5% (95% CI: 86.1–88.8%).12 The estimated negative predictive value of careHPV was 97.9% (95% CI: 96.6–98.8%) and would provide assurance to women testing negative of their minimal risk of cervical precancer. Indeed, the high negative predictive value even over periods of 10 or more years means most (80% or more) older women testing HPV negative are provided reassurance of minimal risk. Such reassurance means women can be screened only one or two times in their lifetime while clinical management (e.g., a second test, colposcopy or immediate treatment) can be focused on those who test HPV positive.16
We evaluated the performance of the careHPV test in a low-resource rural setting anticipated to represent the actual implementation of the test. In the first careHPV studies,12–14 highly skilled QIAGEN Gaithersburg scientists travelled to the study locale to set up the careHPV system and train local technicians using a 3–5 day training protocol followed by an additional 5–7 days of observation of clinical testing. In this study, QIAGEN Gaithersburg staff did not train the technicians or set-up or closely monitor the laboratory as in other Research and Development trials.12–14 Kits (including reagents) were stored according to QIAGEN Research and Development recommendation but not consistently refrigerated prior to testing.
Overall, we experienced little difficulty training technicians. In fact, the second technician, who was trained by the first one, eventually outperformed the first. In this study, we were able to monitor performance remotely through remote review of log files stored within the careHPV test controller, a quality control mechanism available only in Research and Development trials where the controllers were modified to allow access to the log files via USB port. It is noteworthy that in separate occasions when the first technician's performance waned and when some careHPV kits were exposed to high temperatures in an accidental fire, the test controller correctly indicated failure, showing problems with reagents or technician performance.
Outside of the technical functioning of the careHPV test, we did encounter errors with laboratory technicians. These included mistakes transcribing the specimen ID from the tube to the form as well as transcribing the careHPV result from the controller to the form. These problems were resolved through minor protocol modifications and the subsequent errors were minimal.
Although the colposcopy clinic was in the same location as enrollment screening and held at various days and times, we achieved only 69.0% compliance with follow-up colposcopy despite repeated direct intervention and a concerted community effort. In particular, we focused our final efforts to bring back 33 women with a high-grade or cancer cytology result, and 14 (42.4%) still did not attend colposcopy. Such poor compliance and limited access to health services may warrant forgoing colposcopic confirmation of lesions in exchange for immediate treatment with a safe, effective and low-cost technique such as cryotherapy.
Our results suggest that the careHPV test performs well in challenging remote settings with minimal interventions and has high specificity and adequate sensitivity. Incorporating careHPV into screening must now face the larger programmatic challenges of implementing large-scale programs using a batched test such as: access and affordability, transportation, lack of constant power supply in low resource settings, participant organization and follow-up and the likely need for self-sampling to make the screening program practical in some settings.17
This work was supported in part by the Intramural Research Program of the National Cancer Institute, National Institutes of Health, Department of Health and Human Services and NIH contract #HHSN261200900303P. For the careHPV™ test (QIAGEN Gaithersburg, Inc.; Gaithersburg, MD) component of this study, QIAGEN Gaithersburg trained an NCI researcher and donated and shipped reagents, equipment and supplies. NCI investigators (Julia Gage, Mark Schiffman) who lead the study were responsible for data collection and analysis and had final authority and independence with drafting the manuscript. Philip Castle serves on a Data and Safety Monitoring Board for Merck and has received HPV tests and testing for research at a reduced or no cost from Qiagen and Roche. Mark Stoler has served as a consultant to the manufacturers of HPV tests including Qiagen, Roche, Gen-Probe, BD, Ventana and others. Paul Eder worked at Qiagen during part of this study and owns Qiagen stock.