Model Comparison and Results
Figures 1 and 2, FIGURE 2 compare the predicted number of deaths from lung cancer in women and prostate cancer in men obtained from the two statistical methods, PF and SSM, without any subjective selection of the most plausible estimate from PF. In Figure 1A, we used the data on observed number of female lung cancer deaths 1969 to 1994 to fit the two models, and we extrapolated one-, two-, and three-year-ahead projections for 1995, 1996, and 1997. Both models fit the observed data (1969 to 1994) very well. However, the predictions from the SSM model for 1995 to 1997 are closer to the observed data than are the predictions from the PF. Furthermore, as expected, the predictions from both models veer further away from the observed as time progresses. Because we did not want our validation to depend entirely on what was happening in any particular calendar year, the analysis was repeated for subsequent years. In Figure 1B, we used observed data 1969 to 1995 to extrapolate mortality numbers for 1996, 1997, and 1998, and in Figure 1C we used observed data from 1969 to 1996 to extrapolate mortality numbers for 1997, 1998, and 1999. Finally, Figure 1D shows how the actual projections would occur in practice, projecting out to the future where we currently have no data to validate the results. This panel uses the most recent available data (1969 to 2000) at the time this report was written and projects through 2003. Data from 1969 to 2001 are used in CFF 2004 to project through 2004.
Figures 2A to D illustrate the comparison of the SSM and PF models for prostate cancer. Before 1990, the SSM fits are more erratic than are those from PF, because of sensitivity to random error. However, after 1990, the projections from the SSM model are much closer to the observed values than are those from the PF method.
Table 1 compares the projected mortality counts for 1999, based on three methods (PF, CFF, and SSM) with the observed counts for eight cancer sites each in men and women. All the methods use observed data through 1969 and project to 1999 using three-year-ahead projections. Among the 16 sex-specific and site-specific projections, the SSM-generated projections were closer to the observed for nine sites, CFF-generated projections for five sites, and PF-generated projection for one site (PF and SSM were tied for one site).
Figures 3 and 4 compare the predictions from the PF, CFF, and SSM models for the years 1995 to 1999 for selected cancer sites using data up to three years before the prediction year. To make these figures comparable, although on different scales, the vertical axes are all drawn approximately ±25% from the average observed values. On this relative scale, the SSM follows the observed trend closer than do the other methods for male and female lung cancer, female breast cancer, male colorectal cancer, and prostate cancer. The best prediction overall is for colorectal cancer, followed by lung, breast, and prostate. Note that the PF and CFF methods in female colorectal cancer consistently underestimate whereas the SSM estimates bounce over and under the observed values. The extra variability of the SSM model in female colorectal cancer is a reflection of variation in past observations.
Table 2 shows a similar comparison for all cancer sites combined. The SSM model gives better predictions compared with both the PF and CFF methods for the years 1997 to 1999.
Using the squared deviation as the measure of error between the observed and predicted death counts, we compared the accuracy of the PF, CFF, and SSM methods. These quantities are non-negative and become larger with increased discrepancy, giving a proportionally greater penalty to larger discrepancies. We believe that this measure of deviation is appropriate because a large error seems more serious than several smaller ones. We calculated squared deviations of PF, CFF, and SSM from the corresponding observed values for each of the three-year predictions for the years 1997, 1998, and 1999 for the comprehensive set of cancer sites reported in Cancer Facts & Figures. The three-year predictions use data from 1969 until 1994, 1995, and 1996, respectively (with the exception of CFF, which uses data only from 1979). Then, for a particular site, the deviations so obtained are averaged for the three years. Table 3 shows the results for selected cancer site and sex combinations. In general, the average squared deviations for SSM are smaller than those for the PF and CFF methods.
Table 4 shows a summary of how the methods perform for a three-year period. The entries in the table are the squared deviations averaged over the comprehensive set of cancer sites reported in Cancer Facts & Figures.
In addition, to determine whether one method is better depending on the rarity of cancer deaths, we grouped all the sites into four categories according to the number of observed deaths in 1999. We averaged the squared deviations over all the cancers in a certain group for the years 1997 to 1999. Table 5 shows the results. In general, SSM performs better regardless of the rarity of the cancer.
Finally, Table 6 shows summary statistics for the comparison of the PF, CFF, and SSM methods when applied to the state-level data. The predicted values were adjusted so that the sum of the state predictions matches the corresponding national prediction. For each of the cancer sites listed, we averaged the squared deviations over all 50 states and the District of Columbia for the years 1997 to 1999.
Table 6 shows that the PF method performs better than the SSM at the state level (which in turn performs better than the CFF method), although the improvement is slight in most cases.