The Cost-effectiveness of Australia's Active After-school Communities Program

Authors


(marj.moodie@deakin.edu.au)

Abstract

The objective of this study was to assess from a societal perspective the cost-effectiveness of the Active After-school Communities (AASC) program, a key plank of the former Australian Government's obesity prevention program. The intervention was modeled for a 1-year time horizon for Australian primary school children as part of the Assessing Cost-Effectiveness in Obesity (ACE-Obesity) project. Disability-adjusted life year (DALY) benefits (based on calculated effects on BMI post-intervention) and cost-offsets (consequent savings from reductions in obesity-related diseases) were tracked until the cohort reached the age of 100 years or death. The reference year was 2001, and a 3% discount rate was applied. Simulation-modeling techniques were used to present a 95% uncertainty interval around the cost-effectiveness ratio. An assessment of second-stage filter criteria (“equity,” “strength of evidence,” “acceptability to stakeholders,” “feasibility of implementation,” “sustainability,” and “side-effects”) was undertaken by a stakeholder Working Group to incorporate additional factors that impact on resource allocation decisions. The estimated number of children new to physical activity after-school and therefore receiving the intervention benefit was 69,300. For 1 year, the intervention cost is Australian dollars (AUD) 40.3 million (95% uncertainty interval AUD 28.6 million; AUD 56.2 million), and resulted in an incremental saving of 450 (250; 770) DALYs. The resultant cost-offsets were AUD 3.7 million, producing a net cost per DALY saved of AUD 82,000 (95% uncertainty interval AUD 40,000; AUD 165,000). Although the program has intuitive appeal, it was not cost-effective under base-case modeling assumptions. To improve its cost-effectiveness credentials as an obesity prevention measure, a reduction in costs needs to be coupled with increases in the number of participating children and the amount of physical activity undertaken.

Introduction

Obesity interventions in the after-school hours care setting offer potentially greater returns than interventions in other child care settings given the large amount of discretionary time available to increase physical activity and potential to reduce snack food consumption during this period (1). Although there are a range of obesity initiatives in Australia focusing on after-school hours care programs (such as “Nikego” http:www.nike.comnikebiznikegolearn_whygo.jsp, “Eat Smart Play Smart” http:www.noshsa.org.aucurrent_issueseat_smart.htm, and “Nutrition Ready-to-Go at Out-of-School Hours Care” http:www.noshsa.org.aucurrent_issuesnutrition_project.htm), evidence-based research in the after-schools hours setting is limited.

In 2004, the then Australian Government allocated Australian dollars (AUD) 90 million over 4 years to an after-school physical activity program known as Active After-school Communities (AASC) program (2), as part of a package of primarily school-based measures designed to tackle childhood obesity. AASC was part of the wider “the Building a Healthy, Active Australia” initiative designed to address declining physical activity levels in children (3). The program was premised on the fact that about 40% of Australian children aged 5–14 years miss out on outside-school-hours sporting activity (4). In launching the program, the government cited evidence about the optimality of the after-school time slot for providing physical activity opportunities, and the daily recommended requirements for physical activity.

Given the former Australian government's commitment of substantial funding to the roll out of the program and the absence of other evidence-based interventions in the child care setting, the cost-effectiveness of the AASC project was modeled and evaluated as part of the Assessing Cost-Effectiveness in Obesity (ACE-Obesity) project (5) funded by the Victorian Government, Department of Human Services.

Methods and Procedures

Overview

The economic evaluation methods are detailed elsewhere (R. Carter, M. Moodie, A. Markwick et al., unpublished data) (6). A cost-effectiveness evaluation was undertaken from a societal perspective, and the incremental cost-effectiveness ratio (ICER) calculated as the cost AUD per BMI unit saved and disability-adjusted life year (DALY) saved. The intervention was modeled as if applied to the Australian population for 1 year in steady-state operation (i.e., it is implemented and working at its full effectiveness potential, and trained personnel or infrastructure are available). The time horizon for measuring the associated health-care cost-offsets and DALY benefits was rest of life or 100 years. All costs and benefits were discounted at 3% in accordance with the recommendations of the US Consensus Panel on cost-effectiveness (7). The reference year was 2001. In addition to the technical results, the intervention was assessed by the stakeholder Working Group (comprising funders, academics, obesity experts etc.) against a series of second-stage filter criteria in order to incorporate additional factors important to resource allocation decisions.

The intervention

The AASC program is auspiced by the Australian Sports Commission, and provides small grants to participating schools and organizations to deliver the program. Expressions of interest, including a school needs analysis, were invited from all Australian primary schools (government, Catholic, and independent) and child care benefit approved out-of-school hours care (OSHC) services operating during the 3:00–5:30 pm timeslot. Physical activity co-ordinators were appointed to work with national, state, and regional sporting organizations to develop and deliver a physical activity program specific to the needs of each school/service. Selected sites were required to offer 2–3 sessions per week, depending on student numbers, for 8 weeks for each of four school terms per year.

Current practice

The intervention focused on the recruitment of children who were previously inactive during the after-school period, so the current practice comparator equated with no intervention.

Assessment of benefit

The first stage of benefit assessment involved estimation of the health gain attributable to the intervention using the DALY, a metric which incorporates both mortality and morbidity impacts and facilitates comparison of the burden across different diseases. This required calculation of the increase in physical activity due to the intervention, conversion to BMI gain as children and then conversion to DALYs and cost-offsets over their lifetime (5).

A range of available data was used to model the likely change in the BMI of individual program participants (Table 1). The increased energy expenditure for a child new to physical activity in the after-school timeslot was calculated by subtracting the average energy costs of playing sport (5.0 metabolic units) from the energy costs of sitting quietly (1.0 metabolic unit) (8). The net 4.0 metabolic units was then multiplied by the assumed weight (kg) of the target age children and the assumed time spent in physical activity at the AASC program to derive the increased energy expenditure (kJ/day). Given the absence of definitive evidence it was assumed that there was no effect of the increased physical activity in the after-school period on energy expenditure at other times or on energy intake levels (9,10,11,12,13,14). The validated method of Swinburn et al. (15) was used to convert changes in energy balance to changes in weight. The resultant change in BMI was then converted to DALYs saved over the lifetime of the child using the methodology reported elsewhere (5).

Table 1.  Modeled data to estimate reduced BMI for a single “average” child who participate in the AASC program
inline image

As there were no definitive data available, it was assumed that 50% of program participants were not previously active after-school and would reap the benefit from their involvement. A further 20% were assumed to become more active in the after-school period (i.e., the program supplements other physical activity which they are already doing after-school), so a benefit was attributed. For the remaining 30% of children, the program was assumed to be a substitute for some other physical activity which they were previously doing, and no benefit was applied. This latter group would include those children attending the program at an OSHCs who previously had been participants in the centre's after-school care program which provided opportunities for physical activity.

The second stage of benefit assessment is dealt with later under the heading “second-stage filter analysis”.

Simulation of the intervention

The intervention was simulated on the basis of the service delivery model operated in Victoria by the Australian Sports Commission, although where information was available on differences between states, this was taken into account in the modeled cost-effectiveness analysis.

The program was open to all Australian primary schools and to child care benefit approved OSHCs, but was always delivered in the after-school hours time slot. Where a school delivered the program itself, this was either because it did not have a child care benefit approved OSHCs, or alternatively, it was predicted that the program was likely to attract more students than the onsite OSHCs was registered to handle. Although the actual intervention was provided at no extra cost to families, those attending the program at an OSHC were required to pay the after-school care fee.

The Victorian experience of recruiting schools/OSHCs to the AASC program was extrapolated to the Australian situation (Figure 1). The number of potential sites at a national level (3,300) was based on the AASC's appointment of 165 regional co-ordinators with an accepted average ratio of one co-ordinator per 20 sites. As sites were required to provide for a minimum of 15 and a maximum of 50 children per session, an average of 25 children per site was assumed. This translated to a total of 82,500 children involved in the program. The 70% who gained the program benefit were assumed to be equally spread across all grades from Prep to Grade 6.

Figure 1.

Recruitment to the AASC program. The number of children participating and the number receiving a benefit shown here do not take into account any uncertainty around these parameters. AASC, Active After-school Communities; OHSC, out-of-school hours care.

Assessment of costs

Pathway analysis was used to identify the component activities of the intervention in order to ascertain the associated resource utilization (Figure 2). The costs included, unit costs and their sources, and the assumptions employed are specified in Supplementary Table S1 online. All costs were adjusted to real prices in the 2001 reference year using the relevant consumer price index (16).

Figure 2.

Intervention pathway. AASC, Active After-school Communities; ASC, Australian Sports Commission; OHSC, out-of-school hours care.

Given a societal perspective, all costs to the health sector, participants and families, and other sectors involved in the delivery of the intervention were included (Figure 2). Because the intervention was assumed to be operating in “steady state” (that is, fully implemented without workforce or learning curve issues), costs associated with the initial research, design and set-up of the intervention and the pilot phase and the development of program materials were excluded.

Uncertainty analysis

Uncertainty analysis was conducted to address issues of uncertainty in the results due to sampling error and the need to make assumptions given the lack of evidence for some important parameters (Table 2). Simulation-modeling techniques (using the @RISK software and Monte Carlo simulations) were used to facilitate the presentation of a 95% uncertainty range around the health benefits, costs, and ICERs.

Table 2.  Uncertainty analysis
inline image

Sensitivity analysis

Sensitivity analysis was undertaken to test the impact of changing key design features of the intervention. The following scenarios were modeled as univariate sensitivity tests: (i) reduction in the ratio of sites per regional co-ordinator from 1:20 to 1:30; (ii) reduction in the number of state level co-ordinators from 18 to 11; (iii) application of the same wage rate to all site co-ordinators irrespective of the site being a school or OSHC; (iv) a combination of scenarios (i), (ii), and (iii); (v) all participants receive full intervention benefit.

Second-stage filter analysis

This involved consideration by the stakeholder Working Group of issues that either influenced the degree of confidence that could be placed in the cost-effectiveness ratio (“strength of evidence”), or broader issues that needed to be taken into account in decision making about resource allocation (“equity,” “acceptability to stakeholders,” “feasibility of implementation,” “sustainability,” and “potential for side-effects”) (R. Carter, M. Moodie, A. Markwick et al., unpublished data) (6).

Results

The marginal benefit per participant was a decrease of 0.07 BMI units for individual boys and girls in Prep to Grade 4 and 0.08 BMI units for boys and 0.09 for girls in Grades 5 and 6 (Table 3).

Table 3.  Cost-effectiveness results
inline image

Based on current assumptions, the ICERs were marginal in terms of net costs per DALY saved (AUD 82,000) (Table 3). Of the 3,000 iterations modeled, there was only an 8.9% chance that the ICER would be less than the cost-effectiveness benchmark of AUD 50,000 per DALY saved (with cost-offsets). The key sources of uncertainty around the ICERs were the number of children enrolled per school (r = −0.67) and grant funding per school (r = 0.399).

Of the intervention cost of AUD 40.3 million, the largest components were the costs of the regional co-ordinators, and the cost of the funding grants to selected sites. Apart from the fees payable by parents whose children were new to OSHCs, most costs fell on government and varied in relation to the number of sites recruited to the program.

Although each of the cost-cutting measures modeled improved the intervention's cost-effectiveness credentials, none of them alone were sufficient to bring it under the cost-effectiveness threshold (Table 3). If the intervention targeted only children who were previously inactive in the after-school timeslot, meaning that 100% of participants received the full intervention benefit, it would approach cost-effectiveness. However, this scenario is most unlikely.

A consideration of second-stage filters for the intervention is summarized in Table 4. The key decision points were around the lack of evidence of effectiveness, sustainability and potential for negative side-effects.

Table 4.  Second-stage filter analysis
inline image

Discussion

The AASC program was the major initiative in the Howard government's childhood obesity package; substantial levels of funding were directed to the program's roll out. Just recently, the current Federal Government acknowledged its support for the program as an obesity prevention strategy by recommending its continuation and expansion to more sites across the country (17). However, although the AASC program has intuitive appeal, it was not cost-effective in terms of its effect on obesity in children on the basis of the assumptions made in this modeled evaluation. Even when the cost-offsets arising from future reductions in obesity-related disease were taken into account, the ICERs did not come close to approaching the usually acceptable threshold level of AUD 50,000 per DALY saved. No comparative ICERs were available to place these results in a broader cost-effectiveness context.

The evidence around stand-alone physical activity interventions in the school setting is generally weak, and furthermore, most such studies do not assess the impact on overall physical activity levels nor have BMI as an outcome (18). Generally, evidence from studies within the school setting have shown no significant effect of physical activity programs on BMI (19,20), although there were promising results (in terms of reduction in skinfold thickness) from one Australian study involving high levels of physical activity (21). The evidence suggests that the level of physical activity required to significantly lower BMI is in excess of what a school today could manage within the curriculum. A recent study of an after-school program in Spain, showed that even with substantial extra physical activity (three 90-min sessions per week for 24 weeks), there was improvement in the children's body fat mass, but not BMI (22). Although there is a paucity of physical activity programs which show effectiveness in the school setting in terms of reducing BMI gain, there is a complete absence of such studies which have been subjected to economic evaluation (23).

Although the intervention was designed as an obesity prevention initiative, there would need to be considerable changes in terms of key design features to make it cost-effective from this respect. Its cost-ineffectiveness was a product of its high-cost structure, coupled with the small level of benefit gained per participant.

Furthermore, in the absence of available data, the assumptions made in the intervention modeling around program participation and effectiveness were generous. Broad assumptions were made about the numbers of children who might be new to physical activity in the after-school period. Although the program was intended to attract “inactive” children, no information was forthcoming on the strategies employed to gain their participation. There is also a potential danger that children already attending OSHCs may be discouraged from continuing to attend, which could have the effect of moving them to the “nonactive” group. However, despite these significant gaps in data availability, the ACE-Obesity Working Group considered it important to model the intervention given that it represented a major element of the then Commonwealth's government obesity strategy.

A major concern was the top-heaviness of the intervention in terms of costs, and in particular, its heavy reliance on a large number of regional co-ordinators. Scenario analysis was used to illustrate ways in which the intervention costs could potentially be reduced, but cost-cutting measures alone were insufficient to make the program cost-effective. Such measures would need to be combined with other initiatives to increase participation of previously inactive children both in terms of numbers involved and the time spent engaged in physical activity to render significant improvement in the intervention's performance. The modeling assumed that the program provided for 1 h of physical activity in accordance with the minimum requirement set down under the conditions of the funding grants to approved providers. Anecdotal evidence, however, suggests that the children are not necessarily active for the full hour, given that they spend some time receiving instructions, observing skills demonstrations, etc. However, even if the full hour was spent being “physically active”, this additional amount of physical activity is insufficient to make significant gains in terms of lowering BMI. The recent Spanish after-school study (22) offered higher levels of physical activity than the AASC program, yet resulted in no BMI differences between the intervention and control groups.

In terms of the second-stage filters, key decision points of the intervention were “strength of evidence,” “sustainability,” and “potential for negative side-effects”. The lack of data surrounding the program's effectiveness was a key limitation. Although data may be potentially forthcoming from the Australian Sports Commission evaluation about the numbers of participants new to physical activity after school, the mean benefit per child would still be reliant on modeling as participants were not subjected to “before” and “after” anthropometric measurements. The long-term “sustainability” of the intervention is also of concern, given the need for ongoing funding and support and reliance on a large complement of regional co-ordinators who account for 25% of the program costs. (The program continues to operate under the new Labor government.) There may also be some potential for negative side effects, although there is no hard evidence about this. Likewise, whilst there is scope for a number of positive outcomes from the program, none of these benefits have been proven.

To improve the AASC program's cost-effectiveness credentials as an obesity prevention measure, a reduction in the cost of operating the program needs to be coupled with a rise in the number of participating children, and an increase in the amount of physical activity undertaken.

SUPPLEMENTARY MATERIAL

Supplementary material is linked to the online version of the paper at http:www.nature.comoby

Acknowledgments

The Assessing Cost-Effectiveness in Obesity (ACE-Obesity) project was funded by the Victorian Government Department of Human Services, Australia. We thank members of the ACE-Obesity Working Group for their input into the project: Michael Ackland (Deputy Chair), Bill Bellew, John Catford, Elizabeth Develin, Helen Egan, Bonnie Field, Tim Gill, John Goss, Robert Hall (Chair), Brian Harrison, Kellie-Ann Jolly, Mark Lawrence, Amanda Lee, Tony McBride, Karen McIntyre, Jan Norton, Anna Peeters, Theo Vos, Melissa Wake, Rowland Watson. We thank the officers of the Australian Active After-school Communities program, Australian Sports Commission, for their input. The opinions, analysis, and conclusions in this article are those of the authors and are not necessarily endorsed by the Victorian Government Department of Human Services, the Victorian Minister for Health or the Government of Victoria.

Disclosure

The authors declared no conflict of interest.

Ancillary