Measuring what gets done: Using goal attainment scaling in a vocational counseling program for survivors of childhood cancer

Abstract Background Childhood cancer survivors face education and employment challenges due to physical, cognitive, and psychosocial effects of the disease and treatments, with few established programs to assist them. The objectives of this study were to describe the implementation of Goal Attainment Scaling (GAS) to evaluate an educational and vocational counseling program established for survivors of childhood cancer, and analyze patterns of program engagement and client outcomes, stratified by demographic and diagnostic characteristics. Methods A population‐based retrospective cohort study of childhood cancer survivors who were engaged with the Pediatric Oncology Group of Ontario's School and Work Transitions Program (SWTP) between January 2015 and December 2018 was utilized. Survivors were followed from SWTP engagement until May 30, 2019 to capture goal attainment. Individual goals were summarized across various demographic, disease, and treatment strata. Results In total, 470 childhood cancer survivors (median age = 17.9, 58% male) set 4,208 goals in the SWTP during the study period. The mean length of observation was 130.8 weeks (SD = 56.9). Overall, 68% of the goals were achieved. Eighty‐three percent of the goals related to further education. Clients diagnosed with a solid tumor set the most goals on average, followed by those with central nervous system tumors and leukemia/lymphoma. Conclusions The SWTP assists childhood cancer survivors in realizing their academic and vocational goals. Application of GAS in this setting is a feasible way to evaluate program outcomes. From the volume and breadth of the GAS goals set and achieved, the overall success of the SWTP appears strong.


| BACKGROUND
Most children diagnosed with cancer in developed nations will be cured. 1 Consequently, the size of the childhood cancer survivor population continues to expand. It is estimated that there are over 400,000 childhood cancer survivors (0.11% of the population) in the United States and approaching 500,000 in Europe, of whom 24% have survived more than 30 years. [2][3][4] In Ontario, Canada, the childhood cancer survivor population has been estimated to be 19,900 as of 2017. 5 Childhood cancer survivors are at an elevated risk for the development of multiple and diverse chronic health conditions. 6,7 Childhood cancer survivors are also at increased risk for difficulty achieving success in employment and education, related to neurocognitive changes as a result of the disease or its treatment. 8,9 Neurocognitive problems affecting learning and memory may be exacerbated by physical, emotional, and psychosocial effects of individual cancer experiences. [10][11][12] The Pediatric Oncology Group of Ontario (POGO), a collaborative network and official advisor to the provincial health system on childhood cancer control, launched what is currently called the School and Work Transitions Program (SWTP) in 2002, 13 to provide support for childhood cancer survivors in Ontario, Canada. The SWTP provides survivors of childhood cancer (hereafter referred to as "clients") support to mitigate barriers to education and vocational success. Services are provided largely through individual counseling but also through presentations, publications, and workshops for clients and other professionals; development of positive community partnerships; advocacy work and ongoing research to ensure continuous improvement of service to its childhood cancer survivor population. 14,15 Individual client needs and the diverse goals and client specific outcomes require tailoring of support services and render the measurement of SWTP success difficult. Although the SWTP demonstrated qualitative evidence of success in a small group of clients, 16 as the program grew it became more challenging to capture differences in service provision across counselors and sites, and consistent monitoring and evaluation of the program's success was lacking.
In an effort to better track individualized goals and related outcomes of each client, identify similarities and differences across clients, and to act as markers of success, the SWTP implemented Goal Attainment Scaling (GAS 17 ) in 2013. GAS, first introduced in the late 1960s, was developed as a general method to evaluate mental health outcomes. Soon, the general applicability of GAS as a measurement and evaluation tool for a variety of healthcare service delivery outcomes was realized. At its core, GAS enables quantitative analysis of qualitative data (i.e., measures personally meaningful change) resulting from many forms of intervention. GAS requires outcome scales to be developed that are tailored to the individual and specific to the context in which it is being applied. GAS has been extensively utilized in various aspects of rehabilitation across various age and patient groups including adult brain tumor patients. [18][19][20][21][22] The objectives of this study were to describe the implementation of GAS in the SWTP, analyze patterns of program engagement, and use GAS to measure outcomes among SWTP clients, stratified by demographic and diagnostic characteristics.

| METHODS
In this population-based retrospective cohort study, participants were childhood cancer survivors who were referred to and initially engaged (defined as responding to an initial contact request by a SWTP counselor) with the SWTP between January 1, 2015 and December 31, 2018. Participants were followed from date of initial engagement until May 30, 2019. This project was approved by the Research Ethics committee at the University of Toronto (Protocol No. 37234). This study is a secondary analysis of routinely collected data and was completed under a waiver of consent.
Referrals to the SWTP come through Pediatric AfterCare Clinics at tertiary hospitals throughout Ontario (specially designed medical clinics for childhood cancer survivors), other health care providers, and selfreferral. SWTP counselors regularly attend AfterCare Clinics to ensure clinic staff and potential clients have an understanding of the program and services offered to encourage and facilitate client referrals. The referral process requests that one or more referral reasons, selected from five broad areas (secondary school, post-secondary school, employment/volunteerism, scholarships, or community services) be identified which are then prioritized by the counselor and client. Following referral, potential clients are contacted by one of the SWTP counselors in the province and enrolled in the program if they are interested. Counselors work with clients to define and support their academic, vocational, or community services support goal(s) via in-person meetings, telephone calls, or through electronic correspondence. Clients with SWTP experience prior to 2015 were excluded from analyses as the GAS system was not fully implemented. Referred clients who chose not to engage with the program after initial contact were also excluded since no goals were set. The timing and amount of support provided to clients varies as client needs change.
GAS was used to quantify clients' progress in achieving goals. SWTP counselors worked with clients to set feasible goals that were client actions and were agreed upon by both client and counselor. Generally, the goals were personally meaningful and followed the SMART framework, that is, specific, measurable, attainable, relevant, and time-bound. 23 In this way, clients were aware of what was expected of them but were not explicitly informed that their actions were being scored, this ensured that clients did not feel self-conscious about being monitored or scored and given we were using GAS to evaluate program outputs and not individual clients. A default time frame of 2 weeks for goal attainment was the norm, with adjustment for circumstances that might demand a different time frame (e.g., college application deadlines). There were no restrictions on the number of goals a counselor could set with an individual client. Example goals include preparing/drafting a resume over 2 weeks or developing and submitting applications for post-secondary education over 4 weeks. Three levels of attainment from 0 (target reached) to +2 (achievement beyond the specified goal), and two levels of insufficiency −1 (some achievement but goal not reached) to −2 (no action toward goal) were specified. If the time elapsed without the client completing the goal, the counselor scored the goal −2 and would either re-establish the same goal with the client at the next interaction or reassess the client's ability to meet the original goal and modify it by breaking the goal into smaller actions to assist the client with attainment.
Members of the implementation team and all counselors underwent a one-day training session on the development and implementation of GAS goals. During this face-to-face, hands-on training counselors were able to develop goals using client scenarios, followed by group discussion on how the goals could be improved. As part of the SWTP, counselors meet regularly to discuss program issues and during the preliminary implementation phase, time during these meetings was devoted to collectively discussing GAS goals. Any questions that were not able to be addressed among their own community of practice were discussed with the implementation team. As new counselors joined the program, the SWTP Manager provided in-depth orientation and training to GAS goal setting to ensure consistency across the program.
To begin using GAS to measure SWTP outcomes, all client goals set in a preliminary period of 6-8 months were examined by two members of the implementation team (JDP and BW) to identify common domains of practice and goals set with clients across sites. The creation of standard goals was necessitated by counselor time constraints and the associated additional time needed to create each GAS goal de novo. After reviewing several hundred GAS goals set by five counselors, repetition in goals was evident. From this effort, 15 standard goals were established ( Table 1). The inclusion of the last standard goal permits the counselor to create a new goal should they determine that none of the standard goals are appropriate. In addition, four broad practice domains were identified to assist in collecting information on the context of the individual goals; (1) community resource, (2) education, (3) employment/volunteering, and (4) SWTP engagement. Each domain contained sub-domains, such as academic accommodation or course completion within the education domain, full details are provided in Table 4. These domains, sub-domains, and goals were integrated into the SWTP electronic charting system, so that counselors selected each via a series of drop-down menus and set the date for goal adjudication. This permitted the charting system to send electronic reminders to counselors to score the goal at the appropriate intervals.

| Data sources
There were two main sources of information for this analysis. First, the SWTP collected detailed records via a custom electronic charting system that assists in client record management to capture details of the GAS goals set and assessed. Cancer diagnosis and treatment information for SWTP clients was obtained from the POGO Networked Information System (POGONIS).

| Analysis
Overall, the analysis as planned was descriptive in nature with the number, type, and outcome of the GAS goals examined and stratified by demographic and disease factors. The analysis stratified participants into three broad diagnostic groups according to the first primary cancer (leukemia/Lymphoma, central nervous system (CNS) tumors, and other extracranial solid tumors). These groups were selected due to general physical and cognitive deficits experienced as resulting from diagnosis and treatment. Length of engagement with SWTP was examined by categorizing the duration into five periods: 0-<1, 1-<2, 2-<3, 3-<4, and 4+ years. Given that the length of engagement with SWTP was thought to impact the number and type of goals set, a stratified analysis was undertaken that examined the outcomes in the first and second half of the total length of SWTP engagement. Only clients that had GAS goals set within both first-half and second-half of their engagement period were considered for the stratified split-half analysis. Raw GAS scores were converted to T-scores following the Keresuk and Sherman method before all analyses. 24 Briefly, Tscores were calculated assuming an equal weight for each T A B L E 1 Standardized GAS goals and scores for the SWTP program. goal and accounting for the number of goals included in each T-score estimate (which varies based on the number of goals by each client or within various strata). Therefore, the T-score is a linear transformation of the sum of the individual goal scores represented on a scale from 0 to 100, with a mean of 50 and standard deviation of 10.

| RESULTS
There were 470 study participants (clients) who had their first referral and engaged with SWTP between January 1, 2015 and December 31, 2018. Of these, 259 participants had multiple engagements (re-engagement) with SWTP over the period of observation. Table 2 provides demographic and disease characteristics of the study participants stratified by diagnostic group. The majority of participants were diagnosed with leukemia/lymphoma (56.0%) followed by CNS (22.8%) and extracranial solid tumors (21.2%). At the time of SWTP referral, the median age was 17.9 years and this was consistent across diagnostic categories. Overall, the median age at diagnosis was 7.7 years, but this ranged from 6.1 years for the non-CNS tumors to 10.9 years for the CNS tumors. Relapsed disease occurred in 11.5%, with slightly higher relapse proportions noted in the CNS and solid tumor groups. Of the participants who did not have CNS tumors, 56.2% received CNS-directed therapy, defined as intrathecal chemotherapy and/or cranial radiotherapy. The proportion of CNS-directed therapy was highest in participants diagnosed with leukemia/lymphoma (71.5% overall; 90.7% among acute lymphoblastic leukemia patients and 78.6% among non-Hodgkin lymphoma patients), than solid tumors (16.0%). Just over 11% of participants received a hematopoietic stem cell transplant (HSCT). Forty-seven percent of the HSCTs were allogeneic and all occurred among patients in the leukemia/ lymphoma group. When considering differences in demographic and disease characteristics by length of engagement with SWTP (data not shown), those participants with engagements of 4+ compared to 1-2 years had a higher proportion of males (63.8% vs. 53.7%) and a diagnosis of leukemia/lymphoma (61.7% vs. 50.4%). There was a lower proportion of solid tumors (19.2% vs. 25.6%) and the median age at diagnosis was also lower (5.5 years vs. 9.8 years) in the 4+ versus 1-2 years engagement groups. Table 3 outlines the detailed GAS outcomes overall and stratified by domain and diagnostic group. In total, 4,208 goals were set among the 470 participants. On average, participants were followed for 130.8 weeks (SD = 56.9) setting an average of 9.0 goals per participant with 68% of these goals being achieved or surpassed (defined as a GAS score ≥0 or a T-score ≥ 50). Educational goals were set by most participants (83%), followed by SWTP engagement (55%) (reflecting re-engagement), employment/volunteering (29%), and community resources (8%).
The solid tumor group set the highest average number of goals (11.5 goals per person) followed by the CNS group (9.4) then the leukemia/lymphoma group (7.8). Goal achievement was similar among the diagnostic groups ranging from 67-70% achievement. The CNS group set the highest proportion of community resources goals (16%) followed by solid tumors (9%), and leukemia/lymphoma (5%), with each group having similar levels of goal achievement. Similarly, the CNS group set the highest proportion of education goals (87%) followed by leukemia/ lymphoma (82%) and solid tumors (80%), although the CNS group achieved less of their educational goals (67% of goals achieved) compared to the other groups (each with 70%). The solid tumor group set the highest proportion of employment/volunteer goals (35%), followed by leukemia/lymphoma (29%) and solid tumors (24%) with the leukemia/lymphoma group having lowest proportion of goal achievement (62%) followed by solid tumors (68%) and CNS (72%).
The split-half analysis assessing if the length of engagement with SWTP impacted on the number and type of goals set, utilized only 394 participants (some participants did not set goals in the first and second half of their engagement with SWTP therefore not permitting analysis T A B L E 3 GAS outcomes of n = 470 SWTP clients who engaged in the program with first referral date between 01JAN2015 and 31DEC2018.   *Achieved goal defined as a GAS score ≥ 0 versus <0.; **Split half analysis outcomes using n = 394 clients who utilized SWTP long enough to permit split-half analysis.

T A B L E 3 (Continued)
T A B L E 4 GAS outcomes for each sub-domain of n = 470 SWTP clients who engaged in the program with first referral date between 01JAN2015 and 31DEC2018.  Table 4 presents a similar analysis of the GAS outcomes, but examines each sub-domain stratified by diagnosis group. Overall, when considering goals set in the education domain, 29% (811/2837) were for postsecondary exploration and 20% were for scholarship applications. When examining by diagnostic group, the proportion of post-secondary exploration goals was greater than scholarship application for the leukemia/ lymphoma and CNS groups (32% and 19% and 28% and 16%, respectively) but was reversed for the solid tumor group (22% and 27%). Overall, when considering goals set in the employment/volunteer domain, 30% (200/675) were for employment support. When examining by diagnostic group, the proportion of employment support goals was similar for the leukemia/lymphoma (31%) and solid tumor group (32%) but lower for the CNS group (22%).

| DISCUSSION
The POGO's SWTP is a unique counseling support program for survivors of childhood cancer. Given that the structure of the program is fundamentally an individual one-to-one counseling experience, assessing global program outcomes is difficult. GAS was successfully introduced to the counseling service. SWTP created a standardized set of goals and integrated this into the electronic charting system, allowing us to characterize the academic and vocational goals of childhood cancer survivors, understand SWTP counselors' impact in helping survivors achieve their goals, and identify gaps in service provision. Application of GAS in this setting is feasible as evidenced by the high average number of goals set for each client.
The characteristics of the clients who engage with SWTP are reasonably representative of the childhood cancer survivor population in Ontario with the notable difference being the higher proportion of male clients. Whether this is due to referral bias, or that males seek the support offered by the SWTP more often than females is unclear and requires further research. Female participation has been more common in late-effects research and counseling intervention programs for childhood cancer survivors. 25 When considering the pattern of goals set and the success of achieving them, it was surprising that the CNS group did not have the highest number of goals set or the lowest success. Given the often complex physical, cognitive, and social deficits that are associated with surviving a CNS tumor, 29 and anecdotal evidence from the counselors, we would have expected a different pattern. However, the limited goals may reflect clients' lower achievement expectations and the successes attained may be due to the counselor's ability to break goals into smaller parts to facilitate achievement. It is encouraging to know that both the SWTP and the GAS system are flexible in implementation to be sensitive to the unique needs of this patient population. Similarly, as expected the leukemia/ lymphoma patients set the smallest number of goals but unexpectedly had goal success similar to CNS patients and below solid tumor patients. Given many leukemia/lymphoma patients experience fewer relapses and secondary malignancies when compared to the solid tumor patients, we would have expected higher goal achievement in this group. Still, higher levels of exposure to CNS directed therapy compared to the solid tumor patients may impact goal achievement. Future research should explore possible mechanisms.
As expected, the number of goals set in the first half of the client engagement period was higher than the second half possibly owing to counselors making the initial goals smaller and less complex until they learn more about the individual client and the client's personal commitment to achieving the goals.
Previous work has identified that childhood cancers survivors utilize special education services nearly three times more frequently than sibling controls and overall educational attainment is reduced and related to specific diagnostic and treatment sub-groups. [30][31][32][33][34][35][36] It is clear when looking at the sub-domains, client's access SWTP most often for assistance with post-secondary education (e.g., exploring options, courses, and scholarship applications). Providing connection to postsecondary education was one of the driving forces at the inception of the program. It appears that the SWTP is meeting this need.
The small number of goals set in the community resources domain should not be interpreted as a lack of interest by the clients for assistance in these areas.
Although only a small number of clients set goals in this area, access to these community services is vital and can be complex. Clients may not be aware of available services and resources in the community and the SWTP program serves a special niche in providing clients with assistance in identifying and engaging with these difficult to access resources.
A 2015 review highlighted the special psychosocial, health-promotion, and neurocognitive needs of adult survivors of childhood cancer. 12 Within this systematic review the authors were not able to identify any studies that aimed at supporting educational or employment outcomes congruent with the goals of the SWTP. Identified studies examined neurocognitive interventions aimed at improving educational performance that most often focused on increasing working memory, cognitive flexibility, and attention as well as psychosocial interventions focusing on social competence. All interventions were short, lasting only hours to weeks and none provided the long-term one-to-one counseling that is characteristic of the SWTP. A recent review examining education support noted substantial proportions of childhood cancer survivors received education support, but most supports lacked any evaluation of their effectiveness thereby calling into question their utility. 37 Given the paucity of literature in this area, it is difficult to find outcome evaluations suitable for comparison to those presented herein.
The SWTP offered to survivors of childhood cancer in Ontario, Canada appears to be unique. The program has good engagement with clients, who come to the program with multiple goals. The overall outcome of the program appears positive, is valid across diagnoses and for several groups of issues. The success is likely a result of the oneto-one counseling model where client-counselor relationships can drive individual progress toward education and employment goals to bolster and support survivors toward a productive and rewarding future. Similar programs to the SWTP delivering direct-to-client services to survivors of childhood cancer should be considered as part of routine long-term follow-up care.