Developing a sustainable child and family service system after a community tragedy: Lessons from Sandy Hook

Abstract This article describes a systematic approach to assessing community services post‐Sandy Hook shooting. An evaluation team was invited to develop a sustainability plan for community services in Newtown. Service organizations, providers, and families were interviewed. Descriptive statistics were used to characterize the range of services; respondent perspectives were coded using content analysis. We found that Newtown has a broad array of community services, but respondent groups varied in their perceptions of service adequacy. Consensus existed about core components of an ideal service system, including centralizing access; coordinating care, personalizing and tailoring services for families, and providing evidence‐based care. The strategic community assessment approach developed here may inform how communities examine their service capacity and develop sustainability plans post‐disaster.

than shootings (Cohen, Deblinger, Mannarino, & Steer, 2004;Kataoka et al., 2003;Stein et al., 2003). Trainings to support delivery of evidence-based treatments at scale (i.e., community or statewide) have also been studied (Hoagwood et al., 2007). Even so, community-based mental health services tend to be significantly underused by those most in need (Wang et al., 2007(Wang et al., , 2008. As a consequence of the increase in mass shootings and other disasters, disaster planning in communities has expanded, and there are many toolkits, trainings, and technical assistance supports available (American Academy of Pediatrics, 2012;American Red Cross, 2016, Federal Emergency Management Agency, 2016Brymer et al., 2006;Palinkas, 2012Palinkas, , 2015Shannon, 2015). However, these programs and supportive materials largely focus on identifying immediate and short-term community needs, training and deploying appropriate staff after a disaster, and safety issues. These efforts do not address service system issues, including the need to buttress existing services, construct a new and higher quality system responsive to the demands of the new healthcare environment, or address the longterm needs of communities that have experienced these tragedies. In fact, little is known about rebuilding or redesigning sustainable service systems for children and families after a tragedy.
In During the 3 years after this event, the Town of Newtown, like other communities where mass tragedies have occurred, received both public (e.g., federal, state, and local) and private funding to meet the needs of its citizens. The Town of Newtown distributed some of these public and private resources, and other resources went directly to local community organizations, the school district, and local mental health service providers. Anticipating the need for a sustainability plan once federal funding ended, the OVC asked NYU researchers to assist. Specifically, we were asked to (a) describe the type and organization of services that currently exist; (b) gather perspectives from both the services providers and the families affected by the tragedy on the adequacy of existing services; and (c) make recommendations, based on the data generated, for improving and sustaining high-quality, integrated services. This article summarizes our approach, our findings, and our recommendations. The approach we took may be useful in assisting other communities to systematically assess the architecture of their system and to strengthen and sustain high-quality services. The institutional review board evaluated this project and deemed it a quality improvement effort because of its strong focus on local assessment to improve service quality.

Study samples
To gather the data necessary to answer the questions posed by OVC, we interviewed community service organizations, private behavioral health providers, and both victim and witness families.

Organizations
In collaboration with town leadership and local stakeholders, the research team developed a list of all local organizations to interview, including direct service providers, medical providers, community-based organizations, faith-based organizations, and representatives from local government. The First Selectman (the title for the town's mayor) sent an introductory letter to leadership at all identified organizations. The research team then followed up by phone and e-mail to provide additional information about the project, answer questions, and schedule interviews. Because we were interested in the agency perspective and prior research suggests that no one individual has the complete organizational perspective, we conduced group interviews when agencies requested that multiple staff be interviewed. A total of 30 organizations were invited to participate in interviews (see Table 1). Group interviews were conducted with 56 individuals from 28 organizations (the number of participants per group interview ranged from one to six, with an average of two per organization). Leadership at two organizations (both medical providers) did not return our calls and emails.

Private providers
A list of all 57 behavioral health and private mental health practitioners within a 5-mile radius of Newtown, CT, was generated. To ensure an adequate representation of the opinions of the private mental health providers serving the Newtown community, we estimated that it was necessary to interview 25% of the 57 individuals identifiable through public records. Given that we wanted this sample to be unbiased and that we had no information on a probable response rate, we randomly selected samples of providers, contacted them to arrange telephone or in-person interviews, and then randomly selected additional samples from the provider list until we approached our target sample of 14 interviews. The First Selectman sent the same introductory letter to all selected private providers. Of the 39 randomly selected individual providers we could reach (two could not be contacted), 14, or 35.9%, agreed to participate and were interviewed. Like the entire sample of 57 providers, the 14 interviewed were largely master's-level licensed clinical social workers (LCSWs) (35.7%) or other master's-level clinicians (35.7%). However, clinicians from other disciplinary backgrounds (e.g., PsyD, PhD) were overrepresented (28.6% vs. less than 15% in the full sample) in the group we interviewed.

Families
A total of 28 families were invited to participate in interviews (17 victim families, 11 witness families). Although there were 26 victim families (i.e., families whose children died) and 12 witness families (i.e., families whose children directly witnessed the shooting), we included only those families who currently live in Newtown. The First Selectman sent an e-mail to the victim families that provided basic project information and the research team's direct contact information, for anyone interested in participating. The research team also sent the same letter to these families through their monitored media request website. The First Selectman and the Executive Director of the Newtown-Sandy Hook Community Foundation sent an e-mail to the witness families that provided basic project information and the research team's direct contact information, for anyone interested in participating. Families were contacted only once and the research team never directly contacted a family prior to the family contacting the team. Of the 28 contacted families, 10 (58.8%) victim families and 4 (36.4%) witness families chose to participate.

Measures
Before the scheduled interview, organizations and private practitioners were sent a questionnaire asking a range of questions about their client population, organizational structure, sources of revenue, types of services offered, types of treatments offered, and staff training opportunities. Questions were adapted from ongoing research with New York State behavioral health agencies (Chor et al., 2014;Chor, Wisdom, Olin, Hoagwood, & Horwitz, 2015;Olin et al., 2015a,b). The in-person interviews asked about barriers and facilitators to implementing and sustaining services, current and future needs of the community, local professional networks, and characteristics of an ideal services system.
The family interview asked seven open-ended questions on services families had received, match of services to needs, most and least helpful services, services needed but not received, and what would be an ideal set of services.
Responses were recorded and questions were followed-up with structured probes when necessary.

Coding of family responses
To develop codes for the open-ended questions in the family interviews, two of the authors (SSO and SMH) first used qualitative content analysis techniques (Hsieh & Shannon, 2005), using two randomly selected family interviews. These codes were then used to code two additional, randomly selected family interviews with complete agreement. The project staff coded the remaining interviews (10), after training, with 30% being recoded to check for accuracy. Any discrepancies were reconciled through a discussion between coders.

Analysis
Categorical measures were summarized using counts and percentages, while continuous measures were described . Counts and percentages were used to summarize bivariate associations between the primary revenue source and type of treatment offered (evidence-based and nonevidence based). Data were analyzed using SAS (version 9.3).

Demographic characteristics of the community providers and services
We interviewed 56 individuals across the seven groups of community organizations as well as 14 solo practitioners.
Most providers had advanced degrees (i.e., a master's degree or more), and, with the exception of the police, clergy and foundations/other community-based organizations, most of the individuals interviewed were female. While the majority had been in their current position 5 years or less, they had been in their current fields 10 years or more. The single most common certification was that of an LCSW. The solo practitioners' characteristics were similar to those of the organization interviewees. Among solo practitioners, five were LCSWs, five were licensed professional counselors or licensed marriage and family therapists, and four had "other" clinical licensure. Organizations that employed staff ranged from a low of 2-20 staff to a high of ≥ 100 staff, with most providing direct services. The majority of behavioral health care organizations anticipated reduced staffing (57%) after December of 2015.
We looked at the degree to which services were tailored or personalized to meet victim needs (Table 2). Surprisingly, few of the 12 specific mental health services were tailored to meet victim needs. For example, among the seven community-based organizations providing behavioral health services, only 3 of 12 services (i.e., family therapy, home-based family therapy, and crisis services) were reported by at least 50% of agencies delivering these services to be tailored for victims. A similar lack of tailoring services for victims was reported by these agencies for specific Home-based / family preservation 1/2 (50.0) n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a Other MH services n/a 1/1 (100) n/a n/a 1/2 (50.0) n/a n/a n/a Evidence-based treatments include psychotropic medication management, eye movement desensitization and reprocessing, cognitive behavioral therapy, trauma-focused cognitive behavioral therapy, cognitive behavioral intervention for trauma in schools, child-parent psychotherapy, cognitive processing therapy, and child and family traumatic stress intervention. Treatments with weak evidence include biofeedback/neurofeedback, animal-assisted therapy, emotional freedom technique / tapping, acupuncture, support groups, parenting groups, reiki, massage therapy, and meditation/yoga. Treatments with no evidence include wrap-around services, trauma systems therapy, somatic experiencing, brainspotting, music therapy, art or other expressive therapies, play therapy, aromatherapy, internal family systems, and MNRI R . a These data are available for 12 of the 14 solo practitioners.

Offer any EB treatments
health, substance abuse treatment, and social services (data not shown). Similar patterns were observed among the solo practitioners. Further, only 1 of 28 organizations mentioned offering bereavement or grief counseling to families.
This absence of tailored services for victim's families was also observed for the wellness, recreation, and financial services (data not shown).
We next inquired about the availability of treatments that have been examined empirically and found to have a strong evidence base (evidence-based practices or EBPs) for the treatment of trauma, stress, anxiety, or depression (Table 2). Only four of the seven behavioral health organizations (i.e., 57%) and two of the medical organizations (i.e., 40%) offered evidence-based trauma treatments. In contrast, the school system and the solo practitioners (83%) offered EBPs. When we examined treatments for trauma with a limited or weak evidence base, we found that five of seven (71%) behavioral health organizations, five other organizations, and 42% of solo practitioners also offered treatments with a weak evidence base. We also examined the availability of services with no evidence base and found that five of seven (71%) behavioral health agencies and half of the solo practitioners offered services with no evidence base.

What does Newtown need to create an ideal service system?
We asked respondents about the characteristics they believed were needed to construct an ideal service system in their community (Table 3). The two areas provider organizations cited most frequently were (a) developing centralized community supports and (b) improving collaboration/coordination among community providers. The solo practitioners also endorsed a wide range of strategies including centralized community support, education and training, collaboration/coordination among different service providers, and the creation of an integrated care service with a continuum of services (Table 3).
Finally, we asked the provider organizations about the necessary ingredients to make the ideal service system a reality (Table 3). Both organizations and solo practitioners felt funding and transparency about money was important.
Organizations also thought communication or dialogue among stakeholders, leadership or a decision maker, willingness to set aside differences, and communication and coordination of services through a single point of entry were important. Solo practitioners thought development of competency in identifying trauma-related needs and coordinated care with a single point of entry were important (Table 3).

Association of chief sources of funding with use of EBPs
When we examined the association of type of funding with whether organizations offered evidence-based mental health treatments, we found that three of five (or 60%) behavioral health organizations reported offering EBPs if they received federal/state/government (F/S/G) funding, but four of five (or 80%) also offered treatments with weak or no evidence base (Table 4). The one behavioral health organization with charitable funds as a major source of funding offered only treatments with weak or no evidence base. The one medical provider whose primary source of revenue was F/S/G offered no EBPs, while, of the other four medical providers relying on other sources of income (private and other), 50% offered some EBPs. In contrast, 83.3% of solo practitioners, all of whom relied on direct payments or insurance, offered EBPs. Two thirds (4/6; 67%) of organizations that relied on F/S/G funding reported use of treatments with weak or no evidence base, while less than half (8/18; 44%) of those who rely primarily on other direct payments or insurance used these treatments.   Overall, 50% of the victim families reported that the services they received were not well matched to their needs.

Victim family and witness family perspectives
Only 20% of the victim families gave a "very well" rating when asked how well the services matched their needs. Among the witness families interviewed, half rated their services as having matched their needs very well. Both sets of families reported that they needed services that they did not receive (50% victim families, 75% witness families; Table 5).
There was a range of features that distinguished helpful services. For victim families, services were helpful when they addressed a specific need; in addition, they found having a point person and money/entitlements for services helpful. For witness families, having a point person, services that were trauma specific, and addressing a specific need were the most important features.
The characteristics of services that made them least helpful for victim families included the service itself not matching the family's needs, difficulties with reimbursement, and mismatch between provider expertise and family need. Service and provider expertise mismatches were the most frequently mentioned characteristic by witness families interviewed. On a scale ranging from 1 (not at all) to 10 (a great deal), victim families rated the overall benefit from services as 4.3 ± 3.2, and witness families rated overall benefit as 8.3 ± 2.1. Although the data were not collected in a format that allowed us to test the association between benefit and the specific services received, notes from the interviews suggest that specific evidence-based trauma-focused services sought outside of the community were perceived to be most beneficial. On the individual level, the three most frequently mentioned features of an ideal system were individualized services for each family, that services be offered to families rather than families having to seek them out, and having a single point of contact. This is consistent with provider perceptions about service access being the greatest current need.
Witness families had few recommendations about an ideal service system at the community level. Their recommendations focused on the provider/organization level, where they also thought that expert trauma services and additional mental health services were important, but added that there should be accessible (i.e., local) expertise to facilitate long-term continuity of care. They agreed that differentiated services (i.e., for siblings vs. witness children) were needed in schools and also thought that schools needed trauma expertise and academic supports. At the individual level, the important features of an ideal system for witness families included individualized services for each family, a single point of entry, and having services offered to families rather than families having to seek them out on their own.

DISCUSSION
Events like Sandy Hook are impossible to predict but tragically are occurring with some regularity. Like other mass shootings, the Sandy Hook tragedy affected the entire country because of its prominence in the public eye and media. It is not surprising, therefore, that when the Office for Victims of Crime offered assistance to Newtown after the tragedy, they deemed the entire community "a victim" in order to ensure that funds would flow quickly into the town. This decision, however, likely had consequences for how the system was shaped, what services were provided, and to whom they were provided. It likely also affected how the providers, victim families, and witness families ultimately perceived the services. In addition, before our evaluation, a "division of labor" had been set up between the Department of Education's support for assisting the school system in responding to the tragedy and the Department of Justice's response to assisting the community. This created some challenges. We were asked to focus on the community set of services, not the array of services already being provided in the schools, some of which included community outreach.
Our findings highlighted several aspects of service system redesign that may be relevant to other communities as they assess their capacities for providing sustainable services after a community tragedy. Newtown was unusually well resourced in the aftermath, with funds from federal, state, and private donations. While funding matters, we learned that coordinated and systematic attention to implementation as a separate step in a community's healing is essential.
It involves a unique set of tasks, responsibilities, and planning. Developing a communitywide plan for implementation of networked community services needs to involve all major stakeholders sharing a common goal. implementation. Some of the core components across these natural experiments include centralizing access to services; coordinating care by creating teams that work within and across agencies to provide efficient and nonduplicative care; and personalizing services to improve the match between patient's needs and the care they receive. Interestingly, providers and families in Newtown also identified these themes, being implemented across other U.S. communities, as key features for an improved future system. However, the context of having experienced a tragedy of unprecedented proportions, having to rebuild a sense of community again, and dealing with the wide range of very different personal needs-from those who lost a child to those who were shocked and saddened by the event-brought the challenges of these system issues into stark relief.
Newtown's community service system was broad and included a wide array of different community servicesmental health, substance use, medical, social services, schools, faith-based, and safety-focused. It was by most standards adequately staffed, with unusually high levels of advanced degrees and years of tenure in the system. The quality of the services, however, was uneven, with significant gaps and unmet needs. Access was largely decentralized with different entry points for school services, community-based treatment services, and wellness services. Evidence-based practices were offered in more than half (57%) of the behavioral health agencies, 40% of medical providers, in the school system, and by the majority of the solo providers. Yet services were not individualized, and few personalized or tailored services were available. Surprisingly, only one of the 28 agencies offered bereavement or grief counseling.
There was, however, considerable agreement from providers and families about what services were needed to improve the system. Both providers and families agreed that improving access through centralization of services, improving coordination across agencies, and individualizing services were paramount. There were also suggestions to make monetary decisions about allocation of funds more transparent and provide training on effective organizational practices to all levels of staff: frontline, mid-management, and leadership.

Limitations
It is important to note the key limitations that may have influenced our findings. Although we were able to interview more than 90% of the provider organizations in Newtown, reaching solo practitioners was much more challenging.
Because of the likelihood of self-selection, our findings from this group of respondents should be interpreted with caution. It is likely that those solo practitioners who agreed to participate were self-selected based on valuing research.
In addition, because we adhered to OVC's classification of the entire town as "victim," we took a broad approach in understanding services provided to anyone in the town and thus do not have information about which tier of victims these solo practitioners served. Further, while victim families were well represented, witness families were much more reluctant to participate in this evaluation. It is thus likely that the views of the witness families who agreed to participate may not fully represent the range of perspectives from this important group.

Conclusion
The opportunity for an external group of evaluators to assess an entire array of community services within a single town and develop a sustainability plan was very unique. Our approach, which was to sample the entire set of agencies, providers, and families and use semistructured interviews to compare and contrast perspectives from providers and families, yielded data that captured, descriptively, the town's capacities, gaps, and community goals. Despite variability in views about the adequacy of services within the community, there was consensus across providers and families about key components of an ideal service system, including developing an infrastructure for coordination and centralization of services and resources to support all families, but especially those most impacted by the tragedy. Providers and families also recognized the importance of community leadership in sustaining trust and providing a foundation for an ideal service system.
That consensus existed around the elements needed to build a sustainable system is important and bodes well for making these changes happen. That healthcare system leaders have also identified these same components nationally (Berwick, 2016;Halfon et al., 2014;Pronovost & Bo-Linn, 2012) is noteworthy. National organizations and federal agencies are developing resources and guidelines to inform stronger community policies to deal with tragedies. The core ingredients of community strengthening in the aftermath of a tragedy have been identified and there is now general consensus on these ingredients. Building them a priori into a coordinated, integrated, and sustainable system that is ready for mobilization when tragedy strikes is the next step.