SEARCH

SEARCH BY CITATION

Keywords:

  • Mental health services;
  • pre-school;
  • service evaluation;
  • observational study

Abstract

  1. Top of page
  2. Abstract
  3. Key Practitioner Message:
  4. Introduction
  5. Background to the study
  6. Method
  7. Results
  8. Reported behavioural/emotional changes in the child
  9. Discussion
  10. Acknowledgements
  11. References

Background:  Specialist CAMHS-based services for pre-school age children are being introduced in the UK using different models of care. The clinical value of these new services requires assessment.

Method:  Over 20 months a Primary Mental Health Specialist (Under 5s) service operating in South West UK was evaluated on a variety of themes including effectiveness. Recruited clients completed questionnaires on their own well-being (on two occasions) and their child’s behaviour (on three occasions) over the intervention period.

Results:  Of the 67 carers assessed at or shortly after recruitment using the General Health Questionnaire, 55.2% were at high risk of having a clinically significant affective mental condition. Despite a trend towards improvement, neither the GHQ-12 total score nor the proportion meeting the clinical criterion was statistically significantly reduced. The main index of child behaviour and emotional state demonstrated a statistically significant reduction in the number of disturbance-indicating behaviours over the service intervention period. The largest changes occurred between recruitment and the 4th visit. Selection bias cannot be excluded.

Conclusions:  Significant changes determined over a relatively short period of intervention is consistent with an effect of service, but direct attribution demands care as no control groups were included.


Key Practitioner Message:

  1. Top of page
  2. Abstract
  3. Key Practitioner Message:
  4. Introduction
  5. Background to the study
  6. Method
  7. Results
  8. Reported behavioural/emotional changes in the child
  9. Discussion
  10. Acknowledgements
  11. References
  • • 
    Obtaining measures of clinical improvement over a service period for the under-fives age group is desirable and feasible
  • • 
    The psychological health of the carer is an important outcome in achieving work targets with the under-fives
  • • 
    Attributing emotional and behavioural changes to service interventions is important but methodologically challenging
  • • 
    Building evaluation protocols, including longer term family follow-up, into current practice may be extremely valuable in ascertaining overall service impact

Introduction

  1. Top of page
  2. Abstract
  3. Key Practitioner Message:
  4. Introduction
  5. Background to the study
  6. Method
  7. Results
  8. Reported behavioural/emotional changes in the child
  9. Discussion
  10. Acknowledgements
  11. References

Most Child and Adolescent Mental Health Services (CAMHS) are faced with a large referred body of school-age children exhibiting disruptive behaviour, frank conduct disorder or self-harm. In comparison, infant attachment disorders are generally not seen as equivalently urgent, with resource-intensive interventions perceived to be preventive rather than therapeutic (Barrows, 1997) and, consequently, of lower priority. For these reasons and, perhaps, with a consequent lack of early years expertise, CAMHS services have historically focused their waiting lists on school-age children, leaving an age-gap in service provision. Provision of support for parents of pre-school age children with behavioural and emotional difficulties in England has largely fallen to the Health Visiting service. However, an increased focus on infancy and early childhood is now embedded in the National Service Framework for Children, Young People and Maternity Services (NSF, 2004). Standard 2 refers specifically to the importance of secure attachments and emotional development in pre-school children, and Standard 9 to promoting parent-child relationships, early intervention and CAMHS support for primary care services. The evidence-base cited to support these standards is, however, very limited.

Background to the study

  1. Top of page
  2. Abstract
  3. Key Practitioner Message:
  4. Introduction
  5. Background to the study
  6. Method
  7. Results
  8. Reported behavioural/emotional changes in the child
  9. Discussion
  10. Acknowledgements
  11. References

A city-wide service of four full-time equivalent Primary Mental Health Specialist posts for the Under 5s (PMHS) was formed in 2004 to support the needs of very young children and their parents in cases where specifically behavioural or emotional problems had emerged. The service, whilst liaising closely with Primary Care, was based in CAMHS and conformed, approximately, to the ‘outreach from Tier 3′ model described by Macdonald et al. (2004). These posts were to act as an ‘interface between Tier 1 and specialist/core CAMHS’ and to support and strengthen existing Tier 1 provision by building capacity in health visitors and all those working with young children in providing direct and indirect clinical interventions and in facilitating access to CAMHS. The service is broadly modelled on a ‘Watch, Wait and Wonder’ approach (Muir, 1992) focusing on the child-parent relationship through increasing parental awareness in order to achieve behavioural and emotional change. It adopts a pattern of negotiated engagement with the parent(s) and includes elements of child-led play. Consultations most often take place in the home and emphasise the value of child observation in recognising parental anxiety within the relationship and the child’s responses to it. PMHS staff have received training through the Tavistock Clinic postgraduate courses in Psychoanalytic Observational Studies and/or Infant Mental Health. Postholders receive regular personal supervision sessions from child psychotherapists, whilst overall clinical supervision is provided by a consultant child psychotherapist working within one of the CAMHS teams.

In 2006 a formal evaluation of process, activity, and effectiveness of the new service was conducted (Pollock & Horrocks, 2008). One component of the evaluation quantitatively assessed the clinical impact of service on the children and their mothers. A paper reporting a qualitative evaluation of users’ perceptions and experiences is in preparation.

Method

  1. Top of page
  2. Abstract
  3. Key Practitioner Message:
  4. Introduction
  5. Background to the study
  6. Method
  7. Results
  8. Reported behavioural/emotional changes in the child
  9. Discussion
  10. Acknowledgements
  11. References

The primary aim was to assess change in reported child behaviour and maternal well-being over the period of service intervention. The evaluation was designed to meet best practice recommendations proposed by the Royal College of Psychiatrists,1 the CAMHS Outcome Research Consortium,2 the National Institute of Mental Health in England report on priority setting (McCombie & Chilvers, 2005), and the Every Child Matters documents (NSF, 2004).

Measures of clinical change

The primary criteria for selecting assessment instruments were that they should have been formally validated on a similar population, could be completed quickly by the carer, and covered the full age-range of the pre-school age child. Current CORC-recommended core measures were unsuitable for this purpose. The instrument adopted to assess child behaviour was the ‘Ages and Stages (Social-Emotional) Questionnaire’ (ASQ:SE), a parent-completed child-monitoring system used for screening for social-emotional difficulties in pre-school age children. It is unique as a screening instrument in covering the whole pre-school age range, although interpretive derived scoring systems are required to assess longitudinal changes across age-bands. It is used in practice to identify children who meet criteria for referral to specialist services and has been extensively validated on a diverse population in the US (Squires, Bricker, & Twombly, 2003).

The instrument chosen to assess the emotional well-being of the carer was the short form General Health Questionnaire, developed by Goldberg as a screening instrument to assess risk of a current psychiatric disorder, diagnosable by subsequent psychiatric interview (Goldberg, 1972). The GHQ has been widely validated in numerous populations and settings (Vieweg & Hedland, 1983).

Assessments were timed for soon after the first service intervention (Baseline/Stage 1); after four substantive contacts with the client (Stage 2), and at discharge back to the referring person (usually the health visitor), or 6 months after the first appointment, whichever was the earlier (Stage 3).

Client contact

Verbal consent to contact new clients was provided by PMHS staff at the earliest point in their initiation of service and, following the receipt of signed consent forms, clients were asked to complete questionnaires individually or have them completed during interview according to their preference. PMHS staff and their administrators informed the researchers of withdrawal from service, refusals, the timing of 4th visits and discharge or referral on. The timing of questionnaires was:

Recruitment (Stage 1):ASQ:SEGHQ-12
4th visit (Stage 2):ASQ:SE 
Discharge/6 months:ASQ:SEGHQ-12

Ethical approval was granted by the Local Research Ethics Committee and the Ethics Committee of the University of the West of England.

Data management and analysis

Quantitative data were analysed using Microsoft Excel, SPSS version 13 and STATA version 7. Data from individual PMHS workers were combined and analysed on a service basis.

Results

  1. Top of page
  2. Abstract
  3. Key Practitioner Message:
  4. Introduction
  5. Background to the study
  6. Method
  7. Results
  8. Reported behavioural/emotional changes in the child
  9. Discussion
  10. Acknowledgements
  11. References

Compliance and questionnaire completion rates

Over a 20-month period 76 consecutive clients initially agreed to participate in the evaluation but signed consent forms were obtained only in 68 cases. PMHS staff vetoed 6 clients they considered too disturbed or unable to participate in the evaluation. During the evaluation 22 (32%) clients were lost by withdrawal from service through client choice or residential change, or were discharged through loss of contact, non-attendance, referral onwards or early resolution. The flow diagram details the numbers at each stage of the evaluation (Figure 1). Four clients who had not completed Stage 2 questionnaires returned to complete Stage 3 and the total number of clients completing all three stages was 12.

image

Figure 1.  Flow diagram indicating client numbers at different study stages

Download figure to PowerPoint

Possible attrition bias resulting from differences in the types of families who stayed in the service for different stages of the intervention requires consideration. Analysis showed that whilst those clients lost to the study in later stages had more severe initial problems than those remaining in the study to discharge or 6 months, the differences did not approach statistical significance (Mann Whitney tests: = −0.88 to −1.29, = .2 to 0.38 for the three outcome measures).

Different client problems, caseload pressures and clinical decisions, together with variation in appointment compliance and PMHS absences, ensured great variation in the frequency and duration of service provision to individual client families. From the 23 Stage 1 to Stage 2 durations with known dates, the mean period was 88.3 days or 12.6 weeks (range: 29–195 days, SD = 46.6 days). The 16 Stage 1 to Stage 3 known durations had a mean of 170.5 days or 24.4 weeks (range 61–252 days, SD = 65.0 days).

Client characteristics

The age of index children being seen in 2007 ranged from birth (or even antenatal first appointments in two cases) to 5 years of age (mean 2.7 years, SD = 1.41 years).

The mean number of service contacts per case opened and closed in 2007 was 5.1 (SD = 4.5). One half of these cases were seen less than six times, but 20.2% of the cases first seen in 2007 were ongoing at the end of the data collection period. Behavioural difficulties, challenging behaviour, or aggressive behaviour together formed the most prevalent category (49%) of behaviour instigating referral to the PMHS service, followed by relationship, bonding or attachment difficulties, feeding, sleeping and crying (39%).

Maternal well being

Indication of a clinically significant emotional condition requiring professional care is generally accepted at GHQ-12 total scores of 4 or above as used in the annual Health Survey for England. Thirty-nine service clients (55.2%) met this criterion at baseline. A total of 16 carers submitted two completed GHQ questionnaires separated by periods ranging from 7 to 37 weeks (mean 22.6 weeks, SD 10.1). A very wide range of score changes was observed (−21 to +10) with a mean score change indicating a health improvement (negative value of change) of 3.4 points (SD = 8.3), but this difference did not reach statistical significance (Wilcoxon Signed Ranks test, = −1.53, = .13). Consideration must, however, be given to the low proportion of carers providing data on follow-up. It is unclear how attrition bias might have affected any general direction of change as women whose wellbeing improved (increased likelihood of negative GHQ score changes) might have either been less or more likely to participate. At re-assessment on discharge (or after 6 months) the proportion of carers reaching clinical significance in their GHQ scores had changed little (6/16 versus 7/16).

Reported behavioural/emotional changes in the child

  1. Top of page
  2. Abstract
  3. Key Practitioner Message:
  4. Introduction
  5. Background to the study
  6. Method
  7. Results
  8. Reported behavioural/emotional changes in the child
  9. Discussion
  10. Acknowledgements
  11. References

Analysis of change in outcome faced two difficulties: differences between the specific age-banded questionnaires used, and cases dropping out of the service, the evaluation, or both, over time. Different age-banded questionnaires differed in the number of questionnaire items, the maximum score and the referral threshold. Children were referred from infancy to age 5 and individual children were monitored over time periods that sometimes covered different age-banded questionnaires. Whilst each age-band was independently validated for referral, there was little information available from the validation study that could confirm the equivalence of referral threshold criteria across age bands. For this reason the primary outcome adopted was the observed score as a percentage of the ASQ: SE threshold score for referral for that age band, 100% indicating the child’s score was equal to the threshold score.

Outcome changes were analysed as ‘stage totals’ (examining the outcome statistics recorded for any client contributing data to any stage) and ‘stage transitions’ (examining the outcome changes only for the same clients contributing data at two or more stages). In the former a clear downward ‘trend’ over the PMHS intervention period is illustrated in the ASQ:SE results presented as the percentage of the clinical intervention cut-off threshold (outcome measure 1, see Table 1). The mean long-term (Stage 1 to Stage 3) absolute difference of 66.3% indicates a substantial reduction in the carers’ reporting of their children’s behaviour as disturbed. At recruitment 75.0% of the 68 ASQ:SE scores were above the clinical intervention threshold. By Stage 2 this had reduced to 60.0% and by Stage 3, 35.3%. By discharge (or 6 months after the first appointment) the mean ASQ:SE score was only just above (by 6.9% of the score) the published cut-off for clinical intervention, a drop from 73.2% above.

Table 1.   ASQ:SE total scores as a % of the clinical intervention threshold (cut-off) at the three stages of PMHS intervention (100% = threshold score for referral at that age)
StageRecruitment (Stage 1)After 4th visit (Stage 2)Discharge or 6  months (Stage 3)
ASQ:SE total score as a % of cut-offASQ:SE total score as a % of cut-offASQ:SE total score as a % of cut-off
N682516
Mean173.2136.7109.1
Median152.5128.677.4
SD102.881.987.3

The results of a non-parametric analysis of changes in ASQ:SE scores in the same cases over time (stage transitions) is shown in Table 2. A significant or a highly statistically significant reduction in the degree to which the recorded total ASQ:SE score surpassed the intervention threshold was observed for both intervention time periods, with most of the change over the whole intervention period occurring over the first four service visits.

Table 2.   Testing the significance of changes in ASQ:SE total scores as a % of the clinical intervention threshold (cut-off) between different periods of PMHS service intervention
Period comparisonWilcoxon matched pairs Signed Ranks test
Zp (2-tailed)
  1. Note: A negative Z value indicates a lower adjusted ASQ:SE score in the later stage.

Comparison of Stage 1 (recruitment) to Stage 2 (4th visit) = 25−3.56<.001
Comparison of Stage 1 (recruitment) to Stage 3 (discharge or 6 months) = 16−2.10.04

The apparent anomaly of a larger change between Stage1 and Stage 2 than between Stage 1 and Stage 3 is probably an artefact deriving partly from case attrition: cases lost between Stages 2 and 3 had rather higher baseline (Stage 1) ASQ:SE values (mean rank 18.5) than those remaining in the study (mean rank 13.2, Mann Whitney test, = −1.78, p .07). As the change in this measure was strongly correlated with baseline (Stage 1) ASQ:SE values (rsp = 0.5, = .01), it seems that the Stage to Stage results were affected by a subgroup that started with more severe problems, exhibited marked improvements within four visits, and dropped out from the study, leaving a less-changed remainder in the final analysis. The duration of the PMHS service was by negotiation with the client and the evaluation protocol timing depended on continuing client engagement.

Discussion

  1. Top of page
  2. Abstract
  3. Key Practitioner Message:
  4. Introduction
  5. Background to the study
  6. Method
  7. Results
  8. Reported behavioural/emotional changes in the child
  9. Discussion
  10. Acknowledgements
  11. References

In the current study over half of the 67 carers met the criterion of being at high risk of having a clinically significant affective mental condition. As a screening device no diagnostic certainty can be attributed to GHQ scores but this measure meets conventional interpretations used in surveys of mental health. Comparison with a national sample of women aged 18–40 from the 2005 Health Survey for England indicated a 3.6 times higher prevalence in the study sample. Only a small number of clients were available for longitudinal assessment of change over the service intervention period up to discharge or 6 months, but neither the GHQ-12 total score nor the proportion meeting the clinical criterion was statistically significantly reduced. High levels of maternal psychological ill health are likely to reflect and be reflected in their children’s behaviour and emotional well-being and should be included in assessment of clinical effectiveness in such services. The role of the PMHS staff in managing (or at least working around) psychiatric disorder in carers demands recognition as they were neither trained for, nor did they have the time to prioritise this task, their activities being focused on the linked, but distinct topic of carer-child attachment, parenting, and child behaviour.

Over the PMHS intervention period improvements were noted in parent-reported measures of their child’s behaviour and emotional state and their own concerns, as encapsulated in the ASQ:SE score as a percentage of referral threshold. Most improvement occurred over the first four PMHS visits as judged by adjusted ASQ:SE scores changes. Attributing these changes to the PMHS intervention demands caution as no control groups were included. However, significant changes determined over a relatively short period of intervention is of interest and consistent with a clinical effect. Making the reasonable assumption that families were not already on an improving trajectory at the time of referral, the results over the short-term are encouraging as this intermediary assessment stage was introduced specifically to argue for the attribution of change to the service, rather than extraneous or simply temporal factors. Furthermore, the degree of improvement in the child (as reported by the carer) was highly correlated with the baseline ASQ:SE score, suggesting that the service effect might be dependent on the acuteness of the case. Interpretation is, however, confounded by the fact that child behaviour measures were carer-reported, that data limitations restrict generalisation, and that control groups were absent. Formal attribution of any clinical improvement to the intervention of the PMHS service has to remain cautionary.

A survey of existing services in England and Scotland identified 42 parent-infant mental health services for under fives, although it is unclear how complete this survey is.3 Evaluation (or audit) was infrequently reported and only two referred specifically to clinical outcome in the survey. Notwithstanding the probable incompleteness of the survey and the possible inclusion of some unstated clinical effectiveness measures in the remaining evaluations, it seems that relatively little evaluation of under 5s mental health services has addressed direct clinical change in a quantitative fashion. Evidence-based assessment of clinical effectiveness in existing services is methodologically complex and poses many research challenges, not least because of ethical considerations in including suitable controls. Nevertheless, it is essential that these services are subject to clinical scrutiny and measurement of effectiveness if possible, and that evaluations go further than simply recording stakeholder opinion and process. The current study is an attempt to redress the balance in this challenging area of research.

Acknowledgements

  1. Top of page
  2. Abstract
  3. Key Practitioner Message:
  4. Introduction
  5. Background to the study
  6. Method
  7. Results
  8. Reported behavioural/emotional changes in the child
  9. Discussion
  10. Acknowledgements
  11. References

The inception of this work owes much to Paul Barrows who provided clinical guidance for the service and PMHS staff, to the PMHS workers themselves for assisting in recruitment and data collection, and to the administrative staff supporting them. The clients themselves are owed much for completing questionnaires and, in some cases, for agreeing to be interviewed. We are also grateful to Rosalind Bennet for her advice in describing the PMHS role.

References

  1. Top of page
  2. Abstract
  3. Key Practitioner Message:
  4. Introduction
  5. Background to the study
  6. Method
  7. Results
  8. Reported behavioural/emotional changes in the child
  9. Discussion
  10. Acknowledgements
  11. References
  • Barrows, P. (1997). Parent-infant psychotherapy: A review article. Journal of Child Psychotherapy, 23, 255264.
  • Goldberg, D. (1972). The detection of psychiatric illness by questionnaire. Maudesley Monograph No. 21. London: Oxford University Press.
  • Macdonald, W., Bradley, S., Bower, P., Kramer, T., Sibbald, B., Garralda, E., & Harrington, R. (2004). Primary mental health workers in child and adolescent mental health services. Journal of Advanced Nursing, 46, 7887.
  • McCombie, C., & Chilvers, S. (2005) Research in child and adolescent mental health services: Results of research priority setting exercise. NIME. January.
  • Muir, E. (1992). Watching, waiting, and wondering: Applying psychoanalytic principals to mother-infant intervention. Infant Mental Health Journal, 13, 319328.
  • NSF (2004). National service framework for children, young people and maternity services: Change for children - every child matters. London: Department of Health.
  • Pollock, J.I., & Horrocks, S. (2008). Evaluation of the primary mental health specialist (infant mental health) programme in North Bristol NHS Trust and United Bristol Healthcare NHS Trust. Bristol: University of the West of England.
  • Squires, J., Bricker, D., & Twombly, E. (2003). The ASQ:SE user’s guide. Baltimore, MD: Paul Brookes.
  • Vieweg, B.W., & Hedland, J.L. (1983). The General Health Questionnaire (GHQ): A comprehensive review. Journal of Operational Psychiatry, 14, 7885.