Assessment of the convergent validity of the Questions About Behavioral Function scale with analogue functional analysis and the Motivation Assessment Scale

Authors


Johnny L. Matson Ph.D., Department of Psychology, 236 Audubon Hall, Louisiana State University, Baton Rouge, LA 70803, USA (e-mail: johnmatson@home.com).

Abstract

The present study examined the convergent validity of the Questions About Behavioral Function (QABF) scale, a behavioural checklist for assessing variables maintaining aberrant behaviour, with analogue functional analyses and the Motivation Assessment Scale (MAS). The two checklists were more highly correlated with each other than either checklist with results from the analogue sessions, and the QABF was more highly correlated with analogue sessions than the MAS. Using analogue sessions, the experimenters failed to ascertain behavioural function for a number of subjects because the behaviour problems in question were low frequency/high intensity and failed to appear during the course of the analysis, pointing out a limitation of this technology. These findings, taken together with recent research outlining the psychometric properties of the QABF, seem to support the use of the QABF in a hierarchical model of functional analysis. The implications of the findings are discussed.

Introduction

The development of treatment interventions for severe behaviour problems in people with intellectual disability (ID) has benefited from the information obtained from functional assessment procedures. Functional assessment procedures assist staff in identifying the contingent relationship between target behaviours and environmental antecedents and consequences (Sturmey 1996). In addition, the assessment may specify reinforcer quality and duration, and the behaviours to be targeted (Iwata et al. 1990; Sprague & Horner 1995; Linscheid et al. 1996). The results of functional assessment can be used by clinicians to develop a functionally appropriate treatment package (Sprague & Horner 1995).

Three primary methods of functional assessment have been identified in the literature. These include descriptive assessments (e.g. Aberrant Behavior Checklist data and scatterplots), analogue assessments (e.g. experimental functional analyses) and indirect assessments (e.g. interviews and behavioural checklists). Each assessment method has associated strengths and weaknesses. Descriptive assessments allow for objective identification of events surrounding behaviour, and can be used to identify idiosyncratic forms of reinforcement and existing schedules of reinforcement (Mace 1994). Unfortunately, these assessments are limited in that the results are correlational in nature (Mace 1994; Sprague & Horner 1995). Mace & Lalli (1991) and Lerman & Iwata (1993) found that descriptive analyses yielded results that were inconsistent with experimental sessions. In general, descriptive assessments can be time consuming, yield data that is difficult to interpret and require extensive training (Sturmey 1996).

Analogue functional analyses provide an empirical demonstration of functional control, and a reliable match between function and treatment (Van Houten & Rolider 1991; Mace 1994; Sprague & Horner 1995). The experimental functional analysis procedures described by Iwata et al. (1994) involves comparing the rates of target behaviours across different stimulus conditions (e.g. play and demand). The validity of this approach has been reported for self-injurious behaviour (Day et al. 1988), aggression (Slifer et al. 1986), stereotypy (Sturmey et al. 1988) and property destruction (Carr & Durand 1985).

Analogue functional analyses also have inherent limitations, including the cost, the training required, the amount of time required and the availability of necessary resources (Lennox & Miltenberger 1989; O’Neill et al. 1990; Iwata et al. 1994; Sprague & Horner 1995; Sturmey 1995). The use of this methodology would also not be appropriate for high-intensity or low-frequency behaviours (Sturmey 1995; Linscheid et al. 1996; Vollmer & Smith 1996; Pyles et al. 1997). Martin et al. (1999) found poor test–retest reliability for the experimental functional analysis and reported difficulty interpreting functions.

Interviews and behavioural checklists are another technique used by clinicians to identify functions of problem behaviour. These methods require a third party (e.g. parent, caregiver or teacher) who knows the client well to respond to questions about client behaviour. This approach to assessment provides the clinician with an efficient method of ascertaining behavioural function; however, the problem behaviour is not observed first hand. These methods are cost and labour effective, and are amenable for use with large groups in institutional settings (Durand & Crimmins 1988; Van Houten & Rolider 1991).

Indirect methods also have noticeable weaknesses. The reliability and validity of indirect functional assessments are poor overall. One of the most widely used indirect functional assessments, the Motivation Assessment Scale (MAS; Durand & Crimmins 1988), has poor psychometric properties (Bihm et al. 1991; Sigafoos et al. 1994; Sturmey 1994).

The present study was designed to further validate the Questions About Behavioral Function (QABF; Matson & Vollmer 1995) scale for clinical use. The QABF, a recently developed behavioural checklist, was designed to identify functions of aberrant behaviour, the preliminary psychometrics of which have been good (Matson et al. 1999; Paclawskyj et al. Year?). The QABF, like other behavioural checklists, allows for an efficient assessment of observable behaviour. To establish convergent validity, the present authors compared the QABF to the MAS (Durand & Crimmins 1988) and analogue functional analyses (Iwata et al. 1994), which are more traditional and widely used methods of determining behavioural function.

Subjects and methods

Participants

Thirteen participants referred for a formal functional analysis were studied, and the cohort was similar in size to that of Durand & Crimmins (1988), who reported on the convergent validity of the MAS. The sample contained both females (53.8%) and males (46.2%), and individuals diagnosed with profound ID (92.3%). Target behaviours included self-injury (n = 6), aggression (n = 4), tantrums/verbal aggression (n = 1) and stereotypy (n = 2).

Procedure

Experimental functional analyses

Informed consent from a parent or guardian was obtained for those participants for whom a parent or guardian could be identified and located. Criteria for terminating the sessions was set a priori, but that criteria was never met.

Each participant was included in a formal functional analysis, using procedures described by Iwata et al. (1982). Four stimulus conditions were alternated using a multi-element design: (1) attention; (2) demand; (3) tangible; (4) alone/ignore; and (5) toy play. The tangible condition was not run in two cases where preferred items could not be identified, and the alone condition was not conducted for the three participants who displayed aggressive behaviour.

Data were collected using one-minute intervals. For those behaviours where a clear start and stop time was difficult to ascertain, occurrence/non-occurrence data were collected. For all other cases, frequency of responding was recorded. When results were unclear (n = 1), extended alone sessions were conducted to determine if the behaviour persisted in the absence of social contingencies. Each condition was conducted in an empty day room located in the client’s home or workshop. The therapist for the session sat next to or behind the client, as appropriate for the condition. One or two observers collected data from the far side of the room, and did not interact with the client until the session terminated.

In the attention condition, one therapist sat near the client reading a magazine, and provided verbal and physical attention only following the occurrence of the target behaviour. In the toy play condition, the participant had preferred items available, and the therapist provided attention once every 30 s contingent on a 5-s absence of target behaviours, ignoring target behaviours. In the demand condition, the therapist presented functional demands (e.g. folding towels or stacking cups) to the client once every 30 s using a three-step hierarchy of prompts (i.e. verbal, modelling and physical guidance). The therapist allowed the client to escape the demand for 30 s contingent on their target behaviour. In the alone/ignore condition, the therapist sat behind the client or out of the client’s line of sight, and ignored the target behaviour. Finally, in the tangible condition, the therapist gave the client a highly preferred object to interact with for 2 min prior to the session’s start. The therapist then removed the item and returned it to the client for 30 s contingent on any target behaviour.

Two to three graduate students conducted each functional analysis session and a total of 257 sessions were conducted. Students were trained in functional analysis procedures prior to the onset of this study by achieving at least 80% inter-observer agreement with another trained student. These students were blind to the results of the QABF and MAS until the experimental sessions were complete. Reliability was obtained for 54.1% of the sessions. The reliability of frequency data was computed by dividing the smaller frequency by the larger frequency within each interval, multiplying by 100%, and averaging across intervals. Reliability for the occurrence/non-occurrence data was calculated by dividing the number of agreements by the number of intervals and multiplying by 100% (Vollmer et al. 1995). Reliability averaged 95.7% (range = 61.7–100%).

The maintaining variables for the problem behaviour as assessed by the functional analyses were determined using the visual inspection criteria developed by Hagopian et al. (1997). Essentially, this set of procedures identifies the differentiated condition(s) by comparing responding across the test conditions (i.e. demand, tangible, attention and alone) relative to the control condition (toy play). The percentage agreement on interpretation was computed by dividing the number of exact agreements on function by the number of agreements and disagreements. The percentage agreement between the two raters was 92.3%.

Behaviour rating scales

A graduate student administered the QABF and the MAS to an informant who had known the client for at least 6 months. The examiner and informant were blind to the results of the analogue sessions until all assessment was completed for that client. The QABF and MAS were administered during or up to one month after the experimental sessions were conducted.

The QABF includes a section on physical illness or discomfort not assessed by analogue functional analysis. Therefore, each client’s record was reviewed to determine if a physician identified a medical diagnosis as a contributing factor to the client’s problem behaviour. In each medical chart, the following sections were examined: active illnesses; major problems; minor problems; progress notes for the past year; and physician’s orders. Physical illness was ruled out as a maintaining variable for the maladaptive behaviours of the 13 participants.

Primary functions, as identified by the QABF, were determined by the methodology described in the user’s manual (Matson & Vollmer 1995). A scoring profile (available with the measure) was completed for each QABF. If one subscale received a greater total score than the others it was labelled as the primary maintaining variable. If two subscales had identical total scores, then both were identified as primary maintaining variables.

Primary functions as identified by the MAS were determined by following the methodology of Durand & Crimmins (1988). The mean scores were computed for each subscale and rank ordered with the primary ranking selected for convergent validity conclusions. Two functions were identified as primary when tie scores occurred.

Spearman rank-order correlation coefficients were computed between the subscale scores of the QABF and MAS. Since the interpretation of the analogue sessions produced categorical data, total percentage agreement on function was calculated by dividing the number of exact agreements by the number of agreements plus disagreements.

Results

The following results of the analogue functional analyses reflect interpretations based on the criteria outlined by Hagopian et al. (1997). Figures 1–5 depict the functional analysis results for participants grouped by functional analysis outcome. Figure 1 includes three participants whose problem behaviour appeared to be maintained by negative reinforcement in the form of escape from demands. Figures 2 and 3 include four participants, all of whose behaviour appeared to be maintained by automatic reinforcement. Three participants’ behaviour was maintained by multiple sources of reinforcement, as shown in Fig. 4. Figure 5 depicts the functional analyses with undifferentiated results.

Figure 1.

Participants whose problem behaviour is maintained by negative reinforcement in the form of escape from demands: (a) intervals with screaming; (b) rate of aggression; and (c) rate of self-injury.

Figure 2.

Participants whose problem behaviour is maintained by automatic reinforcement: (a) rate of self-injury; (b) rate of self-injury; and (c) rate of hand-mouthing.

Figure 3.

Participant whose problem behaviour is maintained by automatic reinforcement: intervals of stereotypy.

Figure 4.

Participant whose problem behaviour is maintained by multiple sources of reinforcement: (a) rate of aggression; (b) rate of self-injury; and (c) rate of self-injury.

Figure 5.

Participants whose functional analysis yielded undifferentiated results: (a) rate of aggression; (b) rate of aggression; and (c) rate of self-injury.

For the cases categorized as ‘undifferentiated’ (Fig. 5), there were insufficient data to suggest a primary function. A review of behavioural records for the three participants from the 3 months prior to the functional analyses indicated that participant 3 exhibited no incidents of aggression during that time, participant 6 exhibited an average of 0.9 incidents of aggression per day and participant 7 displayed an average of 0.03 self-injurious behaviours per day. Therefore, for these three participants, the target behaviours were of sufficient intensity to warrant referral for a functional analysis, but too infrequent to be assessed by this method.

Table 1 compares the functions of the target behaviour for each participant as identified by the analogue sessions, QABF and MAS. The QABF and analogue sessions agree on what occurred in 56.3% of cases, the MAS and analogue sessions on 43.8% of cases, and the QABF and MAS on 61.5% of cases.

Table 1.  Identified behavioural functions in the participants: (MAS) Motivation Assessment Scale; and (QABF) Questions About Behavioral Function scale
 Behavioural function(s)
ParticipantAnalogue sessionsMASQABF
1Attention, tangibleAttention, tangibleAttention, tangible
2Tangible, escapeTangibleTangible, escape
3UndifferentiatedAttentionAttention
4AutomaticAutomaticAutomatic
5EscapeAutomaticAutomatic
6UndifferentiatedTangibleAutomatic, tangible
7UndifferentiatedAutomaticAutomatic
8AutomaticEscapeAutomatic
9Automatic, escapeEscapeEscape
10AutomaticAutomaticEscape
11EscapeEscapeEscape
12AutomaticEscapeEscape
13EscapeAutomaticEscape

Moderate correlations were observed between the QABF and MAS. Unfortunately, some of these subscales were similar while others were not. Table 2 outlines the Spearman rank-order correlation coefficients between the MAS and QABF. However, given the small sample size, these correlations must be interpreted with caution, although this is a larger number of subjects relative to some other analogue functional analysis studies. With an total number of 13, a correlation would have to exceed approximately 0.475 to achieve statistical significance at the P < 0.05 level (Glass & Hopkins 1996).

Table 2.  Spearman rank-order correlations between the Motivation Assessment Scale (MAS) and the Questions About Behavioral Function scale (QABF)
 MAS
QABFSensoryEscapeAttentionTangible
  • *

    P < 0.01.

  • **

    P < 0.001.

Attention0.247 −0.2780.5120.410
Escape0.4660.508 −0.1350.126
Non-social0.794**0.0380.2690.657*
Physical0.796**0.0490.3380.815**
Tangible0.540 −0.1920.4350.857**

Discussion

Convergent validity was assessed through comparison of the QABF to analogue functional analysis. In addition, to determine the agreement and degree of validity of the QABF in relation to the MAS, comparisons were made between the MAS, analogue sessions and QABF. The assessment of convergent validity resulted in lower agreement between the QABF and analogue sessions than would be ideal. Agreement between the MAS and analogue sessions was still lower, while agreement between the QABF and MAS was highest. It would appear that the two checklists tap similar content domains, but do not correspond well to analogue sessions. Still, it is important to consider that the QABF has thus far been demonstrated to have very good psychometric properties (Paclawskyj et al. Year?).

A factor affecting the current results is the inclusion of cases with low-frequency behaviours resulting in functional analysis outcomes that were undifferentiated. In the present study, 23% of the participants had undifferentiated functional analysis results. Low-frequency but high-intensity behaviours are under-investigated in current functional analysis research (Sturmey 1995), and therefore, this is an area that would benefit from attention from researchers. The inclusion of participants with undifferentiated analysis results had a significant impact on the convergent validity portion of the present study. Exclusion of the data from the three participants with the undifferentiated functions would serve to increase the percentage agreement between the QABF and analogue sessions to 69.2%, and the agreement between the MAS and analogue sessions to 53.8%. The percentage agreement between the QABF and MAS would remain stable at 61.5%.

The results of the present study have implications for selecting effective procedures to identify the functions of problem behaviour. Researchers have previously focused of the use of analogue procedures to identify behavioural function. Unfortunately, some procedures are often not practical or appropriate to the behaviour of concern. The present authors have found the results of the QABF to have a moderate agreement (69.2%), when the undifferentiated cases are not included, with the analogue analyses. This represents a better agreement than what was found with the more widely used MAS (53.8%). Therefore, these data further indicate the use of a checklist, specifically the QABF, within a hierarchical model of functional assessment. Hence, more individualized and effective function-based treatments can be developed, and assist in programming for individuals with ID and behaviour problems. Future research may examine the relative treatment efficacy of interventions based on either the results of the QABF or analogue analyses.

Acknowledgements

This study is based on portions of a dissertation submitted by the first author (T.R.P.) in partial fulfilment of the degree of Doctor of Philosophy in the Department of Psychology at the Louisiana State University, Baton Rouge, LA, USA. The authors would like to thank the staff at Pinecrest Developmental Center for their assistance in the study.

Ancillary