• This study was completed with support from the Korn Learning, Assessments, and Social Skills (KLASS) Center at The University of Tennessee.

Correspondence to: Christopher H. Skinner, University of Tennessee-EPC, BEC 535, Knoxville, TN 37996–3452. E-mail: cskinne1@utk.edu


The success of Response-to-Intervention (RTI) and similar models of service delivery is dependent on educators being able to apply effective and efficient remedial procedures. In the process of implementing problem-solving RTI models, school psychologists have an opportunity to contribute to and enhance the quality of our remedial-procedure evidence base. In this article, we describe and analyze how the broad-scale implementation of RTI may allow school psychologists to collaborate with others to apply, develop, adopt, and adapt contextually valid remedial and research design procedures. To capitalize on this opportunity, graduate training in school psychology must be enhanced to focus on the application of repeated measures design in applied settings using more precise and sensitive measurement and evaluation procedures. Such strategies should prevent us from advocating for procedures that cannot be applied in educational contexts and/or are ineffective. This will also encourage comparative effectiveness studies that can be used to determine which procedures remedy problems the quickest.

Across models of service delivery, educators continue to apply different theories, strategies, and procedures designed to remedy skill deficits before they become so severe that they interfere with future learning and/or require the implementation of more intense services (Deno & Mirkin, 1977; Fletcher, Coulter, Reschly, & Vaughn, 2004; Fuchs & Fuchs, 2006; Marston, Muyskens, Lau, & Canter, 2003; Shapiro, 1989; VanDerHeyden, Witt, & Barnett, 2005). Whereas Response-to-Intervention (RTI) models offer a relatively new approach for determining special education eligibility, other services, including collaborative problem-solving teams, consultation, and extended school-year services, have provided educators with the opportunity to address similar remediation goals (Curtis, Curtis, & Graden, 1988; Deno, 1995; Kratochwill, Elliott, & Callan-Stoiber, 2002; Rosenfield & Gravois, 1996; Shapiro, 2011). An element common to most RTI models involves the use of standardized procedures for universal screening (e.g., assessing all students within a local school or district using brief rate measures, such as words correct per minute) to establish data bases that allow educators to make across-student comparisons and ultimately identify students who may be in need of additional services (Compton, Fuchs, & Fuchs, 2006; Fuchs, Fuchs, & Compton, 2004).

With RTI and other models, once students with skill deficits are identified, additional data are used to guide a variety of decisions about the intervention plan used to remedy the problem (e.g., setting, personnel, and time). Although the data collected, analysis procedures, and criteria for making formative decisions vary across these models (Hughes & Dexter 2011; VanDerHeyden et al., 2005), these decisions are based on students’ learning levels (i.e., current levels of achievement without regard for time spent learning) and responsiveness data (Shapiro, 2011). Thus, RTI is similar to earlier data-based decision models in that once a remedial procedure (i.e., intervention) is defined and implemented, skill development is frequently assessed and these data are used to make formative decisions, including whether to maintain, alter, supplement, and/or change the specific remedial procedures that were selected and utilized (e.g., Deno & Mirkin, 1977; Shapiro, 1989).

When applying RTI models, reliable and valid procedures are needed to assess both skill development levels (i.e., current achievement) and learning rates (i.e., amount of learning divided by time spent learning); however, without access to effective and sustainable remediation procedures, educators may find they have little learning or growth to measure. Educators and researchers have stressed that remedial strategies and procedures be scientifically supported, data based, empirically validated, and/or evidence based (Detrich, Keyworth, & States, 2007; Drake, Latimer, Leff, McHugo, & Burns, 2004; Fuchs & Fuchs, 2006; Kazdin, 2004; Sheridan, 2001; Shriver & Watson, 2005; Stoiber & Kratochwill, 2000). Additionally, educators and researchers have stressed that these remedial procedures and/or interventions be implemented with integrity, as valid conclusions about student responsiveness to an intervention cannot be made without clear evidence that remedial procedures were implemented as intended (Gresham, Gansle, & Noell, 1993; Gresham, MacMillan, Beebe-Frankenberger, & Bocian, 2000).

RTI: An Opportunity to Enhance our Remedial-Procedure Evidence Base

An evaluation of student “responsiveness” also can be conceptualized as an evaluation of the remedial procedure utilized. These evaluations provide an opportunity for school psychologists to become involved in the process of collecting empirical evidence that supports the effectiveness and applicability of particular interventions. Resources for evaluating the effectiveness of remedial procedures may have been scarce with earlier remediation models (e.g., problem-solving teams). However, with RTI, because responsiveness evaluations are often used to inform high-stakes decisions regarding special education eligibility, more resources may be available to evaluate the effectiveness of various remedial procedures and to document that these interventions are applied with integrity. Thus, our central thesis is that the broad-scale implementation of RTI provides school psychologists with the opportunity to contribute to the process of developing, adopting, adapting, and empirically validating remedial procedures.

Because no single remediation procedure is likely to be effective for all students, educators need a pool of validated remedial procedures (Deno & Mirkin, 1977; Saecker, Skinner, Brown, & Roberts, 2009; Shapiro, 2011). Just as a variety of procedures are needed to meet the various needs of a diverse student population, this pool of remediation procedures also must address various contextual factors, such as the willingness of teachers to carry out a treatment (e.g., acceptability), the size of the class, student time available to devote to the procedure, and other human resource constraints, such as the availability of teachers to apply interventions (Foster & Skinner, 2011; Skinner & Skinner, 2007). Consequently, rather than concentrating on how RTI may be used to evaluate existing standard protocol interventions, we will focus our discussion on how the implementation of problem-solving models of RTI may provide school psychologists with the opportunity to become involved in the development and validation of novel and contextually valid remedial procedures.

The process of placing students into special education based on failure to respond to validated remedial procedures can also be viewed as a call for researchers to scientifically evaluate learning strategies, procedures, and interventions. Consistent with this focus, we describe how the implementation of RTI can provide the impetus for school psychologists to collaborate with teachers to enhance the empirical validation process and the pool of empirically validated interventions. Working with teachers in school-based settings requires school psychologists to focus on developing contextually valid intervention procedures that can be developed, applied, and sustained in specific classrooms. Similarly, we will need to apply and develop contextually valid research methods (e.g., single-subject designs) that can be used to validate remedial procedures within school and classroom contexts.

In addition to addressing contextual valid intervention and research procedures, we analyze how the implementation of RTI should encourage school psychologists to focus on applied measurement issues. Thus, we describe how the large standard error of measure associated with some commonly used repeated measures affects our ability evaluate remedial procedures in a timely fashion. Additionally, we analyze how this applied focus on measuring responsiveness should encourage researchers to develop better repeated measures that can be used to evaluate learning rates and provide an indication of which remedial procedures work best.

By collaborating with teachers on remedial-procedure development, implementation, and evaluation, we can (1) decrease the probability that practitioners in the field develop and promote remedial procedures that cannot be applied in context, (2) enhance school psychologists’ credibility with educators who must apply remedial procedures, (3) improve remedial research design and evaluation procedures, (4) increase our pool of effective, empirically validated remedial procedures, (5) enhance our ability to evaluate remediation efforts more rapidly, and (6) enhance our ability to quickly identify which interventions remedy problems.

Taking Advantage of the Opportunity

To take full advantage of this opportunity to contribute to the advancement of remedial procedures, school psychologists may need additional training in modifying remedial procedures to fit applied context, repeated measures designs, and measuring learning rates. Although applying similar assessment, identification, remediation, evaluation, and decision-making procedures across groups (e.g., all second-grade students in a school system) may help address many students’ needs (Hughes & Dexter, 2011), in other instances, standardized procedures may need to be modified because of contextual concerns (e.g., not enough computers in a classroom) or idiosyncratic student needs or preferences. The ability to modify remediation procedures such that they are effective and able to be applied in a specific context requires an understanding of how specific intervention components or strategies interact with one another and ultimately influence learning. Therefore, in addition to being able to identify and apply empirically validated interventions, school psychologists must have sufficient understanding of (1) general procedures (e.g., self-evaluation), (2) theories about why these general procedures work, and (3) how they can be integrated with other intervention components. Although some of this training can come from basic coursework in learning, preservice school psychologists would benefit from practical opportunities to analyze complex applied interventions, break them into their component parts, use their understanding of learning principles/theories to identify the function of each component, and describe how the components interact to form a cohesive intervention. Through these kinds of activities, school psychology students will be in better positions to contribute to the process of modifying empirically validated interventions to address students’ learning difficulties within a specific context.

School-based practica that allow preservice school psychologists to work with struggling students (e.g., nonresponders under RTI) should provide opportunities to develop, evaluate, and modify remedial procedures. In such placements, students should be challenged to develop procedures that are effective and require little time to effect positive change. This challenge will not only influence their remediation strategy selection and development, but should also influence how they evaluate their remediation strategies. Specifically, in evaluating their procedures’ effectiveness, school psychologists should be trained to use more precise measures that reflect the amount of time students spend in specific remedial activities (e.g., learning rates) to determine which procedures remedy deficits most rapidly (Skinner, 2008; Skinner, 2010).

Contextually Valid Remedial Procedures

Evidence of a remedial procedure's internal, external, and contextual validity should enhance educators’ ability to evaluate the procedure and determine whether it is worth using or an alternative procedure should be applied. Internal validity is supported by evidence that a specific independent variable (e.g., the intervention selected), as opposed to other variable(s), caused a change in a dependent variable (i.e., the outcome). The internal validity of any given remedial procedure would be established via evidence that within a particular context, the procedure enhanced the targeted student's skill development. To convince educators that an internally valid procedure can help them remedy their students’ problems, evidence of the procedure's external validity is also needed. In this case, evidence is needed to suggest that the remedial procedure will be effective with students who were not study participants (i.e., generalization across students), in other settings, when applied by other educators/researchers, and perhaps when addressing different skills and/or larger or smaller skill deficits (Campbell & Stanley, 1966; Ringeisen, Henderson, & Hoagwood, 2003; Skinner & Skinner, 2007).

Contextually Valid Interventions

Establishing an intervention's internal and external validity is critical to convincing educators that the procedure may help them remedy a student's academic skill deficit(s), but it is not sufficient. We will use the term contextual validity herein to refer to evidence that serves to enhance educators’ confidence that they can apply a remedial procedure (e.g., they possess or can obtain the skills and resources needed to apply the procedure) and sustain the application of this procedure, given presenting contextual variables (Blondin, Skinner, Parkhurst, Wood, & Snyder, 2012; Foster & Skinner, 2011; Skinner & Skinner, 2007). Some examples of common threats to contextual validity include the time, materials, and training needed to learn, develop, implement, and sustain prevention/remediation activities (Foster & Skinner, 2011).

Negative side effects associated with internally and externally valid remedial procedures represent another class of threats to contextual validity (Blondin et al., 2012; Foster & Skinner, 2011; Ringeisen et al., 2003). Within the context of their schools and classrooms, educators may have concerns over the impact of remedial procedures on other activities (Detrich et al., 2007; Kazdin, 2004; Kratochwill & Shernoff, 2004; Shriver, 2007; Skinner & Skinner, 2007). For example, re-allocating a substantial amount of time from recess and physical education to remedial reading procedures may hinder students’ social skills development (Skinner, 2008). Moreover, new remedial procedures being applied can interact with and adversely affect standard classroom instruction and learning (Popkin & Skinner, 2003). For example, teaching students to solve basic math facts using counting procedures may inhibit their ability to develop automaticity with math facts (Poncy, Skinner, & O'Mara, 2006; Ysseldyke, Thill, Pohl, & Bolt, 2005). Some remedial procedures may cause improvements in select students, while adversely affecting peers (Fudge, Skinner, Williams, Clark, & Bliss, 2008; Skinner, Skinner, & Burton, 2009). Although these concerns are valid, because educational researchers do not routinely assess for negative side effects, data addressing the concerns are often nonexistent, scant, incidental, or collected using indirect measures, such as posttreatment questionnaires.

Collecting and Disseminating the Evidence Base

Because idiosyncratic and contextual differences are often apparent, educators would benefit from a broad pool of validated intervention procedures. However, the proportion of journal articles that directly evaluate the effectiveness of remedial procedures is small relative to the amount of narrative or correlational articles available (Floyd et al., 2011; Gresham et al., 2000; Harrison, 2000; Hseih et al., 2005; Robinson, Skinner, & Brown, 1998; Seethaler & Fuchs, 2005). For example, a review of four major school psychology journals shows that only 8% to 11% of the articles involved the empirical evaluation of interventions (Bliss, Skinner, Hautau, & Carroll, 2008). Additionally, many journals have editorial policies that prioritize the publication of articles establishing the internal validity of novel strategies and/or procedures (i.e., demonstrating a novel intervention-caused change), as opposed to establishing the external and/or contextual validity of procedures (e.g., demonstrating how the procedures can be adapted and applied in another classroom or with another teacher who wants to target additional behaviors yet still be effective; see Blondin et al., 2012) that have already been shown to cause learning (Detrich et al., 2007). Because establishing external and contextual validity frequently requires numerous replication and extension studies, relatively few remedial procedures have evidence of all three types of validity (Skinner & Skinner, 2007). Fortunately, some journal editors have begun to focus on these issues (see Shriver & Watson, 2005, and Wodrich, 2009).

Educators and researchers have posited several reasons for the lack of applied, field-based, intervention research published in school psychology journals. For example, field-based intervention research requires substantial time and cooperation (Reschly, 2000). To conduct intervention studies, researchers often must first obtain informed consent from parents and teachers, assent from child participants, and institutional approval from both university and school system human subjects boards. Interventions must be developed and implemented, and appropriate assessment procedures must be used to gauge intervention effectiveness. Researchers must also verify the quality of both the intervention (e.g., treatment integrity) and the outcome measures (e.g., reliability, validity, inter-rater agreement). Typically, methodological (e.g., research design elements) and statistical procedures are needed to control for threats to internal validity (Bliss et al., 2008; Skinner, 2004). Although it may be easier and more efficient to write a narrative article or produce methodologically sound descriptive or correlational research than to conduct a study designed to validate the efficacy of an intervention or prevention program (Strein, Cramer, & Lawser, 2003), the broad-based implementation of RTI may mitigate many of the difficulties associated with empirically validating interventions.

Contextually Valid Repeated Measures Designs

Although many education professionals have some understanding of how true experimental designs can be used to evaluate treatments, often they have been unable to use these procedures in their applied settings because random assignment of students to remedial procedures may not be possible and is likely to be met with resistance (Campbell & Stanley, 1966; Gresham, 2004; Kazdin, 2004; Keith, 2000). One difference between RTI and other remedial models of service delivery is the emphasis on repeated, systematic assessment of student performance to evaluate remediation effectiveness and guide formative decisions. Although these data are often used to make numerous decisions regarding a student's educational program, they can also be used to evaluate remediation procedures. School psychologists can assist the process of enhancing our remedial-procedure evidence base by addressing threats to internal, external, and contextual validity during these responsiveness evaluations.

The repeated measurement procedures used during RTI can allow for a more rapid and precise evaluation of learning (responsiveness) than traditional pretest–posttest designs (Christ, 2006; Gresham & Noell, 1998; Skinner, 2004). Therefore, RTI assessment procedures may provide a rich data base that allows for the application of time-series methods (e.g., single-subject designs) to control for threats to internal validity. As RTI procedures are applied across students (e.g., 15% commonly receive some remedial service), they allow for numerous replications across students, teachers, classrooms, target behaviors, and other contexts that may yield evidence of external and contextual validity (Skinner, 2004; Skinner & Skinner, 2007). As remedial procedures may not be applied concurrently across students (e.g., students move in and out of remedial services at different times), the implementation of RTI should allow educators/researchers to establish the internal validity of remedial procedures by applying nonconcurrent multiple baseline designs (Christ, 2007; Skinner, 2004; Winn, Allin, Hawkins, & Skinner, 2004).

Enhancing the Data Base via Collaboration and Adaptation

Although not all educators and educational researchers have been trained in single-subject design procedures, school psychologists should have some training and experience with these repeated-measures, time-series designs (Keith, 2008; Skinner, 2004). Given that school psychology professionals frequently work with general educators and others responsible for delivering remedial procedures, opportunities should exist to collaborate in their efforts to develop, implement, and validate remedial procedures, from which new applied research methods and analysis procedures are likely to emerge. For example, educators and researchers have already begun merging single-subject design methods with group statistical procedures (Gorman & Allison, 1997; Olive & Smith, 2005; Parker, 1995; Parker & Vannest, 2012).

If only exact replications of specific remedial procedures with strong validity are applied, then it is unclear how practitioners collaborating with researchers are going to help develop and validate novel and effective remedial procedures. This problem is not as serious as it appears. Because educators rarely can (or perhaps rarely should) apply the exact same procedures, problem-solving models of remediation are evolutionary and will result in the development of entirely new strategies, procedures, and interventions (Skinner, in press). Terms such as scientifically supported strategies are often used to refer to a general class of procedures (e.g., increasing opportunities to respond, self-directed learning, and differential reinforcement). Although a very specific procedure or intervention may have been empirically validated, in practice, educators often have to adapt operationally defined procedures to fit their target student(s), target behavior(s), and context (e.g., apply a strategy three times per week for 20 minutes as opposed to five times per week for 10 minutes). Thus, in practice, educators are rarely implementing exact replications of procedures carried out by previous researchers; rather, they are re-applying and adapting general strategies as they develop, apply, and evaluate their novel remediation procedures (Skinner, 2004). In many instances, effective and creative teachers may not even recognize that they have developed novel strategies or procedures.

Working collaboratively with educators should help to encourage researchers to refine or adapt existing remediation strategies so that they can be implemented with integrity and sustained in the classroom/school context (Carroll, Skinner, McCleary, Hautau von Mizner, & Bliss, 2009; Detrich et al., 2007; Skinner & Skinner, 2007). Additionally, as various educational professionals and researchers collaborate to remedy academic skills, they are likely to share theories, strategies, and procedures that lead to the development of novel concepts and procedures (Parker, Skinner, Booher, & Crisp-Turner, 2010). These variations in remediation and evaluation procedures can provide additional support for the internal, external, and contextual validity of a general strategy, while providing additional theoretical evidence regarding causal variables that influence learning (Skinner, 2010). Thus, the evolutionary process of adapting interventions to environments, contexts, students, and goals should help to facilitate the development of original learning theories and interventions.

Applied Remediation Research Training

Many professional training programs focus on teaching preservice educators research methods that allow them to conduct studies provided they can select their problems, have enough time to plan procedures, have enough resources to implement these procedures, and are able to control for any extraneous factors. Applied school-based remediation research is often reactive (meaning the researcher is reacting to a real-world problem as opposed to selecting the problem he or she wants to study), requires rapid work (so children do not fall further behind), must be implemented with few resources (as opposed to grant-funded work), and occurs within a natural setting, as opposed to a context where experimenters can control the environment. As teachers attempt to remedy students’ skill deficits, they often develop and apply novel procedures that have strong contextual validity (otherwise they would not develop or apply them), and many teachers may want to share these procedures with their peers. Educators who have been pitched remedial procedures that 1) have not been validated, 2) result in disappointment when they discover they cannot apply the validated procedure, and/or 3) do not result in enough learning given the cost (e.g., student instructional time used) may be reluctant share their novel procedure with peers unless they have strong evidence of its validity (e.g., effectiveness, efficiency).

As most graduate school psychology programs provide training in basic learning theories, applied academic interventions, and collaborative problem-solving models, school psychologists are encouraged to take a more active role by working with educators to develop remedial procedures, validate these procedures, and disseminate the validation evidence. The broad-scale implementation of RTI affords an opportunity for all school psychologists to engage in such activities. Traditional social science training in quasi-experimental and experimental designs may prove useful in evaluating standard protocol remedial curricula and/or procedures that have already been developed. Additionally, single-subject design procedures may allow school psychologists to use RTI data to evaluate developed, applied, and altered remedial procedures using problem-solving models. Such evaluations will allow us to continue and enhance the process of empirically validating many novel strategies and procedures that teachers develop and apply every day.

Measuring Learning Rates

Establishing that a remedial strategy or procedure “works” (e.g., it is better than nothing) is important, but when faced with a pool of valid remedial strategies or procedures, educators are ultimately interested in knowing what “works best” (Skinner, Belfiore, & Watson, 1995/2002). Comparative effectiveness studies (i.e., studies that compare the effects of one procedure with another) may inform educators which valid remedial procedures work best (Skinner, 2008). Often, research designed to compare intervention effectiveness requires a much greater degree of precision than research designed to establish effectiveness (Barlow & Hayes, 1979). Fortunately, researchers and educators have begun to focus on precise measures of both behavior change and time spent learning (Cates et al., 2003; Christ, 2006; Skinner, Belfiore, Mace, Williams, & Johns, 1997). This enhanced precision will help to improve our ability to prevent and remedy learning problems by allowing researchers and educators to develop theories and procedures based on patterns in the data that have previously been hidden by imprecise measures of learning (Skinner, 2008). This precision can come from modifying both our measures of instructional time and our measures of behavior change.

Precise Measures of Cumulative Instructional/Learning Time: The Horizontal Axis

Learning has frequently been defined as a relatively permanent change in behavior or behavioral potential brought about by experience. Because all experiences depend on and require time, the amount of time spent engaged in the learning experience is a constant variable in all learning. Additionally, learning time is a critical factor that must be taken into account when evaluating the efficacy of interventions (Skinner, Belfiore, & Watson, 1995/2002). In defining the construct of learning as amount of skill development divided by time spent learning, some researchers have used the term “learning rate” (Skinner, Belfiore, & Watson, 1995/2002). Many researchers evaluating learning procedures have established that a procedure is effective, but typically they have either neglected to measure learning rates or measured them imprecisely (Bramlett, Cates, Savina, & Lauinger, 2010). Evaluations that provide some indication that a procedure enhanced learning may have important theoretical implications, but because they provide little or no indication of the remediation procedure's impact on learning rates their applied value is questionable (Skinner, 2008, 2010; Skinner, Belfiore, & Watson, 1995/2002).

Applied Significance of Learning Rates

The applied case for precisely measuring learning rates has come to the forefront as we attempt to apply RTI models. All remedial services require student time (Cates et al., 2003; Skinner, 2010). As time cannot be manipulated, it must be re-allocated (Skinner, Belfiore, & Watson, 1995/2002). For example, those receiving remedial services in extended-school-year (summer school) or extended-school-day (after-school tutoring) programs have their time re-allocated from other activities (e.g., summer camp, learning to swim, organized team sports, playing computer game). However, with most RTI models, time applied to remedy skill deficits is re-allocated from other in-school activities (e.g., music, recess, physical education, and/or some other academic area). Although this is acceptable to some, other stakeholders (e.g., teachers, students, parents, administrators, citizens) often express concern about the negative side effects associated with removing a child from typical school activities. These concerns are legitimate, as quality of life and future productivity may be affected by the habits and skills developed during recess (social skill development), physical education, and music and art classes (Skinner, 2008). Thus, re-allocating school time is a contextual validity limitation associated with many RTI service-delivery models (Skinner 2010).

Numerous remedial procedures can be applied via RTI; however, because time from one school activity is re-allocated to activities designed to remedy deficits, the application of RTI may enhance educators’ and applied researchers’ attention to the time allotted to remedial procedures (Skinner, 2008). For example, the focus on remedial time is embedded in many RTI models, where early-tier procedures may require educators to allocate a portion of time (e.g., an additional 30 minutes per day) for remedial activities. If these procedures do not remedy a skill deficit, then students may move to a more advanced tier, where even more time is allocated to remediation (e.g., an additional 60 minutes per day). Thus, within RTI models, dosage or intervention intensity is often equivalent to time allocated for remediation. Both the requirement to re-allocate time during the school day and the focus on instructional time as intensity (e.g., if lower tier procedures do not work, then more time-consuming or intensive procedures are applied in the next tier) should provide an opportunity for school psychologists and others who are interested in validating remedial strategies and procedures to more precisely measure learning as a function of the time the student spends engaged in remedial procedures.

Two Pictures and a 432-Word History Are Worth 2,000 Words

In 1993, educators and researchers used an alternating treatments design to compare the effects of two taped-words (TW) interventions on students’ sight-word reading (Skinner, Johnson, Larkin, Lessley, & Glowacki, 1995). Two sets of equivalent words were identified and read into a tape recorder. One set was read at a rate of one word per second; the other was read at a rate of one word every 5 seconds. During the intervention, students were instructed to read the words along with the tape. The theoretical implications being investigated were related to hypothetical causal models (i.e., neurological impress and modeling) which suggest that student reading fluency would be enhanced by the more rapid readings.

The Skinner, Johnson et al. study was accepted for publication in 1994 and published in 1995. While awaiting publication, the researchers realized the procedure for evaluating the two TW interventions was imprecise and from an applied perspective, might mislead readers interested in knowing which procedure resulted in superior learning. Figure 1 provides Bob's results from the initial Skinner, Johnson et al. (1995) article. Visual analysis of Figure 1 suggests little difference in learning rates (slopes) across the interventions. Note that in Figure 1, the horizontal axis is labeled “School Days.” In a subsequent article designed to correct any misunderstanding caused by the earlier Skinner, Johnson et al. article (1995), Skinner, Belfiore, and Watson, (1995/2002) reconstructed and re-analyzed Bob's data (see Figure 2); however, rather than using a crude measure of instructional time (e.g., cumulative instructional sessions or school days), the researcher applied a more precise measure of actual time spent in each remedial procedure, defined as the cumulative number of instructional seconds. Figure 2 shows that in the first 2,640 seconds allotted for learning under each intervention, the rapid-rate TW intervention doubled Bob's word reading fluency relative to the slow-rate (5 seconds) TW intervention.

Figure 1.

Number of words Bob mastered per cumulative school days.

Figure 2.

Number of words Bob mastered per cumulative instructional seconds.

Skinner, Belfiore et al. (1995/2002) demonstrated how precisely measuring learning rates allowed researchers to establish that the rapid rates of presentation (1 second) would result in more rapid increases in Bob's word reading. Other educators and researchers have conducted similar studies and shown that strategies and procedures that appear to enhance learning rates (when researchers evaluate them using crude measures of instructional time) actually retard learning (Cates et al., 2003; Joseph & Nist, 2006; Nist & Joseph, 2008; Schisler, Joseph, Konrad, & Alber-Morgan, 2010; Skinner et al., 1997). Others have compared intervention effectiveness more precisely by holding instructional time constant across competing remedial procedures (e.g., Carroll, Skinner, Turner, McCallum, & Masters, 2006; Grafman & Cates, 2010; Poncy, Skinner, & Jaspers, 2007; Skinner et al., 1997).

RTI and Learning Rates

Although many researchers have focused on internal validity, (i.e., what works), as educators and researchers implementing RTI models adopt a model of efficiency (i.e., optimal balance among treatment dose/exposure, intervention cost, and intervention effect), moving forward they will need to focus on developing and evaluating remedial procedures that enhance skills or learning most rapidly (Skinner, Fletcher, & Henington, 1996). This focus on precise measurement of learning rates has begun to spread from applied university researchers (Cates et al., 2003; Joseph & Nist, 2006; Skinner, Belfiore et al., 1997) to RTI researchers who have begun to evaluate and compare interventions as they are applied for different specific intervals or schedules (Bryant et al., 2008; Fielding, Kerr, & Rosier, 2007; O'Connor, Harty, & Fulmer, 2005). For example, such data may inform us that intervention x may be superior to y when both are applied for 30 minutes every other day, but y is superior to x when both are applied for 15 minutes every school day (Skinner, 2008). Such nuanced comparisons cannot be answered when instructional time is measured as sessions or school days (Skinner, 2010).

Looking ahead, the advent of RTI models is likely to sustain and enhance this focus on learning time and precise measures of learning rates. As educators implement RTI models of service delivery, stakeholders will begin to demand the most efficient treatments so their students can return to their regular schedule of activities more quickly (Skinner, 2008). Consequently, applied researchers, especially those conducting relative effectiveness studies (e.g., comparing remedial procedures) will be encouraged by practitioners to alter their dependent variables (i.e., measures of learning outcomes) by incorporating precise measures of instructional and/or learning time so that remedial strategies that result in the most rapid remediation of deficits can be identified (Skinner, 2010).

Given the demand for identifying the most rapid remediation procedures, school psychologists must do more than emphasize what works, but rather, must direct our attention toward what works best (Skinner, in press). Thus, as the field of school psychology moves forward, training and requiring school psychologists to more carefully consider the amount of student learning time empirically validated procedures require and the amount of time they require relative to other effective procedures is critical. Although many school psychologists may be trained on strategies and procedures that can supplement interventions and enhance learning, we must consider the additional time these extra procedures require. For example, trainers should challenge their students to examine empirically validated remedial procedures (what works) and strategies that may allow them to enhance learning rates by altering validated procedures so they require less time (see Skinner et al., 1996; Skinner et al., 1997). Finally, applied school psychology researchers must be properly trained and subsequently required to apply evaluation procedures to include more precise measures of cumulative learning time (Skinner, 2008).

Better Measures of Behavior Change: The Vertical Axis

In Figure 2, the horizontal axis provides a precise indication of time spent with the remedial procedures, whereas the vertical axis reflects change in the dependent variable or learning that has occurred. Enhancing the precision of both axes will improve our ability to evaluate remedial procedures. With RTI, as educators make more decisions (e.g., whether to continue or alter remedial procedures), including high-stakes decisions (e.g., are they failing to respond and in need of special education services) based on learning rates or student responsiveness, the demands for more valid, reliable, sensitive, and precise estimates of student skill development (vertical axes in Figures 1 and 2) are likely to grow.

One commonly used and extensively researched measure of learning (i.e., vertical axis measures) is words correct per minute (WCPM) within the context of oral reading fluency (Reschly, Busch, Betts, Deno, & Long, 2009). Although correlations (ranging from .85 to .95) suggest that WCPM is a reliable measure (Marston, 1989), it is not uncommon for educators who routinely use CBM procedures to comment on the amount of within-student variability or “bounce,” associated with these data; they have trouble understanding how a student could improve dramatically or lose skills so quickly. Of course, these levels of skill change did not occur so quickly; rather, the observed changes are most likely measurement error (Neddenriep, Poncy, & Skinner, 2011). One reason for the strong reliability estimates is that the majority of the reported studies have focused on the consistency of inter-student comparisons (e.g., how students rank relative to each other across two or more measures), which is appropriate when evaluating a measure for screening purposes. However, far fewer educators and researchers have examined the consistency of intra-individual data (e.g., comparing the student's performances across multiple data points). Intra-individual consistency of measures is a critical consideration in making appropriate formative decisions, including evaluating the effectiveness of a given remedial procedure with a specific student (Christ, 2006; Hintze, Owen, Shapiro, & Daly, 2000; Poncy, Skinner, & Axtell, 2005).

Sensitivity: A Double-Edged Sword of Rate-Based Measures

The extent to which WCPM data are sensitive to small changes in learning or skill development influences their usefulness for evaluating remedial procedures. For example, educators would like to expediently determine how effective remediation procedures are so they do not needlessly expend valuable time and resources, and possibly hinder student development with ineffective procedures. Although researchers have found evidence that WCPM is a reliable and sensitive measure for estimating both the level (Christ & Silberglitt, 2007; Poncy et al., 2005) and trend (Christ, 2006; Hintze et al., 2000) of reading skill development, the high reliability-like coefficients associated with WCPM scores can have a large standard error of measurement (Christ & Silberglitt, 2007; Francis et al., 2008; Poncy et al., 2005), which suggests that WCPM data may be sensitive to other factors.

WCPM data have been shown to be sensitive to a variety of extraneous variables, including testing conditions, directions, probe difficulty levels, and administration and scoring errors (Colón & Kranzler, 2006; Derr-Minneci & Shapiro, 1992; Hintze & Christ, 2004; Poncy et al., 2005). Derr-Minneci and Shapiro (1992) found that numerous testing conditions, including who administered the test (teacher or psychologist), where it was given (teacher's desk or psychologist's office), and the task demand (timed and untimed probes) influenced WCPM scores. Colón and Kranzler (2006) changed one sentence in the administration directions from “Do your best reading” to “Read as fast as you can without making mistakes” and found an average increase of 36 WCPM in fifth-grade students, an alarming finding considering that Ardoin and Christ (2008) found that fifth-grade students grew an average of 32 to 33 WCPM over the course of a year. Poncy et al. (2005) found that 81% of the variability in WCPM scores was due to student skill, whereas 10% of the variability was due to the unequal passage difficulty of the probes utilized. Although 10% variance accounted for by passages may appear small, administering one probe to estimate the student's level of skill development yields a standard error of measure of ± 18 WCPM, a 36 WCPM range in scores.

Enhancing Evaluation With Superior Measures of Learning

Poncy et al. (2005) suggested that when educators are monitoring progress with weekly probes, it is not uncommon to find that a child reads 67 WCPM one week, 81 WCPM the next week, and 53 WCPM the final week. Such patterns may be perplexing to educators who do not understand how the student's performance could fluctuate so vastly. Attributing this large degree of bounce as a function of measurement error may be accurate, but it does nothing to address educators’ concerns over the prospect of making inappropriate decisions based on highly variable data. Also, imprecise measurement can make it nearly impossible for researchers to make fine distinctions, such as determining which interventions cause the most rapid remediation.

The widespread implementation of RTI has raised concerns over measurement error and has encouraged the development of strategies and procedures that may reduce this error. Some of these sources of error are more easily addressed than are others. For example, administration conditions (e.g., directions, timing procedures, testing location, administrator qualities) can be controlled, to some extent, with strict administration guidelines. However, given that merely seeing a stopwatch can influence WCPM scores (Derr-Minneci & Shapiro, 1992), it is likely that there are other sources of error that we have yet to identify. Perhaps practitioners who frequently administer these measures can help to improve the process of reducing measurement error by identifying other variables that they suspect may influence WCPM performance.

To address error caused by passage difficulty, educators and researchers have begun to field test passages to develop a pool of more equivalent passages (Christ & Ardoin, 2009). Given the amount of error variance accounted for by variations in probe difficulty (see Poncy et al., 2005), this will likely result in significant and meaningful reduction in error or “bounce” observed when testing students using CBM procedures. An alternative to field testing probes is to increase the number and/or frequency of assessments. Poncy et al., (2005) found that the standard error of measurement of ± 18 WCPM for a single probe was reduced to ± 8 WCPM when performance across five probes was averaged. Similarly, Christ (2006) found that increasing the number of administered probes to a child significantly reduced the standard error of slope (i.e., SEb) when calculating the student growth rate. Combining both procedures by developing more equivalent probes and increasing the number and frequency of assessment should help educators and researchers to make more precise and rapid assessments of skill development. Additionally, researchers have begun to develop and evaluate alternative brief rate measures designed to evaluate reading and other skills that may prove superior to current measures (e.g., Hale et al., 2011; Miura-Wayman, Wallace, Ives-Wiley, Tichá, & Espin, 2007).

School psychologists receive training associated with traditional assessment procedures and psychometric properties that indicate the quality of the measures (e.g., reliability and validity); however, these traditional procedures may be misleading for evaluating the quality of repeated measures used to evaluate responsiveness (Ardoin, Roof, Klubnick, & Carfolite, 2008). Thus, traditional psychometric training may need to be supplemented with current research on commonly used RTI measures, including research addressing the amount of error associated with RTI measures, factors that contribute to this error (e.g., nonequivalent alternate forms, poor administration, and scores procedures), with an added emphasis on the importance of applying strategies that reduce this error, especially when making high stakes decisions (Christ, 2006; Christ & Silberglitt, 2007). For example, Christ, Zopluoglu, Long, and Monaghen (2012) found that even with carefully matched probes it may be necessary to administer 14 oral reading fluency assessments (one per week) in order to make decisions about responsiveness. Practicing school psychologists who are aware of these findings will position themselves to be more successful in encouraging more frequent responsiveness assessments during responsiveness evaluations. Furthermore, an understanding of the limitations of our evaluation systems may encourage the development of stronger evaluation procedures.

Implications for School Psychology Moving Forward

Many school psychology training programs require supervised consultation experiences, whereby graduate students work with educators to identify problems and develop, apply, and evaluate remedial procedures for the purpose of effecting positive changes in functioning. Despite this training, many practicing school psychologists may find that there is little support for them to become involved in the intervention development and validation process. As the development and application of RTI models provides a window of opportunity for practitioners to become more involved in such activities, it may be helpful for school psychologists to develop and share action plans that support colleagues who want to assert themselves into this role.

When working with teachers to remedy a specific student's academic problems, some school psychologists may find that they do not have the skills needed to demonstrate cause-and-effect relationships between remedial procedures and their measured behavior change. Applied researchers have and continue to focus their efforts of the development of single-subject design procedures to address this concern. Because those trained in single-subject design procedures may be better positioned to use evaluations of responsiveness to empirically validate interventions, we reiterate Keith's (2008) assertion that all school psychologists should receive training in single-subject design procedures.

Although validating remedial procedures is important, it is not in and of itself sufficient. For most stakeholders (parents, teachers, students), the goal of all remediation is to eliminate deficits as rapidly as possible so that the students can be successful in their development of subsequent skills and be allowed to engage in typical classroom activities, including recess, art, and music (Skinner, 2008, 2010). The ability to generate the evidence base needed to increase educators’ ability to remedy skill deficits as rapidly as possible will also be influenced by our ability to develop and apply sensitive, valid, reliable, and efficient measures of meaningful skill development, and precisely evaluate and compare student learning rates under different remedial procedures. The process of developing alternative assessment procedures is ongoing, and we are encouraged by the amount of effort that researchers are applying to developing and improving repeated measurement procedures. Thus, we believe our training programs and research activities are making meaningful progress on this front.

Having conducted and disseminated several studies demonstrating how more precise measures of instructional time change our conclusion regarding intervention effectiveness, we have noticed that practitioners almost always agree that we need to more precisely measure learning rates. Although many trainers and researchers also support these efforts, a focus on learning or instructional time may require many of us to reconsider the relative strength of some our preferred and effective (large effect size) interventions when these interventions require too much time, given the degree of learning they occasion. Perhaps our most important recommendation is that we shape practitioners and researchers to be less vested in particular procedures, strategies, programs, or curriculum and more focused on enhancing the effectiveness of these procedures by making them more efficient (Cates et al., 2003; Skinner et al., 1996; Skinner et al., 1997; Skinner, Ford, & Yunker, 1991). In addition to discouraging researchers from promoting procedures that cause slight improvement over other procedures but require much more student time, we should train and require applied researchers to employ more precise measures of learning rates so that practitioners can identify which procedure remedies deficits most rapidly (Skinner, 2008; Skinner, Belfiore, & Watson, 1995/2002).


There is evidence that the process of using RTI data to validate and compare remedial procedures has started and has influenced some school psychology researchers and trainers to focus on contextually valid remedial procedures that cause more learning in less time (Skinner, 2010). We encourage school psychologists and others to take advantage of this opportunity to work collaboratively with educators to use the wide-scale application of RTI to develop, validate, and disseminate remedial procedures. The widespread application of RTI may make it easier for school psychologists to become vested participants in the process, thereby improving the breadth and quality of our evidence-based procedures. Furthermore, school psychologists may be more inclined to apply evidence-based procedures when they are able to regularly contribute to that evidence base (Carroll et al., 2009). Consider how rich our remedial-procedure research base would become if we validated even 10% of the strategies or procedures developed and successfully applied by educators in their day-to-day activities. School psychologists may find these collaborative efforts proceed more smoothly when they apply or adapt remedial and research-design procedures to fit the presenting context, collect data that allows educators to compare among scientifically support remedial procedures, and select or develop those procedures that result in more rapid learning.