Food Safety Practices Assessment Tool: An Innovative Way to Test Food Safety Skills among Individuals with Special Needs

Authors

  • Elena T. Carbone,

    1. Author Carbone is with Univ. of Massachusetts, Dept. of Nutrition, 100 Holdsworth Way, Chenoweth Lab, Amherst, MA 01003–9282. Author Scarpati is with Univ. of Massachusetts, School of Education (Emeritus). Author Pivarnik is with Univ. of Rhode Island, Nutrition and Food Sciences Dept. Direct inquiries to author Carbone (Email: ecarbone@nutrition.umass.edu)
    Search for more papers by this author
  • Stanley E. Scarpati,

    1. Author Carbone is with Univ. of Massachusetts, Dept. of Nutrition, 100 Holdsworth Way, Chenoweth Lab, Amherst, MA 01003–9282. Author Scarpati is with Univ. of Massachusetts, School of Education (Emeritus). Author Pivarnik is with Univ. of Rhode Island, Nutrition and Food Sciences Dept. Direct inquiries to author Carbone (Email: ecarbone@nutrition.umass.edu)
    Search for more papers by this author
  • Lori F. Pivarnik

    1. Author Carbone is with Univ. of Massachusetts, Dept. of Nutrition, 100 Holdsworth Way, Chenoweth Lab, Amherst, MA 01003–9282. Author Scarpati is with Univ. of Massachusetts, School of Education (Emeritus). Author Pivarnik is with Univ. of Rhode Island, Nutrition and Food Sciences Dept. Direct inquiries to author Carbone (Email: ecarbone@nutrition.umass.edu)
    Search for more papers by this author

Abstract

Abstract:  This article describes an innovative assessment tool designed to evaluate the effectiveness of a food safety skills curriculum for learners receiving special education services. As schools respond to the increased demand for training students with special needs about food safety, the need for effective curricula and tools is also increasing. A Food Safety Education for High School and Transition Special Needs Students curriculum served as the basis upon which our assessment tool was developed. The project was a collaborative effort by food safety and education professionals in Connecticut, Rhode Island and Massachusetts. This USDA-funded initiative emerged from teacher-generated data that identified critical gaps in food safety knowledge and skills among students with disabilities (SWD) receiving special education services. As an adjunct to this curriculum, a Food Safety Practices Assessment Tool was developed to: 1) conduct observations of students as they demonstrate food safety practices, and 2) use this information to design classroom-based learning activities that are aligned with students’ Individual Education Plans (IEP). Pilot data suggest that the tool is valid and reliable for use in a kitchen-based setting. This is the first known tool of its kind to test food safety skills of individuals with special needs in a real-world environment. Further testing is needed to determine the usefulness of the tool for broader audiences.

The Centers for Disease Control and Prevention (CDC) estimates that foodborne diseases cause approximately 48 million illnesses, 130,000 hospitalizations, and 3,000 deaths each year (Scallan and others 2011a; Scallan and others 2011b). Federal agencies such as the US Dept. of Agriculture (USDA), US Food and Drug Administration (FDA), and CDC have responded to this public health issue by developing and implementing guidelines and interventions to address the need for highly qualified food safety workers. Education initiatives such as FightBac!® (Partnership for Food Safety Education 2009) and Be Food Safe (United States Dept. of Agriculture Food Safety Inspection Service 2008) have been developed for consumers and food service personnel to increase knowledge about food handling practices that can reduce risk of illness. While these initiatives reflect the critical importance of food safety knowledge through education and training for the general public (United States Dept. of Health and Human Services 2009), little is known about the effectiveness of training and testing programs on participants in general, and even less among employees with unique learning characteristics. An often over looked segment of this population are students with disabilities (SWD), particularly high school and transition students with special educational needs. For the purposes of this article, students receiving special education services are identified as those who have a disability based on either a state or federal definition and for whom appropriate diagnostic and evaluation teams have concluded that special education services are needed to make reasonable academic progress in school (Individuals with Disabilities Education Improvement Act 2004). These students are placed on Individual Education Plans (IEP) that guide their school activities and identify the services they need to access the general school curriculum. Transition students are those individuals receiving special education services who have IEP objectives beginning at age 14 that specifically address the skills and services they will need to be successful when they leave high school (Individuals with Disabilities Education Improvement Act 2004). The food safety skills used as the basis of the Food Safety Practices (FSP) Assessment Tool described in this article are examples of IEP-based transition objectives, specifically designed to meet industry-approved food safety requirements.

Students with Disabilities (SWD) in Food Service

National data collected by the federal government (Sanford and others 2011) point to the food service industry as a major player in the employment success of post secondary SWD. Among this group, 17% worked in food service, 11% worked as a skilled laborer, and 10% were employed as a cashier. Youth were significantly more likely to work in food service than in most other types of jobs (Sanford and others 2011).

Since food service is among the most viable employment options for SWD, handling foods safely is a critical pre-employment skill. Demonstration of knowledge as a preventive strategy for reducing foodborne illness risk is also fundamental to the FDA Food Code (United States Dept. of Health and Human Services 2009). Since 1997, the FDA Food Code has required demonstration of knowledge of foodborne disease along with food handling procedures determined to reduce the risk of food related illnesses.

Given the special learner characteristics of SWD, reliance on traditional paper and pencil evaluations might not provide useful information about certain skills (Elliott and Thurlow 2006). Real-life demonstrations of safe food handling practices can result in more beneficial assessment data of student skill development (Stiggins 1987).

A Food Safety Education for High School and Transition Special Needs Students curriculum served as the basis upon which our assessment tool was developed (Madaus and others 2010). The project was a collaborative effort by food safety and education professionals in Connecticut, Rhode Island and Massachusetts. This USDA-funded initiative emerged from teacher-generated data that identified critical gaps in food safety knowledge and skills among students with disabilities (SWD) receiving special education services. As an adjunct to this curriculum, the FSP Assessment Tool was developed to: 1) conduct reliable observations of students as they demonstrate food safety practices and 2) use this information to design classroom activities that are in keeping with students’ IEPs. This is the first known tool of its kind to test food safety skills of individuals with special needs in a real-world setting.

Performance Assessment and Food Safety Skills

Validity of the FSP Assessment Tool was based on observing students demonstrating what they know or what they can do. In contrast to assessments that limit how students respond by requiring them to select a response in a multiple choice or true or false fashion, performance assessments require students to construct their responses while they are active participants in the assessment process. Students are therefore responsible for constructing their responses and creating an answer or a product rather than simply filling in a blank, selecting a correct answer from a list, or deciding whether a statement is true or false. The focus of performance assessment is on measuring students’ learning as part of real kitchen-based activities representing cooking situations that demand attention to safe food handling practices. The appeal of performance assessment is that it is criterion-referenced. This approach illustrates how well students are performing on specific goals or standards, rather than how their performance compares to a norm group of students nationally or locally, and is therefore intended to show individual growth and inform instruction. Finally, using performance assessment with SWD when handling food refers to their specific skills and competencies in relation to a continuum of agreed upon standards of proficiency or excellence and requires professional rater judgment in its design and interpretation (Haney and Madaus 1989; Stiggins 1987).

Information gathered on the FSP Tool not only provides criterion performance indicators, or benchmarks, of what students know (and need to improve upon), but it can also provide useful insights into instructional design needs, materials development, and define opportunities for success. For example, hand washing involves several steps; therefore, a student may successfully demonstrate all or a portion of those steps. A benchmark for adequate performance is established. In this case, it would assume to be set at 100%.

Methods

Once the study was approved by the Univ. of Massachusetts (UMass) Institutional Review Board and a setting with high school SWD was selected to pilot test the tool, an observation study was designed in three phases. In phase I, an “Observation Template” was created to establish a framework against which all observed data were compared. Design of the template adhered to the philosophy and guidelines of what is commonly referred to as Universal Design for Learning (UDL) (Hehir 2009). UDL assumes all learners have unique cognitive learning styles and that a “one size fits all” approach ignores these differences and is incapable or inflexible of meeting the individual needs of many students, especially those with disabilities. The template was also designed in consultation with the Culinary Arts Manager around four existing lunch meals to reflect a variety of foods prepared by the students (see Figure 1). Organization centered around the four core practices for safe food handling established by Fight Bac!: 1) Clean, 2) Separate, 3) Cook, and 4) Chill (Partnership for Food Safety Education 2009). Using these four general categories, specific safe food handling skills were identified and, for each of the four meals selected for inclusion in the pilot test, the possible opportunities for these behaviors to be demonstrated were recorded. After consulting with the Culinary Arts Manager, we knew for example, if using a cutting board was part of the preparation process and how many times it should be used for each particular meal selected. In this way a realistic baseline was obtained, against which future observations could be compared. The draft template was sent to the project Advisory Committee for review and comment.

Figure 1.

–Observed lunch menu items

After the template was revised, the next step (Phase II) was to establish an observation system. Safe food handling behaviors were observed and recorded using both a time sampling and an interval recording system. Intervals are pre-established time segments during which each of the four food safety behaviors were likely to be demonstrated. We established a 5 to 10 minute time frame for the intervals once the template was designed and the possible opportunities for each behavior to occur were identified. A time sampling methodology was used to collect data about the degree to which students demonstrated the selected food safety skills (Primavera and others 1997). Because this approach involves intermittent observation, the observer is able to study a behavior without having to set an amount of time aside to observe continually. Thus, it is a practical way to estimate the overall occurrence of a behavior. Time sampling is similar to interval recording in that the observation time is divided into intervals (in this case 5 to 10 minutes); however, in time sampling, the behavior is recorded only if it occurs at the end of the time period when the observer is required to check if the behavior has been demonstrated or not. For instance, if proper washing of fruits and vegetables is being observed, it would be recorded as occurring/not occurring at the end of each interval. When the specified amount of time has expired, the observer(s) looks at the student and determines whether or not the behavior is occurring. Time sampling is appropriate for behaviors that are longer in duration (such as proper washing of fruits and vegetables). If the behaviors are of shorter duration (such as checking proper cooking temperature of a meat product), the time sampling method would be adjusted to an interval recording method where behaviors are designated as either “occurring” or “not occurring”during each interval. In either case, total occurrences of the specified behavior are summed for each period, divided by the number of recording intervals, and multiplied by 100 to establish a percentage. For example, if washing fruits and vegetables occurred four times during an eight segment recording interval, a value of 50% would be assigned to this behavior: (4 occurrences/8 segments) = 50%.

In Phase III, pilot “template” observation data were collected for each of the four meals selected during Phase I. A number of specific guiding principles and practices were adhered to throughout the data collection process (see Table 1) (adapted in part from Patton 2002). Three to six students were observed as a group preparing and/or cooking meals during each of the 60-minute meal preparation periods pre-established by the facility. Two authors (Carbone and Scarpati) independently observed the students. Check marks were made on the FSP Assessment Tool to indicate if a food safety skill had been demonstrated.

Table 1.  –Principles and practices informing data collection.
Principles Practices
  1. (Adapted in part from Patton 2002)

Observation procedures should not be a distraction from instruction.Clearly describe the observation procedure.
Observation should not take undue time away from instruction.Clearly define the behavior/s to be observed.
Data collection should provide a learning experience for students.Establish a scoring and recording system.
Results should lead to frequent changes and improvements in instructional process.Create an opportunity for observers to learn and practice the observation technique.
The process should not create undo anxiety for students.Establish a behavioral recording system that is best suited to monitor the behavior/s.
The process should encourage cooperation.Clearly define the schedule of when behavior/s will be observed and the observation intervals (periods).
The process should respect the privacy and dignity of students.Create an opportunity to observe and record students’ behavior.
 Conduct reliability testing.
 Graphically represent the data.

Reliability

Inter-rater or inter-coder reliability examines the extent to which different observers using the same instrument get equivalent results (Singleton and others 1993). Inter-rater reliability checks the stability of data collected with the observational forms and should be completed on at least three separate occasions during the initial data collection phase if possible (Singleton and others 1993). In this study, two independent raters (Carbone and Scarpati) used the FSP Assessment Tool at the same time, on the same students, based on the same observable behavior. Both raters were knowledgeable about proper use of the instrument and in the food safety behaviors and skills prior to implementation. Inter-rater reliability (that is, percent of agreement) was calculated as follows:

image

For example, if Rater #1 and Rater #2 agreed that hand washing was observed a total of 8 times over a 10-minute interval session but disagreed about its occurrence during two intervals, the overall reliability rating would be 80% {8/8 + 2 = 80%}. Data collection and systematic behavior observation procedures used in this study provided an objective basis for making reliable decisions and documenting student behaviors and performance.

Results and Discussion

A total of 15 students (9 male, 6 female) between the ages of 15–19 years old agreed to participate in the pilot testing of the FSP Assessment Tool. All participants had an Individual Education Plan (IEP) and attended a high school for students with learning, emotional, and behavioral disabilities. Their specific learner characteristics were aligned with state and federal definitions of these disabilities and included average or above intellectual ability, achievement deficits, selective attention difficulties, and emotional reactions to situations that might interfere with their learning and/or the learning of others. No other specific student data were available due to privacy issues and legal constraints regarding access to IEP information.

Four different lunch meals were observed over a nine-week period. Each observation session took approximately one hour. This observation study was exploratory in nature; data were collected over time in an iterative fashion, with each observation session informing the next. Therefore, changes were made to both the content and format of the FSP Assessment Tool throughout this pilot study. After the first observation (meal #1), for instance, we discovered the need to add a space for “other” food safety practices in the Clean and Cook categories. We also added a new category (Other) to reflect personal hygiene practices that were not reflected in the four Fight Bac! practices. There was also a need to add more space on the observation form to better accommodate note taking.

Two observers (Carbone and Scarpati) independently observed food safe skills during meal preparation opportunities. Inter-rater reliability rates were calculated for two meals (#1 and #3). For meal #1, inter-rater reliability rates were as follows: Clean-90% agreement; Chill -100% agreement; Separate-N/A; Cook-100% agreement; and Other-90% agreement. For meal #3, overall agreements were: Clean-95%; Chill-100%; Separate-100%; Cook-100%; and Other-90%. Reliabilities were relatively easy to establish given the clear and well defined safe food handling behaviors, the discrete nature of their occurrence, and the moderate to low rates at which they occurred. Therefore, reliability checks were only taken for two meals.

All behaviors were equally weighted and the benchmark for adequate performance was set at 100%. General observation data for each food safety practice demonstrated during the pilot study are shown in Table 2. These data suggest that cooking and chilling foods, and cleaning practices had the lowest overall performance rates. Specific food safety practices with the poorest performance rates were cleaning spills and work surfaces, and pre-washing vegetables/fruits (Clean), keeping foods that need to be refrigerated at the appropriate temperature and chilling hot foods properly (Chill). Proper tasting practices were also inconsistently demonstrated (Other). Overall, properly separating foods and utensils and personal hygiene practices were more consistently demonstrated. An example of more complete observation data from meal #1, with specific comments and suggested corrective actions, are provided in in the case study example Figure 2.

Table 2.  –Observation data for student-based food safety practices.
Food Safety Category and Observable Practices Meal #1a Meal #2 Meal #3a Meal #4 Overall Rates (%)b
  1. aMeals for which inter-rater reliability checks were performed.

  2. bOverall performance rates reflect data for each food safety practice across all meals observed.

  3. cExp = expected number of observed practices.

  4. dObs = actual number of observed practices.

  5. eTotal observed/total expected =%

  6. fNA = not applicable (not observed).

CleanExp cObsd%eExp cObsd%eExp cObsd%eExp cObsd%e 
 Wash hands properly4375661006583536081
 Clean prep surfaces22100221003267313370
 Pre-wash vegs/fruits4375425011100437569
 Wash boards/utensils11100111002210022100100
 Clean spillsNAfNANANANANA11100215067
 Other (rinsed lids)11100NANANANANANANANANA100
Practice Performance Rate (%)  83  85  85  6378
ChillExp cObsd%eExp cObsd%eExp cObsd%eExp cObsd%e 
 Keep refrigerated foods21502210011100215071
  at proper temp             
 Check freezer tempNANANANANANANANANANANANANA
 Chill hot foods properlyNANANA1110021501110075
Practice Performance Rate (%)  50  100  67  6773
SeparateExp cObsd%eExp cObsd%eExp cObsd%eExp cObsd%e 
 Separate cuttingNANANA111001110022100100
  boards             
 Separate utensils forNANANA111001110011100100
  meat/fish             
 Proper storage ofNANANA111001110011100100
  meat/fish             
Practice Performance Rate (%)     100  100  100100
CookExp cObsd%eExp cObsd%eExp cObsd%eExp cObsd%e 
 Use thermometer to check internal cookingNANANA100100NANANA0
  temp             
 Other (cook eggs until11100NANANANANANANANANA100
  yolks/whites are firm)             
Practice Performance Rate (%)  100  0  0   33
Other (personal hygiene)Exp cObsd%eExp cObsd%eExp cObsd%eExp cObsd%e 
 Wear proper hair3267658355100548084
  restraint             
 Wear gloves331006610054805510095
 Demonstrate properNANANANANANANANANANANANANA
  care of cuts             
 Demonstrate proper215011100NANANANANANA67
  tasting practices             
Practice Performance Rate (%)  75  92  90  9088
Figure 2.

–Meal #1 case study example

The assessment technique and instrument described here provides educators working with SWD with a useful way to collect information about student performance and effective instruction. This information can help reveal inconsistencies in instructional techniques that may not otherwise present themselves in traditional paper and pencil evaluations. In the context of current educational reform and its emphasis on outcomes, accurate measures of student performance are critical to making informed decisions about students, teachers, and schools (US Dept. of Education No Child Left Behind Act 2001). For SWD in particular, performance outcomes are inherent in their IEPs and reflect the effectiveness of the curriculum. Useful and accurate performance data generated from the Tool fits well into the outcomes sections of IEPs and it is an accurate way to monitor students’ progress while not overwhelming teachers and other practitioners with yet another “test” to give to students.

Focus of this study was on the development of the tool and its feasibility of use. A limited number of meals and students were observed; therefore, generalizability of the data should be viewed with caution. However, the findings do provide information around which instruction can be built. For example, review of the pilot data suggest that properly separating cutting boards and utensils is a food safety behavior that was consistently practiced, while the other behaviors may need to be targeted for additional instruction. The FSP Assessment Tool can reveal a wide range of observable food safety practices. Identification of problem areas can easily be made and corrective actions taken. The Tool can also be tailored to accommodate the needs of the particular facility.

In addition to the small sample size, a caveat we offer is the same for any assessment that it is a sample of student performance across a specified time frame and context, and that it is not expected to measure every aspect of a specific safe food activity. Context played an important role when we designed the observation form. We kept in mind that observers would likely operate in a variety of kitchens. Some will be large enough to allow for easy and fluid avenues for observers to move around while checking for safe food skills, while others might be small and restrict observer(s) to specific locations while collecting data. Our pilot test location was small, with equipment (for example, ovens,) and work spaces (for example, prep tables) not in close proximity to sinks and wash areas. As observers, we were restricted to one general location to prevent interference with the natural work flow. Despite these restraints we were able to collect sufficient data in a reliable manner.

Many aspects of recording the occurrences of a skill make it user friendly because it rarely interferes with classroom activities and a simple “yes” or “no” check may be all that is needed. Furthermore, occurrence data produces a count that can easily be transferred to a graph or other sheet and interpreted to monitor progress. A major challenge, however, to observing specific food safety skills within the general categories described here (that is, Clean, Chill, Separate, Cook) is that they are likely to be “low frequency behaviors.” In other words, it would be unreasonable to expect them to occur very often. For example, it is logical to assume that if washing a cutting board is not an expected skill for that observation period, then it would not be demonstrated. If it was demonstrated, it would likely only occur once, or possibly twice during that time. The FSP Assessment Tool was designed for observing skills as they are exhibited and the sample of individuals being observed was relatively small. To maximize the accuracy of the observation tool, we offer several guidelines to prepare and organize an observation session (see Table 3).

Table 3.  –Observation session organization.
Procedure Example
Identify food safe behaviorsClean, Chill, Separate, Cook
Define food safe behaviorsSpecify what each behavior will “look like” when observed (for example, What do you see when hands are washed, or foods are separated?) Behaviors should clearly observable, measurable and specific.
Define events related to the target behaviorsIdentify events that might influence the defined behaviors (e.g, time of day, kitchen organization, menu instructions, teacher behavior, number of students in kitchen, etc.).
Establish measurement strategiesDetermine how to measure and record food safe behaviors (e.g, as descriptive narrative, checklist or interval recording techniques). Establish inter-rater reliability.
Collect, analyze, graph dataCompile results based on measurement strategy. Prepare results to inform curriculum.

Universal Design and Assessment

As noted earlier, the FSP Assessment Tool was developed in adherence to the guiding principles of Universal Design for Learning (UDL) (Hehir 2009). Many traditional assessments, such as written classroom tests, are not designed in this fashion and therefore have to provide accommodations (for example, more time, special materials, etc.) in order to allow SWD fair and equitable access to the test information. In contrast, the FSP Assessment Tool is flexible and applicable to all students engaged in a safe food curriculum, and it uses frequent measures to determine progress and inform instruction. Teachers and classrooms will benefit from establishing high expectations while presenting meaningful and appropriate activities for all students. For more information about UDL, visit the Center for Applied Special Technology (CAST), Universal Design for Learning Research and Demonstration Center (www.cast.org).

Conclusion

Finally, this assessment approach, while specifically tailored to a safe food curriculum, can be readily adapted to many performance skills that are part of any well-rounded food management training curriculum. Our intention was to present a logical, valid and reliable way of determining the effectiveness of a safe food instruction program by observing SWD demonstrate specific skills in real time across real food handling situations. With more accurate assessment data and better informed instruction, SWD are more likely to gain access to valuable employment in food service and ultimately better access to a community and productive life.

Acknowledgements

This study was funded by a grant from the USDA Integrated Research, Education and Competitive Grants Program under agency award #2005–51110-03275. This study was assigned Contribution #5141 by the USDA at the Univ. of Rhode Island, Agricultural Experiment Station.

Ancillary