Evaluation of Food Protection and Defense Outreach Education Programs


  • John M. Shutske,

    1. Author Shutske is with the Univ. of Wisconsin-Madison College of Agricultural and Life Sciences and Cooperative Extension, 140 Agricultural Hall, 1450 Linden Drive, Madison, WI 53706, U.S.A. Author Pierquet is with the Univ. of Minnesota School of Public Health and National Center for Food Protection and Defense, 420 Delaware St. SE, Minneapolis, MN 55455, U.S.A. Author Michel is a former student at the Univ. of Minnesota in the Dept. of Chemical Engineering, 151 Amundson Hall, 421 Washington Ave., SE, Minneapolis, MN 55455, U.S.A. Authors Rasmussen and Olson are with the Centers for Public Health Education and Outreach, 2221 Univ. Avenue SE, Suite 350, Minneapolis, MN 55414, U.S.A. Direct inquiries to author John M. Shutske (e-mail: shutske@wisc.edu).
    Search for more papers by this author
  • Jennifer Pierquet,

    1. Author Shutske is with the Univ. of Wisconsin-Madison College of Agricultural and Life Sciences and Cooperative Extension, 140 Agricultural Hall, 1450 Linden Drive, Madison, WI 53706, U.S.A. Author Pierquet is with the Univ. of Minnesota School of Public Health and National Center for Food Protection and Defense, 420 Delaware St. SE, Minneapolis, MN 55455, U.S.A. Author Michel is a former student at the Univ. of Minnesota in the Dept. of Chemical Engineering, 151 Amundson Hall, 421 Washington Ave., SE, Minneapolis, MN 55455, U.S.A. Authors Rasmussen and Olson are with the Centers for Public Health Education and Outreach, 2221 Univ. Avenue SE, Suite 350, Minneapolis, MN 55414, U.S.A. Direct inquiries to author John M. Shutske (e-mail: shutske@wisc.edu).
    Search for more papers by this author
  • Laura Michel,

    1. Author Shutske is with the Univ. of Wisconsin-Madison College of Agricultural and Life Sciences and Cooperative Extension, 140 Agricultural Hall, 1450 Linden Drive, Madison, WI 53706, U.S.A. Author Pierquet is with the Univ. of Minnesota School of Public Health and National Center for Food Protection and Defense, 420 Delaware St. SE, Minneapolis, MN 55455, U.S.A. Author Michel is a former student at the Univ. of Minnesota in the Dept. of Chemical Engineering, 151 Amundson Hall, 421 Washington Ave., SE, Minneapolis, MN 55455, U.S.A. Authors Rasmussen and Olson are with the Centers for Public Health Education and Outreach, 2221 Univ. Avenue SE, Suite 350, Minneapolis, MN 55414, U.S.A. Direct inquiries to author John M. Shutske (e-mail: shutske@wisc.edu).
    Search for more papers by this author
  • Ruth Rasmussen,

    1. Author Shutske is with the Univ. of Wisconsin-Madison College of Agricultural and Life Sciences and Cooperative Extension, 140 Agricultural Hall, 1450 Linden Drive, Madison, WI 53706, U.S.A. Author Pierquet is with the Univ. of Minnesota School of Public Health and National Center for Food Protection and Defense, 420 Delaware St. SE, Minneapolis, MN 55455, U.S.A. Author Michel is a former student at the Univ. of Minnesota in the Dept. of Chemical Engineering, 151 Amundson Hall, 421 Washington Ave., SE, Minneapolis, MN 55455, U.S.A. Authors Rasmussen and Olson are with the Centers for Public Health Education and Outreach, 2221 Univ. Avenue SE, Suite 350, Minneapolis, MN 55414, U.S.A. Direct inquiries to author John M. Shutske (e-mail: shutske@wisc.edu).
    Search for more papers by this author
  • Debra Olson

    1. Author Shutske is with the Univ. of Wisconsin-Madison College of Agricultural and Life Sciences and Cooperative Extension, 140 Agricultural Hall, 1450 Linden Drive, Madison, WI 53706, U.S.A. Author Pierquet is with the Univ. of Minnesota School of Public Health and National Center for Food Protection and Defense, 420 Delaware St. SE, Minneapolis, MN 55455, U.S.A. Author Michel is a former student at the Univ. of Minnesota in the Dept. of Chemical Engineering, 151 Amundson Hall, 421 Washington Ave., SE, Minneapolis, MN 55455, U.S.A. Authors Rasmussen and Olson are with the Centers for Public Health Education and Outreach, 2221 Univ. Avenue SE, Suite 350, Minneapolis, MN 55414, U.S.A. Direct inquiries to author John M. Shutske (e-mail: shutske@wisc.edu).
    Search for more papers by this author


ABSTRACT:  This analysis documents the outcomes and impacts from a series of food protection and defense educational programs conducted over a 3-y period for private and public sector food system professionals. Several measures were used to determine the professions of participants; their improvements in skills and abilities that resulted from workshops; the audiences' most valued program content; practice changes resulting from educational program participation; abilities to recognize and change food system vulnerabilities; and changes in knowledge levels. Findings indicate that the knowledge level and interest within the target audience progressed over the 3-y period from basic awareness level through more complex and higher order skills and competencies such as being able to walk through a food production, processing, distribution, or retail facility and identify specific vulnerabilities and make specific risk control recommendations. Pre- and posttest scores from the 3rd y of educational activities indicate that baseline awareness levels of core content on food protection and defense is now high among those most likely to attend these types of events, and that participants need to be challenged with additional higher-level education to promote specific skills. This program series proved successful in building important relationships among food industry and regulatory partners. Analysis of the food security investigation (FSI) series will be useful in the development and delivery of new education and outreach efforts. It is hoped that the analysis and discussion will provide the motivation to further develop a set of competencies that can be used to ground educational efforts that focus on securing and defending our food system.


Food protection and defense is an issue of concern because of the threat of intentional contamination with chemical or biological agents (Elad 2005) and because of other possible interruptions, fear, and economic damage that could be caused by intentional threats and actions (Rasco and Bledsoe 2005). Throughout this paper, the term food protection and defense will be used, even though other terms are commonly used to describe actions to protect our food system from the farm to the consumer's dinner plate from intentionally caused harm. Table 1 contains additional information on some of these existing terms.

Table 1—.  Definitions for food defense, terrorism, and security concepts.
Food Defense (NCFPD, 2006)A collective term used by the FDA, USDA, DHA, and so on, to encompass activities associated with protecting the nation's food supply from deliberate or intentional acts of contamination or tampering. This term encompasses other similar verbiage (that is bioterrorism, counterterrorism, and so on).
Food Terrorism (WHO, 2002)“An act or threat of deliberate contamination of food for human consumption with chemical, biological, or radio-nuclear agents for the purpose of causing injury or death to civilian populations and/or disrupting social, economic, or political stability.”
Food Security (USDA, 2007)“Access by all people, at all times to sufficient food for an active and healthy life. Food security includes at a minimum: the ready availability of nutritionally adequate and safe foods, and an assured ability to acquire acceptable foods in socially acceptable ways.”

In 2003, faculty members of the Univ. of Minnesota Extension Service were 1st asked by community leaders in a large and rapidly growing metropolitan county north of Minneapolis, Minnesota, to develop a half-day educational workshop for approximately 30 people. The focus of this initial session was to create a basic awareness and to discuss the potential intentional terrorist threats to the food system. Workshop leaders also discussed the audience members' various roles in prevention, preparedness, and response to an intentional food-related event. The audience for this presentation was a local (county and city) public health committee that included people from both the public and private sector. It included representatives from groups such as retail food, healthcare, public health, law enforcement, and the Extension Service. The event generated significant audience interest and discussion, media attention, and follow-up requests for similar and expanded educational offerings in other parts of the state in the years that followed.

As a result of this successful event, a larger initiative was created in 2004 that focused on terrorism and related preparedness issues within all sectors of the food system from the farm to the table. This included a history of food-related terrorism events and special risks associated with imported food as a result in the changing trends in consumer demands and globally based patterns in the sourcing of food products (Heath 2006). Educational content also covered newly created food protection and defense regulatory requirements (Crutchley and others 2007). The 2004 programs were funded primarily with resources from the CDC-funded Univ. of Minnesota Center for Public Health Preparedness. This program was continued in 2005 and 2006, culminating in 2006 with a series of 3 unique, experiential educational events titled “FSI Minnesota” (Food Security Investigation). This title was chosen to market the program by playing off of a popular television series, though “food protection and defense” is a preferred term. From this point onward, the entire 3-y educational program series will be referred to as FSI.

Little published information is available that documents the outcomes or impacts from educational programs that focus on food protection and defense and preparing professionals to deal with the threats of food terrorism. In fact, a review of research literature databases in the food, agricultural, public health, and medical fields done by the authors that examined search terms that included food protection, food defense, or food security combined with “educational program evaluation” (and related terms) found only 1 reference by Levin and others (2005). In this article, the authors described an evaluation of a program designed to engage local leaders in efforts to protect the food supply.

This article summarizes the evaluation data from this 3-y (2004 to 2006) series of FSI educational programs. Data from the 3 y of program activities were summarized and analyzed in 2007 and early 2008.


Educational methods

Educational workshop activities were designed and conducted during the 3 y from 2004 through 2006. Planning input was provided each year by a range of stakeholders that included public health, state and federal regulatory personnel, private sector leaders in the food industry, and others with knowledge about agricultural and food systems. Input was gathered both through formal steering committees and informal discussions. Workshops were designed during the 3 y to meet 5 basic objectives:

  • 1Develop relationships among and define the unique roles and responsibilities of public health, food industry, regulatory, and education professionals in addressing food protection and defense vulnerabilities.
  • 2Identify specific areas (such as storage, transportation, and production) for specific food products and ingredients in the farm-to-table food system that are most vulnerable to intentional attack.
  • 3Identify personnel-related food protection and defense risk factors using case studies from historical events that have occurred in the United States and internationally.
  • 4Design and/or recommend specific control strategies to prevent, mitigate, or respond to food protection and defense vulnerabilities and potential events.
  • 5Describe potential occupational health and safety risks to workers that would stem from an intentional food system attack and explain other linkages between worker protection and food protection and defense.

A more detailed description of the curricula for each of the 3 y is provided in Table 2 along with the additional information provided in the sections that follow describing the general themes for each year.

Table 2—.  Brief description of curriculum for 3 y of educational programming.
YearCurriculum description/outline
Year 1Introduction to public-sector entities involved in food protection and defense; historical overview of food protection/defense related events; detailed discussion and examples of specific types of vulnerabilities and hazards including direct product contamination, contamination of input products, introduction of pathogens (including those that directly affect food animals and plants); small group food system vulnerability identification and risk assessment/prioritization based on local region and related food industry resources; detailed discussion of organizational capabilities and completion of an event “response” template.
Year 2Review of Year 1 content; public sector organizations representing food, agricultural, and animal/livestock industries present detailed information on specific capacities for preventing and responding to food protection/defense-related events; fundamentals of risk/crisis communication; tabletop exercise focused on details of response to a food-system emergency event examining issues of causation, response, logistical problems, and organizational coordination.
Year 3Field trips held over 3-wk period (3 different days over 3 different weeks); field trips preceded by brief (25 min) review and overview of basic food system vulnerabilities; field trip sites included dairy farm, milk processor/bottler, fruit and vegetable processor/fresh wholesaler, retail grocers (including large supermarket and small co-op), seafood wholesale/retail store, immigrant-owned ready-to-eat processor of tamales, and large-scale food warehousing and distributions owned by supermarket and international airport; field trip hosts conducted walkthroughs and worked with participants to identify and document vulnerabilities and recommended fixes.

Educational themes for year 1 The 1st year's series of half-day workshops attracted 150 participants from 10 different professional backgrounds. This 1st year's FSI programs coincided with a major restructuring of the state's Extension system, where the system changed from being county-based to regionally based. Workshops were held in partnership with newly opened Extension regional centers in Mankato, Marshall, Crookston, St. Cloud, and Rochester. The newly-assigned Extension Service Regional Directors helped market the program to local and regional stakeholders. Workshop sessions included overview lecture information about food system vulnerabilities. Guided discussions helped the audience identify and assess critical local and regional food system assets and risks.

Specific content was developed for Year 1 to highlight the following issues:

  • 1An understanding of the complexity of farm-to-table food systems including the increasing global influence on food products and choice.
  • 2Food products or ingredients as a potential target for chemical, biological, or radiological agent contamination.
  • 3The threat of possible intentional introduction of animal or plant disease agents as a means to disrupt the food system.
  • 4The use of agricultural input products and other assets (fuel, fertilizer, pesticide application equipment) for illicit purposes including as potential contaminants to raw ingredients and inputs (and also as a mechanism to engage those involve in food production at the farm level).

Lectures and facilitated discussions were conducted by university faculty and by staff who represented the Minnesota Departments of Agriculture and Public Safety. The state's Board of Animal Health also provided lecture content. In addition to lectures and discussions on vulnerability identification, participants worked in teams to complete an emergency contact information template containing information on critical 1st-action steps and contacts to respond to a food system event. The program activities included filling out the emergency response template and discussion of the “State Duty Officer Program,” a state-operated, single point-of-contact entity for both public and private sector entities to contact when state-level assistance is needed or when a state-level notification is required. These sessions also included a brief overview of incident command systems (ICS) protocols as a means to structure a response to a local, state, or national response to a food system emergency.

One important outcome of the Year 1 events was that they fostered new professional relationships between agencies and businesses that had not previously existed even though these individuals were often located in the same county or city. In some cases, these entities that had never previously become acquainted were even co-located in the same building. These relationships are viewed by many as being critical in efforts to protect consumers as well as the food industry from an intentional event (Applebaum 2004). These newly created relationships will be discussed in the results section.

Educational themes for year 2 In Year 2, objectives included the development of deeper and more complex vulnerability recognition and response system skills as compared to the Year 1 workshops. These were structured as full-day events, and drew 41 participants in 2 Minnesota sites (Marshall and West St. Paul). The planning team for the Year 2 series realized that audience numbers would decrease as the level of “intensity” and depth of content increased beyond the basic awareness level materials created for the 1st year's programs.

The events during the 2nd year averaged just over 20 people per site as compared to 30 the previous year. A 3rd workshop was done in Year 2 for approximately 30 participants in North Dakota, though information from that program will not be summarized in this article.

A notable difference between the first 2 years' events was that a tabletop exercise was developed in Year 2. This exercise was conducted in the 2nd half of each full-day session. The focus of the exercise and simulated event was 2-fold. It was designed to help participants understand complexities and interconnections of local, regional, state, and national, and global food systems and associated response systems. The exercise also provided a venue that allowed program participants to practice specific actions and skills.

The tabletop exercise focused on an intentional event that involved botulism toxin contamination of livestock feed (hay in a field contaminated by an aerial application unit); a similar threat at a milk processing plant; and, one at a retail facility (hotel restaurant). The goal was to create an understanding of the possible implications for the regional (and national) milk system at the farm, processor, retail, and consumer levels. This exercise was later systematically evaluated and peer-reviewed by the RAND Health group, a research division of the RAND Corp. This review's results as well as the specific criteria used by the reviewing agency can be found at: http://web3.rand.org/health/php/show_exercise.asp?eid=114. The 14 evaluation criteria that were used during RAND's evaluation are shown in Table 3. The evaluation results which correspond to these 14 points are found in Figure 1 for this exercise.

Table 3—.  RAND Public Health's Evaluation Criteria used in peer review of tabletop exercise developed for year 2. (Criteria number corresponds to Figure 1.)
Criteria nrCriteria descriptor (rated on Figure 1 as Excellent, Good, or Fair)
 1Goals of the exercise are clearly stated
 2Objectives of the exercise are clearly stated
 3Exercise objectives are appropriate given the goals of the exercise.
 4Exercise addresses each of its objectives.
 5Exercise objectives are measurable within the context of the exercise.
 6Scenario used in the exercise is appropriate given the goals and/or objectives of the exercise.
 7Exercise scenario is internally consistent.
 8Exercise scenario is a realistic depiction of the capabilities and resources likely to be available to a participating health jurisdiction.
 9Exercise documentation gives clear guidance as to who should participate in the exercise, and which other organizations or functions need to be simulated.
10Exercise is designed to engage all invited participants.
11Exercise guidance and materials are adequate to allow others to easily replicate the exercise.
12Exercise is designed to result in action items.
13Exercise is designed to solicit feedback from participants.
14Exercise, as designed, can be completed within the scheduled timeframe.
Figure 1—.

RAND Public Health Preparedness Evaluation results of FSI tabletop exercise (Source: http://web3.rand.org/health/php/show_exercise.asp?eid=114).

Educational themes for year 3 The workshop series was further refined and deepened in Year 3 with many of the same educational objectives, but the 3rd year's events incorporated new activities and learning experiences. Changes were made based on participant feedback and comments in Year 2 including an expressed interest to participate in field-based activities where participants could practice newly learned skills in a real-world setting, ideally moving beyond tabletop exercises.

The Year 3 workshops were held on 3 different days over 3 wk to allow for a structure that focused each session on a different sector of the food “system” from the farm level to final retail sale to consumers. The workshops filled to capacity with 25 participants on each of the 3 d (75 total). The workshops featured a brief (25 min) introductory overview lecture followed by 4 to 6-h food industry-based tours with each tour covering multiple business entities within 1) retail; 2) processing and production and; 3) distribution. Facilities included several retail food facilities (large, small, and specialty markets), a processing facility for frozen dinner entrée products, a milk bottling plant, a dairy farm, a large fruit and vegetable wholesaler and processor, and 2 major food distribution facilities (one associated with a major international airport and the other with a large supermarket chain).

As was the case with previous programs in 2004 and 2005, one key was to provide a venue to link professionals who represented multiple disciplines together and to build specific food system preparedness skills and competencies. As part of this program's evaluation, during site visits and facility walkthroughs, each participant was asked to identify and record (on a carbon-copy form) specific, observed food protection/defense security vulnerabilities. Then, they were asked to identify and record specific control strategies to help address and fix each vulnerability.

Data collection and analysis methods

Educational program evaluation information was collected at each of the FSI workshops during all 3 y (10 total events). These data were summarized at the end of each year to guide the subsequent year's workshops. Later, after all workshops were completed, the Univ. of Minnesota's Institutional Review Board (IRB) was sought, allowing the development team to simultaneously examine all 3 years' worth of data for publication. The IRB determined the analysis to be exempt based on minimal risk (no identifiers and because the data were connected only to an evaluation of an educational program). Specific goals of the aggregated summary and analysis of the 3 years' of workshop data included to determine:

  • 1Characteristics of professions/disciplines attending.
  • 2Improvements in specific skills and abilities resulting from workshops.
  • 3Content deemed by participants as most/least valuable.
  • 4Specific reported practice changes resulting from participation in educational events.
  • 5Types of food system vulnerabilities that participants identified and documented at field locations (as a measure of their ability to recognize a range of threats).
  • 6Knowledge changes based on pre-/posttest scores.

Professional demographic information was collected simply by asking participants to self-identify their primary profession by checking off a single profession from a predetermined categorical list. Participants were asked to indicate which parts of the content were the most valuable to them. They were also asked to indicate the likely professional practice changes that they would implement as a result of attending these events. This information was recorded by participants on postevent written surveys with questions measured using a 1 to 5 Likert scale (1 being the lowest; 5 being the highest).

Assessments of the participants' ability to identify and recommend control strategies for specific food system vulnerabilities were collected in Year 3 using the form created by the course leaders. These forms were printed in duplicate so that the participant and the course leaders could each keep a copy. The participatory, field-trip based workshops with food system site tours were purposely planned to develop professionals' specific skills in identifying food protection vulnerabilities. The analysis of the specific items that they identified was a primary indicator of the program's outcomes. Each individual was provided a sheet to capture their observations viewed to be vulnerabilities. They were then asked to recommend 1 or more specific control strategies to correct the corresponding vulnerability. In the analysis, each item identified received 1 point, and the specific items on the participant sheets were categorized by the authors into 1 of 4 categories. These 4 categories were:

  • 1Regulatory and emergency planning (for example, no written plan to respond to an observed food contamination event).
  • 2Product and ingredient distribution and handling (for example, no protocol to test input ingredients from external vendors).
  • 3Worker/staff and related personnel management issues (for example, lack of training on food protection for front line workers and supervisors).
  • 4Physical infrastructure and facilities (for example, lack of security devices such as locks or surveillance equipment on loading docks or near vulnerable process equipment).

In the analysis of identified vulnerability information, data were entered and analyzed using Microsoft Excel spreadsheet software and Minitab 14 for statistical analysis. For the vulnerability information recorded by course participants on the duplicate forms, each unique separate vulnerability identified by participants received 1 point. These were later coded into specific categories as part of the analysis.

Pre- and posttests of knowledge were conducted in Year 3 with the field-based classes. The test data reflects test answers that were “self-graded,” and it should be noted that the tests were kept simple, with the primary purpose to assess baseline levels and especially to foreshadow and create interest, prior to departing for the field trips. The posttests also served as a vehicle for wrap-up discussion at the end. More limitations on the pre-/posttest method will be discussed in the results section.


Characteristics of participants' professions/disciplines

Figure 2 shows that of the 226 total program participants over the entire 3-y FSI series with 10 events in total, public health professionals accounted for the highest percentage (39%) followed by professionals in the food processing industry (15%) and emergency management (in the public sector) with 12%.

Figure 2—.

Total attendance by profession for workshops.

A critical objective of these educational programs was established early in the planning process to “develop relationships among and define the unique roles and responsibilities of public health, food industry, regulatory, and education professionals in addressing food protection and defense vulnerabilities.” Clearly, such partnerships are critical in addressing any type of major public health vulnerability related to our food supply (Applebaum 2004). This outcome was clearly achieved in this series based both on the mix of audience members that the programs attracted, as well as the structured format for planning together, participating in the exercises, and traveling together to visit facilities and develop control recommendations.

One anecdotal piece of evidence showed that this is likely to have a longstanding impact, was that at one Year 1 regional site, there is a large food manufacturing and distribution company (that markets nationwide) in 1 of the cities where these programs were held. In that workshop, food safety and “security” staff from this company attended the program. The event was the first time that the corporate staff had ever met public sector personnel from both the city and county who are responsible for emergency management planning, public health, and preparedness. These participants together discussed and explored specific issues related to chemical storage facility vulnerabilities near 1 of the major manufacturing sites and began to discuss follow-up joint planning and sharing of resources and expertise. This example was repeated often throughout the Year 1 program sites as well as in the following 2 y of the program.

Improvements in skills/abilities

The next question was intended to evaluate the degree to which participants felt they could better perform specific tasks as a result of having attended the FSI workshops (many of these abilities were then “exercised” and actually measured in Year 2 and 3). These scores (ranked from 1 to 5) were based on the specific, stated competencies targeted at each year's set of workshops. Over the 3 y, the specific competencies listed below ranked the highest based on mean scores:

  • Year 1: Identify potential sources of threats that are both intentional and unintentional.
  • Year 2: Year 2: Identify specific business and community vulnerabilities to both intentional and unintentional hazards.
  • Year 3: Year 3: Describe elements of food system homeland security issues that involve workers and employee protection strategies

A trend emerged over the 3 y. Each year, the scope of food protection and defense content narrowed and intensified from general awareness level information in the 1st year where participants learned about principal potential food system threats, to later being able to describe specific vulnerabilities for businesses (Year 2) and then examination of the detailed roles of workers (as both potential threats and victims) and other critical assets in food-related events (Year 3). Note that in Year 3, other specific categories of vulnerabilities were covered, and these will be discussed in later sections.

Content deemed as most/least valuable

Participants also were asked to identify the most and least valuable parts of the workshops in written evaluations. These were written by participants as narrative and later categorized, and despite a large number or participants not responding to this question about “least valuable,” some themes appeared:

  • Year 1: Most valued: general workshop content.
  • Least valued: incident command system.
  • Year 2: Most valued: emergency contact information.
  • Least valued: incident command system.
  • Year 3: Most valued: site visits with hands-on learning experience.
  • Least valued: repetitiveness of key information at multiple tour sites.

A higher number of comments were made about the features of each session of most value to the participants as compared to comments about least value. Overall, participants agreed these workshops offered information that was relevant to their needs as professionals working on the issues of food protection and defense. There was consistency in the first 2 y with “incident command system” (ICS) overview being the least valued. This content had been included as a means to provide some conceptual framework for understanding how a large-scale catastrophic food system event might be handled and was included in all 3 years' programs. Interestingly, during the Year 3 field trip sessions, while traveling between sites in a large bus, discussions occurred about the importance of organizational structures like the ICS structure. This renewed interest that was absent in Years 1 and 2 was due, at least in part, to the participants' observations and experiences from the previous year's events that surrounded Hurricane Katrina.

Specific practice changes resulting from participation

Participants were asked in an open-ended question how the workshops would influence or change their activities performed within their practice in the workplace. Just over 50% of the total participants responded to this question. The relatively low response rate on this question most likely occurred because it was open-ended and required additional writing (as opposed to being a multiple choice question). Analysis of comments showed the most prevalent response to this question was a generally stated “increased awareness” of food protection and defense and risk assessment (awareness is not necessarily a practice change). However, in the 3rd year, the workshop participants increasingly indicated that their frequency of communication with other food protection and defense partners and stakeholders partners would increase based upon the knowledge gained from attending the site tours (and presumably from the time spent traveling together as a group and the professional networking and relationship building that occurred).

Food system vulnerabilities and corrective actions at field locations

Figure 3 illustrates the how the percentages of vulnerabilities that participants identified varied across the 3 food sectors based on the structure of the 3 distinct sets of tours (retail; production/processing; and distribution). Among the 75 participants in these 3 weeks' worth of tours, 89% of those who attended were able to identify at least 1 vulnerability related to the sites and operations they observed.

Figure 3—.

Percent of participants identifying specific categories of vulnerabilities for each course.

Consistently, across the 3 food sectors, items most often identified were related to security in the site's physical infrastructure and facilities with 100%, 42%, and 84% identifying physical site security issues in retail, distribution, and production/processing, respectively. This would be expected, given that these items are often very tangible and concrete, and can be readily observed through visual checks without a more detailed understanding or discussion of the issue with the manager-tour guide. Staffing-related vulnerabilities were identified with the least prevalence. Issues of staff and personnel are more complex to identify through visual observation and include items such as worker training; background checks of employees; and efforts to maintain strong relationships and good working conditions for employees. Even though staff and personnel vulnerabilities often cannot be identified easily through observation, the participants were encouraged in advance to politely ask about these issues, and likewise, tour-guide hosts within the facility were encouraged to discuss the plans and protocols they had in place.

Knowledge changes

Finally, the pre- and posttest scores from the Year 3 workshops were of limited initial value with respect to workshop evaluation. The real intent in designing the pre- and posttests had been to provide a “teaser” and introduction to content, as opposed to having significant value in measuring complex skill/awareness gains. Since not all participants in the Year 3 program had attended similar past programs from this series or others, the pretest was also used to make sure everyone had some baseline level of core knowledge before departing to the field trip sites.

Even though the value of the pre-/posttest scheme for measuring knowledge change is limited here, the data did indicate that many food system and regulatory professionals attending these workshops had already attained relatively high levels of basic awareness for food protection and defense as was indicated by high baseline (pretest) scores. Figure 4 shows only small differences in the pre- and posttest scores for the 2006 workshop. No statistically significant changes were observed based on paired sample t-tests of mean scores. Improvements in awareness levels measured by higher posttest scores were quite modest. In some respects, like many educational programs (of any type), people most likely to attend and actively participate are already engaged at a relatively high level.

Figure 4—.

Pre-/posttest score comparisons for basic knowledge.

It is also important to note that pre- and posttests were not designed to evaluate specific skills and competencies developed in the field trips while observing, asking questions, identifying vulnerabilities, and making specific suggestions for change is a skill not easily measured on a pre- and posttest of knowledge. The summary of identified vulnerabilities and participant-recommended specific corrective changes, in the previous section is obviously a better measure of program impact than this pre- and posttest measure.


Prior to this analysis, very few food protection and defense educational programs had been described in the research literature. This article was developed as a result of having conducted and evaluated a constellation of 3 year's worth closely related educational programs that were intended for professionals working on issues of food protection and defense from both the public and private sectors. These programs were not initially intended to be part of a “series,” and so the methods that were used both for educational content delivery and for program evaluation evolved each year as the development and delivery team gained experience in this relatively new field if inquiry. It is hoped that these experiences will inform future educators and researchers as food protection and defense programs continue to develop and as outcome and evaluation strategies become more refined.

The program series described attracted 226 food system professionals including both the public and private sector over a 3-y period. Bringing these diverse stakeholder groups together to learn about an issue was challenging, but food protection and defense seems to be an ideal body of content since it is of such vital importance to all citizens regardless of the motivation of the program participants. Motivations of participants in these sessions included economic (for food businesses wishing to protect customers and their markets) as well as regulatory/public health (particularly among regulatory officials charged with food protection and defense responsibilities).

Many different types of educational strategies were used in an attempt to engage adult learners from a range of disciplines. Ota and others (2006) state that, “by using combinations of adult learner techniques and strategies, Extension educators can create training experiences that will enhance the learning of participants.” Further, these researchers state that well-designed adult learning activities that incorporate techniques such as educational games and case studies (such as the table top exercises in Year 2) and guided problem-based learning (such as the field trips and assessment of vulnerabilities and corrective measures) makes it more likely that adult learners will retain “what they have learned and apply it in their work environment,” (Ota and others 2006).

Educational activities in this series were designed to: develop relationships; help participants learn to identify specific vulnerable nodes within the food system; identify worker-related food protection and defense risk factors; prepare participants to better understand how to design and/or recommend specific food-related control strategies; and to better understand health and safety risks to workers that would stem from an intentional food system attack. Workshop components included the tabletop exercise, the small-group problem solving activities to identify and assess regional risks/threats, and the food system field trips.

Educators who attempt to replicate these types of learning activities need to focus on real-world problems and situations, and realize that working with private industry and with regulatory partners (in activities such as designing the tabletop exercise) is time and resource intensive and require a high degree of trust to be successful.

Anecdotally, since the FSI field trips in 2006, it has become somewhat more difficult to get students and adult learners into food-related facilities. In particular, food processors have become much more hesitant to allow field trips such as the ones done in this series for reasons of homeland security. It would seem that educational efforts aimed at food protection and defense are therefore working (even if it makes facility access more difficult to educators).

Countless new relationships were catalyzed and developed as a result of this educational effort, though the exact changes and impacts that result from these types of efforts are difficult or impossible to fully quantify. Participants often expressed that the events were the “1st time” they'd ever had an opportunity to work across public–private sector boundaries, or even across boundaries within various regulatory agencies. Anecdotally, at a state-level, among the organizers of these events, new initiatives have been continued including additional educational programs; joint participation in statewide emergency planning activities, and; grants and funded outreach and applied research among partners first brought together through this study. Most importantly, this series of educational programs and the processes to create them built trust among those who were involved, an important component of any major effort to protect homeland security.

Since new relationships were an important impact of this program (though difficult to measure), it would be worthwhile for educators who develop future educational programs efforts in food protection and defense to create and test methods to better quantify and describe the specific impacts of these new relationships. This type of “relationship” outcome and impact evaluation is being done in current programs at the Univ. of Minnesota after the experiences described in this article. It has involved determining the participants' “intent” to pursue activities associated with their newly created relationship and then following up to learn what actually has actually occurred via surveys, interviews, and other methods. A more sophisticated approach could also include efforts to examine the networks and webs of personal and agency contacts, communication, and shared work that results from participating together in these programs.

The 3 years' worth of workshops also were successful in examining multiple issues and vulnerabilities within the food system. This was accomplished through general awareness-level workshops, tabletop exercises/simulations of potential food system emergency events, and bringing participants going into the field to study and learn specific skills related to vulnerability assessment and control. The heavy emphasis was also placed in the final year on front line workers as both potential threats/vulnerabilities as well as “assets” to improve food protection efforts. These programs were also unique in dealing with worker occupational exposures as an issue related to food protection and defense, as workers are the ones most likely to first identify an issue and then be exposed to harmful agents (such as biological or chemical agents).

The data from Year 3 indicate that it is relatively easy for participants to demonstrate an understanding of food system vulnerabilities that involve the physical characteristics of a system that they are observing. In the future, additional thought should be given to finding ways to measure whether or not food system professionals fully understand the importance of issues pertaining to personnel management, internal worker/management relationships, and other factors that have an impact of the conditions within a given facility.

For those considering future research in this area including both educators as well as those agencies that provide funding to support these types of efforts, a previous review of food protection and defense educational programs indicates that most programs available nationwide are still focused on basic awareness level generation such as what was done in this series in Year 1 (Campbell and others 2005). The review by Campbell and others (2005) also describes the fact that public sector funding agencies sometimes rely heavily on process and activity evaluation methods for educational programs that are based on how many people attend events and that these events actually occurred. This method for program evaluation creates a natural incentive to develop programs that generate the biggest audience possible, and the largest group still consists of professionals who want baseline knowledge as opposed to learning more intense and complex skills.

Funding agencies need to recognize that “body counts” of people who attend programs is probably not the best measure of impact, and in fact, a program having significant impact in skill building and behavioral change may actually have a much smaller audience base. This issue is being further examined at the Univ. of Minnesota through the Natl. Center for Food Protection and Defense, the Center for Public Health Preparedness, and other groups who are working to explore the higher level “competencies” that we need to develop in food protection and defense professionals such as those that were developed in Year 3.

The authors hope that this educational program evaluation and analysis is helpful in generating new educational program concepts. The evaluation of 3 y worth of distinctly different types of activities is complex, and the data are not necessarily comparable from year to year. As was documented over the 3 y, learning outcomes progressed from basic awareness level information and knowledge gains to more complex skills associated with identifying specific physical, personnel, and procedural and regulatory-based vulnerabilities, and then being able to make specific recommendations to close vulnerabilities and gaps. As new programs are developed, it is important that educators and program designers begin to identify specific, measurable skills and competencies and ways to evaluate changes resulting from actual changes in behaviors, processes, and working conditions. Measuring these changes is difficult since the changes needed often are relatively large in scale and may take long time periods to evolve, and the motivation to change may come from several cumulative motivations that might include regulation, educational events, and other methods that build awareness.


The FSI Minnesota educational program series focused on bringing together public and private sector food system professionals with a primary focus on food protection and defense. A total of 226 individuals learned about a range of issues, gaining baseline levels of awareness and then later learning specific protocols and techniques for identifying and correcting vulnerabilities. While we would expect to see a continued progression in the sophistication of educational and outreach efforts, funders and policy makers must be aware that traditional measures of program success including numbers who attend these events are not true indicators of progress. Likewise, as programs become more sophisticated, the size of the audience will diminish, though there are still significant numbers of people who need to learn these new higher level skills.

Hopefully, this analysis of the FSI series will be useful in the construction of new education and outreach efforts. In addition, it is hoped that the analysis and discussion will further reinforce the need and provide the motivation to carefully construct a set of competencies that can be used to ground educational efforts that have the focus of securing and defending our food system.


The educational programs described in this article were supported in part by grant/cooperative agreement nr U90CCU524264, D. Olson, PI, from CDC. The data analysis for this study was partially supported by the USDA-CSREES National Integrated Food Safety Initiative (NIFSI) Special Emphasis Grant on Food Safety and Defense. Grant No. 2005-51110-02332. The contents are solely the responsibility of the authors and do not necessarily reflect the official views of CDC.