Development of a validated exam to assess physician transfusion medicine knowledge

Authors


Abstract

Background

There is evidence that physicians lack adequate transfusion medicine knowledge. To design needs-based educational interventions to address this gap, a validated assessment tool is required. Previously published exams have not been created or validated using rigorous psychometric methods.

Study Design and Methods

A modified Delphi method was used to achieve consensus regarding the essential knowledge and skills for physicians who transfuse blood products. To ensure content validity, members of an international organization of transfusion medicine experts (Biomedical Excellence for Safer Transfusion [BEST] Collaborative) participated in the exam design process. An exam, based on the most highly rated topics, was created and administered to individuals with a priori expected basic, intermediate, and expert levels of transfusion medicine knowledge. Rasch analysis, a psychometric technique used in high-stakes medical licensure and board testing, was used to determine exam accuracy and precision.

Results

Thirty-six topics achieved ratings sufficient to be considered for inclusion in the exam (content validity index > 0.8). A 23-question exam was administered to 49 individuals. Mean scores for individuals with expected basic, intermediate, and expert knowledge were 42, 62, and 82%, respectively (p < 0.0001). The exam achieved good fit with the Rasch model.

Conclusion

A validated exam has now been created to accurately assess transfusion medicine knowledge. This exam can be used to determine knowledge deficits and assist in the design of curricula to improve blood product utilization.

Abbreviations
ASCP

American Society of Clinical Pathology

CVI

content validity index

PGY

Postgraduate Year

TACO

transfusion-associated circulatory overload

A number of studies have shown a high rate of inappropriate blood product use.[1-3] These findings are likely in part due to inadequate knowledge of transfusion medicine.[4] While medical schools have transfusion medicine curricula, the amount of time spent teaching this material can vary greatly.[5, 6] In one study from the United States, while 83% of the 86 surveyed administrators reported that their school had transfusion medicine lectures, almost half of this teaching amounted to only 1 or 2 hours.[5] Only 29% reported small group sessions related to transfusion medicine. In addition, there are no published data that have determined the efficacy of these different curricula.

A validated transfusion medicine exam would enable a better assessment of educational methods, trainee knowledge, and areas for improvement. While transfusion medicine exams have been published, they were not designed or validated using accepted psychometric techniques.[7-10] As noted in an editorial on the development of one of the more recent of these exams, “several critical points about the experimental design and conduct of the study raise questions about the quality of the results and/or data and whether all conclusions and recommendations are truly warranted.”[11] In this study, we used rigorous analytical methods to develop a validated transfusion medicine exam.

Materials and Methods

Identifying topics for the exam

Exam content was determined using a modified Delphi method that is a structured approach for achieving consensus among experts.[12] The experts were members of the Biomedical Excellence for Safer Transfusion (BEST) Collaborative, an international research organization that works to explore ways to improve transfusion practice (http://www.bestcollaborative.org). The group, representing a diversity of interests and high levels of accomplishment, has scientific and blood center members from 10 countries. While there is some turnover, during the survey period described below, approximately 80% of the membership remained constant.

BEST members were presented with an initial survey that asked the following open question:

“What knowledge or skills related to transfusion medicine are absolutely essential for physicians who are not transfusion medicine specialists (e.g., internists, cardiologists, surgeons, neurologists) but whose practice includes the transfusion of blood products? (List as many as 10 items).”

To provide specific responses the following additional instructions were given:

“Please be as specific as possible and avoid broad subject headings. Examples: Too non-specific (avoid if possible): ‘Adverse reactions’; ‘Immunohematology’; ‘Indications for red cell transfusion’; Specific (preferred): ‘Post-transfusion purpura’; ‘Testing for a biphasic hemolysin’; ‘Red cell transfusion threshold for (a specific indication).’ ”

After removal of redundant and nonspecific responses, this list of topics was then collated for a second survey in which BEST members were asked to rate the randomly presented topics on a scale of 1 (very low importance) to 6 (very high importance/essential) for inclusion on an exam to be administered to all physicians who transfuse blood products. As the results from this survey would be used to determine content for a physician exam, only results from BEST physician members were used to calculate a content validity index (CVI) for each rated item. The CVI is equal to number of respondents rating a topic a 4, 5, or 6 divided by the total number of respondents. A CVI of greater than 0.8 was considered appropriate for inclusion on the exam.[13] All surveys were administered through SurveyMonkey (surveymonkey.com) and were anonymous.

Developing exam questions

The exam questions were developed in conjunction with the American Society of Clinical Pathology (ASCP). The ASCP oversees both the Pathology Resident In-Service Exam (RISE) and certification exams for laboratory technologists.[14] As such, the organization has decades of experience in exam design and psychometrics. Following guidelines for the Pathology RISE, BEST members submitted questions for inclusion in the exam. To focus on the most pertinent issues and limit exam length, questions were primarily based on topics with a CVI of greater than 0.9 although topics with a CVI of greater than 0.8 were also included. All submitted questions required a reference supporting the correct answer. These questions were vetted by the authors and refined to create a 23-question pilot exam. Question selection was based on quality and to ensure diversity in the topics covered.

Piloting the exam

The exam was piloted on individuals from three different hospitals (one in Canada and two in the United States). The following individuals were included and divided into three a priori categories:

  1. Expected basic knowledge: Postgraduate Year-1 (PGY-1) internal medicine and pathology residents.
  2. Expected intermediate knowledge: non–transfusion medicine specialists with an interest in transfusion medicine (e.g., members of hospital transfusion committees, hematologists, anesthesiologists, critical care physicians); PGY-2 and above clinical pathology or anatomic and clinical pathology residents or fellows.
  3. Expected expert knowledge: transfusion medicine physicians.

The exam was administered through SurveyMonkey. Although several demographic questions were asked, no identifying information was requested. As such, the examinees remained anonymous.

Statistical analysis of pilot data

Classical test theory involves direct comparison of scores (i.e., percent correct) among examinees as an indication of accuracy (i.e., experts should have higher mean scores).[15] For this purpose, we determined mean scores for each expected expertise group and analyzed the data using a one-way analysis of variance (ANOVA; Microsoft Excel, Seattle, WA).

To provide a more detailed assessment of accuracy and reliability, Rasch psychometric analysis was performed.[15-17] This approach provides greater information than classical test theory in regard to specific item difficulty and student ability. Rasch analysis is routinely used in high-stakes testing such as for medical certification and licensure.[17]

The Rasch model assumes a logarithmic relationship between student ability and item difficulty:

display math

Using the above formula, for a question with a set difficulty, one can predict how often individuals with a particular ability should select the correct answer. For example, an individual with the highest ability would be expected to get the answer correct almost 100% of the time while an individual with average ability would be expected to get the question right only 50% of the time. Rasch analysis determines how well, from exam administration data, each question and each individual fit the model.

Fit can be determined using chi-square analysis for the comparisons of the acquired data to the Rasch model. This value is related to the measured variance divided by the predicted variance adjusted for the number of observations. As such, a question fit value of 1.3 indicates there is 30% more variance than predicted by the Rasch model.[16] While recommendations vary for acceptable question fit scores, there is general agreement that questions with a fit value of more than 1.5 have more variance than expected and should be reevaluated.[18] While not necessarily indicating an item needs to be removed, the basis for the misfit should be determined during the question reevaluation process.

Fit values for examinees can also be determined. In addition, an overall examinee reliability measure can be calculated and interpreted in a similar fashion to Cronbach's alpha.[19] The Rasch model's value for reliability tends to be an underestimate when compared to Cronbach's alpha.[20]

Results

Determining topics for exam inclusion

Of the 71 BEST members e-mailed, 34 responded (48%) to the first open-question survey. Characteristics of the respondents are shown in Table 1. There were 289 potential topics listed for an average of approximately nine per respondent. After removing redundant and nonspecific responses from the first survey, the second rating survey consisted of 78 topics. Of 76 BEST members e-mailed, 36 responded and of these 27 were physicians (36%). Characteristics of the survey respondents are shown in Table 1. As the topics for rating were those already indicated as important by BEST members, CVI scores showed a distribution to the right (median, 0.77). However, there was evidence of topic discrimination with only 36 of the initial 74 topics achieving a CVI of more than 0.8 and 16 achieving a CVI of more than 0.9 (Table 2). In addition, although the topics were presented randomly, related topics were ranked highly (e.g., management of dyspnea, transfusion-associated circulatory overload [TACO], and transfusion-related acute lung injury [TRALI]). The topics were presented and deemed acceptable by the BEST membership at the October 2010 BEST meeting.

Table 1. Demographics of survey respondents
CharacteristicOpen-question surveyRating survey
Respondents3427
Response rate (%)4836
Countries represented1010
Degrees (%)  
MD5563
MD/PhD3027
PhD150
Specialty (%)  
Transfusion medicine6678
Hematology2215
Other127
Work setting (%)  
Blood center2833
Hospital blood bank4130
Hospital transfusion service2426
Other711
Experience in years (mean [range])24 [0-40]24 [7-40]
Table 2. Topics with CVI of more than 0.8 and more than 0.9*
  1. PT = prothrombin time; PTT = partial thromboplastin time.
Transfusion thresholds
*RBC transfusion thresholds in acute anemia
*RBC transfusion thresholds for nonbleeding, hospitalized patients without cardiac conditions
RBC transfusion thresholds in hospitalized patients with cardiac conditions
*PLT transfusion thresholds for invasive or surgical procedures
*PLT transfusion thresholds for bleeding patients
PLT transfusion thresholds for prophylaxis (nonbleeding patient)
*Plasma transfusion thresholds for invasive or surgical procedures
Plasma transfusion thresholds for prophylaxis (nonbleeding patients)
Indications for cryoprecipitate
Understanding that there is no target Hb level at which an RBC transfusion is indicated in all patients
*Understanding the relationship between Hb level, tissue oxygenation, and role of RBC transfusion
*Principles of bedside assessment of a patient's need for transfusion
Transfusion reactions
*Diagnosis and management of TRALI
*Diagnosis and management of TACO (fluid overload)
*Management of dyspnea during and/or after transfusion
*Understanding processes for reporting transfusion reactions
Diagnosis and management of acute hemolytic transfusion reactions
Diagnosis and management of transfusion-related anaphylaxis
Diagnosis and management of allergic transfusion reactions (nonsimple urticarial, nonanaphylaxis)
Management of fever during and/or after transfusion
Understanding how to obtain informed consent for transfusion
Understanding transfusion-transmitted infectious disease risk
Safe administration of transfusions
*Understanding the importance of correct recipient identification
*Understanding the importance of proper collection of blood samples for transfusion
Knowing correct procedures for ordering or requesting blood products
Knowing how to appropriately transfuse blood products (e.g., rate, fluid compatibility)
Nonserologic testing
*Understanding relationship of PLT counts to bleeding risk
Knowing the dose response to RBC transfusion (1 unit = 1 g/dL Hb increase)
Understanding causes or interpretation of prolonged PT/PTT
Blood bank testing
Understanding risks of un-cross-matched blood
Principles of ABO/Rh blood group compatibility for RBCs
Other
*Principles of warfarin reversal (e.g., blood products, vitamin K)
*Management of patients requiring massive transfusion
Indications for irradiated blood
Knowing how and when to contact a transfusion medicine specialist
Indications for using iron as opposed to blood transfusion

Exam development, piloting, and validation

An exam consisting of 23 questions was created with input from BEST Collaborative members and the ASCP. The topic for each question and the answer reference are shown in Table 3. All of the questions were based on topics with at least a CVI of more than 0.8 and 15 of the 23 questions (65%) were based on topics with a CVI of more than 0.9. Twelve of the 16 topics with a CVI of more than 0.9 were included in the exam.

Table 3. Exam question topics and references
QuestionTopic(s)Reference
  1. * CVI of more than 0.9.
 1RBC transfusion thresholds in acute anemia*Napolitano et al.[21]
 2Management of dyspnea, diagnosis and management of TACO*Mazzei et al.[22]
 3Indications for irradiated bloodAlyea and Anderson[23]
 4PLT transfusion thresholds for invasive or surgical procedures*Dzik[24]
 5RBC transfusion thresholds in hospitalized patients with cardiac conditionsCarson et al.[25]
 6Management of dyspnea, diagnosis and management of TRALI*Kopko and Popovsky[26]
 7PLT transfusion thresholds for invasive or surgical procedures*Nester and AuBuchon[27]
 8Diagnosis and management of acute hemolytic transfusion reactionsDavenport[28]
 9RBC transfusion thresholds for nonbleeding, hospitalized patients without cardiac conditions*Menitove[29]
10Diagnosis and management of allergic transfusion reactions (nonsimple urticarial, nonanaphylaxis)Vamvakas[30]
11Plasma transfusion thresholds for invasive or surgical procedures*Dzik[24]
12Understanding processes for reporting transfusion reactions*Heddle and Webert[31]
13Understanding transfusion-transmitted infectious disease riskKlein and Anstee[32]
14Plasma transfusion thresholds for invasive/surgical procedures*Dzik[24]
15Understanding processes for reporting transfusion reactions*Kopko and Popovsky[26]
16RBC transfusion thresholds in acute anemia*Napolitano[21]
17Understanding transfusion-transmitted infectious disease riskRamirez-Arcos and Goldman[33]
18Management of patients requiring massive transfusion*Nester and AuBuchon[27]
19PLT transfusion thresholds for prophylaxisKlein and Anstee[34]
20Principles of warfarin reversal*Keeling et al.[35]
21Understanding the importance of correct recipient identification and understanding the importance of proper collection of blood samples for transfusion*Davenport[28]
22Management of dyspnea, diagnosis and management of TRALI*Kopko and Popovsky[26]
23Knowing the dose response to RBC transfusion (1 unit = 1 g/dL Hb increase)Circular of Information[36]

To pilot the exam, 101 individuals were sent an e-mail invitation. Forty-nine individuals completed the exam for an overall response rate of 49%. Demographics of the respondents are shown in Table 4. The group of individuals with expected intermediate knowledge was made up of 19 pathology residents at an average training level of PGY-3 (range, 2-5) and nine non–transfusion medicine physicians (five hematology-oncology, two medicine, one anesthesiology, one critical care). The mean scores and ranges of the three a priori defined basic, intermediate, and expert knowledge categories were 42 (30-48), 62 (30-87), and 82% (61%-96%). Demonstrating accuracy, a one-way ANOVA demonstrated a significant difference in exam scores among the three groups (p < 0.0001).

Table 4. Demographics of pilot exam respondents*
  1. * Data are reported as number/response rate %, number (%), or percent.
Respondents49/49
A priori expected knowledge level 
Basic9 (18)
Intermediate28 (57)
Expert12 (24)
Self-rated knowledge level 
Beginner16 (33)
Intermediate21 (43)
Advanced8 (16)
Expert4 (8)
Country 
Canada31
United States69

In regard to Rasch analysis, the mean examinee fit and item fit were 0.99 and 1.00, respectively. The exam reliability was 0.79. One question demonstrated a fit score above 1.5. This question (Question 20) involved the appropriate treatment for a patient with a high international normalized ratio on warfarin. Only 42% of transfusion medicine experts chose the correct response compared to 56% of PGY-1 residents. Not all transfusion medicine specialists may be familiar with nontransfusion management of warfarin as they may not routinely order medications for patients. After input from BEST membership the item was retained, as it was agreed that such knowledge is very important for patient management related to transfusion medicine.

Rasch analysis also allows determination of the effect of question removal on exam quality. Based on question length, topic and fit values, we selected three questions (Questions 5, 11, and 23) for which we believed removal would shorten time to complete the exam without affecting quality. Upon reanalysis of this 20-item test, a one-way ANOVA still demonstrated a p value of less than 0.0001. The average item and examinee fit values were 1.0 and 0.99. The reliability was 0.80.

Discussion

The first step in curricular design is a needs assessment,[37] that is, determining the extent of an educational problem and what specific areas may need improvement. Although there is evidence of limited and ineffective physician training in transfusion medicine, there are no available validated tools to accurately perform a needs assessment. In this study, we have created a rigorously validated exam to gauge transfusion medicine knowledge.

While transfusion medicine exams have been published, they have generally been created by small groups of individuals without use of an accepted approach for reaching expert consensus.[7-10] In our study, to ensure content validity, we used a modified Delphi method with a large international group of transfusion medicine experts, with an average of more than 20 years of practice experience, to determine exam topics. The anonymity associated with this method allows for the best possible input as there is no concern for stronger personalities driving content.[12] Previous exams have also had minimal assessment of accuracy and precision. We used Rasch analysis, a tool used in validation of high-stakes medical testing, to demonstrate the exam has adequate performance to distinguish different transfusion medicine knowledge levels.[15-17] Rasch analysis also enabled question removal without affecting exam quality.

One possible limitation of our study is that our content experts were primarily made up of transfusion medicine specialists and not the end-users (i.e., the physicians transfusing the products). As the transfusion medicine experts that determined content came from a variety of practices and countries, we believe that their expert opinions represent a broad enough perspective to ensure appropriate topics for inclusion. As inclusion of topics was based on relevance for all physicians who transfuse blood products, at the very least, the core topic list and exam can be used as a framework to build on in developing specialty-specific curricula and assessment tools.

In summary, we have created, to our knowledge, the first extensively validated transfusion medicine exam. We believe that the topic list will be helpful in curriculum design and the exam will be useful for individuals trying to assess transfusion medicine knowledge in their work or educational setting. The exam is available by contacting the corresponding author.

Acknowledgment

The authors thank Mark Fung, MD, PhD, for assistance in recruiting individuals to take the pilot exam.

Conflict of Interest

The authors report no conflicts of interest or funding sources.

Ancillary