Exploring the feasibility of using very short answer questions (VSAQs) in team‐based learning (TBL)

Team‐based learning (TBL) currently relies on single best answer questions (SBAQs) to provide immediate feedback. Very short answer questions (VSAQs) are a reliable and discriminatory alternative that encourage learners to use more authentic clinical reasoning strategies compared to SBAQs. However, the challenge of marking VSAQs has limited their integration into TBL; we therefore explored the feasibility of VSAQs within a TBL session.

understanding of their content knowledge, as well as encouraging team competition which is key for knowledge acquisition and retention. 2 Traditionally the mechanism by which feedback is provided has been with paper scratch cards, although digital versions have been developed. 3 SBAQs have, until now, more usually been used in TBL as their format allows straightforward provision of immediate feedback. 1 However, there is no theoretical reason why alternative question formats could not be used so long as it is also possible to generate immediate feedback.
Very short answer questions (VSAQs) are a novel question format that encourage learners to use more authentic clinical reasoning strategies when compared to SBAQs. 5 Like SBAQs, VSAQs have a clinical vignette followed by a lead-in question. However, instead of having an answer list, the candidate must provide their own free-text answer of between one and five words. The candidate's answers are marked against the assessor's pre-approved answers. 6 Any answers that do not match the pre-approved options can then be reviewed retrospectively to consider if they should be added to future lists of approved answers. VSAQs have been demonstrated to be an acceptable, reliable and discriminatory assessment tool which avoid some of the pitfalls of SBAQs, such as cueing. 5,6 A key aspect of TBL is that the questions used in the RATs should reflect those used in summative assessments. 1 There is therefore a need to ensure that VSAQs can be used in TBL to reflect their increasing use in undergraduate assessments.
The team discussion has been proposed as a crucial part of how TBL offers advantages over alternative learning strategies. 7 'Peer elaboration' during the discussion provides a means for students to help each other understand difficult concepts and gain a new perspective of the knowledge processed during previous TBL components. 7 Group discussion has been shown to be a strong activator of prior knowledge and helps to establish student's understanding of a topic. 7 The use of VSAQs in TBL may therefore enrich group discussions when there are multiple plausible answers in a clinical scenario that need to be debated, thereby enhancing the learning potential of the session.
The requirement to provide immediate feedback during the tRAT has thus far been a barrier to using VSAQs, as the free-text nature of the answers has previously made automated marking challenging. With the advent of digital assessment management systems, VSAQs can be reliably and acceptably marked electronically. 6 However, the automated marking of VSAQs has not yet been utilised in the context of TBL. The aim of the study was to explore the feasibility of real-time marking of VSAQs and their incorporation into TBL.

| ME THODS
We designed our TBL session using established guidelines described by Parmelee et al. 1 A session developing an understanding of the differential diagnosis of headache was created using an online TBL platform (Learning Activity Management System, LAMS). We developed a function within LAMS to allow immediate marking of the VSAQs during the TBL session by the instructor. Students' answers to the iRAT appeared in an 'answer queue' visible only to the instructor as they were submitted. The instructor was able to categorise the students' answers as either 'correct' or 'incorrect' by dragging and dropping them in to the appropriate columns ( Figure 1). These answers are saved and automatically recognised by the software, so that during the tRAT the students receive immediate feedback. Ethical approval for the study was granted by the Imperial

College London Medical Education Ethics Committee (MEEC). In
December 2019, all third-year medical students at Imperial College London (n = 324) were invited to take part in this study.
As part of the iRAT and tRAT, students completed five VSAQs and five SBAQs. The five VSAQs were marked by the session instructor as described; the five SBAQs were automatically machine marked. The tAPP exercises were unchanged from the standard format. Immediately following the session, students were asked to complete a questionnaire about their experiences of using VSAQs in TBL.
We analysed the results from the post-session questionnaire, the mean time taken to mark the VSAQs and range in number of attempts made per question type.

| RE SULTS
In total, 209 third-year medical students at Imperial College London participated in the study. Using the newly developed technology, instructors were able to mark all VSAQs in real time during the iRAT, allowing immediate feedback to be provided during the tRAT. Two members of faculty facilitated three sessions each. The mean time taken to mark the five VSAQs was 422 seconds (SD 73 seconds). It was feasible for one instructor to mark the five VSAQs while coordinating the session.
For VSAQs, the number of attempts made by students before the correct answer was identified ranged from 1 to 38 compared to 1 to 4 for SBAQs ( Figure 2). VSAQs have a potentially unlimited number of attempts as students can choose from the options that their team members provided in the iRAT or enter new answers generated from their team discussion. We allowed the teams to input as many answers as they wished until they either entered the correct answer or chose to move on to the next question. Students were reminded at the beginning of the TBL session that VSAQ answers are always between one and five words in length, and to consider this when making their attempts.
Of the 209 students who took part in the session, 141 (67.5%) responded to the post-session questionnaire. Of the post-session questionnaire respondents, 85.1% (n = 120) agreed that VSAQs were a better representation of how they would be expected to answer questions in clinical practice. 71.6% (n = 101) agreed that using VSAQs in TBL sessions helped to emphasise group discussions (Figure 3).

| DISCUSS ION
Our study is a pilot in one institution, but it demonstrates the feasibility of incorporating VSAQs in TBL using innovative technology. In addition, the post-session questionnaire data suggest that students responded positively to the introduction of VSAQs into TBL. This represents an important first step in the process of evaluating the merits of VSAQs in TBL.

This represents an important first step in the process of evaluating the merits of VSAQs in TBL.
The post-session questionnaire responses indicate that students felt the use of VSAQs in TBL to be more representative of clinical practice. This is in keeping with previous work on VSAQs, where students have repeatedly reported that they are a more authentic assessment method than SBAQs. 5,6,8 Given that a benefit of TBL is its ability to incorporate the aspects of clinical problem F I G U R E 2 Number of attempts made by students before the correct answer was identified across all questions in the tRAT by question format. VSAQs: very short answer questions. SBAQs: single best answer questions solving, 1 VSAQs may present a way of further increasing this advantage of TBL.

Students felt the use of VSAQs in TBL to be more representative of clinical practice.
Students also appear to have a wider range of attempts to reach the correct answer in VSAQs compared to SBAQs. We hypothesise that this may lead to an enrichment in their team discussions during TBL, as the focus of discussion is not limited to the five options available. Students perceive a valuable part of TBL to be the inter-learner discussions, allowing them to hear peer answer explanations and aiding recognition of mistakes in their own understanding. 9 These team discussions may form part of the psychological mechanisms that lead to the reconsolidation of knowledge that occurs during TBL. 7 The unlimited scope of potential answers that students can produce with VSAQs may have potential to enrich these discussions, and this hypothesis is supported by the students' responses in the post-session questionnaire. However, we acknowledge that a limitation of our study is that we were not able to evaluate the quality of each team discussion to reach a consensus for a VSAQ answer versus a SBAQ answer. Future investigations should study the quality of the discussions through a content analysis. It would also be valuable to evaluate student engagement when using VSAQs in TBL compared to SBAQs, using a validated classroom observation tool, as has been done in traditional TBL. 10 Future studies evaluating the range of incorrect answers submitted by students could also provide further insight into the breadth of the team discussions.
The unlimited scope of potential answers that students can produce with VSAQs may have potential to enrich these discussions.
Although we have reported that students agreed VSAQs were of value, we cannot compare this with student experiences in TBL sessions where VSAQs were not used. A key area of further study would be to compare student perceptions of TBL sessions with and without VSAQs.
The students at our institution had prior experience of TBL, alongside a wide range of other teaching modalities. Although 89.4% agreed or strongly agreed that they were experienced with the format of TBL, it would be beneficial to introduce VSAQs into an undergraduate course that is focused around TBL. 3 This would allow us to see how 'TBL expert students' respond to the change in question format, and if they perceive the same advantages across multiple TBL sessions.

| CON CLUS ION
We present an innovative solution for integration of VSAQs into TBL sessions, allowing students to receive immediate feedback and benefit from a more authentic question format that may enrich their group discussions. Although we demonstrate the application of VSAQs in TBL in a medical undergraduate setting, the technology and strategies used are generalisable to any setting where TBL is already used. We present an innovative solution for integration of VSAQs into TBL sessions.

FU N D I N G SO U RCE S
The software implementation for this project was made possible by Nanyang Technological University's EdeX learning and teaching project grant.

PRE VIOUS PRE S ENTATIONS
The results of this study were presented as a pre-recorded oral presentation at AMEE 2020: The Virtual Conference, September 2020.  and have agreed to be accountable for all aspects of the work.

ACK N OWLED G EM ENTS
The authors thank the LAMS Foundation for releasing the VSAQ feature for TBL as an open source software. They acknowledge Ernie Ghiglione from LAMS Education Asia Pte Ltd for his support on the design, implementation and delivery of this project.

CO N FLI C T O F I NTE R E S T
The authors report no conflict of interest.

E TH I C A L A PPROVA L
Ethical approval for the study was sought from and granted by