Understanding dental students’ use of feedback

Introduction: Feedback can enhance learning and

S e e h t t p://o r c a .cf. a c. u k/ p olici e s. h t ml fo r u s a g e p olici e s.Co py ri g h t a n d m o r al ri g h t s fo r p u blic a tio n s m a d e a v ail a bl e in ORCA a r e r e t ai n e d by t h e c o py ri g h t h ol d e r s .

| INTRODUCTION
The educational literature surrounding feedback suggests that it is highly valued by students for its ability to enhance learning.1,2 Hattie & Jaeger3 suggest that feedback is an important aspect of learning and Ramsden4 states that "effective" comments on students work characterise good teaching practice.As such student evaluation of feedback is often a central metric for the evaluation of teaching quality.5 However, it would appear that the previously stated opinions are not shared by all students; a significant proportion fail to fully engage with the feedback process or do so only when their performance fails to align with their self-assessed ability.6-8Moreover, a positive correlation between overall learner/student satisfaction with an educational programme and satisfaction specifically with feedback does not seem to exist.9 Students have indicated that feedback is only worthwhile if it is applicable to future assignments.10-12It is hypothesised that in professional and/or vocational programmes (such as dentistry), feedback that has a direct impact on enhancing future performance should be commonplace.Additionally and with direct reference to clinical practice, in particular it is believed that students would place high value on this type of feedback; but is this really the case?
What is not clear from the literature is how students (particularly dental students) actually use feedback and whether this varies depending on its context and method of delivery.

| Aim
The aim of this research was to explore dental students' actual use of feedback in a variety of contexts.
In order to achieve this, the following objectives were identified; • Explore how students make use of feedback.
• Explore whether the context in which feedback is provided influences the value students ascribe to it.
• Explore whether the context in which feedback is provided influences the way students use it.

| METHODS
Qualitative methodology was considered the most appropriate means of achieving the projects aims.Focus groups were undertaken in three UK dental schools, and these were facilitated by clinical academic staff from the participating schools.Focus groups were selected as the method of choice, in preference to individualised interviews, as they have been shown to allow interactions between the participants that enhance the depth of the data.13,14To encourage students to freely express their opinion without prejudice, facilitators were selected to be out with the host institution, that is at school A focus groups were facilitated by the researchers from schools B and C, at school B focus groups were facilitated by researchers from school A and C, whilst at school C focus groups were facilitated by researchers from schools A and B.
Topic guides were developed for the focus groups (see Appendix S1).These were reviewed and iteratively modified in conjunction with emerging themes as the study progressed.Two focus groups were planned for each institution with an expectation that if saturation of data had not been obtained after completion of 6 focus groups, further groups would be arranged.Of the two focus groups undertaken at each institution, one had a topic guide that focussed on academic feedback (written assignments/assessments) whilst the second was focussed predominantly on clinical performance.The research team considered data saturation to have been reached when the last focus group reiterated many of the themes that emerged previously.
Student participants were recruited using a purposive sampling strategy to ensure diverse representation across the programmes.
Students were given written details regarding what would be expected of them during the study, and those willing to participate then provided informed written consent.Ethical approval was granted by each of the Universities involved.
Focus groups were audio-recorded, transcribed and anonymised to avoid the possibility of individual participant identification.
Transcripts were checked to ensure that verbatim transcription had taken place.Thematic analysis utilising an inductive interpretivist approach was undertaken.15 On completion of the data collection phase, the researchers familiarised themselves with the transcript of the initial focus group prior to participation in a thematic analysis alignment process.
During this process, the researchers worked through the transcript together and discussed the emergent themes, sought agreement and achieved alignment prior to coding and the further identification of emergent themes and subthemes within the remaining transcripts.A thematic map was also produced.
Each of the research team then took responsibility for providing a primary analysis of one of the remaining five transcripts, before passing their coded transcript to another member of the team for confirmation.The distribution of transcripts was controlled to ensure that researchers did not analyse or cross-check a transcript from their own school.Any disagreement between the pairs of researchers was referred to a third member (the lead researcher) for mediation.In this way, multiple researchers played a part in the collection and analysis of data, thus ensuring researcher triangulation.16Thematic analysis was undertaken manually and when completed the thematic map was updated.

| RESULTS
Six focus groups were undertaken; two at each of the participating schools.In total, 72 students took part in the study with 24 students from each of years 2 to 5 participating from each of the three schools (Tables 1 and 2).
Thirty-six students (12 from each school) participated in the focus group considering mainly academic (non-clinical) work with students from years 2 to 5 of the 5-year programme at each school participating.First-year students were excluded as the timing of the research precluded them having received any significant feedback.
Thirty-six students (again, 12 from each school) participated in a focus group considering feedback relating to the performance of clinical procedures.This group included only students from the clinical years of the respective programme.themes were seen to be emerging, and the final two focus groups were conducted to ensure equal representation of the three schools.
The research team were thus satisfied that saturation of themes had been fully achieved.
Initial analysis identified five emergent overarching themes which influenced the usefulness and use of feedback: value, utility, accessibility, variability and understanding.Further analysis identified subthemes.The pertinence and relevance of the overarching emergent themes and associated subtheme findings are presented in turn.
In order to contextualise the results, details of the type of feedback provided at each institution are provided in Appendix S2.

| Emergent theme 1: Value
In order for students to use feedback and find it useful, it had to be valued by them.Value was perceived and described through several emergent subthemes.The first two were learner centric and therefore perceived not to be influenced by the nature or context of the actual feedback.
The learner stance and/or their intrinsic learner style affected the value that individuals placed on feedback and, therefore, the likelihood of them accessing the feedback and using it.Some students alluded to being disinterested in feedback if they had passed an assessment and could therefore be described as being strategic in their approach to learning; seeing it only as a means by which to cross an assessment threshold and progress within the programme.For some, this was also articulated as an apparent lack of ambition to be the best they could be, recognising that within dentistry there was a very clear standard that must be reached in order to graduate and be registered with the UK's governing body (the General Dental Council).
We're just trying to get to that level where we're safe and we're good practitioners.
Some students, however, expressed the desire to use feedback in order to improve their performance regardless of the outcome of assessment.In this respect, there was a stated tendency to focus on the more negative areas of feedback as these were seen to be of more value in identifying where most improvement could be gained.This was particularly the case for feedback on clinical performance.
I always forget about good feedback and just focus on the negative stuff, to be fair.This is how you improve.
…some people looking at feedback will just click straight to the low scores and ignore everything that went well.
Within value, the next emergent subtheme related to positioning around assessment thresholds.This appeared to influence whether feedback was accessed and utilised, with students just below a pass/fail or "classification" threshold particularly seeking and using feedback to assist them in identifying how much, and where, improvement was needed to reach the next level.In these circumstances, a lack of "adequate feedback" was seen to be particularly frustrating.

When you've been one mark off getting the higher of the two (grades on offer), I think that's frustrating when you don't get adequate feedback or written feedback because that's when you want to know how close you are and you don't really know how close you are.
Some students seemed to attribute less value to written or recorded feedback on clinical performance.This could be related to the perception that within a clinical environment, as well as verbal and visual feedback from teachers, students are also constantly receiving multisensory feedback (sight, feel, muscle memory, trepidation) that could be profoundly and deeply imprinted in their minds.This was attributed to a general lack of experience by some and in others particularly to situations when things had not gone well.
…our lack of experience makes everything more memorable …when I'm actually doing the prep (preparation) you will find that you will have some muscle memory from before and you will know how it's meant to look A further subtheme emerged around the notion of respect.
Students clearly valued feedback more when delivered by an "expert", both clinical and academic.It was noted that, although likely to respect the opinion of peers less, students recognised that peer feedback had a limited value in certain situations.

Peers obviously haven't got the same expertise as all the staff members in terms of writing about research essays.
Quite often, with regards to the content, the people who mark it are experts in the field anyway.
Like my peers might be able to mark, say, the way it's written and the use of English equally as well and so I'd value that if it was done well but obviously with regard to the content, through no fault of their own, peers probably can't get the same scope on that as a marker would.
Similarly, there was respect shown towards the unique perspective of patients and there was recognition of the value that patient feedback could provide to students.

Yeah and I think it's a good perspective to see because you can be the dentist or the dental student, you can be the clinician, but you're not the patient and the patient's
the one that's getting it done on them, the patient in practice is the one that's going to be paying for it or complaining about it or thanking you for it, so hearing their opinion about the treatment, how you deal with them, I think is really important.
Within the subtheme of respect, the students talked extensively about the value of feedback in demonstrating the assessor's respect for the student, their academic effort and the worth of their work.
Where feedback was inadequate this was perceived by the students to portray a lack of respect.In some instances, this went as far as speculating whether assessors had actually read the work when there was no tangible evidence of feedback on large sections of the submission.

Yeah, it's quite frustrating when you've spent ages doing the work and you get the feedback back and, for example, I had one page annotated and the rest of the essay hadn't even been annotated. I almost felt… I might be wrong but I almost felt like they just hadn't read it because the only thing that was filled
out was the matrix.
Finally, within the theme of value, several students stated that positive feedback was valued because of its ability to provide them with affirmation of their progress and provided emotional reassurance.Feedback in this instance was valued for boosting confidence, whilst negative feedback or low grades could adversely impact on confidence.
This was particularly apparent in relation to clinical performance.

| Emergent theme 2: Future applicability
The second major theme on students' use of feedback related to the pragmatics of feedback being able to be applied in the future.Within this theme, there were four subthemes: quality and quantity, individualisation, timeliness and transferability.
The amount of feedback and its quality strongly influenced the students' perception on its future applicability.Limited feedback via single words, "ticks," a summary comment or overarching grade was seen of being of virtually no use for identifying areas for improvement.Likewise, feedback that seemed inconsistent with the overall grades awarded was perceived as having little usefulness.This applied to both students who were at the pass/fail threshold but also to those who aspired to raise their achievement from an already good level to an even better one.It was also noted that students appeared particularly frustrated by a lack of constructive feedback when awarded grades were high.Yeah, so its drip feed all the way through and you do get some clinicians that are marking as you go through which is really useful.Now, we just get feedback all the time and we're constantly inputting it and so I think we're more aware that we're getting it and we're using it a lot more in reflecting on it better.
In contrast to clinical feedback, it was recognised that a time delay of only a few days impacted on the ability of students to recall and relate to written assignments and thus find utility in the feedback.

By the time you get your feedback you've kind of forgotten what's gone into the piece of work anyway so it's less relevant
The transferability of feedback to future assignments was perceived as an essential attribute; this was of particular relevance to written assignments.Some students recognised that, whilst topic or subject areas may change, feedback on writing and other academic skills could be valuable.In contrast, students very much recognised the transferability of feedback on clinical work and, for that reason, found it useful and tended to apply it more readily to future clinical sessions.

| Emergent Theme 3: Accessibility
Unsurprisingly, students were only able to utilise feedback if it was readily accessible to them; however, the perception of accessibility was subject to some variability and perhaps divergent from assessors' opinion.Three subthemes were identified.
The format in which feedback was delivered, in particular whether this was in hard copy, verbal or electronic, influenced students' perception of its accessibility.Most stated a preference of accessing feedback on academic assignments and examinations in an electronic format via their phones, tablets or laptop computers.
The convenience of access and "retrievability" of feedback delivered through remote electronic media was noted to be particularly useful.

you picked it (a piece of paper) up from the office, if you lost it you would never see it again because the information is stored electronically, we can always go back and read it we'd be more likely to look at it again because you can access it whenever you want to
Students perceived and reported on the fact that electronic copies could be more easily retained and found within virtual learning environments, or within their personal computers, and were more readily retrievable.Thus, this format of feedback was more likely to be sought out for application to future work.Paper copies or even self-made and self-stored photographic images of previous feedback were less likely to be looked at.

I take photos but hardly ever look at any of them if you asked me to go and find my feedback on a sheet in a pile of paper, it's probably just put somewhere and I'm not going to see it again
Hard paper copies, their accessibility and their retrievabililty seemed very much dependent on individual student "systems" of organisation and document storage, ranging from "a box in the garage" to complex and well-structured systems of recording feedback that enabled students to retrieve and utilise it more readily.

…unless it's all in a folder or easily accessible you tend not to use it
The legibility of written feedback was essential for students and, on occasions, presented real challenges related to the assessor's handwriting and their writing medium.This again linked with a preference to electronically delivered feedback where the notes are typed or audio recorded.

three times I have got a sheet and I can't read it because it's in pencil and very smudgy sometimes you can't even read their writing…and so it's pointless
Although there was no overall consistency or desire to receive audio feedback, many students believed that they would simply make written notes from such feedback and to which they would re-access it at some point in the future if required or desired to do so.

| Emergent theme 4: Variability
The fourth emergent theme related to the variability of students' experience of feedback.In each of the focus groups (and regardless of context), the learners clearly felt very strongly about this theme with frequent dissatisfaction and disquiet being expressed.The students identified that, in their experience, the quality and quantity of feedback was largely dependent on the individual assessor and it was therefore down to some degree of chance as to whether they gained helpful feedback.

I've seen other people's feedback that had paragraphs
and paragraphs written … and then some had one sentence.
the feedback we receive depends on the examiner you get Variability was also determined to be somewhat unfair, in that those who had failed seemed to have had more "time" devoted to giving them feedback.

| Emergent theme 5: Understanding
The fifth theme related to understanding and had two subthemes.
The first encapsulates the divergence that may exist between institutional processes, student needs and their interpretation of process.Where the institutional processes seemed at odds with the learners' needs, it became apparent that the students developed "workarounds" in order to record feedback they received.
if they've annotated our essay we have to give them back so most of us take a picture of them we have notebooks that we make notes in, most people do that The final subtheme was an understanding, gained by some students, of additional purpose to the giving and receiving of feedback, that is the

| A model for the provision of feedback
Following analysis, the interrelationships between themes and students use of feedback were considered and illustrated as a model (Figure 1).

Future applicability
Future assignments

Future career
Respect for learner Justifcation for grades

Staff training
Course design

Use of learning technology
It has been argued that feedback is only effective if it results in enhancement of future performance of the learner.17In developing a model for feedback, we considered that the learner and their context or stance should therefore be at the forefront.Examples of subthemes that were seen to be influential on learner's stance included their positioning around assessment thresholds, their inherent learning style and whether their self-assessment of performance aligned to that of their assessors.The ability of institutional processes to affect the learner stance was not identified within the students who took part in this study.
It was clear from our analysis that in order to engage with feedback the learner must place value on it and that such value is influenced by several factors including the learner stance.The credibility of the feedback was strongly influential on the value that learners ascribed to it.Factors such as alignment of feedback to the final grade, status and demeanour of the person giving the feedback and consistency of approach to the quantity and quality of feedback were all seen to contribute to credibility.Students identified that the credibility of feedback could be influenced by institutional processes such as staff training, marker alignment, understanding student expectations, and appropriate use of marking matrices and free-text comments in the justification for awarded marks.
In order to place value on feedback, we observed that students needed to understand its purpose and the processes that underpinned it.For some, recognising that developing an ability to receive, accept and act upon feedback was a secondary, but nonetheless essential, skill that contributed to the development of their professional identity.
The future applicability and the students' understanding of future relevance influenced the value that students placed on feedback.The learning institution was seen to have the ability to influence this through the configuration of a "scaffolded" assessment framework.However, it was also recognised that the transferability of skills and knowledge may not always be obvious to students and institutional process could also influence this perception.
An important relationship between "being felt to be valued" and the quality and quantity of feedback was also identified, with students expressing the opinion that good quality feedback affirmed for them the respect that the institution had for their work and the effort that it had entailed, and respect for them as a part of that institution.The value of the feedback is therefore shown in the model to affect learner stance.
Valuing available feedback is, however, not in itself, sufficient to enable usage if the accessibility of that feedback is inadequate.
In line with other studies, 11,18 we identified that accessibility was influenced by the format of feedback, its legibility and ease of retrieval, all of which are informed by the institutional processes that exist.
Based upon the findings of this study, we suggest that within the proposed model there are factors which may not be in the immediate and obvious control of the institution, such as the individual students learning style and their positioning around assessment thresholds.However, the wider curriculum does perhaps have the potential to influence positively learner understanding of the feedback process and its purpose, variability, future applicability and accessibility of feedback.Subject to further validation this model may be widely transferrable and useful for institutions wishing to review or design feedback processes.

| DISCUSSION
Our analysis suggests that in order to use feedback the learner needs first to place value on it and that the value of developmental feedback is influenced by learner stance.Are they a deep learner who wishes to continually enhance performance, or a strategic learner aiming to reach the minimum standard required to "pass"?
Institutions can take steps to influence learner stance by raising awareness of learning styles and study skills19 but, perhaps of more value, is the institutional ability to enhance and develop student feedback literacy.20In this study, institutional processes for the provision of feedback had commonality but were not identical (see Appendix S2).There was no distinction in the way these processes were perceived or valued by students.
To value feedback, the learner needs to feel respected.Students in this study, as elsewhere,12,35 were more likely to access qualitative feedback to understand why they had been positioned around a threshold grade or if they had received an unexpected outcome.Sutton25 suggests that grades are the "prism through which feedback is read" and that students will read feedback to seek justification for the grade they have been awarded.

| Limitations and Future Study
The use of focus groups brought depth and richness to the data collection phase; however, a limitation of only using focus groups might be considered as a lack of methodological triangulation.Further forms of qualitative data collection or mixed methods approach were considered but discounted due to concerns of creating evaluation fatigue.As an alternative way of increasing credibility, researcher triangulation was utilised, whereby multiple researchers played a part in the data collection and/or analysis.
The research team made no attempt to purposefully sample students who had failed examinations previously, nor did we record these details from participants.It could be considered that this is a further limitation of the study as students with this contextual background may have held different opinions and utilised feedback differently.Nonetheless, we did identify the significance of student positioning around assessment thresholds as an emergent theme, perceiving that this altered the value placed on any feedback given and thus the way in which students used it.In reality, the concept of "failing" was clearly articulated by many of the participants, albeit not necessarily in relation to summative assessments; students talked of failing to achieve an expected grade boundary or a sense of failure when clinical procedures had gone badly.We are therefore confident that data saturation was achieved around this theme.
Although this study is of particular interest to dental schools, we would suggest that the findings may be transferable to other clinical healthcare education programmes and beyond; however, this remains to be tested.In a similar vein, whilst the authors can see no reason why the model would not be valid in non-UK institutions, this cannot be categorically stated without further testing.
What is apparent from the study is that further exploration of the influence of learner-centric factors is required.In particular, further consideration of how a student's short-and long-term learning goals may impact upon their use of feedback and how the design of marking criteria and assessment matrices can be influential in fostering engagement with feedback which has clear relevance to future learning and assessment.

| CONCLUSION
Overall, feedback was valued by the students but its use varied, due to context (clinical, oral and written) and method of delivery.
An understanding by education providers of how the interrelationship of factors such as accessibility and future applicability influence its use is important for improving the quality of feedback in dental education.
Pl e a s e n o t e: C h a n g e s m a d e a s a r e s ul t of p u blis hi n g p r o c e s s e s s u c h a s c o py-e di ti n g, fo r m a t ti n g a n d p a g e n u m b e r s m a y n o t b e r efl e c t e d in t his ve r sio n.Fo r t h e d efi nitiv e ve r sio n of t hi s p u blic a tio n, pl e a s e r ef e r t o t h e p u blis h e d s o u r c e.You a r e a d vis e d t o c o n s ul t t h e p u blis h e r's v e r sio n if yo u wi s h t o cit e t hi s p a p er. Thi s v e r sio n is b ei n g m a d e a v ail a bl e in a c c o r d a n c e wit h p u blis h e r p olici e s.
an unintended negative impact and caused them to question their own insight into their performance.It would personally make me panic because I thought I did okay on one and then I'd failed it.And then I kept like going over it and going over it and going over it and I'm like 'What did I do wrong?' and it actually took away my confidence for the next year I did my OSCE.Clinical feedback was seen to be particularly useful when given at the time of a procedure being undertaken or being provided immediately afterwards.The timeliness, as well as frequency, and volume of feedback given around clinical work was recognised as being helpful.

FIGURE 1 A
FIGURE 1 A model for the provision of feedback to students, demonstrating factors within institutions' control that can impact the utility of feedback

TABLE 2
Distribution of participating students across year groups at each of the schools It became apparent that the three schools had a common practice of delivering generic group feedback after OSCE style examinations.Students reported this type of (group) feedback was not helpful and stated a preference for individual feedback at a station/ question level that would allow them to identify specific areas of personal weakness.For many, this was because they felt unable to relate the generic feedback to their own examination performance.
Where people get 70 the marker might think 'oh they must be happy with that'.People are…but there's still another 30%…even though you are making into like a high boundary, you're not getting further because people haven't commented constructively.So they mentioned most of you guys struggled with station 8 but didn't tell us how we could have done better or what they correct answers were…it was a bit pointlessFor some students, this lack of specific personalised feedback had utilisation of the feedback process to develop other skills by virtue of exemplifying good and bad practice, for example personal reflection, communication, teaching, team working and supporting colleagues.
25evious authors report that students place greater importance on grades or marks obtained for their assignments, rather than feedback on their performance.18,26Thiswas not the case in the present study which supports the findings of Douglas et al,6 who demonstrated that students valued constructive qualitative feedback as much as grades, especially in relation to clinical skills.Grades were still considered useful as they allowed students to benchmark themselves and acted as a catalyst for further development.25 passing and failing grades and the number and depth of comments made by assessors for written coursework.Although unmeasured in the current study, the perception of feedback inequality may affect how students value it.The dental schools involved have strategies to identify, support and provide targeted training for underperforming students, and this may well be recognised by others as a bias