Summary
Email received from Barry Wolfenson, Group President, Advanced Wound Care & Drug Development, Derma Sciences.
Q1. Can you please let me know, regarding the 132 Cochrane Reviews on the topic of Wounds, aside from the one titled “Honey as a topical treatment for wounds”, how many include studies within the review that are authored by authors of the Cochrane Review itself?
Authors response: This is a very general question; you have not identified the 132 reviews you are referring to and we do not routinely record that information outside of the review. However, any included trial which has a trialist who is also an author of the Cochrane review must declare this in the Declaration of Interests section of the review.
Q2. Given that one of your stated core principles is to minimize bias, can you please let me know your organizational guidelines regarding authorship of reviews and whether or not authors should be able to review their own studies? To a lay person, there seems to be quite a potential for conflict here.
Authors response: In the Cochrane Wounds Group we adhere strictly to the policy that if an author of a Cochrane review is also a trialist of an included trial in that same review they must declare this in the Declaration of Interests section of the review. This declaration is also made in the online Declaration of Interests form which has to be completed by all authors before a protocol or a review can be published on the Cochrane Library.
Q3. As reviewers of the Jull et al study (Randomized clinical trial of honey-impregnated dressings for venous leg ulcers), how did you judge the relative bias of the study author’s ad hoc decision to remove incidence of infections from the adverse events table?
Authors response: This question is directly with respect to a trial we conducted, not the systematic review. We did not remove incidence of infections from adverse events. The incidence of infection was analysed separately as specified a priori in the protocol and statistical analysis plan in response to interest in whether honey may prevent infections in venous leg ulcers.
Q4. With regard to the above, the removal of infections from the adverse events section somehow allowed the study authors to also remove the costs associated with hospitalizations associated with those infections from their overall cost of care analysis. However, the other costs associated with hospitalization (not having to do with infection) remained. When the costs associated with hospitalizations due to infection were included in the analysis, the honey arm was less expensive. Only after the costs associated with the additional hospitalizations associated with infection were removed (ad hoc) was the honey arm more expensive. Given that the decision to remove infections was ad hoc, how did you judge the relative bias of the statement by the study authors that the honey arm was likely more expensive?
Authors response: This question is directly with respect to a trial we conducted, not the systematic review. Incidence of infection was included in all the cost analyses. An analysis for sensitivity of the base case (all costs for ulcer treatment) to the small number of patients hospitalized for treatment related to their ulcer was undertaken. We do not know what the exact purpose of the hospitalization was i.e. whether it was related to infection or not, only that it was self-reported by the patients as related to their ulcer. As explained in the paper, the difference was likely due to random variability rather than use of honey or not. The decision to sensitivity test the base case was not ad hoc.
Q5. As reviewers of the Jull et al study, how did you judge the relative bias of the study regarding the author’s conclusion that honey dressings should not be considered for venous leg ulcer healing given that all the primary and secondary endpoints favored the honey arm (although not statistically)? It would seem, based on the results of the study, that honey should be included into the control group of alginate, hydrofiber, hydrocolloid, foam, non-adherent, iodine, and silver-based dressings as dressings that have demonstrated efficacy as adjuvants to compression therapy. Again, given the positive nature of the evidence derived in this study, how did you judge the potential bias within the authors’ conclusion?
Authors response: This question is directly with respect to a trial we conducted, not the systematic review. There were no statistically significant differences between the groups on the primary outcome, nor on any of the secondary outcomes. In fact the results for the blinded verification of healing suggest barely any difference between groups (absolute difference 0.5%, 95%CI -10.6% to 11.3%) on the primary outcome. Routine use of honey dressings is therefore not superior to other dressings for promoting healing compared to usual care. This conclusion does not preclude non-routine use of honey.
Q6. You included a second study in your Cochrane Review to address the management of venous leg ulcers. This second study, by Gethin et al, allowed only those wounds with greater than 50% slough coverage. In the larger Jull et al study of potentially likely to heal venous leg ulcers, the authors stated that the positive results (not statistically significant) favoring honey should be “generalizable” to all venous leg ulcers, regardless of wound bed appearance, unless new data shows otherwise. In the Gethin study, the wounds in both arms had average slough coverage ranging from 78% - 86%, and had average wound areas of 9.9cm2 to 10.5cm2. However, in the Jull study, there is no notation of slough coverage and the wounds ranged in average area from only 2.6cm2 to 2.7cm2. Thus, the wound bed appearance was dramatically different between these two studies. If the authors of the Jull study thought it was wise to view different wound bed appearance as indicative of a different patient set, why did you chose to combine results of these two vastly different studies? It should be noted that in the Gethin study on harder-to-heal wounds, even a much smaller number of patients resulted in the honey arm providing statistically significant healing benefit over control. Based on what the authors of the Jull et al study stated, shouldn’t your conclusions from these two studies have been: Honey provides a non-significant healing benefit for routine venous leg ulcers, and provides a significant healing benefit for non-routine venous leg ulcers?
Authors response: The commentator states two studies were “vastly different” but mistakenly compares median ulcer size reported in Jull et al to mean ulcer size reported in Gethin & Cowman, two statistics that are not comparable. The median ulcer area reported in Gethin & Cowman was 5.4 and 4.2 cm2. Ulcer duration is also an important consideration when considering likelihood of healing. The mean duration of ulceration prior to enrolment in the Gethin and Cowman study was 39 weeks and 30 weeks, which was similar to the means in Jull et al (39 and 48 weeks). To suggest therefore that Gethin & Cowman recruited harder-to-heal wounds compared to Jull et al is not accurate. These studies were not dissimilar in terms of populations and there is no evidence to suggest that they were different in terms of wound bed appearance – one trial provided information on this factor, another did not. In the Gethin & Cowman study, the statistically significant difference only emerged in the adjusted analysis. The unadjusted analysis for Gethin and Cowman is presented in the systematic review and shows no significant difference (RR 1.33, 95%CI 0.82 – 2.16).
Q7. Regarding honey versus SSD, why should the results of these studies be disregarded as you suggest due to other studies having shown benefit of hydrocolloids and silicone based dressings over SSD? Does this mean that you recommend that no other studies be done comparing one arm to SSD in the treatment of partial thickness burns?
Authors response: There is evidence that suggests SSD cream applied from time treatment initiated until healing may delay healing. (Thomas SS et al. J Wound Care 1995;4:218-20. Wyatt D et al. J Trauma 1990;30:857-65. Bugmann P et al. Burns 1998;24:609-12.) It appears the issue may be how SSD cream is used, not whether it is used. Our focus in the review was in trying to understand whether honey conferred a benefit. Using an appropriate comparator is pertinent to such an analysis and it is not clear that SSD applied daily from treatment initiation to healing is an appropriate comparator.
Q8. Regarding the above, which do you think is the more commonly used product on partial thickness burns, SSD or hydrocolloids? Are you aware of the commonplace usage on SSD in burn centers? If so, why would you suggest disregarding the results of studies which show a benefit over SSD? If not, and you find this to be true (that SSD is still commonly used in burn centers), would you change your decision to disregard the evidence derived from studies utilizing SSD as a control arm?
Authors response: See above
Q9. Also regarding the honey vs SSD studies, can you provide a more thorough clarification on why the CHIT method employed by the investigator disqualified these studies from your analysis?
Authors response: The use of the chit method did not disqualify any trial from consideration. Authors of trials that did not describe the method for generating the random sequence were approached for further information. The author of 11 trials replied stating the sequence was “manually generated random numbers by the chit method”, without providing any further information about what was meant by this approach with respect to sequence generation. We sought the advice of broadly experienced senior biostatistician who could not infer from this response what method of sequence generation was used. We therefore stated in the Risk of bias tables “method of generation of random sequence not reported. The author informed us that the sequence was generated by the ‘chit method’, but it is not clear what this method is.”
Q10. Of the 6 additional studies cited as rationale for this “updated” Cochrane Review, 5 out of the 6 are positive in favor of honey. Why is only the one negative study included in your updated conclusion? How did the biases of this study, as well as the overall structure of the study, compare with the other 5 studies which were not included in your updated conclusion? Were these differences the reason you only cited the negative study?
Authors response: When a review is updated we run the searches again and assess the resulting output. For this update we identified six additional studies which met the inclusion criteria published in the review, and all six studies were included in the review with the results of those studies presented.
Q11. Would someone be wrong if they stated it appears as though your inclusion of only the one negative study, and lack of inclusion of the 5 positive studies, appears to be “cherry picking”?
Authors response: A systematic review is conducted according to a defined, peer reviewed and published protocol to minimize bias. Any studies which are excluded have to be identified in the Table of Excluded studies and the reasons for the exclusion made clear. I am assuming you are referring to the six additional studies which were included in the last update of this review. All six studies are “included studies”.
Q12. Regarding the one negative study, which was on the treatment of cutaneous Leishmaniasis (a wound resulting from a bite from a sand fly indigenous to the Middle East), the study authors note that further studies should be done using standardized medical grade honeys, such as Leptospermum Scoparium (Manuka Honey). They acknowledge the fact that they used plain local honey could have been the reason for not achieving positive results. Why did you not include this in your conclusions or anywhere in your description of the study?
Authors response: The purpose of the review was to summarise the available evidence, not to restate authors’ interpretations.
Q13. Regarding the negative study on honey vs early excision and grafting of mixed partial and full thickness ulcers, to the best of your knowledge, how many studies do you know that compare one dressing vs this surgical standard and show improved results? Given the rest of the consistently positive results of the other honey studies cited for management of partial thickness burns, do you think the results of the early excision and grafting study suggest that honey would in any way be dangerous to use on partial thickness wounds that do not require early excision and grafting? If so, why?
Authors response: This review does not set out to present results of any dressing when compared with early excision and grafting of mixed partial and full thickness ulcers. Clearly there was sufficient equipoise for a trial to be conducted comparing early excision and grafting versus treatment with honey and delayed grafting as necessary. This trial was therefore included in the review.
Q14. Can you please provide the total numbers of the following: How many studies cited in your review had a statistically significantly positive outcome for the honey arm? How many studies cited in your review had a positive outcome (although not significantly) for the honey arm? How many studies had a negative outcome (although not significantly) for the honey arm? How many studies had a statistically significantly negative outcome for the honey arm?
Authors response: The data extracted from the trials and presented in the results section is provided in the review. Counting the number of statistically significant studies is not a sensible approach to summarising evidence when meta-analysis is possible.
Q15. Given the numbers above, why did you feel that it was important to include in your updated conclusion the following statement; “…purchasers should refrain from providing honey dressings for routine use until sufficient evidence of effect is available”? This statement seems cautionary in its advice. What is the caution that you believe purchasing agents should take into account when considering routine use of honey based dressings?
Authors response: In the light of insufficient evidence a cautionary approach is warranted. Routine use of honey dressings refers to use of honey as a first line dressing.
Q16. Given that Andrew Jull is not only is the author of the key study cited in the Cochrane Review, but that he is also on the editorial board for the wounds group reviewing all Cochrane Reviews on wound care, this question is directed specifically to him in his role as an editor: Due to the lack of well controlled RCTs, the vast majority of Cochrane Reviews on wound dressings / technologies (with the exception of total contact casts) have negative conclusions. The majority of these conclusions all seem to have a “boiler plate” type of statement, basically stating “there is not enough evidence to suggest the use of one product over another.” Given the recent updated conclusion in the review of honey dressings, do you think, if any of the other reviews are similarly “updated” (including those on negative pressure, silver based dressings, alginates, hydrocolloids, foams, etc.), without any meaningful positive results from well controlled studies, do you think that clinicians should similarly be advised to refrain from routine use of those dressings/technologies for routine use until sufficient evidence of effect is available? If not, then why?
Authors response: Andrew Jull is an Editor of the Cochrane Wounds Group; he does not review all Cochrane reviews in wound care and reviews are distributed amongst all the editors and each review is peer reviewed by a number of editors and reviewers. It is not appropriate for the authors of this review to suggest how other authors should write their reviews.
Q17. Why was the statement, “purchasers should refrain from providing honey dressings for routine use until sufficient evidence of effect is available” subsequently changed?
Authors response: This change was in response to feedback received from Barry Wolfenson on behalf of Derma Sciences (14th June 2013) expressing concern at the wording of the Authors' conclusions - as follows: “There is insufficient evidence to guide clinical practice in other types of wounds, and purchasers should refrain from providing honey dressings for routine use until sufficient evidence of effect is available.” This statement had been present in the Authors' conclusions since the review was first published in 2008. Cochrane guidance from the Handbook states "The primary purpose of the review should be to present information, rather than to offer advice". Having considered the feedback and received the advice of the Cochrane Collaboration, we have modified this section to read: "…There is insufficient evidence to guide clinical practice in other areas, and health services may wish to consider avoiding routine use of honey dressings until sufficient evidence of effect is available". Although the primary purpose of a review is to present information, rather than to offer advice, review authors are invited to interpret the findings of their review in the "implications for practice" section. This section is clearly labelled as "authors' conclusions".
Q18. Do you believe the following statement is intended to provide advice; “…Health services may wish to consider avoiding routine use of honey dressings until sufficient evidence of effect is available”? If not, how do you interpret this statement to not be providing advice on usage?
Authors response: Review authors are invited to interpret the findings of their review in the "implications for practice" section and these are clearly labelled as "authors' conclusions". It seems perfectly reasonable where no compelling evidence of benefit has been identified to suggest that decision makers consider this finding. The use of the terms "consider" and "routine" clearly indicate that this is not intended to be a didactic recommendation, simply something for consideration by decision makers.
Q19. How many times is this type of advice given in any of the other Cochrane Reviews regarding wound dressings? Please only limit your response to those Cochrane Reviews that had negative conclusions, such as on alginates, foams, silver dressings, negative pressure, etc… If the answer is 0 (zero), what do you think is the distinction between the evidence presented for honey-based dressings and the evidence presented for these other dressings/technologies?
Authors response: There is no blueprint for the precise wording of this section, and it is not unusual for Cochrane Reviews to present text that provides guidance that decision makers should consider the absence of high quality evidence of benefit when making treatment decisions.
Q20. Specifically regarding your statement in the Cochrane Review that clinicians may consider refraining from routine use on honey-based dressings, can you please clarify:
20.1 For routine venous leg ulcers: Should clinicians refrain from using honey-based dressings? If so, why?
Authors response: It is reasonable to highlight the apparent failure of routine use honey dressings to deliver such an important outcome as healing rate.
20.2 For chronic/stalled venous leg ulcers: Should clinicians refrain from using honey-based dressings? If so, why?
Authors response: All venous leg ulcers are chronic. The evidence from the trials does not address “stalled” venous ulcers.
20.3 For other routine ulcers such as pressure ulcers: Should clinicians refrain from using honey-based dressings? If so, why?
Authors response: The results of the review demonstrate clear uncertainty in respect of this outcome.
20.4 For other chronic/stalled ulcers such as pressure ulcers: Should clinicians refrain from using honey-based dressings? If so, why?
Authors response: The evidence from the single available trial does not address “stalled” pressure ulcers.
20.5 For partial thickness burns that do not require early excision and grafting: Should clinicians refrain from using honey-based dressings? If so, why?Authors response: The
results of the review demonstrate clear uncertainty in respect of this outcome.
20.6 For any challenging wounds where a clinician would typically use negative pressure, silver-based dressings, iodine-based dressings, alginates, or foams: Should clinicians refrain from using honey-based dressings? If so, why?
Authors response: There were no trials where the wounds were defined as ‘challenging’.
20.7 Do you consider chronic wounds (whether they be venous leg ulcers, diabetic foot ulcers, pressure ulcers, or dehisced surgical wounds) to be “routine”?
Authors response: It is not clear what is meant by “routine” in this context – none of the conditions are uncommon.
20.8 Do you consider partial thickness burns that are treated in a burn center to be “routine”?
Authors response: It is not clear what is meant by “routine” in this context.