How is research publishing going to progress in the next 20 years? Transcription of Session for Editors, Associate Editors, Publishers and Others with an Interest in Scientific Publishing Held at IADR Meeting In Seattle on Wednesday, 20 March 2013
On March 20th 2013, a one-hour session for Editors, Associate Editors, Publishers and others with an interest in scientific publishing was held at the IADR International Session in Seattle. Organised by Kenneth Eaton and Chris Lynch (Chair and Secretary, respectively, of the British Dental Editors Forum), the meeting sought to bring together leading international experts in dental publishing, as well as authors, reviewers and students engaged in research. The meeting was an overwhelming success, with more than 100 attendees. A panel involving four leading dental editors led a discussion on anticipated developments in publishing dental research with much involvement and contribution from audience members. This was the third such meeting held at the IADR for Editors, Associate Editors, Publishers and others with an interest in scientific publishing. A follow up session will take place in Cape Town on 25 June 2014 as part of the annual IADR meeting. The transcript of the meeting is reproduced in this article. Where possible speakers are identified by name. At the first time of mention their role/ position is also stated, thereafter only their name appears. We are grateful to Stephen Hancocks Ltd for their generous sponsorship of this event. For those who were not able to attend the authors hope this article gives a flavour of the discussions and will encourage colleagues to attend future events. Involvement is open to Editors, Associate Editors, Publishers and others with an interest in scientific publishing. It is a very open group and all those with an interest will be welcome to join in.
Ken Eaton - Chair
Ladies and gentlemen I would like to welcome you. It is great that there are so many of you here. The panel is made up of four editors: Will Giannobile, Rex Holland, Stephen Hancocks and Peter Robinson. The proposal is that we think about how research publishing is going to progress in the next 20 years. The proceedings are being recorded and it is the intention to publish them as a record for the future. A huge number of issues may or may not be relevant. The ones that are now on the screen have been suggested by our panel. I will quickly run through them just to get you thinking:
- Open (or free) access
- Expert analysis
- Vast explosion in the number of papers compared with how things were 20 years ago
- Quantity without quality
- The ethics of the publications, the integrity, also of the publisher and the publication
- Should we review literature quickly and accurately and use agreed research protocols to make studies more comparable
- Stronger peer review
- Getting reliable peer reviewers, should we pay them? Maybe but where is the money coming from?
- International accepted code of publishing responsibility
- Encouragement of research reporting the reproducibility of previously published studies
- Democratisation versus quality and the whole quality issue
- Secondary publishing, the broader use of electronic discussion of recently published studies.
- In 20 years’ time will most of the methods sections will be in the form of a video with a short commentary so that people can see exactly what happened?.
- Clinical trial registration
We have a lot of baggage from the past but let's go forward into the future with good ideas and let's see collectively in the next 55 min what we can achieve.
I would like each of the panellists now just to make very brief introductory statements. Will [Giannobile] can we start with you?
Will Giannobile (Editor – Journal of Dental Research)
First I would like to thank you Kenneth for organising this gathering and I think that as you had mentioned it was a couple of years ago that you first brought together many editors from across the globe looking at dentistry and dental publications and really getting together as a group understanding a lot of the uniqueness of dental publications but also addressing some of the broader issues that impact all of the journals across biomedical research. So really we see this as a gathering with editors and those involved with the peer review process looking at ways that we can improve the whole process, understanding the changeability right now in our current climate and ways that we can work together in furthering the publication process. So certainly you have provided a lot of good discussion points and so I am really looking forward to this session and again thank you for bringing us together.
Rex Holland (Editor of the Archives of Oral Biology)
The most persistent problem an editor faces is finding quality referees. This is becoming increasingly difficult partly because refereeing gets little recognition. The most obvious way around this problem is to pay referees. Paid reviews would likely be done first and fast in the hope that one would be invited again. This would be particularly appreciated in countries such as the USA where faculty are obliged to earn part of their salary from outside sources. This would also provide a measurable parameter (earnings) for promotion purposes. This already goes on with some of the newer journals that extract submission fees.
The second major concern is linked and is the apparently high level of publishing misconduct. This has attracted much more attention and there are a number of international groups addressing it. Particularly interesting is the number of papers that are now retracted. Not all retractions are due to misbehaviour but many are. Depressing evidence of this is available online (http://retractionwatch.wordpress.com, http://ori.dhhs.gov) Twenty years on will see a reduction of this problem with the wide availability of antiplagiarism software and search engines that can detect duplication and ‘salami’ science.
One other change that would be beneficial would be a change in attitude of reviewers and editors towards reports that repeat studies that have already been done and towards that contain negative but valid data. With, effectively, an unlimited number of clouds finding somewhere to publish these should not be difficult.
Stephen Hancocks (Editor-in-Chief of the British Dental Journal)
Thank you very much Ken and thank you ladies and gentlemen for coming along today. It is good to see so many of you. I have absolutely no idea where we will be in 20 years’ time. But I often think that it is useful when looking ahead to actually look back 20 years and if we look back to 1993 incredibly we wouldn't even really have had the Internet. We wouldn't have had email, most of us wouldn't have had cell phones or mobile phones and so we would have been busy faxing one another. There is a gentleman at the front here looking completely quizzical, it is like what world did I come from? I lived in a cave just outside London! But the explosion of technology and of communication, possibilities that have happened in the last 20 years has been vast. I remember there was an FDI congress in Amsterdam in 1989 and editor Helmut Hydt, editor of the Dental Journal of South Africa, burst into a meeting saying it was all going to change. Helmut was talking about a page layout programme called PageMaker. He said ‘this is going to change everything we do, it is going to change everything’. That was 1989 and now we just take all this completely for granted. I think that is important because where we are going to be in 20 years’ time is in a lot of ways going to depend on what we can actually do technically, physically and so forth.
But having said that, there are some fundamentals that will still be in place and they have already been mentioned. One of course is veracity. How do we know where the truth is? It takes two people to tell the truth and that is why we have referees. I am second in the queue after Rex for the thousand pound refereeing fees that sounds great. I'm told it was dollars, you see it is going down already! So there are some fundamentals, how do we really sift out the truth? So there are going to be some fundamentals that I think will still be in place. I think that there are going to be a lot of things that we are going to have to carry on adapting to whilst keeping those fundamentals in place. High up on the agenda are things like plagiarism and so forth because they are human attributes. They are still going to be things that we need to take care of.
Peter Robinson (Editor of Bio-Med Central Oral Health)
Thank you very much Ken for giving me this opportunity to talk. I was recently appointed as the section editor for oral epidemiology and health services research for BMC Oral Health, which hopefully you all know is an open access journal. The general model is that authors pay to publish in BMC Oral Health. That model seems to be a model for the future. It is difficult to see very far ahead but certainly by the indication by the number of submissions to the BMC Oral Health and the other journals in the BMC suite that seems to be the way ahead. My appointment as section editor marks a threshold in the number of manuscripts submitted to BMC Oral Health.
My appointment also marks a real effort by the publishers to separate the editorial decisions from the revenue stream. That is to say since BMC generates its revenue by accepting manuscripts they don't want to be directly involved in deciding how many and which ones are accepted.
Online publication presents an interesting conundrum and I was pleased to see that Ken had used some of the information that I had given him, when producing the list of topics that we might discuss today. For us at BMC quality is interesting because we are not constrained by space in the same way as the print journals are. Hence there is less pressure on us to be selective, unlike the situation say at JDR. The general policy is that we will publish anything that represents an incremental advance. I am slightly anxious about that partly because I teach critical appraisal to postgraduates across England and I have learnt that many readers rely to a greater or lesser extent on publication as an indication of the quality of what they are reading. That is to say that appraisal skills amongst readers are worse than they are amongst reviewers, not universally good. So there is a great deal of pressure on a journal such as our own and how we are going to overcome that.
Thank you panellists for your wise words. Now the floor is open. We have made a lot of points, and I now call on you individually to give your opinions. Shall we start with the peer-review situation? That is one of the burning issues I think. Hello, there yes, Rowena what is your question?
Rowena Milan (BDJ)
Can I just ask the question to Rex Holland maybe, how many review requests do you receive per month.
Right, so in Rex's world you get paid a thousand dollars per review, that is six thousand dollars, which you would be quite happy to take up I would expect.
I think that we ought to be careful and think about this, in a world where some journals will pay reviewers and others won't, the journals that don't are going to have to jump on the bandwagon first otherwise they are not going to get any reviews, how is that going to affect academia as a whole in a world where having 6 thousand dollars a month for peer reviewing.
Rowena thank you very much. So what are the implications of paying reviewers and what they might be? One of them is who is going to pay and also where is the money coming from because our current system of subscription to journals doesn't really accommodate this.
In response to the point that is being paid about the making payments to reviewers, assuming that it takes around three hours to do a thorough review, then when one takes into account university overheads, senior staff salary etcetera, then perhaps one thousand pounds per review is not too wide of the mark. The issue is that in an era where journals are asking us to review for free but then also asking us to pay to have articles available on open access then to me there is an element of publishers potentially having their cake and eating it so to speak – a situation that is unsustainable.
Unknown member of the audience
Unfortunately a relatively small number of researchers review papers. A lot of people say I don't have time. So I think that if you for more than once, twice, three times refuse to review papers it might be an option to block these people from having their paper published in your journal.
Yes, well this is an idea which we suggested about 2 years ago when we had the inaugural meeting of editors and publishers and the principle was submit one, review one and that is one way of looking at it, I think that we would have to get a consensus amongst all the quality journal publishers and editors to do that but that could be a way forward, Rex?
Well the submission fee could balance the refereeing fee and there is another consideration about reviewing, which is that there is no credit for it. You come up for 10-year promotion or whatever, retirement, and they count papers, they can count the number and amount of grants, but the contribution to refereeing is not measurable easily and so it doesn't count when it should, it is a very important part of personal assessment. If there was money involved they could count that money and it would become measurable and also certainly in the USA a lot of faculty members are obliged to earn part of their salary, usually that would come from grants but it could also come from refereeing as well.
Michael Glick (JADA)
Eight years ago I actually rewarded my reviewers with continued education credits, so they received something for it, but it didn't cost me that much. Yes they can do it but we also actually evaluate the quality of the review and if the quality is good enough they get the credit, if not they won't.
Well I think that is absolutely excellent because we have all had absolutely dreadful reviews, you know the stupid comment this is a great paper accept it, which is absolutely meaningless to us but we all get them don't we.
How would you handle accepting payment for reviews and conflict of interest?
Well that is an interesting point and I think that if you felt there was a conflict of interest you would have a reason for not accepting the review because it could be said that the reviewer would not be very objective. Clearly if you are working for another group that is working in a similar area, there is a potential conflict of interests.
But what if you perceive there is no conflict and someone else does? I think that you have to discuss that with other editors. Editors round here how would you feel about that though?
Yes, this is an interesting discussion. I do think that payment for reviewing is not very wide spread, and I don't see it as actually becoming very common except possibly for the fee-for-service, for example open-access environment I think that it may come in to play (when authors are paying for manuscript submissions). It certainly is a form of professional service just like reviewing grants, writing letters of support for individuals and as Michael has mentioned there are other ways I think that we can provide recognition. I think that this is very important because it does appear to be a thankless job often times. At this meeting on Friday we have both a reviewer and editorial reception and we publically recognise those reviewers who have carried a heavy load in terms of the reviewing process. Also it is probably on a weekly basis that I am writing letters of support for those reviewers for their 10-year promotion process, to recognise that these individuals have performed reviews.
As Michael has mentioned with his journal we have a quality assessment, so every review gets what is called an R score. It is a pretty traditional combination of the quality and the timeliness of the review. Those are the most important in our overall assessment of a reviewer. I do like this point of those individuals who submit articles to the journal should expect that they would also go into the reviewer pool. Many of you have been asked to review manuscripts based on those criteria. But we look at those very closely because we know the matrix of timeliness of review and getting quality reviews is critically important from the vantage points of the author and the journal's reputation looking to continue to enrich that pool.
Regarding paying reviewers, I think that what is happening is that the journal editors and the section editors and with JDR, all the associate editors, make the final decisions on the large majority of manuscripts. We have to do heavy triaging of manuscripts because we know of reviewer fatigue. We cannot send every manuscript we receive out for peer review and with JDR, approximately 50% of the manuscripts are returned without external review. The editors make a decision based on impact or potential methodological issues. We direct the authors to another type of journal because we do understand it is such a limited pool of reviewers (that are not able to review all of the papers we receive). And so the only context that we have discussed paying reviewers, and I was an associate editor for a journal that paid biostatisticians in particular, and they are almost functioning like an associate editor in a way because of this. It could be viewed as a conflict of interest if you were just a hired gun to review. We have had some of these models where there is a very limited pool in certain areas. All of our clinical trials and systematic reviews are reviewed by biostatisticians and clinical trialists. Right now, in our current environment, we have had some discussions at the board level for you know a very modest stipend because no one is really doing it for the money. But you can't ask someone to review 10 papers a month. We do try to never give a reviewer more than one paper at a time, we sometimes abuse the biostatisticians in that regard but most everyone else we try to calibrate that.
Unknown member of the audience
Let me throw something else into the midst, if we demand financial disclosure of individuals that submit, so if you now receive a thousand dollars, is it ethical not to disclose this in the same way, as the sponsorship of a study by someone? There is financial interest because the reviewer is still anonymous and you still get paid, is that ok? So as a reviewer would you be willing to have your name associated with review published with a disclosure that you were paid to review it?
I think that it is a very interesting point. For openness and transparency it would be best that the name of the reviewer was associated with the paper and the fact that they have been paid for the review. This is what governance and transparency would suggest.
Steve Rosenstiel (Journal of Prosthetic Dentistry) (industry)
I think that if we can get closer matches between the peer reviewers and the subject, that certainly helps and as a reviewer and as an area that I am working in it is an easier review, it is probably a more accurate review and I know that Elsevier has nice software which helps us get a list of individuals that you might not know personally, might not have reviewed for your journal before but they are a very close match to the topic that is being reviewed, and I think that is something in the future that we might be able to get better peer reviewers even if they are not the familiar faces that we are currently using.
Yes I think that a lot of us will have that, we have got little lists, I know that I have, of particular people who I know to be reliable and that have their particular topics of knowledge and I would have thought that, most of you must do that surely rather than just a random selection of people.
Steve Mason (GSK)
Just a philosophical question really. Do you think that you might arrive a situation where you actually have professional reviewers who by virtue of their career and development have then just simply become a professional reviewer on a schedule, on a quality. And I think that you know that the question was raised in terms of 20 years’ time whether that would be a future stage that could potentially arise?
Well I think that could be a way forward and certainly people have recently retired who aren't totally disenchanted with dentistry and want a break from the golf or the garden. It might be a thing that they could do along the way of life, and we are all living longer. Of course the dangers is: Are they keeping up to date in their area and this is the problem isn't it. I mean when you stop doing clinical dentistry all clinicians say ‘oh he is a dry fingered dentist’ but actually he might be in a position that he can read more as a result and be more up to date. So it is an interesting development, and I think that could happen, yes you have got a point there.
Gordon Proctor (Archives of Oral Biology)
You mentioned the Elsevier software and I was going to make the same point. It provides a very good match between reviewer, potential reviewer, people who you may not have heard of but who are publishing in exactly the right areas for the sort of paper that has been submitted. On that basis I don't see us going to a situation where we will be paying the reviewers in the future. The whole review process would become democratised as there are lots of potential reviewers out there. The papers you generally have a problem with are those which don't quite fit into a clear category and maybe a niche, that could be a problem sometimes. Generally there is plenty of choice of potential reviewers. You just have to give them a try, people who have submitted their own papers and I guess once you have tried them and it has been a poor review then you don't go back there again.
I think that it is a very good point and it is something that we look at in terms of it is a key point if you have, you know if there is an interesting paper, you know those of us involved in science typically will want to review it, if it is very interesting, especially if you are active in the field you can benefit and learn in the process and I think that is one of the most valuable aspects. We tend to see the papers just in a rough estimate, those papers that tend to be more impactful people may agree to review very quickly and then on the opposite side if we cannot get a reviewer and I was just talking to an associate editor, they had gone through 12 individuals and so if these are experts who don't want to review it then those papers probably should just be triaged. So it is a balance, if it is pulling teeth to try to get reviewers, maybe this is not a strong paper for the journal.
There is another aspect, as Peter was saying about teaching critical appraisal to post grads. We are all doing this now and so that way is the way forward I think that we are actually training people to review because a lot of us (holding eyes on the audience) were just sent a paper and it was presumed that our expertise was such that we could get on and review it. So that might well be a helpful way forward, I don't know if anybody has any comments on this idea?
I think Ken if I could just come in, certainly on the British Dental Journal we do a lot of the positive sounding things that have been mentioned in the room already in terms of CE or CPD credits, we don't need to do that because our General Dental Council (the national regulatory body) recognises reviewing as a CE activity. I think that there is a parallel with tennis linesmen and umpires isn't there, I think that years ago we had this debate about whether it should be amateur people who judge the lines at Wimbledon and now I think that they are all professional, am I correct? Are there tennis fans?
Well actually I can help you there because I have friends who have been referees, umpires and line judges and some are professional but the majority are actually still amateurs who have had intensive training. There is a hierarchy and pecking order and you don't get to judge the lines at Wimbledon until you have actually worked your way through the minor and lesser tournaments. In short there is formal training.
Right, well thanks, that is, I am sure Andy Murray will be very assured.
The formal training was the point that I was going to make. None of this is an exact science and in fact science itself isn't exact. The measurement is exact but what isn't exact is the interpretation of it and that is actually what we are dealing with when publishing research. We are dealing with somebody's opinion on what it is they have just observed and measured. I was invited to a celebration two or 3 years ago for the Cochrane Foundation to do my usual thing of introducing the proceedings and making a few sideways remarks. I said the thing that amuses me about Cochrane reviews is that they start with a particular subject and find that there are 350 papers on this subject. Then they apply their really strict criteria and that reduces it to 104 and then they go through and look at it again and it reduces it to 76, and then it goes down to six and then they read these six and they say on the basis of this they say what we need is more research! And everybody in the audience chuckled merrily and said ‘oh it's Stephen taking a sideways look at things again. ‘Thank you very much Stephen’. Now over to our first panel of the day for orthodontists’ The orthodontist came on and said well we started with 376 papers, and then we reduced it to 17 and then we reduced it to three and our conclusion was that there wasn't enough research to make a decision one way or the other and so we need more research. And nobody laughed. They were like yes that is very true, that is very true.
So none of this is exact and it is not exact on the side of the people writing, doing research and writing it. It is also not exact on our side either. Really what we have been discussing today is how we can make checks and balances to make sure that what we publish is about the best it can be both in terms of the quality of the research and its relevance and also the things that we have done in order to try and make it publishable, useful, readable and so forth. I don't think that by paying referees, by having all these declarations that you are ever going to get to the truth of conflict of interest, where people just frankly don't like this author because once they pushed into the queue at the salad bar in Selfridges or something. You are never going to find that stuff out, it is human nature. I enjoy the novels of Agatha Christie, particularly Miss Marple detective novels. What we are talking about here is detection as well, investigation, based a lot of her observations of what went on in her English country village. It goes on in my English country village. It is always the same people after a council meeting who put the chairs away. There are 500 people in the village but only three put the chairs away. It's the same in our world. There are 500 people who know all about the histopathology of gammaglobulin but only two will ever review papers for you. So again it is about human nature. We are never going to get it absolutely right but we as editors and publishers need to make sure that what we do is as right and as balanced as it can be and I think that all these things that we have been talking about today add into that mix and it is terrific that we are able to talk about them openly and then be able to apply them with the various techniques and methods that we have got.
Well Stephen thank you very much for that, I am going to move us onto another topic that you have introduced: the veracity and integrity of the author, publication and publisher. We have been talking about the problems with plagiarism, is it a growing problem? Can we improve the detection software? One of the universities where I have an honorary chair, all the assignments that our students do go through plagiarism software as a matter of course. Does that happen at your universities?
They have another software package and I have been able to avoid being a detective!
I think that it might be useful just exploring this whole business about fraud and publication, plagiarism and all the other difficulties there, and Rex you have made some opening comments on that very topic earlier on, so what does the audience feel about this? Do you think it is a growing problem?
Michael Glick (JADA)
I can only tell what we get, I probably read around 1300 manuscripts a year and it is very clear, without mentioning countries or cultures, that in some cultures plagiarism is looked upon more as something positive than negative, let's put it that way. Some countries where I had 90% of submissions are plagiarised and the question is then how much identification, how much overlap is considered plagiarism? Some people say well you know copy you can copy methods, how can I write it any other way, this is plagiarism. Well maybe that is copyright rather than plagiarism, so I think that we have enormous problem which is going to have even more problems in the future of plagiarism, yes it is terrible, it is terrible.
I agree with Michael, except that I see it more as a collision of academic cultures. I am sorry to sound so politically correct about this, but I have a lot of students from educational cultures who do this a lot.(comment from audience and laughter).
As it is a collision of cultures, as soon as we start to use the word ‘plagiarism’ it is a form of imperialism really that we are imposing a set of values on people, and I don't think that helps us solve the problem. Now maybe our standard is what they aspire to but it is really very difficult and I think that we would have to think of seeing it in other ways than we are right and they are wrong. Instead, perhaps we should see it as an educational issue and that we need to harmonise standards.
Can I respond to that, I have had a discussion with an editorial board a couple of years ago about an Independent Review Board (IRB), and I said can we accept an IRB that is different to ours if it comes from another country, and it is exactly the same thing, the same issue.
And I got a publication from a country that didn't mention an IRB or its equivalent (an Ethics Committee) and I rejected it and then I subsequently received the review board approval after this rejection with the signature of the university. This was acceptable to the university concerned but was it ethically acceptable? I think that if they publish in our journals then we set the standard and that is it. It is our definition, unfortunately.
It is really just a question of attribution isn't it, I mean copying is copying, if you are in school and you look over and copy something that your neighbour has written it is wrong. However, in some cultures it is seen as ‘homage’!
If you miss out a pronoun in some countries in your final exam, you know from the original text, you will lose marks. It is homage.
Jane Ryley (Elsevier)
We have had some recent discussions, clearly with the advent of all these electronic submission systems it has allowed for autosubmission of papers to become much more global, we are seeing a lot of it (plagiarisation) over all of our dental journals. A couple of recent examples, just to echo what you said, is that it is considered by some of these students if they are called on in papers, they consider it a form of flattery and they are paying homage to their professors and respect the academics within their specialities, so I think that it is an ongoing problem.
It is, it is very difficult but we hope that with the passage of time and internationalisation that we can improve things there because it is a big problem. Can I move us on then and I think this topic here is very relevant because this is an IADR meeting and this is where it could happen perhaps. It is an agreement to standardise research protocols to make studies more comparable. I would look perhaps to IADR and the different interest groups there to think about this but what do you think about that idea?
That's extremely difficult to do. Anytime that I have been involved in some sort of consensus meeting on the development of guidelines we can never get people to agree on those sorts of things. So it seems almost impossible, and the examples that you give with evidence-based dentistry in trying to perform a systematic review and then meta-analysis, going from a gazillion papers and then you have three that fit within a specific protocol. Within the different disciplines of research methodology and education training programmes at universities within the educational systems, it may be possible, but I think for the editors and the journals to mandate this sort of thing it becomes very difficult.
I think that I agree with you but I would say that epidemiology is one area where you can possibly note some progress and so that you are actually using the same systems for assessing disease in the population, and that is one area where we could do quite a lot more than we have been doing.
When it comes to epidemiological studies we now require the following of STROBE guidelines for observational studies, clinical trial registration, CONSORT guidelines and these have gradually gotten into many of the different journal systems requiring them and that does improve journal quality. But those guidelines also provide a lot of flexibility in terms of how the materials are presented. So guidelines have already been a positive step to address some of the quality-related issues that you are bringing up.
And looking to the next 20 years as STROBE and CONSORT have been developed in the last 10 years might there be something else happening. Is anybody in the audience working on anything like that at present that they would bring to us?
Man for South Africa
I think it is getting back to the idea of you know globalisation of information and how information could be useful outside of the United States or you know, trying to get a protocol for me sort of innovation from outside the mainstream where there are new ideas. Innovation can happen outside of that mainstream, and so trying to set this up again is not respecting the fact that new knowledge, new protocols can come from outside and I think that we have to be careful. Yes 90% of the journals come from the United States. However, 90% of the work probably does not come from the United States, and a lot of comes from the western world.
I think that your data are wrong. I remember reading Will's paper that he sent me and it is not 90% from the United States by any means. The percentage has actually declined has it not Will over the last 20 years? What was it, about 71%, 20 years ago and it is now about 30%. So there has been a big change. However, I think that you are absolutely right, any guideline can be very limiting to new techniques and new knowledge but the point is, is that the guideline is not a law, it is not absolute and if you don't follow the guideline as long as you give a clear indication of why you haven't followed the guideline then that should be acceptable but you have to give that explanation. This is the way to overcome the problem.
I think that is an excellent example of where judgement comes in rather than just is it black or is it white, does it fall one side of the barrier and what we are talking about again here is opinion, human nature, interpretation and therefore one would hope that the systems that we have got in place now, certainly in the British Dental Journal would actually recognise that here was something that didn't quite fit the mould. We would want to recognise something as different but not necessarily a bad thing by asking ‘is that actually where this particular branch of dental science is going?’ It is that classic thing of you don't want, as an editor, to be the guy that turned down the speaking movies, don't want to be the guy that turned down the paper that might have changed the whole of dental science. So I take your point absolutely that if we are too rigid about these things then we could be missing some very important research and data.
I am interested in the panel's views on what would be better than reporting protocols, I mean the number of times that you look at a review and a large number of studies have to be discarded purely because they haven't said what it was that they did and so you can't compare. And so even if there is not an agreed protocol for carrying out the study itself, as long as you reported exactly what you have done then that would be a step forward, what would the panel do?
It is getting better isn't it though? Stephen's jokes aside, things like CONSORT and STROBE have pushed quality forwards. With a PhD student planning studies now, we give them CONSORT and say ‘Make sure that your trial meets these criteria, both when you plan it and when you report it’. So to give an example, I have been involved in the Cochrane reviews of powered toothbrushes. In our first iteration, published about 10 years ago, there were only 40 papers and that represented all of humanity's work on comparing powered with manual toothbrushes. And yet in the next round there will be nearly as many again so that there have been really as many good quality studies in the last 10 years as there had been in the previous two millennia.
Brenda Heaton (Boston University)
I think that we have come a long way since STROBE and CONSORT reporting guidelines but I think that there could be a higher level of rigour from the review process or even on the submission process and be more clear on design. You see so many studies, case control studies that are not case control studies by design which makes things like reviews very very difficult and not to mention better analysis or other things. I think that is something very simple to be taken care of in the review process or the submission process to, in the same way we use biostatisticians use epidemiologists.
Bring a little bit more clarity to design because I think that limitations are under reported because of lack of clarity and the design for example and a lot of things get promoted.
Isn't that the real value of many systematic reviews? Not that they come up with a concrete result but that they find out the lack of data and maybe they should just try and emphasise more the standardisation of criteria. Anybody who has ever done one will realise that they throw away probably 10 times as much data as they include. The interesting thing is that somebody said that they had, the second time around in a study they found as many new worthwhile papers to include in a systematic review and I have had that experience too and I suspect a lot of people have. So maybe it is the first systematic review that needs to be done and then the worthwhile ones will be done later.
On the same topic and I think that you have brought out something very important, we don't tend to publish negative results, so if I do a systematic review I am not going to find it. I call this the evil dolphin theory, you know dolphins are wonderful animals because if you fall overboard and they save you, but there are these evil dolphins that take you out to sea and you drown and you never find out about those dolphins. And it's the same thing here, we do our systematic review but as publishers we don't publish negative results and we are never going to get to the truth. And I would challenge all of us to stop doing it.
It would be interesting to know how many people do now publish negative results because there is nothing wrong with negative results as long as the controls are adequate so that we know that that is the real result. It wouldn't be worth doing the experiment if there wasn't the possibility of having a negative result.
Yes my question is actually directed at Will. I know that you spoke at the ADA within the last couple of weeks regarding registration of clinical trials. Could you give us your thinking is on that.
Yes there was a conference held in Chicago a couple of weeks ago that Michael Glick organised to bring together some of the manufacturing communities and dental editors regarding clinical trial registration. Clinical trial registration is a requirement for the Journal of Dental Research and many other journals. So there is a consortium of journals in the early 2000′s that actually came together, the Lancet and the New England Journal, requiring it. The reason for clinical trial registration is to simplify things, to provide clarity and transparency on the design of clinical trials in terms of outcome measures. A priority defining primary outcome measures as x and y is designed to help promote patient safety because of unfortunate examples within the drug manufacturer community where the primary outcomes were changed.
That is one part of it. The second part is to help patients to become more involved in the process and so during recruitment some patients will enquire regarding the process. So we are doing it in terms of clarity and it helps the editors a great deal when this is done a priority to identify what those outcome measures are and then the whole series of secondary analysis can be done. We started implementing this, almost 2 years ago. It wasn't requiring a priority registration because I think that dental community is behind the medical establishment, but we are requiring clinical trial registration and we anticipate in the next year that it would be a priority registration at the time of the initiation of the study.
Michael you might want to add, as you compiled a lot of the conclusions from that meeting, anything else that you would want to include, I know JADA is also requiring clinical trial registration.
They felt that by doing this they would improve the quality of information and could they be excluded from this requirement and that is what the discussion was all about.
Well now we are getting near to the end of the session and I know that a lot of us are going to rush off to other things at half past one. There are three things which I need to say. The first is that to thank you so much for coming, we had an amazing turnout, a completely full room and people standing at the back. I hope that it has met your expectations. This was the third meeting that we have held at an IADR meeting for editors, associate editors, publishers and those interested in the publication process. I would like to float the idea, would you collectively feel that it would be a good thing to approach IADR to ask if we can have a regular session? I think that if we do then I will go for this sort of time because later on in the programme it gets incredibly difficult as there are so many clashes with others things. Is that the feeling from the audience, would you like that to happen? Anybody think that it is not a good idea? OK well I will chat with the panellists and we will get something together. We hopefully have all written down your names and email addresses because it is always a nightmare trying to find the list from a year ago or 2 years ago and make sure that everybody knows about it other than just seeing it in the programme. If you have got any ideas I will leave my email address on the screen. Do please send me an email, tell me what you think, what your ideas are and we will try and take this forward. Now that red arrow (computer screen cursor) has poised over the little thing which said funding and that is to remind me, we have a sponsor for today because the cost of recording all this and all the technical things is quite substantial. A big thank you to Stephen Hancock's organisation for providing the sponsorship and as they say, used to say in American television at the end of a programme and now a few words from our sponsor.
Thank you very much Ken and I echo all of Ken's words really, I am taking my British Dental Journal hat off now and putting my own publishing company hat on, Stephen Hancocks Limited, and if I can just get you to take one of these and pass them around, which I think and hope is very relevant to IADR. It is called World Dental Posters and it is at www.worlddentalposters.com. Basically it is an open-access site for anybody who has had a poster presented at a peer reviewed meeting, we get back to that concept again of peer reviewed meeting, to upload their poster. The first 100 posters are free and thereafter it will be £25 or $40 per poster, so again a modest sum I think from authors. The posters will go onto the open-access site, they will be listed according to author's name, title and key words and they will be up there for as long as the author wants them to be up there, there will be no further charge of any sort. So the idea really is to get as much science out there as possible to as many people as possible as economically as possible, the idea came from my partner who said what happens to all those old posters that we see at IADR and all the other meetings around the world? The answer which came back from many poster authors was, well they end up rolled up in the back of our wardrobe or in the corner of the department's cupboards or tripping up our secretaries or whatever, so to actually have a longer life of these posters would be a terrific idea. So that is my message and I am delighted that my company was able to help to sponsor today's meeting, do please go on line, have a look, www.worlddentalposters.com and we hope to see lots of your posters on the site very soon. Thank you very much.
Well Stephen thank you very much for all the sponsorship and also a big thankyou to our technician there, hopefully we haven't caused him to many problems and hopefully he is going to give me the disc and so there is a clear record of what we have said and we will try and publish these proceedings, thank you all very much.