Faced with the concern that an increasing number of airway management devices were being introduced into clinical practice with little or no prior evidence of their clinical efficacy or safety, the Difficult Airway Society formed a working party (Airway Device Evaluation Project Team) to establish a process by which the airway management community within the profession could itself lead a process of formal device/equipment evaluation. Although there are several national and international regulations governing which products can come on to the market and be legitimately sold, there has hitherto been no formal professional guidance relating to how products should be selected (i.e. purchased). The Airway Device Evaluation Project Team’s first task was to formulate such advice, emphasising evidence-based principles. Team discussions led to a definition of the minimum level of evidence needed to make a pragmatic decision about the purchase or selection of an airway device. The Team concluded that this definition should form the basis of a professional standard, guiding those with responsibility for selecting airway devices. We describe how widespread adoption of this professional standard can act as a driver to create an infrastructure in which the required evidence can be obtained. Essential elements are that: (i) the Difficult Airway Society facilitates a coherent national network of research-active units; and (ii) individual anaesthetists in hospital trusts play a more active role in local purchasing decisions, applying the relevant evidence and communicating their purchasing decisions to the Difficult Airway Society.
The Difficult Airway Society (DAS) is one of the largest specialist societies in anaesthesia in the UK, representing ∼3000 members. Through its main Committee and annual general meetings, DAS supports research and audit, formulates clinical guidelines and provides advice to the profession as a whole on a range of matters relating to airway management (see http://www.das.uk.com). Several recent articles [1–3] have explained how novel types of airway management devices were appearing in hospitals with apparently little or no prior evidence of their clinical efficacy. These concerns caused DAS to assemble a special working party – the Airway Device Evaluation Project Team (ADEPT) – which included co-opted representatives of industry and anaesthetist-scientists with extensive experience in airway-related research.
The need for objective evidence that devices perform effectively is both intellectual and practical. Like all other clinicians, anaesthetists need to know that the drugs and equipment they use actually work, and the appropriate method by which to establish this is a clinical trial. Although some trial data may exist in preliminary form (e.g. industry reports), it is important for substantive evidence to be presented in an organised manner by formal publication in the scientific literature so that there is assurance of peer review, appropriate ethical conduct and statistical analysis, with the opportunity for readers to add comment via correspondence pages . The fact that no published trial is statutorily necessary before an airway management device can be marketed  means that, by definition, purchasing or using many devices currently lies outwith evidence-based practice.
Such intellectual considerations apart, there are many practical examples where devices have failed or performed poorly after introduction to the market. The replacement of re-usable products by single-use ones brought to light the reality that simply manufacturing a device to a broadly similar size or specifications did not guarantee similar outcomes (we are unaware of new single-use devices that actually performed better than their multi-use predecessors). Thus, a single-use bougie can be shown to perform extremely poorly in both clinical trials and manikin studies [6–8]. Single-use connectors for carbon dioxide monitoring marketed as ‘low dead space devices’ actually caused potentially dangerous airway obstruction, something that would have been apparent with a simple clinical trial . As previous authors have stressed, such post-marketing surveillance is a poor means of ensuring patient safety, especially when there has been no formal pre-marketing exposure to patients [10, 11]. It is not only anaesthetists, as users of the equipment, who need the prior evidence. Manufacturers of the devices also need to know early which ones do not work, so that they avoid wasted effort in marketing substandard products.
Previous suggestions have focussed upon changing the regulations that bring products to market [1, 2], but these require major changes in legislation across the European Union to implement. The focus of ADEPT was therefore very different: to provide guidance to anaesthetists and other healthcare professionals in regard to the best approach to take when making purchasing decisions regarding airway-related devices, rather the regulation of the marketing of these devices.
The key questions
When making a purchasing device that is based upon evidence, a relevant consideration is: what sort of ‘evidence’ is needed? This is an almost philosophical question concerning the nature of ‘proof’. But in turn, the answer to it informs the underlying issue of how to develop the infrastructure for obtaining this evidence. This also requires understanding of the research governance structures for clinical trials (see http://www.rdforum.nhs.uk/) and methods of procurement processes within the NHS (see http://www.dh.gov.uk/en/Managingyourorganisation/NHSprocurement/index.htm).
In this article, we explain the reasoning behind the strategy that ADEPT has finally adopted (which now constitutes DAS guidance to its members), so that readers can better identify with our conclusions.
What sort of ‘evidence’ is needed?
The scientific community regards the randomised controlled trial (RCT) as the gold standard for establishing efficacy of a treatment or intervention . Ideally a large, double-blind multicentre study, the RCT’s strength lies in rigorously eliminating bias. Patients are randomly assigned (thus minimising any consistent differences between their characteristics) and are managed identically (ideally by blinded researchers) such that the only consistent difference between the groups is the treatment or intervention of interest. The data are also analysed in a blinded manner. This design gives confidence that any differences are real, uncontaminated by bias, confounding factors, or chance.
Since the 1970s, the rise in so-called evidence-based medicine has generated classifications of the ‘strength’ of evidence, in which a meta-analysis of RCTs lies in the highest class, with the RCT just below (see: http://www.cebm.net/). Weak sources of evidence (i.e. those contaminated by observer bias or chance), such as individual case reports or expert opinion, lie very much lower in the hierarchy (Table 1).
Table 1. Evidence-based medicine hierarchies of evidence. The strongest is Level 1a; the weakest is Level 5. Adapted from the Centre for Evidence-Based Medicine at: http://www.cebm.net/index.aspx?o=1025). Notwithstanding several other classifications of types of research evidence, this table represents a useful summary of categorisation.
Level of evidence
Type of study
RCT, randomised controlled trial.
Systematic review of RCTs
All-or-none study (i.e. when all patients died before the therapy became available, but some now survive on it; or when some patients died before the therapy became available, but none now die on it)
Systematic review of Level 2b cohort studies
Single cohort study or low-quality RCT
Outcomes studies that investigate outcomes of healthcare practices using epidemiology to link outcomes (e.g. quality of care, quality of life) with independent variables such as geography, income or lifestyle, etc
Systematic review of Level 3b studies
Single case-control or historical-control study
Case report or case series
Expert opinion or ideas based on theory, on bench studies or first principles alone
There is little doubt that RCTs and their meta-analyses constitute very persuasive evidence. However, RCTs are not the only source of evidence and it has gradually become apparent that perfectly pragmatic decisions can be made on ‘lesser’ levels of evidence [13–18]. This argument has been articulated most clearly by Professor Sir Michael Rawlins, Chairman of the National Institute for Health and Clinical Excellence (NICE), in his Harveian Oration of the Royal College of Physicians . Rawlins was emphatic that the enormous and growing costs of RCTs (in both money and time) did not always justify the information gained. Akin to arguing that RCTs were sometimes sledgehammers used to crack a nut, he suggested that particularly for questions of a pragmatic nature, several alternative methods of generating evidence were equally valid. Among these are non-inferiority trial designs [20, 21] (which can use historical controls or registries when the data are stored for general use ) or non-randomised trials which are better at detecting harm (which may often be the main interest in many airway device-related studies) .
The high level of evidence generated by an RCT certainly resolves all doubts pertaining to a scientific question but for a pragmatic course of action, the lack of gold standard evidence should not paralyse all decision making . The real question is which lesser levels of evidence are useful. The psychologists Twersky and Kahneman distinguished human decisions made on optimal evidence vs those made on only satisfactory evidence . The second – also termed ‘bounded rationality’– is a process constrained by limitations of time, resources, effort and expense, etc , and it seems entirely appropriate to employ in decisions about selecting airway devices. In deciding which level of evidence is ‘satisfactory’, Table 1 can help. After discussion, consensus and using first principles, ADEPT identified some levels of evidence as falling well below any minimum requirement for justifying our decisions. Thus, the lowest ranked (Level 5) consists only of expert opinion . The next lowest (Level 4) consists of case reports with no identifiable control group or clear endpoint. It was not possible to construct an argument that these were satisfactory.
However, higher levels of evidence were, at least to some extent, justifiable. Case-control studies at Level 3b can be adequately powered (e.g. using the Poisson distribution for observational data ) and (as discussed above) data from historical controls can be used in a non-inferiority design to set the margins of acceptability. Traditionally used to monitor the progress of disease, such study designs can also be adapted to examine interventions such as use of airway management devices. Importantly, because it has a control or reference arm, Level 3b is the minimum level of evidence that can be subjected to a systematic review, which in turn helps yield Level 3a evidence. Level 3b can thus be regarded as the minimum ‘quantum’ of evidence that can be assimilated into a wider evidence base . Level 2 and higher levels are of course acceptable and where an RCT is performed, its results will undoubtedly be useful. To be pragmatic, any minimum level of evidence that we select must be a point of balance between what is achievable and what is meaningful. To illustrate this: Level 5 is easily achievable but barely meaningful; Level 1 RCTs are difficult to achieve but very meaningful. For the reasons given, Level 3b seems an appropriate point of balance.
The value of evidence is always a subjective judgement (i.e. some people are satisfied with low levels of evidence, but others remain sceptical despite high levels) . There is no evidence that confirms Level 3b, rather than another Level, is an appropriate minimum level of evidence, but three notes are supportive of ADEPT’s conclusion. First, there must exist some level of evidence that can be regarded as minimum. Second, there exists no justification for Levels 4 and 5. Third, currently the statutory level of evidence required before an airway device can be legitimately marketed is lower than Level 5 (i.e. no published report at all) . Therefore, selecting Level 3b as a minimum represents a huge advance in the evidence required.
ADEPT Guidance on selecting airway management devices
Appendix 1 gives the full text of ADEPT’s conclusions, in the form of DAS’s guidance to its members. If widely applied and followed by DAS members who sit on hospital procurement committees, this Guidance would automatically exclude all those devices that Cook  and the Centre for Evidence-Based Purchasing [28, 29] found were unsupported by any evidence; ADEPT and DAS believe such exclusion would be proper and safe. The ADEPT Guidance does not mean that devices supported by this minimum level of evidence are automatically acceptable. Individual anaesthetists/purchasers (who remain ultimately responsible for making the final decisions) may quite reasonably require much higher levels of evidence than Level 3b. Therefore, Level 3b evidence is a sine qua non: a necessary but not of itself sufficient criterion for equipment selection. It is also very much a starting point: as the culture within the specialty changes, ever higher levels of evidence might be expected.
The ADEPT Guidance is perhaps unusual in another sense. Generally, professionals might expect their societies to issue advice on how to act: e.g. what steps to take when there is a cardiac arrest, or manoeuvres to employ in a failed tracheal intubation, etc  (i.e. in the manner of standard operating procedures). Or, they might expect specific advice that sets limits of acceptable device performance (e.g. ‘Only accept a device if it succeeds in > x% of cases or causes harm in < y% of cases’). Or, they might have hoped that DAS would stipulate clearly which specific devices to use and which to avoid (in the manner of ‘blacklists’ or ‘white lists’). For very good reasons (discussed further below), the ADEPT Guidance is different. It offers advice to support members’ personal decision making. Anaesthetists can find themselves in many different situations, from managing a failed/difficult tracheal intubation to sitting on a procurement committee. Just as ‘remember oxygenation first’  is useful to recall in the former, ‘remember the minimum level of evidence needed’ is a useful reminder in the latter. Furthermore, in an era when (both anecdotally and in our experience), trusts are making equipment purchasing decisions solely on cost , the ADEPT Guidance helps restore the place of ‘evidence’ as a proper factor and lends some professional support to those members who refuse to use devices that lack a minimum evidence base.
The ADEPT Guidance gives considerable latitude to the judgement of the individual anaesthetist. It does not stipulate that any published evidence must take a specific form (e.g. in its trial methodology) or that the publication must be of a particular type (e.g. conference abstract vs full paper vs letter, etc). Nor is there any restriction on the journal type or quality (e.g. a high vs low impact factor journal or an English- vs foreign-language publication). It is also acknowledged by DAS that sometimes, it is not always clear whether a trial has attained the minimum required level of evidence. Even within Level 1 RCTs, there can be important differences in methodological quality between trials, hence the development of the Jadad score . Therefore, it will remain entirely at the discretion of anaesthetists how they assess the evidence presented to them. Some trust procurement groups will inevitably judge that the evidence presented is inadequate, whereas others will judge that the same evidence is sufficient to purchase the device.
Reconciling ADEPT Guidance with the Medical Devices Directive
The Medical Devices Directive is the relevant regulation that covers the placing of medical equipment on the market and putting it into service. . Its Annex 10 dictates that suitable ‘clinical investigations’ should be undertaken, but the terminology is sufficiently vague as to leave open to different interpretations core issues such as the role of scientific peer review and open access to any clinical outcomes data . Naturally, DAS would like to see both the wording of this Directive and its application to reflect more closely modern evidence-based practice, but DAS has very little direct influence, as the Directive’s contents require extensive negotiations with all international stakeholders and agreement of all member states of the European Union, before any amendment can be made.
However, DAS has exploited the fact that, although the Directive dictates what can be legitimately sold, it cannot have any jurisdiction on what should be bought. For this reason, the ADEPT Guidance is framed in a way that is aimed at fellow professionals who are prospective purchasers and users of the equipment, and not at regulatory authorities responsible for overseeing the placement of products onto the market. To put it in another way, just because a device is for sale, it does not mean we have to buy it. As users, we may apply whatever logic we wish in our purchasing decision and (rather than a decision based, say, on colour or cost, or on tossing a coin) DAS has chosen to apply a logic based on evidence.
Two recent suggestions are relevant, but notably these were concerned with changing how devices should be brought to the market (i.e. changing the Medical Devices Directive) and not concerned with offering advice to professional purchasers. Cook proposed a three-stage process before devices could be marketed . In the first stage, he proposed manikin studies to exclude major ‘obvious’ issues before exposing patients to any risk; the second stage should be a rigorous pilot study in patients to exclude major safety issues and ‘exclude inefficacy’. The third stage should be the key RCT against the current ‘gold standard’ device. Writing about supraglottic airways, Cook’s proposals could be widened to apply to all airway management devices, such as laryngoscopes, facemasks, bougies and other intubating aids, exchange catheters, cricothyroidotomy equipment, etc. He suggested that any alteration in the design of the device (including copies of existing designs) should entail restarting the process, with CE marking awarded after the second stage and marketing allowed after the third stage.
Wilkes et al. proposed a scheme whose main elements were: (i) a national ‘device evaluation centre’ (for co-ordinating assessments of airway devices and retaining a database of results); (ii) a national ‘panel of experts’ (for critically appraising of the evidence and establishing a ‘gold standard’ list of acceptable devices); and (iii) purchasing groups in each hospital trust (for selecting devices from the ‘gold standard’ list) . The authors recognised that because European legislation does not allow additional hurdles to be placed beyond those prescribed by the Medical Devices Directive, their proposed scheme to bring devices to market would have to be voluntary, albeit co-ordinated by national organisations such as the Association of Anaesthetists of Great Britain & Ireland and DAS . However, for any national organisation to specify by name which devices were ‘acceptable’ in the way suggested might indeed be construed as creating additional hurdles beyond that of CE marking.
As it is not concerned with how devices are brought to the market, DAS’s ADEPT Guidance has avoided any confrontation with the Medical Devices Directive. As purely professional advice with no statutory standing, it does not require major pan-European agreement of governments or regulatory authorities. Nothing in the Guidance seeks to recommend (or avoid) named devices and therefore, it creates no barriers to free trade. Consistent with almost all guidance now issued to medical practitioners, ADEPT makes explicit the need to base all professional decisions upon objective evidence to a minimum standard.
However, it is to be expected that anaesthetists will follow the advice from their national professional society. The consequence of widespread adoption of the ADEPT Guidance by the profession should help create an important infrastructure. First, we predict that if anaesthetists heed ADEPT’s Guidance, then they should logically reject all airway-related equipment that has not been the subject of a published, case-controlled trial to a Level 3b standard of evidence. They might (ideally) review all existing airway devices in their trusts and remove all those that do not attain the same minimum evidence base. Thus, in turn, it should logically become an incentive for industry to perform or support suitable clinical trials as early as possible, as these (rather than less evidence-based forms of marketing) will form the basis of persuasive arguments to the profession. Third, to assist industry to obtain this now-needed evidence, DAS itself will evolve into an organisation that facilitates a national network of research units offering a platform for the conduct of clinical trials.
Creating the infrastructure: the evolution of a DAS-facilitated research network
This process requires input from: (i) individual anaesthetists, especially those who advise procurement committees; (ii) industry; and (iii) DAS itself.
Input from anaesthetists: the importance of trust procurement committees
Anaesthetists in individual trusts interested in improving the quality of airway management will acquire new responsibilities: (i) to judge to which devices DAS’s ADEPT Guidance should be applied; (ii) to assess for themselves the level of evidence available for the devices in question; (iii) to reject/exclude from use devices that do not meet the minimum evidence criterion; (iv) to advise their local trust procurement committees; (v) for devices supported by minimum evidence, to conduct a local trial designed to assess whether the device is suitable locally; and (vi) to feed back to DAS the results of any local trial and of the purchasing decision made.
Procurement in the National Health Service (NHS) is complex but generally for airway devices, each trust has its own committee responsible for ordering and negotiating a price (i.e. there is no centralised NHS purchasing) . The procurement group (usually comprised of managers and finance officers) normally receives clinical advice to help ensure that equipment is acceptable to users (Fig. 1, ‘Old’). However, adherence to this process may be poor: Gregory et al. reported that the majority (∼55%) of UK hospitals did not use anaesthetists’ preference or advice as the main factor in selecting supraglottic airway devices .
Henceforth if anaesthetists adopt ADEPT Guidance (Appendix 1), a new process will result (Fig. 1, ‘New’). Whenever there is no published evidence relating to a device, it will be automatically rejected. Only for those devices judged to meet the minimum evidence criterion, will a local trial be useful to inform the final purchasing decision. Feedback of this decision will enable DAS to create a central record of which devices have been purchased, and also the evidence upon which those decisions were based. This database would acts as a resource (the collective ‘wisdom of the crowd’ ) for others facing similar decisions. Issues of safety often take longer to emerge and may require more concerted surveillance based upon such information.
In summary, Fig. 1 outlines the manner in which DAS would like to see purchasing decisions made; Appendix 2 outlines answers to questions anaesthetists may be likely to ask in the context of the ADEPT Guidance.
Input from industry: the importance of promoting clinical trials
We predict that industry will be incentivised by this culture change within the specialty to promote early publication of at least one clinical trial result to the requisite standard of evidence. It will therefore become in industry’s best interests to work with airway management specialists (as an extension of what it does now) to design and offer funding support for proper device evaluations. As the currency of marketing discussions with professionals will henceforth revolve around ‘evidence’, it is the higher quality evidence that will be valued more highly in any decision making, and this will itself create a further incentive to design trials yielding levels of evidence much higher than Level 3b. As objectivity is important, assessments independent of the manufacturer will also be likely to have higher value. Trial outcomes where the new device performs poorly will be as important to manufacturers as to clinicians, as a sensible company will wish to save on further investments in a product that is quickly established not to work.
Input from DAS: the importance of establishing infrastructure
The notion that specialists societies should transform themselves into organisations that facilitate ‘national research networks’ was one explicit recommendation of the Royal College of Anaesthetists’ Academic Strategy (Pandit) Report [37, 38]. Cancer Research UK is a very successful example of a professional society evolved into a large research network and it defines its strategy as having the twin aims of: (i) providing the right environment for cancer research; and (ii) providing the right people to conduct the research (see: http://www.cancerresearchuk.org/). The concept of creating collaborative ‘research capacity’ in this manner as a formal entity to achieve goals is relatively new to anaesthesia [39, 40].
To redress this deficiency, DAS is inviting all anaesthetists active in airway management research (including audit [41, 42]) formally to register an interest in working together within an inclusive national network . We expect that the most UK centres or departments with a track record of airway management research will form the primary active ‘research units’, together forming a core research capacity that will grow as new centres are recruited.
Second, DAS is planning to support a secretariat (by appointment of staff) to act as a single point of contact for industry and co-ordinate requests for clinical trials with the ‘research units’. Tripartite agreements about the feasibility, timescale and costs of projects between industry, DAS and the research unit will include DAS overheads (e.g. the costs to DAS of the secretariat, ethics and other regulatory body templates, databases, clinical trial registration and sponsorship); staff costs (e.g. research time for consultants or a research fellow or nurse); travel (e.g. for independent trial supervision or the costs of disseminating results at a conference); and publishing costs. Many individual centres and DAS already have a long and amicable history of managing such costs in relation to large annual general meetings, so the details of any legal agreements between the units and the DAS secretariat as to how this funding is managed should be a formality. These tripartite arrangements will also create distance between the industry funder on the one hand, and the anaesthetic researchers on the other, so that there is proper independence of the project. We do not anticipate prohibitive additional costs to industry (e.g. in the order of ∼£25–50K for a device-related research project of ∼12 months’ duration), recalling that these trials will often be designed to generate a minimum level of evidence. It is indeed desirable to keep costs low but equally, the real (i.e. ‘full economic’) costs of undertaking this essential work need to be recognised much better than they have been in the past. Any dividends from overheads will be used by DAS to increase its direct grant support for non-industry commissioned or ‘pure’ research through competitive grant rounds (via the National Institute for Academic Anaesthesia, NIAA). The partner-status of the NIAA with National Institutes of Health Research (NIHR) will further enhance this funding support via the Local Clinical Research Networks (see: http://www.crncc.nihr.ac.uk/).
It is our intention that individual units will be free to define the research question, develop the trial protocol and write the final paper; DAS will act simply as a facilitator, channelling industry requests to the relevant research unit (Fig. 2). This scheme does not preclude industry from approaching units separately and securing research agreements without DAS’s involvement. Equally, industry might choose to conduct trials abroad or in-house. Moreover, it does not preclude individual units from declining to join the DAS network, or offering to undertake the work for no cost, if they feel it is in their interests to do so.
Strengths, weaknesses and threats to the strategy
The main strengths of DAS’s ADEPT strategy are that it is ‘bottom-up’, professionally led and restores the primary role of evidence in decision making. It is achievable without recourse to upheavals in national regulations. It empowers clinicians and, in the way shown, helps DAS evolve into a more active, national organisation. The clinical trials are likely to be relatively low-cost and to yield results within a relatively short and relevant timescale.
However, one strength may also be a weakness: ADEPT’s decision to leave many judgements to individual discretion was a pragmatic one (Appendix 2), and arguably, there is not enough dictat from the centre. If the majority of anaesthetists are apathetic and/or disagree with ADEPT then they will continue to purchase and use devices that lack evidence. At this point, the strategy will fail.
Another threat is that some trusts may continue to ignore anaesthetic opinion, prioritising instead the financial considerations. So long as DAS is informed of such developments, it will be in a position to offer some assistance (see Appendix 2) and the DAS Committee is now prepared to take a more active role in the support of anaesthetists who are promoting high quality airway management in their respective centres.
Manufacturers who see themselves as partners in an effort to establish the true efficacy or niche of their devices will wish to engage in the network outlined in Fig. 2. However, some may try to use a non-evidence-based approach to marketing their products.
An interesting challenge will arise if adoption of ADEPT’s Guidance results in such a large surge in demand for device-related trials that it outstrips the UK’s research capacity in this field. The Pandit Report noted that only ∼10% of UK anaesthetic consultants had any research experience or interest, so this outcome is conceivable [37, 38]. However, because of the way in which the proposed infrastructure will engage with the national research governance framework, such high demand will in fact fuel growth in our research capacity. This will be most welcome and will help reverse the decline in academic anaesthetic capacity that has hitherto been such a concern .
It is important that DAS takes careful account of the costs of its proposed secretariat, and DAS Committee members (who currently give up their time voluntarily to help develop the society) may need to organise their efforts even more carefully so that they can give more commitment to what will be an expanding and busier organisation.
The Difficult Airway Society does not view these problems as insurmountable. We believe that DAS’s ADEPT Guidance is logical and designed to ensure that there is some minimum evidence base to inform decisions. The infrastructure that will evolve by adopting this Guidance will go some way to increasing the involvement of anaesthetists in clinical trials. The longer-term aim is to obtain ever higher quality evidence and not remain satisfied with the minimum level of evidence. It is hoped that the change in culture brought about by DAS’ ADEPT Guidance – especially the establishment of a national research network – will help attain that goal.
Acknowledgements and competing interests
The views expressed in this article represent the opinions of the DAS National Committee and of BAREMA (but not of any other individual professional organisations, hospitals or manufacturers). The strategy outlined has been endorsed by the DAS National Committee whose members also include: Dr Chris Frerk; Dr Ravi Dravid; Dr Jairaj Rangasami; Dr Alistair McNarry; and Dr Ravi Bhagrath. Further details, including a form for feedback of purchasing decisions, are available from the DAS website at http://www.das.uk.com. The authors are all members of ADEPT, and JJP, EO’S, AK and PG are members of the DAS National Committee. HC is Secretary of BAREMA, a trade association representing member-companies involved in the manufacture, sale and distribution of anaesthetic and respiratory equipment (see http://www.barema.org.uk/). JJP is an Editor of Anaesthesia and this article has undergone an additional external review as a result.
The Difficult Airway Society (DAS)’s ADEPT Guidance on the minimum level of evidence to support the selection or purchase of an airway management device (wording adapted from Safety Guidelines on the Management of Anaesthetic-related Equipment ).
DAS’ ADEPT Guidance on selecting airway-related equipment
All airway-related equipment under consideration must fulfil the minimum criterion that there exists for it at least one source of ‘Level 3b’ trial evidence concerning its use, published in peer-reviewed scientific literature
A literature review will determine whether such evidence exists or it might be provided by the company marketing the device. ‘Level 3b’ refers to the quality of evidence, in the hierarchy of evidence suggested by the Center for Evidence-Based Medicine (CEBM), from a trial (not necessarily randomised) with a control or reference arm (which can be a historical control). The Level 3b criterion recommended by DAS is a necessary, but not of itself sufficient, criterion by which to select a device for purchase. Therefore, when presented with a choice of several devices, each of which fulfils the Level 3b criterion, the local Medical Devices Management Group will need additionally to consider other types of evidence when making its final choice or recommendation
Organisations such as the former Centre for Evidence-based Purchasing (CEP, part of the NHS Purchasing and Supply Agency) and Emergency Care Research Institute (a US-based organisation) have published reports on various categories of equipment. For up-to-date information, purchasers should contact and visit other users, trade exhibitors and sometimes factories. In particular, previous performance by the manufacturer in terms of delivery, stock held, training provided and response to problems are rarely published, but details could be obtained from users in other trusts. Although none of this data reaches the minimum level of 3b evidence that DAS recommends all devices must meet, it is helpful (along with other types of published studies) in helping to choose between devices for which Level 3b evidence does exist
The local Medical Devices Management Group, including the anaesthetic equipment officer, should then carefully consider the options from the information available. Once a choice is made, a local trial may help ascertain whether the equipment meets local needs and whether it fits well into the local environment (e.g. in terms of sterilisation services, storage, etc) and may help local users gain familiarity with it (if they have not already). As many users as possible should be encouraged to participate and to feedback to the equipment officer on an agreed form (such forms are often a compulsory part of the purchasing procedure). Note, however, that the local trial cannot be published without formal ethical approval and therefore cannot substitute for the recommended Level 3b evidence referred to above
The local Medical Devices Management Group is asked to feedback to DAS the final purchasing decision and the evidence (i.e. the published papers) upon which that decision was made, so that DAS can develop a database of equipment and its supporting evidence, to facilitate the processes outlined above
Some likely common questions concerning the practical application of a minimum level of evidence to selection of airway-related equipment, with the Difficult Airway Society (DAS) response.
What is ‘Level 3b’ evidence?
In the formal evidence hierarchy (http://www.cebm.net), this is a case-controlled or historical-controlled clinical trial
Which devices will need a minimum level of evidence in support of their purchase or use? Supraglottic airways? Tracheal tubes? Laryngoscopes? Oropharyngeal airways? Bougies? Facemasks?
You need to decide for which devices you need to apply the minimum level of evidence. DAS encourages that use of every relevant device needs to be supported by the minimum level of evidence. Certainly for any device whose performance is questioned or controversial or where poor performance would influence clinical practice, a minimum level of evidence should apply
When should a device be considered ‘new’ for the purposes of applying the standard of evidence?
You need to decide when a device is ‘new’ or simply an exact copy of an existing device. For any device that is not an exact replica, there is potential for a change in performance and you need to decide whether new evidence is needed
Some devices are very established, yet there appears to be no Level 3b trial ever conducted. Should we remove these devices from use?
You need to decide whether to remove such devices from use. Certainly for any device whose performance has been questioned or controversial, a minimum level of evidence should apply
The manufacturer of an established device (supported by at least one Level 3b trial or higher) has made a modification to it that seems trivial. Should we now insist upon seeing evidence from a new trial?
You need to decide if the modification is in fact trivial. If you feel device performance could be affected, then it is reasonable to ask for evidence from a new trial
The manufacturer has provided evidence in support of a device: it appears to be a short letter (or abstract presented at a conference) briefly describing a trial in a foreign-language journal. Is this Level 3b evidence?
You need to apply your training in critical appraisal skills to assess the quality of evidence and make a judgement whether the evidence describes a case-controlled trial. You need to judge whether the evidence provided helps you make the judgement you are asked to make. After you make your purchasing decision, please communicate to DAS your decision, your reasons and the evidence upon which it was based (see the DAS website for the feedback form)
The manufacturer has provided clear evidence of the device studied in a Level 3b trial. However, the trial was studying an aspect of device performance that we do not think very relevant to the true purpose of the device, and despite its publication, we think the trial was of poor design. Should we purchase the device?
If the device has been studied in any published case- or historical-controlled trial to Level 3b evidence, then it fulfils the minimum criteria for purchase. However, you are entitled to judge that this trial did not answer the question you are interested in. You may feel that higher levels of evidence are, in fact, required. Recall, the Level 3b evidence is a necessary but not sufficient criterion for purchasing the device. After you make your purchasing decision, please communicate to DAS your decision, your reasons and the evidence upon which it was based (see the DAS website for the feedback form)
A new device has proved very popular locally especially after our local ad hoc trial, but there is no published clinical trial evidence relating to it. Should we purchase and use the device?
You are entitled to purchase and use any device with a CE mark. However, you need to be aware that you are using a device that falls short of the minimum evidence requirements stipulated by your specialist society. If device performance or your clinical practice later comes into question, you may be called upon to justify and explain why the minimum evidence you applied fell short of what DAS recommended. After you make your purchasing decision, please communicate to DAS your decision, your reasons and the evidence upon which it was based (see the DAS website for the feedback form) – in this case, that local approval superceded formal trial evidence
Local clinicians do not approve of a new device, yet managers have purchased it against our advice because it is cheap, saying it has a CE mark. Will DAS support us in representations with our managers?
Yes. If the device has not been assessed in a Level 3b trial, then DAS can write to remind the Chief Executive and/or Medical Director of our guidance on the minimum level of evidence that we have judged is needed. If the Level 3b criterion has been met, then DAS can write in more general terms outlining the importance of airway management to patient outcomes and reminding the trust that anaesthetists are the relevant specialists from whom proper advice should be taken. DAS will not, however, make any comment on any specific device(s)
Will DAS help us decide which devices are supported by Level 3b evidence, which trials constitute Level 3b and which do not or help us make purchasing decisions? Or will certain devices be approved by DAS?
No. However, as colleagues in different hospitals return to us the feedback forms (available from DAS website), DAS will be able to generate a database of purchasing decisions, devices and the evidence supporting these decisions. Colleagues will then be able to use this database as a resource to assist their own, independent decisions on purchasing and use. DAS will not recommend, approve or disapprove of any device(s)