Commentary on Bitton et al. (2010): ‘Yes, but will it work?’—the need for a formal evaluation process for the Framework Convention on Tobacco Control


‘Yes, but will it work?’ is a common rejoinder when new ideas, concepts, and even better mousetraps are introduced. The paper by Bitton et al. in this issue of Addiction[1] asks this question of the World Health Organization (WHO)'s Framework Convention on Tobacco Control (FCTC), the world's first public health treaty and, even more specifically, of Article 14 of the FCTC, dealing with the development of comprehensive national programs for the treatment of tobacco dependence.

The FCTC provides a detailed roadmap for this signal achievement in global tobacco control and disease prevention. It came into force in 2005, and Article 14 will be considered for adoption in November of 2010 in Punta del Este, Uruguay, as the 169 countries which have ratified the treaty meet to consider specific Articles and provisions of the FCTC, such as development of cessation treatment guidelines and further work on smuggling, and to assess progress in meeting successfully the goals of the treaty.

It is this issue—assessment of success—which Bitton et al. address and for which, 5 years after the FCTC became legally binding, they lament that we still must be asking the question: ‘Yes, but will it work?’. The reason that this question is still outstanding is that, despite the significant efforts, and ultimate success, of the WHO and the negotiating Parties in crafting a very strong treaty, no formal mechanism has been put in place by the COP to monitor and rigorously evaluate its effects.

As Bitton et al. note, efforts have been undertaken, in four separate surveys, to assess the entire FCTC and/or the cessation treatment aspects of it (Table 1). Two of the surveys are ‘formal’, i.e. sanctioned or conducted by the WHO and by the Conference of the Parties (COP—the name for the decision-making body, and its meetings, representing all nations which have ratified the FCTC) and two are ‘ad hoc’, i.e. conducted by elements of civil society. However, they also note that these surveys have different goals, different questions, different sources of respondents and different time-frames and, perhaps most importantly, unstable sources of support. All this adds up to a chronic inability to answer the ‘will it work?’ question regarding the FCTC or its specific provisions, such as Article 14, with a sufficient degree of validity and reliability. The inability to answer this question adequately will make it exceptionally difficult, in turn, to improve the treaty as it progresses

Table 1.  Brief summary of the four Framework Convention on Tobacco Control (FCTC) and/or Article 14 surveys.
1. World Health Organization (WHO) Tobacco Free Initiative MPOWER Survey (
  Officially sanctioned by WHO
  Has stable source of support (Bloomberg Initiative)
  Covers most countries
  Covers six important FCTC policies
  Includes plan for repeated surveys
  Primarily government respondents
  Insufficient civil society input
  Missing data/lack of detail
  Does not address the full range of FCTC provisions
2. FCTC Party Reports (
  Officially sanctioned by the Conference of the Parties (COP)
  Reports provided on a regular basis
  Primarily government respondents
  Insufficient civil society input
  Missing data
  Inconsistent report formats
  Does not provide objective evaluation data
3. Framework Convention Alliance Shadow Reports (
  Mix of government and civil society respondents
  Provides counterbalance to officially sanctioned surveys
  Covers a wide range of FCTC provisions
  Inconsistent quality
  Unstable source of support
4. Raw Treatment Survey (
  Detailed scoring system
  Good model for Article-specific evaluation
  One-time survey
  Small, convenience sample

Evaluation challenges are not unique to the FCTC—many treaties, whether regarding trade, arms control, protection of the environment, etc. find that measuring success of the treaty is an afterthought, often underfunded, and consisting of a hodgepodge of sometimes disconnected survey instruments and questionnaires [2]. As with the FCTC, the Parties to these treaties are usually required to provide regular reports to the treaty Secretariat, but these are progress reports, as in the case of the FCTC, and not true measures of how well the treaty is succeeding in reaching its goals.

Therefore, despite good intentions and reasonable efforts, the ‘Yes, but will it work?’ question remains open for the FCTC. However, this treaty is too significant an undertaking and too important for global health—after all, more than 5 million will die this year alone from tobacco use—to allow a ‘let's hope for the best’ approach to its evaluation, an outcome favored by no one involved in the FCTC process.

So, as Lenin said so eloquently in 1902 when faced with similarly challenging problems, what is to be done? Several suggestions can be made:

  • • The Uruguay COP should make a clear and unequivocal statement in support of a formal process for the rigorous, well-funded, regularly conducted evaluation of the FCTC. There should be both officially sanctioned and independent arms of the FCTC evaluation. The officially sanctioned arm could be an expanded version of the WHO Tobacco Free Initiative (TFI) MPOWER Survey (a 32-question instrument to measure implementation of six tobacco control policies [3], supplemented by the COP's progress reports. The independent arm of the evaluation could be an expanded version of the civil society-driven Framework Convention Alliance [4] Shadow Report system, if sufficient independent funding can be identified, and/or a series of independent surveys, such as that described by Raw et al. [5] to evaluate Article 14 of the treaty;
  • • The FCTC Secretariat (the WHO-housed administrative arm of the Parties to the treaty) could improve the usefulness of the Party report system by agreeing on a fixed format; requiring responses which tie more closely to outcome, rather than process, measures; and reducing respondent burden by substituting more closed-ended questions; similarly, the MPOWER survey could be made even more valuable by expanding its reach beyond the six areas it now covers, given the breadth of issues addressed by the FCTC and international tobacco control needs [6]; and the FCA Shadow Report system could continue to make a significant contribution by, perhaps, focusing its reporting upon specific provisions of the FCTC implementation, such as Article 14 and working through such existing, valuable resources as, the only website offering science-based information and advice about tobacco dependence treatment in multiple languages;
  • • Consideration could be given by the COP and its Secretariat to commissioning separate, and ongoing, evaluations of Articles—such as Article 14—as they are adopted, as this would allow more precise measurement of these vital treaty provisions;
  • • Consideration could be given by the COP, its Secretariat, and TFI, to consolidating funding for the evaluation of the treaty from the many, although dispersed, supporters which are currently funding aspects of the evaluation, and to seeking new sources of support—including the pharmaceutical industry, if proper controls can be established; and
  • • The Parties should commit to a regular review of the evaluation data—both the official and the independent arms—and to making adjustments in the implementation of the treaty based on the evaluation data.

Certainly, as Bitton et al. note, there are positive and encouraging signs in their data regarding the potential for a strong evaluation of the FCTC. They applaud the impressive support for the current evaluation surveys from such diverse parties as the FCTC Secretariat, WHO's Tobacco Free Initiative, the Bloomberg Initiative, the Framework Convention Alliance, the Society for Research on Nicotine and Tobacco, the International Union Against Cancer/Globalink, the Swedish International Development Agency, Cancer Research UK, ASH/US, the Open Society Institute, the Society for the Study of Addiction and the International Non-Governmental Coalition Against Tobacco.

The support of the Bloomberg Initiative in working with TFI to conduct the MPOWER Survey and the FCA in its efforts to provide an independent evaluation through their Shadow Report system have been particularly helpful. Both will be vital in assessing the success of Article 14, should it be adopted at the Uruguay COP. Both the COP and its Secretariat and the WHO–TFI understand and support the need for a strong FCTC evaluation, but have been hindered in their efforts by limited funding and, as noted above, a lack of commitment on all sides to a systematic approach to evaluation of the treaty, incorporating both officially sanctioned and independent arms.

The FCTC is a magnificent achievement, with the potential to save literally millions of lives—but in order to make it a living document, capable of casting its effects deep into this century, a strong, systematic evaluation plan must be put into place which will provide a definitive answer to the question ‘Yes, but will it work?’.

Declaration of interest