Research Article
A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomes
Abstract
Calibration is one of the main properties that must be accomplished by any predictive model. Overcoming the limitations of many approaches developed so far, a study has recently proposed the calibration belt as a graphical tool to identify ranges of probability where a model based on dichotomous outcomes miscalibrates. In this new approach, the relation between the logits of the probability predicted by a model and of the event rates observed in a sample is represented by a polynomial function, whose coefficients are fitted and its degree is fixed by a series of likelihood‐ratio tests. We propose here a test associated with the calibration belt and show how the algorithm to select the polynomial degree affects the distribution of the test statistic. We calculate its exact distribution and confirm its validity via a numerical simulation. Starting from this distribution, we finally reappraise the procedure to construct the calibration belt and illustrate an application in the medical context. Copyright © 2014 John Wiley & Sons, Ltd.
Citing Literature
Number of times cited according to CrossRef: 41
- Isabelle Vock, Lisandra Aguilar-Bultet, Adrian Egli, Pranita D Tamma, Sarah Tschudin-Sutter, Independent, external validation of clinical prediction rules for the identification of extended-spectrum β-lactamase-producing Enterobacterales, University Hospital Basel, Switzerland, January 2010 to December 2016, Eurosurveillance, 10.2807/1560-7917.ES.2020.25.26.1900317, 25, 26, (2020).
- Anna Zamperoni, Carlotta Rossi, Stefano Finazzi, Paolo Del Sarto, Matteo Mondini, Giovanni Nattino, Daniele Poole, Guido Bertolini, Case-mix affects calibration of cardiosurgical severity scores, Minerva Anestesiologica, 10.23736/S0375-9393.20.14280-9, 86, 7, (2020).
- Virginie Lemiale, Stéphanie Pons, Adrien Mirouse, Jean-Jacques Tudesq, Yannick Hourmant, Djamel Mokart, Frédéric Pène, Achille Kouatchet, Julien Mayaux, Martine Nyunga, Fabrice Bruneel, Anne-Pascale Meert, Edith Borcoman, Magali Bisbal, Matthieu Legrand, Dominique Benoit, Elie Azoulay, Michaël Darmon, Lara Zafrani, Sepsis and Septic Shock in Patients With Malignancies, Critical Care Medicine, 10.1097/CCM.0000000000004322, 48, 6, (822-829), (2020).
- Nadir Yehya, Michael O. Harhay, Margaret J. Klein, Steven L. Shein, Byron E. Piñeres-Olave, Ledys Izquierdo, Anil Sapru, Guillaume Emeriaud, Philip C. Spinella, Heidi R. Flori, Mary K. Dahmer, Aline B. Maddux, Yolanda M. Lopez-Fernandez, Bereketeab Haileselassie, Deyin Doreen Hsing, Ranjit S. Chima, Amanda B. Hassinger, Stacey L. Valentine, Courtney M. Rowan, Martin C. J. Kneyber, Lincoln S. Smith, Robinder G. Khemani, Neal J. Thomas, Predicting Mortality in Children With Pediatric Acute Respiratory Distress Syndrome, Critical Care Medicine, 10.1097/CCM.0000000000004345, 48, 6, (e514-e522), (2020).
- J. Sainz Cabrejas, C. García Fuentes, C. García Juarranz, A.M. González López, L. Maure Blesa, J.C. Montejo González, M. Chico Fernández, Evaluation of quality of care in trauma patients using international scoring systems, Medicina Intensiva (English Edition), 10.1016/j.medine.2020.05.002, (2020).
- Peter C. Austin, Frank E. Harrell, David Klaveren, Graphical calibration curves and the integrated calibration index (ICI) for survival models, Statistics in Medicine, 10.1002/sim.8570, 39, 21, (2714-2742), (2020).
- Isabel Rosalie Arianne Retel Helmrich, Hester F. Lingsma, Alexis F Turgeon, Jose-Miguel Yamal, Ewout W Steyerberg, Prognostic Research in Traumatic Brain Injury: Markers, Modeling and Methodological Principles, Journal of Neurotrauma, 10.1089/neu.2019.6708, (2020).
- Umberto Benedetto, Shubhra Sinha, Matt Lyon, Arnaldo Dimagli, Tom R Gaunt, Gianni Angelini, Jonathan Sterne, Can machine learning improve mortality prediction following cardiac surgery?, European Journal of Cardio-Thoracic Surgery, 10.1093/ejcts/ezaa229, (2020).
- P. Ghorbani, T. Troëng, O. Brattström, K. G. Ringdal, T. Eken, A. Ekbom, L. Strömmer, Validation of the Norwegian survival prediction model in trauma (NORMIT) in Swedish trauma populations, BJS (British Journal of Surgery), 10.1002/bjs.11306, 107, 4, (381-390), (2019).
- E. C. McIlveen, E. Wright, `M. Shaw, J. Edwards, M. Vella, T. Quasim, S. J. Moug, A prospective cohort study characterising patients declined emergency laparotomy: survival in the ‘NoLap’ population, Anaesthesia, 10.1111/anae.14839, 75, 1, (54-62), (2019).
- Alishah Mawji, Samuel Akech, Paul Mwaniki, Dustin Dunsmuir, Jeffrey Bone, Matthew O. Wiens, Matthias Görges, David Kimutai, Niranjan Kissoon, Mike English, Mark J. Ansermino, Derivation and internal validation of a data-driven prediction model to guide frontline health workers in triaging children under-five in Nairobi, Kenya, Wellcome Open Research, 10.12688/wellcomeopenres.15387.1, 4, (121), (2019).
- Marine Flechet, Fabian Güiza, Isabelle Scharlaeken, Dirk Vlasselaers, Lars Desmet, Greet Van den Berghe, Geert Meyfroidt, Near-Infrared–Based Cerebral Oximetry for Prediction of Severe Acute Kidney Injury in Critically Ill Children After Cardiac Surgery, Critical Care Explorations, 10.1097/CCE.0000000000000063, 1, 12, (e0063), (2019).
- Seth A. Berkowitz, Kara E. Rudolph, Sanjay Basu, Detecting Anomalies Among Practice Sites Within Multicenter Trials, Circulation: Cardiovascular Quality and Outcomes, 10.1161/CIRCOUTCOMES.118.004907, 12, 3, (2019).
- An Jacobs, Marine Flechet, Ilse Vanhorebeek, Sören Verstraete, Catherine Ingels, Michael P. Casaer, Gerardo Soto-Campos, Sascha C. Verbruggen, Koen F. Joosten, Fabian Güiza, Greet Van den Berghe, Performance of Pediatric Mortality Prediction Scores for PICU Mortality and 90-Day Mortality*, Pediatric Critical Care Medicine, 10.1097/PCC.0000000000001764, 20, 2, (113-119), (2019).
- Vincenzo Russotto, Sergi Sabaté, Jaume Canet, Development of a prediction model for postoperative pneumonia, European Journal of Anaesthesiology, 10.1097/EJA.0000000000000921, 36, 2, (93-104), (2019).
- Michaël Darmon, Anne-Sophie Truche, Moustapha Abdel-Nabey, David Schnell, Bertrand Souweine, Early Recognition of Persistent Acute Kidney Injury, Seminars in Nephrology, 10.1016/j.semnephrol.2019.06.003, 39, 5, (431-441), (2019).
- J. Sainz Cabrejas, C. García Fuentes, C. García Juarranz, A.M. González López, L. Maure Blesa, J.C. Montejo González, M. Chico Fernández, Valoración de la calidad asistencial al traumatismo grave mediante comparación con estándares internacionales, Medicina Intensiva, 10.1016/j.medin.2019.02.002, (2019).
- Marine Flechet, Stefano Falini, Claudia Bonetti, Fabian Güiza, Miet Schetz, Greet Van den Berghe, Geert Meyfroidt, Machine learning versus physicians’ prediction of acute kidney injury in critically ill adults: a prospective evaluation of the AKIpredictor, Critical Care, 10.1186/s13054-019-2563-x, 23, 1, (2019).
- J. Fronczek, K. Polok, P.J. Devereaux, J. Górka, R.A. Archbold, B. Biccard, E. Duceppe, Y. Le Manach, D.I. Sessler, M. Duchińska, W. Szczeklik, External validation of the Revised Cardiac Risk Index and National Surgical Quality Improvement Program Myocardial Infarction and Cardiac Arrest calculator in noncardiac vascular surgery, British Journal of Anaesthesia, 10.1016/j.bja.2019.05.029, (2019).
- Giovanni Nattino, Stanley Lemeshow, Gary Phillips, Stefano Finazzi, Guido Bertolini, Assessing the Calibration of Dichotomous Outcome Models with the Calibration Belt, The Stata Journal: Promoting communications on statistics and Stata, 10.1177/1536867X1801700414, 17, 4, (1003-1014), (2019).
- Miroslav Stojadinovic, Ivan Vukovic, Milos Ivanovic, Milorad Stojadinovic, Dragan Milovanovic, Damnjan Pantic, Slobodan Jankovic, Optimal threshold of the prostate health index in predicting aggressive prostate cancer using predefined cost–benefit ratios and prevalence, International Urology and Nephrology, 10.1007/s11255-019-02367-z, (2019).
- Albert R. Moore, Paul M. Wieczorek, Jose C. A. Carvalho, Association Between Post–Dural Puncture Headache After Neuraxial Anesthesia in Childbirth and Intracranial Subdural Hematoma, JAMA Neurology, 10.1001/jamaneurol.2019.2995, (2019).
- Sharon E Davis, Robert A Greevy, Christopher Fonnesbeck, Thomas A Lasko, Colin G Walsh, Michael E Matheny, A nonparametric updating method to correct clinical prediction model drift, Journal of the American Medical Informatics Association, 10.1093/jamia/ocz127, (2019).
- Michaël Darmon, Aurélie Bourmaud, Quentin Georges, Marcio Soares, Kyeongman Jeon, Sandra Oeyen, Chin Kook Rhee, Pascale Gruber, Marlies Ostermann, Quentin A. Hill, Pieter Depuydt, Christelle Ferra, Anne-Claire Toffart, Peter Schellongowski, Alice Müller, Virginie Lemiale, Djamel Mokart, Elie Azoulay, Changes in critically ill cancer patients’ short-term outcome over the last decades: results of systematic review with meta-analysis on individual data, Intensive Care Medicine, 10.1007/s00134-019-05653-7, (2019).
- Naomi Datson, Matthew Weston, Barry Drust, Warren Gregson, Lorenzo Lolli, High-intensity endurance capacity assessment as a tool for talent identification in elite youth female soccer, Journal of Sports Sciences, 10.1080/02640414.2019.1656323, (1-7), (2019).
- Jejo D. Koola, Sam B. Ho, Aize Cao, Guanhua Chen, Amy M. Perkins, Sharon E. Davis, Michael E. Matheny, Predicting 30-Day Hospital Readmission Risk in a National Cohort of Patients with Cirrhosis, Digestive Diseases and Sciences, 10.1007/s10620-019-05826-w, (2019).
- Robyn M. Busch, Olivia Hogue, Michael W. Kattan, Marla Hamberger, Daniel L. Drane, Bruce Hermann, Michelle Kim, Lisa Ferguson, William Bingaman, Jorge Gonzalez-Martinez, Imad M. Najm, Lara Jehi, Nomograms to predict naming decline after temporal lobe surgery in adults with epilepsy, Neurology, 10.1212/WNL.0000000000006629, 91, 23, (e2144-e2152), (2018).
- Gloria Maria Custodio de Carvalho, Tacyano Tavares Leite, Alexandre Braga Libório, Prediction of 60-Day Case Fatality in Critically Ill Patients Receiving Renal Replacement Therapy, SHOCK, 10.1097/SHK.0000000000001054, 50, 2, (156-161), (2018).
- Guido Bertolini, Giovanni Nattino, Carlo Tascini, Daniele Poole, Bruno Viaggi, Greta Carrara, Carlotta Rossi, Daniele Crespi, Matteo Mondini, Martin Langer, Gian Maria Rossolini, Paolo Malacarne, Mortality attributable to different Klebsiella susceptibility patterns and to the coverage of empirical antibiotic therapy: a cohort study on patients admitted to the ICU with infection, Intensive Care Medicine, 10.1007/s00134-018-5360-0, 44, 10, (1709-1719), (2018).
- Marco Carbone, Alessandra Nardi, Steve Flack, Guido Carpino, Nikoletta Varvaropoulou, Caius Gavrila, Ann Spicer, Jonathan Badrock, Francesca Bernuzzi, Vincenzo Cardinale, Holly F Ainsworth, Michael A Heneghan, Douglas Thorburn, Andrew Bathgate, Rebecca Jones, James M Neuberger, Pier Maria Battezzati, Massimo Zuin, Simon Taylor-Robinson, Maria F Donato, John Kirby, Robert Mitchell-Thain, Annarosa Floreani, Fotios Sampaziotis, Luigi Muratori, Domenico Alvaro, Marco Marzioni, Luca Miele, Fabio Marra, Edoardo Giannini, Eugenio Gaudio, Vincenzo Ronca, Giulia Bonato, Laura Cristoferi, Federica Malinverno, Alessio Gerussi, Deborah D Stocken, Heather J Cordell, Gideon M Hirschfield, Graeme J Alexander, Richard N Sandford, David E Jones, Pietro Invernizzi, George F Mells, Caradog Thomas, Meshbah Rahman, Tom Yapp, Chin Lye Ch'ng, Melanie Harrison, Richard Sturgess, Roman Galaska, Chris Healey, Jessica Whiteman, Marek Czaijkowski, Catherine Gray, Anton Gunasekera, Pranab Gyawli, Purushothaman Premchand, Steven Mann, Keith Elliott, Kapil Kapur, Alan Watson, Graham Foster, Paul Trembling, Javaid Subhani, Rory Harvey, Roger McCorry, Carolyn Adgey, Lucie Hobson, Caroline Mulvaney-Jones, Richard Evans, Thiriloganathan Mathialahan, David Ramanaden, Jaber Gasem, Greta Van Duyvenvoorde, Christopher Shorrock, Katie Seward, Paul Southern, Jeremy Tibble, Ruth Penn, David Gorard, Jane Maiden, Rose Damant, Altaf Palegwala, Susan Jones, Graeme Alexander, George Mells, Richard Sandford, Jessica Whiteman, Sunil Dolwani, Martin Prince, Valeria Silvestre, Matthew Foxton, Eleanor Dungca, Harriet Mitchison, Natalie Wheatley, Ian Gooding, Helen Doyle, Mazn Karmo, Melanie Kent, Sushma Saksena, Delyth Braim, Minesh Patel, Susan Lord, Roland Ede, Alison Paton, Andrew Austin, Nicola Lancaster, Joanna Sayer, Andrew Gibbins, Karen Hogben, Chris Hovell, Neil Fisher, Martyn Carter, Konrad Koss, Janine Musselwhite, Florin Muscariu, Andrzej Piotreowicz, Alexandra McKay, Charles Grimley, David Neal, Lai Ting Tan, Guan Lim, Jacqueline Brighton, Carole Foale, Aftab Ala, Athar Saeed, Kerry Flahive, Gordon Wood, Paula Townshend, Chris Ford, Jonathan Brown, Jean Kordula, Jane Bowles, Mark Wilkinson, Caroline Palmer, John Ramage, Harriet Gordon, James Featherstone, Jo Ridpath, Theodore Ngatchu, Sass Levi, Syed Shaukat, Joy Sadeghian, Ray Shidrawi, Bronwen Williams, George Abouda, Sarah Jones, Claire Duggan, Abigail Hynes, Mark Narain, Ian Rees, Imroz Salam, Mary Crossey, Simon Taylor-Robinson, Ashley Brown, Carolyn MacNicol, Simon Williams, Elva Wilhelmsen, Paul Banim, Parizade Raymode, Andrew Chilton, Debasish Das, Hye-Jeong Lee, Howard Curtis, Michael Heneghan, Markus Gess, Emma Durant, IM Drake, Rebecca Bishop, Mervyn Davies, Rebecca Jones, Mark Aldersley, Noma Ncube, Alistair McNair, Raj Srirajaskanthan, Sambit Sen, Rebecca Casey, George Bird, Mike Mendall, Caroline Cowley, Adrian Barnardo, Paul Kitchen, Kevin Yoong, Kelly Amore, Dawn Sirdefield, Jacky Orpe, Ray Mathew, George MacFaul, Aruna Wrigth, Amir Shah, Chris Evans, Janie Keggans, Bridget Bird, Gwen Baxter, Subrata Saha, Katharine Pollock, Maggie Hughes, Peter Bramley, Emma Grieve, Karin Young, Andrew Fraser, Ashis Mukhopadhya, Kate Ocker, Peter Mills, Francis Hines, Chris Shallcross, Joy Wilkins, Leonie Grellier, Stewart Campbell, Kirsty Martin, Andrew Bathgate, Caron Innes, Alan Shepherd, Simon Rushbrook, Talal Valliani, Robert Przemioslo, Helen Fairlamb, Chris Macdonald, Anne Eastick, Jane Metcalf, Elizabeth Tanqueray, Udi Shmueli, Becky Holbrook, Andrew Davis, Julie Browning, Asifabbas Naqvi, Kirsten Walker, Tom Lee, Juliette Verheyden, Susan Slininger, Stephen D Ryder, Roger Chapman, Jane Collier, Denise O'Donnell, Lizzie Stafford, Kate Williamson, Linda Kent, Howard Klass, Mary Ninkovic, Linda March, Matthew Cramp, Diane Simpson, Christine Dickson, Nicholas Sharer, Maria Hayes, Patrick Goggin, Mary Quinne, Sallyanne Pearson, Barbara Hoeroldt, Linda Jones, Alice Wright, Jonathan Booth, Alison Loftus, George Lipscomb, Hannah Dewhurst, Emma Gunter, Earl Williams, Anna Fouracres, Liz Farrington, Lyn Graves, Hyder Hussaini, Bill Stableforth, Suzie Marriott, Reuben Ayres, Marina Leoni, Andrew Burroughs, Eileen Marshall, Douglas Thorburn, David Tyrer, Kate Martin, Martin Lombard, Imran Patanwala, Lola Dali-Kemmery, Victoria Lambourne, Julia Maltby, Samir Vyas, Julie Colley, Bal Shinder, Saket Singhal, Jayne Jones, Marisa Mills, Dermot Gleeson, Mandy Carnahan, Jeff Butterworth, Kerenza Boulton, Natalie Taylor, Keith George, Tim Harding, Julie Tregonning, Andrew Douglass, Carly Brown, Gayle Clifford, Simon Panter, Denise Gocher, Jeremy Shearman, Gary Bray, Maria Hamilton, Graham Butcher, Daniel Forton, John Mclindon, Janette Curtis, Debashis Das, Tracey Shewan, Matthew Cowan, Gregory Whatley, Mariam Nasseri, Bob Grover, Nurani Sivaramakrishnan, Samantha Ducker, Kathryn Houghton, David Jones, Laura Griffiths, Sherill Tripoli, Maxton Pitcher, Ervin Shpuza, Nikki White, Deb Ghosh, Andrew Douds, Marie Green, Matthew Brookes, Lourdes Cumlat, Voi Shim Wong, Karen Warner, Kimberley Netherton, Adtya Mandal, Snjiv Jain, Hemant Gupta, Pradeep Sanghi, Steve Pereira, James Neuberger, Bridget Gunson, Gideon Hirschfield, Reina Teegan Lim, Susan Gallagher, Darren Clement, Alison Brind, Gill Watts, Mcdonald Mupudzi, Mark Wright, Jane Gitahi, Fiona Gordon, Denis Gocher, Esther Unitt, Hilary Pateman, Sally Batham, Toby Delahooke, Allister Grant, Jill Conder, Andrew Higham, Mark Cox, Lynn O'Donohoe, Lynn Currie, Alistair King, Metod Oblak, Carole Collins, Simon Whalley, Marie Quinn, Yolanda Baird, Isobel Amey, Jocelyn Fraser, Andy Li, Donna Cotterill, Andrew Bell, Alan Watson, Amit Singhal, Ian Gee, Sandra Greer, Yeng Ang, Rupert Ransford, Joanna Allison, James Gotto, Simon Dyer, Helen Sweeting, Charles Millson, Pietro Invernizzi, Marco Carbone, Laura Cristoferi, Giulia Bonato, Federica Malinverno, Francesca Bernuzzi, Domenico Alvaro, Giancarlo Labbadia, Maria Consiglia Bragazzi, Pietro Andreone, Luigi Muratori, Francesco Azzaroli, Annarosa Floreani, Andrea Galli, Mirko Tarocchi, Edoardo Giannini, Luca Miele, Antonio Gasbarrini, Antonio Grieco, Giuseppe Marrone, Maria Francesca Donato, Luca Valenti, Fabio Marra, Marco Marzioni, Luca Maroni, Cristina Rigamonti, Massimo Zuin, Pier Maria Battezzati, Antonino Picciotto, Pretreatment prediction of response to ursodeoxycholic acid in primary biliary cholangitis: development and validation of the UDCA Response Score, The Lancet Gastroenterology & Hepatology, 10.1016/S2468-1253(18)30163-8, (2018).
- Stefano Skurzak, Greta Carrara, Carlotta Rossi, Giovanni Nattino, Daniele Crespi, Michele Giardino, Guido Bertolini, Cirrhotic patients admitted to the ICU for medical reasons: Analysis of 5506 patients admitted to 286 ICUs in 8 years, Journal of Critical Care, 10.1016/j.jcrc.2018.03.018, 45, (220-228), (2018).
- Giovanni Nattino, Stanley Lemeshow, Gary Phillips, Stefano Finazzi, Guido Bertolini, Assessing the Calibration of Dichotomous Outcome Models with the Calibration Belt, The Stata Journal: Promoting communications on statistics and Stata, 10.1177/1536867X1701700414, 17, 4, (1003-1014), (2018).
- Daniele Poole, Greta Carrara, Guido Bertolini, Intensive care medicine in 2050: statistical tools for development of prognostic models (why clinicians should not be ignored), Intensive Care Medicine, 10.1007/s00134-017-4825-x, 43, 9, (1403-1406), (2017).
- Luigina Mortari, Roberta Silva, Analyzing How Discursive Practices Affect Physicians’ Decision-Making Processes: A Phenomenological-Based Qualitative Study in Critical Care Contexts, INQUIRY: The Journal of Health Care Organization, Provision, and Financing, 10.1177/0046958017731962, 54, (004695801773196), (2017).
- Pedro Emmanuel Alvarenga Americano do Brasil, Sergio Salles Xavier, Marcelo Teixeira Holanda, Alejandro Marcel Hasslocher-Moreno, José Ueleres Braga, Does my patient have chronic Chagas disease? Development and temporal validation of a diagnostic risk score, Revista da Sociedade Brasileira de Medicina Tropical, 10.1590/0037-8682-0196-2016, 49, 3, (329-340), (2016).
- D. Poole, J. B. Carlisle, Mirror, mirror on the wall…predictions in anaesthesia and critical care, Anaesthesia, 10.1111/anae.13537, 71, 9, (1104-1109), (2016).
- Giovanni Nattino, Stefano Finazzi, Guido Bertolini, A new test and graphical tool to assess the goodness of fit of logistic regression models, Statistics in Medicine, 10.1002/sim.6744, 35, 5, (709-720), (2015).
- R. Raj, T. Brinck, M. B. Skrifvars, L. Handolin, External validation of the Norwegian survival prediction model in trauma after major trauma in Southern Finland, Acta Anaesthesiologica Scandinavica, 10.1111/aas.12592, 60, 1, (48-58), (2015).
- Peter C. Austin, Ewout W. Steyerberg, Bootstrap confidence intervals for loess‐based calibration curves, Statistics in Medicine, 10.1002/sim.6167, 33, 15, (2699-2700), (2014).
- Giovanni Nattino, Stefano Finazzi, Guido Bertolini, Comments on ‘Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers’ by Peter C. Austin and Ewout W. Steyerberg, Statistics in Medicine, 10.1002/sim.6126, 33, 15, (2696-2698), (2014).
- Ben Van Calster, Ewout W. Steyerberg, Calibration of Prognostic Risk Scores, Wiley StatsRef: Statistics Reference Online, 10.1002/9781118445112, (1-10), (2014).
