Performance of the fullPIERS model in predicting adverse maternal outcomes in pre-eclampsia using patient data from the PIERS (Pre-eclampsia Integrated Estimate of RiSk) cohort, collected on admission
B Payne, B428A-4500 Oak Street, Vancouver, BC, V6H 3N1, Canada. Email firstname.lastname@example.org
The fullPIERS (Pre-eclampsia Integrated Estimate of RiSk) model is a promising tool for the prediction of adverse outcomes in pre-eclampsia, developed using the worst values for predictor variables measured within 48 hours of admission. We reassessed the performance of fullPIERS using predictor variables obtained within 6 and 24 hours of admission, and found that the stratification capacity, calibration ability, and classification accuracy of the model remained high. The fullPIERS model is accurate as a rule-in test for adverse maternal outcome, with a likelihood ratio of 14.8 (95% CI 9.1–24.1) or 17.5 (95% CI 11.7–26.3) based on 6- and 24-hour data, respectively, for the women identified to be at highest risk (predicted probability ≥30%).
Pre-eclampsia, generally characterised by gestational hypertension and proteinuria, is a state of exaggerated systemic inflammation that remains a leading direct cause of maternal and perinatal morbidity and mortality worldwide.1 The only cure for pre-eclampsia is delivery, but expectant management remote from term is often used to improve perinatal outcomes. Caregivers are faced with the balancing act of attempting to gain the benefit of time for the fetus while undertaking uncertain maternal risk. Remote from term, time may bring benefits such as the optimum antenatal corticosteroid effect, whereas at any gestational age time brings the opportunity for the patient to be transferred to a higher level facility. However, the magnitude of maternal risk associated with pre-eclampsia may be unclear.
The fullPIERS (pre-eclampsia integrated estimate of risk) model is a recently developed tool for predicting adverse maternal outcomes after the diagnosis of pre-eclampsia.1 This tool is meant to aid caregivers in determining maternal risk in the setting of pre-eclampsia, in order to guide decisions around triage, transport, and treatment, in combination with an assessment of neonatal risk based on gestational age at presentation. Developed with data from over 1900 women, the fullPIERS model accurately predicted adverse maternal outcomes within 48 hours of being deemed eligible for the study, without significant over-fitting (area under the receiver operating characteristic curve, AUC ROC, 0.88; 95% CI 0.84–0.92), and also up to 7 days after being deemed eligible for the study (AUC ROC > 0.7).1
One critique of the model is that the worst values recorded during the first 48 hours after being deemed eligible for the study were used to predict complications within that same 48-hour time frame. Arguably, a model based exclusively on information available at the time of admission would be more clinically useful.2 Therefore, we undertook this study to assess how the fullPIERS model performs in the prediction of adverse maternal outcomes when the predictor variables are all obtained within either 6 or 24 hours of admission for pre-eclampsia.
The data used in this study were obtained from the PIERS database. The PIERS study was a multicentre prospective study designed to identify predictors of adverse maternal outcomes in women with pre-eclampsia. Recruitment to the cohort occurred from 1 September 2003 and 31 January 2010. The detailed methodology and results of the fullPIERS model development have been published elsewhere.1 In summary, women were included in the cohort if they met a research definition of pre-eclampsia or HELLP syndrome. Pre-eclampsia was defined as: (i) hypertension (blood pressure, BP, ≥140/90 mmHg, twice, ≥4 hours apart, after 20+0 weeks) and either proteinuria (urine protein ≥2+ by dipstick, ≥0.3 g/day by 24-hour urine collection, or >30 mg/mmol by urinary protein:creatinine ratio) or hyperuricaemia (value greater than local upper limit of normal for non-pregnant individuals); or (ii) superimposed pre-eclampsia (pre-existing hypertension with any of the following: clinician-determined accelerated hypertension, systolic BP ≥ 170 mmHg, diastolic BP ≥ 120 mmHg, new or accelerated proteinuria, or new hyperuricaemia). HELLP syndrome was defined as haemolysis, elevated liver enzymes, low platelets, with or without hypertension, and proteinuria.3 Women were excluded from the cohort if they were admitted to hospital in spontaneous labour or if any element of the combined adverse maternal outcome occurred prior to their meeting the eligibility criteria or before the collection of predictor variables was possible.
Study centres included four in Canada, one in New Zealand, one in Australia, and two in the UK. The PIERS study also involved standardisation of assessment and surveillance guidelines around the care of women with pre-eclampsia or gestational hypertension across all centres. The details of these guidelines have been published elsewhere.1
Data quality and missing data
Customised case report forms and databases were used by all participating sites. Data were obtained from the patient’s medical records, and predictor variables were collected within 6 and 24 hours of hospital admission. If data was missing, the method of last observation carried forward was used: any preceding observation recorded within 2 weeks (for laboratory values) or within 12 hours (for clinical assessments) of admission was regarded as current, unless replaced by a more recent value. A single imputation method was used to fill any missing values for the highly informative predictor variable, oxygen saturation by pulse oximetry (SpO2). The imputed value of 97% was based on the median value for the cohort.
Strategies to reduce misclassification and missing values included abstractor training, checking of data collection methods, monitoring of data logic, and re-abstraction of charts (5% at random, and all cases of suspected or confirmed adverse outcomes).
The combined maternal adverse outcome included maternal mortality and any of the following maternal morbidity: hepatic dysfunction, haematoma, or rupture; one or more seizures of eclampsia; Glasgow coma score <13; stroke; reversible ischaemic neurological deficit; transient ischaemic attack; posterior reversible encephalopathy syndrome; cortical blindness or retinal detachment; need for positive inotrope support; infusion of a third parenteral antihypertensive; myocardial ischaemia or infarction; acute renal insufficiency or failure; dialysis; pulmonary oedema; SpO2 <90%; requirement of ≥50% fractional inspired oxygen (FiO2) for more than 1 hour; intubation (other than solely for caesarean section); transfusion of any blood product; severe thrombocytopoenia (<50 × 109/l) in the absence of blood transfusion; and placental abruption. The list of adverse maternal outcomes was developed by iterative Delphi consensus.1 Full definitions of all components of the adverse maternal outcome are available online at https://piers.cfri.ca
Details regarding the development process of the fullPIERS model are published elsewhere.1 Developed with data from 1935 women with complete data, the final fullPIERS equation is: logit(π) = 2.68 + (−5.41 × 10−2 × gestational age at eligibility) + 1.23(chest pain or dyspnoea) + (−2.71 × 10−2 × creatinine) + (2.07 × 10−1 × platelets) + (4.00 × 10−5 × platelets2) + (1·01 × 10−2 × aspartate transaminase) + (−3·05 × 10−6 × AST2) + (2.50 × 10−4 × creatinine × platelets) + (−6.99 × 10−5 × platelets × aspartate transaminase) + (−2.56 × 10−3 × platelets × SpO2). A fullPIERS probability calculator is available on the study website.1
For all patients in the database, performance of the fullPIERS model was assessed by limiting predictor variable data to the worst values of the available data, either within 6 hours of admission or within 24 hours of admission. If no post-admission value was available in the specified time frame, the most recent pre-admission value was used if available. Patients with missing values required for the fullPIERS equation (aside from SpO2, which was filled with a value of 97% as described above) were excluded from the assessment of model performance. These predictor variables were used to predict outcomes occurring within 48 hours of admission to hospital, as in the development of the model,1 because this was felt to be a clinically useful time frame for treatment with corticosteroids or to allow transport from rural and remote areas.
We evaluated stratification capacity, calibration ability, and classification accuracy.4 Stratification capacity is reflected in the model’s ability to separate the population into low- and high-risk groups accurately. Calibration refers to the extent to which the risks calculated reflect the actual percentage of women with the outcome in each group. Classification accuracy is the extent to which persons with outcomes are identified as likely to have those outcomes. We calculated the AUC of the receiver operating characteristic (ROC) curve for the <6-hour and <24-hour time frames and likelihood ratios. We considered the following categories for the interpretation of the AUC ROC: non-informative (AUC = 0.5); poor accuracy (0.5 < AUC ≤ 0.7); moderate accuracy (0.7 < AUC ≤ 0.9); high accuracy (0.9 < AUC < 1); and perfect accuracy (AUC = 1).5 Likelihood ratios were calculated according to the method of Deeks and Altman6 for a multicategory diagnostic test. This method allows the calculation of likelihood ratios for each risk group individually, and is not directly related to the sensitivity and specificity of the dichotomised test result. The following categories for the interpretation of the likelihood ratios were used: informative (LR < 0.1 or > 10); moderately informative (LR 0.1–0.2 or 5–10); and non-informative (LR 0.2–5.0). ROC curve analyses were performed with spss 18.0 (SPSS Inc. Released 2009. PASW Statistics for Windows, Version 18.0. Chicago: SPSS Inc.), and Microsoft excel™ 2007 was used to generate risk stratification tables.
Between 1 September 2003 and 31 January 2010, data for 2023 women (2221 fetuses) were entered into the fullPIERS database from eight international sites. There were 261 (13%) combined adverse maternal outcomes at any time after study inclusion criteria met. A detailed description of demographic characteristics, clinical measures, interventions, pregnancy outcomes, and adverse outcomes was previously published.1 A total of 1935 women with complete data were used for the development and internal validation of the original model. For this study, there were 1829 women with complete data within 6 hours of eligibility, and 1891 women with complete data within 24 hours of eligibility.
In the group of women with complete data within 6 hours of admission, 91 (5.0%) had an adverse maternal outcome within 48 hours of admission. The median gestational age in this group was 36.0 weeks (interquartile range, IQR, 33.0–38.3 weeks) and the median maternal age was 31 years at estimated date of delivery (IQR 27–36 years). In the group of women with complete data within 24 hours of admission, 97 (5.1%) had an adverse maternal outcome within 48 hours of admission. The median gestational age in this group was 36.5 weeks (IQR 32.0–38.6 weeks) and the median maternal age was 31 years at estimated date of delivery (IQR 27–36 years). In both groups the most common adverse maternal outcomes were the haematological outcomes of requiring a blood transfusion and severe thrombocytopenia (platelet count <50 × 109) in the absence of blood transfusion, followed by the occurrence of pulmonary oedema. Using the platelet count as an example, in the 6-hour cohort the average time from admission to laboratory assessment was approximately 6 minutes, and in the 24-hour cohort it was approximately 100 minutes. The average time from eligibility to outcome in the full cohort was 23.25 hours.
The fullPIERS model predicted, with moderate accuracy, adverse maternal outcomes within 48 hours of eligibility, using predictor variables available within 6 hours of admission (AUC ROC 0.76; 95% CI 0.72–0.81), and within 24 hours of admission (AUC ROC 0.81, 95% CI 0.77–0.86). An AUC ROC >0.75 was maintained in both time periods when the data set was limited to just women admitted at <34 weeks of gestation, or when considering all outcomes except blood transfusion.
Table 1 presents the risk stratification capacity, calibration ability, and classification accuracy of the fullPIERS model when using data from within: (i) 6 hours or (ii) 24 hours of admission. fullPIERS limited to the initial 6 hours of data retained its stratification capacity, with 68% of women classified as low risk (predicted probability of combined adverse maternal outcome <2.5%), and 3% of women classified as highest risk (predicted probability ≥30%). fullPIERS limited to the initial 24 hours of data also successfully stratified the women into clinically relevant risk categories, with 65% of the women classified as low risk and 4% of women classified as highest risk. The calibration of the model in both data sets was accurate in high-risk groups, but there was a slight overestimation of risk in the lowest risk groups (Table 1). Assessing classification accuracy using a predicted probability of the outcome of ≥10% as the cut-off point, fullPIERS identified 56.7% of women who subsequently had adverse outcomes as being at high risk, whereas only 8.5% of women were incorrectly identified as being high risk when using the initial 24 hours of data. When limited to data within 6 hours of eligibility, fullPIERS identified 44% of women who subsequently had adverse outcomes as being at high risk, whereas only 7.2% of women were incorrectly identified as high risk (at a predicted probability of the outcome of ≥10%).
Table 1. Risk stratification, assessing the value of the fullPIERS model in risk prediction, by predicted probability of adverse maternal outcome within 48 hours
| Using predictor variable data within 6 hours of eligibility |
|Total||1829||91||5.0% (6.0–7.0)|| || || || |
| Using predictor variable data within 24 hours of admission |
|Total||1891||97||5.1% (4.1–6.1)|| || || || |
Using this threshold of ≥10% predicted probability to define a positive test resulted in negative predictive values of 96.9 and 97.5%, and positive predictive values of 24.4 and 26.6%, for the data collected within 6 and 24 hours of admission, respectively. When this threshold for a positive test was increased to ≥30% predicted probability, the negative predictive values of the test were 96.2 and 96.6%, and the positive predictive values were 43.6 and 48.6%, for the data collected within 6 and 24 hours of admission, respectively.
Table 1 also presents the likelihood ratios for each risk group. When data from within either 6 or 24 hours of admission were used, the fullPIERS model was highly informative, with likelihood ratios of 14.8 (95% CI 9.1–24.1) and 17.5 (95% CI 11.7–26.3), respectively, for women in the highest risk group (predicted probability of the outcome ≥30%).
We undertook this study to assess the ability of the fullPIERS model to predict adverse maternal outcomes when using only clinical predictor variables available within either 6 or 24 hours of admission. Although some deterioration in model performance was observed, the fullPIERS model applied within these restricted time periods remained a useful tool for stratifying women with pre-eclampsia according to their risk for adverse outcomes. The fullPIERS model is therefore a risk prediction tool that should be useful for clinical decision-making on: triage of patients for immediate delivery, versus expectant management; administration of corticosteroids remote from term, allowing time for patient transfer; and randomising women with pre-eclampsia in a clinical trial. Our findings provide evidence that fullPIERS largely retains its ability to predict adverse maternal outcomes within 48 hours when using data available within 6 or 24 hours of admission, instead of the worst values during the first 48 hours. The AUC ROC of 0.76 for the 6-hour data and 0.81 for the 24-hour data were slightly lower than the AUC ROC of 0.88 for the 48-hour data previously published,1 but these results still reflect moderate accuracy (AUC 0.7–0.9).5 The apparent change in performance probably reflects the fact that predictors improve with closer temporal proximity to the maternal adverse outcome.
Further assessment of the model using risk stratification tables confirms that it retains good stratification capacity and good calibration ability. Classification accuracy (i.e. the proportion of patients with subsequent adverse events who are identified as high risk versus low risk, based on a chosen threshold) remained reasonable, although not as good as in the original model. When using the initial 6 hours of data, and choosing a threshold for the predicted probability of adverse outcome of 10% to define a positive test, fullPIERS correctly identified 44% of women who subsequently had adverse outcomes as being high risk, as compared with 69% with the original 48-hour time frame. The false-positive rate remained essentially the same at this 10% threshold (7.2 versus 7.3%), corresponding to a 24.4% positive predictive value and 96.9% negative predictive value.
In both time frames presented here, the likelihood ratios associated with the highest risk group showed excellent performance (i.e. >10) of fullPIERS as a rule-in test. That is, if the fullPIERS predicted probability is >30%, clinicians can have confidence that the woman is at high risk of an adverse maternal outcome, and should adjust their management accordingly. However, the likelihood ratios for the lowest risk group in this analysis (<1% predicted probability) appeared to be meaningfully different than those obtained in the original publication (0.28 and 0.24 versus 0.09, respectively). Nevertheless, the overlap in confidence intervals around these point estimates (0.17–0.51 and 0.13–0.45 versus 0.03–0.27, respectively) precludes conclusions of significant differences in performance. These apparent differences in performance in the lowest risk group suggest that the model does not perform as well as a rule-out test for women who are at low risk of adverse outcomes when using data collected within 6 or 24 hours of admission, but it does still perform well as a rule-in test for women at high risk of adverse outcomes.
Some of the predictors in the fullPIERS model represent early stages of adverse maternal outcomes. For instance, elevated serum creatinine and liver enzymes predict acute renal and hepatic failure, respectively. This suggests that fullPIERS represents an early diagnosis rather than a prediction. The issue here is partly semantic, with a diagnosis of mild/biochemical derangement of the kidney or liver predicting overt renal or hepatic failure. In fact, diagnosis and prognosis/prediction are closely linked; the Apgar score is both diagnostic of the need for resuscitation and predictive of death in the neonatal period.7 Independent of whether the fullPIERS model represents early diagnosis or prognosis, given the fact that the predictor variables used in this study were measured, on average, 1 hour after admission, whereas the outcomes occurred, on average, 24 hours after admission, there would be adequate time to alter management based on the model results.
One limitation of this study results from the use of a combined adverse maternal outcome. One concern when using a combined adverse maternal outcome is that there is variation in the severity of the components of the outcome. For example, the requirement for a blood transfusion is not considered as severe an event as eclampsia. When a large proportion of events making up the combined adverse outcome are the ‘less significant’ outcomes, concern arises that we are only predicting these less severe events. The most common component of the combined adverse maternal outcome to occur in this study was the requirement of blood transfusion, which accounted for 30% of all adverse events. This limitation was addressed through the sensitivity analysis performed, showing that the model performance remained similar when excluding the most prevalent outcome. A second limitation of this study is that it assesses the performance of the fullPIERS model at only two specified time frames, from admission, looking forwards in time. Although this best reflects clinical practice, to look backwards in time from the occurrence of an adverse outcome may provide more insight into the course of illness in pre-eclampsia. Therefore, one future direction is to assess what point in time, relative to the occurrence of an adverse outcome, the fullPIERS predicted probability begins to increase.
This study provides evidence that the fullPIERS model maintains its utility as a prediction tool to be used as a rule-in test for adverse maternal outcomes within 48 hours of admission, when clinical predictor variables are limited to data available within 6 and 24 hours of admission with pre-eclampsia. Although much of its predictive value is maintained with data collected closer to admission, fullPIERS performance does improve with serial monitoring and inclusion of the worst values available, suggesting that the model could be used with data available upon admission, and then adjusted as further data become available.
Disclosure of interests
PvD is a consultant to Alere International for work that is unrelated to the PIERS project.
Contribution to authorship
BP, SH, LAM, and PVD developed the initial research question. BP, SH, JAH, TL, and KSJ contributed to the statistical methodology and analysis. All authors contributed to the writing of the final article and the interpretation of results.
Details of ethics approval
Ethics approval was obtained at all sites: University of British Columbia Clinical Research Ethics Board and Children’s & Women’s Hospital Ethics Board (REB# H07-02207); Ottawa Hospital (REB# 2008152-01H); Queen’s University (REB# OBGYN-138-04); Sherbrooke University (REB# 04-096-M3); Leed’s (NHS) East (REB# 04/Q1206/129); Nottingham (NHS), approval obtained as Continuous quality improvement (CQI) process and no file number given; Perth University and Department of Health for the Government of Western Australia (REB# 1538/EW); and Christchurch Women’s Hospital (REB# CTR/04/09/171).
We would like to acknowledge the funding support provided by the Canadian Institute of Health Research (CIHR; operating grants, salary, PvD and JAH), UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development & Research Training in Human Reproduction, Preeclampsia Foundation, International Federation of Obstetricians and Gynecologists (FIGO), Michael Smith Foundation for Health Research (salary, LAM and PvD), and the Child and Family Research Institute (salary award, PvD and KSJ).
We are very grateful for the effort made to collect the data involved in this study by all of the study coordinators and site investigators. The other members of the PIERS Study Group are: in Canada, J Mark Ansermino, Samantha Benton, Geoff Cundiff, M Joanne Douglas, Andrée Gruslin, Dany Hugo, Shoo K Lee, Paula Lott, Jean-Marie Moutquin, Annie B Ouellet, James A Russell, Dorothy Shaw (for FIGO), Graeme N Smith, D Keith Still, George Tawagi, and Brenda Wagner; in New Zealand, M Peter Moore; in the UK, Fiona Broughton Pipkin, Pamela Loughna, and James J Walker; in the USA, William A Grobman and Eleni Tsigas (for the Preeclampsia Foundation); and at the WHO, Mario Merialdi and Mariana Widmer.