External Validation of the fullPIERS Model for Predicting Adverse Maternal Outcomes in Pregnancy Hypertension in Low- and Middle-Income CountriesNovelty and Significance
The hypertensive disorders of pregnancy are leading causes of maternal mortality and morbidity, especially in low- and middle-income countries. Early identification of women with preeclampsia and other hypertensive disorders of pregnancy at high risk of complications will aid in reducing this health burden. The fullPIERS model (Preeclampsia Integrated Estimate of Risk) was developed for predicting adverse maternal outcomes from preeclampsia using data from tertiary centers in high-income countries and uses maternal demographics, signs, symptoms, and laboratory tests as predictors. We aimed to assess the validity of the fullPIERS model in women with the hypertensive disorders of pregnancy in low-resourced hospital settings. Using miniPIERS data collected on women admitted with hypertensive disorders of pregnancy between July 2008 and March 2012 in 7 hospitals in 5 low- and middle-income countries, the predicted probability of developing an adverse maternal outcome was calculated for each woman using the fullPIERS equation. Missing predictor values were imputed using multivariate imputation by chained equations. The performance of the model was evaluated for discrimination, calibration, and stratification capacity.
Among 757 women with complete predictor data (complete-case analyses), the fullPIERS model had a good area under the receiver-operating characteristic curve of 0.77 (95% confidence interval, 0.72–0.82) with poor calibration (P<0·001 for the Hosmer–Lemeshow goodness-of-fit test). Performance as a rule-in tool was moderate (likelihood ratio: 5.9; 95% confidence interval, 4.23–8.35) for women with ≥30% predicted probability of an adverse outcome. The fullPIERS model may be used in low-resourced setting hospitals to identify women with hypertensive disorders of pregnancy at high risk of adverse maternal outcomes in need of immediate interventions.
Hypertension during pregnancy is one of the top 3 causes of maternal morbidity and mortality worldwide.1,2 The hypertensive disorders of pregnancy (HDPs), which include preeclampsia, superimposed preeclampsia, gestational hypertension, and chronic hypertension, complicate ≈5% to 10% of pregnancies.1,3 Maternal complications that result from HDPs include stroke, eclampsia, and renal dysfunction; and adverse fetal outcomes include stillbirth, preterm delivery, and cerebral palsy.4 These severe consequences of the HDPs make them a global health burden, especially in the low- and middle-income countries (LMICs) where >90% of HDP-related deaths occur.2,5 To reduce this burden, there is a need to correctly identify women at high risk of developing adverse outcomes in time to avoid their occurrence. Accurate risk assessment can aid decision making around the management of HDPs, including timing of delivery, administration of antenatal corticosteroids for acceleration of fetal pulmonary maturity or Magnesium sulfate for seizure prophylaxis, and maternal transfer to a higher level of care.1,3
To facilitate risk stratification and improve the management of HDPs, the fullPIERS model (Preeclampsia Integrated Estimate of Risk) was developed to predict adverse maternal outcomes occurring in the 48 hours after hospital admission with preeclampsia in high-income countries. The adverse outcomes predicted by the model included major organ dysfunction and death.6 The fullPIERS model is based on maternal demographics, signs, symptoms, and laboratory tests, with the final model consisting of 6 predictor variables: gestational age, chest pain or dyspnoea, oxygen saturation (Spo2), platelet count, serum creatinine, and serum aspartate aminotransferase (AST). On internal validation, the fullPIERS model predicted an adverse maternal outcome within 48 hours of hospital admission with an area under the receiver-operating characteristic curve (AUC ROC) of 0.88 (95% confidence interval [CI], 0.84–0.92).6,7 A preliminary external validation using the PETRA study (Preeclampsia Eclampsia Trial) cohort of high-risk women was also reassuring (AUC ROC: 0.97; 95% CI, 0.94–0.99).8
To ensure the generalizability of the fullPIERS model before it is implemented into clinical practice to improve maternal care,9,10 we sought to assess the model’s potential for use in a LMIC setting where the majority of HDP-related morbidity and mortality occur. The objective of this study was to use data from the miniPIERS cohort,11 collected prospectively in LMICs, to assess the broader validity of the fullPIERS model.
Ethical approval for this validation study was obtained from the Research Ethics Board of the University of British Columbia (CREB no: H07-02207). The PIERS projects were undertaken as a consented research and as a continuous quality improvement project depending on local ethics committee requirements.6
fullPIERS Cohort (Development Cohort)
The methods and results of the fullPIERS model have been published.6 In brief, the cohort consists of 2023 women diagnosed with preeclampsia who were admitted into tertiary hospital units, from September 2003 to January 2010 in 4 well-resourced countries: Canada, New Zealand, Australia, and the United Kingdom.6 Preeclampsia was defined as hypertension and one of proteinuria, hyperuricemia, or HELLP syndrome (Hemolysis Elevated Liver enzyme Low Platelet).6 An adverse maternal outcome referred to a composite of maternal death or morbidity, as determined by Delphi consensus for the fullPIERS study6 and outlined in Table S1 in the online-only Data Supplement. Women were excluded if they had already experienced an adverse maternal outcome before hospital admission or data collection or if they were admitted in spontaneous labor.
miniPIERS Cohort (Validation Cohort)
The methods and results of the miniPIERS study have been published.11 In brief, the cohort consists of 2081 women who were admitted to a participating hospital unit with a HDP (ie, preeclampsia, gestational hypertension, or chronic hypertension) and who had not yet experienced an adverse maternal outcome, from July 2008 to March 2012 in 5 LMICs: Fiji, Uganda, South Africa, Brazil, and Pakistan. Preeclampsia was defined as in the fullPIERS cohort; gestational hypertension was defined as blood pressure ≥140/90 mm Hg (at least one component, twice, ≥4 hours apart, ≥20+0 weeks) without significant proteinuria, and chronic hypertension as blood pressure ≥140/90 mm Hg (at least one component, twice, ≥4 hours apart, <20+0 weeks’ gestation). Adverse maternal outcomes were defined as in fullPIERS (Table S1). Women were excluded from the cohort if they experienced an adverse outcome before hospital admission or data collection or if they were admitted in spontaneous labor.
The distribution of patient characteristics in the development (fullPIERS) and validation (miniPIERS) cohorts were compared using χ2 test for nominal data and Mann–Whitney U test for continuous data. Univariate comparison of patient characteristics between the women in the validation cohort who experienced an adverse outcome and those who did not was also performed. A P value <0.05 was considered to be statistically significant.
The fullPIERS logistic regression equation for the prediction of adverse maternal outcomes from preeclampsia: logit(pi)=2.68+(−5.41×10−2; gestational age at eligibility)+1.23(chest pain or dyspnea)+(−2.71×10−2; creatinine)+(2.07×10−1; platelets)+(4.00×10−5; platelets2)+(1.01×10−2; aspartate transaminase)+(−3.05×10−6; AST2)+(2.50×10−4; creatinine×platelet)+(−6.99×10−5; platelet×aspartate transaminase)+(−2.56×10−3; platelet×Spo2).
Using the worst values (predefined in the model development study as the highest or lowest where appropriate)7 for the model predictors recorded within 24 hours of admission to HDP, the
fullPIERS equation was applied to the miniPIERS data, and the predicted probability of adverse outcomes for each individual with complete predictor data (complete case) was calculated. Before assessing the performance of the model, the model intercept was updated (baseline adjustment)10 because of the difference in the adverse maternal outcome rates between the fullPIERS (6.5%) and the miniPIERS population (12.5%).6,11
Missing Data and Sensitivity Analyses
To be consistent with the fullPIERS study, missing Spo2 values were imputed with 97%, the population median for women without adverse outcomes.6
After imputation of missing Spo2 data, complete-case analysis was used to assess model performance in the validation cohort, meaning only women with complete predictor data were included. However, to determine whether any bias in the model performance was present because of missing data, sensitivity analyses were performed using multiple imputations by chained equations to generate plausible values for the missing variables.12–20 More details on the imputation technique are given in the online-only Data Supplement.
We also conducted a sensitivity analysis using data of women admitted with only preeclampsia to assess the discriminatory performance of the model in this subgroup.
Performance Evaluation in the Final Validation Cohort
The performance of the model was evaluated based on discrimination and calibration ability and stratification accuracy.13,14 Discriminative ability was assessed using the AUC ROC and was interpreted using the following criteria: noninformative (AUC≤0.5), poor discrimination (0.5<AUC≤0.7), and good discrimination (AUC>0.7).15 Calibration was assessed by estimating the slope on a calibration plot of predicted versus observed outcome rates in each decile of predicted probability.13 Similar to the AUC ROC, a calibration slope of 1 was interpreted as ideal, >0.5 to <0.7 as poor, and ≤0.5 as noninformative. Calibration was also assessed based on the fit of the model in the validation cohort using the Hosmer–Lemeshow goodness-of-fit test, in which a P value >0·05 signifies a good fit between the model and data.14 The stratification capacity of the model to classify the women into low- and high-risk categories was assessed using a classification table with generated risk groups (defined based on categories established in the model development study).16,17 The true and false positive rates, negative predictive values, and positive predictive values were computed for each group. The likelihood ratios were calculated for each group using the Deeks and Altman method for a multicategory diagnostic test.18
All statistical analyses were performed using R version 3.1.3 (The R Project for Statistical Computing).
Simulation studies recommend at least 100 events and 100 nonevents for adequate power in validation studies.19 This number of events was calculated to give 80% power at the 5% significance level. We used this guideline to determine whether we had adequate statistical power in our study.
Of the 2081 women in the miniPIERS cohort, 261 women (12.5%) developed an adverse maternal outcome(s) within 48 hours of hospital admission with a HDP. Seven hundred and fifty-seven women (36.4%) women had information for all variables in the fullPIERS model, and these women were used for this validation study (complete-case analysis).
Of the 757 complete cases, 109 women (14.4%) had an adverse maternal outcome(s) within 48 hours of hospital admission. The most common adverse outcomes encountered were blood transfusion (52 women), eclampsia (14 women), and pulmonary edema (18 women). Other notable outcomes are listed in Table S2. There was no case of maternal death recorded in the validation data set.
Women in the miniPIERS validation cohort versus the fullPIERS development cohort were different with regard to demographics and pregnancy characteristics (ie, slightly younger, more often parous, and less likely to be a smoker or have a multiple pregnancy), clinical measures (ie, lower dipstick proteinuria, lower platelet count, and lower creatinine), interventions (ie, more likely to receive antenatal corticosteroids, antihypertensive therapy, and MgSO4), and outcomes (ie, shorter admission to delivery interval, higher infant birth weight but a higher infant mortality before hospital discharge; Table 1).
Within the miniPIERS validation data set, women who had adverse outcome (versus those who did not) were slightly younger, were more often nulliparous, and had hypertensive disorders of greater severity, including higher blood pressure, more frequent antihypertensive therapy and MgSO4, early gestational age at delivery, and lower infant birth weight compared with women without an adverse outcome (Table 2).
Data Completeness and Imputation Analysis
Seven hundred and fifty-seven women (36.4%; 568 preeclampsia and 189 with other HDPs) in the miniPIERS data set had complete fullPIERS variables. All women in the miniPIERS cohort had data for the gestational age at eligibility and chest pain/dyspnea; missing Spo2 values (1423, 68.3%) were substituted with 97% similar to the fullPIERS model development, and multiple imputations were performed for missing platelet count (1297, 62.3%), serum creatinine (1282, 61.6%), and AST (923, 44.4%). Imputation of missing values did not seem to alter the model performance significantly (online-only Data Supplement).
Within 48 hours of eligibility, the fullPIERS model predicted an adverse maternal outcome in the miniPIERS validation cohort with good discriminative performance as indicated by an AUC ROC of 0.77 (95% CI, 0.72–0.82; Figure 1). There was no significant change in the model performance using only cases with preeclampsia.
Figure 2 shows the calibration plot of the fullPIERS model when applied to the miniPIERS validation cohort. The calibration performance of the model was poor with a slope of 0.67 and intercept of −0.53, showing underestimation of risk at the lower risk ranges and overestimation of risks at the high-risk ranges. The Hosmer–Lemeshow test indicated a poor fit of the model’s expected outcomes with those observed in the validation cohort (P<0·05).13 Table 3 presents tabular information about calibration and classification accuracy. In the fullPIERS development cohort, more women (35%) fell into the predicted risk category of <1.0% than any other category, whereas in the miniPIERS complete-case validation cohort, the 5.0% to 9.9% range was the most common (with 23.5% of women). The majority of women who experienced an adverse outcome in both cohorts were in the predicted risk category of ≥0.30 (ie, 59% for fullPIERS and 50% for miniPIERS). Thus, the model classified a greater proportion of women without outcomes into the middle group, indicating lower stratification accuracy for the low-risk groups, although stratification accuracy remained good for the high-risk group in the validation cohort.
Table 4 presents the negative and positive predictive values and the true and false positive rates for the different risk groups. Using the highest predicted probability cutoff of 0.30, the category into which most women with adverse outcomes fell, the likelihood ratio was moderate at 5.9 (95% CI, 4.2–8.4), with a positive predictive value of 50% (95% CI, 0.40–0.60). Overall, the negative predictive values remained high (>90%) across all the risk.
We externally validated the fullPIERS model using the miniPIERS cohort of women in low-resourced settings for the prediction of adverse maternal outcomes related to the HDP within 48 hours of hospital admission. The model had good discriminative ability with AUC ROC of 0.77 (95% CI, 0.72–0.82) within 48 hours of admission, but this was significantly lower than its original performance in the development cohort (AUC ROC: 0.88; 95% CI, 0.84–0.92). Despite updating the model intercept to account for the baseline differences in adverse outcomes between the development and validation cohorts, the fullPIERS model had a poor fit in the miniPIERS data set reflected by the poor calibration performance. However, the fullPIERS model performed moderately as a rule-in test in the highest probability risk group with likelihood ratio of 5.9 (95% CI, 4.23–8.35).18
The decrease in the discriminative performance of the model in this study is in contrast with the fullPIERS validation study by Akkermans et al,8 which reported a high discriminative performance of the model with AUC ROC of 0.97 (95% CI, 0.94–0.99). The study used the PETRA cohort collected in the Netherlands, which is similar to the fullPIERS development cohort in that both cohorts were derived from tertiary centers in high-income settings, with similar management for women with HDPs. Compared with our validation cohort and the development cohort, the prevalence of adverse maternal outcomes in their study was also high (34%).
A possible reason for the decrease in the performance of the fullPIERS model in our study was the heterogeneity between the development cohort and our validation cohort. Differences between the development cohort and our validation cohort existed in the inclusion criteria, outcome prevalence, data collection settings (high-resourced versus low-resourced countries), and predictor distribution such as AST and platelet count (Table 1). Such low- and middle-income settings as our validation cohort settings are more likely to have more comorbidity, lower socioeconomic status, less availability of resources, and differences in disease management compared with high-income settings (reflected by more cointerventions and the shorter admission to delivery interval in the validation cohort shown in Table 1). Such factors may result in case-mix differences and may also alter the effect of the predictors on the outcome.2,7 Therefore, the extreme predictions observed in the calibration slope may have been as a result of differences in the predictor effects in the validation and development cohorts.10 These factors may have resulted in the reduced performance of the model.10,13
Another study by Agrawal and Maitra that assessed the validity of the fullPIERS model in a low-income setting reported a high likelihood ratio (17.53) for ruling out adverse outcomes.21 However, the rate of adverse outcomes (18.3%) and management of HDPs in their study cohort differed from the fullPIERS development cohort and our cohort. In addition, the study was underpowered and did not report AUC ROC.
A strength of our study is that this the first study to externally validate the fullPIERS model in a broader population (in a low-resourced setting with any HDP) using a fully powered sample size. Although internal and external validation using a similar patient cohort are important, validating a model in a different geographical setting is needed to evaluate the generalizability of the prediction model in other settings with a more diverse group of patients.10 This external validation study conducted using data from LMICs is particularly useful as most of the global burden of mortality and morbidity from the HDPs is borne by low-resourced settings.
The observed likelihood ratio (5.9) at the highest classification group suggests that the fullPIERS model can be used as a moderate rule-in tool for adverse outcomes from preeclampsia and other HDPs in low-resourced settings. For clinical practice in these settings, the recommended predicted probability of 0.3 can also be used as the optimal cutoff point to guide decisions around the need for immediate interventions. Half of the women with an adverse outcome fell in this risk category while the model still maintained a good likelihood ratio at a low false positive rate (7.6%). This has the added advantage of focusing limited resources on those who most need assistance in LMICs.
The major limitation of this study is the high proportion of missing values because the miniPIERS data were not originally collected explicitly for the purpose of this study. Using only complete-case analysis can lead to biased estimates of the predictions if the validation subset is not truly representative of the population at risk.20,22 Imputation of all missing values did not show any significant change in the model performance. Therefore, it is unlikely that selection bias contributed significantly to the drop in performance of the fullPIERS model in the complete-case analysis compared with the development performance. Even when missing values were excluded, the complete-case analysis had sufficient power (109 outcomes) to externally validate the model as recommended by simulation studies.19
Of note, most of the variables were missing because laboratory measurements for preeclampsia and the other HDPs are usually ordered based on the severity of other clinical measurements. This clinical management practice reflects the scarcity of resources in LMIC public hospitals and should draw attention to the need for lower cost point-of-care laboratory measurement techniques for these important laboratory measures. In the validation cohort, we demonstrated that there were worse clinical measures and pregnancy outcomes observed in the women with complete laboratory data compared with those with missing laboratory results (Table S4). This suggests that clinicians in these settings are able to identify higher risk women based on clinical assessment alone but that there remains a delay in timely intervention, so women continue to experience poor outcomes. Reducing the delay between assessments of laboratory measures and intervening when indicated should improve these women’s outcomes.
The fullPIERS model showed moderate utility for the prediction of adverse maternal outcomes in women with HDPs in our validation cohort collected in low-resourced setting hospitals, with some limitations in the lower risk groups. The stratification accuracy and discriminative ability of the fullPIERS model within the highest risk group makes it a valuable tool to aid clinicians in the identification of women at highest risk of adverse outcomes and allow for timely delivery of appropriate interventions such as transfer to a higher level of care for delivery and administration of antenatal corticosteroids.3 To determine applicability of the model in other well-resourced settings, future validation studies using more similar cohorts to that in which the model was developed are still needed.19,23
We are grateful to the members of the miniPIERS working group.11
Sources of Funding
This study was supported by the Canadian Institutes of Health Research (CIHR operating grants). The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit the article for publication.
The sponsors of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report.
The online-only Data Supplement is available with this article at http://hyper.ahajournals.org/lookup/suppl/doi:10.1161/HYPERTENSIONAHA.116.08706/-/DC1.
- Received November 4, 2016.
- Revision received December 2, 2016.
- Accepted January 6, 2017.
- © 2017 American Heart Association, Inc.
- Wulf S,
- Johns N,
- Lozano R
- Payne B,
- Hodgson S,
- Hutcheon JA,
- Joseph KS,
- Li J,
- Lee T,
- Magee LA,
- Qu Z,
- von Dadelszen P
- Akkermans J,
- Payne B,
- von Dadelszen P,
- Groen H,
- Vries Jd,
- Magee LA,
- Mol BW,
- Ganzevoort W
- Moons KG,
- Kengne AP,
- Grobbee DE,
- Royston P,
- Vergouwe Y,
- Altman DG,
- Woodward M
- Steyerberg E
- Payne BA,
- Hutcheon JA,
- Ansermino JM,
- et al
- van Buuren S,
- Groothuis-Oudshoorn K
- Pepe MS,
- Feng Z,
- Janes H,
- Bossuyt PM,
- Potter JD
- Deeks JJ,
- Altman DG
- Cummings P
- Agrawal S,
- Maitra N
Novelty and Significance
What Is New?
In this article, we have externally validated the fullPIERS model (Preeclampsia Integrated Estimate of Risk from high-income countries) for predicting maternal adverse outcomes from preeclampsia using data from multiple settings in low- and middle-income countries.
Our study is adequately powered and shows a moderate prediction performance of the model at the prerecommended predicted probability cutoff of ≥30%.
We have also assessed the performance of the model after imputation, which has not been done by the previous studies. Even on imputation of missing values, the model still identified high-risk women moderately.
What Is Relevant?
Hypertension in pregnancy is a major contributor to maternal morbidity and mortality, especially in low- and middle-income countries. Identifying the women at highest risk of adverse maternal outcome from hypertensive disorders of pregnancy is crucial in the settings to avert severe complications.
This study supports the existing literature and provides evidence that the fullPIERS model might be a useful tool in low-resourced settings. This finding is important to aid in reducing maternal morbidity and mortality resulting from hypertensive disorders of pregnancy in such areas where these events occur the most.