Advertisement

Risk Prediction Models for Inadvertent Intraoperative Hypothermia: A Systematic Review

Published:October 16, 2021DOI:https://doi.org/10.1016/j.jopan.2021.02.011

      Abstract

      Purposes

      Inadvertent intraoperative hypothermia (core temperature <36°C) is a common surgical complication with several adverse events. Hypothermia prediction models can be a tool for providing the healthcare staff with information on the risk of inadvertent hypothermia. Our systematic review aimed to identify, demonstrate, and evaluate the available intraoperative hypothermia risk prediction models in surgical populations.

      Design

      This study is a systematic review of literature.

      Methods

      We systematically searched multiple databases (Ovid MEDLINE, Web of Science, Embase, and Cochrane Center Register of Controlled Trials). Two reviewers independently examined abstracts and the full text for eligibility. Data collection was guided by the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS checklist), and methodological quality and applicability were assessed by the Prediction model Risk Of Bias ASsessment Tool (PROBAST).

      Findings

      A total of 3,672 references were screened, of which eight articles were included in this study. All the models had a high risk of bias since most of them lacked model validation. Also, they failed to report the model performance and final model presentations, which restricted their clinical application.

      Conclusions

      The researchers should present models in a more standard way and improve the existing models to increase their predictive values for clinical application.

      Keywords

      Inadvertent intraoperative hypothermia, defined as a core body temperature less than 36.0°C during a surgical procedure, is a common but preventable surgical complication.
      • Yi J.
      • Zhan L.
      • Lei Y.
      • et al.
      Establishment and validation of a prediction equation to estimate risk of intraoperative hypothermia in patients receiving general anesthesia.
      In a recent study, 44.3% of surgical patients undergoing general anesthesia became hypothermic.
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      Hypothermia is associated with numerous adverse events, involving disturbed drug metabolism,
      • Leslie K.
      • Sessler D.I.
      • Bjorksten A.R.
      • Moayeri A.
      Mild hypothermia alters propofol pharmacokinetics and increases the duration of action of atracurium.
      surgical site infection,
      • Mason S.E.
      • Kinross J.M.
      • Hendricks J.
      • Arulampalam T.H.
      Postoperative hypothermia and surgical site infection following peritoneal insufflation with warm, humidified carbon dioxide during laparoscopic colorectal surgery: A cohort study with cost-effectiveness analysis.
      postoperative ileus,
      • Choi J.W.
      • Kim D.K.
      • Kim J.K.
      • Lee E.J.
      • Kim J.Y.
      A retrospective analysis on the relationship between intraoperative hypothermia and postoperative ileus after laparoscopic colorectal surgery.
      postoperative cardiovascular events,
      • Giuliano K.K.
      • Hendricks J.
      Inadvertent perioperative hypothermia: Current nursing knowledge.
      increased bleeding risk,
      • Kander T.
      • Schött U.
      Effect of hypothermia on haemostasis and bleeding risk: A narrative review.
      and increased consumption of red blood cells.
      • Lester E.
      • Fox E.E.
      • Holcomb J.B.
      • et al.
      The impact of hypothermia on outcomes in massively transfused patients.
      Also, hypothermia leads to more postoperative Intensive Care Unit (ICU) admissions, longer postanesthesia care unit (PACU) stays and prolonged postoperative hospital days.
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      Therefore, more importance should be attached to inadvertent hypothermia. In this era of personalized medicine, given that all patients do not have the same risk of hypothermia, the models developed to predict the risk of hypothermia are necessary.
      Risk prediction models use predictors to evaluate the probability that a condition or disease will occur in the future.
      • Moons K.
      • Wolff R.F.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration.
      One of the oldest and most famous prediction models in health care is the Apgar score, which provides a systematic and rapid assessment method to present newborns’ condition. Moreover, many multivariable prediction models have been widely developed to estimate individuals’ risk for cardiovascular disease events,
      • Carrillo-Larco R.M.
      • Altez-Fernandez C.
      • Pacheco-Barrios N.
      • et al.
      Cardiovascular disease prognostic models in Latin America and the Caribbean: A systematic review.
      diabetic complications,
      • Haider S.
      • Sadiq S.N.
      • Moore D.
      • Price M.J.
      • Nirantharakumar K.
      Prognostic prediction models for diabetic retinopathy progression: a systematic review.
      and so on. In the area of hypothermia, researchers identified the association between the occurrence of hypothermia and relevant variables such as age,
      • Kleimeyer J.P.
      • Harris A.
      • Sanford J.
      • Maloney W.J.
      • Kadry B.
      • Bishop J.A.
      Incidence and risk factors for postoperative hypothermia after orthopaedic surgery.
      BMI (body mass index),
      • Emmert A.
      • Gries G.
      • Wand S.
      • et al.
      Association between perioperative hypothermia and patient outcomes after thoracic surgery: A single center retrospective analysis.
      baseline temperature,
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      ,
      • Kleimeyer J.P.
      • Harris A.
      • Sanford J.
      • Maloney W.J.
      • Kadry B.
      • Bishop J.A.
      Incidence and risk factors for postoperative hypothermia after orthopaedic surgery.
      duration of anesthesia,
      • Emmert A.
      • Gries G.
      • Wand S.
      • et al.
      Association between perioperative hypothermia and patient outcomes after thoracic surgery: A single center retrospective analysis.
      and ambient temperature.
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      ,
      • Li Y.
      • Liang H.
      • Feng Y.
      Prevalence and multivariable factors associated with inadvertent intraoperative hypothermia in video-assisted thoracoscopic surgery: A single-center retrospective study.
      Although numerous studies examined the predictors and developed hypothermia prediction models, no studies have systematically reviewed the predictive performance and clinical applicability of these models.
      Accordingly, the objective of this systematic review was to identify and evaluate current risk prediction models for intraoperative hypothermia in surgical populations. We attempt to present the characteristics of the models and appraise the predictive competency and applicability of these models to provide a reference for future research and clinical practice.

      Method

      Protocol and registration

      We conducted this systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)
      • Moher D.
      • Shamseer L.
      • Clarke M.
      • et al.
      Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.
      guidance (Supplemental 1) and present this review in accordance with the Cochrane methodology and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS checklist).
      • Moons K.G.
      • de Groot J.A.
      • Bouwmeester W.
      • et al.
      Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.
      This review protocol was also registered in the PROSPERO register of systematic reviews (NO. CRD42020206643).

      Search strategy

      Electronic databases (Ovid MEDLINE, Web of Science, Embase, and Cochrane Center Register of Controlled Trials) were searched from inception to August 2020 to identify original studies describing the construction and validation of hypothermia risk models. No language or date restrictions were applied, and articles were translated when necessary. The search strategy is shown in Supplemental 2. Briefly, the search included terms related to “hypothermia,” “intraoperative,” and “risk prediction.” Additional potential articles were located by manually searching the citations and reference lists of all included articles.

      Eligibility criteria

      Study inclusion criteria were as follows: (i) the model was developed or validated in surgical patients, (ii) the model involved at least two predictors, (iii) the outcome was inadvertent intraoperative hypothermia. Exclusion criteria were as follows: (i) nonhuman or animal studies, (ii) a cross-sectional study without any actual outcome analyses, (iii) the outcomes were perioperative hypothermia events (eg, the occurrence was not preoperative or postoperative).

      Study selection

      Two reviewers independently screened titles and abstracts to determine which articles could potentially be included. Results were pooled, and any discrepancies were resolved by consensus. The same procedure was applied to assessing the eligibility of the full-text articles. Finally, the citations and references of the included articles were examined to assure any additional eligible prediction models with the proposed characteristics were identified.

      Data extraction

      Two reviewers independently extracted information using standardized forms. The list of extracted items was based on the CHARMS checklist.
      • Moons K.G.
      • de Groot J.A.
      • Bouwmeester W.
      • et al.
      Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.
      The information extracted from each study included country, study design (eg, retrospective, prospective cohort), surgery type (eg, abdominal surgery, thoracic surgery), final predictors, total/events (sample size/the number of positive events), temperature measuring site (eg, tympanic membrane, nasopharyngeal), model method (eg, logistic regression model, Cox proportional hazards model), model performance (eg, AUC: area under the receiver operating characteristic curve, H-L: Hosmer-Lemeshow test), model presentation (eg, equation, risk scale), and validation.

      Risk of bias and applicability

      The Prediction model Risk Of Bias ASsessment Tool (PROBAST) was used to estimate the risk of bias and applicability.
      • Moons K.
      • Wolff R.F.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration.
      Risk of bias to occurs when shortcomings in study design, implementation, or analysis may lead to systematically distorted assessments of model values, which is compromised of four domains: participants, predictors, outcome, and analysis.
      • Wolff R.F.
      • Moons K.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess the risk of bias and applicability of prediction model studies.
      Applicability contains the first three areas. Risk of bias indicates whether systematic errors exist in the study, while applicability determines whether the model matches the research question of the systematic review.
      • Moons K.
      • Wolff R.F.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration.
      ,
      • Wolff R.F.
      • Moons K.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess the risk of bias and applicability of prediction model studies.
      The “worst score counts” principle was applied to the overall appraisal, meaning that one bad item in a domain made the overall evaluation bad.
      • Palazón-Bru A.
      • Mares-García E.
      • López-Bru D.
      • et al.
      A critical appraisal of the clinical applicability and risk of bias of the predictive models for mortality and recurrence in patients with oropharyngeal cancer: Systematic review.

      Synthesis of results

      Descriptive statistics of data were presented in tables to provide a synthesis of pivotal characteristics of the prediction models. Because of the heterogeneity among studies, we could not conduct a meta-analysis.

      Results

      Study selection

      Through database searching and snowballing, a total of 3,672 articles were identified. After removing duplicates and excluding studies that did not meet criteria based on titles and abstracts, 35 reports remained for full-text evaluation. Eight articles were selected for a narrative synthesis after full-text assessment against the review criteria. The search process is shown in a PRISMA flow diagram (Figure 1).
      Figure 1
      Figure 1Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart for studies included in the review. From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed1000097. This image is available in color online at www.jopan.org.

      Study characteristics

      The key characteristics of each study are shown in Table 1. Half of the studies (n = 4) were conducted in Asian countries,
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      ,
      • Kasai T.
      • Hirose M.
      • Yaegashi K.
      • Matsukawa T.
      • Takamata A.
      • Tanaka Y.
      Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia.
      • Kim E.J.
      • Yoon H.
      Preoperative factors affecting the intraoperative core body temperature in abdominal surgery under general anesthesia: An observational cohort.
      • Yang L.
      • Huang C.Y.
      • Zhou Z.B.
      • et al.
      Risk factors for hypothermia in patients under general anesthesia: Is there a drawback of laminar airflow operating rooms? A prospective cohort study.
      and the rest originated from Europe (n = 3) or North America (n = 1).
      • Emmert A.
      • Gries G.
      • Wand S.
      • et al.
      Association between perioperative hypothermia and patient outcomes after thoracic surgery: A single center retrospective analysis.
      ,
      • Rincón D.A.
      • Valero J.F.
      • Eslava-Schmalbach J.
      Construcción y validación de un modelo predictivo de hipotermia intraoperatoria [Construction and validation of a model to predict intraoperative hypothermia].
      • Desgranges F.P.
      • Bapteste L.
      • Riffard C.
      • et al.
      Predictive factors of maternal hypothermia during Cesarean delivery: A prospective cohort study. Facteurs prédictifs d'hypothermie maternelle durant la Césarienne: une étude prospective de cohorte.
      • Chalari E.
      • Intas G.
      • Zyga S.
      • et al.
      Preoperative factors affecting the intraoperative core body temperature in elective hysterectomy under general anesthesia.
      The oldest model was reported in 2002,
      • Kasai T.
      • Hirose M.
      • Yaegashi K.
      • Matsukawa T.
      • Takamata A.
      • Tanaka Y.
      Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia.
      and the newest one in 2019.
      • Chalari E.
      • Intas G.
      • Zyga S.
      • et al.
      Preoperative factors affecting the intraoperative core body temperature in elective hysterectomy under general anesthesia.
      The majority of the models (n = 6) were developed using data from prospective cohort studies, although retrospective data (n = 2) were also used.
      • Emmert A.
      • Gries G.
      • Wand S.
      • et al.
      Association between perioperative hypothermia and patient outcomes after thoracic surgery: A single center retrospective analysis.
      ,
      • Kasai T.
      • Hirose M.
      • Yaegashi K.
      • Matsukawa T.
      • Takamata A.
      • Tanaka Y.
      Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia.
      In addition, most studies were conducted with patients undergoing general anesthesia,
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      ,
      • Yang L.
      • Huang C.Y.
      • Zhou Z.B.
      • et al.
      Risk factors for hypothermia in patients under general anesthesia: Is there a drawback of laminar airflow operating rooms? A prospective cohort study.
      ,
      • Rincón D.A.
      • Valero J.F.
      • Eslava-Schmalbach J.
      Construcción y validación de un modelo predictivo de hipotermia intraoperatoria [Construction and validation of a model to predict intraoperative hypothermia].
      though surgeries with other types of anesthesia were also included, such as abdominal surgery, cesarean delivery, thoracic surgery, and open hysterectomy.
      • Emmert A.
      • Gries G.
      • Wand S.
      • et al.
      Association between perioperative hypothermia and patient outcomes after thoracic surgery: A single center retrospective analysis.
      ,
      • Kasai T.
      • Hirose M.
      • Yaegashi K.
      • Matsukawa T.
      • Takamata A.
      • Tanaka Y.
      Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia.
      ,
      • Kim E.J.
      • Yoon H.
      Preoperative factors affecting the intraoperative core body temperature in abdominal surgery under general anesthesia: An observational cohort.
      ,
      • Desgranges F.P.
      • Bapteste L.
      • Riffard C.
      • et al.
      Predictive factors of maternal hypothermia during Cesarean delivery: A prospective cohort study. Facteurs prédictifs d'hypothermie maternelle durant la Césarienne: une étude prospective de cohorte.
      ,
      • Chalari E.
      • Intas G.
      • Zyga S.
      • et al.
      Preoperative factors affecting the intraoperative core body temperature in elective hysterectomy under general anesthesia.
      The number of predictors per study ranged from 3 to 8, and 18 different predictors were identified in the prediction models. The most common predictors were age, baseline temperature, and weight. The number of participants per study varied between 104 and 1840, and the number of positive events per study varied between 51 and 472. Most of models measured tympanic membrane temperature (n = 6),
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      ,
      • Kasai T.
      • Hirose M.
      • Yaegashi K.
      • Matsukawa T.
      • Takamata A.
      • Tanaka Y.
      Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia.
      ,
      • Kim E.J.
      • Yoon H.
      Preoperative factors affecting the intraoperative core body temperature in abdominal surgery under general anesthesia: An observational cohort.
      ,
      • Rincón D.A.
      • Valero J.F.
      • Eslava-Schmalbach J.
      Construcción y validación de un modelo predictivo de hipotermia intraoperatoria [Construction and validation of a model to predict intraoperative hypothermia].
      • Desgranges F.P.
      • Bapteste L.
      • Riffard C.
      • et al.
      Predictive factors of maternal hypothermia during Cesarean delivery: A prospective cohort study. Facteurs prédictifs d'hypothermie maternelle durant la Césarienne: une étude prospective de cohorte.
      • Chalari E.
      • Intas G.
      • Zyga S.
      • et al.
      Preoperative factors affecting the intraoperative core body temperature in elective hysterectomy under general anesthesia.
      except Yang et al.
      • Yang L.
      • Huang C.Y.
      • Zhou Z.B.
      • et al.
      Risk factors for hypothermia in patients under general anesthesia: Is there a drawback of laminar airflow operating rooms? A prospective cohort study.
      measured nasopharyngeal temperature and Emmert et al.
      • Emmert A.
      • Gries G.
      • Wand S.
      • et al.
      Association between perioperative hypothermia and patient outcomes after thoracic surgery: A single center retrospective analysis.
      did not report a measuring site.
      Table 1Study Characteristics
      Study IDCountryStudy DesignSurgeryFinal Predictors (n)Total/ EventsMeasuring SiteModel MethodModel PerformancePresentationValidation
      Kasai 2002JapanRetrospective cohortOpen abdominalAge, height, weight, preoperative systolic blood pressure, preoperative heart rate (5)400/200TympanicLogistic regressionSensibility: 81.5%

      Specificity: 83.0%
      EquationExternal
      Rincon 2008ColombiaProspective cohortGeneral anesthesiaAge, weight, duration of operation, ambient temperature, baseline temperature (5)200/93TympanicLogistic regressionAUC: 0.83 (95%CI: 0.78-0.89)

      Sensibility: 84%

      Specificity: 73%
      Risk scaleExternal
      Kim 2014South KoreaProspective cohortAbdominal surgeryPreoperative baseline temperature, weight, preoperative heart rate, age (4)1 h: 147/51

      2 h: 147/68

      3 h: 147/80
      TympanicLogistic regression1h: R2=0.658, p<0.001

      2h: R2=0.631, p<0.001

      3h: R2=0.665, p<0.001
      NRNone
      Yang 2015ChinaProspective cohortGeneral anesthesiaAge, operating room, surgery type (3)1840/472NasopharyngealLogistic regressionNRNRNone
      Desgranges 2017FranceProspective cohortCesarean deliveryThe volume of fluids, baseline temperature, core temperature at skin incision, warming modes, obesity (5)359/81TympanicLogistic regressionAUC: 0.87 (95%CI: 0.83-0.90)

      H-L: p=0.99
      NRNone
      Yi 2017ChinaProspective cohortGeneral anesthesiaMagnitude of surgery, the volume of fluids, duration of anesthesia, warming modes, BMI, ambient temperature, baseline temperature (7)830/NR =39.9%TympanicLogistic regressionBrier score: 0.21

      C-statistics: 0.79

      H-L: p=0.56
      EquationExternal
      Emmert 2018GermanyRetrospective cohortThoracic surgeryDuration of operation, body surface area, the volume of fluids (3)339/218NRLogistic regressionNRNRNone
      Chalari 2019GreeceProspective cohortOpen hysterectomyBMI, ASA, type of anesthesia, anesthetic drugs, opioid drugs, premedication, ropivacaine, muscle relaxants (8)104/56TympanicCox proportional hazardsNRNRNone
      AUC, area under the receiver operating characteristic curve; R2, coefficient of determination; H-L, Hosmer-Lemeshow; NR, not reported.
      The major modeling method was logistic regression, except one used Cox proportional hazards.
      • Chalari E.
      • Intas G.
      • Zyga S.
      • et al.
      Preoperative factors affecting the intraoperative core body temperature in elective hysterectomy under general anesthesia.
      Four models presented discrimination with the AUC value or the C-statistic ranging from 0.79 to 0.87.
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      ,
      • Kasai T.
      • Hirose M.
      • Yaegashi K.
      • Matsukawa T.
      • Takamata A.
      • Tanaka Y.
      Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia.
      ,
      • Rincón D.A.
      • Valero J.F.
      • Eslava-Schmalbach J.
      Construcción y validación de un modelo predictivo de hipotermia intraoperatoria [Construction and validation of a model to predict intraoperative hypothermia].
      ,
      • Desgranges F.P.
      • Bapteste L.
      • Riffard C.
      • et al.
      Predictive factors of maternal hypothermia during Cesarean delivery: A prospective cohort study. Facteurs prédictifs d'hypothermie maternelle durant la Césarienne: une étude prospective de cohorte.
      Two models presented calibration using the H-L test.
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      ,
      • Desgranges F.P.
      • Bapteste L.
      • Riffard C.
      • et al.
      Predictive factors of maternal hypothermia during Cesarean delivery: A prospective cohort study. Facteurs prédictifs d'hypothermie maternelle durant la Césarienne: une étude prospective de cohorte.
      Kim et al.
      • Kim E.J.
      • Yoon H.
      Preoperative factors affecting the intraoperative core body temperature in abdominal surgery under general anesthesia: An observational cohort.
      developed three models, describing the incidence of intraoperative hypothermia in three different periods (1, 2, and 3 hours during surgery) and demonstrated goodness of fit with the coefficient of determination (R2) ranging from 0.631 to 0.665. The remaining studies did not report the model performance.
      • Emmert A.
      • Gries G.
      • Wand S.
      • et al.
      Association between perioperative hypothermia and patient outcomes after thoracic surgery: A single center retrospective analysis.
      ,
      • Yang L.
      • Huang C.Y.
      • Zhou Z.B.
      • et al.
      Risk factors for hypothermia in patients under general anesthesia: Is there a drawback of laminar airflow operating rooms? A prospective cohort study.
      ,
      • Chalari E.
      • Intas G.
      • Zyga S.
      • et al.
      Preoperative factors affecting the intraoperative core body temperature in elective hysterectomy under general anesthesia.
      Three models reported a regression equation or a risk scale and were prospectively applied to other patients to conduct external validation.
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      ,
      • Kasai T.
      • Hirose M.
      • Yaegashi K.
      • Matsukawa T.
      • Takamata A.
      • Tanaka Y.
      Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia.
      ,
      • Rincón D.A.
      • Valero J.F.
      • Eslava-Schmalbach J.
      Construcción y validación de un modelo predictivo de hipotermia intraoperatoria [Construction and validation of a model to predict intraoperative hypothermia].
      However, the remaining models presented insufficient information about model presentation and validation.
      • Emmert A.
      • Gries G.
      • Wand S.
      • et al.
      Association between perioperative hypothermia and patient outcomes after thoracic surgery: A single center retrospective analysis.
      ,
      • Kim E.J.
      • Yoon H.
      Preoperative factors affecting the intraoperative core body temperature in abdominal surgery under general anesthesia: An observational cohort.
      ,
      • Yang L.
      • Huang C.Y.
      • Zhou Z.B.
      • et al.
      Risk factors for hypothermia in patients under general anesthesia: Is there a drawback of laminar airflow operating rooms? A prospective cohort study.
      ,
      • Desgranges F.P.
      • Bapteste L.
      • Riffard C.
      • et al.
      Predictive factors of maternal hypothermia during Cesarean delivery: A prospective cohort study. Facteurs prédictifs d'hypothermie maternelle durant la Césarienne: une étude prospective de cohorte.
      ,
      • Chalari E.
      • Intas G.
      • Zyga S.
      • et al.
      Preoperative factors affecting the intraoperative core body temperature in elective hysterectomy under general anesthesia.

      Risk of bias assessment

      A summary of the risk of bias assessment based on the PROBAST tool
      • Moons K.
      • Wolff R.F.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration.
      is shown in Table 2. All studies had an overall high risk of bias (n = 8). In the domain of participants, most of studies had a low risk of bias (n = 6),
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      ,
      • Kim E.J.
      • Yoon H.
      Preoperative factors affecting the intraoperative core body temperature in abdominal surgery under general anesthesia: An observational cohort.
      • Yang L.
      • Huang C.Y.
      • Zhou Z.B.
      • et al.
      Risk factors for hypothermia in patients under general anesthesia: Is there a drawback of laminar airflow operating rooms? A prospective cohort study.
      • Rincón D.A.
      • Valero J.F.
      • Eslava-Schmalbach J.
      Construcción y validación de un modelo predictivo de hipotermia intraoperatoria [Construction and validation of a model to predict intraoperative hypothermia].
      • Desgranges F.P.
      • Bapteste L.
      • Riffard C.
      • et al.
      Predictive factors of maternal hypothermia during Cesarean delivery: A prospective cohort study. Facteurs prédictifs d'hypothermie maternelle durant la Césarienne: une étude prospective de cohorte.
      • Chalari E.
      • Intas G.
      • Zyga S.
      • et al.
      Preoperative factors affecting the intraoperative core body temperature in elective hysterectomy under general anesthesia.
      whereas two retrospective studies showed a high risk of bias.
      • Emmert A.
      • Gries G.
      • Wand S.
      • et al.
      Association between perioperative hypothermia and patient outcomes after thoracic surgery: A single center retrospective analysis.
      ,
      • Kasai T.
      • Hirose M.
      • Yaegashi K.
      • Matsukawa T.
      • Takamata A.
      • Tanaka Y.
      Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia.
      In the domain of predictors, only three models performed well.
      • Kasai T.
      • Hirose M.
      • Yaegashi K.
      • Matsukawa T.
      • Takamata A.
      • Tanaka Y.
      Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia.
      • Kim E.J.
      • Yoon H.
      Preoperative factors affecting the intraoperative core body temperature in abdominal surgery under general anesthesia: An observational cohort.
      • Yang L.
      • Huang C.Y.
      • Zhou Z.B.
      • et al.
      Risk factors for hypothermia in patients under general anesthesia: Is there a drawback of laminar airflow operating rooms? A prospective cohort study.
      The reason for poor performance was that some predictors (eg, duration of surgery, the volume of fluids) were not available at the moment of prediction. For outcome, the risk of bias was often low (n = 7). However, a high risk of bias was frequently scored in the analysis domain, mainly because these models lacked a validation process.
      Table 2Risk of Bias Assessment Based on the PROBAST Tool
      Risk of BiasApplicabilityOverall
      StudyParticipantsPredictorsOutcomeAnalysisParticipantsPredictorsOutcomeRisk of BiasApplicability
      Kasai 2002++++++
      Rincon 2008++++++
      Kim 2014+++++
      Yang 2015+++++++
      Desgranges 2017++++++
      Yi 2017++++++
      Emmert 2018?+?+?
      Chalari 2019++++++
      + Indicates low risk of low concern regarding applicability, - indicates high risk of bias or high concern regarding applicability, and ? indicates unclear risk of bias or unclear concern regarding applicability.

      Discussion

      This systematic review presented an overview of currently available hypothermia risk prediction models. We reported eight models specifically developed for common surgeries such as abdominal surgery, cesarean delivery, and thoracic surgery. The results showed that existing articles did not report model information in a complete and transparent way. Some studies failed to show pivotal modeling data, such as sample size considerations, the parameters of calibration and discrimination, and mathematical equations, which might lead to a high risk of bias.
      The risk of bias assessment demonstrated that all studies, whether or not they conducted external validation, had a high risk of bias. The primary bias came from the domain of analysis due to inappropriate sample size and incomplete handling of missing data. This finding is in agreement with previous research.
      • Haider S.
      • Sadiq S.N.
      • Moore D.
      • Price M.J.
      • Nirantharakumar K.
      Prognostic prediction models for diabetic retinopathy progression: a systematic review.
      ,
      • Cai R.
      • Wu X.
      • Li C.
      • Chao J.
      Prediction models for cardiovascular disease risk in the hypertensive population: A systematic review.
      From the perspective of methodology, although overall sample size matters, number of participants paired with the outcome (positive events) is even more significant. In prediction model studies, sample size calculations have usually been based on the number of Events Per Variable (EPV, positive events/candidate predictors).
      • Moons K.
      • Wolff R.F.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration.
      ,
      • Sami W.
      • Alrukban M.O.
      • Waqas T.
      • Asad M.R.
      • Afzal K.
      Sample size determination in health research.
      The EPVs indicate not the number of predictors enrolled in the final model but rather the total number of predictors measured during any phase of the model implementation. For this reason, EPVs of at least 20 are less likely to have overfitting.
      • Ogundimu E.O.
      • Altman D.G.
      • Collins G.S.
      Adequate sample size for developing prediction models is not simply related to events per variable.
      Overfitting can arise when the number of positive events is small compared with the number of final predictors, resulting in the incidence of an outcome event being underestimated in low-risk participants and overestimated in high-risk participants.
      • Meier K.
      • Parrish J.
      • D'Souza R
      Prediction models for determining the success of labor induction: A systematic review.
      Nevertheless, only two studies met this criterion in our included models.
      • Yi J.
      • Lei Y.
      • Xu S.
      • et al.
      Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
      ,
      • Yang L.
      • Huang C.Y.
      • Zhou Z.B.
      • et al.
      Risk factors for hypothermia in patients under general anesthesia: Is there a drawback of laminar airflow operating rooms? A prospective cohort study.
      Issues related to missing data were also problematic. When a study does not mention missing data, patients with any missing information probably are omitted from analyses. Because statistical packages automatically eliminate samples with any missing data in the process of data analysis, this can undermine the validity of results by biasing estimation, lowing power, and ultimately reducing confidence in the models.
      • Moons K.
      • Wolff R.F.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration.
      ,
      • Bell M.L.
      • Floden L.
      • Rabe B.A.
      • et al.
      Analytical approaches and estimands to take account of missing patient-reported data in longitudinal studies.
      Overall, these shortcomings in the methodological quality increase the potential bias when evaluating the predictive performance of a model. Therefore, guidelines for the reporting of risk prediction research should be taken into account. The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative is a checklist involving a minimum set of items that authors should report to inform the readers about how the study is designed and carried out.
      • Collins G.S.
      • Reitsma J.B.
      • Altman D.G.
      • Moons K.G.
      Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement.
      This protocol can be a guidance to standardize the reporting way of prediction models. Moreover, researchers are encouraged to register their study on relevant websites, such as Clinical Trials.gov.
      Our results showed that most studies did not include model validation, which might impact the comparison of these models’ predictive values. However, this finding is consistent with existing research.
      • Haider S.
      • Sadiq S.N.
      • Moore D.
      • Price M.J.
      • Nirantharakumar K.
      Prognostic prediction models for diabetic retinopathy progression: a systematic review.
      ,
      • Meier K.
      • Parrish J.
      • D'Souza R
      Prediction models for determining the success of labor induction: A systematic review.
      ,
      • Heestermans T.
      • Payne B.
      • Kayode G.A.
      • et al.
      Prognostic models for adverse pregnancy outcomes in low-income and middle-income countries: A systematic review.
      Model validation is a crucial step to ensure that models perform stably in new participants, which can be divided into two categories (internal and external validation). Internal validation means that the same data are used for both development and validation, such as cross-validation and bootstrapping. External validation aims to evaluate existing models using data external to the construction sample (eg, from different populations).
      • Wolff R.F.
      • Moons K.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess the risk of bias and applicability of prediction model studies.
      Both of these methods can quantify the models’ bias. The model performance usually consists of discrimination (eg, the c-statistic) and calibration (eg, the H-L test). However, in our study, only three models used the AUC and/or H-L test for model performance to present the results of model validation. Meanwhile, only these three models showed the final models using an equation or a risk scale.
      According to Heestermans et al.,
      • Heestermans T.
      • Payne B.
      • Kayode G.A.
      • et al.
      Prognostic models for adverse pregnancy outcomes in low-income and middle-income countries: A systematic review.
      a lack of exact model presentations made external validation by other scholars difficult. Researchers may find it challenging to apply these models to clinical populations and cannot examine the models’ values. Therefore, to make the implementation of models within a healthcare setting more convenient, it is necessary to present the final models in a user friendly way. For example, a mathematical equation, a score chart, or a nomogram. Furthermore, these models undergoing the continued external data test also improve their clinical application.
      • Heestermans T.
      • Payne B.
      • Kayode G.A.
      • et al.
      Prognostic models for adverse pregnancy outcomes in low-income and middle-income countries: A systematic review.
      ,
      • Haniffa R.
      • Mukaka M.
      • Munasinghe S.B.
      • et al.
      Simplified prognostic model for critically ill patients in resource limited settings in South Asia.
      Accordingly, the performance tests and presentations of models are required to report.
      In our study, only three models performed well in the domain of predictors. The main reason was that some predictors were unavailable when the model would be intended to be applied, such as duration of surgery, the volume of fluids, or duration of anesthesia. Healthcare staff usually estimate the risk of intraoperative hypothermia before surgery. However, the above data can be obtained after the operation is over. In such situations, it is hard for medical staff to use these models for prediction, and the applicability of models would be limited.
      For a prediction model to be applied to clinical settings, all enrolled predictors need to be obtained at the moment of prediction. Variables that cannot be obtained in time would inflate actual model values since such variables are measured closer in time to the outcome evaluation and are more likely to be strongly associated with the outcome.
      • Moons K.
      • Wolff R.F.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration.
      Hence, this kind of flaw in the predictor domain needs to be mentioned and solved. Previous research suggested that only variables that were readily accessible to healthcare professionals can be considered in derivation, and variables that were challenging to determine objectively should be excluded.
      • Heestermans T.
      • Payne B.
      • Kayode G.A.
      • et al.
      Prognostic models for adverse pregnancy outcomes in low-income and middle-income countries: A systematic review.
      However, this approach may undermine the predictive competency of models. For this shortcoming, we recommend consultation with clinical healthcare workers who have rich experience to identify relevant predictors, such as duration of surgery and the volume of fluids. Certainly, the accuracy of assessment depends on many factors, and the construction and validation of related assessment tools may be a reasonable starting point for further studies.

      Recommendations

      Given the high risk of bias surrounding the performance and methodology of published models, no models can be recommended for clinical practice at this time. However, the findings of this study can make valuable recommendations for the construction and improvement of further models. First, the sample size should be large enough to minimize overfitting. Thus, the EPV of at least 20 should be recommended and adopted. Second, samples with missing data should be handled appropriately instead of being directly excluded. Simply removing involved samples with incomplete information results in biased model performance since the analyzed individuals are a selective rather than a random sample of the original full sample. Hence, an appropriate method for handling missing data is recommended. According to Held et al.,
      • Held U.
      • Kessels A.
      • Garcia Aymerich J.
      • et al.
      Methods for handling missing variables in risk prediction models.
      the multiple imputation with fixed or random intercepts for cohorts was an optimal method to impute the systematical missing data since it minimized biased results with correct P values and standard errors.
      Third, future research should calibrate, extend, or update the existing models to strengthen their predictive performance rather than only constructing new models. And within this process, the additional evaluation value of certain predictors can be considered to enhance model performance. In addition, some new modeling methods can be explored, such as structural equation models and machine learning. Fourth, the EPVs should be clearly defined and accurately analyzed, and all final variables should be truly reported. Some assessment tools can be studied to assist researchers in predicting relevant variables. Finally, further prediction model research should be designed in a more user-friendly way, reported more clearly, and shown more completely. Widely recognized guidelines, including the TRIPOD Initiative
      • Collins G.S.
      • Reitsma J.B.
      • Altman D.G.
      • Moons K.G.
      Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement.
      and PROBAST tool,
      • Moons K.
      • Wolff R.F.
      • Riley R.D.
      • et al.
      PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration.
      can be a standard model to offer guidance. Additionally, healthcare institutions can integrate the models into electronic medical records, which can be convenient for practitioners to apply models and further facilitate model validation.
      Considering that no published hypothermia prediction model can be recommended for clinical use, future models are needed to determine the rigorous criteria of sample size to ensure the appropriate EPVs. Also, the complete model performance and presentation should be demonstrated following the standard modeling statements. Further studies are encouraged to validate and strengthen existing hypothermia risk prediction models rather than developing entirely new models.

      Strengths and limitations

      To our knowledge, this is the first systematic review that provides a comprehensive overview of hypothermia risk prediction models, evaluating their quality using the PROBAST tool. The results of this study can offer valuable insights for researchers to improve the models’ reporting quality. Nevertheless, some limitations of our review should be acknowledged and considered. First, we did not perform a meta-analysis due to the heterogeneity of the included research that met our study criteria, which limited the quantitative comparison of these models. Second, most research studies did not report model performance, which might impact these models’ predictive value assessment. These weakness are eagerly need improvement in further studies.

      Conclusions

      Our systematic review identified eight risk prediction models for inadvertent intraoperative hypothermia in surgical patients, and these models were reported following the guidance of the CHARMS and PROBAST statements. All the models had a high risk of bias, which undermines their prediction value in clinical practice. Accordingly, we encourage future studies to present models in a more standardized approach following relevant guidelines and include enough participants to validate models.

      Reference

        • Yi J.
        • Zhan L.
        • Lei Y.
        • et al.
        Establishment and validation of a prediction equation to estimate risk of intraoperative hypothermia in patients receiving general anesthesia.
        Scient Rep. 2017; 7: 13927https://doi.org/10.1038/s41598-017-12997-x
        • Yi J.
        • Lei Y.
        • Xu S.
        • et al.
        Intraoperative hypothermia and its clinical outcomes in patients undergoing general anesthesia: National study in China.
        PloS One. 2017; 12e0177221https://doi.org/10.1371/journal.pone.0177221
        • Leslie K.
        • Sessler D.I.
        • Bjorksten A.R.
        • Moayeri A.
        Mild hypothermia alters propofol pharmacokinetics and increases the duration of action of atracurium.
        Anesthesia Analgesia. 1995; 80: 1007-1014https://doi.org/10.1097/00000539-199505000-00027
        • Mason S.E.
        • Kinross J.M.
        • Hendricks J.
        • Arulampalam T.H.
        Postoperative hypothermia and surgical site infection following peritoneal insufflation with warm, humidified carbon dioxide during laparoscopic colorectal surgery: A cohort study with cost-effectiveness analysis.
        Surg Endosc. 2017; 31: 1923-1929https://doi.org/10.1007/s00464-016-5195-0
        • Choi J.W.
        • Kim D.K.
        • Kim J.K.
        • Lee E.J.
        • Kim J.Y.
        A retrospective analysis on the relationship between intraoperative hypothermia and postoperative ileus after laparoscopic colorectal surgery.
        PloS One. 2018; 13e0190711https://doi.org/10.1371/journal.pone.0190711
        • Giuliano K.K.
        • Hendricks J.
        Inadvertent perioperative hypothermia: Current nursing knowledge.
        AORN J. 2017; 105: 453-463https://doi.org/10.1016/j.aorn.2017.03.003
        • Kander T.
        • Schött U.
        Effect of hypothermia on haemostasis and bleeding risk: A narrative review.
        J Int Med Res. 2019; 47: 3559-3568https://doi.org/10.1177/0300060519861469
        • Lester E.
        • Fox E.E.
        • Holcomb J.B.
        • et al.
        The impact of hypothermia on outcomes in massively transfused patients.
        J Trauma Acute Care Surg. 2019; 86: 458-463https://doi.org/10.1097/TA.0000000000002144
        • Moons K.
        • Wolff R.F.
        • Riley R.D.
        • et al.
        PROBAST: A tool to assess risk of bias and applicability of prediction model studies: Explanation and elaboration.
        Ann Int Med. 2019; 170: W1-W33https://doi.org/10.7326/M18-1377
        • Carrillo-Larco R.M.
        • Altez-Fernandez C.
        • Pacheco-Barrios N.
        • et al.
        Cardiovascular disease prognostic models in Latin America and the Caribbean: A systematic review.
        Global Heart. 2019; 14: 81-93https://doi.org/10.1016/j.gheart.2019.03.001
        • Haider S.
        • Sadiq S.N.
        • Moore D.
        • Price M.J.
        • Nirantharakumar K.
        Prognostic prediction models for diabetic retinopathy progression: a systematic review.
        Eye (London, England). 2019; 33: 702-713https://doi.org/10.1038/s41433-018-0322-x
        • Kleimeyer J.P.
        • Harris A.
        • Sanford J.
        • Maloney W.J.
        • Kadry B.
        • Bishop J.A.
        Incidence and risk factors for postoperative hypothermia after orthopaedic surgery.
        J Am Acad Orthopaed Surg. 2018; 26: e497-e503https://doi.org/10.5435/JAAOS-D-16-00742
        • Emmert A.
        • Gries G.
        • Wand S.
        • et al.
        Association between perioperative hypothermia and patient outcomes after thoracic surgery: A single center retrospective analysis.
        Medicine. 2018; 97: e0528https://doi.org/10.1097/MD.0000000000010528
        • Li Y.
        • Liang H.
        • Feng Y.
        Prevalence and multivariable factors associated with inadvertent intraoperative hypothermia in video-assisted thoracoscopic surgery: A single-center retrospective study.
        BMC Anesthesiol. 2020; 20: 25https://doi.org/10.1186/s12871-020-0953-x
        • Moher D.
        • Shamseer L.
        • Clarke M.
        • et al.
        Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.
        Systematic reviews. 4. 2015: 1https://doi.org/10.1186/2046-4053-4-1
        • Moons K.G.
        • de Groot J.A.
        • Bouwmeester W.
        • et al.
        Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.
        PLoS Med. 2014; 11e1001744https://doi.org/10.1371/journal.pmed.1001744
        • Wolff R.F.
        • Moons K.
        • Riley R.D.
        • et al.
        PROBAST: A tool to assess the risk of bias and applicability of prediction model studies.
        Ann Int Med. 2019; 170: 51-58https://doi.org/10.7326/M18-1376
        • Palazón-Bru A.
        • Mares-García E.
        • López-Bru D.
        • et al.
        A critical appraisal of the clinical applicability and risk of bias of the predictive models for mortality and recurrence in patients with oropharyngeal cancer: Systematic review.
        Head Neck. 2020; 42: 763-773https://doi.org/10.1002/hed.26025
        • Kasai T.
        • Hirose M.
        • Yaegashi K.
        • Matsukawa T.
        • Takamata A.
        • Tanaka Y.
        Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia.
        Anesthesia Analgesia. 2002; 95https://doi.org/10.1097/00000539-200211000-00051
        • Kim E.J.
        • Yoon H.
        Preoperative factors affecting the intraoperative core body temperature in abdominal surgery under general anesthesia: An observational cohort.
        Clin Nurse Specialist CNS. 2014; 28: 268-276https://doi.org/10.1097/NUR.0000000000000069
        • Yang L.
        • Huang C.Y.
        • Zhou Z.B.
        • et al.
        Risk factors for hypothermia in patients under general anesthesia: Is there a drawback of laminar airflow operating rooms? A prospective cohort study.
        Int J Surg (London, England). 2015; 21: 14-17https://doi.org/10.1016/j.ijsu.2015.06.079
        • Rincón D.A.
        • Valero J.F.
        • Eslava-Schmalbach J.
        Construcción y validación de un modelo predictivo de hipotermia intraoperatoria [Construction and validation of a model to predict intraoperative hypothermia].
        Revista Espanola de Anestesiologia y Reanimacion. 2008; 55: 401-406https://doi.org/10.1016/s0034-9356(08)70610-8
        • Desgranges F.P.
        • Bapteste L.
        • Riffard C.
        • et al.
        Predictive factors of maternal hypothermia during Cesarean delivery: A prospective cohort study. Facteurs prédictifs d'hypothermie maternelle durant la Césarienne: une étude prospective de cohorte.
        Can J Anaesthesia = Journal Canadien D'anesthesie. 2017; 64: 919-927https://doi.org/10.1007/s12630-017-0912-2
        • Chalari E.
        • Intas G.
        • Zyga S.
        • et al.
        Preoperative factors affecting the intraoperative core body temperature in elective hysterectomy under general anesthesia.
        Clin Exp Obstetr Gynecol. 2019; 46: 560-564
        • Cai R.
        • Wu X.
        • Li C.
        • Chao J.
        Prediction models for cardiovascular disease risk in the hypertensive population: A systematic review.
        J Hypertension. 2020; 38: 1632-1639https://doi.org/10.1097/HJH.0000000000002442
        • Sami W.
        • Alrukban M.O.
        • Waqas T.
        • Asad M.R.
        • Afzal K.
        Sample size determination in health research.
        J Ayub Med Coll Abbottabad. 2018; 30: 308-311
        • Ogundimu E.O.
        • Altman D.G.
        • Collins G.S.
        Adequate sample size for developing prediction models is not simply related to events per variable.
        J Clin Epidemiol. 2016; 76: 175-182https://doi.org/10.1016/j.jclinepi.2016.02.031
        • Meier K.
        • Parrish J.
        • D'Souza R
        Prediction models for determining the success of labor induction: A systematic review.
        Acta Obstetricia et Gynecologica Scandinavica. 2019; 98: 1100-1112https://doi.org/10.1111/aogs.13589
        • Bell M.L.
        • Floden L.
        • Rabe B.A.
        • et al.
        Analytical approaches and estimands to take account of missing patient-reported data in longitudinal studies.
        Patient Rel Outcome Measures. 2019; 10: 129-140https://doi.org/10.2147/PROM.S178963
        • Collins G.S.
        • Reitsma J.B.
        • Altman D.G.
        • Moons K.G.
        Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement.
        BMJ (Clin Res ed.). 2015; 350: g7594https://doi.org/10.1136/bmj.g7594
        • Heestermans T.
        • Payne B.
        • Kayode G.A.
        • et al.
        Prognostic models for adverse pregnancy outcomes in low-income and middle-income countries: A systematic review.
        BMJ Global Health. 2019; 4e001759https://doi.org/10.1136/bmjgh-2019-001759
        • Haniffa R.
        • Mukaka M.
        • Munasinghe S.B.
        • et al.
        Simplified prognostic model for critically ill patients in resource limited settings in South Asia.
        Crit Care (London, England). 2017; 21: 250https://doi.org/10.1186/s13054-017-1843-6
        • Held U.
        • Kessels A.
        • Garcia Aymerich J.
        • et al.
        Methods for handling missing variables in risk prediction models.
        Am J Epidemiol. 2016; 184: 545-551https://doi.org/10.1093/aje/kwv346