|Year : 2022 | Volume
| Issue : 1 | Page : 19-24
External validation of 4C ISARIC mortality score in critically ill COVID-19 patients from Saudi Arabia
Waleed Tharwat Aletreby1, Shahzad Ahmad Mumtaz1, Saima Akhtar Shahzad1, Intekhab Ahmed1, Mohammed Ali Alodat1, Mohamed Gharba2, Zohdi Ahmed Farea1, Ahmed Fouad Mady3, Waqas Mahmood1, Huda Mhawish1, Majd Munia Abdulmowla4, Rehab Mohammed Nasser5
1 Critical Care Department, King Saud Medical City, Australia
2 Critical Care Department, The Wollongong Hospital, NSW, Australia
3 Critical Care Department, King Saud Medical City; Anesthesia Department, Faculty of Medicine, Tanta University, Tanta, Egypt
4 College of Medicine, Alfaisal University, Riyadh, Saudi Arabia
5 Internal Medicine Department, Prince Mohammed Bin Abdulaziz Hospital, Riyadh, Saudi Arabia
|Date of Submission||18-Aug-2021|
|Date of Decision||09-Nov-2021|
|Date of Acceptance||06-Dec-2021|
|Date of Web Publication||12-Jan-2022|
Waleed Tharwat Aletreby
Critical Care Department, King Saud Medical City, Riyadh
Source of Support: None, Conflict of Interest: None
Background: ISARIC mortality score is a risk stratification tool that helps predict the in-hospital mortality of COVID-19 patients. However, this tool was developed and validated in a British population, and thus, the external validation of this tool in local populations is important.
Objectives: External validation of the ISARIC mortality score in COVID-19 patients from a large Saudi Arabian intensive care unit (ICU).
Methods: This is a retrospective study that included all adult patients with COVID-19 admitted to the ICU of King Saud Medical City, Riyadh, Saudi Arabia, from March 2020 to June 2021. Patients who were pregnant or had pulmonary tuberculosis/human immunodeficiency virus were excluded along with patients with missing variables. Data were collected to calculate the ISARIC mortality score and then fitting receiver operator characteristic curve against patients' outcome.
Results: A total of 1493 critically ill COVID-19 patients were included. The mortality was 38%, the area under the curve of the score was 0.81 (95% confidence interval [CI]: 0.79–0.83, P < 0.001) and the cutoff value correctly classified 72.7% of the cohort. The cutoff value of >9 had sensitivity of 70.5% (95% CI: 66.6–74.3); specificity, 73.97% (95% CI: 71–76.8); positive predictive value, 62.4% (95% CI: 59.5–65.2) and negative predictive value, 80.2% (95% CI: 78.2–82.4).
Conclusion: The ISARIC score was found to have excellent predictive ability for mortality in critically ill COVID-19 patients in our Saudi Arabian cohort. A cutoff score of >9 was the optimal criterion.
Keywords: COVID-19, ISARIC, mortality, Saudi Arabia, survival, sensitivity and specificity
|How to cite this article:|
Aletreby WT, Mumtaz SA, Shahzad SA, Ahmed I, Alodat MA, Gharba M, Farea ZA, Mady AF, Mahmood W, Mhawish H, Abdulmowla MM, Nasser RM. External validation of 4C ISARIC mortality score in critically ill COVID-19 patients from Saudi Arabia. Saudi J Med Med Sci 2022;10:19-24
|How to cite this URL:|
Aletreby WT, Mumtaz SA, Shahzad SA, Ahmed I, Alodat MA, Gharba M, Farea ZA, Mady AF, Mahmood W, Mhawish H, Abdulmowla MM, Nasser RM. External validation of 4C ISARIC mortality score in critically ill COVID-19 patients from Saudi Arabia. Saudi J Med Med Sci [serial online] 2022 [cited 2023 Feb 7];10:19-24. Available from: https://www.sjmms.net/text.asp?2022/10/1/19/335657
| Introduction|| |
The COVID-19 pandemic has had a significant impact worldwide, with >262 million cases reported as of December 1, 2021.,,,, The spectrum of clinical presentation of COVID-19 infection is ample, ranging from being asymptomatic, which accounts for the majority of cases, to mild cases requiring minimal support, up to critically ill cases requiring intensive care unit (ICU) admission and mechanical ventilation. Life-threatening COVID-19 presentations are commonly characterized by acute respiratory distress syndrome, multiorgan failure, hyperinflammatory responses and thromboembolic manifestations, among others. Therefore, it is comprehendible that the mortality rate in severe and critically ill COVID-19 patients is high, particularly when complemented with poor prognostic factors such as comorbidities, elevated inflammatory biomarkers and old age.,
The variability in presentation necessitated the development of risk stratification tools that would allow early identification of COVID-19 patients at higher risk of mortality, using readily available objective criteria.,, Accordingly, Knight et al. utilized The International Severe Acute Respiratory and emerging Infections Consortium (ISARIC) World Health Organization Clinical Characterization Protocol to develop and validate such a tool, the ISARIC 4C mortality score (hereafter referred to as ISARIC), which likely outperforms other risk stratification models. The ISARIC mortality score utilizes variables that are readily available on hospitalization, thereby avoiding reliance on parameters such as radiological imaging or those that only become available after ICU admission., The model has been demonstrated to have a high discriminatory ability for in-hospital mortality and categorizes patients into four categories of severity with a uniformly increasing mortality risk [Supplementary Table 1].
However, the tool's development and validation were primarily done for the population in the United Kingdom. For generalizability, there is a need to validate the score within different populations. To the best of our knowledge, no study from Saudi Arabia has previously validated this tool, and thus, the current study was performed to validate the performance of the ISARIC score within the setting of a Saudi Arabian ICU.
| Methods|| |
Study design, patients, and setting
This is a retrospective study that included all adults with COVID-19 admitted to the ICU of King Saud Medical City, Riyadh, Saudi Arabia, between January 1, 2020, and June 30, 2021; the first case was recruited on March 20, 2020. The study was approved by the hospital's institutional review board.
King Saud Medical City is a 1200-bed government hospital, with an ICU capacity that has been increased from 127 to 300 beds since the beginning of the COVID-19 pandemic. The ICU is closed, operated round the clock by intensivists and has a nurse:patient ratio of 1:1. For the pandemic, almost half of the beds were converted to single rooms, and all beds are fully equipped with capabilities of invasive and noninvasive monitoring and ventilation. The admission criteria of COVID-19–positive patients to the ICU were broad, as the center serves as a COVID-19 referral center of the region. COVID-19 was confirmed by nasopharyngeal reverse transcription–polymerase chain reaction (RT-PCR), and patients were admitted if they had any of the following: supplemental oxygen of any kind to maintain peripheral oxygen saturation (SpO2) ≥92%, mechanically ventilated, any level of consciousness disturbance or respiratory rate ≥28 per minute. The criteria were the same during both waves of COVID-19 and in line with the COVID-19 management recommendations of the Saudi Ministry of Health.
Inclusion and exclusion criteria
All adult (≥18 years) patients with a nasopharyngeal RT-PCR–confirmed COVID-19 admitted to the ICU of our hospital were included in this study. Nasopharyngeal RT-PCR was repeated for all patients who previously tested positive either before ICU discharge or after 2 weeks of ICU admission, whichever occurred first.
Pregnant women as well as those known to have pulmonary tuberculosis (PTB) and human immunodeficiency virus (HIV) were excluded from this study, as common COVID-19 medications such tocilizumab or antiviral drugs may not be offered to these patients. After the completion of data collection, patients with missing variables that preclude the calculation of ISARIC score were also excluded.
For calculating the ISARIC score [Supplementary Table 2], the following variables were collected from the electronic database of our ICU and/or manually from the patients' medical records: age; gender; number of comorbidities; respiratory rate, SpO2 on room air, and Glasgow coma scale (GCS) at hospital admission; first available blood urea level (mmol/L); and C-reactive protein (CRP) (mg/L). In addition, ethnicity, smoking status, and nature of comorbidities were also noted. The collection of data was performed by a group of data collectors supervised by authors: IA, ZS, MA and WA. The electronic records of the hospital were used by the data collectors to gather demographic data, COVID-19 test and laboratory results along with all relevant dates such as admission and discharge, while data of the condition upon hospitalization (such as GCS and SpO2) were gathered manually from the patients' files, and the aforementioned authors verified the same.
The primary outcome of the study was the performance of ISARIC score in our settings by evaluating its discriminatory ability of survivors and deceased in the all-cause hospital mortality outcome. Secondary outcomes included comparison between subjects with score above and below the optimal criterion derived from the analysis with regard to mortality rate, ICU length of stay (LOS) and survival analysis.
Statistical tests were performed using the STATA® statistical software (Release 16; StataCorp LLC, College Station, TX, USA). For descriptive purposes, the demographic and clinical characteristics of survivors and nonsurvivors are presented separately. Continuous variables are summarized as mean ± standard deviation and discrete variables as frequency and percentages. Continuous variables were compared by Student's t-test or Wilcoxon rank-sum test, as appropriate, while discrete variables were compared by Chi-square test or Fisher's exact test, as appropriate. Comparative results are presented with corresponding 95% confidence interval (CI) and P value.
The primary outcome was evaluated by fitting receiver operator characteristic (ROC) curve using known outcomes as reference variable and ISARIC score as classification variable, through 1000 bootstrap with replacements. The area under the curve (AUC) and P value are presented. Furthermore, the analysis identified the optimal criterion associated with Youden's index, which was used as a cutoff value for the ICU survival analysis, presented as Kaplan–Meier curve and log-rank P value, in addition to sensitivity, specificity, positive predictive values (PPV) and negative predictive value (NPV), and a 2 × 2 contingency table. All tests were two tailed, and P < 0.05 was considered statistically significant.
| Results|| |
During the study period, there were 1749 ICU admissions of confirmed COVID-19 [Figure 1]. Of these, 32, 12, 5 and 2 patients were excluded because they were aged <18 years, were pregnant, had PTB and were HIV positive, respectively. Another 205 (11.7%) patients were excluded due to missing CRP values precluding score calculation. Accordingly, the study finally included 1493 patients.
[Table 1] depicts the demographic and clinical characteristics of the cohort and comparison between survivors and nonsurvivors. The majority of the patients were Asians (42%) followed by Saudis (28.2%) and Middle Eastern (24.4%). There were 567 (38%) hospital deaths, with a mean age of 53.2 ± 14.1 years and the majority (76.3%) being males.
|Table 1: Demographic and clinical characteristics of the cohort and comparison between survivors and nonsurvivors|
Click here to view
Survivors had significantly different values than nonsurvivors in all variables used in the calculation of ISARIC score, in the direction of a lower score, with the exception of gender distribution, and significantly lower prevalence of all recorded comorbidities, except for chronic kidney disease, ischemic heart disease and smoking. Only Saudis and Asians had significantly higher survival than mortality rates. The average ICU LOS of the cohort was 10.5 ± 9 days, being significantly shorter for survivors compared to nonsurvivors (8.1 ± 7.2 vs. 14.3 ± 10.2, 95% CI of difference: −7 to −5.3, P < 0.001). The cohort had a mean ISARIC score of 8.6 ± 5.3; survivors had significantly lower score than nonsurvivors (6.3 ± 4.3 vs. 12.4 ± 4.3, 95% CI of difference: −6.5 to −5.6; P < 0.001). [Supplementary Figure 1] shows the numbers of patients, mortality rate and median (interquartile range) ISARIC score for different modes of supplemental oxygen.
The fitted ROC curve of ISARIC score against known outcomes had AUC of 0.81 (95% CI: 0.79–0.83, P < 0.001) [Figure 2]. The associated criterion with Youden's index was a score of >9; accordingly, survival based on that cutoff value estimated by the Kaplan–Meier curve showed a median survival of 22 days with ISARIC scores ≤9 and 16 days with scores >9, and the associated log-rank test was highly significant [Figure 3]. The sensitivity and specificity with scores >9 were 70.5% (95% CI: 66.6–74.3) and 73.8% (95% CI: 71–76.8), respectively, the PPV and NPV was 62.4% (95% CI: 59.5–65.2) and 80.4% (95% CI: 78.2–82.4), respectively, and the cutoff value correctly classified 72.7% of the cohort [Supplementary Table 3].
|Figure 2: Area under the curve of 4C ISARIC score discriminatory ability. Area under the curve = 0.81 (95% confidence interval: 0.79–0.83, P < 0.001)|
Click here to view
|Figure 3: Kaplan–Meier survival based on cutoff of >9. Significantly higher survival for patients with 4C ISARIC score <9|
Click here to view
Dividing the enrolled patients into two groups based on ISARIC score of ≤9 and >9 showed significantly higher mortality in the >9 group compared to ≤9 (62.4% vs. 19.6%, 95% CI: 38%–47.4%; P < 0.001). Similarly, the group with a score of ≤9 had ICU LOS of 9.1 ± 8.2 days, which was significantly shorter than that of the group with score >9 (LOS: 12.3 ± 9.7 days) (95% CI: 2.3–4.1, P < 0.001) [Supplementary Table 4].
| Discussion|| |
In the population of our study, the ISARIC score was found to have excellent predictive ability for mortality risk, with an AUC of 0.81 (95% CI: 0.79–0.83, P < 0.001). This study excluded cases with missing data (11.2%) precluding calculation of ISARIC score, although modern imputation methods (such as multiple imputation) are likely to produce valid estimates; however, only when critical assumptions have been met, which may have not been true for our data, they still carry the potential to cause bias. Enrolled patients in our analysis were of multiple ethnicities, representing the general population of Saudi Arabia, and thus providing a good external validation cohort for the ISARIC score.
The AUC in our study was slightly higher than that reported by the original development and validation study (AUC = 0.786; 95% CI: 0.781–0.79). Although this is uncommon in external validation studies, higher AUCs were similarly reported by van Dam et al. (AUC = 0.84; 95% CI: 0.79–0.88) and Wellbelove et al. (AUC = 0.83; 95% CI: 0.71–0.95). Other studies have reported AUC values similar to the validation study or lower. This variation among studies – although minimal – utilizing the same prediction model may be a reflection of the variations in the studied populations, with regard to their demographic characteristics, clinical severity and sample size. For example, in our study, various ethnic groups were included, the age range was wide (from 18 to 104 years) and there was an unequal gender distribution (males: 76.3%). Furthermore, our cohort included only patients admitted to the ICU, which means that they presented a more critical picture of the disease, while the original study enrolled all hospital admissions. Regardless of those variations, all studies in the literature, including ours, found the discriminatory ability of ISARIC score to be acceptable or excellent.
Our external validation, similar to the original study, showed rising mortality rates across groups of severity [Supplementary Figure 1] and [Supplementary Table 1], that is, a directly proportional relationship between mortality risk and increase in score. This reflects that the model performs optimally, especially when considering that the higher mortality rates were higher within all groups in our study compared with the original study. In our analysis, the optimal cutoff value associated with Youden's index was a score >9, with this value correctly classifying 72.7% of the cohort.
The diagnostic parameters (sensitivity, specificity, PPV and NPV) of the cutoff of >9 in our model were considerably lower compared to the same cutoff value in the British study. This could possibly be due to inclusion of only critically ill COVID-19 patients, which makes predicting outcome more difficult than mild cases, given the larger number of complications that could be the indirect cause of death. This is also reflected in the fact that our study had a higher mortality rate compared with the study by Knight et al. (38% vs. 32.2%, respectively). One of the most important diagnostic parameters is NPV, which indicates the probability of survival in patients with scores ≤9. In our model, NPV was 80.2% (95% CI: 78.2–82.4), which provides a reasonable risk probability to guide clinical decision making; this discrimination was supported by our finding of a significantly higher survival of patients with scores ≤9, shorter ICU LOS, and lower mortality.
This study has several limitations. First, the inherent limitations of the retrospective study design are applicable. Second, this was a single-center study, reflecting the management in only our ICU, and thus, our results may not be extrapolated to other centers in Saudi Arabia or non-ICU patients. Third, not using imputation methods on missing data may have reduced our sample size and impacted our results. Fourth, despite working with a quite diverse population, our cohort lacked several ethnicities, and thus, our results may not be applicable in other regions of the world. Finally, the generalizability of our results may be limited to only critically ill patients, as non-ICU admissions were not included. Further multicenter studies with a larger sample size and including those with varied severities are required to validate the score in the larger Saudi population and possibly explore predictors of mortality in COVID-19 patients.
| Conclusion|| |
The ISARIC score was found to have excellent predictive ability for mortality in critically ill COVID-19 patients in our Saudi Arabian cohort. A cutoff score of >9 was the optimal criterion.
The study was approved by the institutional review board of King Saud Medical City, Riyadh (reference no.: H1R1-24-Feb21-04, dated: April 8, 2021). The requirement of patient consent was waived in view of the retrospective study design. The study followed the general principles outlined by the Declaration of Helsinki, 2013.
Data availability statement
The datasets generated during and/or analyzed during the current study are not publicly available, but are available from the corresponding author on reasonable request.
This article was peer-reviewed by two independent and anonymous reviewers.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Aletreby WT, Alharthy AM, Faqihi F, Mady AF, Ramadan OE, Huwait BM, et al
. Dynamics of SARSCoV2 outbreak in the Kingdom of Saudi Arabia: A predictive model. Saudi Crit Care J 2020;4:79-83. [Full text]
Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, et al
. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med 2020;382:727-33.
Ali R, Qayyum F, Ahmed N, Haroon MZ, Irshad R, Sajjad S, et al
. Isaric 4c mortality score as a predictor of in-hospital mortality in COVID-19 patients admitted in ayub teaching hospital during first wave of the pandemic. J Ayub Med Coll Abbottabad 2021;33:20-5.
World Health Organization. WHO coronavirus (COVID-19) dashboard. Available from: https://covid19.who.int/
. [Last accessed on 2021 Dec 01].
Mady A, Aletreby W, Abdulrahman B, Lhmdi M, Noor AM, Alqahtani SA, et al
. Tocilizumab in the treatment of rapidly evolving COVID-19 pneumonia and multifaceted critical illness: A retrospective case series. Ann Med Surg (Lond) 2020;60:417-24.
Alharthy A, Aletreby W, Faqihi F, Balhamar A, Alaklobi F, Alanezi K, et al
. Clinical characteristics and predictors of 28-day mortality in 352 critically ill patients with COVID-19: A retrospective study. J Epidemiol Glob Health 2021;11:98-104.
Knight SR, Ho A, Pius R, Buchan I, Carson G, Drake TM, et al
. Risk stratification of patients admitted to hospital with COVID-19 using the ISARIC WHO Clinical characterisation protocol: Development and validation of the 4C mortality score. BMJ 2020;370:m3339.
van Dam PM, Zelis N, van Kuijk SM, Linkens AE, Brüggemann RA, Spaetgens B, et al
. Performance of prediction models for short-term outcome in COVID-19 patients in the emergency department: A retrospective study. Ann Med 2021;53:402-9.
Covino M, De Matteis G, Burzo ML, Russo A, Forte E, Carnicelli A, et al
. Predicting in-hospital mortality in COVID-19 older patients with specifically developed scores. J Am Geriatr Soc 2021;69:37-43.
Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al
. Prediction models for diagnosis and prognosis of COVID-19: Systematic review and critical appraisal BMJ 2020;369:m1328.
Song Y, Zheng S, Li L, Zhang X, Zhang X, Huang Z, et al
. Deep learning enables accurate diagnosis of novel coronavirus (COVID-19) with CT images. IEEE/ACM Trans Comput Biol Bioinform 2021;18:2775-80.
Tolksdorf K, Buda S, Schuler E, Wieler LH, Haas W. Influenza-associated pneumonia as reference to assess seriousness of coronavirus disease (COVID-19). Euro Surveill 2020; 25 (11):2000258. [doi: 10.2807/1560-7917.ES.2020.25.11.2000258].
Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol 2010;5:1315-6.
Newgard CD, Haukoos JS. Advanced statistics: Missing data in clinical research – Part 2: Multiple imputation. Acad Emerg Med 2007;14:669-78.
Mittag N. Imputations: Benefits, Risks and a Method for Missing Data (Harris School of Public Policy, University of Chicago, 1155 E. 60th
Street, Chicago, IL): Harris School Of Public Policy, University of Chicago; 2013. Available from: https://home.cerge-ei.cz/mittag/papers/Imputations.pdf
. [Last accessed on 2021 Aug 13].
Wellbelove Z, Walsh C, Perinpanathan T, Lillie P, Barlow G. Comparing the 4C mortality score for COVID-19 to established scores (CURB65, CRB65, qSOFA, NEWS) for respiratory infection patients. J Infect 2021;82:414-51.
Verma AA, Hora T, Jung HY, Fralick M, Malecki SL, Lapointe-Shaw L, et al
. Characteristics and outcomes of hospital admissions for COVID-19 and influenza in the Toronto area. CMAJ 2021;193:E410-8.
[Figure 1], [Figure 2], [Figure 3]