Establishment and validation of a dynamic nomogram to predict short-term prognosis and benefit of human immunoglobulin therapy in patients with novel bunyavirus sepsis in a population analysis study: a multicenter retrospective study

Abstract

Objective

This study aims to develop a dynamic nomogram model using machine learning to improve short-term prognosis prediction and identify patients who would benefit from intravenous immunoglobulin (IVIG) therapy.

Methods

A multicenter retrospective study was conducted on 396 patients diagnosed with SFTS. Univariate and multivariate Cox regression analyses identified significant predictors of mortality. Machine learning models, including Random Survival Forest, Stepwise Cox Modeling, and Lasso Cox Regression, were compared for their predictive performance. The optimal model, incorporating consciousness, LDH, AST, and age, was used to construct a dynamic nomogram. The nomogram’s performance was validated in training, validation, and external test sets. Additionally, the impact of IVIG therapy on survival was assessed within high-risk groups identified by the nomogram.

Results

The dynamic nomogram demonstrated excellent predictive performance with an AUC of 0.903 in the training set, 0.933 in the validation set, and 0.852 in the test set, outperforming SOFA and APACHE II scores. Calibration curves confirmed the model’s accuracy. In the high-risk group, the hazard ratio (HR) for death for those who injected immunoglobulin versus those who did not was 0.569 (95% CI 0.330–0.982) in the nomogram model.

Conclusion

The dynamic nomogram effectively predicts short-term prognosis and identifies the population that benefits from IVIG therapy in patients with novel bunyavirus sepsis. This tool can aid clinicians in risk stratification and personalized treatment decisions, potentially improving patient outcomes.

Introduction

Severe fever with thrombocytopenia syndrome (SFTS) is an emerging infectious disease caused by a novel Bunyavirus. It is characterized by fever, low peripheral blood leukocyte and platelet counts, and multi-organ failure [1]. This disease, also known as novel bunyavirus sepsis, has a mortality rate of 20–30% and poses significant challenges in clinical management [2].

From a critical care perspective, sepsis—defined as organ dysfunction caused by a dysregulated host response to infection—provides a useful framework for understanding the severe illness caused by this virus [3, 4]. In clinical practice, predictive models are essential tools for stratifying patients based on their risk of adverse outcomes. These models facilitate better treatment decisions, more effective surveillance, and improved implementation of risk reduction strategies. Traditional scoring systems, such as the Sequential Organ Failure Assessment (SOFA) and the Acute Physiology and Chronic Health Evaluation II (APACHE II), have been widely used for prognostic prediction and stratification in critically ill patients [5,6,7]. However, these systems have notable limitations. The SOFA score, while simple and broadly applicable, may no longer be as effective in the context of modern clinical practice. The APACHE II score, which assesses mortality risk based on data collected at admission or during the first 24 h in the ICU, is not suitable for rapid assessment due to the extensive clinical data required [8,9,10,11].

Moreover, recent advancements in machine learning offer new opportunities for developing more accurate and personalized predictive models [12, 13]. Machine learning techniques, such as Random Survival Forests and Lasso regression, have been shown to improve predictive performance and enable personalized predictions in various clinical settings [14, 15]. These methods have been applied to cardiovascular system diseases and sepsis prognosis, but their use in novel bunyavirus sepsis remains limited. Previous studies have explored the application of machine learning in predicting the prognosis of SFTS, but they often focused on epidemiological aspects and lacked a comprehensive, clinically useful scoring system [16,17,18].

Another critical aspect of managing SFTS is the identification of effective treatments. Currently, there is no specific antiviral therapy for SFTS. Intravenous immunoglobulin (IVIG) has garnered attention for its immunomodulatory effects, with some studies suggesting potential benefits in treating SFTS [19]. However, the populations that may benefit most from IVIG therapy remain poorly defined.

Given these gaps in knowledge and clinical practice, the aim of this study was to develop and validate a dynamic nomogram model to predict the short-term prognosis of patients with novel bunyavirus sepsis. We compared the effectiveness of various machine learning models and traditional scoring systems (SOFA and APACHE II) to identify the optimal approach. Additionally, we sought to identify patient populations that would benefit most from IVIG therapy, thereby informing clinical decision-making and potentially improving patient outcomes.

Methods

Study population

A total of 302 patients admitted to Nanjing Second Hospital between June 2016 and September 2023 and 104 patients from The First Affiliated Hospital of Wannan Medical College in 2023 were included in this study. Their clinical and laboratory data at admission were retrospectively analyzed. Patients with other virus infections or serious chronic diseases were excluded. SFTS patients were diagnosed based on the presence of acute fever (with a temperature of 38 °C or higher) and platelet count < 100 × 10⁹/L), with lab-confirmed SFTS virus (SFTSV) infection by qRT-PCR. The study protocol was approved by the ethics committee of The Second Hospital of Nanjing. We randomized the study population into a train set and a validation set (7:3), 104 patients from The First Affiliated Hospital of Wannan Medical College as an external test set. Their demographics and treatments were balanced between the two sets. The derivation set were divided into survival and death groups.

Data collection

In this retrospective study, the clinical and laboratory (blood routine, biochemistry, coagulation) data of the 406 patients diagnosed with SFTS on admission were collected from the Second Hospital of Nanjing and The First Affiliated Hospital of Wannan Medical College Case Data System. Patients from The Second Hospital of Nanjing were used as Unit1 group with a total of 292 cases as the model training and validation set, and patients from The First Affiliated Hospital of Wannan Medical College were used as Unit2 group with a total of 104 cases as the external test set. The patients were categorized into survival and death groups. (Fig. 1) Data on demographics, clinical symptoms, laboratory tests, and treatments were collected. APACHEII score and SOFA score were calculated during the first 24 h after admission to the intensive care unit. The primary outcome was death from all causes during hospitalization.

Fig. 1

Study flow chart

Construction of the dynamic nomogram

To develop a robust predictive model for short-term prognosis in patients with novel bunyavirus sepsis, we employed several statistical and machine learning techniques. These methods were chosen based on their unique advantages in handling complex datasets and identifying significant predictors of mortality:

• 1. Univariate and Multivariate Cox Regression Analyses: These traditional survival analysis methods were used to identify significant predictors of mortality. They provide interpretable results and allow for the assessment of the proportional hazards assumption, which is crucial for survival data.

• 2. Stepwise Cox Modeling: This method iteratively selects variables based on statistical significance, balancing model complexity and predictive power. It helps in identifying the most relevant predictors while avoiding overfitting.

• 3. Lasso Cox Regression: Lasso regression is particularly useful for datasets with many correlated variables. It performs variable selection by shrinking less important coefficients to zero, resulting in a more parsimonious model. This method is advantageous for handling multicollinearity and improving model interpretability.

• 4. Random Survival Forest: This machine learning technique is robust to non-linear relationships and interactions between variables. It can handle large datasets and complex interactions without requiring strict assumptions about the underlying data distribution.

By comparing these methods, we aimed to select the optimal model that best balances predictive accuracy and clinical interpretability. The final dynamic nomogram was constructed using the most significant predictors identified through these analyses.

Firstly, In the training sets, patients in the death and survival groups were compared at baseline, One-way COX regression analysis was performed for each factor included, and variables that were statistically significant (P < 0.05) or clinically valuable according to professional judgment were included in stepwise COX regression analysis, lasso regression analysis, and Random Forest Survival Analysis models, whereas multifactorial COX regression analysis was performed to screen for independent influences. Correlation coefficients were applied to assess the covariance between the variables involved in the construction of the model. Dynamic nomograms based on Occam's Razor's Law were obtained in the training set and developed into an APP application that can be accessed at this URL https://ykahh.shinyapps.io/DynNomapp/. The optimal model should be the one that achieves the best performance with the fewest or most accessible variables [20].Mortality risk stratification based on Nomogram scores was performed to facilitate clinician assessment of the application.

Validation of the nomogram

The performances of the nomogram, SOFA and APACHE II Score were compared in terms of AUC in the training, validation sets and test sets. Calibration curves to assess the accuracy of model predictions. The C-index and the Integration of the Brier Score (IBS) of the Nomogram model in the training set, validation set and external test set.

Immunoglobulin efficacy analysis

Clinical survival outcomes were assessed by the Kaplan–Meier analysis and prognostic groups were compared by the log-rank test. Hazard Ratio (Hazard Ratio) to assess the proportion of risk of death in the treated vs. non-treated group.

Statistical analysis

Statistical analyses were performed using the R software (version4.3.1 R Foundation for Statistical Computing, Vienna, Austria). In the univariate analysis, categorical data were analyzed by x² test. Continuous data in a normal distribution were analyzed by independent t test and presented as $\overline{X}$ ± s. Continuous data in a non-normal distribution were analyzed by the Mann Whitney test and their medians were compared (P25, P75). Pearson's chi-square test and Fisher's exact test were used where appropriate when comparing differences in proportions between groups.

Missing values were addressed with multiple imputation in the process of logistic regression and model construction with five interpolations. The imputation technique involves creating multiple viral load of the data and replacing missing values with imputed values through a suitable random sample from their predicted distribution. Independent risk factors were identified by three algorithms Cox regression analysis and multivariate Cox regression analysis to predict mortality. The stepwise Cox regression analysis model introduces and excludes variables iteratively using forward selection and backward elimination. The LASSO regression analysis model applies L1 regularization to penalize regression coefficients, thereby performing variable selection and coefficient shrinkage, and employs tenfold cross-validation to determine the optimal penalty parameter, using lambda.1se as the best λ value. The random survival Cox model constructs the model after optimizing the number of nodes. Correlation coefficient plots were used for multifactor covariance determination. Area under the curve of the receiver operating characteristic (AUROC) and calibration curves for model performance evaluation. Nomogram construction based on optimal modeling. The X-tile software (version 3.6.1, Yale School of Medicine) was used for the analysis of optimal cutoff values [21]. All statistical tests were two-sided with a statistical significance level set at P values of < 0.05. The main packages R of the study design are gtsummary, tidyverse, survminer, randomForestSRC, nomogramFormula, survex and ggplot2.

Ethics approval

The study was approved by The Second Hospital of Nanjing ethics committees (2023-L-S-023).

Results

Patient description

During the study period, 396 patients (Unit1: 292, Unit2:104) were diagnosed with SFTS, of whom 98 died at a median time of 4 days after admission. The overall mortality rate was 24.7% (Table 1), with as many as 79 deaths within 7 days of admission in the mortality group (80.6%). Viral load, IgM and IgG antibodies not measured in external test set patients.

Table 1 Demographic and clinical characteristics of patients

Table 2 shows that there was no significant difference between the training and validation groups in terms of population epidemiologic characteristics and clinical treatment strategies. (Table 2).

Table 2 linical baseline comparison of patients in the training and validation groups

Feature selection and nomogram modeling

Univariate Cox regression analysis showed that advanced age, presence of altered consciousness, high level of APACHEII score and SOFA score, high viral load, high heart rate, application of antihypertensive drugs, need for hormonal application, need for anti-bacterial and fungal infections, need for renal replacement and respiratory therapy and elevated levels of U, TBil, ALT, AST, LDH, ALP, Cr, and D. Dimer were the lethal risk factors for outcome, and high levels of platelets, human albumin and PNI were protective factors. (Table 3) The above variables, in addition to APACHEII and SOFA scores, were used for modeling of the three algorithms. Stepwise Cox regression analysis model using stepwise forward and backward analysis (AIC = 474.78) and multifactorial Cox survival analysis (a < 0.05) screened the influencing factors involved in model construction including consciousness, antifungal, CRRT, U, ALT and D-Dimer, and the Lasso Cox regression analysis model selected lambda.1se (0.147) as the optimal λ value (Fig. 2A and B), the corresponding influencing factors include consciousness, LDH and AST, the optimal number of nodes for stochastic survival analysis was 4 (Fig. 2D), and the influencing factors involved in the model construction were consciousness, ALT, AST and viral load and rank the variables in order of importance. (Fig. 2C) According to the clinical practice, the factor of age was included in the construction of the three models mentioned above. Multifactorial Cox analysis showed early altered consciousness was independent risk factor for lethal outcome. The heat map of correlation coefficients shows that the variables are not correlated and the VIF values of the variables are less than 5. (Table 4, Fig. 3 Correlation coefficient graph).

Table 3 Single-factor cox survival analysis

Fig. 2

Cox_model_Lasso (A/B) and Cox_model_rfsrc (C/D)

Table 4 Multi-factor Cox regression of three model screening outcome variables and VIF values for each factor

Fig. 3

Correlation coefficient graph

Stepwise Cox regression model, lasso Cox regression model and stochastic survival analysis model based on independent influences and Cox hypothesis testing (Fig. 4). The global Schoenfeld tests for the stepwise Cox regression model, the lasso Cox regression model, and the stochastic survival analysis model were 0.0109, 0.7729, and 0.04393, respectively. We therefore chose the variables included in the the lasso Coxregression model, i.e., consciousness, LDH and AST, and added age to construct a 7-day Nomogram model for survival analysis. (Fig. 5) Comparison of the importance of age, consciousness, LDH, and AST in the training set, validation set, and test set models reveals consciousness as the most important indicator (Fig. 6) The five variables mentioned above were involved in the construction of a dynamic predictive model, which was deployed on the website at https://ykahh.shinyapps.io/DynNomapp/.

Fig. 4

Shows the results of the Cox hypothesis tests for the three machine learning models, where the P-value of the Lasso Cox regression model is greater than 0.05, indicating that it satisfies the Cox proportional hazards assumption

Fig. 5

Shows the 7-day survival prediction nomogram based on the Lasso Cox regression model, which includes the four variables: level of consciousness, lactate dehydrogenase, aspartate aminotransferase, and age

Fig. 6

Compares the importance of the four variables in the prediction model: level of consciousness, aspartate aminotransferase, lactate dehydrogenase, and age, where level of consciousness is the most important indicator. Comparison of' Variable lmportance (the Training Set (A), Validation Set (B), and Test Set (C)

Evaluation and validation of machine learning models

In the Training set, the average Time-AUC values of Nomogram model, Cox_model_SOFA and Cox_model_APACHEII were 0.903, 0.0.83, 0.745 in 7 days; Validation set:0.933, 0.824, 0.794; Test set: 0.852, 0.908, and 0.86 (Fig. 7, see Appendix 1 for specific values).Comparison of the AUC size of the Nomogram model with cox_model_SOFA and cox_model_APACHEII in the Test set showed no statistical difference (P > 0.05), respectively (Fig. 8, see Appendix 2 for specific values). The C-index of the Nomogram model in the training set, validation set and external test set are 0.878, 0.883 and 0.821 respectively. The Integration of the Brier Score (IBS) of the Nomogram model is 0.089, 0.066 and 0.07 for the training set, the validation set and test set, respectively. Calibration curves were depicted for both of the training, validation sets and test sets, and the calibration plot revealed good predictive accuracy between the actual probability and predicted probability. (Fig. 9).

Fig. 7

Time-AUC (A: Train, B: Validation, C: Test)

Fig. 8

Comparison of the AUC size of the Nomogram modelwith cox model SOFA (A) and cox model APACHEII (B) in the Test set (P > 0.05)

Fig. 9

Calibration curves (A: Train set, B: Validation set, C: Test set)

Exploring the population benefits of human IVIG

We applied X-title software to explore the cutoff values for age, AST, LDH and SOFA scores, which were 63, 489.9, 1494, and 7, respectively. (Appendix 3) Greater than or equal to the cutoff value was considered a high-risk group, and vice versa for low-risk group. In a cohort of 396 patients, the median follow-up times for the high-risk and low-risk groups were 24 days and 11 days, respectively. The 7-day survival probabilities were 24% (95% CI 15–39%) and 88% (95% CI 85–92%) for the high-risk and low-risk groups, respectively. The median survival time for the high-risk group was 3 days. The Nomogram prediction model constructed from the development cohort was used to classify the development cohort, the external test cohort and all patients into a low-risk population and a high-risk population, respectively, and survival curves were plotted and log-rank tests were performed. From the results of the analysis, it was known that there was a significant difference (P < 0.05) between the survival curves of the two populations (Fig. 10). In the total population group, in the high and low risk subgroups of the Nomogram prediction model and the SOFA prediction model, survival curves were plotted according to whether immunoglobulin was applied or not, and log-rank tests were performed. From the results of the analyses, it was found that there was a significant difference (P < 0.05) between the survival curves of the high-risk groups only in both models (Fig. 11), and hazard ratio (HR) for death for those who injected immunoglobulin versus those who did not was 0.569 (95% CI 0.330–0.982) in the Nomogram prediction model (P < 0.05). In the SOFA prediction model, the hazard ratio (HR) of death for non-applicants was 1.938 (95% CI 1.137–3.305) (P < 0.05). Further exploring the significant difference between survival curves with and without immunoglobulin application in the awareness-only group in the high-risk group according to the cutoff values of each variable in the Nomogram prediction model (Fig. 12) and the hazard ratio (HR) of death for those who did not apply was 1.868 (95% CI 1.168–2.987) (P < 0.05).

Fig. l0

Survival curves by risk group Overall survial curves for the Train set (A), Test set (B) and Complete set (C) by cutoff point

Fig. 11

Suryival curves by immunoglobulin groups Overall surivival curves for Nomonpram modle (A) and SOFA modle (B)

Fig. 12

Survival curves by immunoglobulin groups Overall surivival curves for each variable of Nomongram modle (A: age, B: consciousness, C: AST, D: LDH)

Discussion

In this study, a dynamic Nomogram modle for predicting the short-term prognosis of STFS patients was developed and deployed on a website for reference application by clinicians. After risk stratifying the population, the human immunoglobulin benefit population was further explored.

We screened the factors that satisfy the Cox proportional risk assumption (Fig. 4) by applying three machine learning methods for the construction of Nomogram modle. The heat map of correlation coefficients shows that the variables are not correlated and the VIF values of the variables are less than 5. (Table 4, Fig. 3). The final independent risk factors were age, awareness, LDH, and AST (Fig. 5), and the importance of the above variables the above four factors contributed to the predictive ability of the model in the order of awareness, AST, LDH, and age, and had consistent performance in the training set, validation set, and external test set (Fig. 6). Studies have also shown that CNS manifestations (adjusted odds ratio [OR] 30.26) are important risk factors for the fatal outcomes of SFTS patients, and have been recognized by the Ministry of Health of the People's Republic of China as an important clinical manifestation for identifying critically ill patients. It has also been recognized by the Ministry of Health of the People's Republic of China as an important clinical manifestation for the identification of critically ill SFTS patients [22, 23]. The pathogenic mechanism of encephalitis in patients with SFTS is unclear, and studies have shown that the virus is able to cross the blood–brain barrier into the skull to replicate, thus directly damaging neurons in brain tissue [24, 25]. SFTS patients tend to exhibit multiple organ damage or failure, and elevated LDH and AST are important biomarkers reflecting cell damage, death, and organ failure, and have been shown to be closely associated with the severity of virus-related diseases and mortality [26,27,28]. An elder age indicated a higher mortality, which may be due to the fact that most elderly patients have a combination of underlying diseases, an increased risk of severe infections, a low organ compensatory capacity, and an impaired immunity [29, 30]. Studies have confirmed that viral load is an equally important risk factor for death [31]. However, this factor did not satisfy the Cox proportional risk assumption (Fig. 4C) in the randomized survival analysis in this study. In addition, from the perspective of clinical practice, considering that some healthcare institutions in primary or developing regions may not have the ability to test viral load or the measurement value is not uniform in different regions, which leads to the lack of value of the factor for wide application to a certain extent.

The study presents compelling evidence for the superiority of Nomogram model in predicting short-term outcomes for Novel Bunyavirus Sepsis compared to traditional scoring systems like SOFA and APACHEII. This is supported by the higher AUC values (Fig. 7) and lower Brier Scores observed in the study, indicating enhanced accuracy and reduced deviation from actual outcomes. Although the SOFA score had the highest Time-AUC in the external test set, further comparisons with the Nomogram model and APACHE II scores showed no statistically significant differences (Fig. 8). Furthermore, the dynamic nature of Nomogram model allows for real-time adjustments based on evolving patient data, offering a significant advantage over static scoring systems. The ability of Nomogram model to automatically identify and prioritize influential prognostic factors, as demonstrated in studies by Dong et al. [14] and Jiang et al. [12], facilitates personalized risk stratification and treatment planning.

The study found that in the high-risk group identified by the nomogram (cutoff value of 116.4), patients treated with IVIG had a significantly higher survival rate than untreated patients, with a risk ratio of 1.756 (95% CI 1.019–3.028). This suggests that IVIG may have an important role in the treatment of SFTS, especially in high-risk patients. Some studies have shown IVIG’s potential benefits in SFTS include neutralizing the virus, modulating the immune response, and regulating the complement system [19, 32]. The significance of this study is that it provides an important tool for risk stratification and individualized treatment decisions for SFTS patients. By identifying high-risk patients and administering IVIG therapies, patient prognosis can be improved and treatment outcomes enhanced. In addition, Given the importance of state of consciousness for prognostic prediction and the stability and widespread use of the SOFA score for prognostic prediction, this study showed that those who developed impaired consciousness or had a SOFA score of 7 or more were also immunoglobulin beneficiaries, which is consistent with the results of earlier studies of the effectiveness of human immunoglobulin in the treatment of viremia or sepsis [33, 34].

This study demonstrates that the 7-day survival probability among high-risk patients is only 24%. The dynamic column-line diagram model developed in this study provides an important reference value for prognostic prediction and therapeutic decision-making in patients with novel Bunyavirus sepsis. However, the study has some limitations. First, the relatively small number of patients, especially in the external test set, may limit the model's external validity and generalizability. Second, as a retrospective study, it is prone to informational and selection biases. Additionally, the study focused on clinical bioindicators and may have overlooked other potential factors, such as patients' immune status, genetic backgrounds, and socioeconomic status. These aspects need further refinement and improvement in future studies.By expanding the sample size, optimizing the selection of variables, and adopting prospective studies, the accuracy and reliability of the model can be improved so that it can be better applied to clinical practice and improve patient prognosis.

Dynamic column-line graphical modeling was effective in predicting the short-term prognosis of patients with novel Bunyavirus sepsis and identifying patient groups that could benefit from human immunoglobulin therapy. The machine learning model demonstrated higher accuracy and reliability in predicting the short-term prognosis of patients with novel bunyavirus sepsis compared to traditional scoring systems (SOFA, APACHE II). Human immunoglobulin therapy is efficacious in high-risk patients and significantly improves patient survival.

Appendices

Appendix 1

	Model	Times	AUC	se	Lower	Upper
1	Nomogram	1	0.875	0.04604767	0.7847482	0.9652518
2	Nomogram	2	0.8794643	0.03968527	0.8016826	0.957246
3	Nomogram	3	0.8697275	0.07037675	0.7317916	1
4	Nomogram	4	0.8673917	0.05559553	0.7584265	0.9763569
5	Nomogram	5	0.820086	0.06730968	0.6881615	0.9520106
6	Nomogram	6	0.8313215	0.0617311	0.7103307	0.9523122
7	Nomogram	7	0.8192192	0.06404957	0.6936844	0.9447541
8	SOFA	1	0.95375	0.03379834	0.8875065	1
9	SOFA	2	0.9278274	0.03023919	0.8685597	0.9870951
10	SOFA	3	0.9146571	0.03521436	0.8456382	0.983676
11	SOFA	4	0.9159971	0.04636838	0.8251167	1
12	SOFA	5	0.8775346	0.05635113	0.7670884	0.9879807
13	SOFA	6	0.8874747	0.05324462	0.7831171	0.9918322
14	SOFA	7	0.8795861	0.05520026	0.7713956	0.9877766
15	APACHEII	1	0.95	0.02746136	0.8961767	1
16	APACHEII	2	0.827381	0.07186861	0.6865211	0.9682408
17	APACHEII	3	0.8567439	0.05170202	0.7554099	0.958078
18	APACHEII	4	0.8753759	0.043317	0.7904761	0.9602756
19	APACHEII	5	0.8267672	0.05871053	0.7116967	0.9418377
20	APACHEII	6	0.8452463	0.05454121	0.7383475	0.9521451
21	APACHEII	7	0.8409632	0.05551288	0.73216	0.9497665

	Model	Times	AUC	se	Lower	Upper
1	Nomogram	1	0.8737179	0.02862183	0.8176202	0.9298157
2	Nomogram	2	0.9089434	0.02457669	0.860774	0.9571129
3	Nomogram	3	0.9185626	0.02123215	0.8769483	0.9601768
4	Nomogram	4	0.9186333	0.02099024	0.8774932	0.9597734
5	Nomogram	5	0.9065475	0.02472241	0.8580925	0.9550025
6	Nomogram	6	0.9094313	0.02407116	0.8622527	0.9566099
7	Nomogram	7	0.8867189	0.02882815	0.8302168	0.9432211
8	SOFA	1	0.8121795	0.06669188	0.6814658	0.9428932
9	SOFA	2	0.8385114	0.05144513	0.7376808	0.939342
10	SOFA	3	0.8178867	0.04559279	0.7285265	0.9072469
11	SOFA	4	0.8375132	0.03750367	0.7640074	0.911019
12	SOFA	5	0.8206032	0.03852817	0.7450894	0.8961171
13	SOFA	6	0.8320381	0.03734823	0.758837	0.9052393
14	SOFA	7	0.8530483	0.03454148	0.7853482	0.9207484
15	APAHEII	1	0.7288462	0.0555797	0.6199119	0.8377804
16	APAHEII	2	0.7154915	0.05312927	0.61136	0.8196229
17	APAHEII	3	0.7439336	0.04645165	0.65289	0.8349771
18	APAHEII	4	0.7600819	0.04106366	0.6795986	0.8405652
19	APAHEII	5	0.7487865	0.03989944	0.670585	0.826988
20	APAHEII	6	0.7675135	0.03873847	0.6915875	0.8434395
21	APAHEII	7	0.7522338	0.03955836	0.6747008	0.8297668

Nomogram	1	0.9487952	0.03322147	0.8836823	1
Nomogram	2	0.9629877	0.02185734	0.9201481	1
Nomogram	3	0.971173	0.0168295	0.9381878	1
Nomogram	4	0.9242137	0.04749708	0.8311211	1
Nomogram	5	0.9176545	0.04367461	0.8320538	1
Nomogram	6	0.9044511	0.0428194	0.8205266	0.9883756
Nomogram	7	0.9008942	0.04352179	0.8155931	0.9861954
SOFA	1	0.935241	0.02934965	0.8777167	0.9927652
SOFA	2	0.8729012	0.07822914	0.7195749	1
SOFA	3	0.8818133	0.06032423	0.76358	1
SOFA	4	0.7815648	0.10796062	0.5699659	0.9931637
SOFA	5	0.7497259	0.10198337	0.5498421	0.9496096
SOFA	6	0.7760061	0.09571886	0.5884006	0.9636116
SOFA	7	0.768767	0.09614232	0.5803315	0.9572025
APACHEII	1	0.7861446	0.10414901	0.5820163	0.9902729
APACHEII	2	0.7424444	0.0914684	0.5631697	0.9217192
APACHEII	3	0.8117419	0.07624332	0.6623077	0.9611761
APACHEII	4	0.8219565	0.06864271	0.6874193	0.9564938
APACHEII	5	0.7858111	0.07194703	0.6447975	0.9268247
APACHEII	6	0.8073238	0.06854026	0.6729873	0.9416602
APACHEII	7	0.8004681	0.06956199	0.6641291	0.9368071

Appendix 2

	Times	Model	Reference	delta.AUC	se	Lower	Upper	P	Contrast
1	1	cox_model_APACHEII	Nomogram	0.1425	0.14060528	− 0.13308128	0.4180813	0.3108331	cox_model_APACHEII—Nomogram
2	2	cox_model_APACHEII	Nomogram	− 0.03125	0.11963259	− 0.26572557	0.2032256	0.7939256	cox_model_APACHEII—Nomogram
3	3	cox_model_APACHEII	Nomogram	0.006202095	0.08269078	− 0.15586886	0.168273	0.940212	cox_model_APACHEII—Nomogram
4	4	cox_model_APACHEII	Nomogram	0.030816966	0.05786965	− 0.08260547	0.1442394	0.5943633	cox_model_APACHEII—Nomogram
5	5	cox_model_APACHEII	Nomogram	0.027996222	0.05776649	− 0.08522401	0.1412165	0.6279284	cox_model_APACHEII—Nomogram
6	6	cox_model_APACHEII	Nomogram	0.053260328	0.05692244	− 0.0583056	0.1648263	0.3494458	cox_model_APACHEII—Nomogram
7	7	cox_model_APACHEII	Nomogram	0.064672484	0.06065494	− 0.05420902	0.183554	0.286317	cox_model_APACHEII—Nomogram

	Times	Model	Reference	delta.AUC	se	Lower	Upper	P	Contrast
1	1	SOFA_model	Nomogram	0.14625	0.139489	− 0.127143423	0.4196434	0.29442223	SOFA_model—Nomogram
2	2	SOFA_model	Nomogram	0.06919643	0.08770277	− 0.102697834	0.2410907	0.43011903	SOFA_model—Nomogram
3	3	SOFA_model	Nomogram	0.06411523	0.06661443	− 0.066446659	0.1946771	0.33580727	SOFA_model—Nomogram
4	4	SOFA_model	Nomogram	0.07143817	0.05246643	− 0.031394153	0.1742705	0.17332493	SOFA_model—Nomogram
5	5	SOFA_model	Nomogram	0.07876358	0.05136076	− 0.02190165	0.1794288	0.12514378	SOFA_model—Nomogram
6	6	SOFA_model	Nomogram	0.0954887	0.05124713	− 0.004953837	0.1959312	0.0624203	SOFA_model—Nomogram
7	7	SOFA_model	Nomogram	0.10329533	0.05510236	− 0.004703312	0.211294	0.06084666	SOFA_model—Nomogram

Appendix 3