Prediction of 12-month recurrence of pancreatic cancer using machine learning and prognostic factors

Abstract

Background and aim

Pancreatic cancer is lethal and prevalent among other cancer types. The recurrence of this tumor is high, especially in patients who did not receive adjuvant therapies. Early prediction of PC recurrence has a significant role in enhancing patients’ prognosis and survival. So far, machine learning techniques have given us insight into favorable performance efficiency in various medical domains. So, this study aims to establish a prediction model based on machine learning to achieve better prediction on this topic.

Materials and methods

In this retrospective research, we used data from 585 PC patient cases from January 2019 to November 2023 from three clinical centers in Tehran City. Ten chosen ensemble and non-ensemble algorithms were used to establish prediction models on this topic.

Results

Random forest and support vector machine with an AU-ROC of approximately 0.9 obtained more performance efficiency regarding PC recurrence. Lymph node metastasis, tumor size, tumor grade, radiotherapy, and chemotherapy were the best factors influencing PC recurrence.

Conclusion

Random forest and support vector machine algorithms demonstrated high-performance ability and clinical usability to improve doctors’ decisions in achieving different therapeutic and diagnostic measures.

Introduction

Pancreatic cancer (PC) is known as one of the most lethal types of cancer [1]. It appears through non-invasive epithelial proliferation in the pancreatic ducts, also known as pancreatic intraepithelial neoplasia [2, 3]. This malignancy ranks twelfth and eleventh among all types of cancer in men and women, respectively [4, 5]. It has an upward trend regarding prevalence and is estimated to reach 18.6 per 100,000 in 2050, posing significant public health challenges [6]. With a 5-year survival rate of 10%, this cancer is becoming one of the most important causes of cancer-related death worldwide [7]. According to GLOBOCAN, PC is the seventh most common type of cancer in terms of mortality. Also, this disease will rank second in cancer deaths by 2030 [8,9,10]. One of the crucial causes of PC-related death is the high recurrence rate of this disease. Also, current monitoring techniques used for PC patients in the postoperative period, such as clinical symptom monitoring, CT scan, and tumor markers, are inefficient and insensitive to deal with disease recurrence [11].

PC has 79% recurrence in patients who have undergone surgery, and despite other cancer types, it has a poor prognosis even after performing surgery in combination with adjuvant therapies [12]. Also, this cancer type has an 83.7% and 87% recurrence rate associated with 7.8 and 13.4 months in patients who have undergone surgery [13]. So, due to the poor prognosis of this disease and the insufficiency of current screening techniques to prevent PC recurrence, we require more efficient strategies to better prognosis and increase PC patients’ survival [14,15,16,17]. Previous studies have shown that machine learning (ML) techniques can significantly improve prognosis in different healthcare fields, such as cancer, by establishing efficient prediction models [18,19,20].

So far, ML techniques have been used in various clinical aspects of PC, such as predicting the prognosis [21], survival prediction [22], clinical outcomes [23], response to treatment [24, 25], and diagnostic procedures [26]. In some previous studies, these techniques have also been leveraged to predict PC relapse. Lee et al. used ML to predict PC recurrence based on pathological and surgical factors [27]. In the current study, we used therapeutic factors such as radiotherapy and chemotherapy data in addition to the pathologic factors considered in Lee’s study to establish a more comprehensive prediction model with better accuracy. Elarre used this technique to predict individual recurrence in PC patients after preoperative treatment [28]. Elarre’s study focused more on therapeutic factors, such as radiotherapy and paid little attention to pathological factors. In the current study, we used more detailed pathological factors and the modified Glasgow Prognostic Score (mGPS) in addition to the pathological and therapeutic factors used in Elarre’s study. Hayashi et al. used the supervised ML technique and histopathological data to predict PC recurrence [29]. In the current study, we used pathological and therapeutic factors in addition to histopathological data to establish prediction model for PC recurrence.

Early prediction of PC recurrence will assist clinicians in making better decisions to enhance the prognosis of patients by optimizing the various clinical and therapy measures, eventually leading to increased PC patients’ survival [28]. Therefore, this study aimed to establish a prediction model based on the ML techniques for the 12-month recurrence of PC patients to better prognosis.

Methods

Study roadmap

An overview of the methodology used in the current research is depicted in Fig. 1.

Fig. 1

The overall schema of the methodology used

Based on Fig. 1, after defining the topic and population of the current research, we first defined the study’s database, including the records and input and outcome features. Second, we preprocessed the current database to enhance data quality and better analysis. To this aim, we excluded duplicate records and refined the current database using the imputation and exclusion processes to solve the missing value and data error issues. Third, we leveraged the feature selection process using binary logistic regression and chose the best-ranking features to obtain the most efficient model. Fourth, we used selected ensemble and non-ensemble ML algorithms to establish prediction models for predicting PC recurrence. Fifth, we developed the models using the Grid search method to gain the best-tuned combination of ML algorithms’ hyperparameters with higher performance for prediction purposes. We used the hold-out strategy to partition the data for the mining process and divided the data into training, testing, and validation sections. We used confusion-based performance criteria to understand the best model with higher predictive competency and compared various ML algorithms’ performance. Finally, to depict the clinical applicability and interpretability of the best model obtained, we assessed the predictive ability of the features using the relative importance.

Database and study population

The study population in this retrospective study were PC patients who underwent PC resection and adjuvant treatment. They were referred to three clinical centers in Tehran City for recurrence evaluation from January 2017 to November 2023. The PC recurrence status after 12 months of surgery was recorded in their medical records. The patients’ data belonging to centers were registered in one integrated Excel (XLSX) database. This file included 585 rows, comprising 274 and 311 records associated with positive and negative cases regarding PC recurrence after 12 months of diagnosis.

Feature description

The input prognostic factors used to establish the prediction model for PC recurrence included age, gender, BMI (Body Mass Index), surgery method, chemotherapy, radiotherapy, lymph node metastasis, vascular resection, resection margin, peritoneal cytology, resectability, duration of hospital stay, modified Glasgow Prognostic Score (mGPS), tumor grade, T-stage (tumor size), N-stage (lymph node invasion), M-stage (Metastasis state), and histological type. The current study’s outcome feature was PC’s recurrence status after 12 months of diagnosis. It had two values in the current database, including 12-month relapse and non-relapse, which were specified as codes 1 and 0, respectively.

Database preparation

We used some preprocessing steps to enhance the data quality in the current database. First, any duplicate data was investigated and removed. The duplicate cases were associated with data on the same patient with a different ID, stored in two or more rows in the current database. Second, we handled the missing value issue. In this respect, we performed two main tasks: if the missing data in records was more than 10%, we excluded that record from further analysis.

On the other hand, considering the qualitative data in the current database, we used the mode of feature gained by other records to fill in the value of the same feature belonging to the missing record. Moreover, there was some data error in some records in the current database. Considering the qualitative data, the data error was specified as any feature values not categorized as the predefined values in the database. To address this, we deleted these values from the current database and then treated them as missing. So, we filled them using the mode of feature belonging to other records.

Feature selection

In the current study, we used the Feature selection (FS) technique to obtain the best factors for prediction purposes. FS is an efficient and effective preprocessing step for ML and data mining processes. It has a valuable role in improving the performance of algorithms, better understanding and cleaning of data, constructing more straightforward and more understandable models, and increasing the speed of training algorithms [30,31,32]. We used the binary logistic regression as a multivariable FS technique with the Enter method to achieve the best factors for the 12-month PC recurrence prediction. The P < 0.05 was considered for the FS threshold and obtaining the best factors influencing it.

Model establishment and assessment

We used ML algorithms to develop prediction models for 12-month PC recurrence among patients. The selected ensemble, including Random Forest (RF), LightGBM, Bagging and XG-Boost, and non-ensemble algorithms, including Support Vector Machine (SVM), K-Nearest Neighbor (K-NN), Logistic Regression (LR), Artificial Neural Networks (ANNs), J-48 and Naïve Bayes (NB) were utilized to this aim. To assess the performance efficiency of the ML algorithms, we used the Positive Predictive Value (PPV), Negative Predictive Value (NPV), sensitivity, specificity, accuracy, and F-Score. We also presented the confusion matrix results (Fig. 2) for the best-performing models to investigate their classicability more precisely. In Fig. 2, True Positive (TP) and True Negative (TN) are the positive and negative cases regarding PC recurrence correctly classified by the algorithm. The False Negative (FN) and False Positive (FP) point to incorrectly classified these cases. The Receiver Operator Characteristics (ROC) curve was also assessed to compare the predictability of the ML models and select the best model for prediction purposes.

Fig. 2

Confusion matrix

Grid search method

The current study used the Grid search to adjust the ML algorithms’ hyperparameters during training. This method is an exhaustive search based on a predefined subset of the hyperparameter space. This way, the combinations of hyperparameters with different values in different training scenarios for optimization are used to test ML algorithms’ performance. Contrary to statistical methods and genetic algorithms that leverage statistical methods to optimize the hyperparameters’ values based on data used, the optimization process in this method would be conducted by a predefined range by a user. So, this method would give us more detailed and specific behavior on algorithm performance during the training process in different scenarios for testing the algorithms’ performance [33, 34].

Hold-out strategy

One strategy to split data during the training process is Hold-out. This way, the data is randomly partitioned into two independent sections: a training set and a test set, or a training set and test set, including test and validation sets. The proportion between the training and test data is not specified precisely. However, two-thirds and one-third of the data are usually used to train and test the algorithms. In other training scenarios, 70% and 30% of data are leveraged for training and testing purposes. The training set is used to derive the model, whose performance is estimated using the test set [35, 36]. The current research split the data into three sections: training, testing, and validation. 70% of the data were used for algorithms’ training, and 15% and 15% were considered for testing and validation, respectively.

Feature assessment

Feature assessment has a significant role in enhancing the explanation of ML algorithms and clinical applicability. The current study used the high-performing ML algorithms’ Relative Importance (RI) to analyze and choose the best factors influencing the 12-month PC recurrence. The RI is considered a more straightforward and usual method for selecting the best predictors and clinical applicability, and it has been utilized in other biomedical research [37,38,39].

Results

Database preparation and patients’ characteristics

After checking the database regarding duplicate cases, 6 cases with the same identifiers belonging to the same patients were excluded from the study. Thirteen cases with more than 10% lost data in their features were removed from the database. The lost data of 20 records with less than 10% missing value in their features were filled by the mode of the same feature obtained by other records. Also, there were data errors in 20 records that were first excluded and then embedded by the mode of the same feature. Finally, 566 records were analyzed in the current study, including 268 and 298 cases associated with relapsed and non-relapsed PC, respectively. 287 and 279 cases have belonged to women and men, respectively. The details of PC patients’ characteristics included in the current study are presented in Table 1.

Table 1 The characteristics of the relapsed and non-relapsed PC patients

Based on Table 1, age, chemotherapy, radiotherapy, resectability, resection margin, tumor grade, T-stage (Tumor size), N-stage (Lymph node invasion), M-stage (Metastasis state), mGPS, lymph node metastasis, and vascular resection showed a meaningful difference between two different groups statistically (P < 0.05). The factors, including gender (P = 0.21), BMI (P = 0.1), duration of hospital stay (P = 0.08), surgery method (P = 0.06), histological type (P = 0.08), and peritoneal cytology (P = 0.08) did not attain difference. Also, the skewness and kurtosis of data associated with all variables were in ranges of [-2 2] and [-3 3], respectively, indicating the approximately normal data distribution for analysis.

Feature selection

The results of the FS process using the binary logistic regression are shown in Table 2.

Table 2 The analysis of data using the binary logistic regression

As Table 2 shows, according the multivariable regression analysis, the factors including age (β = 0.266, OR = 1.103, 95% CI=[1.081–1.116]), duration of hospital stay (β = 0.217, OR = 1.262, 95% CI=[1.115–1.377]), surgery method (β = 0.544, OR = 1.523, 95% CI=[1.298–1.752]), chemotherapy (β = 0.98, OR = 2.474, 95% CI=[1.974–3.003]), radiotherapy (β = 1.082, OR = 3.185, 95% CI=[2.542–3.724]), resectability (β = 0.732, OR = 2.195, 95% CI=[1.745–2.656]), resection margin (β = 0.885, OR = 2.221, 95% CI=[1.883–2.654]), histological type (β = 0.283, OR = 1.135, 95% CI=[1.087–1.198]), tumor grade (β = 0.656, OR = 1.805, 95% CI=[1.522–2.031]), T-stage (β = 0.723, OR = 1.878, 95% CI=[1.597–2.127]), N-stage (β = 0.705, OR = 1.856, 95% CI=[1.552–2.096]), M-stage (β = 0.624, OR = 1.579, 95% CI=[1.399–1.825]), mGPS (β = 0.411, OR = 1.426, 95% CI=[1.396–1.641]), lymph node metastasis (β = 0.513, OR = 1.498, 95% CI=[1.399–1.696]), vascular resection (β = 0.425, OR = 1.44, 95% CI=[1.417–1.482]), and peritoneal cytology (β = 0.225, OR = 1.027, 95% CI=[1.018–1.041]) with P < 0.05 were considered as the best factors for predicting PC recurrence. In contrast, factors including gender and BMI with P > 0.05 did not show any importance.

Model establishment and evaluation

The results of the performance evaluation of ML algorithms in total mode (average of training, testing, and validation states) are shown in Table 3. Also, the ranges of values used to optimize the algorithms’ hyperparameters using the Grid search method are shown in Table 4.

Table 3 The performance evaluation of selected ML algorithms

Table 4 The ranges of hyperparameters used for optimization

According to Table 3, RF with a PPV of 92.05%, NPV of 91.72%, sensitivity of 90.67%, specificity of 92.95%, accuracy of 91.87%, and F-Score of 91.35% obtained better performance efficiency than other ML algorithms. SVM with a PPV of 90.84%, NPV of 90.13%, sensitivity of 88.81%, specificity of 91.95%, accuracy of 90.46%, and F-Score of 89.81% gained favorable performance in addition to RF for predicting the relapsed and non-relapsed cases. In this respect, XG-Boost with a PPV of 82.64%, NPV of 83.72%, sensitivity of 81.72%, specificity of 84.56%, accuracy of 83.22%, and F-score of 82.18% obtained almost satisfactory performance with all criteria more than 0.8. LightGBM obtained a PPV of 79.48%, NPV of 81.54%, sensitivity of 79.48%, specificity of 81.54%, accuracy of 80.57%, and F-score of 79.48%. K-NN with a performance in PPV of 66.20%, NPV of 72.04%, sensitivity of 70.90%, specificity of 67.45%, accuracy of 69.08%, and F-score of 68.47% and LR with a PPV of 63.77%, NPV of 68.28%, sensitivity of 65.67%, specificity of 66.44%, accuracy of 66.08%, and F-score of 64.71% were satisfactory to some extent. All of the performance criteria belonging to bagging ranged from 70 to 80%, and the performance of J-48 and ANNs obtained performance between 60% and 70%. The NB had the worst performance ability with a PPV of 58.89%, NPV of 63.18%, sensitivity of 59.33%, specificity of 62.75%, accuracy of 61.13%, and F-Score of 59.11%. Overall, RF and SVM obtained higher performance (approximately 90%) based on various criteria and were regarded as the best-performing models for predicting PC recurrence. The confusion matrices of SVM and RF as the best models for predicting PC recurrence are shown in Fig. 3.

Fig. 3

The confusion matrices of best-performing models

In Fig. 3, the codes 0 and 1 point to negative and positive cases, respectively. According to this, SVM obtained a TP of 238, FN of 30, TN of 274, and FP of 24. So, SVM classified correctly 238 out of 268 and 274 out of 298 associated with positive and negative cases, respectively. RF classified correctly 243 out of 268 and 277 out of 298 on positive and negative PC recurrence, respectively.

The ROC of ML algorithms in predicting PC recurrence in training, testing, and validation modes are depicted in Figs. 4, 5 and 6. We reported the 95% confidence interval for the algorithms’ Area under the ROC (AU-ROC).

Fig. 4

The algorithms’ ROC in training mode

Fig. 5

The algorithms’ ROC in validation mode

Fig. 6

The algorithms’ ROC in testing mode

Figure 4 shows the ROC of ML algorithms in training mode. In this condition, RF with an AU-ROC of 0.97 and 95% CI= [0.95–0.99] and SVM with an AU-ROC of 0.96 and 95% CI= [0.93–0.98] outperformed others on the prediction of recurrence. XG-Boost performed almost satisfactorily on this topic with an AU-ROC of 0.85 and 95% CI= [0.82–0.89]. In contrast, NB achieved lower performance than the other algorithms in training mode with an AU-ROC of 0.67 and 95% CI = [0.63–0.7]. Based on Fig. 5, RF with an AU-ROC of 0.86 and 95% CI= [0.84–0.9] and SVM with an AU-ROC of 0.85 and 95% CI= [0.83–0.88] achieved favorable performance in the validation mode. Also, XG-Boost, with an AU-ROC of 0.76 and 95% CI= [0.75–0.77], obtained almost satisfactory performance in this condition. The worst performance in this mode belonged to NB with an ROC of 0.53 and 95% CI= [0.51–0.56]. In the testing mode (Fig. 6), RF with an AU-ROC of 0.82 and 95% CI= [0.8–0.85], SVM with an AU-ROC of 0.81 and 95% CI= [0.78–0.84], and XG-Boost with an AU-ROC of 0.79 and 95% CI= [0.75–0.83] were considered as the best performing models. On the contrary, NB with an AU-ROC of 0.55 and 95% CI= [0.52–0.59] obtained the lowest performance for predicting PC recurrence.

Feature assessment

We leveraged RF and SVM as high-performing models to evaluate the importance of each factor influencing PC recurrence. The results of scoring the factors based on RI for these models are presented in Figs. 7 and 8, respectively.

Fig. 7

The RI of the factors based on RF

Fig. 8

The RI of the factors based on SVM

According to Fig. 7, The factors, including lymph node metastasis (RI = 0.48), tumor grade (RI = 0.46), radiotherapy (RI = 0.46), and chemotherapy (RI = 0.43) obtained more predictive power than other factors for predicting PC recurrence. In this model, the factors of age (RI = 0.12) and duration of hospital stay (RI = 0.22) had lower predictive efficiency than others. Based on SVM (Fig. 8), the factors, including T-stage (tumor size) (RI = 0.48), tumor grade (RI = 0.45), radiotherapy (RI = 0.43), lymph node metastasis, and chemotherapy (RI = 0.41) were considered as best predictive factors for PC recurrence. In this model, similar to RF, age (RI = 0.16) and duration of hospital stay (RI = 0.21) obtained lower predictive performance than others.

Discussion

In the current research, we aimed to establish a prediction model for predicting PC recurrence. Hence, we got assistance from the ML techniques fed by prognostic factors to achieve this aim. The prognostic factors leveraged in the current study were categorized as pathological, therapeutic, and demographical features. Also, we leveraged the selected ensemble, such as bagging, RF, LightGBM, and XG-Boost, and non-ensemble algorithms, including ANNs, J-48, NB, K-NN, LR, and SVM to establish prediction models for 12-month PC recurrence. The current results revealed that the RF with a PPV of 92.05%, NPV of 91.72%, sensitivity of 90.67%, specificity of 92.95%, accuracy of 91.87% and F-Score of 91.35%, and SVM with a PPV of 90.84%, NPV of 90.13%, a sensitivity of 88.81%, specificity of 91.95%, accuracy of 90.46% and F-Score of 89.81% gained more satisfactory performance than other ML algorithms regarding various performance criteria. They also obtained an AU-ROC of more than 0.9, higher than the other models, indicating a higher performance ability for predicting PC recurrence. We also determined the best features regarding the recurrence predictability based on RF and SVM as high-performing models. The features, including lymph node metastasis, tumor grade, radiotherapy, chemotherapy, and tumor size, were recognized as the best factors for predicting PC recurrence.

So far, few studies have been conducted on leveraging the ML approach to predict PC recurrence. Lee et al. used ML for predicting PC recurrence after surgery. They used a multi-center database, namely the Korea Tumor Registry System (KOTUS). RF and Cox proportional-hazards models were utilized to construct prediction models for recurrence. Similar to this study, they considered prognostic features, including pathological factors, to predict recurrence.

On the contrary, they had no therapy factors for prediction purposes. RF and Cox obtained a C-index of 0.68050 and 0.7738 for predicting PC recurrence, respectively. The tumor size, tumor stage, and lymphovascular invasion were gained as the best factors influencing PC recurrence. According to their study, by providing essential prognostic factors regarding PC recurrence, RF could bring more predictive insight for clinical decision-making [27]. In the current study, we used a multi-center database similar to the Lee. Also, RF was considered the most favorable model for predicting PC recurrence. As an ensemble algorithm, RF has achieved satisfactory prediction performance in many biomedical studies [20, 40, 41]. Also, in the current study, similar to Lee, the tumor size, tumor stage, and lymphovascular invasion were recognized as crucial factors for predicting PC recurrence. In the current study, we used therapeutic features such as adjuvant therapy, which Lee et al. did not consider these factors in their study.

Baek et al. attempted to predict disease-free survival of PC recurrence using multi-omics data. To achieve this aim, they used two biological features belonging to four types of omics data and seven clinical features to build a prediction model. Based on their study, LR, with an accuracy of 0.762 and AU-ROC of 0.795, obtained more performance than other ML algorithms [13]. In this study, we used more prognostic factors than Baek despite the lack of omics data in the current database. Unlike Baek et al.‘s research, we used a feature importance strategy to rank prognostic factors. This subject is crucial in enhancing the model’s explainability and clinical usability. The current ML models with higher than 90% predictive performance obtained more efficiency than Baek’s study (AU-ROC of 0.795) for predicting PC recurrence.

Elarre utilized ML techniques to predict the individual risk of PC relapse in intensified preoperative treatment. The pathological and therapeutic factors were considered to predict PC relapse. Their study showed that LR with an AU-ROC of 0.75 obtained a more favorable performance than others [42]. In the current study, we attempted to use more detailed prognostic factors, especially more pathological ones, to predict PC recurrence. The current study demonstrated that RF and SVM with an AU-ROC of more than 0.9 obtained more performance efficiency than Elarre’s study.

Leveraging more prognostic factors has an advantageous impact on ML algorithms’ performance efficiency in predicting PC recurrence. So far, some studies that have been conducted on this topic have not considered therapeutic factors to this aim. In the current study, we included these factors for analysis and concluded that radiotherapy and chemotherapy are crucial in predicting PC recurrence. Therefore, more standardized and advanced treatment protocols can be achieved by physicians based on more efficient clinical decisions using these factors, eventually leading to better patient prognosis. Moreover, pathological features such as tumor size and grade were recognized as essential factors, showing their clinical usability in PC recurrence. So, introducing more advanced tools and methods to better assess these pathological factors in clinical environments is essential in early PC prediction and achieving more efficient and effective preventive strategies.

Limitations and future implications

Despite the current study’s benefits, we had some limitations that should be addressed. In this research, we gathered prognostic data from three clinical centers. Collecting more data records from various clinical centers will improve our performance efficiency and generalizability. We leveraged some methods to fill in the missing data, so we suggest collecting the actual data as possible for future studies. Also, some data error issues in the current database were detected and solved in the preprocessing step, which may affect the algorithms’ performance and generalizability. Due to the retrospective nature of the current research, some factors may not be regarded; for example, the genomic data was absent in the current database, so they were not analyzed for mining purposes. Future studies should consider these factors to obtain better interpretability and performance efficiency. Another limitation of the current study was the database’s outcome variable (12-month PC recurrence). In this respect, we recommend assessing more disease-free survival recurrence; for example, the 6-month and 2-year periods are better for assessing the prognosis situations of PC patients. Leveraging the external validation process is crucial to measuring the models’ interoperability and applicability in other clinical settings. Due to the lack of external data from other clinical centers, this study could not use this process. So, we recommend using this in future studies, especially through the data from other geographic regions. It will ensure the ML models’ high accuracy and clinical applicability in populations from other regions.

Conclusion

This study leveraged the combination of various prognostic factors to train ML algorithms and establish a prediction model for PC recurrence. Based on the results, SVM and RF gained favorable competency in the early prediction of PC recurrence. So, these algorithms can be fruitful for physicians in making better decisions to improve PC patients’ prognosis through promoting treatment and diagnostic measures in clinical settings.