Development and Validation of a Random Forest Risk Prediction Pneumothorax Model in Percutaneous Transthoracic Needle Biopsy

Background

Computed tomography (CT)-guided percutaneous transthoracic needle biopsy (PTNB) is a common method used to establish the pathological diagnosis of pulmonary nodules or masses. CT-guided PTNB has high diagnostic accuracy and an acceptable complication rate [1]. However, pneumothorax occurs more often than other complications [2–4], with an incidence rate ranging from 4.0% to 62.0% [5,6]. As a result of the pneumothorax, patients often suffer from restlessness, shortness of breath, asphyxia, hypotension and cold sweat, shock, and even unconsciousness. Pneumothorax can also become a life-threatening condition requiring urgent intervention. Therefore, a preliminary screening tool that can be used by interventional physicians to predict the postoperative pneumothorax for patients undergoing CT-guided PTNB is essential to improve patient selection, preoperative preparation, procedural techniques, and postoperative management [7,8], such as using oxygen delivery by facemask or nasal cannula, shortening the length of the aerated lung traversed by the needle, and extending the period of vital signs monitoring.

In recent years, there has been a surge of interest in utilizing artificial intelligence and machine learning algorithms in the field of medicine for early disease diagnosis and risk prediction [9–12]. However, the majority of current machine learning models focus on the diagnosis and localization of pneumothorax on either chest radiographs or CT after the PTNB [13,14]. Only a few predictive models are available for determining the risk of developing pneumothorax after PTNB. Zhao et al and Wang et al developed logistic regression (LR) models to predict the risk of developing a pneumothorax [7,15]. The types of the pneumothorax model were still unitary, and the accuracy and sensitivity of these models were poor [15].

Previous studies demonstrated that random forest (RF) could adapt to complex clinical conditions and accurately predict or diagnose diverse cases [16,17]. LR is a generalized linear model, while RF is a nonlinear model. Furthermore, RF is less prone to overfitting and more robust when compared with LR [18]. To the best of our knowledge, RF risk prediction models for pneumothorax after CT-guided PTNB based on patient and lesion characteristics, as well as procedure-related risk factors, have not yet been explored.

Therefore, this study aimed to develop and validate an effective RF-based pneumothorax prediction model for patients undergoing CT-guided PTNB using the risk factors previously reported in the literature and based on our clinical experience [15–17].

Material and Methods

ETHICS:

This retrospective study was approved by our institutional review board (Approval number: PJ2020-003-02). Written informed consent was obtained from all patients enrolled for the CT-guided PTNB.

DATA COLLECTION:

The data were collected from 2 different institutes in China. The data from institution A were used to develop the algorithm, while the data from institution B were used to validate the algorithm (Figure 1).

From May 2016 to May 2017, 1056 consecutive patients who underwent CT-guided PTNB at institute A were enrolled, and the rate of pneumothorax was 11.3%. We excluded patients presenting with pleural effusion, patients with non-solitary pulmonary nodule or mass, lesions with a maximum diameter smaller than 5 mm, and lesions located in either the mediastinum, pleura, or chest wall, as well as patients with incomplete data also (Figure 1). A total of 245 patients (174 males and 71 females) with a mean age of 61.6 years (range: 28–85, standard deviation (SD)±9.9) were finally included in the study and used to develop the algorithm. Forty-four percent (n=108) of these patients had a pneumothorax, while the rest (n=137) had no pneumothorax. The patient and lesion characteristics of the development data are summarized in Table 1.

A total of 124 consecutive patients who underwent CT-guided PTNB between March 2017 and January 2021 at institute B were enrolled, and the rate of pneumothorax was 54.0%. Seventy-eight percent (n=97) of these patients met the exclusion criterion (Figure 1). The patient group consisted of 62 males and 35 females, and the mean age was 62.8 years (range: 20–86 years, SD±12.0). The patient and lesion characteristics of the external validation data are summarized in Table 2.

BIOPSY TECHNIQUE:

Six interventional radiologists with 3–5 years of experience performed all CT-guided PTNBs procedures according to a standard protocol [19]. All CT scans in institute A were acquired using a 16-detector MX, Phillips Medical Systems, Best, Netherlands CT scanner using 120 mAs, 120 kV, and a 2.0–3.0 mm slice thickness. Biopsies were performed using the coaxial method with a 19-gauge coaxial guide needle (C2016B, Bard) and a 20-gauge automated core biopsy needle (MN2016, Bard). In institute B, all CT scans were acquired with a scanner (LightSpeed VCT; GE Healthcare, Waukesha, Wis) using 140 mAs, 120KV, and a slice thickness of 1.0–2.0 mm. Biopsies were also performed using the coaxial method with a 17-gauge coaxial guide needle (C1816A, Bard Peripheral Vascular, Inc.) and an 18-gauge automated core biopsy needle (MC1816, Bard Peripheral Vascular, Inc.).

MEASURED VARIABLES:

Based on information in the literature, 15 patient-, lesion-, and procedure-related risk factors were identified and data on them were collected for analysis [13,16,17]. Patient-related risk factors included age and sex. Lesion-related risk factors included lesion size (the maximum short-axis diameter), lesion location, lesion depth (the shortest distance from the costal pleura to the edge of the lesion), presence of a cavity inside the lesion, emphysema or bullae adjacent to the biopsy track, and pneumonitis around the lesion. When the lesion is in direct contact with the pleura, the lesion depth is 0.0 mm. Biopsy procedure-related risk factors included length of the aerated lung traversed by the needle (the distance from the pleural puncture point to the edge of lesion along the needle path) [20], angle of pleural puncture, procedure time (total needle insertion and removal time), biopsy position, the crossing of the pleural indentation or interlobar fissure, and the number of biopsies acquired. The measurement methods used to calculate the risk factors for developing pneumothorax are illustrated in Figure 2. Pneumothorax was classified as either mild, moderate, or severe according to the largest distance between parietal pleura and visceral pleura on axial CT images [21] (Figure 2C).

MODEL DEVELOPMENT AND FEATURE SELECTION:

A total of 245 patients were randomly divided using a ratio of 8: 2 into the training and testing datasets. The 15-RFs were first used to develop 4 pneumothorax risk models: LR, support vector machine (SVM), decision tree (DT), and RF. The output of the models was then labeled as either positive (1) or negative (0) for pneumothorax. Then, the more relevant risk factors were selected using RF feature selection. RF is an ensemble of classification trees. Each individual RF tree was set as internal nodes and leaves. The selected feature was used to make a decision on how to divide the data set into 2 separate sets with similar responses. The internal nodes features were selected using a criterion based on the Gini impurity classification. The importance of the feature, also known as the Gini importance, was computed as the (normalized) total reduction of the criterion brought by that feature [22]. To determine the optimum risk factor subset, the relationship between the AUCs of the RF model and the number of risk factors were evaluated. Finally, we redeveloped these models using the most relevant risk factors. All models on the development dataset were then applied to the external validation dataset obtained from institute B. All models were built using the Python environment (version 3.8, Beaverton, Ore) by using the sklearn, numpy, pandas, and scipy packages.

STATISTICAL ANALYSIS:

Statistical analyses were performed using SPSS version 26 (IBM Analytics Products, USA). Continuous variables were summarized as mean and standard deviation (mean±SD), while categorical variables were summarized using frequencies and percentages. Correlation analysis was performed by Pearson correlation analysis. The Mann-Whitney U test and chi-square test were used in between-group comparisons. The models’ performance was assessed by calculating the area under the curve (AUC) of a receiver operating characteristic curve (ROC). The DeLong test was used to statistically compare the ROC curves of all models [23]. The sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV) for all developed models were also calculated. For all statistical tests, a P value of 0.05 or less was considered to be statistically significant.

Results

CLINICAL CHARACTERISTICS:

To develop and validate the 4 models (LR, SVM, DT, and RF) for the prediction of pneumothorax, we used the development data of 245 patients and external validation data of 97 patients. Within the development dataset, pneumothorax was mild in 86 (79.6%), moderate in 18 (16.7%), and severe in 4 (3.7%) patients, while in the external dataset, pneumothorax was mild in 48 (88.9%), moderate in 3 (5.6%), and severe in 3 (5.6%) patients. There was no postoperative delayed pneumothorax. Very often, patients with mild pneumothorax are not provided any treatment and are instead put under observation. The placement of a chest tube (n=28) is only considered if the pneumothorax becomes symptomatic or more severe. The detailed characteristics of the development and external validation data are summarized in Tables 1 and 2.

FEATURE SELECTION:

The risk factors are shown in order of the importance of pneumothorax in Figure 3A. The predictive performance (AUCs) of RF models for each number of risk factors are shown in Figure 3B. The AUCs of RFs reached an approximate plateau when the top-6 risk factors were introduced. In addition, univariate analysis showed a significant difference in sex between 2 groups (P=0.003). Considering the clinical circumstance, so the top-7 ranked risk factors were identified as the optimum risk factor subset. The length of the aerated lung traversed was identified as the most important risk factor for developing pneumothorax, followed by angle of pleural puncture, lesion depth, lesion size, age, procedure time, and sex.

PERFORMANCE OF PREDICTION MODELS:

The sensitivity, specificity, accuracy, PPV, and NPV were calculated for all models using both development and validation datasets, which are summarized in Tables 3 and 4. The prediction performance of the models was assessed using the AUC, as illustrated in Figure 4. According to DeLong’s test for difference in ROC curves, the LR, DT, and RF models based on either 15-RFs or 7-RFs achieved similar classification performance (P>0.05). In contrast, the SVM model had significantly lower levels based on 7-RFs (P<0.05). All models yielded an AUC above 0.599, with the RF model achieving the highest accuracy, with an AUC of 0.921. In the external validation dataset, based on 7-RFs, the RF model achieved an AUC of 0.914 (P<0.05), which was significantly higher than all other models, as shown in Table 5.

Discussion

In this study, we developed the first pneumothorax prediction model using the RF algorithm based on the patient, lesion, and procedure-related risk factors. Furthermore, to minimize the risk of overfitting the RF models, we evaluated the model’s diagnostic performance using a separate external validation dataset. We also compared the performance of the RF model with other commonly used prediction models, including LR, SVM, and DT.

The 7-RFs RF model proposed in our study achieved the best performance, with an AUC of 0.914 on the validation dataset, indicating that the RF model fit well with the relationship between risk factors and pneumothorax. Therefore, this model may be used clinically to reduce the risk of developing pneumothorax by changing controllable factors such as the angle of pleural puncture and also by improving the postoperative management of high-risk patients.

To date, among the various machine learning methods, only multivariate LR analysis has been used to develop a predictive model for pneumothorax after a CT-guided needle biopsy. Zhao et al established a predictive LR model based on 6 risk factors, which resulted in an AUC, sensitivity, specificity, and accuracy of 0.735, 0.568, 0.796, and 0.725, respectively [15]. However, this model was not externally validated. In another study, Wang et al used a multivariable LR to develop a predictive model based on 5 risk factors filtered by using the lasso regression method. The model had good predictive ability in both the development dataset (AUC=0.801) and in the external validation dataset (AUC=0.738) [7]. In comparison, our LR model achieved a lower AUC, slightly higher sensitivity, and marginally lower specificity and accuracy. This variation may be due to the use of different risk prediction factors. In the present study, the RF model demonstrated higher sensitivity, specificity, accuracy, PPV, NPV, and AUC when compared with the LR, SVM, and DT models in both development and validation datasets for both 15-RFs and 7-RFs (Tables 3–5). Moreover, for all models except for SVM, there was no statistically significant difference in the ROC curves between the 15-RFs and 7-RFs models in both the development and external validation datasets (Table 5). These findings may suggest that a predictive model based on only 7-RFs is sufficient and may be implemented clinically, hence facilitating clinical decision-making.

Our findings identified the length of the aerated lung traversed as the most important risk factor for the development of pneumothorax, followed by angle of pleural puncture, lesion depth, lesion size, age, procedure time, and sex (Figure 3A). An interesting finding of this study was that among the 7 critical risk factors identified in our study, 3 of them – length of the aerated lung traversed, angle of pleural puncture, and procedure time – are related to the puncture procedure. This implies that the risk of developing pneumothorax may be reduced by optimizing the surgical procedure.

Previous studies found that the length of the lung parenchyma traversed by the biopsy tract was the most important factor for pneumothorax [24,25], as also identified in our study. In our study, the measurement method of lesion depth was different from that in the past, and lesion depth was positively correlated with the length of the aerated lung traversed in both development and validation datasets (r=0.658, P<0.001; r=0.673, P<0.001). This means that the deeper lesions may require a longer biopsy needle path. A longer needle path may require frequent adjustments of the biopsy needle position, which may result in tearing of the pleura and eventually increase the risk of developing pneumothorax [26,27]. A smaller needle pleural angle [28] and a longer puncture time were also found to be correlated with the risk of developing pneumothorax [29]. Consistent with previously published studies [30–34], patients with smaller lung lesions and older patients had an increased risk for developing pneumothorax in our study. A study by Kim et al found that patients above the age of 60 years had a significantly higher risk of developing a pneumothorax [34]. Males were found to be more at risk of developing pneumothorax in our study, in concordance with the studies by Lee et al and Hiraki et al [33,35].

Our study has a number of limitations that have to be acknowledged. First, some factors, such as the radiologists’ experience, needle size, and other complications, such as hemorrhage or hemothorax, were not included in the models. The model proposed in this study can only be used to predict the occurrence of postoperative pneumothorax, not the severity. However, the prediction of pneumothorax following PTNB is more important for patients exhibiting oxygen desaturation, as they require immediate catheter insertion. Therefore, further stratified analysis involving more participants with moderate or severe pneumothorax is required. The small sample size in our study may have resulted in selection bias or operator biases, potentially limiting the generalizability of the research findings. To further validate our results, additional studies including data from multiple institutions are warranted. Future studies should also compare the performance of our RF model with other machine learning models. Future models should also establish prediction thresholds. Finally, a prospective study on the association of the predictions with the resultant outcome is needed to clinically validate the prediction accuracy of this model.

Conclusions

The RF models resulted in high precision accuracy in the prediction for pneumothorax and performed better than the LR, SVM, and DT models. There was no statistically significant difference between the 15-RFs and 7-RFs RF models, suggesting that the 7-RFs model is sufficient to clinically determine the risk for developing pneumothorax. The 7-RFs RF model may be used to facilitate preemptive decisions in patients deemed at higher risk of developing pneumothorax, which may include procedure optimization and better postoperative management.