Machine Learning (ML) in healthcare

ML models were used successfully in several studies to predict patient outcomes and to forecast the probability of health events such as disease progression, adverse events, hospitalization and mortality. One example is a random forest model, an ensemble learning method that builds multiple decision trees and merges their predictions. ¹

The first ML project we’ll discuss used a random forest algorithm to enhance the identification of key covariates for inclusion in mechanistic models.1 In this study the ML model predicted metastatic relapse in patients with early-stage breast cancer. Using a dataset of 642 patients and 21 clinical-pathological features, the authors conducted a random survival forest analysis to preselect a minimal set of the most predictive covariates. The random survival forest algorithm enabled the assessment of each covariate’s predictiveness by means of the forest-average minimal depth, a metric that quantifies a covariate’s importance based on its positioning within the trees of the forest. The preselected covariates were then evaluated in the mechanistic model using a backward selection procedure, thus reducing computational time and the risk of overfitting compared to performing a backward elimination with the full set of available covariates. This model was shown to accurately fit the data with a predictive performance comparable to Cox regression. The authors proposed a personalized prediction tool for routine management of patients with breast cancer by providing informative estimates of the invisible metastatic burden at the time of diagnosis and forward simulations of metastatic growth.

Ibtissem et al. ² at Certara developed the mlcov R package for covariate selection. First, the dataset, consisting of empirical Bayesian estimates of individual parameters and covariate sets, is randomly split into five folds. Second, the covariate selection is performed by applying the Lasso algorithm³ to reduce irrelevant or redundant covariates followed by the Boruta algorithm⁴ to identify relevant covariates. Third, a voting mechanism across folds determines the final selected covariates based on their robustness. Finally, residual plots are employed to evaluate the covariate-parameter relationships. After the covariate selection, an XGboost model is trained on the selected covariates with the remaining trends between residuals and unselected covariates examined to detect any significant trends or relationships that could be captured by additional covariates.

This framework was evaluated using few patient data examples and compared with the traditional stepwise covariate modeling (SCM) approach. In one example, covariate impact was tested on clearance (CL/F), volume of distribution (V/F), and absorption rate constant (Ka), and the same set of covariates were tested for both SCM and mlcov:

• CL/F: weight (WGT), albumin (ALB), creatinine clearance (CRCL), sex, race and ethnicity;
• V/F: WGT, ALB, sex, race and ethnicity
• Ka: age, formulation (FORM), device.

While SCM identified ethnicity and sex for CL/F, mlcov did not, likely due to their correlations with race and body weight, respectively. For Ka, SCM identified the variable, device for Ka, while mlcov did not identify any covariate, with no trends in the residual plots. It is noteworthy to mention that the identified variable “device” by SCM did not demonstrate any significant impact on the extent of absorption (shown in the bioequivalence study).

Model-based meta-analysis or MBMA is a quantitative approach that uses published summary-level clinical data along with sponsors’ internal data. As a standard tool for the model-informed drug development (MIDD) framework, it can be applied to inform key drug development decisions. MBMA overcomes many limitations of traditional meta-analytic methods by providing a quantitative framework leveraging pharmacological information (dose, time, population characteristics, endpoint correlations, etc.) to explain observed variability in trial outcomes and allowing for appropriate comparisons of treatment options within a disease area. A standard MBMA includes data from randomized clinical trials, but there has been a growing interest in applying to real-world data⁵. A hierarchical structure can be considered to incorporate the heterogeneity between treatments, studies and study designs⁶.

Constructing databases is a major impediment to conducting MBMA. ML methods have great potential to further expand the implementation of MBMA in drug development by, for example, automating many steps of the systematic review process, including search, screening, data extraction and augmentation. These methods have been utilized in the AI-powered CODEX platform⁷ that creates and maintains highly curated, indication-specific clinical outcomes databases.

The three applications referenced here illustrate the potential ML models possess as powerful tools to extract, analyze and interpret patient data.

However, it should be noted that patient data are inherently complex, and the performance of ML models is directly influenced by the quality of the data used for training. ML model performance could be impaired by missing data, errors in data entry, and biases in the dataset to generalize reliable predictions. MBMA results can also be significantly impacted by potential bias from published summary-level data. This includes issues like the poor quality of study design, the lack of publications from studies with negative outcomes, and errors during data extraction, etc. The AI-powered CODEX platform noted above offers one potential solution for ensuring the quality of the database.

Another challenge is that many ML models, particularly deep learning algorithms, are more difficult to interpret than traditional statistical models such as regression models. A lack of transparency of the ML models would make it challenging to understand how decisions are made based on these models, which can hinder their use in guiding clinical decisions or informing public health policy. In healthcare, where decisions can have serious or even life-threatening consequences, it is essential to ensure that ML models are both reliable and explainable. Therefore, accountability and transparency of ML algorithms are critical to evaluate model performance and build trust among researchers, clinicians, policymakers and patients. This includes understanding not only the input data if they can be generalized to the wider population as well as the ML model features which could influence study results. While there are a few existing tools such as “InterpretML”⁸ and “AI Explainability 360”⁹ that can help interpret ML model results, human intervention remains essential for careful design of ML models, thoughtful interpretation and communication of analysis results for successful and responsible application in healthcare studies.

In conclusion, ML models provide powerful tools to analyze and interpret patient data collected in both clinical trials and real-world settings. When used appropriately, these models can support clinical decision-making and enhance patient care. However, the successful application of ML models in healthcare studies relies on access to high-quality, representative data, careful model selection, and the ability to generate interpretable and accountable analysis results.