Modeling Tutorials

Go Back to PredictMod Project.

Step-by-Step Tutorials

The following tutorials are recommended for those interested in creating and submitting models to the PredictMod platform.

1. Metagenomic ML Tutorial
2. EHR ML Tutorial

Recommended Pipeline Steps

Please use the existing tutorials on the wiki and supplement your pipeline with the information below. Please indicate which of this content, if any, is more useful than what is online so that we can revise accordingly.

Environment Setup

1. Install VS Code, Miniconda, and Python (version 3.11 is recommended).
   1. Miniconda is great for creating isolated environments. Learn more at https://www.anaconda.com/docs/getting-started/working-with-conda/conda-intro-tutorial.
2. Make sure relevant packages are imported:
   1. scikit-learn, matplotlib, numpy, pandas, shap, xgboost, seaborn, and joblib packages.

Data Preparation

1. Import the dataset. The ML-ready dataset should:
   1. Include all required variables for model training
   2. Include a clear R/NR column
2. Run initial descriptive statistics
3. Split the data into train/test
4. Handle missing data (if needed)
   1. Simple imputation (median/mode) should be sufficient for first-time modelers, KNN is recommended but more complex.
5. Conduct feature selection
   1. Select a method (stepwise selection using random forest (RF) or logistic regression (LOGR) is recommended)
   2. Select a threshold (a maximum threshold of 10 or 15, or a percentage threshold of 5% or 1% is recommended)

Modeling & Evaluation

1. Select at least 3 model algorithms for training and comparison (I recommend RF, gradient boosting (XGB), & LOGR)
   1. Decision tree classifier (DTC) & support vector machine (SVM) algorithms are also recommended.
   2. Information about each model algorithm is thoroughly documented at https://scikit-learn.org/stable/api/sklearn.ensemble.html.
2. Record accuracy, AUROC, f1, precision, recall, and confusion matrices for each model.
3. Generate a SHAP summary plot to visualize feature importance
4. OPTIONAL: conduct hyperparameter tuning using random search, grid search, cross validation, etc.

Single-Patient Predictions

1. Once trained, create a separate .py script to run predictions using the model of choice.
   1. Save the fitted scaler, trained model, and feature names to pickle files using joblib, then load them into the new file.
2. Check for missing data, extra columns, and reorder columns as needed to match the training dataset.
3. Apply the saved scaler using .transform().
4. Generate a class prediction and print the results clearly.
   1. OPTIONAL: generate a probability score in addition to the class prediction.
5. Validate the script by testing several single-patient predictions.

Documentation

1. Each step of your script should include a comment indicating the intended function.
2. Each model must be accompanied by a BioCompute Object.