Modeling Tutorials - Revision history

Lorikrammer: /* Modeling & Evaluation */

2026-04-01T20:33:07Z

Modeling & Evaluation

← Older revision		Revision as of 20:33, 1 April 2026
Line 37:		Line 37:
	# Record accuracy, AUROC, f1, precision, recall, and confusion matrices for each model.		# Record accuracy, AUROC, f1, precision, recall, and confusion matrices for each model.
	# Generate a SHAP summary plot to visualize feature importance		# Generate a SHAP summary plot to visualize feature importance
	# ~~OPTIONAL~~: ~~conduct~~ hyperparameter tuning using random search, grid search, cross validation, etc.		# Recommendations for improving model performance:
			## Conduct hyperparameter tuning using random search, grid search, cross validation, etc.
			## For those interested in more advanced methods, such as SHAP or LIME, are recommended.
			## For small datasets, techniques for oversampling such as SMOTE, GANs, etc. are recommended.

	=== Single-Patient Predictions ===		=== Single-Patient Predictions ===

Lorikrammer: /* Data Preparation */

2026-02-24T16:40:44Z

Data Preparation

← Older revision		Revision as of 16:40, 24 February 2026
Line 27:		Line 27:
	## Simple imputation (median/mode) should be sufficient for first-time modelers, KNN is recommended but more complex.		## Simple imputation (median/mode) should be sufficient for first-time modelers, KNN is recommended but more complex.
	# Conduct feature selection		# Conduct feature selection
	## Select a method (~~I recommend~~ stepwise; random forest (RF) or logistic regression (LOGR))		## Select a method (stepwise selection using random forest (RF) or logistic regression (LOGR) is recommended)
	## Select a threshold (~~I recommend~~ 5% or 1%)		## Select a threshold (a maximum threshold of 10 or 15, or a percentage threshold of 5% or 1% is recommended)

	=== Modeling & Evaluation ===		=== Modeling & Evaluation ===

	# Select 3 model algorithms for training and comparison (I recommend RF, gradient boosting (XGB), & LOGR)		# Select at least 3 model algorithms for training and comparison (I recommend RF, gradient boosting (XGB), & LOGR)
	## Decision tree classifier (DTC) & support vector machine (SVM) algorithms are also ~~fine~~		## Decision tree classifier (DTC) & support vector machine (SVM) algorithms are also recommended.
			## Information about each model algorithm is thoroughly documented at https://scikit-learn.org/stable/api/sklearn.ensemble.html.
	# Record accuracy, AUROC, f1, precision, recall, and confusion matrices for each model.		# Record accuracy, AUROC, f1, precision, recall, and confusion matrices for each model.
	# Generate a SHAP summary plot to visualize feature importance		# Generate a SHAP summary plot to visualize feature importance
Line 43:		Line 44:
	## Save the fitted scaler, trained model, and feature names to pickle files using joblib, then load them into the new file.		## Save the fitted scaler, trained model, and feature names to pickle files using joblib, then load them into the new file.
	# Check for missing data, extra columns, and reorder columns as needed to match the training dataset.		# Check for missing data, extra columns, and reorder columns as needed to match the training dataset.
	# Apply the saved scaler using .transform() ~~only (not fit~~.~~transform())~~		# Apply the saved scaler using .transform().
	# Generate a class prediction and print the results clearly.		# Generate a class prediction and print the results clearly.
	## OPTIONAL: generate a probability score in addition to the class prediction		## OPTIONAL: generate a probability score in addition to the class prediction.
	# Validate the script by testing several single-patient predictions.		# Validate the script by testing several single-patient predictions.

Lorikrammer: /* Environment Setup */

2026-02-24T16:37:45Z

Environment Setup

← Older revision		Revision as of 16:37, 24 February 2026
Line 15:		Line 15:
	## Miniconda is great for creating isolated environments. Learn more at https://www.anaconda.com/docs/getting-started/working-with-conda/conda-intro-tutorial.		## Miniconda is great for creating isolated environments. Learn more at https://www.anaconda.com/docs/getting-started/working-with-conda/conda-intro-tutorial.
	# Make sure relevant packages are imported:		# Make sure relevant packages are imported:
	## ~~REQUIRED:~~ scikit-learn, matplotlib, numpy, pandas, shap, xgboost, seaborn, and joblib packages.		## scikit-learn, matplotlib, numpy, pandas, shap, xgboost, seaborn, and joblib packages.

	=== Data Preparation ===		=== Data Preparation ===

Lorikrammer: /* Recommended Pipeline Steps */

2026-02-24T16:37:16Z

Recommended Pipeline Steps

← Older revision		Revision as of 16:37, 24 February 2026
Line 12:		Line 12:
	=== Environment Setup ===		=== Environment Setup ===

	# Install VS Code, Miniconda, and Python (3.11).		# Install VS Code, Miniconda, and Python (version 3.11 is recommended).
			## Miniconda is great for creating isolated environments. Learn more at https://www.anaconda.com/docs/getting-started/working-with-conda/conda-intro-tutorial.
	# Make sure relevant packages are imported:		# Make sure relevant packages are imported:
	## REQUIRED: scikit-learn, matplotlib, numpy, pandas, shap, xgboost, seaborn, and joblib packages.		## REQUIRED: scikit-learn, matplotlib, numpy, pandas, shap, xgboost, seaborn, and joblib packages.

Lorikrammer at 16:28, 24 February 2026

2026-02-24T16:28:25Z

← Older revision		Revision as of 16:28, 24 February 2026
Line 1:		Line 1:
			<small>Go Back to [[PredictMod\|PredictMod Project]].</small>

	== Step-by-Step Tutorials ==		== Step-by-Step Tutorials ==
	The following tutorials are recommended for those interested in creating and submitting models to the PredictMod platform.		The following tutorials are recommended for those interested in creating and submitting models to the PredictMod platform.

Lorikrammer: /* Step-by-Step Tutorials */

2026-02-24T16:27:35Z

Step-by-Step Tutorials

← Older revision		Revision as of 16:27, 24 February 2026
Line 2:		Line 2:
	The following tutorials are recommended for those interested in creating and submitting models to the PredictMod platform.		The following tutorials are recommended for those interested in creating and submitting models to the PredictMod platform.

	~~EHR~~ ML Tutorial		# [[Metagenomic ML Tutorial]]
			# [[PredictMod ML Pipeline Tutorial\|EHR ML Tutorial]]
	~~Metagenomic~~ ML Tutorial

	== Recommended Pipeline Steps ==		== Recommended Pipeline Steps ==

Lorikrammer: Created page with "== Step-by-Step Tutorials == The following tutorials are recommended for those interested in creating and submitting models to the PredictMod platform. EHR ML Tutorial Metagenomic ML Tutorial == Recommended Pipeline Steps == Please use the existing tutorials on the wiki and supplement your pipeline with the information below. Please indicate which of this content, if any, is more useful than what is online so that we can revise accordingly. === Environment Setup ==..."

2026-02-24T16:14:37Z

Created page with "== Step-by-Step Tutorials == The following tutorials are recommended for those interested in creating and submitting models to the PredictMod platform. EHR ML Tutorial Metagenomic ML Tutorial == Recommended Pipeline Steps == Please use the existing tutorials on the wiki and supplement your pipeline with the information below. Please indicate which of this content, if any, is more useful than what is online so that we can revise accordingly. === Environment Setup ==..."

New page

== Step-by-Step Tutorials ==
The following tutorials are recommended for those interested in creating and submitting models to the PredictMod platform.

EHR ML Tutorial

Metagenomic ML Tutorial

== Recommended Pipeline Steps ==
Please use the existing tutorials on the wiki and supplement your pipeline with the information below. Please indicate which of this content, if any, is more useful than what is online so that we can revise accordingly.

=== Environment Setup ===

# Install VS Code, Miniconda, and Python (3.11).
# Make sure relevant packages are imported:
## REQUIRED: scikit-learn, matplotlib, numpy, pandas, shap, xgboost, seaborn, and joblib packages.

=== Data Preparation ===

# Import the dataset. The ML-ready dataset should:
## Include all required variables for model training
## Include a clear R/NR column
# Run initial descriptive statistics
# Split the data into train/test
# Handle missing data (if needed)
## Simple imputation (median/mode) should be sufficient for first-time modelers, KNN is recommended but more complex.
# Conduct feature selection
## Select a method (I recommend stepwise; random forest (RF) or logistic regression (LOGR))
## Select a threshold (I recommend 5% or 1%)

=== Modeling & Evaluation ===

# Select 3 model algorithms for training and comparison (I recommend RF, gradient boosting (XGB), & LOGR)
## Decision tree classifier (DTC) & support vector machine (SVM) algorithms are also fine
# Record accuracy, AUROC, f1, precision, recall, and confusion matrices for each model.
# Generate a SHAP summary plot to visualize feature importance
# OPTIONAL: conduct hyperparameter tuning using random search, grid search, cross validation, etc.

=== Single-Patient Predictions ===

# Once trained, create a separate .py script to run predictions using the model of choice.
## Save the fitted scaler, trained model, and feature names to pickle files using joblib, then load them into the new file.
# Check for missing data, extra columns, and reorder columns as needed to match the training dataset.
# Apply the saved scaler using .transform() only (not fit.transform())
# Generate a class prediction and print the results clearly.
## OPTIONAL: generate a probability score in addition to the class prediction
# Validate the script by testing several single-patient predictions.

=== Documentation ===

# Each step of your script should include a comment indicating the intended function.
# Each model must be accompanied by a [https://wiki.biocomputeobject.org/Main_Page BioCompute Object].