BioXpress pipeline README - Revision history

Hivelabwikiadmin: /* Major Changes from v-4.0 */

2024-10-17T14:49:26Z

Major Changes from v-4.0

← Older revision		Revision as of 14:49, 17 October 2024
Line 355:		Line 355:

	[[Major updates to the BioXpress from the previous version (v-4.0)]]		[[Major updates to the BioXpress from the previous version (v-4.0)]]


			==Post-processing for OncoMX and Glygen==

			''Processing done for integration of BioXpress data into OncoMX and Glygen.''

			===Processing for OncoMX===
			The final output from BioXpress v-5.0 is available on the OncoMX-tst server at the path:
			<code>/software/pipeline/integrator/downloads/bioxpress/v-5.0/</code>

			For OncoMX, the <code>de_per_tissue.csv</code> is used to report gene expression per tissue, however [https://data.oncomx.org data.oncomx.org] hosts both per tissue and per study datasets. The files are processed with the recipe pipeline. The recipes filter for all genes that are successfully mapped to uniprotkb accession IDs.

			'''Recipes'''
			<code>human_cancer_mRNA_expression_per_study.json</code>
			<code>human_cancer_mRNA_expression_per_tissue.json</code>

			The output is available on the OncoMX-tst server at the path:
			<code>/software/pipeline/integrator/unreviewed</code>

			'''Final output files'''
			<code>human_cancer_mRNA_expression_per_study.csv</code>
			<code>human_cancer_mRNA_expression_per_tissue.csv</code>

			===Processing for Glygen===
			The final output from BioXpress v-5.0 was modified to align with the previous input for cancer gene expression and now includes the following columns:
			* '''pmid'''
			* '''sample_name'''
			** Same as DOID and name
			* '''parent_doid'''
			** Same as DOID
			** All DOIDs in v-5.0 are parent terms
			* '''parent_doname'''
			** Same as DOID and name
			** All DOIDs in v-5.0 are parent terms
			* '''sample_id'''
			** Taken from previous version, unclear on the origin of these numbers

			The following mapping for the column <code>sample_id</code> was recovered from the previous version and mapped to DOIDs present in v-5.0:

			{\| class="wikitable"
			! sample_name !! sample_id
			\|-
			\| DOID:10283 / Prostate cancer [PCa] \|\| 42
			\|-
			\| DOID:10534 / Stomach cancer [Stoca] \|\| 19
			\|-
			\| DOID:11054 / Urinary bladder cancer [UBC] \|\| 34
			\|-
			\| DOID:11934 / Head and neck cancer [H&NC] \|\| 46
			\|-
			\| DOID:1612 / Breast cancer [BRCA] \|\| 70
			\|-
			\| DOID:1781 / Thyroid cancer [Thyca] \|\| 16
			\|-
			\| DOID:234 / Colon adenocarcinoma \|\| 3
			\|-
			\| DOID:263 / Kidney cancer [Kidca] & Kidney renal clear cell carcinoma \|\| 61
			\|-
			\| DOID:3571 / Liver cancer [Livca] \|\| 60
			\|-
			\| DOID:3907 / Lung squamous cell carcinoma \|\| 33
			\|-
			\| DOID:3910 / Lung adenocarcinoma \|\| 53
			\|-
			\| DOID:4465 / Papillary renal cell carcinoma \|\| 57
			\|-
			\| DOID:4471 / Chromophobe adenocarcinoma \|\| 23
			\|-
			\| DOID:5041 / Esophageal cancer [EC] \|\| 32
			\|}

			The processed file for Glygen is available on the glygen-vm-dev server at:
			<code>/software/pipeline/integrator/downloads/bioxpress/August_2021/human_cancer_mRNA_expression_per_tissue_glygen.csv</code>

Hivelabwikiadmin: /* Publisher Step 2: Run the script de-publish-per-tissue.py */

2024-10-17T14:39:40Z

Publisher Step 2: Run the script de-publish-per-tissue.py

← Older revision		Revision as of 14:39, 17 October 2024
Line 350:		Line 350:
	'''Output'''		'''Output'''
	A csv file with the DESeq output for all tissues, mapped to DO IDs, uniprot accession ids, and refseq ids. The path is specified in the script as one of the hard-coded lines edited during the method.		A csv file with the DESeq output for all tissues, mapped to DO IDs, uniprot accession ids, and refseq ids. The path is specified in the script as one of the hard-coded lines edited during the method.


			== Major Changes from v-4.0 ==

			[[Major updates to the BioXpress from the previous version (v-4.0)]]

Hivelabwikiadmin: /* DESeq Step 3: Run the script run_per_case.sh */

2024-10-17T14:37:43Z

DESeq Step 3: Run the script run_per_case.sh

← Older revision		Revision as of 14:37, 17 October 2024
Line 300:		Line 300:
	* <code>distance_heatmap.png</code>		* <code>distance_heatmap.png</code>
	* <code>pca.png</code> - Principal component analysis plot		* <code>pca.png</code> - Principal component analysis plot

			== BioXpress Publisher Step ==
			'''Step 4 of the BioXpress pipeline'''

			=== General Flow of Scripts ===
			<code>de-publish-per-study.py</code> -> <code>de-publish-per-tissue.py</code>

			=== Procedure ===

			==== Publisher Step 1: Run the script <code>de-publish-per-study.py</code> ====

			'''Summary'''
			The python script <code>de-publish-per-study.py</code> takes the output from running DESeq in the previous step for each TCGA study and combines it into one master file.

			'''Method'''
			Edit the hard-coded paths in the script <code>de-publish-per-study.py</code>:

			* Specify the <code>in_file</code> for the disease ontology mapping file (line ~26)
			* Specify the <code>in_file</code> for the uniprot accession id (protein id) mapping file (line ~40)
			* Specify the <code>in_file</code> for the refseq mapping file (line ~51)
			* Specify the <code>in_file</code> for the list of TCGA studies to include in the final output (line ~72)
			* Specify the <code>deseq_dir</code> for the folder containing all deseq output (line ~80)
			* Specify the path to write the output (line ~135)

			Run the python script:
			<code>python de-publish-per-study.py</code>

			'''Output'''
			A csv file with the DESeq output for all TCGA studies, mapped to DO IDs, uniprot accession ids, and refseq ids. The path is specified in the script as one of the hard-coded lines edited during the method.

			==== Publisher Step 2: Run the script <code>de-publish-per-tissue.py</code> ====

			'''Summary'''
			The python script <code>de-publish-per-tissue.py</code> takes the output from running DESeq in the previous step for each tissue and combines it into one master file.

			'''Method'''
			Edit the hard-coded paths in the script <code>de-publish-per-study.py</code>:

			* Specify the <code>in_file</code> for the disease ontology mapping file (line ~26)
			* Specify the <code>in_file</code> for the uniprot accession id (protein id) mapping file (line ~40)
			* Specify the <code>in_file</code> for the refseq mapping file (line ~51)
			* Specify the <code>in_file</code> for the list of tissues to include in the final output (line ~72)
			* Specify the <code>deseq_dir</code> for the folder containing all deseq output (line ~80)
			* Specify the path to write the output (line ~135)

			Run the python script:
			<code>python de-publish-per-tissue.py</code>

			'''Output'''
			A csv file with the DESeq output for all tissues, mapped to DO IDs, uniprot accession ids, and refseq ids. The path is specified in the script as one of the hard-coded lines edited during the method.

Hivelabwikiadmin: /* Output */

2024-10-17T14:34:59Z

Output

← Older revision		Revision as of 14:34, 17 October 2024
Line 213:		Line 213:
	==== Output ====		==== Output ====
	A folder is generated for each case ID that has a tumor sample and a normal tissue sample. Two files are generated per case: read counts and categories. These files are needed to run DESeq per case.		A folder is generated for each case ID that has a tumor sample and a normal tissue sample. Two files are generated per case: read counts and categories. These files are needed to run DESeq per case.

			== BioXpress DESeq Step ==
			'''Step 3 of the BioXpress pipeline'''

			=== General Flow of Scripts ===
			<code>run_per_study.py</code> -> <code>run_per_tissue.py</code> -> <code>run_per_case.py</code>

			=== Procedure ===
			==== DESeq Step 1: Run the script <code>run_per_study.sh</code> ====
			'''Summary'''

			The python script <code>run_per_study.py</code> provides arguments to the R script <code>deseq.R</code>. The count and category files generated from the Annotation step are used to calculate differential expression and statistical significance. The result is a series of files per tissue including the normalized reads (DESeq normalization method), the DE results and significance, and QC files such as the PCA plot.

			* '''Note:''' This step is time consuming (~2-3 hours of run time)

			'''Method'''

			Edit the hard-coded paths in the script <code>run_per_tissue.py</code>:
			* Specify the <code>in_dir</code> to be the folder containing the final output files of the Annotation steps for per study
			* Specify the <code>out_dir</code>
			* Ensure that the file <code>list_files/studies.csv</code> contains all of the tissues you wish to process - '''Note:''' the studies can be run separately (in the event that 2-3 hours cannot be dedicated to run all the studies at once) by creating separate dat files with specific tissues to run

			Run the shell script:
			<code>sh run_per_study.sh</code>

			* '''Note:''' the R libraries specified in <code>deseq.R</code> will need to be installed if running on a new server or system, as these installations are not included in the scripts

			'''Output'''
			A set of files:
			* log file
			* <code>deSeq_reads_normalized.csv</code> - Normalized read counts (DESeq normalization method applied)
			* <code>results_significance.csv</code> - log2fc differential expression results and statistical significance (t-test)
			* <code>dispersion.png</code>
			* <code>distance_heatmap.png</code>
			* <code>pca.png</code> - Principal component analysis plot, important for observing how well the Primary Tumor and Solid Tissue Normal group together

			==== DESeq Step 2: Run the script <code>run_per_tissue.sh</code> ====
			'''Summary'''

			The python script <code>run_per_tissue.py</code> provides arguments to the R script <code>deseq.R</code>. The count and category files generated from the Annotation step are used to calculate differential expression and statistical significance. The result is a series of files per study including the normalized reads (DESeq normalization method), the DE results and significance, and QC files such as the PCA plot.

			* '''Note:''' This step is time consuming (~2-3 hours of run time)

			'''Method'''

			Edit the hard-coded paths in the script <code>run_per_tissue.py</code>:
			* Specify the <code>in_dir</code> to be the folder containing the final output files of the Annotation steps for per tissue
			* Specify the <code>out_dir</code>
			* Ensure that the file <code>list_files/tissue.dat</code> contains all of the tissues you wish to process - '''Note:''' the tissues can be run separately by creating specific dat files

			Run the shell script:
			<code>sh run_per_tissue.sh</code>

			'''Output'''
			A set of files:
			* log file
			* <code>deSeq_reads_normalized.csv</code> - Normalized read counts (DESeq normalization method applied)
			* <code>results_significance.csv</code> - log2fc differential expression results and statistical significance (t-test)
			* <code>dispersion.png</code>
			* <code>distance_heatmap.png</code>
			* <code>pca.png</code> - Principal component analysis plot

			==== DESeq Step 3: Run the script <code>run_per_case.sh</code> ====
			'''Summary'''

			The python script <code>run_per_case.py</code> provides arguments to the R script <code>deseq.R</code>. The count and category files generated from the Annotation step are used to calculate differential expression and statistical significance. The result is a series of files per case including the normalized reads (DESeq normalization method), the DE results and significance, and QC files such as the PCA plot.

			* '''Note:''' This step is time consuming (~2-3 hours of run time)

			'''Method'''

			Edit the hard-coded paths in the script <code>run_per_case.py</code>:
			* Specify the <code>in_dir</code> to be the folder containing the final output files of the Annotation step for per case
			* Specify the <code>out_dir</code>
			* Ensure that the file <code>list_files/cases.csv</code> contains all of the cases you wish to process

			Run the shell script:
			<code>sh run_per_case.sh</code>

			'''Output'''
			A set of files:
			* log file
			* <code>deSeq_reads_normalized.csv</code> - Normalized read counts (DESeq normalization method applied)
			* <code>results_significance.csv</code> - log2fc differential expression results and statistical significance (t-test)
			* <code>dispersion.png</code>
			* <code>distance_heatmap.png</code>
			* <code>pca.png</code> - Principal component analysis plot

Hivelabwikiadmin: /* Output */

2024-10-17T14:23:18Z

Output

← Older revision		Revision as of 14:23, 17 October 2024
Line 211:		Line 211:
	<code>python split_per_case.py</code>		<code>python split_per_case.py</code>

	=== Output ===		==== Output ====
	A folder is generated for each case ID that has a tumor sample and a normal tissue sample. Two files are generated per case: read counts and categories. These files are needed to run DESeq per case.		A folder is generated for each case ID that has a tumor sample and a normal tissue sample. Two files are generated per case: read counts and categories. These files are needed to run DESeq per case.

Hivelabwikiadmin: /* Method */

2024-10-17T14:23:11Z

Method

← Older revision		Revision as of 14:23, 17 October 2024
Line 198:		Line 198:
	The python script `split_per_case.py` takes case and sample IDs from the sample sheets downloaded from the GDC data portal and splits annotation data so that there is one folder per case with only that case’s annotation data.		The python script `split_per_case.py` takes case and sample IDs from the sample sheets downloaded from the GDC data portal and splits annotation data so that there is one folder per case with only that case’s annotation data.

	=== Method ===		==== Method ====
	Edit the hard-coded paths in `split_per_case.py`		Edit the hard-coded paths in `split_per_case.py`

Hivelabwikiadmin: /* Summary */

2024-10-17T14:23:02Z

Summary

← Older revision		Revision as of 14:23, 17 October 2024
Line 195:		Line 195:
	=== Annotation Step 3: Run the script split_per_case.py ===		=== Annotation Step 3: Run the script split_per_case.py ===

	=== Summary ===		==== Summary ====
	The python script `split_per_case.py` takes case and sample IDs from the sample sheets downloaded from the GDC data portal and splits annotation data so that there is one folder per case with only that case’s annotation data.		The python script `split_per_case.py` takes case and sample IDs from the sample sheets downloaded from the GDC data portal and splits annotation data so that there is one folder per case with only that case’s annotation data.

Hivelabwikiadmin: /* Annotation Step 3: Run the script split_per_case.py */

2024-10-17T14:22:37Z

Annotation Step 3: Run the script split_per_case.py

← Older revision		Revision as of 14:22, 17 October 2024
Line 193:		Line 193:


	== Annotation Step 3: Run the script split_per_case.py ==		=== Annotation Step 3: Run the script split_per_case.py ===

	=== Summary ===		=== Summary ===

Hivelabwikiadmin: /* Output */

2024-10-17T14:21:22Z

Output

← Older revision		Revision as of 14:21, 17 October 2024
Line 191:		Line 191:
	==== Output====		==== Output====
	Read count and category files are generated for each tissue specified in the tissues.csv file.		Read count and category files are generated for each tissue specified in the tissues.csv file.


			== Annotation Step 3: Run the script split_per_case.py ==

			=== Summary ===
			The python script `split_per_case.py` takes case and sample IDs from the sample sheets downloaded from the GDC data portal and splits annotation data so that there is one folder per case with only that case’s annotation data.

			=== Method ===
			Edit the hard-coded paths in `split_per_case.py`

			* Edit the line (line ~29) <code>in_file = "/data/projects/bioxpress/v-5.0/generated/misc/studies.csv"</code> with the version for your current run of BioXpress
			* Edit the line (line ~38) <code>in_file = "/data/projects/bioxpress/v-5.0/downloads/sample_list_from_gdc/gdc_sample_sheet.primary_tumor.tsv"</code> with the version for your current run of BioXpress as well as the same of the sample sheet for tumor samples downloaded from the GDC data portal
			* Edit the line (line ~57) <code>in_file = "/data/projects/bioxpress/v-5.0/downloads/sample_list_from_gdc/gdc_sample_sheet.solid_tissue_normal.tsv"</code> with the version for your current run of BioXpress as well as the same of the sample sheet for normal samples downloaded from the GDC data portal
			* Edit the line (line ~81) <code>out_file_one = "/data/projects/bioxpress/v-5.0/generated/annotation/per_case/%s.%s.htseq.counts" % (study_id,case_id)</code> with the version for your current run of BioXpress
			* Edit the line (line ~82) <code>out_file_two = "/data/projects/bioxpress/v-5.0/generated/annotation/per_case/%s.%s.categories" % (study_id,case_id)</code> with the version for your current run of BioXpress
			* Edit the line (line ~85) <code>in_file = "/data/projects/bioxpress/v-5.0/generated/annotation/per_study/%s.htseq.counts" % (study_id)</code> with the version for your current run of BioXpress

			Run the python script:
			<code>python split_per_case.py</code>

			=== Output ===
			A folder is generated for each case ID that has a tumor sample and a normal tissue sample. Two files are generated per case: read counts and categories. These files are needed to run DESeq per case.

Hivelabwikiadmin: /* Method */

2024-10-17T14:21:00Z

Method

← Older revision		Revision as of 14:21, 17 October 2024
Line 186:		Line 186:
	<code >in_file = "/data/projects/bioxpress/v-5.0/generated/annotation/per_study/%s.htseq.counts" % (study_id)</code>		<code >in_file = "/data/projects/bioxpress/v-5.0/generated/annotation/per_study/%s.htseq.counts" % (study_id)</code>
	with the version for your current run of BioXpress.		with the version for your current run of BioXpress.

			Run the python script <code> python merge_per_tissue.py </code>

			==== Output====
			Read count and category files are generated for each tissue specified in the tissues.csv file.