BioMuta pipeline README

Under construction

This article is still under construction and should not be nominated for deletion.

Originally updated by Ned Cauley (August 2022); currently maintained by Maria Kim (September 2024).

This page will contain an updated version of this BioMuta documentation page.

Description

The Biomuta pipeline gathers mutation data from various sources and combines them into a single dataset under common field structure.

The sources included in BioMuta are:

• The Cancer Genome Atlas (TCGA)
• Clinical Interpretation of Variants in Cancer (CIVIC)
• Catalogue of Somatic Mutations in Cancer (COSMIC)
• International Cancer Genome Consortium (ICGC) (retired)

BioMuta gathers mutation data for the following cancers:

• Urinary Bladder Cancer (DOID:11054)
• Breast Cancer (DOID:1612)
• Colorectal (DOID:9256)
• Esophageal Cancer (DOID:5041)
• Head and Neck Cancer (DOID:11934)
• Kidney Cancer (DOID:263)
• Liver Cancer (DOID:3571)
• Lung Cancer (DOID:1324)
• Prostate Cancer (DOID:10283)
• Stomach Cancer (DOID:10534)
• Thyroid Gland Cancer (DOID:1781)
• Uterine Cancer (DOID:363)
• Cervical Cancer (DOID:4362)
• Brain Cancer (DOID:1319)
• Hematologic Cancer (DOID:2531)
• Adrenal Gland Cancer (DOID:3953)
• Pancreatic Cancer (DOID:1793)
• Ovarian Cancer (DOID:2394)
• Skin Cancer (DOID:4159)

Running the Pipeline

To run the BioMuta pipeine, download the scripts from the HIVE Lab github repo: GW HIVE BioMuta Repository.

Pipeline Overview

Step 1: Download

In the downloader step, mutation lists will be downloaded from each source. Refer to each individual source below for downloading instructions.

Download: TCGA

Annotated variant files are downloaded from the ISB-CGC Big Query repository.

Field descriptions for Big Query output available in field_names_descriptions.csv. Additional field descriptions available on GDC docs.

The list of studies used in TCGA can be found here: List of TCGA studies.

Gain access to data

There are two parts to obtaining data from TCGA:

1. Primary TCGA data

• Available at NCI Genomic Data Commons.
• Accessible through the ISB-CGC Big Query repository (more info TBA). For complete documentation, see the ISB-CGC Read the Docs pages.

2. TCGA controlled-access data

• Hosted at dbGaP. For information on how to get access, see Sharepoint.

Run downloader R script using R Studio

Required script: TCGA_mutation_download.R

Run each line one after the other, instead of the whole script at once.

Running library(bigrquery) and calling this library with bq_project_query() (later in the script) will open a browser to login with Google credentials.

• Use the Google account registered for a ISB-CGC project and with dbGaP authorization.
• After logging in, a token will be saved so that you can login through R Studio instead.

This script will download all mutation data for TCGA.

Since the downloaded file is very large, there might be issues running this script. If this is the case, run the following scripts in the tcga folder:

• TCGA_mutation_download_part1.R
• TCGA_mutation_download_part2.R
• TCGA_mutation_download_part3.R
• TCGA_mutation_download_part4.R

These scripts will download a set of the TCGA studies, so that the downloaded file size is smaller.

Additional information can be found here: TCGA Additional Information.

Download: CIViC

A VCF for the monthly relaease of accepted variants was downloaded from: https://civicdb.org/releases/main

Download: COSMIC

There are three COSMIC mutation datasets for coding mutations:

• COSMIC Complete Mutation Data (Targeted Screens)

A tab separated table of the complete curated COSMIC dataset (targeted screens) from the current release. It includes all coding point mutations, and the negative data set.

• COSMIC Mutation Data (Genome Screens)

A tab separated table of coding point mutations from genome wide screens (including whole exome sequencing).

• COSMIC Mutations Data

A tab separated table of all COSMIC coding point mutations from targeted and genome wide screens from the current release.

The COSMIC Mutations Data set was chosen because it combines both the Targeted and Genome Screens.

Downloaded File: COSMIC_SNPs_June_2022.tsv

NOTE Downloading the mutation datasets requires a COSMIC login. With an academic email address, an account can be created for free and the download can be performed.

Fields

The COSMIC dataset contains a large number of fields, many of which were filtered out in order to speed up processing in subsequent steps.

A ‘simplified’ version of the file was used by selecting specific columns from the original downloaded file using the command line tool **awk**

Fields in Simplified Version

Field Name	Example
Accession Number	ENST00000404621.5
Sample name	H_LV-3334-1316090
Primary site	breast
Mutation CDS	c.644C>G
Mutation AA	p.S215*
Mutation genome position	12:1244466234-124466234

All Fields from COSMIC and Field Descriptions

From 'File Description' drop down menu below 'Cosmic Mutation Data' (on downloads page):

Field Name	Description
Gene name	The gene name for which the data has been curated.
Accession Number	The transcript identifier of the gene.
Gene CDS length	Length of the gene (base pair) counts.
HGNC id	If gene is in HGNC, this id helps linking it to HGNC.
Sample name	Sample id, Id tumor A assigned.
Primary Site	The primary tissue/cancer from which the sample originates.
Site Subtype 1	Further sub classification (level 1) of the sample’s tissue.

Download: ICGC

A VCF for release 28 was downloaded from https://dcc.icgc.org/releases/release_28/Summary

Downloaded File: simple_somatic_mutation.aggregated.vcf.gz

Step 2: Convert

In the convert step, all resources are formatted to the Biomuta standard for both data and field structure.

For each resource, a unique script is used to convert from the raw format provided by the resource, to a format aligned with past versions of the Biomuta pipeline.

With a common format, all resources can then be combined into a master dataset.

See the individual resource pages for details on the conversion:

Convert: TCGA

Scripts

• process_tcga_download.py

Procedure

Run process_tcga_download.py

Summary

The python script `process_tcga_download.py` will take the output of the TCGA download step and:

• Map the data to:

 * uniprot accession
 * doid parent terms

• Rename fields
• Reformat fields:

 * amino acid change and position
 * chromosome id

• Filter out unnecessary fields

Script Specifications

The script must be called from the command line and takes specific command line arguments.

Input

• -i: A path to the input csv to reformat
• -m: A path to the folder containing all mappings
• -d: A path to the tcga study to doid mapping file
• -e: A path to the ENSP to uniprot mapping file
• -o: A path to the output folder

Output

• A data report comparing new AA sites to old AA sites for Biomuta

Usage

• `python process_tcga_download.py -h`

Gives a description of the necessary commands*

• `python process_tcga_download.py -i <path/input_file.vcf> -m <path/> -d <doid_mapping.csv> -e <ensp_mapping.csv> -o <path/>`

Runs the script with the given input TCGA CSV and outputs a formatted CSV*

Additional Notes - tcga

Convert: CIVIC

Scripts

• genomic liftover > convert_civic_vcf.py > map_civic_csv.py

Procedure

Perform liftover of mutations from GRCh37 to GRCh38

Summary

The most recent data release for CIVIC is aligned to the GRCH37 human reference genome. For this update, we are using the human reference genome GRCh38.

To convert coordinates between the two reference genomes, we use a ‘liftover’ tool to remap the genomic coordinates. The CIVIC file is very small in size, so we can use the ENSEMBL online liftover tool: [1](https://useast.ensembl.org/Homo_sapiens/Tools/AssemblyConverter?db=core)

Run the downloaded VCF through the tool with the default parameters (change the file type to VCF).

Redownload the transformed VCF and use that VCF for the next step.

Run convert_civic_vcf.py

Summary

The python script `convert_civic_vcf.py` will convert the VCF formatted file to a CSV file.

With the VCF format, each mutation line in the file can contain multiple annotations and annotation-specific information.

The output CSV format will contain only one annotation per line with associated annotation-specific information.

In order to know how the information for the mutation and annotation fields are structured, a schema describing the fields is provided to the script.

Example Line Transformation

Input VCF lines

mutation A info | mutation A annotation 1 info | mutation A annotation 2 info

mutation B info | mutation B annotation 1 info | mutation B annotation 2 info | mutation B annotation 3 info

Output CSV lines

mutation A info,annotation 1 info

mutation A info,annotation 2 info

mutation B info,annotation 1 info

mutation B info,annotation 2 info

mutation B info,annotation 3 info

Script Specifications

The script must be called from the command line and takes specific command line arguments:

Input

• -i: A path to the CIVIC VCF file
• -p: A prefix used for naming the output files
• -o: A path to the output folder, where the mutation data CSV will go

Output

• A CSV file with mutation data

Usage

python convert_civic_vcf.py -h

Gives a description of the necessary commands

python convert_civic_vcf.py -i <path/input_file.vcf> -s <path/schema.json> -o <path/> Runs the script with the given input VCF and outputs a CSV file.

Run map_civic_csv.py

Summary

The python script map_civic_csv.py will take the output of the TCGA download step and:

Map the data to: uniprot accessions doid parent terms Rename fields Reformat fields: amino acid change and position chromosome id genomic location nucleotide change remove indels transform NA values

Script Specifications

The script must be called from the command line and takes specific command line arguments:

Input

-i: A path to the CIVIC CSV file -m: A path to the folder containing mapping files -d: The name of the doid mapping file -e: The name of the ensp to uniprot accession mapping file -o: A path to the output folder Output

A CSV file with mutation data mapped to doid terms and uniprot accessions

Usage

python map_civic_csv.py -h

Gives a description of the necessary commands

python map_civic_csv.py -i <path/input_file.vcf> -m <path/mapping_folder> -d <doid_mapping_file_name> -e <ensp_mapping_file_name> -o <path/>

Runs the script with the given input CSV and outputs a CSV with mutation mapped to doid terms and uniprot accessions.

Additional notes- CIVIC

Convert: COSMIC

Scripts

• map_cosmic_tsv.py

Procedure

Run map_cosmic_tsv.py

Summary

The python script `map_cosmic_tsv.py` will take the output of the TCGA download step and:

• Map the data to:

 * uniprot accessions
 * doid parent terms

• Rename fields
• Reformat fields:

 * amino acid change and position
 * chromosome id
 * genomic location
 * nucleotide change

Script Specifications

The script must be called from the command line and takes specific command line arguments.

Input

• -i : A path to the cosmic tsv mutation file
• -m : A path to the folder containing mapping files
• -d : The name of the doid to cosmic cancer type mapping file
• -e : The name of the enst to uniprot accession mapping file
• -o : A path to the folder to export the final mapped mutations

Output

• A mutation file with COSMIC mutations mapped to doid terms and uniprot accessions

Usage

map_cosmic_tsv -h

Gives a description of the necessary commands

python map_cosmic_tsv.py -i <path/cosmic_file_name.tsv> -m <path/mapping_folder> -d <doid_mapping_file_name> -e <enst_mapping_file_name> -o <path/output_folder>

Runs the script with the given input file and exports the mapped mutation file.

Additional Notes for COSMIC

Convert: ICGC

Scripts

• genomic liftover (mapvcf_copySA.py) > convert_icgc_vcf.py > map_icgc.py

Procedure

Perform liftover of mutations from GRCh37 to GRCh38 (mapvcf_copySA)

Summary

The most recent data release for ICGC is aligned to the GRCH37 human reference genome. For this update, we are using the human reference genome GRCh38.

To convert coordinates between the two reference genomes, we use a ‘liftover’ tool to remap the genomic coordinates.

Seun performed the liftover and provided the notes listed below.

Genomic Liftover Notes

VCF A VCF (Variant Call Format) file is a text file used to store gene sequence variations. The files often start with lines of metadata, then headers relating to the variants described. Because the standard for formatting and relaying genomic data is always evolving, there are numerous versions and references for VCF files and the dependencies they use.

Fields

Common fields for VCF files

Converting with CrossMap

CrossMap is a program that can convert genome coordinates between different assemblies, such as hg18 (GRCh36) to hg19 (GRCh37). It is made in Python and offered as a webtool, by Ensembl in limited capacity or as a local script for full functionality. This gives extra customizability and the option to convert files over 50 mb, it is necessary to run a local edition of CrossMap.

Crossmap Documentation: [2](http://crossmap.sourceforge.net/)

Requirements

• Python2 or Python3 installed on a Linux server
• Chain file - describes a pairwise alignment between two reference assemblies
• They can be found through UCSC, Ensembl, and other sources
• Compressed files are allowed
• `hg19ToHg38.over.chain` was best tested
• Target, input file - file to be converted in format compatible with CrossMap
• CrossMap supports vcf, bam/cram/sam, maf, and other formats; compressed files are allowed
• Referencefile - fasta format of the wanted genome assembly

Other files used

• `mapvcf` is the script from the package that does the conversion, attached is the version I used. I believe commenting out lines 100:109 is what allowed it to work.
• `hg19ToHg38` is the chain file that I used
• This is the command I used to get the assembly file from UCSC: `wget http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz`
• This is the command I used to unzip the assembly file: `gzip -dk hg38.fa.gz`
• This is the exact command I ran to create the file: `python3 ./local/bin/CrossMap.py vcf /mnt/d/hg19ToHg38.over.chain.gz /mnt/d/icgc_missense_mutations.vcf /mnt/d/icgc_missense_mutations_38_hg7.vcf`
• Of note, there are numerous other assembly and chain files. I tried 3 or 4 of each and the ones linked here were the best. I determined best by both what the script relays and how big the final vcf file were.

Output

Two output files were generated from the liftover and stored on the OncoMX-tst server at `/software/pipeline/integrator/downloads/biomuta/v-5.0/icgc/ - icgc_missense_mutations_38.vcf`

• All mutations with converted coordinates
• `icgc_missense_mutations_38_fail.vcf` - Mutations whose coordinates could not be converted

Only the mutations whose coordinates were successfully converted were carried forward in the pipeline.

Run convert_icgc_vcf.py

Summary

The python script `convert_icgc_vcf.py` will convert the VCF formatted mutation file to a CSV file.

With the VCF format, each mutation line in the file can contain multiple annotations and annotation-specific information.

The output CSV format will contain only one annotation per line with associated annotation-specific information.

In order to know how the information for the mutation and annotation fields are structured, a schema describing the fields is provided to the script.

Example Line Transformation

Input VCF lines

mutation A info | mutation A annotation 1 info | mutation A annotation 2 info

mutation B info | mutation B annotation 1 info | mutation B annotation 2 info | mutation B annotation 3 info

Output CSV lines

mutation A info,annotation 1 info

mutation A info,annotation 2 info

mutation B info,annotation 1 info

mutation B info,annotation 2 info

mutation B info,annotation 3 info

Script Specifications

The script must be called from the command line and takes specific command line arguments:

Input

• -i: A path to the ICGC VCF file
• -s : A schema file containing the field names in the annotations and to use for the output file
• -o: A path to the output folder, where the mutation data CSV will go

Output

• A .csv file with mutation data where each row contains one mutation and one unique annotation

Usage

python convert_icgc_vcf.py -h

Gives a description of the necessary commands

python convert_icgc_vcf.py -i <path/input_file.vcf> -s <path/schema.json> -o <path/>

Runs the script with the given input VCF and outputs a CSV file.

Run map_icgc.py

Summary

The python script `map_icgc.py` will take the output of the VCF convertor script and:

• Map the data to:

 * uniprot accessions
 * doid parent terms

• Rename fields
• Reformat fields:

 * amino acid change and position
 * chromosome id
 * genomic location
 * nucleotide change

Script Specifications

The script must be called from the command line and takes specific command line arguments.

Input

• -i: A path to the ICGC .csv file
• -m: A path to the folder containing mapping files
• -d: The name of the doid mapping file
• -e: The name of the ensp to uniprot accession mapping file
• -o: A path to the output folder

Output

• A .csv file with mutation data formatted to the biomuta field structure

Usage

python map_icgc.py -h

Gives a description of the necessary commands

python map_icgc.py -i <path/input_file.vcf> -m <path/> -d doid_mapping_file.csv -e enst_mapping_file.csv -o <path/>

Runs the script with the given CSV file and outputs a CSV file formatted for the final biomuta master file.

Additional Notes

Step 3: Combine

In the combined step, all resources are combined into a master dataset.

Scripts

• combine_csv.py

Procedure

Run combine_csv.py

Summary

All of the mutation data for each source was converted to a standardized data structure in the convert step.

Now, all of these separate csv files (one for each source) will be combined into a master CSV file.

All CSV files to be combined should be in a folder together with no additional CSV files.

Script Specifications

The script must be called from the command line and takes specific command line arguments.

Input

• -i : The folder containing CSV mutation files to combine
• -o : The folder to output the combined mutation file

Output

• A CSV file combining all CSV files in a given folder

Usage

python combine_csv.py -h

Gives a description of the necessary commands

python combine_csv.py -i <path/> -o <path/>

Runs the script with the given folder and combines all CSV files in that folder

Final Fields