Medical Genetics

Advances in next-generation sequencing technologies allow researchers to generate massive amounts of sequence data. When clinical samples are sequenced using these technologies, novel variants that have causative roles in disease may be identified. dbSNP’s role is to manage information on the location and type of these novel variants, while ClinVar manages the current interpretation of the variants' clinical phenotype.

dbSNP integrates clinical attribute data from ClinVar (i.e., clinical assertions and asserted allele origin) into new and existing human refSNP records, as well as into additional curated attribute data that includes minor allele frequency and variation false positive status.

VCF files generated from dbSNP’s curated records can be used to filter (subtract) known variants from a set of variant calls to identify novel variants or narrow a list of potential causative variants that might warrant further evaluation.

Genome Mapping

Variations are used as positional markers in genetic and physical mapping of nucleotide sequences when they map to a unique location in a genome. In other words, the variations represented by dbSNP records can serve as stable landmarks in the genome even if the variation is fixed for one allele in a sample. When multiple alleles are observed in a sample pedigree, pedigree members can be tested for variation genotypes as in traditional genetic mapping studies. To aid in such mapping efforts, dbSNP updates variation mapping and annotation for each organism with the release of every new genome assembly.

Molecular and Functional Consequences

Variations that occur in functional regions of genes or in conserved non-coding regions might affect transcription, post-transcriptional processing, or a protein product. dbSNP computes the molecular consequence of any sequence change, based on NCBI’s annotation of the genome. Functional consequences may be reported from submitters.

Association Studies

dbSNP annotates variations with significant association to phenotype from Genome Wide Association Studies (GWAS) as reported by dbGAP and provides a detailed catalog of common variations. dbSNP’s GWAS annotations and common variation catalog are used to inform the design of GWAS studies, the creation of variation arrays used in GWAS studies, and the interpretation of GWAS study results.

Creation and Growth

dbSNP was established in September,1998, to address the need for a general catalog of genomic variation that would facilitate the scientific community’s efforts in genetic association studies, gene mapping, and evolutionary biology. Initially, dbSNP was composed of small-scale locus specific submissions defined by flanking invariant sequence. Following the advent of high-throughput sequencing and the availability of complete genome assemblies for many organisms, however, dbSNP now receives a greater number of variants defined by sequence change at asserted locations on a reference sequence.

Evolution in Submitted Content

Because dbSNP was developed before a human reference assembly was available, initial submissions were primarily from human and defined a variant sequence in the context of flanking sequence. There was often little supporting evidence or validation data. As sequencing and other discovery technologies have changed, dbSNP has grown apace, and now includes data from over 300 organisms as well as ample validation data, including multiple independent submissions, frequency data, genotype data, and allele observations. To meet community needs for a centralized variation database, in the spring of 2008, dbSNP began accepting clinical assertions for new and existing variations as well as asserted locations for variation placement. The ClinVar database, now has the role of accepting variation clinical assertion data, and following its own accession process, will route novel variation positions to dbSNP for the assignment of ss (submitted SNP) and rs (refSNP) numbers.

In addition to integrating clinical assertions into refSNP records, dbSNP has introduced other curated attributes into refSNP records, such as minor allele frequency, asserted allele origin, and potential false positive status. It also uses the curated records to generate VCF files that can be employed to filter variation calls for the presence of novel variations and identify potential causative variants.

Usage Evolution

Originally, the data in dbSNP was used only to populate sequence maps since polymorphic marker density was too low to allow further application of the data. By 2007, however, marker density had increased enough to allow for the use of variation data in association studies, high resolution mapping, and a host of other applications, including population evolution and phylogeny studies that continue to further our understanding of genetic relationships and the genomic basis of traits.

dbSNP’s current integration of clinical information into dbSNP records will allow greater application of dbSNP data to the fields of molecular medicine and pharmacogenetics, as well as emerging fields study such as pharmacometabolomics and precision medicine.

Submitted Content

dbSNP accepts submissions from public laboratories and private organizations. dbSNP does not accept synthetic mutations or variations ascertained from cross-species alignments and analysis. Variations > 50 nucleotides in length should be submitted to the Database of Genomic Structural Variation (dbVAR).

dbSNP will not hold data to be released on a particular date or in a particular dbSNP build. If, however, you are submitting non-clinical human data or non-human data and your manuscript requires dbSNP accession numbers (ss numbers) for the review process, we can hold the submitted data until the publication is accepted and you have notified us that dbSNP can release the data. Once notification has been given, dbSNP will release the data during the next build release cycle. See the ClinVar Submission documentation for the asserted clinical variation data hold policy.

A short tag or abbreviation called a “submitter HANDLE” uniquely defines each submitting laboratory and groups the submissions within the database. See dbSNP’s online submission instructions for help preparing a submission.

The 10 major data elements of a submission include:

Alleles

Alleles define each variation class (Table 1). dbSNP defines single nucleotide variants in its submission scheme as G, A, T, or C, and does not permit ambiguous IUPAC codes, such as N, in the allele definition of a variation. In cases where variants occur close to one another, dbSNP permits IUPAC codes such as N, and in the flanking sequence of a variation, actually encourages them. See (Table 1) for the rules that guide dbSNP post-submission processing in assigning allele classes to each variation.

Table 1.

Variation class organizes submissions by allele definition Note: dbSNP has an allele length limitation of <=50bp. Submit alleles >50 nucleotides in length to the Database of Genomic Structural Variation (dbVAR).

Method

Each submitter defines the methods in their submission as either the techniques used to assay variation or the techniques used to estimate allele frequencies. dbSNP groups methods by method class (Table 2) to facilitate queries using general experimental technique as a query field. The submitter provides all other details of the techniques in a free-text description of the method. Submitters can also use the METHOD_EXCEPTION field to describe changes to a general protocol for particular sets of data (batch-specific details). Submitters generally define methods only once in a submission.

Table 2.

Method class organizes submissions by methodological or experimental approach

Population-specific Allele Frequencies

Alleles typically exist at different frequencies in different populations; a very common allele in one population may be quite rare in another population. Also, allelic variants can emerge as private polymorphisms when particular populations have been reproductively isolated from neighboring groups, as is the case with isolated or remote populations.

Frequency data are submitted to dbSNP as allele counts or binned frequency intervals, depending on the precision of the experimental method used to make the measurement. dbSNP contains records of allele frequencies for specific population samples that are defined by each submitter and used in validating submitted variations. See Table 3 for use of allele frequencies in validation.

Table 3.

Validation status codes summarize available validation data

Computed Content

dbSNP releases its content to the public in periodic “builds” that are synchronized with the release of new genome assemblies for each organism (Handbook: Eukaryotic Genome Annotation Pipeline). The dbSNP build process proceeds as follows:

1.

Cluster variations (ss) submitted since the previous build into RefSNPs (rs).

2.

Map the refSNP clusters to the appropriate assembly.

3.

Merge co-locating refSNP clusters when appropriate.

4.

Mark suspected false positive variants (see the “Suspect Variations” section of this chapter for more information on false positive selection).

5.

Compute a functional context for the mapped variants.

6.

Compute minor allele frequency as well as average heterozygosity and standard error.

7.

Compute links to other relevant NCBI resources such as Gene, PubMed, and RefSeq for RefSNP clusters.

8.

Map all clustered variations to RefSeq sequences.

See Figure 1 for a complete graphic description of the dbSNP Build process.

New Submissions and the Start of a New Build

Each build starts with a “close of data” that defines the set of new submissions that will be mapped to genome sequence for subsequent annotation and grouping of variations into refSNPs. The set of new data entering each build typically includes all submissions received since the close of data in the previous build.

Submitted SNPs and Reference SNP Clusters

When a new variation is submitted to dbSNP, it is assigned a unique submitted SNP ID number (ss#). If the variation is submitted with an asserted position, once the ss number is assigned, the asserted position coordinates are remapped to the corresponding coordinates on the current assembly. If the variation is submitted with flanking sequence, dbSNP aligns the flanking sequence of each submitted SNP to its appropriate genomic location(s).

When multiple submissions of the same variation class (Table 1) that have the same weight (uniqueness) map to the same position on the assembly, dbSNP clusters the ss, defines the “reference SNP cluster,” or “refSNP,” and provides the cluster with a unique RefSNP ID number (rs#). If submitted SNPs of more than one variation class map to a single position, then an rs number will be assigned for each variation class at that position. If only one submission maps to a specific position, then its ss is assigned an rs number and is the only member of its RefSNP cluster until another submitted SNP of the same variation class is found that maps to the same position.

A refSNP cluster has a number of summary attributes that are computed over all cluster members (Figure 2), and are used to annotate variations contained in other NCBI resources. See Figures 2A, 2B, 2C and 2D for the location of all summary attributes and internal/external resource links in a refSNP cluster report.

Figure 2.

The refSNP Summary Record (refSNP Cluster Report). The refSNP Summary Record, also known as the refSNP Cluster Report, provides the user with extensive summary attributes that are supplied by the submitter, calculated by dbSNP from submitted data, or (more...)

Figure 2A.

The refSNP Summary Report: Allele Summary and Integrated Maps Sections. The Allele Summary section of the refSNP report provides clinical significance (A), where the phenotype may be viewed by clicking on either the VarView or the OMIM buttons; the allele (more...)

Figure 2B.

The refSNP Summary Report: the GeneView Section. The GenView section has a Display menu at the top. Once menu choices have been made, clicking the “Go” button (A) will generate the GeneView Display. The default setting (“Clinical (more...)

Figure 2C.

The refSNP Summary Report: The GeneView Display. The default GeneView display provides a tabular summary of variations mapped to splice variants of a particular gene. The variation summary is arranged in the sequential order that the variations appear (more...)

Figure 2D.

The refSNP Summary Report: FASTA, Resource Links, Population Diversity and Validation Summary Sections. The FASTA section provides the variation 5’ flanking sequence (A), the allele (B), and the 3’ flanking sequence (C) as provided by (more...)

dbSNP exports the entire dbSNP refSNP set in many report formats to its FTP site, and delivers them as sets of results when a user conducts a dbSNP batch query.

Note: Summary properties derived from asserted data (e.g., clinical assertions, asserted positions, asserted allele origin) are based on experimental evidence and cannot be seen as a confirmation of a particular clinical phenotype, genomic position, or allele origin since NCBI does not independently verify assertions and cannot endorse their accuracy.

Mapping to a Genome Sequence

When a new genome build is ready, dbSNP uses assembly-assembly alignments to remap ss asserted locations and rs locations from the old assembly to the new assembly.

To map submissions without asserted locations to a genome assembly, dbSNP obtains FASTA files for those submitted SNPs submitted prior to the “close of data,” as well as FASTA files for refSNPs in the current build that can’t be remapped, and then maps the submitted SNPs and refSNPs to the genome sequence using the BLAST procedure described in Appendix 2.

If an organism is represented by multiple assemblies then each assembly is annotated. For example, dbSNP annotates two major human assemblies: the Genome Reference Consortium (GRC) Reference assembly, and the haploid hydatidiform mole (CHM1) assembly.

refSNP Clustering and refSNP Orientation

The orientation of a refSNP, and hence its sequence and allele string, is set by the first submitted SNP (ss) used to create a refSNP (rs) cluster. By convention, the cluster exemplar is the member of a refSNP cluster that has the longest flanking sequence or is the first variant with an asserted location assigned in the cluster. If in a later build, a new variant added to the cluster becomes the exemplar and happens to be in reverse orientation to the current orientation of the refSNP, dbSNP preserves the orientation of the refSNP by using the reverse complement of the cluster exemplar to set the orientation of the refSNP sequence.

For those variants that are submitted with an asserted position rather than a flanking sequence, once dbSNP maps and verifies the asserted position, the flanking sequence is derived from the asserted position and used to determine the variant orientation.

Once the clustering process determines the orientation of all member sequences in a cluster, it will gather a comprehensive set of alleles for a refSNP cluster.

Following the dbSNP redesign, dbSNP will keep the use of flanking sequence and exemplars for those organisms that don’t have a mature assembly, but will phase out the “exemplar” concept once creation of asserted positions for those variations that were submitted with flanking sequence has begun.

Note: dbSNP reports all variants and variant alleles on the + strand of the assembly.

Re-Mapping, refSNP Merging and refSNP Splitting

SELECT  *
  FROM [human_9606].[dbo].[RsSplitArch] where rs_2_split = 26
rs_2_split  rs_new_split      split_build_id    create_time last_updated_time
26    78384355    132   2010-08-19 23:38:00     2010-08-19 23:38:00

Suspect Variations

Currently, a variant is flagged as a “suspect”, i.e., a potential false positive when the presence of a paralogous sequence in the genome (1,2) could cause mapping artifacts or if there is evidence suggesting sequencing errors or computation artifacts.

dbSNP will be updating its suspect false positive flagging system in the near future to rank suspect variants according to the amount of supporting evidence available for each refSNP. Those refSNP clusters that are suspect, but have data from multiple submitters indicating a heterozygous state exists, will rank more highly in dbSNP’s new system as a variation to be trusted than a suspect refSNP that has data from a single submission, has multiple submissions that show no evidence of heterozygosity, or with conflicting evidence of heterozygosity.

Molecular Class

dbSNP computes a molecular context for sequence variations by inspecting the flanking sequence for gene features during the contig annotation process, and does the same for RefSeq/GenBank mRNAs.

dbSNP has adopted Sequence Ontology (SO) terms to define its variation molecular classes so as to conform to the standard set by the biological community. The subset of SO terms dbSNP uses as functional classes can be found in Table 4.

Table 4.

Molecular codes for refSNPs in gene features

A variation may have multiple functional classes. Multiplicity will result, for example, when a variation falls within an exon of one transcript and an intron of another for the same gene.

Clinical Assertions

Novel variations and experimental evidence supporting clinical assertions (Table 5) are submitted to ClinVar. dbSNP and ClinVar will continue to support the Human Variation Batch Submission site as a Web-based tool that can be used to submit or update medically important variation submissions.

Table 5.

Clinical Significance organizes submissions by clinical assertion type

When novel variants with supporting clinical evidence are received from ClinVar, dbSNP remaps the asserted positions of the variants to the corresponding coordinates on the current assembly, as well as to cDNA, protein, and RefSeqGene sequences. Once the variants are mapped, dbSNP assigns ssIDs and rsIDs to each.

Data submitted through the Human Variation Batch Submission site that supports a clinical assertion are processed and extracted by ClinVar, which uses the data to assign clinical attributes to novel variants or to update the clinical attributes of existing variants (LSDB).

Once the submitted variants are mapped and their attributes assigned, these data are made available to other NCBI resources, including VarView, ClinVar, and Variation Reporter.

Population Diversity Data

Build Integration

dbSNP annotates the non-redundant set of variations (refSNP cluster set) on reference genome genomic sequences, chromosomes, mRNAs, and proteins as part of the NCBI RefSeq project. dbSNP computes summary properties for each refSNP cluster, which are then used to build fresh indexes for dbSNP in the Entrez databases, and to update the variation map in the NCBI Map Viewer. Finally, dbSNP updates links between dbSNP and BioProject, dbVar, dbGaP, Gene, HomoloGene, Nucleotide, OMIM, Protein, PubChem Substance, PubMed, PubMed Central, VarView, and Variation Reporter.

Public Release

Public release of a new build involves an update to the public database and the production of a new set of files on the dbSNP FTP site. dbSNP makes an announcement to the dbsnp-announce mailing list when the new build for an organism is publicly available.

The dbSNP Redesign: Changes to Clustering

As of this writing, dbSNP is planning a redesign that will introduce a number of fundamental changes to dbSNP’s dataflow. Users are encouraged to review the proposed changes below and to submit any comments and suggestions to vog.hin.mln.ibcn@nimda-pns. Among these changes is a new clustering algorithm.

While improvements continue to be made in genome assemblies, artificial sequence duplications will resolve and collapse, artificially collapsed regions of a genome will expand, and missing sequence regions will be added. In the case of a sequence collapse, the number of hits that a variation has on the assembly may be reduced, while in the case of a sequence expansion or addition, the number of hits that a variation has to the genome may increase.

In order to have the flexibility necessary to deal with these genome assembly changes, dbSNP 2.0 will have a new clustering algorithm that will change the concept of a refSNP as we know it. This new clustering algorithm will change the rules that govern clustering to invert the ss to rs relationship as it currently exists.

At present, multiple ss numbers can exist in the same rs cluster and each ss number in that cluster links to its one corresponding rs number. The new clustering rules, however, will change this relationship in that each rs will represent a unique location. Under these new rules, a single ss number could link to multiple rs numbers if the ss number maps to more than one location.

dbSNP Home Page

dbSNP is a part of the Entrez integrated information retrieval system and may be searched either by using an ID number query, or by using combinations of different search fields and qualifiers.

Variation Reporter

Variation Reporter matches submitted variation calls to variants housed in dbSNP or dbVar, thereby allowing access via a Web search or through an application programming interface (API) to all data and metadata that dbSNP has for the matching variants. If you submit novel variants and there are no matches between your data and the variants housed in dbSNP or dbVar, the Variation Reporter will provide the predicted consequence of each submitted variant.

BLAST

BLAST can be used to match submitted variations with asserted positions to matching dbSNP records (See the instructions at: ftp://ftp.ncbi.nih.gov/pub/factsheets/HowTo_Finding_SNP_by_BLAST.pdf. Query BLAST with the sequence or clone that contains the asserted position of the variant, and then select an appropriate reference database as the BLAST target. The BLAST algorithm will find any existing SNP records that map to the queried sequence, and thence to the variant of interest, if a dbSNP record happens to match it.

Note: If BLAST fails to find a matching dbSNP record for a variation of interest on a queried sequence:

1.

You cannot assume that the variant is novel without further study, because there are several reasons why an existing variant may not yet have a dbSNP record:

a.

The sequence location of the existing variant may be missing from the reference assembly, or the transcript location of the variant has not yet been sequenced.

b.

The existing variant may have been submitted with low sequence quality or ambiguous base calls, which would inhibit placement on the reference assembly.

c.

The variant may exist in the literature, and has not yet been submitted by the author for inclusion in dbSNP. This is particularly true for those variants that were reported in historic literature.

2.

You can use Variation Reporter to get a predicted consequence of human variations to help you in your analysis if the variations have known sequence locations.

SNP Submission Information Queries

If refSNP(rs) or submitted SNP (ss) numbers are not available to use in a search for a dbSNP record, use the “Submission Information” module to construct a query that will select dbSNP variation records based on other available information associated with a submitted variation:

• Information associated with the submitter
• Information about the submitted batch that contains the variation of interest
• Information associated with the method used to assay for the variation (Table 2)
• Information associated with the submitted population
• Information associated with the publication reporting the variant

Search via ClinVar, Gene, or PubMed

There are multiple databases in NCBI that maintain links to dbSNP. Related records in dbSNP can be identified by following the Find related data on the Summary display, or following the links in the Related information section of a single record.

Entrez Programming Utilities (Eutils)

Use Entrez Programming Utilities (E-utilities or Eutils) to query dbSNP and retrieve information via Web services. You can test an Entrez query interactively and then execute that query using Eutils. There are a number of available Eutil programs that cover a wide range of query types. See the Entrez Programming Utilities help documentation for more information.

dbSNP FTP Site

NCBI supports the public distribution of dbSNP data by providing compressed data dumps in a number of different formats. Access to the NCBI FTP site is available via the World Wide Web (ftp://ftp.ncbi.nih.gov/snp/) or anonymous FTP (host ftp.ncbi.nih.gov cd snp). In addition to the data formats described on the FTP README file, which include ASN.1, FASTA, and XML, dbSNP FTP offers two additional formats:

ADA Section 508-Compliance Link

All links provided on the dbSNP homepage are also provided in text format at the bottom of the page to support browsing by text-based Web browsers. Suggestions for improving database access by disabled persons should be sent to the dbSNP development group at snp-admin@ncbi.nlm.nih.gov.

Local Copies of dbSNP

If you wish to create a SQL copy of dbSNP on a local server for direct access, use the directions in Appendix 3 of this chapter to create the tables and indices for dbSNP from the dbSNP schema, data, and SQL statements.

Note: We will be phasing out the relational database architecture of dbSNP during the dbSNP redesign, and are considering replacing it with Service Oriented Architecture (SOA) and a BLOB/CLOB store system in dbSNP 2.0. Storage technology and object schemas, however, are still under design. Since dbSNP 2.0 may not be an SQL based system, we will provide users with an API to access bulk dumps of data for those wanting to create a local copy of dbSNP. Check or subscribe to the dbSNP News and Announcements site for updates regarding the redesign and availability of the data as relational tables or as objects.

Converting a Location on One Assembly or Sequence to Another

NCBI's Genome Remapping Service (Remap) allows you to convert locations from one sequence to another based on alignments. Use Remap if you have identified the location of variation on an assembly, or on a RefSeqGene/LRG, and want to determine the location on a different assembly (or on the genome in the case of the RefSeqGene).

History of Interpretation of the Medical Importance of an Allele

ClinVar archives the relationship reported between variations and phenotype by accessioning and versioning submissions.

Association Studies

dbGaP archives and distributes data from studies that examine the relationship between phenotype and genotype. Such studies include Genomewide Association Studies (GWAS), medical sequencing, and molecular diagnostic assays. Links are available from dbGaP controlled access records to related variation data in dbSNP, but there are no reciprocal links from dbSNP records to dbGaP unless the aggregate data are public. The refSNP report “Association” section will link to association results from the NHGRI GWAS Catalog and/or PheGenI when association data are available.

Histocompatibity

dbMHC provides a platform where users can access, submit, and edit data related to the human Major Histocompatibility Complex, also called the HLA (Human Leukocyte Antigen).

Both dbMHC and dbSNP store the underlying variation data that define specific HLA alleles. dbMHC provides access to related dbSNP records at the haplotype and variation level, whereas dbSNP provides access to related dbMHC records at the haplotype level.

Haplotypes

The International HapMap Project site allows access to its catalog of statistically related variations, also known as haplotypes, for a number of different human populations, and is a useful resource for those researchers looking for variations associated with a particular gene. HapMap haplotypes can be searched by a landmark such as a refSNP number or gene symbol, as well as by sequence region or chromosome region. The resulting HapMap report includes an ideogram with various tracks that can be altered to provide required data, and appropriate tracks in the report will provide direct links to refSNP cluster records.

Variation as Related to Citations, Genes, Phenotypes, and other NCBI Databases

Multiple databases in NCBI can be used to identify variation that meets certain criteria. They may either reference rs numbers explicitly, or provide links from their records to records in dbSNP.

Variation Batch Submission (VarBatch)

VarBatch is an online submission resource for both clinical and non-clinical human variations, and allows the update and annotation of previously submitted variations. When an asserted clinical variation is processed through VarBatch, it is assigned both a dbSNP submitted SNP (ss) accession as well as a ClinVar accession (format: SCV000000000.0), since the ClinVar accession represents the asserted variation/phenotype relationship.

Note: Since VarBatch does not accept frequency, genotype, or population data, submit these data to dbSNP as updates to your VarBatch submission using the dbSNP VCF or Flat File format via email or through a pre-arranged FTP upload once ss numbers are assigned to your submitted variations.

Variation Reporter

Variation Reporter matches submitted variation call data to variants housed in dbSNP or dbVAR, allowing access to all data and metadata that dbSNP has for any known matching variants. If you submit novel variants to the Variation Reporter, and there are no matches between your data variants housed in dbSNP or dbVAR, the Variation Reporter will provide the predicted consequence of each submitted variant.

VarView

VarView reports display detailed variation information associated with a particular gene and are created only for those genes that have asserted clinical variations. VarView can be accessed in two ways:

1.

Through Gene by using the query “gene_snp_clin[filter]” to identify gene records that have a VarView report.

2.

Through dbSNP either by using the “VarView” link displayed in refSNP reports for variations that have asserted clinical significance, or by querying dbSNP using “snp_gene_clin[filter]” to identify variants that have a VarView report.

Once a Gene or dbSNP record has been selected, and the VarView link on the record has been activated, a VarView report will appear that includes:

• A brief description the gene
• A list of all observed rs variants of the gene
• Links to both internal and external resources including locus specific databases (LSDB), OMIM, Gene, and PubMed.

When one of the listed rs variations in the VarView report is selected, the “submission details” section of the report provides a list of ss numbers associated with the selected rs number as well as links to submitter sites and each ss report.

Note: VarView will be replaced by a new Variation Gene Viewer in April 2014. This new resource will allow users to access all of NCBI’s variation data (i.e., dbSNP, dbVar, ClinVar) in a gene-centric fashion.

1000 Genomes Browser

The 1000 Genomes Browser provides access to 1000 Genomes data including variations, genotypes, and sequence read alignments within the context of GRCh37, the reference assembly used by the 1000 Genomes Project for analysis. The browser allows you to configure the display to include multiple data tracks of interest and provides links to related data housed in various NCBI resources. The 1000 Genomes Browser allows users to quickly view alignments supporting a particular variant call and can be used to download and read variant data for small genomic regions of interest.

Access the 1000 Genomes Browser from dbSNP using the 1000 Genomes Browser link in the refSNP report “Integrated Maps” section.

Access dbSNP from the 1000 Genomes Browser using the “hover” feature in either the “Clinical Channel” or “Cited Variant” tracks. Click on the variation rsID that appears.

Appendix 1. dbSNP Report Formats

Appendix 2. Rules and Methodology for Mapping

The appearance of FASTA-formatted genome sequence for a new build of an assembly or the significant accrual of newly submitted SNP data for an organism will initiate a cycle of MegaBLAST and BLAST alignment of the variations to the NCBI genome assembly of the organism.

Appendix 3. How to Create a Local Copy of dbSNP

How to Create a Local Copy of dbSNP

Curently, dbSNP is a relational database that contains hundreds of tables. Since the inception of build 125, the design dbSNP has been altered to a ”hub and spoke” model, where the dbSNP_Main_Table acts as the hub of a wheel, storing all of the central tables of the database, while each spoke of the wheel is an organism-specific database that contains the latest data for a specific organism. dbSNP exports the full contents of the database for the public to download from the dbSNP FTP site. During the dbSNP redesign, however, we will be phasing out the relational database architecture of dbSNP, and are considering replacing it with Service Oriented Architecture (SOA) and a BLOB/CLOB store system in dbSNP 2.0.

Due to security concerns and vendor endorsement issues, dbSNP cannot provide users with direct dumps of dbSNP. The task of creating a local copy of dbSNP can be complicated and should be left to an experienced programmer. The following sections will guide you in the process of creating a local copy of dbSNP, but these instructions assume knowledge of relational databases, and were not written with the novice in mind.

If you have problems establishing a local copy of dbSNP, please contact dbSNP at vog.hin.mln.ibcn@nimda-pns.

Schema: The dbSNP Physical Model

A schema is a necessary part of constructing your own copy of dbSNP because it is a visual representation of dbSNP that shows the logical relationship between the data. It is available as a printable PDF file from the dbSNP FTP site.

Data in dbSNP are organized into “subject areas” depending on the nature of the data. The data dictionary includes a description of the tables in dbSNP as well as tables of columns and their properties. Foreign keys are not enforced in the physical model because they make it harder to load table data asynchronously. In the future, dbSNP will add descriptions of individual columns. The data dictionary is also available online from the dbSNP website.

Resources Required for Creating a Local Copy of dbSNP

Software:

• Relational database software. If you are planning to create a local copy of dbSNP, you must first have a relational database server, such as Sybase, Microsoft SQL server, or Oracle. dbSNP at NCBI runs on an MSSQL server version 2000, but there are users who have successfully created their local copy of dbSNP on Oracle.
• Data loading tool. Loading data from the dbSNP FTP site into a database requires a bulk data-loading tool, which usually comes with a database installation. An example of such a tool is the bcp (bulk-copy) utility that comes with Sybase, or the “bulkinsert” command in the MSSQL server.
• winzip/gzip to decompress FTP files. Complete instructions on how to uncompress *.gz and *.Z files can be found on the dbSNP FTP site.

Hardware:

• Computer platforms/OS

Databases can be maintained on any PC, Mac, or UNIX with an internet connection.

• Disk space

To ascertain the disk space needed for a complete copy of dbSNP for a particular organism, determine the total download file size for the organism as a starting point. You need a minimum of three times of the data file size to have space for creating indices and storing your own working tables. The allocated size of dbSNP human B137 on dbSNP’s internal server is 3TB, while mouse B137 size is about 700GB.

• Memory

The minimum amount of memory required is approximately 4GB.

• Internet connection

dbSNP recommends a high-speed connection to download such large database files.

dbSNP Data Location

The FTP database directory in the dbSNP FTP site contains the schema, data, and SQL statements to create the tables and indices for dbSNP:

• The shared_schema subdirectory contains the schema DDL (SQL Data Definition Language) for the dbSNP_main_table.
• The shared_data subdirectory contains data housed in the dbSNP_main_table that is shared by all organisms.
• The organism_schema subdirectory contains links to the schema DDL for each organism specific database.
• The organism_data subdirectory contains links to the data housed in each organism specific database. The data organized in tables, where there is one file per table. The file name convention is: <tablename>.bcp.gz. The file name convention for the mapping table also includes the dbSNP build ID number and the NCBI genome build ID number. For example, B125_SNPContigLoc_35_1 means that during dbSNP build 125, this SNPContigLoc table has SNPs mapped to NCBI contig build 35 version 1. The data files have one line per table row. Fields of data within each file are tab delimited.

dbSNP uses standard SQL DDL(Data Definition Language) to create tables, views for those tables, and indexes. There are many utilities available to generate table/index creation statements from a database.

Stepwise Procedure for Creating a Local Copy of dbSNP

1.

Prepare the local area

(check available space, etc.)

2.

Download the schema files

a.

Download the following files from the dbSNP shared​_schema subdirectory: dbSNP_main_table, dbSNP_main_index_constraint, and all the files in the shared​_data subdirectory. Together, the files from both of these subdirectories will allow you to create tables and indices for the dbSNP_main_table.

b.

Go to the organism​_schema subdirectory and select the organism for which you wish to create a database. For the purpose of this example, human_9606 has been selected. Once human_9606 is selected, you will be directed to the human organism_schema subdirectory. Download all of the files contained in this subdirectory.

c.

Go to the organism​_data subdirectory, and select the organism for which you wish to create a database. For the purpose of this example, human_9606 has been selected. Once you select human_9606, you will be directed to the human organism_data subdirectory. Download all of the files contained in this subdirectory.

A user must always down load the files located in the most recent versions of the shared_schema and shared_data subdirectories in addition to any organism specific content.

Save all the files in your local directory and decompress them.

View in own window

Hint:

On a UNIX operating system, use gunzip to decompress the files: dbSNP_main_table and dbSNP_main_index_constraint. The files on the SNP FTP site are UNIX files. UNIX, MS-DOS, and Macintosh text files use different characters to indicate a new line. Load the appropriate new line conversion program for your system before using bcp.

3.

Create the dbSNP_main_table

a.

From the shared​_schema subdirectory, use the dbSNP_main_table file to create tables, and use the dbSNP_main_index_constraint files to create indices for the dbSNP main database.

b.

Load all of the bcp files located in the shared​_data subdirectory into the dbSNP_main_table you just created using the data-loading tool of your database server (e.g., bcp for Sybase). See the sample FTP protocol and sample Unix C Shell script (below) for directions.

c.

Create indices by opening the dbSNP_main_index_constraint.sql file. If you are using a database server that provides the isql utility, then use the following command:

isql -S <servername> -U usename -P password -i dbSNP_main_index_constraint.sql

View in own window

Hint:

The “.bcp” files in the shared_data and organism_data subdirectories may be loaded into most spreadsheet programs by setting the field delimiter character to “tab”.

4.

Create the organism specific database
Once the dbSNP_main_table has been created, create the organism specific database using the files in your specific organism’s organism_schema and organism_data subdirectories. Human_9606 will be used for the purpose of this example:

a.

Create the human_9606 database using the following files found in the human_9606 organism​_schema: human_9606_table.sql.gz, human_9606_view.sql.gz, human_9606_index_constraint.sql.gz,
and human_9606_foreign_key.sql.gz

b.

Load all of the bcp files located in the shared​_data subdirectory into the human_9606 database you just created using the data-loading tool of your database server (e.g., bcp for Sybase). See the sample FTP protocol and sample Unix C shell script (below) for directions.

View in own window

Hint:

Use “ftp -i” to turn off interactive prompting during multiple file transfers to avoid having to hit “yes” to confirm transfer hundreds of times.

View in own window

Hint:

To avoid an overflow of your transaction log while using the bcp command option (available in Sybase and SQL servers), select the "batch mode" by using the command option: -b number of rows. For example, the command option -b 10000 will cause a commit to the table every 10,000 rows.

5.

Sample FTP Loading protocol

a.

Type ftp -i ftp.ncbi.nih.gov (Use "anonymous" as user name and your email as your password).

b.

Type: cd snp/database

c.

To get dbSNP_main for shared tables and shared data: Type ls to see if you are in the directory with the right files. Then type “cd shared_schema” to get schema file for dbSNP_main, and finally, type “cd shared_data” to get the data for dbSNP_main.

d.

Type binary (to set binary transfer mode).

e.

Type mget *.gz (to initiate transfer). Depending on the speed of the connection, this may take hours since the total transfer size is gigabytes in size and growing.

f.

To decompress the *.gz files, type gunzip *.gz. (Currently, the total size of the uncompressed bcp files is over 10 GB).

6.

Use scripts to automate data loading.

a.

Located in the loadscript subdirectory of the dbSNP FTP site, there is a file called cmd.create_local_dbSNP.txt that provides a sample UNIX C shell script for creating a local copy of dbSNP_main and a local copy of a specific organism database using files in the shared_schema, and the organism_schema subdirectories.

b.

Also in the the loadscript subdirectory of the dbSNP FTP site, there is a file called cmd.bulkinsert.txt that provides a sample UNIX C shell script for loading tables with files located in shared_data and organism_data subdirectories.

7.

Data integrity (creating a partial local copy of dbSNP)

dbSNP is a relational database. Each table has either a unique index or a primary key. Foreign keys are not reinforced. There are advantages and a disadvantage to this approach. The advantages are that this approach makes it easy to drop and recreate the table using the dbSNP_main_table, which then makes it possible to create a partial local copy of dbSNP. For example, if you are interested only in the original submitted SNP and their population frequencies, and not in their map locations on NCBI genome contigs or GenBank Accession numbers (both are huge tables), then these tables can be skipped (i.e., SNPContigLoc and MapLink). Please remember that mapping tables such as SNPContigLoc will have a build ID prefix and suffix included in its file name. (e.g., SNPContigLoc will be b125_SNPContigLoc_35_1for SNP build 125, and NCBI contig build 35 version 1). Of course, to select tables for a particular query, the contents of each table and the dbSNP entity relationship (ER) diagram need to be understood. The disadvantage of unreinforced references is that either the stored procedures or the external code needs to be written to ensure the referential integrity.