MUMmer 游騰楷杜海倫王慧芬曾俊雄 2007/01/02. Outlines Suffix Tree MUMmer 1.0 MUMmer 2.1 MUMmer 3.0 Conclusion

MUMmer

游騰楷杜海倫王慧芬曾俊雄

2007/01/02

Outlines

• Suffix Tree• MUMmer 1.0• MUMmer 2.1• MUMmer 3.0• Conclusion

Suffix Tree

游騰楷

Trie

• A Trie, also called a prefix tree, is a tree that records the information of strings.

• It likes a dictionary of those strings.

• Advantages comparing to BST: – Search time.

– Space.

– Prefix matching.

– Balance.

Suffix Trie and Suffix Tree

• A suffix tree, Tree(T), is a compact trie that represents all the suffixes of a string T.

Suffix Trie and Suffix Tree(cont.)

a

b

b

aaa

aa

b

b

b

a

baab

baab

ab

abaab

baab

aab

ab

b

Suffix Trie Suffix Tree

Suffix Trie and Suffix Tree(cont.)

• Let |T| = n• Suffix trie needs O(n2) space.

– Consider T = an/2bn/2

• Suffix tree needs O(n) space.– Since every symbol can cause only one branch, total no

des (edges) cannot exceed O(n).

– In a node we only record the starting position of corresponding suffix.

– In a edge we need not to record the whole substring but the starting and ending positions of the substring.

1,1

2,54,5 2,5

Suffix Trie and Suffix Tree(cont.)1:abaab

2:baab

3:aab

4:ab

5:b

Suffix Tree

1:abaab$

3:aab$

2:baab$

5:b$

2,2

3,5

a

baab

baab

ab

Suffix Tree

3,5 4:ab$

2,2

Suffix Tree: Full Text Index

• P occurs in T

P is a prefix of some suffix of T

Path for P exists in Tree(T)

Linear Time Construction

• 1973, Weiner gave the first linear time algorithm. • 1976, McCreight gave a more readable algorithm.• They were all processing from right to left.• In 1992, Ukkonen gave a left-to-right on-line algo

rithm.

On-line Construction of Suffix Trie

• We have O(n2) time to do this, it is quite easy.

• We construct Triei from Triei-1.

• We need to record the current end points of every suffix.

a

a

b

b

Trie(abaaba)

a

b

b

aa

a

b

b

aaa

aa

a

b

b

aaa

aa

b

b

b

a

b

b

aaa

aa

b

b

ba

a

a

Two Lemmas

• We call the current end points of suffix i as Ci.

1. If some Ci ever branch out it will never branch again.

2. If Ci does not branch, those Cj also does not branch for j > i.

Ideas to Find Suffix Tree

• Thus – If one branched we never consider it again.

– The branches have an order from small indices suffix to large indices.

• For a suffix we have 3 phases:1. Going along original tree.

2. Branching a new leaf.

3. Growing the leaf.

Ideas to Find Suffix Tree(cont.)

• 3 phases:1. Going along original tree.

2. Branching a new leaf.

3. Growing the leaf.

• We just record the longest suffix that is not branch yet.

• When it branches, we try to find next suffix.• To do this, we need suffix links.

Suffix Links

• Suffix link is a pointer from an internal node xS to another internal node S. x , and S = *.

• Suffix tree of “abaabab$”:

a b

aabab

ababb

ababb

ba

abaabab$baabab$

aabab$

abab$

bab$

ab$ b$

$

Using Suffix Link to Find Common Substring(1)

a b

a

abab

ababb

ababb

ba

abaabab$

baabab$

aabab$

abab$

bab$

ab$ b$

$

Consider abaaaba

Using Suffix Link to Find Common Substring (2)

a b

a

abab

ababb

ababb

ba

abaabab$

baabab$

aabab$

abab$

bab$

ab$ b$

$

abaaabaabaa


a b

a

abab

ababb

ababb

ba

abaabab$

baabab$

aabab$

abab$

bab$

ab$ b$

$

abaaababaa


a b

a

abab

ababb

ababb

ba

abaabab$

baabab$

aabab$

abab$

bab$

ab$ b$

$

abaaabaaa


a b

a

abab

ababb

ababb

ba

abaabab$

baabab$

aabab$

abab$

bab$

ab$ b$

$

abaaaba


a b

a

abab

ababb

ababb

ba

abaabab$

baabab$

aabab$

abab$

bab$

ab$ b$

$

abaaabaaaba


a b

a

abab

ababb

ababb

ba

abaabab$

baabab$

aabab$

abab$

bab$

ab$ b$

$

abaaaba ababaa

Construct Suffix Tree

• Like matching, we use suffix link to find next suffix.

Suffix Tree(abaabab$)

1,- 2,-

1 step

aba

1,- 2,-

ab

1,-

a

1,12,-

abaa

2,-4,-

1,12,-

abaab

2,- 4,-

1 step

1,12,-

abaaba

2,-4,-

2 step

abaabab

1,12,-

2,-4,-

Branch

abab

1,12,-

2,3 4,-

4,- 7,- Use suffix link to find this, and go down by “bab”

bab

1,12,-

2,3 4,-

4,- 7,- Nearest ancestor internal node

abab

1,12,3

2,3 4,-

4,- 7,-4,-

7,-Nearest ancestor internal node

bab

1,12,3

2,3 4,-

4,- 7,-4,-

7,-

ab

1 step

Suffix Tree(abaabab$)

1,12,2

3,34,-

4,- 7,-4,-

7,-

2,2

8,-3,38,-

abaabab$baabab$

aabab$

abab$

bab$

ab$ b$

8,- $

Time Complexity

• For phase 3, we do nothing.• For phase 1, we totally use O(n) time.• For phase 2, we do O(n) times of branch. The only

bottleneck is how fast can we find next position.

Time Complexity(cont.)

• Consider we branch at the red circle having distance t to its parent.

• Next suffix can at most pass through t internal nodes.

• Use amortized analysis, we can easily find total internal nodes the algorithm passing through is O(n).

t steps

Time Complexity(cont.)

• Thus, constructing suffix tree needs O(n) time and O(n) space.

Applications

• Longest common substring.• Repeating of a pattern in a string.• Approximate matching.• etc.

Suffix Arrayhattivatti

attivatti

ttivatti

tivatti

ivatti

vatti

atti

tti

ti

i

ε

ε

atti

attivatti

hattivatti

i

ivatti

ti

tivatti

tti

ttivatti

vatti

11

7

2

1

10

5

9

4

8

3

6Suffix array of hattivatti: (11, 7, 2, 1, 10, 5, 9, 4, 8, 3, 6)

att binary search

It can construct from suffix tree in linear time. (DFS)

Special Thanks to

• Esko Ukkonen• Hsueh-I Lu

• Wikipedia

• Some figures of this slides are based on their slides.


MUMmer 1.0 –Alignment of whole genomes

D95922019 杜海倫2007/01/02

About The System - MUMmer

• For rapidly aligning whole genome sequences– Assumption: the sequences are closely related

– Output:• Alignment of the input sequences

• Highlighting the exact differences in the genomes

– SNPs, insertions, significant repeats, tandem repeats, reversals

– Main idea• Suffix tree

• Longest increasing subsequence (LIS)

• Smith-Waterman alignment

Alignment step

• Perform a maximal unique match (MUM) decomposition of the two genomes -> Suffix tree

• Sort the MUMs, and extract the longest possible set of matches in the same order -> LIS

• Close gaps -> Smith-Waterman alignment

• Output!

Step 1• Suffix tree

8 3

Step 1 (cont’)Construct a suffix tree T for genome A

Add the suffixes for genome B(implement: A+dummy character+B)

Find out unique matching sequence: an internal node with exactly two child nodes,

such that the child nodes are leaf nodes from different genomes

Find out MUM: For highly similar genomes, set MUM>=50bp

For more distantly related genomes, set MUM>=20bp

Step 2

• Sort, LIS=> O(KlogK) => O(N)– K: the numbers of MUMs

– K<<N/logN

Step 3• Process the gap into one of the four classes– SNP

• Genome A: cgtcataaagt

• Genome B: cgtcctaaagt

– Insert• Genome A: cgtctaaagtggggaaaactctgg

• Genome B: cgtctaaagt. . . . . . . . Ctctgg

• Transposition or simple insertions

– Polymorphic regions• Genome A: cgtctaaagtggggaaaactctgg

• Genome B: cgtctaaagta tgacaggctctgg

• Should be aligned

– Repeat• Genome A: aaggaaggaaggagct

• Genome B: aaggaagg. . . . agct

Result and Discussion

• Comparing two strain of tuberculosis– H37Rv and CDC1551– >99% identical– Be able to catalog

• all SNPs• all insertions of every length• All tandem repeat with different copy numbers

– Performance (DEC Alpha 4100)• 5s for step1• 45s for step 2• 5s for step 3

Result and Discussion (cont’)


• Comparing two Mycoplasma genome– M.genitalium (580074nt) and M.pneumoniae (226000n

t)

– Performance (DEC Alpha 4100)• 6.5s for step1

• 0.02s for step 2

• 116s for step 3

– FASTA: many hours


FASTA

25mers

MUMmer


• Comparing human and mouse– 222930bp of human chromosome 12 (accession no.

U47924) and 227538bp of mouse chromosome 6 (accession no. AC002397)

– Performance -29s• 1.6s for step1



MUMmer 2.1

王慧芬

MUMmer 2.1• Fast algorithms for large-scale genome alignment and

comparison• by Delcher, Phillippy, Carlton and Salzberg• Nucleic Acids Research 2002• http://www.tigr.org/software/mummer/MUMmer2.pdf

Agenda

• Key in MUMmer 1• Improvements in MUMmer 2.1• Technical improvements in MUMmer 2.1• Application to DNA sequence alignment

– Alignment of incomplete genomes

MUMmer 1• Key

– Built a suffix tree containing 2 input sequences

– Find all maximal unique matches (MUMs) between them.

• MUM (Maximal Unique Matches)– A subsequence occurred in 2 exactly matching copies, once i

n each input sequence

– Cannot be extended in either direction

Improvements in MUMmer 2.1• Fast and less memory, by a factor of nearly three

• Able to align DNA or protein sequence

MUM1 MUM2

Time 74s (1GHz) 27s (1GHz)

Mem 293MB 100MB

To align 4.7 Mb genome of E. coli and

3.0Mb large chromosome of V.cholerae

Technical improvements in MUMmer 2.1

• A reduction in amount of memory used to store suffix trees– Kurtz (1999) technique is used

• An alternative algorithm to find initial exact matches• Cluster matches

Alternative to find initial exact matches

• MUMmer 1:– Built a suffix tree containing 2 input sequences

• MUMmer 2:– Chang-Lawler (1994) method is used

• running time is reduced

– Built a suffix tree storing only one sequence (reference)

– 2nd sequence (query) streamed against the suffix tree • memory usage is reduced by at least half

• once the suffix tree is built, arbitrarily long/multiple queries can be streamed.

Alternative to find initial exact matches (cont.)

• MUMmer 2 (cont.):– Identify where the query sequence would branch off from the tree, to find

all matches

– Unique match • Wherever a branch occurs at a tree position with just a single leaf beneath it

– Maximal match• Using suffix links to find next match (extended match)

• By checking the character immediately preceding the start of this match, we can determine whether it is a maximal match

• To find all maximal matches, it is in time proportional to the length of the query

Unique match

Maximal match

• suffix links is used to find extended match

Cluster matches• MUMmer 1:

– Align 2 complete sequences, no rearrangement.

– That is, to find a single longest alignment.

• MUMmer 2:– After matches are identified, the interval length between match

es are checked.

– If the interval length between matches is less than a user-defined gap length, the matches are joined into a cluster.

Alignment of incomplete genomes

• DNA sequencing• Shotgun sequencing • Terms • Finishing• NUCmer (NUCleotide MUMmer)

DNA sequencing

– Human genomes are approx. 3 billion bases.

– Sequencing machine can generate sequences for fragments in 500-600 bp long.

– In order to read DNA, genome is broken up into tiny of pieces (reads), each is read individually.

– After all pieces are read, they are assembled in the correct order.

Shotgun sequencing

• Extract DNA

• Fragment DNA

• Clone DNA

• Sequence both ends of clones– 500-600 bp each read

• Assemble– reads are assembled to reconstruct the genome

• Finish sequencing (close gaps)

Terms• Raw sequence

– Unassembled sequence reads

• Contig– Overlapping reads are joined into longer composite sequences, called contigs

• Finished sequence– Complete sequence of a genome with no gaps and an accuracy of >99.9%

• Full shotgun coverage– Genome coverage in random raw sequence required to produce finished sequ

ence, 8-10 fold (8-10X)

Finishing

• The process of– Determining the order and orientation of all the contigs, and – Generating additional sequence to fill in all the gaps between them

(closing all the gaps)

• Finishing phase is dispensed in many projects.• However, MUMmer 2.1 is used as a base to build a progra

m for the “finishing phase”– Align the multiple contigs to a completed reference genome, and – Align one set of contigs to another set of contigs

NUCmer (NUCleotide MUMmer)

• Built based on MUMmer 2.1 to develop a miltiple-contig alignment program

• 3 steps• Outputs

NUCmer – 3 steps• Step 1

– Input: 2 DNA sequences in 2 multi-fasta files representing partial or complete assemblies.

– Each DNA sequence is represented as contig sequence– Creates a map of all contig positions within each of the multi-fasta files– Concatenates the two files seperately – Runs MUMmer to find all exact matches between two genomes– These matches are mapped back to the separate contigs

NUCmer – 3 steps (cont.)

• Step 2 ： Clustering– MUMs (output of step 1) are clustered together if they are separate

d in user-defined distance

• Step 3– Run a modified Smith-Waterman DP alignment to align the seque

nce between MUMs (output of step 2)

• Result– Alignment of “every sequence contig in the 1st file” to “every seq

uence contig in the 2nd file”– Order, orientation, and coverage identity percentage of contigs


Improvements in MUMmer 3.0

d95922026 曾俊雄

Overview

• Functionality vs. Modularity

What’s New?

• Optimized suffix-tree library (rewrite)• Non-unique maximal matches (New!)• Distant matches (2.1)

Optimized Suffix-Tree

• The most significant improvement.– multi-contig query against multi-contig reference (conti

nue)

– rewrite, more compact (continue)

MUMmer 3.0, page 4

• multi-contig query against multi-contig reference– already in MUMmer 2.1 through Nucmer package

– now imported into the core

• more compact suffix-tree

• Some previous work:– Manber & Myers : 18.8n~22.4n bytes (DNA)

– Kärkkäinen : 15n~18n (?)

– Crochemore and Vérin : 32.7n

– The strmat software package by Knight, Gusfield and Stoye : 24n~28n (string length at most 2^23)

• Basic Idea:– Use different data structure for different kinds of tree

nodes

– duplicated information should be removed

• Terms:– depth

– head position

abab

i

w

a

b the longest w, the smallest i

• Implementation by Stefan Kurtz :– n+5q (6n) integers !!

• for leaf node, depth and head position and some others are not required

– (3+1/16)n integers !!• more observation

a

aW

if the head position of aW is known, the head position of W will satisfy the constraint:

aW.headposition + 1 >= W.headposition

• case 1: =

• case 2: >

aW

aWaW

aW

W W

for case 1, the internal node representing aW doesn’t even need the head position information

Non-unique Maximal Matches

• 1.0 : matches must be unique– may miss

• 2.0 : uniqueness required in reference sequence– still may miss

• now, a command line to generate all maximal matches, regardless of uniqueness– in cost of very large output file

Distant Matches

• 1.0 : only 100% match is allowed– less sensitivity

• 2.1 : distant (not 100%) match is allowed– through extension packages

• 3.0 : improve


Conclusion

• MUMmer is good at alignment between closely related species

• Distant matches are considered as MUMmer improved

• The open source License may have some advantage

• The space problem is still there

Documents

MUMmer 游騰楷杜海倫 王慧芬曾俊雄 2007/01/02. Outlines Suffix Tree MUMmer 1.0 MUMmer 2.1 MUMmer 3.0 Conclusion

MUMmer 游騰楷杜海倫王慧芬曾俊雄 2007/01/02. Outlines Suffix Tree MUMmer 1.0 MUMmer 2.1 MUMmer 3.0 Conclusion