Graph-based Deformable Matching of 3D Line Segments with Application in Protein Fitting 12 1 HANG DOU 1, MATTHEW L BAKER 2, TAO JU 1 1 1 Washington University

1

Graph-based Deformable Matching of 3D Line Segments with

Application in Protein Fitting

HANG DOU1, MATTHEW L BAKER2, TAO JU1

Washington University in St. Louis1

Baylor College of Medicine2

2

Deformation

Applications of Feature Matching

Data alignment

Co-segmentationReconstruction

3

Our Application• Protein fitting• Protein: building blocks of life forms.

…

4

Protein

Atomic structure Low-res structure

5

Protein Fitting

Atomic structure (Cartoon Rendering)

Low-res structure (Volume)

Fitting result

6

Protein Fitting• Secondary structure (alpha helix)

7


8


9

Protein Fitting• Our method• Find feature (helix) correspondence

• Deform the protein guided by the helices correspondence

10

Protein Fitting• Our method• Find feature (helix) correspondence

• Deform the protein guided by the helices correspondence

11

Problem Statement

• Input:• Undirected line segments• |Source| <= |Target|

• Output:• One to one correspondence

• Matching Criteria:• Similar length• As rigid as possible deformation Source Target

12

Problem Statement




13

Problem Statement




14

Previous Work• Matching points: Lots of work• [Fitzgibbon 2003], [Angulov 2004], [Leordeanu 2005], [Chang 2009], [Chertok 2010], [Duchenne 2011], [TAM 2014], ….

15

Previous Work• Matching line segments: very few work• [Abeysinghe 2010]

TargetSource

16

Previous Work• Matching line segments: very few work• [Abeysinghe 2010]• Need line segments forming rigid body clusters

TargetSource

17

Previous Work• Matching line segments: very few work• [Abeysinghe 2010]• Need line segments forming rigid body clusters• Based on a clique-finding algorithm

TargetSource Matching Result

18

Our Work• A novel method for matching 3D line segments• Allowing fully non-rigid deformations

• Technical contributions• Formulating the problem as graph matching• Improved graph matching using continuous relaxation

19



20

Background: Graph Matching• Input• The two graphs: source and target• Affinity score (between node pairs and edge pairs)• Node pair score: fnode(si, tj)• Edge pair score: fedge({si, sj}, {tk, tl})

• Goal• Find mapping π: S -> T that maximize:

• Subject to mapping constraints

Source Target

s2

s1

t2

t1

fnode

fedge

21

Our Graph Construction• Naïve method• Graph nodes:• One undirected segment one node

𝑠1

𝑠2𝑠3

Source

𝑡1

𝑡 2 𝑡 3

𝑡 4

Target

s3s2

s1

t2

t1

t4

t3

22


• Node pair affinity• Length similarity

𝑠1

𝑠2𝑠3

Source

𝑡1

𝑡 2 𝑡 3

𝑡 4

Target

s3s2

s1

t2

t1

t4

t3

23



𝑠1

𝑠2𝑠3

Source

𝑡1

𝑡 2 𝑡 3

𝑡 4

Target

s3s2

s1

t2

t1

t4

t3

24

Our Graph Construction• Naïve method• Graph nodes:• One segment one node


• Edge pair affinity• Rigidity in deformation

𝑠1

𝑠2𝑠3

Source

𝑡1

𝑡 2 𝑡 3

𝑡 4

Target

s3s2

s1

t2

t1

t4

t3

25



• Edge pair affinity• Rigidity in deformation

𝑠1

𝑠2𝑠3

Source

𝑡1

𝑡 2 𝑡 3

𝑡 4

Target

s3s2

s1

t2

t1

t4

t3

26

Our Graph Construction• Issue of ambiguity:

Matching Source Matching Result 1

𝑠1

𝑠2𝑠3

Matching Result 2

𝑠1

𝑠2𝑠3

𝑠1

𝑠2𝑠3

𝑠4 𝑠4

27



𝑠1

𝑠2𝑠3

Matching Result 2

𝑠1

𝑠2𝑠3

𝑠1

𝑠2𝑠3

𝑠4 𝑠4

28



𝑠1

𝑠2𝑠3

Matching Result 2

𝑠1

𝑠2𝑠3

𝑠1

𝑠2𝑠3

𝑠4 𝑠4

29



𝑠1

𝑠2𝑠3

Matching Result 2

𝑠1

𝑠2𝑠3

𝑠1

𝑠2𝑠3

𝑠4 𝑠4

30

Our Graph Construction• Improved graph construction• Graph nodes:• Source• One undirected segment one node

• Target• One undirected segment two nodes

𝑠1

𝑠2 𝑠3

Source Target

𝑡 3

𝑡1

𝑡 2𝑡 2𝑡 3

𝑡1𝑡 4

𝑡 4

S3S2

S1

T1

T2

T3

T4

31



• Node Affinity• Length similarity

𝑠1

𝑠2 𝑠3

Source Target

𝑡 3

𝑡1

𝑡 2𝑡 2𝑡 3

𝑡1𝑡 4

𝑡 4

S3

S1

S2

T1

T3

T4

T2

32




• Edge Affinity• Rigidity in deformation

𝑠1

𝑠2 𝑠3

Source Target

𝑡 3

𝑡1

𝑡 2𝑡 2𝑡 3

𝑡1𝑡 4

𝑡 4

S3S2

S1

T1

T2

T3

T4

33




• Edge Affinity• Rigidity in deformation

𝑠1

𝑠2 𝑠3

Source Target

𝑡 3

𝑡1

𝑡 2𝑡 2𝑡 3

𝑡1𝑡 4

𝑡 4

S3S2

S1

T1

T2

T3

T4

34

Rigidity Measure• Geometric relation descriptor• Encode each pair of line segments (s, s’) as a vector: (, , )• Rigidity between two pairs are differences in the vector

𝛼𝛽

𝜃

𝑑𝑖𝑠𝑡 (𝐬 ,𝐬 ′)𝐬 𝐬 ′

35

Our Graph Construction• No more ambiguity

𝑠1

𝑠2 𝑠3

Matching Source Matching Result

𝑡1

𝑡 2𝑡 2𝑡 3

𝑡1𝑡 4

𝑡 4

𝑡 3

36



37



38

Solve Graph Matching (review)• Formulate graph matching as quadratic assignment• Encode graph affinities in a matrix M

s1,t1

s1,t1

Affinity Matrix (M)

s3,t3

s3,t3…. ….

….

…. s3s2

s1

t2

t1

t4

t3

Source Target

39

Solve Graph Matching (review)• Formulate graph matching as quadratic assignment• Encode graph affinities in a matrix M

s1,t1

s1,t1

Affinity Matrix (M)

s3,t3

s3,t3…. ….

….

…. s3s2

s1

t2

t1

t4

t3

Source Target

40

Solve Graph Matching (review)• Formulate graph matching as quadratic assignment• Encode graph affinities in a matrix M• Matching goal:

s3s2

s1

t2

t1

t4

t3

Source Target

s1,t1

s1,t1

Affinity Matrix (M)

s3,t3

s3,t3…. ….

….

….

41

Solve Graph Matching (review)• Formulate graph matching as quadratic assignment• Encode graph affinities in a matrix M• Matching goal:• Find assignment vector x {0,1}|S|x|T|

• Maximize: xT M x (equation 1)• Subject to mapping constraints. Mapping constraint

Integer constraint

42

Solve Graph Matching (review)• Solve quadratic assignment• Combinatorial optimization• Usually time consuming.

• Continuous relaxation• Compute confidence vector X’ R|S|x|T| without some or all the constraints • X’(a) is the confident of the match a (Si -> Tj)

• Binarize X’ to obtain assignment vector X {0,1}|S|x|T| with all constraints• X(a) is 1 if a is an accepted match and 0 otherwise.• Widely used greedy approach [Leordeanu 2005]: sequentially pick matches with descending

confidence, while avoiding conflict based on mapping constraints.

43

Binarize Confidence Vector (Review)

TargetSource

0.4

0

Confidence Vector

44


0.4

0

Confidence VectorTargetSource

45


0.66

0

TargetSource

0.4

0

Confidence Vector

46


0.66

0

TargetSource

0.4

0

Confidence Vector

47

Binarize Continuous Solution (Review)

0.66

0

TargetSource

0.4

0

Confidence Vector

48


Correct Matches

TargetSource

0.66

0

0.4

0

Confidence Vector

49

Our Continuous Relaxation• Key idea• Only takes the assignments with high confidence• Formulate a smaller matching problem, constrained by the chosen

assignments

• Iterate until we find all the matches• Compute confidence vector (with any chosen continuous method)• Pick matches whose relative confidence (w.r.t. the highest confidence)

value is above a threshold r• Reconstruct the (smaller) affinity matrix with remaining matches• Edge affinity of already picked segments is added to the diagonal of the new matrix

50

0.66

0

Confidence vector in 1st iteration

Our Continuous Relaxation

TargetSource

r = 1

51


0.66

0

r = 1

Confidence vector in 1st iteration

TargetSource

52


0.95

0

r = 1

Confidence vector in 2nd iteration

TargetSource

53


0.95

0

r = 1

Confidence vector in 2nd iteration

TargetSource

54


0.95

0

r = 1

Confidence vector in 3rd iteration

TargetSource

55


0.95

0

r = 1

Confidence vector in 3rd iteration

TargetSource

56


0.71

0

r = 1

Confidence vector in 4th iteration

TargetSource

57


0.71

0

r = 1


TargetSource

58


r = 1


TargetSource

0.99

0

59


0.99

0

r = 1


TargetSource

60

0.99

0


r = 1


TargetSource

61

0.99

0


r = 1


TargetSource

62


Correct Matching Result

TargetSource

63

Experiment Result• Synthetic data and authentic data• Synthetic data: • generated by thin plate spline (TPS) deformation.

• Authentic data: • 16 pairs of proteins from Protein Data Bank.

• Accuracy and function score• Matching accuracy: • ratio of correct matches over all matches.

• Total affinity ratio: • Function score (xT M x) ratio of our method over a benchmark method [Leordeanu 2005].

64

Experiment Result• Synthetic data• Pick one protein as the matching source• Deform the source line segments at different level of normalized bending

energies as matching targets

Source Bend Energy = 0.2 Bend Energy = 0.7

65

Experiment Result• Synthetic data

Leordeanu[2005]Our, r = 0.6

Our, r = 0.3

Our, r = 1.0

Bending Energy

Tota

l Affi

nity

Rati

o

66


Bending Energy

Mat

chin

g Ac

cura

cy

Leordeanu[2005] Our, r = 0.3 Our, r = 0.6

Our, r = 1.0 Abeysinghe[2010]

67


Bending Energy

Mat

chin

g Ac

cura

cy

Leordeanu[2005] Our, r = 0.3 Our, r = 0.6

Our, r = 1.0 Abeysinghe[2010]

68

Experiment Result• Authentic data:

1sx4-A 1ss8-A

Source Target(Our) Target(Leordeanu) Target(Abeysinghe)

69

Experiment Result• Authentic data:

Protein

Mat

chin

g Ac

cura

cy

Leordeanu Our, r = 1.0, 0.6, 0.3 Abeysinghe

70

Conclusion• An algorithm for finding correspondence between two

sets of 3D line segments• Allowing fully non-rigid deformations

• Technical contributions• Formulating the problem as graph matching• Solve matching problem using a iterative continuous-discrete paradigm

• Validation• Test on both synthetic and authentic data

71

Questions?

Documents

Graph-based Deformable Matching of 3D Line Segments with Application in Protein Fitting 12 1 HANG DOU 1, MATTHEW L BAKER 2, TAO JU 1 1 1 Washington University