40
Changjian Li 1,2 Hao Pan 2 Yang Liu 2 Xin Tong 2 Alla Sheffer 3 Wenping Wang 1 1 The University of Hong Kong 2 Microsoft Research Asia 3 University of British Columbia BendSketch: Modeling Freeform Surfaces Through 2D Sketching 通过绘制平面草图创造三维自由形状

BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

  • Upload
    others

  • View
    15

  • Download
    0

Embed Size (px)

Citation preview

Page 1: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Changjian Li1,2 Hao Pan2 Yang Liu2 Xin Tong2 Alla Sheffer3 Wenping Wang1

1The University of Hong Kong 2Microsoft Research Asia 3University of British Columbia

BendSketch: Modeling Freeform Surfaces Through 2D Sketching

通过绘制平面草图创造三维自由形状

Page 2: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Sketch-based 3D Modeling

• Sketch modeling: intuitive and effective

Teddy [Igarashi et al. 1999] True2Form [Xu et al. 2014]Paint 3D [Microsoft Crop.]

2

Page 3: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Our Goal

Model freeform 3D shapes with controlled curvature variation patterns, by

sketching 2D scattered bending lines and other curves.3

Page 4: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Our Goal

3D shape2D sketching

Model freeform 3D shapes with controlled curvature variation patterns, by

sketching 2D scattered bending lines and other curves.4

Page 5: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Origin of Idea

5

Two very similar ideas presented in 2015.

[Pan et al. 2015] [Iarussi et al. 2015]

Fitting surface to 3D curve networks, with the

surface curvature aligned to the curves.

Fitting normal vectors to 2D projected curves,

with the normal variance aligned to the curves.Combine the two, and model curvature controlled

3D surfaces directly from 2D sketching.

Page 6: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Related Work

6

Contour-based methods• Smooth low-frequency surfaces [Igarashi et al. 1999; Nealen et al. 2007; Olsen et al. 2011; Yeh et al. 2016 … ]

• Rounded shape

[Nealen et al. 2007]

Page 7: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Related Work

7

Contour-based methods• Smooth low-frequency surfaces [Igarashi et al. 1999; Nealen et al. 2007; Olsen et al. 2011; Yeh et al. 2016 … ]

• Rounded shape

• Rotational symmetry surfaces [Gingold et al. 2009; Shtof et al. 2013; Chen et al. 2014; Miao et al. 2015 ...]

• Limited variation (generalized cylinder, ellipsoid)

[Nealen et al. 2007]

[Shtof et al. 2013]

Page 8: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Related WorkContour-based methods

• Smooth low-frequency surfaces [Igarashi et al. 1999; Nealen et al. 2007; Olsen et al. 2011; Yeh et al. 2016 … ]

• Rounded shape

• Rotational symmetry surfaces [Gingold et al. 2009; Shtof et al. 2013; Chen et al. 2014; Miao et al. 2015 ...]

• Limited variation (generalized cylinder, ellipsoid)

Surfacing from curve networks• Man-made shapes [Schmidt et al. 2009; Xu et al. 2014; Bessmeltsev et al. 2012; Pan et al. 2015; Zhuang et al. 2013 ...]

• High-regularity and low curvature variation

[Nealen et al. 2007]

[Shtof et al. 2013]

8

[Schmidt et al. 2009]

Page 9: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Related Work

Domain specific methods• Layered surfaces, developable surfaces… [Bessmeltsev et al. 2015; Jung et al. 2015; De Paolo and

Singh 2015 …]

• Uneasy to generalize

Normal from sketches

• Infer normal from 2D sketches [Sykora et al. 2014; Shao et al. 2012; Xu et al. 2014; Bui et al. 2015; Iarussi et

al. 2015]

• Inaccurate to recover shape

9

[Paolo et al. 2015]

[Iarussi et al. 2015]

Page 10: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Related Work

Data-driven methods• Retrieve shape or map sketch to 3D shape [Xu et al. 2013; Xie et al. 2013, Huang et al. 2016;

Nishida et al. 2016]

• Limited by dataset diversity

• Model class-specific shapes

10

[Xu et al. 2013] [Nishida et al. 2016] [Lun et al. 2017]

Page 11: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Bending Lines

Bending lines convey surface bending directions [Eissen and Steur 2011]

• Principal curvature lines

• Used in 3D illustration• Concept design and cartoons

• Non-photo-realistic rendering [Hertzmann and Zorin 2000]

• 2.5D rendering [Iarussi et at. 2015; Shao et al. 2012]

NPR rendering[Hertzmann and Zorin 2000]

11

Concept Design[Eissen and Steur 2011]

Page 12: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Inverse Problem

Infer 3D surface from sketched bending curves

• Technical challenges:

‒ Bas-relief ambiguity: convex/concave

Crater? [Pentland 84] Ash cone [Pentland 84]

12

Page 13: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Inverse Problem

Infer 3D surface from sketched bending curves

• Technical challenges:

‒ Bas-relief ambiguity: convex/concave

‒ Bas-relief ambiguity: depth

Frontal view Side view 13

[Belhumeur et al 99]

Page 14: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Inverse Problem

Infer 3D surface from sketched bending curves

• Technical challenges:

‒ Bas-relief ambiguity: convex/concave

‒ Bas-relief ambiguity: depth

‒ Sparse input strokes

Dense lines Sparse strokes, about 1014

Page 15: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Overview of Our Solution

Direction field

diffusion

15

Convexity

classification

Input 2D

strokes

Sharp

feature

Bending

Constrained

curvature field3D shape

Disambiguate

convexity

Convert

sparse to denseSolve depth

Page 16: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Our Solution

• Convexity: stroke classification

• Observations• Convex - single stroke

16

convex

Page 17: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Our Solution

• Convexity: stroke classification

• Observations• Convex - single stroke

• Same convexity - parallel stroke

• Same/Opposite convexity - sequential strokes

convex

concave

17

Page 18: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Our Solution

• Convexity: stroke classification

• Observations• Convex - single stroke

• Same convexity - parallel stroke

• Same/Opposite convexity - sequential strokes

• Formulation: binary labeling

• Unary term: favor convex

• Binary term: relationship between strokes

𝑚𝑖𝑛. 𝐸 𝑥 =

s∈𝓥

𝜃 𝑥𝑠 +

(𝑠,𝑡)∈𝓔

𝜃(𝑥𝑠, 𝑥𝑡)

18

convex

concave

Page 19: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Our Solution

• Convexity: stroke classification

• Observations• Convex - single stroke

• Same convexity - parallel stroke

• Same/Opposite convexity - sequence strokes

• Formulation: binary labeling

• Unary term: favor convex

• Binary term: relationship between strokes

• Allow user edit stroke label

𝑚𝑖𝑛. 𝐸 𝑥 =

s∈𝓥

𝜃 𝑥𝑠 +

(𝑠,𝑡)∈𝓔

𝜃(𝑥𝑠, 𝑥𝑡)

19

User correction

Page 20: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Our Solution

• Sparse to dense: direction field diffusion [Iarussi et al. 2015]

Projected curvature directions 𝑢, 𝑣

follow the bending stroke directions,

and initialized by harmonic 4-direction fields

[Diamanti et al. 2014].

Sparse strokes Dense direction field

𝐸𝐵𝑒𝑛𝑑𝐹𝑖𝑒𝑙𝑑 𝑢, 𝑣 =1

|𝛺|න𝛺

𝛻𝑢𝑣2 + 𝛻𝑣𝑢

2 ⅆ𝜎

20

Page 21: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Projected curvature tensorsDirection field

Our Solution

21

• Depth: surface from projected curvature tensors

Height field surface 𝑧 from curvature:

Minimize 𝐸𝑚𝑎𝑡𝑐ℎ 𝑧 =1

𝛺𝛺 𝑑𝑁 ∙ 𝑢 − 𝜆𝑢𝑢

2 + 𝑑𝑁 ∙ 𝑣 − 𝜆𝑣𝑣2𝑑 𝜎

Surface & lifted strokes

curvature tensor matching

Page 22: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Surface & lifted strokes

Our Solution

22

• Depth: surface from projected curvature tensors

Height field surface 𝑧 from curvature:

with magnitudes 𝜆𝑢 𝜆𝑣:

constrained by the spatial curvature of lifted strokes.

Minimize 𝐸𝑚𝑎𝑡𝑐ℎ 𝑧 =1

𝛺𝛺 𝑑𝑁 ∙ 𝑢 − 𝜆𝑢𝑢

2 + 𝑑𝑁 ∙ 𝑣 − 𝜆𝑣𝑣2𝑑 𝜎

Minimize 𝐸𝜆 𝜆𝑢, 𝜆𝑣 =1

𝛺𝛺 𝛻𝑢𝜆𝑣

2 + 𝛻𝑣𝜆𝑢2 + 𝛽 𝛻𝑢𝜆𝑢

2 + 𝛻𝑣𝜆𝑣2 𝑑 𝜎

curvature value diffusion

Projected curvature tensors

Page 23: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Our Solution

23

Search (𝑧, 𝜆𝑢, 𝜆𝑣) in a fast iterative process,

solving linear equations per iteration.

curvature surface

Page 24: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Boundary Conditions and Other Strokes

Boundary conditions

‒ Position constraints

‒ Regularity constraints: smoothness

24

Page 25: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Boundary Conditions and Other Strokes

Boundary conditions

‒ Position constraints

‒ Regularity constraints: smoothness

Other supported strokes

‒ Contour stroke: normal constraints

25

Page 26: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Boundary Conditions and Other Strokes

Boundary conditions

‒ Position constraints

‒ Regularity constraints: smoothness

Other supported strokes

‒ Contour stroke: normal constraints

‒ Flat stroke: parallel normal vector

‒ Sharp feature: sharp angle

26

Page 27: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Boundary Conditions and Other Strokes

Boundary conditions

‒ Position constraints

‒ Regularity constraints: smoothness

Other supported strokes

‒ Contour stroke: normal constraints

‒ Flat stroke: parallel normal vector

‒ Sharp feature: sharp angle

‒ Ridge/valley stroke: a set of bending

lines with known convex and concave27

Page 28: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Patches merged

Multi-View Sketching

Sketching on the current view Surface patch for the current view

28

Page 29: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et
Page 30: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Results

Stroke number: 20 ~ 40. Time: < 3s per patch computation.30

Front Back

Page 31: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Results

31

Stroke number: 20 ~ 40. Time: < 3s per patch computation.

Page 32: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Validation

Compare with the ground truth data• 1/8 torus

32

and shoe model

GT result Our result

GT result Our result

Page 33: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

User Evaluation

Users• 7 novice users w/o background

• 2 professional 3D artists

Results• ~30 min learning

• ~10 min creation

• robust to user input diversity

33

Fast prototyping tool

Targets

Sketches

Results

Page 34: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Comparison

• BendField [Iarussi et al. 2015]

• Curvature direction field

• Inaccurate normal field for 3D reconstruction

• Our method:• Complete curvature tensor field (direction, magnitude)

• Real 3D shape

34

Page 35: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Comparison

35

Ours

BendField [Iarussi et al. 2015]

Direction field Normal map 3D shape

Page 36: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Comparison

36

BendField

with editing [Iarussi et al. 2015]

Click to edit

Normal map 3D shape

Ours

Page 37: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Comparison

• Shape from Shading [Barron and Malik 2015]

SfS results Ours

37

Page 38: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Conclusion

An intuitive and effective tool to model freeform shapes with complex curvature variation.

• Technical contributions:• Disambiguate bending stroke convexity

• Estimate bending magnitudes

• Construct surface from sparse bending strokes

38

Page 39: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et

Limitation and Future Work

Limitation• Self-occlusion or layered objects should use multi-view pipeline

Future Work• Improve user interface

• Extract information from reference image

• Integrate with other tools

39

Page 40: BendSketch: Modeling Freeform Surfaces Through 2D Sketching½˜浩.pdf• Non-photo-realistic rendering [Hertzmann and Zorin 2000] • 2.5D rendering [Iarussi et at. 2015; Shao et