Business Identification:Spatial Detection

Alexander DarinoWeek 8

2

Weaknesses to Current Approach

LatitudeLongitude

Geocoding

ReverseGeocoding

Nearby Businesses

Image OCR Detected Text

Business Name

Matching

BusinessIdentification

Business Spatial

Detection

STR Implementation

• STR Implementation: “Automatic Detection and Recognition of Signs From Natural Scenes”

Multiresolution-based potential

characters detection

Character/layout geometry and color properties analysis

Local affine rectification

Refined Detection

Multiresolution-based potential characters detection

Multiresolution-based potential characters detection

Multiresolution-based potential characters detection

STR Implementation

• Original Next Step: Replace with readily available text detector

• Text detectors are not readily available

(Will revisit later)

TEMPLATE-IMAGE SIFT MATCHINGAfter many technical difficulties…

Template

Name George

Font Trebuchet MS

# Levels 3

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 20-24.1(95%)

Image

Name George …

# Levels 3

Peak Threshold 0

Edge Threshold 10

Statistics

Good 1

Bad 0

Total (% G) 1 (100%)

Template

Name George

Font Trebuchet MS

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 25.60547(95%)

Image

Name George …

# Levels 3

Peak Threshold 0

Edge Threshold 10

Statistics

Good 1

Bad 0

Total (% G) 1 (100%)

Template

Name George

Font Trebuchet MS

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 25.60547(95%)

Image

Name George …

# Levels 1

Peak Threshold 0

Edge Threshold 10

Statistics

Good 2

Bad 0

Total (% G) 2 (100%)

Template

Name George

Font Trebuchet MS

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 25.60547(95%)

Image

Name George …

# Levels 1

Peak Threshold 20

Edge Threshold 10

Statistics

Good 2

Bad 0

Total (% G) 2 (100%)

Template

Name George

Font Trebuchet MS

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 25.60547(95%)

Image

Name George …

# Levels 1

Peak Threshold 20

Edge Threshold 6

Statistics

Good 2

Bad 0

Total (% G) 2 (100%)

Template

Name Aiken

Font Trebuchet MS

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 24.439163(95%)

Image

Name George …

# Levels 1

Peak Threshold 20

Edge Threshold 6

Statistics

Good 3

Bad 0

Total (% G) 3 (100%)

Template

Name Delicious

Font Trebuchet MS

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 26.656116

Image

Name George …

# Levels 1

Peak Threshold 20

Edge Threshold 6

Statistics

Good 0

Bad 0

Total (% G) 0 (0%)

Template

Name Prepared

Font Trebuchet MS

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 26.656116

Image

Name George …

# Levels 1

Peak Threshold 20

Edge Threshold 6

Statistics

Good 0

Bad 0

Total (% G) 0 (0%)

Template

Name Foods

Font Trebuchet MS

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 26.656116

Image

Name George …

# Levels 1

Peak Threshold 20

Edge Threshold 6

Statistics

Good 0

Bad 0

Total (% G) 0 (0%)

Template

Name Bruegger’s

Font Arial Rounded MT Bold

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 32.288651

Image

Name Bruegger’s …

# Levels 1

Peak Threshold 20

Edge Threshold 6

Statistics

Good 0

Bad 0

Total (% G) 0 (0%)

Template

Name Bakery

Font Arial Rounded MT Bold

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 29.145470

Image

Name Bruegger’s …

# Levels 1

Peak Threshold 20

Edge Threshold 6

Statistics

Good 0

Bad 0

Total (% G) 0 (0%)

Template

Name Bakery

Font Arial Rounded MT Bold

# Levels 5

Peak Threshold 0

Edge Threshold 10

Scale Cutoff 29.145470

Image

Name Bruegger’s …

# Levels 1

Peak Threshold 20

Edge Threshold 6

Statistics

Good 0

Bad 0

Total (% G) 0 (0%)

SCENE TEXT RECOGNITIONMoving away from SIFT and revisiting

Scene Text Recognition

• Did not hear back from individuals contacted for STR implementation

• Returning to STR Implementation– Further reading indicates that

patches are necessary for subsequent algorithms

– Text detection is not enough: need to implement specified text detector

Multiresolution-based potential

characters detection

Character/layout geometry and color properties analysis

Local affine rectification

Refined Detection

Color Properties Analysis

• Implemented Gaussian Mixture Model (GMM) to obtain μ and σ of foreground/background for: R/G/B/H/I

• Calculated Confidences that component (RGBHI) can be used to recognize characters

Multiresolution-based potential

characters detection

Character/layout geometry and color properties analysis

Local affine rectification

Refined Detection

Original

Redμ1=141.965609756098 σ1=9.9487

μ2=255

σ2=0.2000

𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒=0.011361775502324

Greenμ1=172.337447154472 μ2=255 𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒=0.017056947503074  σ1=4.8463 σ2=0.2000

Blueμ1=122.673512195122 μ2=255 𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒=0.021524159560500  σ1=6.1478 σ2=0.2000

Hueμ1=106.601736628811 μ2=0 𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒=0.017897920959170  σ1=5.9561 σ2=0.2000

Intensityμ1=145.658856368567 μ2=255 𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒=0.051403296762968  σ1=2.1271 σ2=0.2000

Evaluation

• The highest confidence was found in Intensity even though most letters vanish, vs Hue where letters are easily distinguisible

• This suggests text recognition should occur individually per character

• The paper further suggests it needs the patches around the individual characters

Next Step

Next Step

Thank You