Introduction to my Research

Research Review

Kuo-Yen Lo 羅国彦

ロ　コウエン

2013.4.18Sato Laboratory, University of Tokyo

Short Curriculum Vitae

• Personal Homepagehttp://www.hci.iis.u-tokyo.ac.jp/~kylo/

• Period of past yearsUniversity (2006 – 2010)Internship (2010 – 2011)Research Assistant (2012 – 2013)

• Two main research topic: 1. 2D-to-3D conversion2. Photo Aesthetics.

• Wrap up

Short Curriculum Vitae

• Language: English (TOEIC 935), JLPT(N1), Chinese(Native)

• Programming: C/C++, Matlab, Java(android)Technique: SIFT/SURF/HOG, K-means, GMM, kNN, SVM, PCA/LDA/ITML, bad-of-visual word, bilateral filter.

• Have traveled to: USA, Korea.Want to travel to: China, Thailand, Spain

• Why Japan-- historical and cultural connection-- camera companies and electronics maker here-- founded by Panasonic Scholarship

Visual Cues Low-levelMathematics

Machine LearningPsychology

Computer Vision

Overview

2006NTU

2007UPenn

2008OpenCV

2009RoboticsContest

GenderRecognition

Contest

2012ResearchAssistant

ICPR2012

ACCV_w2012

>>2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

National Taiwan University

2006NTU

2007UPenn

2008OpenCV

>>

People in Vision• Yi-Ping Hung(MM12, CVPR11, CHI11, UIST11)• Yung-Yu Chuang(CVPR12*3)• C.J. Lin (Libsvm)• H.T. Lin(ICML12, NIPS12, CVPR11

KDD12 Champion)• Winston H. Hsu(MM12*6)

33,0001928 B.C.

1.6World rank 80

Summer School in UPenn

2006NTU

2007UPenn

2008OpenCV

>>Summer Language ProgramUniversity of Pennsylvania

Join the Lab

2006NTU

2007UPenn

2008OpenCV

>>Biophotonics and Bioimaging Laboratory

Prof. Ta-Te LinOpenGLOpenCV

BorlandC++ Builder

Robotics Contest @ ASABE 2009

2009RoboticsContest

GenderRecognition

Contest

>>• Problem:

Detecting and Positioning the circular obstacle• Technique:

Graphical simulation(OpenGL), Sensor• Material:

Boe-Bot Toolkit, IR sensor, Ultrasonic sensor, Zigbee wireless communication

Robotics Contest @ ASABE 2009

2009RoboticsContest

GenderRecognition

Contest

>>

Please Visit the following link for viewing the video:

http://youtu.be/8EjON8Y2OJ0

Gender Recognition Contest

2009RoboticsContest

GenderRecognition

Contest

>>• Problem:

Recognize the gender with single face image• Technique:

Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm check

Rotate the image 35 degreeto detect all possible tilt face

Gender Recognition Contest

2009RoboticsContest

GenderRecognition

Contest

>>• Problem:

Recognize the gender with single face image• Technique:

Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm check

Use Eye detectorto wrap the face tountilt view.

Gender Recognition Contest

2009RoboticsContest

GenderRecognition

Contest

>>• Problem:

Recognize the gender with single face image• Technique:

Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm check

Utilize Skin color model, Eye and Mouth detector to filter the false-positive result from the V-J face detector.

Gender Recognition Contest

2009RoboticsContest

GenderRecognition

Contest

>>• Problem:

Recognize the gender with single face image• Technique:

Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm check

• Performance1.2 second per 480*320 image

• Result~85% face detection accuracy~75% gender recognition accuracyWin 3rd place among 20 teams (Taiwan and China). Bonus 60,0000yen.

Fish Recognition

2010ISMAB

NTUGraduate

Internship@ TV corp.

>>• Problem:

Tuna species recognition for fishery conservation and management

• Task:Detection and Classification

Bigeye

YellowfinAlbacore

3 spices are considered2011Navy

Fish Recognition>>

• Problem:Tuna species recognition for fishery conservation and management

• Task:Detection and Classification

Fish Image are capturedin certain lighting condition with measurement plate.Body part is smooth,makes it reflect light well.

72%

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

Fish Recognition>>

• Problem:Tuna species recognition for fishery conservation and management

• Task:Detection and Classification

B Y AB 89 10 5Y 9 86 3A 10 9 81

84%

34% 72% 52% 58%

Confusion Matrix

(Head)(Abdomen)(Tail fin) (Tail)

Discriminate part!

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

Yeh, Graduation!>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

2D-to-3D conversion>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

• Problem:Generating 3D videofrom 2D content.

• Inspiration:3D information isrecovered by depthcues

Captured View + Depth

2D-to-3D conversion>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

Reality Comfort

• Accurate depth map• Correct depth order• Real-time processing

• Clear boundary• Temporal smoothness• Visual impression

How people perceive depth?>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

1. Low-level cue 2. Scene Recognition

2D-to-3D conversion>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

Video frame + motion estimation[ICCE 2009]

Approaches1. Depth map by motion2. Depth map by saliency 3. Depth map by prior

information fusion

2D-to-3D conversion>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy Video frame + Saliency map [SDA 2010]

Approaches1. Depth map by motion2. Depth map by saliency 3. Depth map by prior

information fusion

Introduction to Bilateral Filter>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

Bilateral Filter [Tomasi, ICCV98]:

f(x) h(x)

“Bi” lateral = Spatial term + Range term

Introduction to Bilateral Filter>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

Bilateral Filter [Tomasi, ICCV98]:

f(x) h(x)

“Bi” lateral = Spatial term + Range term

Introduction to Bilateral Filter>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

Bilateral Filter [Tomasi, ICCV98]:

f(x) h(x)

“Bi” lateral = Spatial term + Range term

Application of Bilateral Filter>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

“Bi” lateral = Spatial term + Range termSmooth Target Edge-Preserving Result

And this one?

2D-to-3D conversion>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

Prior fusion [Siggraph 2009]1. Decide Geometric perspective2. Integrate Image and Depth map by Bilateral filter

One-year in Navy>>

2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

Academia Sinica

2012ResearchAssistant

ICPR2012

ACCV2012

>>Institute of Information Science(Central Research Academy)

People in Vision• Chu-Song Chen (CVPR12, CVPR11*2)• Mark H. Liao (MM12*2, MM11*2)• Y.-C. Frank Wang (ECCV12, CVPR12)• Yen-Yu Lin (CVPR13, MM12, TPAMI11)

Prof. Chen

PHOTO AESTHETICS CLASSIFICATIONPredicting the visual appealing quality of photos

good?>>

jcar@DPChallenge

good?>>

Mnet @ DPChallenge

Which one is better?>>

Voted by online photo community>>

Average: 5.088 votes

Average: 7.292 votes

Reason?>>

Reason?>>

BoundaryAlternating repetition(Texture)

Contrast Levels of scale

Roughness

Strong centers

Positive space

Local symmetries

The Void

Not-separateness

Good shape

GradientsEchoes

Simplicity and Inner Calm

Deep interlock and ambiguity

Color

Composition

HarmoniumRichness

Application>>

Image Search & Management Photo evaluation system

Embedded Camera system Media analysis

Photo Aesthetics

2012ResearchAssistant

ICPR2012

ACCV2012

>>• Problem:

Recognition the appealing quality ofphoto by computational approaches.

• Technique:Image analysis, Pattern recognition,Crowdsourcing, Psychology, Photography

• Application

ICPR 2012

2012ResearchAssistant

ICPR2012

ACCV2012

>>As a Pattern Recognition Problem…Comparison of feature 1. Edge distribution, Color histogram,

Hue, Saturation.. [Ke, CVPR06]2. SIFT + BOV [Marchesotti , ICCV11]3. Composition layout (Edge + HSV),

Color palette, contrast.. [Proposed]Result

Item Speed on PC Accuracy

CVPR06 0.2s 81%

ICCV11 4s 85%

Proposed 0.16s 84%

[ Photo aesthetics assessment with efficiency ]

Extraction of Color Information

Extract N Dominant colors

(we set N=5)K-Nearest Neighbor

(K=20)

List of Palettes

Dictionary

HQ Palettes Dictionary

LQ Palettes Dictionary

Palettes of Photo

Retrieved by Frequency

Retrieved by Kmeans(Cluster Center)

Proposed (Weighted Kmeans)

Finding the Dominant Colors

Video Demo

2012ResearchAssistant

ICPR2012

ACCV2012

>>[ Intelligent Photographing Interface with On-Device Aesthetic Quality Assessment ]

Please Visit the following link for viewing the video:

http://youtu.be/o8mKuTfO6ao

Discussion 1 Device : On-line assistive camera system

• Contextual Information (Viewing angle)

camera < human• Feedback

from analysis to advice• Human behavior

What do people take?How do people take?

• Computation

Server-based v.s. Device• Market and Needs

Discussion 2 Algorithm: photo aesthetic value assessment

• Definition of photo aesthetics

Expert v.s. Volkswagen• Labeling process

Individual bias and variance.Absolute or Relative evaluationEffect of Labeling order

• Quantify photo aesthetic

Modeling, the Personalization

Thanks for your attention!

10 ratings5.00/7 average

5 ratings4.90/7 average