Lecture: Introduction to Computer...

Introduction25-Sep-2018

1

Lecture: Introduction to Computer Vision

Juan Carlos Niebles and Ranjay KrishnaStanford Vision and Learning Lab

Introduction25-Sep-2018

2Welcome to CS131

Introduction25-Sep-2018

3Jason SalavonGAN experiment on Twitter

Introduction25-Sep-2018

4Mario Klingemann, GAN experiment on Twitter

Introduction25-Sep-2018

5Memo Akten

Introduction25-Sep-2018

6

What you should expect to learn in this class.• What is computer vision?• Which problems fall under this umbrella?• Which applications are possible today and in the near future?• What are common research questions?• What is the history behind these problems and how did it lead to

deep leading?• What tools will help you develop a framework to solve these

problems?

Introduction2

5-S

ep

-20

18

7

CS131 is the introductory course for computer vision

• CS131 (fall, 2017):

– Overview of computer vision and all its applications

–Will prepare you for an industry job in vision

• CS231a (winter, 2018, Prof. Silvio Savarese)

– Advanced Computer Vision

– focusing on 3D vision

• CS231n (spring, 2018):

– Convolutional Neural Networks

Introduction25-Sep-2018

8

Mathematics

ComputerScience

Biology

Engineering

Physics

Psychology

Computer Vision

Neuroscience

Machine learning

Speech, NLP

Information retrieval

Robotics

Cognitive sciences

Algorithms, theory,…

Imageprocessing

Systems, architecture, …

optics

Introduction25-Sep-2018

9

Today’s agenda• Introduction to computer vision• Course overview

Introduction25-Sep-2018

10

Quiz?

Introduction25-Sep-2018

11

What about this?

Introduction25-Sep-2018

12

Image (or video) Sensing device Interpreting device Interpretations

garden, spring,bridge, water,trees, flower,green, etc.

What is (computer) vision?

Introduction25-Sep-2018

13

The goal of computer vision• To bridge the gap between pixels and “meaning”

What we see What a computer sees Sour

ce: S

. Nar

asim

han

Introduction25-Sep-2018

14

Image (or video) Sensing device Interpreting device Interpretations

garden, spring,bridge, water,trees, flower,green, etc.

What is (computer) vision?

Introduction25-Sep-2018

15

1981: Nobel Prize in medicine

Hubel & Wiesel

Introduction25-Sep-2018

16

Potter, Biederman, etc. 1970s

Human vision is superbly efficient

Introduction25-Sep-2018

17Thorpe, et al. Nature, 1996

Introduction25-Sep-2018

18Thorpe, et al. Nature, 1996

150 ms !!

Introduction25-Sep-2018

19

Rensink, O’regan, Simon, etc.

Change Blindness

Introduction25-Sep-2018

20

Rensink, O’regan, Simon, etc.

Change Blindness

Introduction25-Sep-2018

21

Segmentation

Introduction25-Sep-2018

22

Perception

Introduction25-Sep-2018

23

Introduction25-Sep-2018

24

Introduction25-Sep-2018

25

Image (or video) Sensing device Interpreting device Interpretations

garden, spring,bridge, water,trees, flower,green, etc.

What is (computer) vision?

Introduction25-Sep-2018

26

Paintings in 1838

Introduction25-Sep-2018

27

1812: Jacques-Louis-DavidThe Emperor Napoleon at his Study at the Tuileries

Introduction25-Sep-2018

281808: Ingres, La grande baigneuse

Introduction25-Sep-2018

29

“From today, painting is dead”— painter Paul Delaroche

at a demonstration of the Daguerreotype, 1839

Introduction25-Sep-2018

301837: Niépce, First photo of one’s meal

Introduction25-Sep-2018

31

1838

: Bou

leva

rd d

u Te

mpl

e, D

ague

rre

Introduction25-Sep-2018

321838: First selfie, Robert Cornelius

Introduction25-Sep-2018

33DeepDreams[Mordvintsev et al. 2015]

Introduction25-Sep-2018

34

Neural Style Transfer[Gatys et al. 2015]

Introduction25-Sep-2018

35Neural Style Transfer[Gatys et al. 2015]

Introduction25-Sep-2018

36

CycleGAN [Zhu et al. 2017]

Introduction25-Sep-2018

37

The goal of computer vision• To bridge the gap between pixels and “meaning”

What we see What a computer sees

Sour

ce: S

. Nar

asim

han

Introduction25-Sep-2018

38

Origins of computer vision:an MIT undergraduate summer project

L. G. Roberts, Machine Perception of Three Dimensional Solids, Ph.D. thesis, MIT Department of Electrical Engineering, 1963.

Introduction25-Sep-2018

39

What kind of information can we extract from an image?• Metric 3D information• Semantic information

Introduction25-Sep-2018

40

Vision as measurement deviceReal-time stereo Structure from motion

NASA Mars Rover

Pollefeys et al.

Reconstruction fromInternet photo collections

Goesele et al.

Introduction25-Sep-2018

41

sky

water

Ferris wheel

amusement park

Cedar Point

12 E

tree

tree

tree

carouseldeck

people waiting in line

ride

ride

ride

umbrellas

pedestrians

maxair

bench

tree

Lake Erie

people sitting on ride

ObjectsActivitiesScenesLocationsText / writingFacesGesturesMotionsEmotions…

The Wicked Twister

Vision as a source of semantic information

Slid

e c

red

it:

Kri

ste

n G

rau

man

Introduction25-Sep-2018

42

Why study computer vision?• Vision is useful: Images and video are everywhere!

Personal photo albums

Surveillance and security

Movies, news, sports

Medical and scientific images

Introduction25-Sep-2018

43

Special effects: shape and motion capture

Sour

ce: S

. Sei

tz

Introduction25-Sep-2018

44

3D urban modeling

Bing maps, Google Streetview

Source: S. Seitz

Introduction25-Sep-2018

45

3D urban modeling: Microsoft Photosynth

http://photosynth.netSource: S. Seitz

Introduction25-Sep-2018

46

Face detection

• Many digital cameras now detect faces– Canon, Sony, Fuji, …

Source: S. Seitz

Introduction25-Sep-2018

47

Smile detection

Sony Cyber-shot® T70 Digital Still Camera Source: S. Seitz

Introduction25-Sep-2018

48

Face recognition: Apple iPhoto software

http://www.apple.com/ilife/iphoto/

Introduction25-Sep-2018

49

Biometrics

How the Afghan Girl was Identified by Her Iris Patterns

Source: S. Seitz

Introduction25-Sep-2018

50

Biometrics

Fingerprint scanners on many new laptops, other devices

Face recognition systems now beginning to appear more widelyiphone X just introduced face recognition

Introduction25-Sep-2018

51

Optical character recognition (OCR)

Digit recognition, AT&T labs

Technology to convert scanned docs to text• If you have a scanner, it probably came with OCR software

License plate readershttp://en.wikipedia.org/wiki/Automatic_number_plate_recognition

Source: S. Seitz

Introduction25-Sep-2018

52

Google maps: Annotate all houses and streets

Goodfellow et al. 2014

Introduction25-Sep-2018

53

Toys and Robots

Introduction25-Sep-2018

54

Mobile visual search: iPhone Apps

Introduction25-Sep-2018

55

Snapstacles and Google glasses

• That’s Ranjay in undergrad ->

Introduction25-Sep-2018

56

Automotive safety

• Mobileye: Vision systems in high-end BMW, GM, Volvo models – “In mid 2010 Mobileye will launch a world's first application of full

emergency braking for collision mitigation for pedestrians where vision is the key technology for detecting pedestrians.”

Source: A. Shashua, S. Seitz

Introduction25-Sep-2018

57

Vision in supermarkets

LaneHawk by EvolutionRobotics“A smart camera is flush-mounted in the checkout lane, continuously watching for items. When an item is detected and recognized, the cashier verifies the quantity of items that were found under the basket, and continues to close the transaction. The item can remain under the basket, and with LaneHawk, you are assured to get paid for it… “

Source: S. Seitz

Introduction25-Sep-2018

58

Amazon Go

Introduction25-Sep-2018

59

Vision-based interaction (and games)

Microsoft’s Kinect

Source: S. Seitz

Assistive technologies

Sony EyeToy

Introduction25-Sep-2018

60

Augmented Reality

Introduction25-Sep-2018

61

Virtual Reality

Introduction2

5-S

ep

-20

18

62

Vision for robotics, space exploration

Vision systems (JPL) used for several tasks• Panorama stitching

• 3D terrain modeling

• Obstacle detection, position tracking

• For more, read “Computer Vision on Mars” by Matthies et al.

NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of 2007.

Sour

ce: S

. Sei

tz

Introduction25-Sep-2018

63MGMT “When You Die”