1 Remote Engineered Super Resolved Imaging Zeev Zalevsky Faculty of Engineering, Bar-Ilan University, 52900 Ramat-Gan, Israel

1

Remote Engineered Super Remote Engineered Super Resolved ImagingResolved Imaging

Zeev ZalevskyZeev Zalevsky

Faculty of Engineering, Bar-Ilan University, 52900 Ramat-Gan,

Israel

2

Outline

•Introduction

•The “SW Adaptation” Process

•Diffractive type Super Resolution

•Geometrical type Super Resolution

•Hearing with light

•Conclusions

3

Outline

•Introduction





•Conclusions

4

Introduction-Diffraction LimitationIntroduction-Diffraction Limitation

What is Resolution? Resolution is finest spatial feature that

an imaging system can resolve. Resolution of optical systems is restricted

by diffraction (Lord Rayleigh, Abbe), by the geometry of the detector and by the noise equivalence of its pixels.

5

Introduction-Diffraction LimitationIntroduction-Diffraction Limitation

B

ffXMINxRES

22.122.1}{ #

Diffraction limitation of resolution is proportional to the F number of the imaging optics.

6

Introduction- Geometrical LimitationIntroduction- Geometrical Limitation

Geometrical resolution is limited by the number of detector’s pixels and their size.

7

Introduction- Introduction- Noise Equivalent Noise Equivalent ResolutionResolution

Noise equivalent resolution is originated by the internal noises existing within each pixel of the detector (electronic noises, shot noises etc).

8

Outline

•Introduction





•Conclusions

9

SW SW Adaptation ProcessAdaptation Process

If not resolved, is it hopeless?

Types of a priori information:•A single dimensional object•Polarization restricted information•Temporally restricted signal•Wavelength restricted signal•Object shape

No, if No, if A Priori A Priori information information on the object is available!!!on the object is available!!!

10

SW SW Adaptation Process- Adaptation Process- cont.cont.

The Suggested Solution: Having a priori knowledge of the signal may

lead to super resolution using an SW (space-bandwidth) adaptation process:

Adapt the SW of the signal Adapt the SW of the signal to theto the acceptance SW of the acceptance SW of the systemsystem

otherwise0

W),x(Wallfor1),x(SW tresh Ndxd),x(SW

11

Outline

•Introduction





•Conclusions

12

Diff. SR- Time Diff. SR- Time MultiplexingMultiplexing

Time Multiplexing:Conversion of temporal degrees of freedom

to spatial domain (diffraction)Obj. Img.Ap.

G1 G2

Synchron. moving gratings The structure of the rotated grating (for 2-D S.R. effect)

... dt

Recent improvements: •Automatic synchronization (one grating, transmitted twice)•2-D objects, 2-D gratings•Dammann gratings


14

With clear aperture

Without time multiplexing

With time multiplexing


15

Remote Diff. SRRemote Diff. SR

Projected grating

Open aperture

Closed aperture

Reconstruction

16

Diff. SR- Speckle ProjectionDiff. SR- Speckle Projection

Closed aperture

Reconstruction

Coherent Incoherent

17

Remote Diff. SR- via Remote Diff. SR- via BackgroundBackground

Open aperture Closed aperture Reconstruction

18

Remote Diff. SR- via Remote Diff. SR- via background, cont.background, cont.

Closed aperture Reconstruction

Background

Open aperture

Reconstruction

Closed aperture sequence

19

Remote Diff. SR- from Remote Diff. SR- from satellitesatellite

Numerical simulations

Experimental results

20

Remote Diff. SR- via Remote Diff. SR- via rain/dropletsrain/droplets

Open aperture Closed aperture Reconstruction

21

Spatial DLP based SRSpatial DLP based SR

Two possible experimental setups.

DMD1

DMD2

CAMERA

FOV

FOV SR.

22

FOV improvement 3X3FOV improvement 3X3

Sensor size image

Raw images (different DMD positions)

Sensor FOV

10 20 30 40 50 60 70 80 90 100 110

10

20

30

40

50

60

70

80

90

yo1

20 40 60 80 100

20406080

yo2

20 40 60 80 100

20406080

yo3

20 40 60 80 100

20406080

yo4

20 40 60 80 100

20406080

yo5

20 40 60 80 100

20406080

yo6

20 40 60 80 100

20406080

yo7

20 40 60 80 100

20406080

yo8

20 40 60 80 100

20406080

yo9

20 40 60 80 100

20406080

23

FOV improvement 3X3FOV improvement 3X3

Two types of algorithms (LSQR, L1) N = 8 DMDs positions

Original FOV = middle square only

LSQR N=8 L1 N=8

24

Spatial DLP based SRSpatial DLP based SR

The experimental setup.

Experimental results: (a). Image captured with open iris. (b). Image captured with semi closed iris. (c). Reconstructed super resolved image (demonstrating optical zooming of 3x).

(a). (b).

(c).

25

Optical SAR- IOptical SAR- I

Left: Schematic sketch of the proposed configuration. Right: The proposed iterative algorithm.

Remote scene

Detector

Imaging lens

Movement direction

Sufficiently far distance to justify far field approximation

Focus plane 2

Focus plane 1

1 1exp( )A i

2 2exp( )A i

A1 amplitude of image with scale 1

If MSE reaches minimathen stop

Random phase 1

Scale transformation (based on zero padding in Fourier domain) from 1 to 2

A2 amplitude of image with scale 2

2 2exp( )B i

Scale transformation (based on zero padding in Fourier domain) from 2 to 1

1 1exp( )B i

26

Optical SAR- II, Optical SAR- II, SimulationsSimulations

Numerical simulations. (a). Original image. (b). Its reconstruction. (c). The type of reconstruction that is obtained when the phase is wrongly reconstructed.

0 50 100 150 200 250 3000

50

100

150

200

250

300

350

50 100 150 200 250

50

100

150

200

250

50 100 150 200 250

50

100

150

200

250

50 100 150 200 250

50

100

150

200

250

(a). (b). (c).

The unwrapped phase of the Fourier of the original image and the reconstructed phase.

Left: Original object. Middle: Original object Fourier transform. Right: Blurred image.

27

Optical SAR- III, Optical SAR- III, Experimental resultsExperimental results

28

Outline

•Introduction





•Conclusions

29

Geometrical SR- Geometrical SR- Intermediate plane maskIntermediate plane mask

Reconstruction: (a). Field of view border condition. (b). High resolution mask in the intermediate image plane. (c). Low resolution mask in the intermediate image plane. Mask+sensor are shifted for the micro scanning process

(a). High resolution ref.

(b). Low resolution image without SR

(a). (b).

(a). (b). (c).

30

Geometrical SR- Improved Geometrical SR- Improved TechniqueTechnique

ObjectPlane

Intermediate Image Plane

Moving MaskLens CCD

Image Plane

(a) A random mask with size of . (b) Optical configuration including the binary mask. The mask has to be in an intermediate imaging position. The mask can be moved only in one direction to get different images each time.

(a). (b).

31

Geometrical SR- Improved Geometrical SR- Improved TechniqueTechnique

Improvements:•The mask is the only part being shifted (instead of

mask+sensor).

•The SR is achieved, without any spatial loss of

information.

•The reconstruction has a reduced sensitivity to noise.

•Although the movement of the mask is in 1-D, the

obtainable SR is 2-D.

•The movement of the mask does not have to be in

sub pixel steps.

•The recovery time is improved (the reconstruction

process each pixel can be treated separately &

simultaneously).

32

Geometrical SR- Simulations vs. SR Geometrical SR- Simulations vs. SR factorfactor

(a).

(b). (c).

(d). (e).

(f). (g).

(h). (i).

(a).

(b). (c).

(d). (e).

(f). (g).

(h). (i).

(a).

(b). (c).

(d). (e).

(f). (g).

(h). (i).

Simulation results depicting the algorithm dependence on the super resolution factor. Image Size: 256256, Number of images taken during the process = 2×SR factor, percentage of the image covered by the random mask = 50%, Noise variance = 0.001. (a). Reference image.

Low resolution images ((b), (d), (f), (h)) and their corresponding high resolution reconstructions ((c), (e), (g), (i)) for SR factor of 4X4, 8X8, 12X12 and 16x16 respectively.

Super Resolution Factor (Blocked Area 50%, VarNoise 0.001)

(a).

(b). (c).

(d). (e).

(f). (g).

(h). (i).

Dependence of the algorithm and the runtime on the super resolution factor. Number of images taken during the process = 2× SR factor, the size of the ring image is 256×256 pixels. It has been obtained by performing matrix inversions. The size of each matrix is 2×SR factor rows, and SR factor columns.

33

Geometrical SR- Experimental resultsGeometrical SR- Experimental results

Auto Collimator

Folding Mirror

Binary Mask

Relay Lens

Spherical Mirror

Aperture Stop

Detector

Auto Collimator

Manual micrometer

The experimental setup

34

Geometrical SR- Experimental resultsGeometrical SR- Experimental results

Upper row: The left image: Central part of a low resolution image. The right image: Resulting reconstructed higher resolution image.

Lower row: The cross sections.

35

Outline

•Introduction





•Conclusions

36

Hearing with Light:Hearing with Light: FeaturesFeatures

•The ultimate voice recognition system compatible to “hear” human speech from any point of view (even from behind).•There is no restriction on the position of the system in regards to the position of the sound source.•Capable of hearing heart beats and knowing physical conditions without physical contact for measuring.

Opto-Phone: Hearing with Opto-Phone: Hearing with LightLight

37

Features- cont.Features- cont.

•Works clearly in noisy surroundings and even through vacuum.•Allows separation between plurality of speakers and sounds sources.•Works through glass window. •Simple and robust system ( does not include interferometer in the detection phase).


38

Any visible distance

Imaging module

Invisible Laser projection

CameraSensor

Laser


Imaging module


CameraSensor

Laser


Imaging module


CameraSensor

Laser

CameraImaging

lens

Laser

CameraImaging

lens

Laser


39

LetLet’’s listens listen……from 80mfrom 80m

Heart beat pulse taken from a throat

Cell phone

Counting…1,2,3,4,5,6

Face (profile)

Counting…5,6

Back part of neck

Counting…5,6,7

All recordings were done in a very noisy constriction site at distance of more than 80m.

120 140 160 180 200 220 240 260 280 300 320

-1

-0.5

0

0.5

1X - movement

40

Results: Detection of Results: Detection of occluded objects Ioccluded objects I

(a). Camouflaged object. (b). Camouflage without the object. (c). The object (upper left part) and the low resolution camouflaged scenery.

(a). (b). (c).

(d). The spectrogram of the camouflaged object with its engine turned on. (e). The spectrogram of the object with its engine turned on and without the camouflage. (f). The spectrogram of the camouflaged object without turning on its engine.

Spectrogram

Fre

quen

cy [H

z]

Time [sec]

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

0

50

100

150

200

250

300

350

400

450

500

200

400

600

800

1000

1200

1400

1600

1800

2000

Spectrogram

Fre

quen

cy [H

z]

Time [sec]

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

0

50

100

150

200

250

300

350

400

450

500

100

200

300

400

500

600

700

800

900

Spectrogram

Fre

quen

cy [H

z]

Time [sec]

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

0

50

100

150

200

250

300

350

400

450

500

2

4

6

8

10

12

(d). (e). (f).

41

Results: Detection of Results: Detection of occluded objects IIoccluded objects II

0 1000 2000 3000 4000 5000 6000-20

-15

-10

-5

0

5

10Y - pos

Sample

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000-15

-10

-5

0

5

10Y - pos

Sample

0 1 2 3 4 5 6 7 [sec]

(a). The scenario of the experiment. (b). Experimental results: upper recording is of the camouflaged subject. Lower recording is the same subject without the camouflage.

(a). (b).

42

Outline

•Introduction





•Conclusions

43

Conclusions:• Resolution of optical system is restricted by

various terms.• SW Adaptation process is a useful tool for

designing super-resolution systems.• A generalization for handling more types of

resolution restrictions was introduced for large variety of applications.

• Examples of achieving super resolution effects were viewed.

• New approach for “hearing” with light was demonstrated.

Documents

1 Remote Engineered Super Resolved Imaging Zeev Zalevsky Faculty of Engineering, Bar-Ilan University, 52900 Ramat-Gan, Israel