A passive auto-focus camera control system

視覺系統實驗室視覺系統實驗室指導教授：孫永年教授指導教授：孫永年教授授課教授：連震杰教授授課教授：連震杰教授學生：蘇育興（學生：蘇育興（ P7698428P769842833 ））　　　蔡翔宇（　　　蔡翔宇（ P7698463P769846311 ））日期：日期： 2009/12/222009/12/22

Applied Soft Computing 10 (2010) 296–303Applied Soft Computing 10 (2010) 296–303

Chih-Yung Chen, Rey-Chue Hwang, and Yu-Ju Chen

Outline• Introduction• Auto-focus

– Active auto-focus system– Passive auto-focus system

• The algorithms for sharpness measurement– Discrete wavelet transform (DWT)– Morphology ( ex. Dilation & edge enhancement )

• Self-organizing map (SOM) neural network• Experimental results & Conclusion

2

(#) Introduction

Introduction• With the rapid development of digital still camera (DSC) technology,

auto-focus has become an important function and widely used in many mobile devices such as cellular phone, notebook, and personal digital assistance (PDA).

• Generally, the high-level DSC must have the active focus component such as infrared or ultrasonic sensor which can measure the distance between camera and object.

• Undoubtedly, such an active focus component design in DSC will have some drawbacks such as the increasing of product’s cost and more battery power.

4

(#) Auto focus

Active auto-focus system• Most of high-level DSCs provide focusable function which can

improve the quality of image. Basically, the theorem of object captured by camera and the formation of image are shown in Fig. 1.

6

Fig. 1. The diagram of camera focus.

Active auto-focus system (con.)• In the process of photo shooting, the correct lens position could be

manually adjusted based on user’s experience.

• In the active auto-focus system, active component such as infrared rays or ultrasonic component is used to help the lens to reach a suitable position.

• Once the value of p is obtained, the proper value of q for lens position can be automatically controlled by an established motor controller in order to fit Eq. (1).

7

Passive auto-focus system• Passive auto-focus (PAF) is a technique to find the best focused lens

position without active component.• A good sharpness measurement algorithm should obtain the higher

sharpness value for the proper lens position to a well-focused image than an improper one.

8

Fig. 2. The passive focus sharpness curve.

Passive auto-focus system (con.)• In our study, DWT and morphology edge enhancement algorithm

were used for the sharpness measurement.• Furthermore, Self-organizing map (SOM) neural network controller

was designed to control the adjustable lens to reach an accurate position.

9

Fig. 3. The flowchart of the proposed system.

(#) The algorithms for sharpness measurement

Haar transform

11

Fig. A. original image Fig. B. after twice DWT

Haar transform• 1st Horizontal division

12

20 15 30 20

17 16 31 22

15 18 17 25

21 22 19 18

35 50 5 10

33 53 1 9

33 42 -3 -8

43 37 -1 1

Haar transform (con.)• 2nd Vertical division

13

35 50 5 10

33 53 1 9

33 42 -3 -8

43 37 -1 1

68 103 6 19

76 79 -4 -7

2 -3 4 1

-10 5 -2 -9

Lifting 5/3 DWT (1/3)• The procedure of Lifting 5/3 DWT can be mainly divided into two

steps:1. Splitting step: In this step, a raw data R with length n can be divided into two

parts, the odd points r2i+1 and even points r2i. They are denoted as 　 　 and 　 　 , respectively. Their mathematical expressions are given by

2. Lifting step: The high-pass elements 　 　 and low-pass elements 　 　 can be calculated by

14

S i

0

Di

0

S i

1Di

1

Lifting 5/3 DWT (2/3)• In DWT, Lifting 5/3 is the simplest one which is shown in Fig. 4.

15

Fig. 4. Lift 5/3 DWT diagram.

Lifting 5/3 DWT (3/3)

16

Fig. 5. Frequency bands of DWT. (a) The focused image and (b) the blurred image.

Morphology sharpness measurement • For a briefly explanation, only dilation is reported in this paper.• A two-dimensional structuring element U with size ns x ns is

defined. The operations are described as follows:1. Dilation: In image processing, dilation operation can extend the boundary

of an object by removing the low valued regions. After the dilation operation, new pixel 　 can be obtained by the following equation:

2. Edge enhancement: The edge detector of image segmentation usually has a huge computational complexity. However, the morphology edge enhancement listed in Eq. (7) is simple and efficient. The operation of each pixel includes one subtraction and one dilation.

17

' ,x ji

Morphology sharpness measurement (con.)

• All obtained 　 pixels (i.e., the preformed image 　 ) can be summed up for the sharpness measurement and expressed as

18

Fig. 6. The edge enhancement results of the well-focused and blurred images.

x jiˆ , X

The proposed hybrid sharpness measurement

• The ideal curve of sharpness versus lens position could be generally described as Gaussian curve.

19

Fig. 7. The comparison of sharpness curves versus lens positions.

(#) SOM neural network

SOM neural network (1/8)• In the active auto-focus system, the best focused lens position is

determined by the distance between object and camera. In PAF system, the best focused lens position searching needs to consider the optimal focused sharpness value.

• But, the optimal focused sharpness value of a scene is usually unknown before the best focused position is reached.

• In this study, a SOM neural network based controller is designed to find the best focused lens position.

21

SOM neural network (2/8)• The input layer and Kohonen layer are defined as vectors s and k

respectively, and given by

• Each neuron k in Kohonen layer k has the same dimension as input space s. Thus, a weight vector wi of k can be expressed as

22

Fig. 9. SOM neural network model.

SOM neural network (3/8)• k is the training data which is the representative sharpness values

of the fixed lens positions s0, s1, and s2.

23

SOM neural network (4/8)

24

Fig. 8. Sharpness values of different objects with different lens positions.

SOM neural network (5/8)• The best-matching neuron of the input vector can be found by

minimizing the Euclidean distance between s and wi two vectors.• Suppose the best-matching neuron index (i.e., the winner’s index)

of the input layer s is denoted as b(s), then it can be determined by the following condition:

• Once the best-matching neuron was found, the weight vectors then can be updated. The formula of weight vector adjustment is expressed as

25

SOM neural network (6/8)

26

Fig A. Two dimension Kohonen model Fig B. Topological Structure

SOM neural network (7/8)• The pre-defined focused position (i.e. the revolutions of servo

motor) can be given by z as shown in Eq. (17) which has the same dimensions as k.

• In the hardware design, it can be stored in the read-only memory.

• Once the input data is measured, the trained SOM neural network can determine the output category.

27

SOM neural network (8/8)• In the system we proposed, the neural network is firstly trained by

an offline process. The well-trained SOM neural network is used to predict the best focused lens position.

28

Fig. 10. The proposed searching method.

(#) Experimental results

Experimental results (1/5)• Embedded system and hardware implementation

– Adjustable lens module• a CMOS type image sensor with 100 K color pixels (i.e., 352 X 288) was used to

capture images.– FPGA hardware design

• Lift 5/3 DWT, morphology edge enhancement, and pulse width modulation (PWM ) controller are all implemented by a FPGA chip (Altera Cyclone EP1C12Q240C8).

– Microprocessor• The system clock of high-speed 8051 microprocessor used is 100 MHz and the size

of external SRAM is 64 KB.– PDA displayer

• Microsoft .NET Framework. • The software is operated under the Windows Mobile 2003 operation system.

30

Experimental results (2/5)

31

Fig. 11. The proposed PAF platform. Fig. 12. Adjustable lens module.

Experimental results (3/5)• To evaluate the performance of proposed system, this section

presents the results of the sharpness measurement approach and search algorithm. Besides, the simulation results of each hardware function block implemented by Altera Quartus II 5.0 are shown in Table 1.

32

Experimental results (4/5)• Finally, we concluded that the best parameters of SOM can be

obtained at the learning rate 0.977 and at the iteration number 100.• Basically, microprocessor will take much time in the image

capturing and the sharpness measuring for a conventional PAF.

33

Experimental results (5/5)

34

Fig. 15. The images of focus procedure.

(#) Conclusions

Conclusions• In this paper, a passive auto-focus control system was designed and

the new sharpness measurement scheme with SOM neural network based searching strategy was also proposed.

• Advantages– Easy implementation by hardware– No active component– Rapid autofocus– Low image processing complexity

• Drawbacks– However, in real case, the accurate estimation of the sharpness for an image

with lower or excessive illumination is not easy.

36

(#) Conclusions

※※ 因因 datadata 不足，故捨棄不足，故捨棄 SOM neural networkSOM neural network

Search Method (1/6)• 三次曲線與二次曲線近似演算法

二次曲線與三次曲線近似演算法的實作方法可分為以下幾個步驟Step 1: 移動取像直到△ y 發生正負變化

依據對焦量測準則結果的清晰度曲線分佈特性，我們將顯微鏡馬達每次的移動量（ Moving Step ）暫訂為顯微鏡馬達總行程的五分之一，開始移動取像並計算其對焦量測準則數值，紀錄上述的取像位置 x 與對焦量測準則數值 y 直到△ y 發生正負變化。Step 2: 三次多項式近似

下式為三次多項式的數學表示式。若△ y 發生正負變化，則將之前所紀錄資料中對焦量測數值較大的前四筆變化資料取出來用Newton polynomial form 建立可以表示此四筆資料分佈特性的三次多項式

38

dcxbxaxy 23

Search Method (2/6)Step 3: 三次多項式求解

在數學上，三次多項式可以透過一次微分等於零 f’(x)=0 的方式來求的兩相異實根，一次微分等於零是在找切線斜率等於零的點。而三次曲線一次微分等於零的二點之中，其中一點就是我們在找的對焦點。Step 4: 二次多項式近似

下式為二次多項式的數學表示式。將顯微鏡馬達移動到上次多項式近似時所求得的對焦點，取像並計算其對焦量測準則數值，將其紀錄資料中對焦量測數值較大及其左右二筆共三筆變化資料取出來用 Newton polynomial form 建立可以表示此三筆資料分佈特性的二次多項式。39

cbxaxy 2

Search Method (3/6)Step 5: 二次多項式求解

使用矩陣解法可以求得二次多項式的係數 a 、 b 及 c ，依據圖形曲線分佈得知其為一拋物線，我們可以用矩陣方式求解並以頂點公式 求得對焦點的位置。Step 6: 判斷是否已經找到正確的對焦位置

將顯微鏡馬達移動到二次多項式近似時所求得的對焦點，取像並計算其對焦量測準則數值，將其取像位置及其對焦量測數值紀錄下來，並判斷其對焦位置是否與之前計算出的位置一樣，若一樣則表示已找到正確的對焦位置，若否，則繼續重複步驟 4 和步驟5 。

40

ab2

Search Method (4/6)

41

Search Method (5/6)

42

Search Method (6/6)

43

影片 20X 20X001 40X 40X001

搜尋位置 1219 859 980 1111

最佳位置 1211 870 1010 1113

秒數 2sec 2sec 2sec 2sec

20x

44

A. Our Method B. Ground truth

20x001

45

A. Our Method B. Ground truth

40x

46

A. Our Method B. Ground truth

40x001

47

A. Our Method B. Ground truth

(#) Thank you