Real-Time Markerless Square-ROI Recognition based on Contour-Corner for Breast Augmentation

7/30/2019 Real-Time Markerless Square-ROI Recognition based on Contour-Corner for Breast Augmentation

1/6

Real-Time Markerless Square-ROI Recognitionbased on Contour-Corner for Breast

Augmentation L. Rechard, and A. Bade

Abstract Augmented reality (AR) can be described as a technology consists of computer vision and image processing methods which is

used to enhanced user vision and perception. Whereas marker detection and recognition are the two most important steps which are vital in

the field of augmented reality (AR) in order to successfully placed computer generated object into the real world. This paper, aims to present

a real-time markerless identification and verification technique by combining contour-corner approach to detect and extract strong interest

point from the environment as the fundamental component in registering the virtual imagery with its real object without the needs to use

conventional printed marker. To enhance and to combine the current contour and corner detection approach, we proposed smoothing and

adaptive thresholding technique to the input stream and then use subpixel corner detection to obtain better and accurate interest point. Our

method used marker-less approach as to avoid the needs to prepare the target environment and to make our approach more flexible. The

proposed method starts with first getting an input from the real environment thru a camera as visual sensor. On receiving an input image, the

proposed technique will process the image, detects strong interest points from the ROI and recognize a marker by applying enhanced

contour-corner detection.

Index Terms Augmented reality, contour detection, corner detection, interest point

1 INTRODUCTION

ugmented reality (AR) can be described as a technologywhich is developed by using computer vision techniquesand image processing methods in order to enhanced user

vision and perception. Milgram et al. [1] defined AR as aresearch field in which users vision is enhanced by combiningreal world scene and computer generated objects into the

identical real environment space called mixed reality (see Fig.1) Whereas, Azuma (1997) [2] defines an AR as a system thathave the following characteristics:

Combines real object with virtual objects,

Interactive in real-time, and

Registered in three dimensions.

Fig. 1 Milgrams Reality-Virtuality Continuum [1].

Later, in 2002 Mann [3] come up with a two-dimensionalreality-virtuality-mediality continuum as an effort to addanother axis to Milgrams virtuality-reality continuum byadding mediated reality and mediated virtuality (see Fig. 2). In[3], the system can change reality in three ways:-

Add something (augmented reality)

Remove something (diminished reality) or

Alter the reality in some other way (modulated reality)

Fig. 2 Manns reality-virtuality-mediality continuum [3].

Various studies have been conducted to explore thfundamental concept and unique potential of AugmenteReality technology to be applied in mainstream applicatiosuch as in medicine, visualization, maintenance, path planninentertainment and education, industry, military and aircranavigation and yet most of it is marker-based application duto its ease of development and higher rate of accuracy an

success despite its lack of flexibility in unprepareenvironment. In the field of medicine, the use of computegenerated images such as CT-scan or MRI as a vision aid to thsurgeon in surgical planning has increased dramatically ovthe years. With the use of an AR system, we believe that thsurgeon's vision will be enhanced, more data can be obtaineensure proper surgical planning thus avoiding unnecessarcutting in real surgical operation.

Computer vision (CV) on the other hand is a field which strongly related to AR application development and succesAccording to Marr and Nishihara (1978) [4], vision is process that produces, from images of the external world, description that useful to the viewer and not cluttered witirrelevant information. In AR application, CV works hand hand with scene understanding and analysis. Sub-areas of C

A

JOURNAL OF COMPUTING, VOLUME 5, ISSUE 2, FEBRUARY 2013, ISSN (Online) 2151-9617

https://sites.google.com/site/journalofcomputing

WWW.JOURNALOFCOMPUTING.ORG 15


2/6

can be seen in motion detection, face detection, object trackingand recognition etc. The process to correctly combine the realscene with virtual objects and accurately register the mixedscene depends greatly on CV potentials to extract and detectfeatures either naturally from the desired region of interest(ROI) or based on the fiducial marker that present to thecamera. Idris et al., (2009) [5], considered the use of CVtechniques as a starting point in detecting a fiducial marker ornatural marker in order to solve the registration and trackingissue in AR application.

Most AR systems operate based on the printed squaremarker (see Fig. 3) or customized pattern which is locatedphysically on top of the target area or workspace.

Fig. 3. Example of Printed Square Markers.

Therefore, in this paper, we proposed and describe areal-time markerless identification technique designed tocapture real scene, detect strong interest points from extractedcontour, verify a marker and overlay 2D object. Extracting anddetection of features from the real scene will be our concerns inrecognizing a marker.

Our proposed technique uses the combination of contourand corner approaches in order to find unique natural featuresfrom captured frame through a web cam in real time.Generally, feature points [6], edges [7], curves [8], planes andso on, are used for registration. Even though, extraction of suchnatural feature is more difficult than the artificial vision markerbased approach, the users movement range is not limited,ensure real scene augmentation and no preparation needed to

set up the marker. Therefore it is preferable to use thisapproach over the printed marker. The remainder of thispaper, will be discussing about the basic setup of AR system,marker and marker-less detection, contour-corner approachand results for the presented methods will also be given andillustrate.

2 RELATED WORKS

2.1 Simple AR system setup

The basic setup of an AR system consists of the mobilecomputing unit such as PDA or mobile phone andhead-mounted display (HMD) to which a miniaturized camera

is attached. There are two types of HMD generally used;namely optical see-through (see Fig. 4) and video see-through(see Fig. 5).

Fig. 4 Optical See-through Augmented Reality Display [9].

Fig. 5 Video See-through Augmented Reality Display [9].

The miniaturized camera therefore captures the reaenvironment from the users perspective. The AR systemprocesses these live images in real time extracting features toidentify markers. Markers are identified using point and edgedetectors. The features can be artificial printed markers, naturamarker or natural features.

2.2 Vision-based AR

Researchers in computer vision have proposed and developedseveral methods in tracking and detection that can be applied

in AR registration. This method can be sub-divided into threetypes based on the equipment used; sensor-based, visual-basedand hybrid-based. Since the camera was already part of the ARsystem, visual-based methods are used together withvision-based registration technique.

Approaches for visual-based tracking are marker-based (seeFig. 6) and markerless-based method. Used of marker-basedmethods have been discussed in [10], [11], [12], [13]. Whereas[14], [15], [16] have established a markerless-based method forunprepared environment. Most researchers opted for markerapproach, since it is more accurate and reducecomputation-resources [17]. However, for outdoor orunprepared environment, markerless approach is more

suitable despite suffered from computation time-consuming[18, 19].





3/6

Fig. 6 Marker-based ARToolkit Algorithm [20]

In order to be able to detect interesting features from theenvironment, [21], [22], [23] have proposed and developed

corner detectors techniques for visual tracking.

2.3 Corner detection

Corner or interest point detection is a processing stage incomputer vision. Literally, corner point is some kind of visualcorner, where two edges intersect. A good corner detector mustbe able to distinguish between true and false corners, reliablydetecting corner in noised image, accurately determining thelocations of corners and can be utilized in real-timeapplications (Shi and Tomasi, 1994).

Our method is based on Shi and Tomasi detector. Thisdetector is inspired by Harris detector (Harris and Stephens,1988) and its uses smallest eigenvalue. The difference lies in the

cornerness measure or selection criteria which made thisdetector better than the original. The cornerness for Harris wascalculated in (1) as given by Harris and Stephens (1988).

(1)

There are two ways to define corner:

Large response

The locations x with R(x) greater than certain threshold.Local Maximum

The location x where R (x) is greater than those of their

neighbors.

For Shi-Tomasi, it is calculated like this:

(2)

If R(x) is greater than a certain predefined value, it can bemarked as a good feature or a corner.

Classification of feature points using eigenvalues

Fig. 7 Shi-Tomasi detector [24]

Fig. 8. Harris detector [24]

As depicted in Fig. 7, if both eigenvalues are large, then thfeatures vary significantly in both directions and considered aa good feature (corner-like).

Traditional Corner detection flow chart

Fig. 9 shows a general flow chart of a corner detection process

Fig. 9 Corner detection flow chart [25]

2.4 Contour detection

The reason to perform contour detection in general, is significantly reduce the amount of data in an image bdetecting edges in a robust manner. Edge detection defines thboundaries between detected regions in an image. Thinformation needed for higher level image processing. Ahmaand Choi (1999) [26], and Ziou and Tabbone (1998) [27] havdiscussed different types of edge detection method. Our met

od is based on algorithm developed by Canny (1986) [28] sin





4/6

it produce smooth and continuous edges despite sensitive tonoise pixels and higher in computation-time [29].

As depicted in Fig. 10, there are several stages in this algo-rithm as stated in [28].

Fig. 10 Traditional Canny edge detection Algorithm [30]

3 APPROACH TO MARKER-LESS SQUARE-ROI

RECOGNITION

As demonstrated in Fig. 6, marker-based detection algorithm

needs to go through stage (d) and (e) respectively in order toimplant an object for the detected marker. These two steps arethe main inspiration for our proposed method. By combiningContour-Corner approach we believe that we can somehowreplace the needs to have a physical marker in the desiredenvironment. Instead of using a physical marker, we willdefine a Region of Interest (ROI) as a markerless marker, calledSquare-ROI. This Square-ROI need to be hand-drawn manuallyby the user (see Fig. 13). The reason we chose a square isbecause squares naturally produces 4 possible points and these4 points are needed to calculate the pose camera estimation forvisualization rendering purposes. Another reason is that, theorientation of the points can be estimated as intersections of

edge lines. To enhance and to combine the current contour andcorner detection approach, we proposed smoothing and adap-tive thresholding techniques to the input stream and then usesubpixel corner detection to obtain better and more accurate

interest point.

Fig. 11 Framework for Proposed system

As discussed in [31], a markerless recognition system needto fulfill some criteria as stated below:

Easy offline preparation

Automatic initializationAble to detect natural features reliablyReasonable computation timeAccurate augmentationWork in unconstraint environmentFlexible and adaptive to the application needs

3.1 Algorithm Architecture

As illustrated in Fig. 11, we can see that the process startedwith first getting an input from the real environment thru acamera as visual sensor. On receiving an input image, theproposed system finds and detects strong interest point fromthe ROI by applying enhanced contour-corner detection. Fromthe ROI, features such as number of corners and vertices can beextracted and later used to define a marker.

(a)

(b)

Fig. 12 Overview of the proposed method (a) Architecture (b) Marke

identification Illustration (i) draws detected contour (ii) draw detected corne(iii) Identify marker (iv) overlay an object [6]





5/6

4 RESULTS

4.1 Testing Setup

We present the results of our proposed method based on thetest conducted on a mannequin as an input (real environment)in our experiment.

A square is manually hand-drawn on top of the target area

as a region of interest (ROI) in order to place a non-printedmarker or we can say a markerless ROI (see Fig. 13 (a)). Therationale to manually hand drawn a square contour as ROI isto avoid the needs to prepare a printed marker and at the sametime to make our proposed system more flexible withunprepared environment. Thus, we can ensure that onlydesired area of the image is searched for features. As visualizedin Fig. 13 (a), our method will first capture a frame from avideo feed, binaries captured frame, find contour in squareshape, detect interest point and finally identify a marker.

We formulate that number of vertices detected on a contour(square) must be equal with the number of corners detected in

order to identify a marker and upon successfully identified ayellow circle will be overlaid on top of the target area (see Fig.13 (d)). Yellow circle used as a substitute for 3D breast object inthis test.

Fig. 13 Results of the proposed system: (a) Load camera and capture the

environment (b) Convert captured frame into grey-scale (c) Displaydetected corners and square (d) If marker detected (Overlay 2D Yellow

Circle) [6]

4.2 Execution

From our test, we found that there are three (3) factors couldcontribute or influence our proposed method success rate.

Square-ROI size Square-ROI line thickness and Square-ROI distance from camera (viewpoint)

In Table 1, it shows that, the size of Square-ROI used (witeither single or double smoothing) influence thDistance-to-detect features.

TABLE 1

Square-ROI size and Distance-to-detect

Square-ROI size Distance-to-detect Smoothing

17 x 16 (cm) Estimated at 77 (cm) Single17 x 16 (cm) Estimated at 40 (cm) Double

6 x 5 (cm) Estimated at 15 (cm) Single6 x 5 (cm) Estimated at 10 (cm) Double

5 CONCLUSIONBased on our experiment in section 4, the proposed techniqumanaged to do the followings:

Capture real scene through a camera.

Convert captured scene into a grey-scale image.

Detect four (4) vertices and four (4) corners.

Identify and verify a marker based on the extractefeatures

Overlay 2D object. (Yellow circle)

At the moment, proposed technique have the tendency detect unwanted contour and corner as shown in Figure 13 (and required bigger memory (RAM) size estimated at 16GB foa duration of 7 to 14.1 seconds real-time execution.

In the future, we would like to extend our technique properly visualize the coexistence of a real and synthetic 3

breast cancer model that sharing the same real environmenwith the aid of touchless hand and finger interaction to selethe Region of Interest(ROI) in real-time.

ACKNOWLEDGMENT

The authors wish to thank the GRAVSLAB research group ftheir support and advice. This research is supported by a gran(FRGS0295-SG-1/2011) from the Ministry of Higher Educatio(MOHE), Malaysia.

This paper is an extension of work originally reported in "The2013 2nd International Conference on Medical Information andBioengineering (ICMIB2013).REFERENCES

[1] P. Milgram, H. Takemura, A. Utsumi and F. Kishino, Augment

Reality: A Class of Displays on the Reality-Virtuality Continuum

Proceedings of SPIE, Vol. 2351, Telemanipulator and Telepresen

Technologies, Hari Das; Ed. 1994. Pp. 282292.

[2] R. Azuma, Y. Baillot, R. Behringer, S. Feiner , S. Julier, an

B. MacIntyre, Recent advances in augmented reality, IEE

Computer Graphics and Applications ;21(6):3447, 2001.





6/6

[3] S. Mann, Mediated Reality with implementations for everyday life,

Presence 2002, Teleoperators and Virtual Environments.

[4] D. Marr and H. K. Nishihara, Representation and recognition of

the spatial organization of three-dimensional shapes," Proc. R. Soc. of

Lond. B, 200, 269-294, 1978.

[5] M.Y.I. Idris, H. Arof, E.M. Tamil, N.M. Noor and Z.Razak, Review

of Feature Detection Techniques for Simulataneous Localization

and Mapping and System on Chip Approach, Inform. Technol.

Journal, 8(3), pp. 250-262, 2009.

[6] Y. Genc, S. Riedel, F. Souvannavong, C .Akinlar and N. Navab,

Markerless tracking for Augmented Reality: A learning-based

approach, Proc. of the International Symposium on Mixed and

Augmented Reality (ISMAR 02), pp. 295-304, August 2002.

[7] L. Vacchetti, V. Lepetit and P. Fua, Combining edge and texture

information for real-time accurate 3d camera tracking, In

International Symposium on Mixed and Augmented Reality

(ISMAR 04), pp. 48-57, 2004.

[8] A. I. Comport, E. Marchand, and F. Chaumette, A Real-Time Tracker

for Markerless Augmented Reality, In International Symposium on

Mixed and Augmented Reality (ISMAR 03), Tokyo, Japan,

September 2003.[9] J. R. Vallino, Interactive Augmented Reality, PhD Thesis,

University of Rochester, Rochester, NY. November 1998.

[10]J. Rekimoto, Matrix: A realtime object identification and registration

method for augmented reality, In 3rd Asia Pacific Computer Human

Interaction (APCHI98), pages 6368, 1998.

[11]H. Kato and M. Billinghurst, Marker tracking and HMD calibration

for a video-based augmented reality conferencerencing system, In

Proceedings of the 2nd IEEE and ACM intl. workshop on augmented

reality (IWAR99) (p. 85), Washington, DC, USA, 1999.

[12]M. Fiala, ARTag, a fiducial marker system using digital tech niques,

In Proceedings of the IEEE conference on computer vision and

pattern recognition (CVPR05), vol. 2, pp. 590596, Washington, DC,

USA. 2005. doi:10.1109/CVPR.2005.74[13]S. Sanni, Theory and applications of marker-based augmented

reality, Espoo 2012. VTT Science 3. 198 p. + app. 43 p.

[14]I. Skrypnyk and D. G. Lowe, Scene modelling, recognition and

tracking with invariant image features, In Proceedings of the 3rd

IEEE and ACM intl. symposium on mixed and augmented reality

(ISMAR04) (pp. 110119). 2004. doi:10.1109/ISMAR.

[15]G. Klein and D. Murray, Parallel tracking and mapping for small AR

workspaces, In Proceedings of the 6th IEEE and ACM Intl.

Symposium on Mixed and Augmented reality (ISMAR07), Nara,

Japan. 2007.

[16]T. Lee and T. Hllerer, Hybrid feature tracking and user interaction

for markerless augmented reality, In Proceedings of the IEEE virtual

reality conference (VR08) (pp. 145152). 2008. doi:10.1109/VR.2008.4480766

[17]F. Zhou, H. Duh and M. Billinghurst , Trends in augmented reality

tracking, interaction and display: A review of ten years of ismar , In

7th IEEE and ACM International Symposium on Mixed and

Augmented Reality (ISMAR08). 2008.

[18]D. Beier, R. Billert, B. Bruderlin, D. Stichling and B. Kleinjohann,

Marker-less vision based mobile augmented reality, Proc. tracking

of Japan, Oct 2003.

[19]G. Simon and M. Berger, Reconstructing while registering: a novel

approach for markerless augmented reality, Proc. of ISMAR,

Darmstadt, Germany. 2002.

[20]ARToolkit Library. Available

http://www.hitl.washington.edu/artoolkit/documentation/vision.h

tm. Last accessed 10 February 2013 at 10.46 pm.

[21]H. Moravec, Towards automatic visual obstacle avoidance, In

Proceedings of the International Joint Conference on Artificia

Intelligence, pp. 584. 1977.

[22]C. Harris and M. Stephens, A combined corner and edge detector,

Alvey Vision Conference, pp. 147151. 1988.[23]C. Tomasi and T. Kanade, Shape and motion from image streams : A

factorization method-part 3. Detection and Tracking of Poin

Features, Technical Report CMU-CS-91-132. 1994.

[24]D. Frolova and D. Simakov, Matching with Invariant Features,

Advanced Topics in Computer Vision, Spring 2004, Weizmann

Institute of Science. March 2004. Lecture slide.

[25]D. Parks and P. Gravel, Corner detection, Available

http://kiwi.cs.dal.ca/~dparks/CornerDetection/algorithms.htm.

Last accessed 5th March at 2.17 am.

[26]M. B. Ahmad and T. S. Choi, Local Threshold and Boolean Function

Based Edge Detection, IEEE Transactions on Consumer Electronics

vol. 45, no 3, August 1999.

[27]D. Ziou and S. Tabbone, Edge detection techniques: An overview,

International Journal of Pattern Recognition and Image Analysis

8(4):537559. 1998.

[28]J. Canny, A computational approach to edge detection, Pattern

Analysis and Machine Intelligence, IEEE Transactions on

PAMI-8(6):679698, Nov 1986.

[29]S.K. Mahendran, A Comparative Study on Edge Detection

Algorithms for Computer Aided Fracture Detection Systems ,

International Journal of Engineeirng and Innovative Technology

(IJEIT), vol. 2, no. 5, November 2012.

[30]L. Rechard, A. Bade, S. Sulaiman, & S. H. Tanalol, Using Compute

Vision Techniques to Recognize Hand-Drawn Square-ROI as a

Marker in Augmented Reality Application, In Proceedings of the

10th Annual Seminar on Science and Technology 2012 (SNT 2012)

p.212, 2012.[31]Y. Chunrong, A tracking by detection approach for robus

markerless tracking, Available

http://campar.in.tum.de/twiki/pub/Chair/IsmarIndustrialAugmen

tedReality2006/yuan.pdf

Rechard Lee holds a first degree in Computer Science from UniversityPutra Malaysia completed in 2000. He is currently pursuing his MSc degreeby research in Mathematics with Computer Graphics at University MalaysiaSabah. Current Research Group is GRAVSLAB (www.gravslab.com)

Dr. Abdullah Bade holds MSc degree by research in Computer Sciencefrom Universiti Teknologi Malaysia in 2003. He obtained his Doctoratedegree in Industrial Computing (Computer Graphics and Visualization) fromUniversiti Kebangsaan Malaysia in 2008. He has been actively involved in

research and manages to secure several research grants from Ministry oHigher Education and Ministry of Science and Technology Malaysia. Hisresearch interest lies particularly in developing optimized algorithm focollision detection between objects, Deformable Body Simulation, SeriousGames Simulation,Cloth Simulation and Crowd Simulation. Currently, he ispart of the School of Science and Technology, UMS and appointed as aSenior Lecturer and Head of Programmes for Mathematics withComputer Graphics. He spends most of his leisure time on listening sofmusic, surfing internet, reading and travel. Current Research Group isGRAVSLAB (www.gravslab.com)



http://www.hitl.washington.edu/artoolkit/documentation/vision.htmhttp://www.hitl.washington.edu/artoolkit/documentation/vision.htmhttp://kiwi.cs.dal.ca/~dparks/CornerDetection/algorithms.htmhttp://kiwi.cs.dal.ca/~dparks/CornerDetection/algorithms.htmhttp://www.hitl.washington.edu/artoolkit/documentation/vision.htmhttp://www.hitl.washington.edu/artoolkit/documentation/vision.htm

Documents

Real-Time Markerless Square-ROI Recognition based on Contour-Corner for Breast Augmentation