Proceedings of the 9th International Conference on Info~ation Technology andApplications in Biomedicine, ITAB 2009, Larnaca, cyprus, 5-7 November 2009

A Simple Algorithm to Monitor DR for Real Time TreatmentApplications

Banitsas K., Pelegris P., Orbach T., Cavouras D., Sidiropoulos K. and Kostopoulos S.

Abstract- As the demand for effective and reliable telecaresystems increases rapidly over the last years, novel ideasapplied on existing consumer products enables the developmentof innovative solutions that could enhance the user's wellbeing.In this research, we are going to demonstrate the potential of asystem that enables users to monitor their own heart beat ratein real time and use specialised software for personal healthcoaching. In this paper we will explain and demonstrate how toextract heart beat rate information from a user using thecamera of a commercially available mobile phone which willenable us to supply the users of the system with vitalinformation and utilize interactive tools useful for personalhealth coaching. Our industrial partner Health Smart Limitedhave filed a patent [1] for this application, they retain the fullintellectual property of this project.

Index Terms- HBR, camera, mobile, cbt, health coaching

I. INTRODUCTION

THE rapid evolution of technology has given us with theopportunity to gradually provide access of cost effective

sophisticated electronic devices to the public, currently mostpeople can buy a smart phone with a camera for a reasonablecost. From an engineering point of view this createsadditional possibilities of application deployment andservice delivery: a modem mobile phone gives you enoughprocessing power in conjunction with mobility while it canalso be used as a media platform for increased userinteraction [3],[8].

Our industrial collaborator, Health-Smart Limited hassubstantial experience in self-care ICT for prevention andcontrolling Long Term Conditions (LTC) includingcardiovascular and psychological conditions, one of theirpurpose is to empower patients to assess their state of bodyand mind and train them to improve their health and preventLong Term Conditions (LTC), by using inexpensive friendlyconsumer ICT and sensors [2]. We have developed ademonstration based on an invention - patent they had filed

Manuscript submitted 21th August 2009.K.Banitsas .P. Pelegris and K. Sidiropoulos are with the Electrical and

Computer Engineering Department, Brunei University, West London,Uxbridge, Middlesex, UB8 3PH, UK (phone: +44(0) I895266886;(e­mail:{konstantinos.banitsas,panagiotis.pelegris,konstantinos.sidiropoulos}@brunel.ac.uk).

D. Cavouras and S. Kostopoulos are with the Department of MedicalInstruments Technology, Technological Educational Institute of Athens,Ag. Spyridonos Street, Egaleo, 122 10, Greece. (email{cavouras,skostopoulos}@teiath.gr).

978-1-4244-5379-5/09/$26.00 ©2009 IEEE

[1] to extract heart rate (HR) information from a user, usingmobile phone with a camera (without any other sensor).

The importance of this invention is that it enables anyoneto use a standard mobile phone with a camera as a heart ratemonitor and health coach. In this research the deliveredservice is actually pre-emptive in character and aims inenhancing the well-being of the users by providing themwith simple tools, methods, information regarding the stateof their heart and coach them on how to improve theirphysical and emotional state [9]. In the proposed system weare examining the usage of an embedded camera in a mobilephone or PDA as means of measuring the heart rate (HR).This will help us provide near real time feedback to the usersand will assist them in using self coaching software toenhance their well-being.

Although there are numerous researchers active onworking with mobile phone and PDA's implementinghealthcare systems the vast majority of them use the mobilephone and PDA as a processing and media interface unit (orcommunication device) and never as a sensor by itself. Ineach and every case the user has to wear some kind of sensorunit that enables a device to monitor the user's physiologicalsigns. The less intrusive a solution is the more chances it hasto be accepted by people [8]. A touch screen mobile phone /PDA with a simple interface that has no need for extrawearable sensors might prove to be more successful.

The proposed system is compatible with most of thecommercially available modem mobile phones with a videocamera. Basically the camera is used to capture a video fromthe users' fmger and also as a media interface that willgenerate the feedback and provide them with useful selfcoaching advice that can improve their well-being [5]. Inorder to do that they just need to press their fingertip on thecamera while the PDA device captures a short length video.When this is over the device will analyze their heart beatinformation and provide feedback for the actual heart beatrate. Following that personal health coaching software takesover to utilize this information for an actual pre-emptivehealth service. For example software can teach the usershow to breathe properly and help them regulate their heartrate. The process can be repeated over a period of time so

T. Orbach is with Health-Smart Limited, 77b Fleet Road, Hampstead,London, NW3 2QU, UK (e-mail: Tuvi@health-smart.co.uk, www.health­smart.co.uk).

that users can check their progress and see immediate resultson heart beat self regulation. In the future it will be possibleto analyze the HR continuously and provide feedback andcoaching to the users in near real-time.

The key challenge here is to engage the user in a processthat will enhance his well-being providing him withpersonalised health coaching, delivering effective pre­emptive health services. In order to succeed in that you needan easy to use interface, a system that requires no extraeffort like wearing sensor units and a way to provide quickfeedback for the users enabling them to check the effects oftheir "treatment" right away [4]. The user friendliness of selfmonitoring equipment is crucial for the adoption and successof it. Familiarising users with such solutions will help themadapt to more complex systems in the near future. Theongoing standardisation on telecare and Telehealth will soonstart producing practical health monitoring systems based onmultiple sensor readings.

This research was designed to be carried out in twostages: firstly take the input from a PDA and do thecalculations on a PC and finally implement the whole systemin real time in a mobile device. In this paper we demonstratethe process in a desktop environment using video files takenfrom a PDA Mobile phone (Eten Glofish X800) as input.

II . METHODOLOGY

The invention is based on two interesting phenomena:I) Every heart beat creates a wave of blood that reach thecapillaries in the tip of the finger; when the capillaries arefull of blood less light can pass through them. So thechanges in the amount and the colours of light which ispassing through the finger can represent the changes in theshape of the pulse and its timing (the HR).2) Normally to take a photo or a video we need focus andsome distance between the lens and the object. However weare interested only in the quantity of light, therefore there isno need of focus, and it is possible to cover the object lens ofthe camera with the finger tip as shown in Figure 1 . As longas there is a source of light (which can be natural ambientlight or artificial light) which can pass through the finger, itis possible to view the pulsation of the blood as changes inthe amount of light in the video.

The major advantage of this invention is that the user doesnot need any sensors, does not require any external hardwarewhatsoever, it does not require even to focus; one justtouches and cover the lens of the camera with their finger tipand with the right software they can view their HR and learnto improve their health and fitness. It was worth to examineif one can discern any information about the pulse wavewhen placing the finger directly on top of a camera withstrong flash light right next to it.

Figure 1: Pressing the finger on top ofthe camera while the fla sh isturned on

The initial findings where very encouraging, as it isillustrated in Figure 2 the red portion of the finger is clearlyvisible and easily distinguishable from the rest of the picture,unfortunately in this paper we cannot visually demonstratethe fluctuation from the pulse wave as it only appears onvideo but it is important to note that the resolution providesenough information to estimate the heart rate as long as thefinger is kept right on top of the camera.

Figure 2: Sampleframesfromfinger images on PDA cameraarrows denote the pulsating regions.

In order to develop the system we used both desktop andmobile environment applications for testing. The PDAMobile phone used was an Eten Glofish X800. Theresolution used was 320 x 240 pixels with an effective framecapture rate of 25 frames / sec. Although these are not thecurrent highest available characteristics for a mobile phonewith a camera they proved to be adequate for ourexperiments [6].

240 - -- -- --------' -- - - - - - - - - - -

500 600Frames

40030020010020

0'--- ----'.,,----- --,-'-:--- ....,-L,- - --'-- - -'-- --'

---_. _...-------~ ---- -------~ ----------

50 -----------t-----------t- - ------- --------- -- \-----------\---------­

40 ------- --- r---------r---- ----------- ------1-----------1--- ---- -- -

30 ------ -----f--- --------f-- --------- ---- -------j--------- --j----------, ,, ,, ,, ,, ,, ,

Normal ised Amp8Or-- ----r- --.------,--- --,-- - -,---,

70 -------- --- --- -------

200 250Vertical Position

15010050180

0'---- -::'::-- -----'.,,----- - --'-:-------'-- - ----'

220 --- - - - - - - - -- ~- - - - - - - - - - - - ­

200 - - - ---------~-- - ---- -- - - - -

Red Channel

2601 - -----;-----;= :::::;::::= = =:== = =:== =-1

Figure 3: Frame Profile with virtual vertical axis along the middle Figure 4: Crude pulse signal, illustrating unexpected dive

The system's behaviour was initially simulated on Matlabusing videos captured from the camera on the PDA as input.For each video the frames were extracted and a profile wascreated for every frame. We deliberately scanned the profileof a virtual vertical axis right in the middle of the framecalculating light variation along this axis. The sameprocedure was repeated for every frame. However this isonly one method to calculate the changes in amount of light.

Figure 4 represents a crude preview of the pulsevariation, notice how the signal dives at approximatelyframe 210 indicating a possible sudden movement of thefinger or an external source of noise, however thisunforeseen development does not considerably affect theoutcome of the calculation as the information we are lookingfor is basically contained in the frequency of the signalrather than on its amplitude .

500 600Frames

400300

-. _. ••• _; - - - - -- -- t -, ., ., ., . ,, . ,-------_..-._---- ---- .-----------.----------

200100-3

0'--- ----'.,----- --,-'-:--- ----'-- - --'-- - ---'---- ----'

, ,, ,

-0.5 - - - -- --- -f- - --------f, ,, ,, ,, ,-1 ------ - --- -; -- ---- - - -- - ;· ,· ,· , , , ,

-1.5 - --------- - : - -- - - - - - - - -~ ----------: - - - - - - - - - - - : - - - - - --- - - - : - - -- - -----• , I , ,• , I , •• I I , •

-2 -----------f------ ---- -f-- ---- -----i---------- -f--- --------f------- ---• • I ,• • I ,, • I ,

-25 - - - - - - - - - -- r--- -- -- -- --r -----------i-----------i------- -- -- [-- -- ------

1 -----------f------- ----f-, ,, ,, ,, ,0.5 -- -- -- - ' --- -- ~ - -

Normalised Output1.5 r-----,-----r- - -.------,-----,------,

The next step is to normalise the crude signal intosomething more meaningful, which is easily achievableusing smooth differentiation . From this plot it is simple toget the number of peaks, which divided by the running timeof the video will give us the estimation for the heart beatrate. In the given example the result was 61.3 beats / min asshown in Figure 5.

Figure 5: Normalized signal

Our technique is based on determining red channelvariation along a given area of the image. This represents agood estimation on how much light is getting through. Theprocess is followed for all frames and amplitude isnormalised giving the crude pulse signal as shown in Figure4.

An alternative technique would be to process data from allchannels in RGB mode, or take multiple measurements fromdifferent areas of the image to reinforce our result accuracy.However in this work we mostly aim in demonstrating theefficiency of the algorithm in this application as the finetuning is a process that will be explored in the developmentof the final product.

Another issue is that we can directly scan the red channelwithout any other type of conversion ensuring that our inputdata goes through a minimum of transformation andcompression stages giving us the maximum possibleefficient information.

Moreover since our target platform is a mobile devicewith limited resources both in terms of processing power andmemory, it is reasonable to get the input algorithm fairlysimple as long as the results appear to be highly accurate.

III. RESULTS

After the simulation stage, we needed to evaluateaccuracy of the algorithm using real examples and think ofways to improve performance. A custom softwareapplication was built to process the frames from the video,scan for colour variations, locate peaks and calculate beatsper minute.

It appears that videos of 20 seconds produce results witharound 91% average accuracy, with minimum 85% andmaximum 99% compared to the measured HBR values.Samples of 14 - 16 and 18 seconds have an averageaccuracy of 89%, the minimum was 81% at 14 secondsamples and the maximum was 100% at 16 second samples.The results are summarised on Table.1.

Fnlmes 400 16 Sec ID FaslMode

Frames : 400

The samples of 12 and 10 seconds have only 87% and86% accuracy respectively, that is more or less anticipatedsince when calculations are based on data from 10 seconds,any error would appear to be six times higher when you areestimating beats per minute. A simple optimisation that wewill incorporate in future versions is the sharp definition ofstart and end time based on the actual beginning or ending ofa pulse. This would directly result in an improvement of upto 12 pulses for a 10 second video in some cases, this changemight make it possible for shorter streams to give oneenough data for reliable analysis.

The flexibility of the system lets the user decide on therequired resolution, for example a sample of 20 secondswould produce results with a resolution of 3 beats. If a userfor example has 25 beats in 20 seconds based on that samplehe has 75 beats in 60 seconds. If the same user had 26 beatsin 20 seconds then the software would calculate 78 beats in60 seconds. Similarly samples of 10 seconds would have aresolution of 6 beats, being far less reliable since only theinteger portion of the beats is used in the calculations. Thesampling time effectively gives you the flexibility to definethe desired resolution based on your needs.

Figure 6 illustrates the demonstrating application in actionwhile calculating results based on a 16 second sample; thisprocess doesn't take more than 3 seconds on a 2.4 GHzprocessor. The frames are extracted from video files intoPNG images using FFMPEG and are later processed by theapplication to produce the actual results. During testing wefound out that PNG images give much finer detail when itcomes colour compression compared to the standard IPGimages. The sources files from the X800 are in 3GP formatand frames are extracted directly in order to avoid thedetrimental effects of subsequent recompression stages intodifferent compression schemes.

PlAses : 75

Figure 6: Sample desktop application to calculate pulses

In the above example the actual beats per minute where76 and the application calculated 75 in a 16 second sample.In this case the 16 second result was more accuratecompared to the 20 second result which was 78 but in bothcases the results are within the given resolution for 20second samples which are 3 beats per minute.

Table . IMeasured and Calculated HBR for different sampling times

I s.mplelH BR I Me.sure<l ICOIC.1Osec COIC.1Zsec Ie. C.l4Sec COIC.l6Sec ICOIC.lesec ICOIC.lOsec

69 60 65 68 67 63 8378 60 60 64 63 66 66

3 78 66 65 68 67 70 724 75 72 70 68 67 70 695 74 60 60 60 60 63 696 79 72 75 77 78 76 787 72 60 60 60 63 63 638 75 66 65 64 63 63 669 75 66 70 77 75 73 7210 78 66 65 68 67 66 69

Table.l summarizes the heart beat rate calculation resultsfor 10 random video samples. This is still an early stage onthe development of the algorithm and it appears that a fewminor adjustments will have significant impact.

Table .2Relative HBR accuracy percentages

Sample/Accuracy 10sec 12sec 14sec 16sec 18sec 20sec1 0.87 0.94 0.99 0.97 0.91 0.912 0.77 0.77 0.82 0.81 0.85 0.853 0.85 0.83 0.87 0.86 0.90 0.924 0.96 0.93 0.91 0.89 0.93 0.925 0.81 0.81 0.81 0.81 0.85 0.936 0.91 0.95 0.97 0.99 0.96 0.997 0.83 0.83 0.83 0.86 0.86 0.888 0.86 0 .87 0.85 0.84 0.84 0.889 0.88 0.93 0.97 1.00 0.97 0.96

10 0.85 0.83 0.87 0.86 0.85 0.88

IV. DISCUSSION

As life expectancy increases over the last decades so doesthe burden on the health system supporting people withchronic conditions, the only way out of this is implementingtelecare solutions that will manage to increase the quality ofdelivered health care while maintaining low installation andrunning costs [7].

It is not only that health services delivery will change, butalso that the nature of the service itself will change shiftingfrom re-active treatment of conditions to pre-emptive healthcare. Avoiding health risks would be more efficient thansustaining patient's with chronic conditions that could havebeen avoided [2]. This is where health monitoring andcognitive therapy comes into play, providing the users withinformation on how to avoid getting a health condition ratherthan focusing on how to treat it.

Biomedical data acquisition will also playa major role onapplications where monitoring physical condition andalertness will be critical in respect to safety minimizingassociated risks in sensitive and high risk areas likeaerospace and aviation. Cross disciplinary synergies in thepast have produced important advancements in the area ofunobtrusive multi sensorial data acquisition systems [10].Such platforms can provide invaluable input to self coachinghealth systems that will aim in enhancing well being.

The challenge remains both for researchers and theindustry to agree on standards, engineer innovative solutionsthat will cover the needs while minimising the cost and taketelecare to the next level. This application is a subsystem ofa telecare platform that is being developed in collaborationwith our industrial partner Health-Smart Limited, as a stand­alone system though it may serve as a media platform todeliver cognitive therapy treatment and positive psychologyadvice and practice.

Future work includes migrating the application andalgorithm on mobile devices and including it in a largersoftware package that will deliver health-smart solutions.This includes further optimisation of the code to improveaccuracy and increase tolerance to noise sources andcombine the end result with interactive "game-style"applications that will run on a PDA in the context ofpersonal health coaching.

V. CONCLUSION

In this work demonstrated the efficiency of an innovativealgorithm to detect heart beats per minute using inexpensivewidely available hardware. The objective of this research isto provide a simple and fairly accurate tool for pre-emptivehealth services to be delivered on patients using mobiledevices.

Furthermore real time feedback information for a user will

greatly improve his general feeling about using new modemtechnology in order to maintain well-being or improvehealth through specialised self coaching.

The invention will be used in the near future as asubsystem for a self help application that aims to help usersreduce their stress levels and improve their well-being.

ACKNOWLEDGMENT

We would like to express our appreciation to Health­Smart Limited for its important contribution to this paperand the general research that resulted in a patent application[1].

REFERENCES

[1] PCT patent application No. PCT/GB2009/050989 Blood Analysis ­Health-Smart Ltd.

[2] T. Orbach and 1. Vasquez, "Self-care and the need for interactiveICT", Journal of holistic healthcare, vol. 6, pp. 35-39, August 2009.

[3] R. Gururajan, S. Murugesan and 1. Soar, "Bringing MobileTechnologies in Support of Healthcare Recommendations for aHealthy Beginning and Growth," Cutter IT Journal Article, Aug.2005.

[4] A. Duckworth, T. Steen and M. Seligman, "Positive Psychology inClinical Practice," Annual Review ofClinical Psychology, vol. 1, pp.629 - 651, April 2005.

[5] M. Seligman, T. Steen, N. Park and C. Peterson, "Positive PsychologyProgress : Empirical Validation of Interventions", AmericanPsychologist, July 2005.

[6] M. Fischer, Y. Yang Lim, E. Lawrence and L. Ganguli, "ReMoteCare:Health Monitoring with Streaming Video," in 7th InternationalConference on Mobile Business, July 2008.

[7] V. Shnayder, B. Chen, K. Lorincz, T. Fulford-Jones and M.Welsh,"Sensor Networks for Medical Care," Technical Report TR-08-05,Division of Engineering and Applied Sciences, Harvard University,2005.

[8] A. Pantelopoulos and N. Bourbakis, "A Survey on WearableBiosensor Systems for Health Monitoring," 30th Annual InternationalIEEE EMBS Conference, Vancouver, British Columbia, Canada,August 20-24, 2008.

[9] H. Jang, S. Kim and C. Bae, "Personalized Healthcare throughIntelligent Gadgets," 30th Annual International IEEE EMBSConference, Vancouver, British Columbia, Canada, August 20-24,2008.

[10] A. Astaras, PD. Bamidis, C. Kourtidou-Papadeli and N. Maglaveras,"Biomedical real-time monitoring in restricted and safety-criticalenvironments", Hippocratia, vol. 12, suppl. 1, pp 10 - 14, August2008.