A Robust Heart Rate Detection using Smart-phone Video

Arpan PalTCS Innovation Lab

Tata Consultancy ServicesKolkata - 700091, West

Bengal, Indiaarpan.pal@tcs.com

Aniruddha SinhaTCS Innovation Lab

Tata Consultancy ServicesKolkata - 700091, West

Bengal, Indiaaniruddha.s@tcs.com

Anirban Dutta ChoudhuryTCS Innovation Lab

Tata Consultancy ServicesKolkata - 700091, West

Bengal, Indiaanirban.duttachoudhury

@tcs.comTanushyam

ChattopadyayTCS Innovation Lab

Tata Consultancy ServicesKolkata - 700091, West

Bengal, Indiat.chattopadhyay@tcs.com

AishwaryaVisvanathan

TCS Innovation LabTata Consultancy Services

Kolkata - 700091, WestBengal, India

aishwarya.visvanathan@tcs.com

ABSTRACTIn this paper, the authors have presented a smartphonebased robust heart rate measurement system. The systemrequires the user to place the tip of his/her index finger onthe lens of a smart phone camera, while the flash is on. Thecaptured video signal often contains noise generated due to(i) improper finger placement, (ii) imparting excessive pres-sure, which subsequently blocks normal blood circulationand (iii) movement of the fingertip. To mitigate the aboveissues, a two stage approach has been proposed. Firstly,the onset of good video signal is detected by formulating afinite state machine, which employs multiple window shorttime fast fourier transform. Only upon receiving sufficientacceptable video signal, the heart rate is computed. Resultsindicate that the proposed method has successfully identi-fied and rejected noisy video signal, resulting in avoidanceof erroneous output.

Categories and Subject DescriptorsG.4 [Application Software]: Robust System Deployment;H.1.2 [Smart phone based health care Systems]: Well-ness Mesurement; G.3 [System Implementation]: Robustrealization - Fast Fourier transforms (FFT)

General TermsSmartphone based system, Health care, Wellness

KeywordsHeart rate estimation, Photoplethysmogram, Unobtrusivesensing, Mobile sensing, Mobile computing

1. INTRODUCTIONThe proposed system draws inspiration from an old myth

“Prevention is better than cure”. Medical practitioners of-ten suggest that dire consequences of a major disease maybe avoided only if detected at an early stage. The insurancecompanies of the developed countries are also keen to mon-itor physiological parameters of people, round the clock, asthe early detection of abnormalities may potentially reducethe cost of treatment. It is now observed that the demo-graphic age is increasing in both developed and underdevel-oped countries and that the medical ailments are naturallyhigher in elderly people ( [13]). Their physical movement isoften restricted, thereby requiring 24x7 physiological moni-toring. Moreover, in the developing nations, the cost of regu-lar health check up is not affordable for a large section of thesociety. Lack of expert professionals who measure the healthconditions, makes the situation worse. Though there existsstand-alone devices measuring heart rate and other wellnessparameters, those devices incur additional hardware cost.

Hence we observe a need for unobstrusive easy-to-use ap-plication, running on a smart phone and estimating wellnessparameters. The usage of such application can be further ex-tended to fitness programs. E.g., during an exercise session,a person can instantly measure his/her heart rate by puttingthe tip of the index finger on the camera. Instant Heart Rate1 and Cardiograph 2 are popular examples of such smartphone applications. But they tend to output wrong resultsin the presence of weak and unusable data, generated dueto unwanted movement, placement and pressure of the fin-ger. In this paper, we have proposed a robust methodologyto detect and reject inadequate input signals and provide

1https://itunes.apple.com/app/instant-heart-rate-measure/id395042892?mt=82https://itunes.apple.com/us/app/cardiograph-heart-rate-meter/id584393206?mt=12

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.MobileHealth’13, July 29, 2013, Bangalore, India.Copyright 2013 ACM 978-1-4503-2207-2/13/07 ...$15.00.

43

user feedback instead of computing wrong output. Thoughthe proposed method can only provide an estimated figureof such parameter, sufficient accuracy has been obtained forpreventive health care and wellness applications.

This paper is organized as follows. First we talk aboutthe state of the art of the presented approach in Section2. Section 3 focusses on the drawback of the related workand formulates the problem statement. Then we discuss theproposed methodology in Section 4. The experimental setupused and the results generated are presented in Section 5 andSection 6, respectively. Finally, Section 7 draws conclusionand indicates future work.

2. RELATED WORKSMobile phone based health care is a very recent research

topic having vast business application. A number of relatedresearch papers and patents are present in the literature( [1]- [4], [5], [12]). In these methods, Photoplethysmogra-phy (PPG) is used as a non-invasive method for estimatingpulse ( [6]- [8]). PPG is a well-known technique for opticallycapturing the perfusion of blood flow to the periphery andthus detecting the cardiac cycle, based on the principle thata human heart periodically pumps blood during each cardiaccycle. Initial limitations of using restricted light condition( [6], [7]) were overcome using ambient light ( [8]). All thesemethods are based on manual segmentation and heuristics.In [9], authors have used brightness component of the videoto detect PPG, without addressing the problem of improperfinger placement. In [10], authors have used green compo-nent of the pixels as they have used the transmitted light forprocessing. But it is not possible to get the transmitted lightif a normal smart phone camera is to be used to capture thevideo. Inventors have used the parameters like Stroke Vol-ume, Cardiac Output, Pre-Ejection Period, Peak Ejectionrate, Time to Peak Ejection Rate ( [5]). They measured thepulse rate and ECG waveform using the time difference andpeak values obtained from PPG, using standard formulas.In [4] and [11], authors have used wavelet to estimate theheart rate.

3. PROBLEM STATEMENTThus from the study of the above mentioned papers and

patents we can conclude that the common method of esti-mating the heart rate involves three major steps:

1. Video is captured from the mobile phone

2. Average of pixel values were computed on a color plane.

3. Frequency analysis (Fast Fourier Transform or Dis-crete Wavelet Transform) or Time domain Filteringis applied to estimate the heart rate.

Most of the works have used a window cropped from thecentral part of the video to (i) decrease computational com-plexity as compared to full frame processing, (ii) get rid ofthe weak and noisy signal present in the border of capturedvideo.

However, in real system implementation, a major limita-tion was observed in the robustness aspect. Most of thosemethods assume that the captured video signal is free fromnoise. As a result we obtain an erroneous result wheneveruser does not place his/her finger properly.

Figure 1: Waveform of average values of raw R com-ponent

The Fig. 1 shows the average values of the red (R) com-ponent for each video frame. The waveform indicates thatthe first few seconds of the waveform is very noisy due tofinger placement jitter and flash auto-focus adjustment. It ispossible to skip the initial noisy part manually and extractthe clean segment as shown in Fig. 2 where the zoomedwaveform is shown from 15 seconds onwards. This segmentcan be used to analyse and compute the heart rate correctly.However, for the scenario as shown in Fig. 3, the intermedi-ate noisy segments between 25 seconds to 35 seconds cannotbe handled using the above method. Thus the manual skipmakes the process nonadaptive and hence, not desirable asa real-time smart phone application.

Figure 2: Zoomed view of the good segment of Rcomponent

Figure 3: Waveform of average values of raw R com-ponent - Intermediate noisy segments

Though [11] tries to address this robustness issue, it doesso at the cost of high computational complexity. If imple-mented on a smart phone, it will drain precious battery life.So in this paper we have presented a Finite State Machine(FSM) based approach which estimates the signal qualityprior to heart rate estimation, in order to reject the noisyunusable video segments.

VideoAverage

RedPixels

RejectionFSM

Compute

Rejected

HeartRate

Figure 4: Improvement over state-of-the-art

44

The overview of our approach is quite similar to the methoddescribed in state of the art, except the introduction of theRejection FSM block. As shown in Fig. 4, this block signi-fies our contribution in this paper to make the system robustand free from erroneous output.

4. METHODOLOGYThe overview of the deployed system has been presented

in Figure 4. The detailed method is described as follows.

4.1 Average Red PixelsAverage Red Pixels block consists of the following steps:

• Capture a video frame of height h and width w fromthe camera. Then crop the video by considering thepixels having coordinate (x, y) satisfying the followingcriteria

w

2− 16 ≤ x ≤ w

2+ 16 (1)

h

2− 16 ≤ y ≤ h

2+ 16 (2)

• Compute the average R component of all the pixels ofcropped video frame

µR =1

1024

∑x,y

Rx,y∀x, y (3)

• We define a frame to be good frame if µR ≥ σ whereσ is a heuristically obtained threshold value.

• A final frequency analysis is performed to determinethe heart rate of the user.

The following section describes the FSM of the proposedsystem.

4.2 Rejection FSMA variant of a particular form of FSM, commonly known

as Mealy Machine is used in our deployment. In Mealy ma-chine, output values are determined by the current state aswell as the input values. But in the proposed FSM, instead,the next state is governed by current state and the input val-ues. As shown in Fig. 5, the states of the mealy machinesare as follows.

4.2.1 Signal AcquiringThis is the start state. In this state, the mealy machine

waits to get consecutive input video frames for which µR ≥σ. As shown in Fig. 5, once sufficient number (α) of good redframes are received, the FSM proceeds to Frequency Analysisstate.

4.2.2 Frequency AnalysisAs the name suggests, in this state, FSM analyzes the

frequency of µR. To ensure robust input data, µR valuesare stored for 64 consecutive video frames. From each setof 64 values, as shown in algorithm 1, the inputs of shorttime window analysis are built. For each such window, thesystem calculates the following parameters:

1. 64BlockCount : This is a counter indicating the numberof consecutive 64 point blocks generated.

Algorithm 1 Frequency Analysis

1: procedure Frequency Analysis(µR[N ])2: loop 64 frame blocks3: 64BlockCount+ +4: [pLoc, ampRange]← Analyze(µR[N ])5: . see Algorithm 26: pLocMin← min(pLocMin, pLoc)7: pLocMax← max(pLocMax, pLoc)8: pLocRange← pLocMax− pLocMin9: if (pLocRange ≤ ζ &

10: ampRange ≥ δ) then11: if 64BlockCount ≥ ε then12: nextState← Compute13: else14: nextState← FrequencyAnalysis15: end if16: else17: nextState← Rejection18: end if19: end loop20: return nextState21: end procedure

Algorithm 2 Window Analysis

1: procedure Analyze(µR[N ])

2: meanV alue←∑Ni=0

µR[i]N

3: for all i do . get rid of dc value4: normR[i]← µR[i]−meanV alue5: end for6: fOp[N ][2]← FFT (normR[]); . execute FFT7: for all i do8: absFFT ←

√(fOp[i][0])2 + (fOp[i][1])2

9: end for10: filtStart← ceil(lowCutoff × (N)/fps);11: filtEnd← ceil(highCutoff × (N)/fps);12: for i = filtStart→ filtEnd do13: minFFT ← min(absFFT [i],minA)14: maxFFT ← max(absFFT [i],maxA)15: end for16: maxIndex← findIndex(maxFFT )17: pLoc← maxIndex+ filtStart18: ampRange← |maxFFT −minFFT |19: return pLoc, ampRange20: end procedure

2. ampRange: ampRange stores the range of the outputof the current 64-point-FFT. When a user puts exces-sive pressure on the camera lens, the variation of redpixels reduces due to blockage of normal blood circu-lation. If ampRange doesn’t cross an empirically setthreshold value δ, FSM goes to Rejection state.

3. pLocRange: The peak in each 64-point-FFT indicatesthe dominating frequency for those frames. To ensureconsistent input data, the position of this peak needsto be within a small range for all consecutive 64-pointblocks. pLocRange stores the range of the position of allthe peaks received. If pLocRange crosses a empiricallyset threshold value (ζ), FSM goes to Rejection state.

Once 64BlockCount is more than a threshold (ε), FSMgoes in to the Compute state.

45

AverageRed

Pixels

SignalAcquiring

FrequencyAnalysis

Compute

Rejection

newFrame /µR ≥ σ

newFrame &conscR < α /µR ≥ σ &conscR+ +

newFrame /µR ≤ σ

newFrame &conscR ≥ α /NIL

newFrame &framesReceived < 64 /µR ≥ σ &framesReceived+ + newFrame &

framesReceived == 64 /µR ≥ σ &64BlockCount+ + &framesReceived = 0

newFrame /µR ≤ σ |ampRange ≤ δ |pLocRange ≥ ζ

(start again)

64BlockCount ≥ ε

Parameter/Event Explanation

newFrame New video frame receivedµR Average red pixel value for ROIconscR Consecutive number of video frames which satisfies the condition µR ≥ σframesReceived Video frames received in state Frequency Analysis64BlockCount Consecutive blocks of 64 frames received in state Frequency AnalysisampRange Range of the output of the previous 64-point-FFTpLocRange Range of the position of the peaks received for 64BlockCount 64 frames

Figure 5: Mealy machine of PPG based heart rate detection

4.3 ComputeIf the process continues to gather more data, better will

be the accuracy of calculation. However, to get result withina reasonable time, our methodology limits it to a 512-point-FFT. The peak frequency in 512 point FFT is detected asthe final heart rate.

4.4 RejectionRejection state indicates the receipt of poor video signal.

Upon reaching this state, FSM is reset by setting the AverageRed Pixels as the next state.

5. EXPERIMENTAL SETUPThis section presents the experimental setup for demon-

stration of automatic triggering of the proposed algorithm.The experiments are performed using an iPhone on eightusers. The ground truth for the heart rate is measured us-ing a Pulse Oximeter from Etcomm 3. An application isdeveloped on iPhone which initially puts the flash on, fol-lowed by the execution of the algorithm. The user puts thetip of the index finger touching the iPhone surface, such thatit covers both the camera and the adjacent flash, as shownin Fig. 6(a). A 352x288 video stream is captured from the

3http://www.etcomm.cn/en/products_hc-801.html

(a) Placement of fin-gertip

(b) Data Acquir-ing

(c) Result Com-puted

Figure 6: Display of heart-rate on phone

camera at 29.97 frames per second. A segment of 32x32pixels from the center of each video frame is fed into theproposed algorithm. Once our application starts to receivegood signal (Fig. 6(b)), a periodic beep sound is providedfor the user. During data capture, if the signal quality fallsbelow a certain limit, the mealy machine goes back to theAverage Red Pixels state and the beep sound stops, indicat-ing the user to reposition his/her finger. At the end of asuccessful computation, the result is displayed on the screen(Fig. 6(c)).

46

Table 1: Comparison of visual inspection with auto-matic detection (in seconds)

User 1 2 3 4 5 6Visual 8 16 5 7 13 9

InspectionAutomatic 16 21.5 10.5 12.5 18 18.5

6. RESULTSThe experimentation is done in two stages. Firstly, the

video data is logged on iPhone and the algorithm is devel-oped and tested on Matlab, in PC environment. Later thesame algorithm is ported on iPhone and the accuracy ofrealtime heart-rate is measured against the ground truth.

6.1 Automatic Trigger of AlgorithmThe proposed algorithm calculates the signal to noise ratio

in the Red (R) component of the captured video to deter-mine the usable segments of the signal suitable for compu-tation of the heart rate. By introducing our algorithm, weare able to automatically detect the onset of good signal. Acomparison of visual inspection of good periodic signal andautomatic onset detection using the proposed methodology(see section 4) is shown in Table 1. As expected, duringmultiple run of the application, due to unpredictable hu-man behaviour, the acceptable periodic signal starts fromdifferent timings. It can also be observed that automatictriggering point follows the visual inspection. Hence, wecan conclude that the proposed algorithm does a better jobthan any fixed initial offset, which either lead to a unneces-sary large delay or forced to process a noisy signal.

The heart rate is detected based on a peak in the fre-quency domain analysis. A sample frequency response isshown in Fig. 7 where the green waveform is for the au-tomatic triggered case and red is for the manual skip of 20sec.

Figure 7: Frequency Response of 512 point FFTanalysis

6.2 Accuracy of Heart-rate DetectionA comparison of the heart rate detection using Mealy ma-

chine (with FSM) and without using Mealy Machine ((w/oFSM)) against the ground truth for 8 different users is shownin Table 2. The Peak Signal-to-noise ratio (PSNR) in dB aregiven for both the approaches. It can be seen that for thefirst 6 users the detected heart rate is quite close to theground truth.For the last two users, the signal quality is extremely pooras the users failed to place finger properly. Hence the trig-gering has not taken place and the calculation never reachedthe compute state (see Fig. 4). Instead, it is rejected. ThePSNR and heart rate are not calculated for the proposed al-gorithm. Whereas, in the conventional approach, i.e. with-

out (w/o) Mealy machine, the heart rate calculation givescompletely wrong results.

Table 2: Effect of Rejection FSMUser 1 2 3 4 5 6 7 8

Actual 54 66 84 106 80 105 105 80Heart Rate

(per minute)

Heart Rate 53 63 84 98 88 102 38 42w/o FSM

(per minute)Heart Rate 53 63 84 98 88 102 × ×With FSM

(per minute)

PSNR (dB) 18 14 17 24 18 16 13 9w/o FSM

PSNR (dB) 22 15 19 21 14 16 × ×with FSM

6.3 AnalysisWe critically analyze few cases where there are motion

artifacts and improper placements of finger. As shown inFig. 8(a), the waveforms for R component is quite clean fora good data capture for User 1. Whereas in Fig. 8(b), it is abit noisy for the one with motion artifacts for User 2. Thisis also reflected in the PSNR in Table 2.

(a) Good capture

(b) Motion artifacts

(c) Improper placement of finger

Figure 8: Comparison of the average raw R compo-nent

In an extreme case, in Fig. 8(c), where the placement ofthe finger is not proper for user 7, the waveform is very noisyand lacks in periodicity. Hence, PSNR and heart rate arenot computed for this case (Table 2).

7. CONCLUSION AND FUTURE WORKIn this paper, we have presented a smart phone based

system, that can robustly estimate the heart rate of a user,covering the camera with his/her finger tip. The system de-tects whether the finger is placed properly over the cameraand if required, guides the user to properly place it over thecamera of the phone. We have formulated a FSM in order

47

to estimate the finger placement, by judging the quality ofthe input video signal. The proposed system thus reduceserrors in heart rate estimation and produces more consistentresults. Even though the system was first implemented andprototyped on iPhone, it is now being ported to low-costAndroid phones. A much bigger sample data collection isalso planned for the purpose of exhaustive signal analysis.Finally, a user study is also planned to study the usability ofthe application and efficacy of the user feedback mechanism,based on the proposed signal quality assessment methodol-ogy.

8. REFERENCES[1] C. G. Scully, J. Lee, J. Meyer, A. M. Gorbach, Y

Mendelson, and K. H. Chon”Physiological Parameter

Monitoring from Optical Recordings With a MobilePhone, IEEE Transactions on Biomedical Engineering,vol. 59, no. 2, pp.303-306 ,2012.

[2] E. Jonathan and M. J. Leahy, Cellular phone-basedphotoplethysmographic imaging, Journal onBiophotonics, vol. 4, no. 5, pp. 293-296, 2011.

[3] Y. K. Lee, J. Jo, and H. S. Shin, Development andEvaluation of a Wristwatch-type PPG Array SensorModule,, pp. 168-171, ICCE, 2011.

[4] P. Prakash, M. Sareen, R. Anand, Abhinav and S.Anand, Application of wavelets based multiresolutionanalysis to detect relevant points of interest fromfinger-tip photoplethesmography and pressure signal fromthe radial artery, IEEE International BiomedicalEngineering Conference(CIBEC), 2008.

[5] M. A. Sackner, D. M. Inman, Method and System forExtracting Cardiac Parameters from PlethysmographicSignals, US Patent number: 6783498, Issue date: Aug31, 2004.

[6] F. P. Wieringa, F. Mastik, and A. F. van der Steen,Contactless multiple wavelength photoplethysmographicimaging: a first step towards SpO2 camera technology,Ann. Biomed. Eng. 33(8), Page(s) 1034-1041, 2005.

[7] K. Humphreys, T. Ward, and C. Markham, Noncontactsimultaneous dual wavelength photoplethysmography: afurther step toward noncontact pulse oximetry, Rev. Sci.Instrum. 78(4), 2007.

[8] W. Verkruysse, L. O. Svaasand, and J. S. Nelson,Remote plethysmographic imaging using ambient light,Opt. Express 16(26), 2008.

[9] P. Pelegris ,K. Banitsas ,T. Orbach , and K. Marias, ANovel Method to Detect Heart Beat Rate Using a MobilePhone, 32nd Annual International Conference of theIEEE EMBS, Argentina, 2010.

[10] Y. Maeda, M. Sekine,T. Tamura, The Advantages ofWearable Green Reflected Photoplethysmography,Springer J.Med.Syst. (2010).

[11] J. N. Watson, P. S. Addison, Detecting a signal qualitydecrease in a measurement system, US Patentpublication number: US 2010/0079279 A1, Filing date:Sep 30, 2008.

[12] W. Z. Khan, Y. Xiang, M. Y. Aalsalem and Q.Arshad, Mobile Phone Sensing Systems: A Survey, IEEECommunications Surveys and Tutorials, vol. 15, issue.1,pp. 402-427, 2012.

[13] Cohen, Joel E, Human population: the next halfcentury, Science, vol. 302, no. 5648, pp. 1172-1175, 2003.

48