˘ ˇ ˆ˙ - University of Washington · narrowband light source; (2) a comparison of the hemoglobin estimation with a clinical blood draw and another noninvasive optical hemoglobin

��

Edward Jay Wang1, William Li1, Doug Hawkins4, Terry Gernsheimer3, Colette Norby-Slycord3, Shwetak N. Patel1,2

1Electrical Engineering, 2Computer Science & Engineering, 3Division of Hematology

Seattle, WA 98195 {ejaywang1, wli11, bldbuddy3, coletten3,

shwetak1,2}@uw.edu

4Seattle Children’s Hospital4800 Sand Point Way NE

Seattle, WA 98105 [email protected]

��We present HemaApp, a smartphone application that noninvasively monitors blood hemoglobin concentration using the smartphone’s camera and various lighting sources. Hemoglobin measurement is a standard clinical tool commonly used for screening anemia and assessing a patient’s response to iron supplement treatments. Given a light source shining through a patient’s finger, we perform a chromatic analysis, analyzing the color of their blood to estimate hemoglobin level. We evaluate HemaApp on 31 patients ranging from 6 – 77 years of age, yielding a 0.82 rank order correlation with the gold standard blood test. In screening for anemia, HemaApp achieve a sensitivity and precision of 85.7% and 76.5%. Both the regression and classification performance compares favorably with our control, an FDA-approved noninvasive hemoglobin measurement device. We also evaluate and discuss the effect of using different kinds of lighting sources.

�� !��Hemoglobin; Mobile Health; Photoplethysmography; Anemia; Camera; Blood Screening

��"�� !��H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

#��$%&��#$Smartphone-based medical devices have grown increasingly common for heartrate monitoring [8,9,14], pulmonology [15], sleep monitoring [17], point-of-care diagnostics [23], and a variety of telemedicine systems [10,27]. They demonstrate how the computing, telemetry, and sensing capability of modern smartphones make them excellent

candidates for clinical and remote healthcare platforms. A number of applications have even leveraged only the existing sensors on a smartphone to achieve results similar to those from medical devices (e.g., the microphone [15] and camera [7–9,14]). However, there has been little work on performing non-invasive blood screening on a smartphone. Most related work focuses on blood oxygen saturation. In this paper, we present the design and evaluation of a noninvasive technique for sensing hemoglobin levels using an unmodified smartphone camera. By leveraging the absorption properties of hemoglobin and blood plasma at multiple wavelengths of light, our system, HemaApp, can measure hemoglobin concentrations with no augmentation of the phone hardware and can be further improved with minimal augmentation.

Hemoglobin is the protein molecule in the blood that carries oxygen throughout the body. Conceptually, the measure of hemoglobin is a representation of the oxygen carrying capacity of the patient’s blood. This is distinct from oxygen saturation, which measures the oxygen carrying efficiency of

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected] '16, September 12 - 16, 2016, Heidelberg, GermanyCopyright is held by the owner/author(s). Publication rights licensed to ACM.ACM 978-1-4503-4461-6/16/09...$15.00DOI: http://dx.doi.org/10.1145/2971648.2971653

Figure 1: HemaApp is a smartphone application that noninvasively estimates blood hemoglobin concentration usinga smartphone camera. Analysis of the color of the blood in a user’s finger yields an estimate of the user’s hemoglobin level.We evaluated the system using the smartphone’s LED flash andincandescent light bulbs as illuminating sources.

593

UBICOMP '16, SEPTEMBER 12–16, 2016, HEIDELBERG, GERMANY

the blood. Currently, options to monitor hemoglobin include a full blood analysis using a complete blood count (CBC) [6], blood analysis using a finger prick [20], or optical measurements through the finger using a specialized finger probe [1,16,18,24]. Noninvasive measurement is desirable for both sanitation and ease of use when measuring frequently because it avoids puncturing the skin. A blood-screening tool based on unmodified smartphones has the added advantage of being easily deployable and enables previously unconsidered treatment management options given the lack of such technology. For example, HemaApp can help community health workers in developing countries screen for iron-deficient anemia caused by malnutrition. A major barrier for these health workers is the number of medical tools they need to transport with them on foot for each test. A smartphone is now standard equipment used for telemetry of records; as such, a smartphone-based solution helps reduce the burden on these workers and reduces the cost of equipment. Beyond improved deployability in remote areas, the reuse of smartphones also aids people being treated for cases of anemia and need to monitor their condition at home. Often, these patients are treated with iron supplements and return to the hospital for a blood test every few weeks to ensure their treatment is effective. A smartphone hemoglobin test is convenient for at-home monitoring and does not require a patient to purchase a specialized blood testing device that costs hundreds to thousands of dollars. This allows both the patient and the doctor to track the effectiveness of these treatments much more easily and frequently. This can help early detection of any ineffective treatment leading to complications. Lastly, the ability to measure hemoglobin noninvasively is useful for measuring hemoglobin more frequently even in a clinical setting. For example, sickle cell patients often suffer from extreme anemia and need frequent monitoring. However, due to treatment to suppress their production of sickled cells, their veins are often hardened, making blood draw difficult [4].

Our system enables hemoglobin measurement through a chromatic analysis of the blood at the user’s fingertip by measuring the absorption properties of the blood at different wavelengths of light. This is achieved by using the RGB camera with different light sources illuminating the fingertip. We evaluate three different hardware embodiments that vary the level of hardware augmentation necessary (Figure 2): (1) white flash + infrared emitter, (2) white flash + infrared emitter + incandescent lamp, and (3) white flash + custom infrared led array. Most smartphone cameras have a white LED, and many smartphones have started to introduce IR emitters for autofocus, potentially making the first embodiment augmentation free. The second embodiment requires a low overhead modification by introducing an incandescent lamp for an additional IR spectrum. The third embodiment requires the most hardware augmentation by introducing a custom array of LEDs as a phone case.

Figure 2: Three embodiments of HemaApp in increasing hardware augmentation.

Our research team includes experienced hematologists and pediatricians who help conduct a clinical study with 31 patients. The study population includes a wide spread of hemoglobin levels (8 – 16 g/dL), an age spread from 6 to 77 years old, and included skin tones ranging from pale to dark skin. This demographic spread is in-line with initial feasibility studies done for other noninvasive hemoglobin measurement devices [13,19]. We find that the combination of incandescent and LEDs gives the best estimation accuracy of ±1.26 g/dL (R = 0.82). As a comparison, the Masimo Pronto, an FDA-approved optical hemoglobin measurement device, achieves an accuracy of ±1.28 g/dL (R = 0.81).

The main contribution of this paper is a demonstration that a smartphone camera can be used in estimating hemoglobin concentrations noninvasively with minimal augmentation with a few commonly available light sources. This contribution comes in three parts: (1) an algorithm to estimate hemoglobin using a combination of broadband and narrowband light source; (2) a comparison of the hemoglobin estimation with a clinical blood draw and another noninvasive optical hemoglobin measurement system; and (3) a performance analysis of hemoglobin estimation when different combinations of light sources are used.

�'(��'%)$��The ubiquity of smartphones has lowered the access barrier to computing, telemetry, and the sensors needed to enable medical and health care data for non-clinical settings. Prior work in this new space has inspired the development and design of HemaApp. We divide our summary of related works into three sections: smartphone-based sensing, camera as a hematology sensor, and hemoglobin measurement.

�� There are currently three ways in which medical devices integrate smartphones into their systems: with a self-contained attachment that digitally communicates with the phone, with an attachment that supports and uses the phone’s built-in sensors for measurement, and with an app using only the phone’s built-in hardware to process the specimen.

Self-contained attachments are medical devices that have all the necessary hardware to perform the medical analysis, but use the phone as a hub to control the device, receive data, store data, and potentially send data to an outside database for access by hospitals and doctors [25]. This model is gaining popularity in almost all commercial products for personal health monitoring systems like blood pressure

594


monitors [30] and glucose implants [31], often using Bluetooth as the communication method.

Another class of devices uses supplemental hardware in addition to the smartphone. One such example is an attachment that mechanically couples a stethoscope head to a 3D printed attachment that directs the sound to the smartphone’s microphone [23]. More complex attachments, such as an optics system that illuminates a flow cytometer test strip, can be attached to a smartphone camera to perform point of care blood tests [28]. The optics system helps to illuminate the strip with the needed wavelength, the test strip provides the needed chemical reaction to bind to the proteins and compounds of interest in the blood specimen, and the smartphone camera acts as the sensor. The advantage of such systems is that attachments can be tailored to specific tasks (e.g., chemical reaction, illumination, mechanical amplification), and the sensor of the smartphone can be reused for various purposes and recorded onto the phone.

The last class of systems uses only what is on the smartphone. Researchers have explored various ways to monitor respiratory ailments through spirometry [15] and auscultation [23] using the smartphone’s microphone. Sleep disorders like sleep apnea have been successfully monitored using a combination of speaker and microphone on a smartphone as a sonar to detect breathing [17]. The camera has been used to map melanoma and skin lesion [29]. By only using typical hardware on a smartphone, such systems have an advantage in terms of being immediately deployable through software, making these systems attractive for mass deployment due to their ease of installation and low cost.

�� Most relevant to our work is the use of smartphone cameras to noninvasively perform blood analysis. In particular, this type of work relies on the fact that various compounds in the blood produce coloration differences under different concentrations. BiliCam is a system that measures bilirubin, a compound in the blood that is important to monitor during the first few days of a newborn’s life [7]. This system measures the yellowness of the skin in order to compute the concentration of bilirubin, using a sheet of white paper for color balancing. Similar to this work is a student project called Eyenaemia [22], which uses a color calibration card placed next to the eyelid of a patient, and measures the redness of the underside of their eyelid when flipped over. In this case, the concentration of hemoglobin, which is responsible for the red coloration of the blood, affects the red absorption. Our system follows a similar intuition as Eyenaemia, but measures the coloration of the blood at the fingertip. More importantly, Eyenaemia also only focuses on detecting the risk of anemia, but cannot estimate the actual concentration of hemoglobin as in the case of our approach. Various research projects have also explored the use of smartphone cameras with their IR filter removed to perform 1 http://www.hemocue.com/en/solutions/hematology

pulse oximetry, using a custom set of red and infrared LEDs to illuminate the finger tip of a patient and the camera to measure the pulse at the finger under different illuminations [8,11,12,21]. Our work builds upon the insights from these works in the use of smartphone cameras to detect infrared light, but focuses on not modifying the smartphone camera; instead, we explore multiple ways to illuminate the fingertip, ranging from lighting sources commonly available on smartphones to incandescent lighting for a broad spectrum.

�� "��Unlike pulse oximetry, which compares the ratio of oxygenated hemoglobin to deoxygenated hemoglobin, hemoglobin measurement needs to measure all forms of hemoglobin in reference to the blood plasma.

Clinically adopted methods to measure hemoglobin are currently restricted to blood tests. In a lab test, a technician draws about 3mL of blood for a complete blood count (CBC). In a CBC, information about red blood cells (RBC), white blood cells (WBC), and platelets are measured electronically. The hemoglobin concentration is measured optically after the blood is mixed with a chemical agent that changes the solution’s density proportionally to the concentration of hemoglobin. For tests that only require a hemoglobin concentration result, such as those for screening during blood donations, a point of care (PoC) device is often used instead of the more time consuming CBC. An example of a PoC is the HemoCue1 device. The HemoCue requires a finger prick to draw a small amount of blood; using a microcuvette that draws blood and mixes it with a chemical reagent on a test panel, a result can be determined in a minute. This method has been shown to have a rank order of 0.89 at a mean accuracy of ±0.5 g/dL when compared to results from a CBC test [5,20].

Most recently, noninvasive measurement of the hemoglobin concentration through optical measurement of the blood at the fingertip has been developed. Alam et al. has created a finger probe with 6 LEDs that cover multiple wavelengths in the red to IR spectrum (630, 660, 680, 770, 880 and 1300nm) to measure the hemoglobin to water ratio in blood plasma [1,2]. Kraitl et al. conducted a 41 people study using a three LED (670, 810, 1300nm) finger probe, and found similar success. Timm et al. based their technique off of Kraitl et al.’s system to create a device called OxyTrue Hb®. A commercially available system developed by Masimo is similar to Alam et al.’s system, but instead incorporates 7 LEDs and uses a proprietary algorithm [16]. Validation of this system has shown a mean accuracy of ± 1.0 g/dL [16,18]. We draw from these existing works for noninvasive hemoglobinometry in our design of the lighting sources we incorporated in our study. The aim for our system is to provide a similar level of accuracy, but reusing the hardware of a smartphone to make the system more easily deployable.

595

SESSION: HEALTH I

HemaApp is the only noninvasive hemoglobin measurement system that leverages the camera of the smartphone to perform similar measurements as the Masimo Pronto and other finger probe systems, utilizing non-LED sources to remove the need for custom hardware.

�'"��**Figure 3 shows the HemaApp system, consisting of hardware components that record data which are processed by an algorithm to calculate the hemoglobin concentration. The following hardware section describes the choice of light wavelengths and the phone application that controls the lights and records the light reflected from the finger to the camera. The data obtained by the phone app are a series of videos for each light source. The algorithm section describes the process of converting the series of video recordings to RGB time series waveform and the physical intuition behind the machine learning features used in the hemoglobin estimation.

��!��Due to the limited access to low level hardware control of the phone, we choose to develop a hardware add-on to provide the lights for hemachrome analysis. However, unlike other noninvasive optical hemoglobin measurement systems, we limit the lights we choose to ones that can be commonly found on smartphones today and in typical hardware stores. Furthermore, we make no modifications to the phone’s camera itself, unlike prior work’s use of phone cameras for pulse oximetry, which requires the removal of the IR filter of the camera. Instead, we rely on the front facing camera, which tends to have weaker IR cut off than the back facing cameras, making it more suitable for our application.

�� We address a major limitation of using a smartphone camera for doing hemoglobin measurement; a lack in sensitivity to wavelengths above 1000nm. First, we use only IR absorption

2 http://www.tech-led.com/product/smc/

below 1000nm. Water begins to have a response above 940nm and has an initial relative maximum at 970nm. Second, we not only measure the water content in the plasma, but also the proteins that make up about 10% of the blood plasma by volume[14] as a proxy for capturing the plasma volume to compare against hemoglobin concentration.

This is accomplished by leveraging the blue absorption of the plasma. By illuminating the finger with a white LED (which contains a strong blue component) and an IR light at 970nm, the system is capable of capturing the plasma response. We also included another IR LED (880nm) to help capture the different absorption between the various forms of hemoglobin. Both the 970 and 880nm LED are in the range of IR autofocus LEDs equipped in the current generation of smartphones. Our IR LEDs are sourced from the Marubeni SMC2 series.

Even though some new smartphones are beginning to be equipped with IR LEDs, most smartphones only have a white LED. To address this issue, we look towards a commonly available IR source, the incandescent light bulb. Incandescent light bulbs typically have strong IR light emission in the NIR range. Our system uses a candelabra 6W incandescent light bulb that can be found in hardware stores.

The LEDs are placed in a ring around the phone camera, while the incandescent light bulb was placed about 3 inchesabove the finger.

��For the HemaApp prototype, we use a Nexus 5 with an unmodified front-facing camera. For typical optical hemoglobin measurement, each lighting condition is cycled through at a high rate repeatedly to measure the response at each wavelength pseudo-simultaneously. In this way, the lighting conditions are all measured for the same pulse. However, due to the limited sampling rate of the camera and

Figure 3: System overview of HemaApp, which consists of the phone and lighting hardware, data, and algorithm. The HemaAppapplication communicates with an LED attachment. After a user places their finger over the LED and camera, multiple light sources are cycled through. A video is recorded for each light source. The algorithm then extracts the R, G, and B time series waveform foreach video by averaging each RGB channel independently for each frame. The algorithm then extracts machine learning features including peak and trough measurements for each light source, and also interaction terms between light sources. Finally, a SVM based regression is applied to estimate the hemoglobin concentration for the user.

596


the inability to synchronize the lighting circuit with the camera’s frame refresh, it is not feasible to measure all wavelengths of light at the same time without considerable access to low level hardware control. We discuss the tradeoffs of multiplexing and series testing in the discussion and make suggestions on how multiplexing can be achieved based on our own experiments. For our prototype, each lighting condition shines continuously for 15 seconds and cycles to the next light with an assumption that hemoglobin concentration and average blood flow does not change significantly during the course of the test, which takes a few minutes.

The mobile app is built on the Android platform with the Android Camera 2 API, which allows for full control over the white balancing and exposure. This is important because infrared is generally considered an unwanted spectral response that camera applications will detect and rebalance settings to avoid. Prior work has found that certain white balancing settings, such as the “incandescent mode” can be used to avoid the rebalancing [9]. The Camera 2 API allows for full control over the exposure, white balance, and sensor sensitivity. The hardware gains are manually set for each RGB channel using presets that are empirically found for each lighting condition such that the three channels reported a similar level of light. This is necessary because the red channel typically has a much stronger response due to the red blood under the white and incandescent lights. If left to a flat white balance, the auto exposure will be set to the red channel, leaving the G and B channels highly underexposed. Finally, exposure is set using the camera API’s auto-exposure settings. Once the image is auto-exposed, the exposure is locked and a 15-second video is recorded for each of the lighting conditions sequentially.

%��Each light source is cycled through one after another and a 15 second RGB video is recorded for each light source. The exposure, frame rate, white balance gain, and ISO settings are recorded for calibration. At an average resting heart rate of about 75 beats per minute (BPM), around 15-20 pulses were captured for each light source.

�� The algorithm section first explains hemachrome analysis, which is the blood color analysis we base our algorithmic implementation on. We then detail the two stages of the implementation. First, the video processing step extracts the pulsatile signal recorded in each video. The second stage is a feature generation step that involves combining the intensity values extracted from each wavelength’s video. These features are then used with an SVM to train regression models based on the ground truth blood test hemoglobin.

�� Hemachrome analysis is the study of blood coloration to analyze the components in the blood. HemaApp aims to measure the concentration of hemoglobin as compared to the concentration of plasma in the blood.

The Beer-Lambert law states that the absorption of light is proportional to the concentration and thickness of the medium, given by:

�� (1)

where �� is the incident light intensity, � is the absorption coefficient, ��is the concentration, and � is the thickness of the medium that the light travels through. When the finger is illuminated with a single wavelength of light, the measured intensity �� represents the absorption due to tissues, hemoglobin, and plasma:

�� ! �"#�� $% ! �&'(�)(�� *+�� , (2)

where - is the wavelength of the incident light. To obtain the ratio of �./� and �012342�, it is necessary to eliminate the attenuation of the intensity signal due to finger tissue. This is accomplished by measuring the temporal change of the measured intensity as the thickness of the arteries oscillate with respect to the heartbeat.

Figure 4: Light absorbed by living tissue. Adapted Web09[26]. The absorption of light changes due to the change in volume of blood when the heart pulses.

The change in arterial thickness 5� affects only the path length for ./ and 012342. By measuring the ratio of the maximum and minimum intensity of the light received, the effect of the tissue is removed:

�6��7��

��8�9:��

�;��"#�� $% ! �&'(�)(�� *+�� , (3)

Where the ratio of intensities can then be expressed as:

�<�� =>�?�2@�-

�ABCDEF�- �./�- ./ G� H �?12342�- 012342 G� (4)

The measured ratio of maximum and minimum values of intensity thus provide a measure of absorption due to the different components of blood. In theory, a system can use empirically measured absorption coefficients for each

597

SESSION: HEALTH I

compound at a specific wavelength of light to predict the hemoglobin concentration using Equation 4. By measuring the response at multiple wavelengths of light, multiple �<�� can be calculated. Ratios of �<�� across wavelengths over determine the ambiguity of G� allowing an estimate of �./� to be made. However, factors such as the distribution of the emitter, sensitivity of the sensor, and complex reflection properties of tissue make this infeasible. In our case, the incandescent and white LED are very broadband emitters that produce measurements that are poorly modeled by Equation 4. Using machine learned regressions to estimate the �$%�� and �*+�� for each lighting source, we overcome our reliance on predetermined absorption coefficients at specific wavelengths for blood.

�� For each lighting condition, we extract the intensity of the peak and troughs for each pulse. The center section, measuring half the width and half the height of the image, is cropped and an average intensity for each channel is calculated. This is done because the image from the center of the image is the most consistent and stable.

The signal processing technique that extracts �<

1)� High pass filter with a cut off at 0.5Hz. This is to remove the fluctuations due to breathing, which are around 0.2-0.3 Hz.

2)� Apply an FFT on the filtered waveform and extract the dominant peak. The dominant peak gives an estimate of the heart rate.

3)� Using the heart rate as a threshold between successive peaks, false peaks due to the dicrotic notch and the diastolic peak are avoided. The threshold is set at ¾ of the heart rate to avoid missing actual peaks.

4)� Map the index of peaks and troughs onto the unfiltered signal. We need the original magnitudes of the peaks and

troughs to calculate �< =>I&�(J

I�KL�MN

5) Calculate �< for each peak

Figure 5: To calculate the absorption change due to blood, a peak detection algorithm is applied on the temporal signal. An FFT is used to estimate the heart rate. Using the estimated heart rate as the minimum spacing between two peaks, the peaks and troughs of the heart beat signal is extracted.

�� In order to estimate the absorption coefficients at the broadband wavelengths used by our system, the HemaApp system cannot rely on the basic derivation of multi-wavelength hemachrome analysis. The following is a list of the features. The first three features are derived directly from the Beer-Lambert equations in hemachrome analysis. The fourth and fifth features aim to capture nonlinear interactions between wavelengths.

OPQRS OTU : The baseline intensity due to tissue.

OPQRS V OWXYZ[\ O]U : The amplitude of the pulsatile absorption.

^_OPQRS�`

OWXYZ[\�`

Oa : The adjusted amplitude of absorption that

eliminates the effect of tissue.

Oa�]U�bc� bd, O]U�èf O]U�`S : A pairwise ratio of pulsatile absorptions between all wavelengths.

Oa�]UTU�bc� bd, Oa�è Oa�`S

OTU�è OTU�`S

: Absorption difference across

wavelengths, adjusted with baseline.

��A separate SVM regression (SVR) model is trained for each embodiment of our proposed system: (1) white + 970 nm LED, (2) white + 970 nm LED + incandescent light, (3) white + 970 nm + 880 nm LED + incandescent light. We will refer to each embodiment as EMB#. The regressions are made based on ground truth values obtained from a blood test. During development, we find that a linear regression do not produce as good of a result as an SVR, but provides good insight into feature significance. As such, the linear regression is used to help in the feature selection process, as it is easier to interpret the resultant model, but the SVR is used to produce a model for evaluation by employing the features chosen through the linear regression. Table 1 displays the feature list for the three embodiments:

Table 1: Features used for training in each embodiment.

EMB1 EMB2 EMB3

�g��hFi, �<�g��ji� hFk, �<�g��ji� llm,

�g��hFk, �<�g��ji� nom, �<�g��jk� hFi,

�g��nom, �<�g��jk� hFk, �<�g��hFi� hFk,

�<�g��hFi� hFk, �<�g�p��ji� �jk, �<�g�p��ji� �jk,

�<�g�p��hFi� nom, �<�g�p��ji� hFk, �<�g�p��ji� nom,

�<�g�p��hFk� nom, �<�g�p��hFi� llm,

�<�g�p��hFk� llm,

The SVM model is tested using a leave-one-subject out validation. The training is done using the MATLAB implementation of SVM regression with a Gaussian kernel

598


with default parameters, to avoid overfitting due to parameter tuning.

%��$(('��#$+,�(#%��#$To evaluate and inform the design of HemaApp, we conduct a clinical study with three groups of people: healthy students and staff of the university, in-patients at a children’s cancer and transfusion clinic, and in-patients at an adult cancer and bone marrow transplant clinic. Data collections at these sites provide a diverse dataset paired with ground-truth hemoglobin concentration from CBC tests. We note that typical initial feasibility studies for hemoglobin measurement systems involve 10 – 60 people and typically cover population ranges of 8 – 16g/dL hemoglobin concentration [13,19]. In order to recruit patients from beyond these ranges, a system must be validated enough for hospital use during surgical procedures, as people who are well below the 8 g/dL range are likely in need of a transfusion or are already in surgery. These initial feasibility studies are then followed by a series of larger follow up clinical validations in more diverse populations and repeated measurements per person during surgeries such as urologic procedures where hemoglobin concentrations change dramatically [16,19,24]. We base our study design to reflect the feasibility studies of these previous works. Our study include 31 patients in a range of 8.3 g/dL to 15.8 g/dL.

Table 2: Demographic information of subjects.

Participant Demographics (N = 31)

Age (years) 6 – 77 (µ=31, σ=17.5)

Hemoglobin (g/dL) 8.3 – 15.8(µ=12.1, σ=2.2)

Gender Ratio 15 male:16 female

Reported Ethnicity (n, %)

East Asian: 7

South East Asian/Indian: 6

Latino: 3

White: 14

Mixed: 1

Each patient’s data set include a series of videos measuring the absorption change under multiple wavelengths of light. We collect videos within 24 hours of the ground truth CBC blood draw to ensure that hemoglobin measures are as accurate as possible. Within a day, hemoglobin concentration is typically stable within 0.5 g/dL[3]. Patients who have hemoglobin transfusion or heavy bleeding between the study and the blood draw were excluded.

The in-patient clinics are centered on cancer patients, in particular leukemia and bone marrow transplant patients, because these patients tend to be chronically anemic. As such, these patients often have CBCs done as part of their clinical care. By including both children and adult clinics, the study cover a wide range of age and hemoglobin variations, allowing us to validate on different groups of people. Our study population include about 1/3 pale skintone, 1/3 sepia – light brown skintone, and 1/3 dark brown skintone. Due to a low population of people with black skintones in the city we conduct our study, we are not able to include those with very dark pigmentation in our study. However, it should be noted that pigmentation on the underside of the hand tends to be much lighter in comparison to the backside of the hand, as such, it is likely that the effects of very dark pigmentation is covered well enough by our inclusion of dark brown skintones. In our follow up study, we will be expanding to other hospitals nationally and internationally to have a wider spread of pigmentation to validate these claims.

%�� *��In order to evaluate the effects of camera hardware and lighting conditions, we built a setup that allows us to efficiently cycle through all the combinations for every subject in our clinical validation. The setup is an acrylic box that contains a Bluetooth-enabled microcontroller that controls each of the light sources. The top of the box has a 6W incandescent light and a white piece of card stock with a hole cut in the middle. A Nexus 5 smartphone is placed in the box with the camera pointing up to the ceiling of the box. The LED circuit is then placed over the camera. The box is clear except for the portion holding the electronics, which lets ambient light shine through. The box also has a black cover that is used to block out ambient light.

Figure 6: The experimental set up included an acrylic box housing a 6W incandescent light bulb of 3000k color temperature and a Nexus 5 smartphone with an LED array attached to its front facing camera. The array is equipped with white, 880nm, and 970nm LEDs. The participant places their finger over the camera while covering the LEDs.

The subject places the fingertip of the ring finger on their non-dominant hand on the camera. The subject is asked to sit still and not speak during the test to reduce movement. Each

599

SESSION: HEALTH I

lighting condition is then cycled through in the following order: incandescent, white, 970 nm, 880 nm.

The study consists of taking a series of videos of the participant’s finger under various lighting conditions. Tests are done during the day, with no particular control over the ambient lighting conditions. An optical hemoglobin measurement is obtained using the FDA cleared Masimo Pronto 7 right before the recordings with the HemaApp system. The CBC blood test is used as ground-truth data and the optical Hb is used as a source of comparison to a specialized noninvasive device.

�'�&(��

*�� (��The top half of Figure 7 shows the predictions for each HemaApp embodiment comparing to the ground truth CBC, calculated using a leave-one-subject out cross validation. The bottom half of Figure 7 shows the corresponding modified Bland-Altman plot, where the residuals (HemaApp – CBC) are plotted against the CBC hemoglobin. For comparison, the results of the Masimo Pronto are included in each plot.

Results: HemaApp’s hemoglobin estimations correlate with the CBC’s predictions with a rank order correlation of 0.69, 0.74, and 0.82 with a mean error of 1.56 g/dL, 1.44 g/dL, and 1.26 g/dL respectively for each embodiment. The results of the Pronto are also compared to the CBC, which yields a rank order correlation of 0.81 with a mean error of 1.28 g/dL.

Implications: The improvement from EMB1 to EMB2 shows that simply supplementing the smartphone with an incandescent light both improves the correlation and decreases the error significantly. With the addition of an extra IR source, thus including both a 970 nm and 880 nm IR LED, the performance of HemaApp is comparable to the results of the Masimo Pronto.

A Wilcoxon signed rank test and an F-test of the residual variances fails to show statistically significant differences between the EMB3 predictions, Pronto’s Hb predictions, and ground truth CBC values (p>0.05). An N-way ANOVA on the residual magnitude (|HemaApp EMB3 – CBC|) also did not reveal statistically significant effects on the residual magnitude due to age and race (p>0.05).

�� To evaluate how well HemaApp can be used as a tool for screening for anemia, the results of the regression are classified into two groups: anemic and normal. The classification is based on the average expected hemoglobin for each age group and gender.

Figure 8 shows the classification results plotted against age. A different reference line is shown for male and female above the age of 16 as female tends to have lower hemoglobin than males. Those classified correctly are shown in gray. Higher sensitivity signifies better ability to detect the presence of anemia. Higher specificity signifies better ability to detect normal levels of hemoglobin. Table 3 breaks down

Figure 7: (Top) A comparison between predicted hemoglobin levels and ground truth hemoglobin level. (Bottom) A Bland-Altmanplot showing residuals of predicted hemoglobin level against the ground truth hemoglobin. The ±1.96SD is shown for each plot. Embodiments 1, 2, and 3 shown from left to right. Blue represents the HemaApp results and orange represents the Pronto results.

600


the results by sensitivity and specificity for each embodiment and the Pronto. The 31 participants included 14 with anemia. The Pronto did not produce a hemoglobin measurement for one of the anemic participant due to low signal quality, thus the sensitivity results are for 13.

Results: HemaApp’s classification results for each embodiment have higher sensitivity (78.6%, 85.7%, and 85.7%) as compared to the Masimo Pronto (69.3%). In comparison, the Pronto performed better in specificity, 88.2% compared to HemaApp (70.6%, 70.6%, and 76.5%).

Figure 8: Anemia classification of HemaApp using typical hemoglobin reference ranges for each age and gender groups. Colored lines map incorrectly classified points to ground truth classification and hemoglobin predictions.

Table 3: Anemia classification results for each embodiment.

Sensitivity Specificity

EMB 1 78.6% (11/14) 70.6% (12/17)

EMB 2 85.7% (12/14) 70.6% (12/17)

EMB 3 85.7% (12/14) 76.5% (13/17)

Pronto 69.3% (9/13) 88.2% (15/17)

Implications: The results are in favor of HemaApp as a useful screener for anemia. This can be seen in the sensitivity of each embodiment being near or above 80%. The Pronto on the other hand performed better in terms of specificity, at just under 90%. Comparing HemaApp to Pronto, the two systems are calibrated differently. HemaApp is tuned to be more cautious in order to reduce missing patients who are anemic, but at the cost of lower specificity. We believe this is a safer choice for use cases such as community health workers screening in the field, as missing an anemic diagnosis is a higher risk. In comparison, the Pronto’s higher specificity is useful for situations such as in home monitoring. In situations where the patient is measuring frequently, having too many false alarms will result in

unnecessary patient burden and hospitals getting frequent calls from unnecessarily worried patients.

%#��&��#$Our analysis of HemaApp shows that with some augmentation to the current smartphone hardware, our smartphone-based hemoglobin system compares favorably with the Pronto predictions. HemaApp cannot replace the CBC blood test, but can be used as an effective screener to determine whether further blood testing is necessary. The availability of the smartphone makes HemaApp a good candidate for at-home monitoring of chronically anemic patients, and its low cost can help equip community health workers in low resource areas screen for iron deficient anemia. The findings show that by using another ubiquitous item, the incandescent light bulb, the accuracy of HemaApp can be significantly improved. Although such light bulbs are beginning to be phased out in developed countries for more efficient alternatives, they are still easily obtainable as decorative lights in typical hardware stores.

( � �� -��)��.The current results are based on data collected on a Nexus 5 device and using only one brand of incandescent light bulb. As such, for the system to be more accessible, it needs to function on various types of devices and supplemental lighting. Different brands and models of phone use different cameras, lenses, and filters. The strength of the IR filter will strongly influence how strong the IR LEDs need to be for penetration. The necessary adjustments based on these factors need to be investigated before the results of HemaApp can be generalized to different hardwares.

The incandescent light’s properties also present another variable for future exploration. The current light bulb is a 3000K bulb, which places its peak wavelength at about 1000 nm, making it ideal for the detection of water. Another variable is the effect of the age of the light bulb. As light bulbs become less efficient, their emitted spectrum changes. It is possible that in order for the incandescent light bulb method to remain accurate over time, a calibration stage would be necessary after some time. Another issue with using an incandescent light bulb is ambient lighting. When using the data collected when ambient light is allowed into the experimental setup, the results of the best case EMB3 dropped to R = 0.60 and a mean error of ±1.72 g/dL. Although this does not limit the use of EMB2 and 3 in a dark room, it does suggest a user would have to make sure to use a lamp shade to reduce large fluctuations due to changes in ambient lighting. Further experiments will have to be done to determine to what degree ambient light exposure affects the results. We believe using a brighter light bulb and placing the hand closer to the lamp will reduce this problem significantly, limiting the effect of ambient light.

��The current data collection focuses on a range of 8-16 g/dL, which is sufficient for covering certain levels of anemia and

601

SESSION: HEALTH I

normal populations; however, this means HemaApp cannot be used for those who are so gravely anemic that they need immediate transfusion or those with elevated hemoglobin. Our current data collection has come across a few patients in these extreme ranges, but we will need a more targeted recruitment to have enough data to develop the system to work in these ranges. To find people with elevated hemoglobin, we will need to expand our clinical study to include pulmonary clinics with patients that have high hemoglobin due to advanced pulmonary ailments. For obtaining extremely low hemoglobin, our collaborators have suggested taking data at midnight, when most of their transfusion patients are admitted and their hemoglobin measures as low as 4-5 g/dL. For this to work, our system needs to become much quicker, more robust, and self-contained such that running our study does not require the participation of the patient and fast enough that we do not interfere with the patient’s clinical care.

� � �� Only patients with normal red blood cell traits were recruited in our initial study, but we plan to expand our evaluation to include different blood disorders (e.g., sickle cell, thalassemia, and hemolyzing patients). Each of these disorders can potentially cause different optical absorption variations that may or may not be observable by our current measurement technique. Further studies of the optical properties of different hemoglobin types will have to be done through laboratory experiments. On the other hand, patients with hemolysis have ruptured red blood cells that results in free floating hemoglobin in the blood plasma. In the measurement of hemoglobin concentration, it is important to distinguish between hemolyzed and non-hemolyzed hemoglobin. In a laboratory test, the blood is first separated into the plasma and intact red blood cell components before hemoglobin concentration is determined. HemaApp has not been validated to differentiate the two conditions to produce an adjusted measurement. This is not possible for our system.

��!��"��A major hurdle to making the system truly deployable is in making a system that can ensure data quality. All the data collection is conducted either by the development team or trained data collection assistants who have experience using prototype medical devices. The interface produces a real-time visualization of the camera feed that they interpret at the time of data collection to ensure that the patient has fully covered the camera and LEDs and the waveform being recorded is void of excessive movement.

To enforce proper finger placement and reduce movement, a finger cuff can be designed to center the finger over the camera. A snug fit for such a cuff would be ideal, but it cannot be so tight that the finger loses circulation. Furthermore, the finger cuff will also function as a cover to block the ambient light. A small window can then be integrated into the top of the cuff to be opened for the incandescent light.

An automated signal quality detector will also be necessary for detecting whether a signal is stable enough for analysis. The usual components in the signal consists of the pulse signal and the breathing signal, both of which are typically periodic between 0.3 – 2 Hz. An automated system can analyze for sudden DC shifts caused by the finger shifting, resulting in a non-periodic signal alteration. This signal quality detector then determines whether a segment of collected data is useable for analysis, prompting the user to perform the data collection again if it failed.

�$�(&�#$HemaApp is a smartphone application that noninvasively monitors the hemoglobin concentration using the smartphone’s camera and different lighting sources. Monitoring hemoglobin is a standard clinical tool for both screening and assessing a patient’s response to treatment. We evaluate HemaApp on 31 patients ranging from 6 – 77 years of age using three different embodiments of our system. The first uses LEDs commonly found on smartphones. The second augments the phone with an incandescent light source. Finally, the third both augments the phone with an incandescent source and uses a custom LED setup. The resulting hemoglobin predictions yield a rank order of 0.69, 0.74, and 0.82 correlation with the gold standard blood test. In screening for anemia, HemaApp can achieve 85.7% sensitivity and 76.5% specificity when augmented with incandescent lights and IR LEDs. Comparing to an specialized FDA-approved noninvasive hemoglobin measurement device, which performed at a rank of 0.81 and an sensitivity and specificity of 69.3% and 88.2%, our results for HemaApp are promising.

To further validate our results, a larger national and international study that includes a more global demographic will need to be deployed. The current results can only be interpreted under the scope of normal hemoglobin phenotypes. However, many of the clinical use cases that would highly benefit from daily monitoring of hemoglobin and anemia require the same efficacy for hemoglobins that are deformed due to changes in its protein structure.

This study is a first step in transferring hemachrome analysis techniques to a smartphone platform, introducing hemoglobin monitoring to a wide audience, from those who need at home monitoring to community health workers.

��$)('%/'"'��We thank all the volunteers, and all publications support who wrote and provided helpful comments on previous versions of this document. In particular, we thank Manoj Gulati, Alex Mariakakis, Lauren Oickle, Hayley Young, Jim Taylor, and Jim Stout for their help with the coordination and data collection during the clinical study. This work is performed under the approval from the University of Washington Human Subject Division and reliance agreement with the Seattle Children’s Hospital: IRB HSD #49667.

602


�'-'�'�'

1. Omar Abdallah, Mohammed Natsheh, Kawther Abo Alam, Qasem Qananwah, Ahrned a Nabulsi, and Armin Bolz. 2010. Concentrations of Hemoglobin Fractions Calculation Using Modified Lambert-Beer Law and Solving of an Ill-Posed System of Equations. Spie 7715, 1, 1–8. http://doi.org/10.1117/12.854603

2. Kawther Abo Alam. 2011. Fuzzy Logic Hemoglobin Sensors. Ph.D Dissertation. Karlsruhe, Germany.

3. A. Brown and A. L. Goodall. 1946. Normal variations in blood haemoglobin concentration. The Journal of Physiology 104, 4, 404–407. http://doi.org/10.1113/jphysiol.1946.sp004133

4. H. Franklin Bunn. 1997. Pathogenesis and Treatment of Sickle Cell Disease. The New England Journal of Medicine 337, 11, 762–769. http://doi.org/10.1146/annurev.nutr.23.011702.073027

5. H. Gehring, C. Hornberger, L. Dibbelt, et al. 2002. Accuracy of Point-of-Care-Testing (POCT) for Determining Hemoglobin Concentrations. Acta Anaesthesiologica Scandinavica 46, 8, 980–986. http://doi.org/aas460809 [pii]

6. Beverly George-Gay and Katherine Parker. 2003. Understanding the Complete Blood Count with Differential. Journal of Perianesthesia Nursing 18, 2, 96–117. http://doi.org/10.1053/jpan.2003.50013

7. Lilian de Greef, Mayank Goel, Min Joon Seo, et al. 2014. Bilicam: Using Mobile Phones to Monitor Newborn Jaundice. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing - UbiComp ’14 Adjunct, 331–342. http://doi.org/10.1145/2632048.2632076

8. Mathew J. Gregoski, Martina Mueller, Alexey Vertegel, et al. 2012. Development and Validation of a Smartphone Heart Rate Acquisition Application for Health Promotion and Wellness Telehealth Applications. International Journal of Telemedicine and Applications 2012. http://doi.org/10.1155/2012/696324

9. E. Jonathan and Martin Leahy. 2010. Investigating a Smartphone Imaging Unit for Photoplethysmography. Physiological measurement 31, 11, N79–83. http://doi.org/10.1088/0967-3334/31/11/N01

10. Warren a Kaplan. 2006. Can the Ubiquitous Power of Mobile Phones be Used to Improve Health Outcomes in Developing Countries? Globalization and Health 2, 1, 9. http://doi.org/10.1186/1744-8603-2-9

11. Walter Karlen, J. Mark Ansermino, Guy A. Dumont,

and Cornie Scheffer. 2013. Detection of the Optimal Region of Interest for Camera Oximetry. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2263–2266. http://doi.org/10.1109/EMBC.2013.6609988

12. Walter Karlen, Joanne Lim, J. Mark Ansermino, Guy Dumont, and Cornie Scheffer. 2012. Design Challenges for Camera Oximetry on a Mobile Phone. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2448–2451. http://doi.org/10.1109/EMBC.2012.6346459

13. Jens Kraitl, Ulrich Timm, Elfed Lewis, and Hartmut Ewald. 2010. Optical Sensor Technology for a Noninvasive Continuous Monitoring of Blood Components. Library 7572, 757209–757209–10. http://doi.org/10.1117/12.843153

14. Sungjun Kwon, Hyunseok Kim, and Kwang Suk Park. 2012. Validation of Heart Rate Extraction Using Video Imaging on a Built-in Camera System of a Smartphone. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2174–2177. http://doi.org/10.1109/EMBC.2012.6346392

15. Eric C Larson, Mayank Goel, Gaetano Boriello, Sonya Heltshe, Margaret Rosenfeld, and Shwetak N Patel. SpiroSmart : Using a Microphone to Measure Lung Function on a Mobile Phone. Figure 1.

16. Mark R. MacKnet, Martin Allard, Richard L. Applegate, and James Rook. 2010. The Accuracy of Noninvasive and Continuous Total Hemoglobin Measurement by Pulse CO-oximetry in Human Subjects Undergoing Hemodilution. Anesthesia and Analgesia 111, 6, 1424–1426. http://doi.org/10.1213/ANE.0b013e3181fc74b9

17. R; Rajalakshmi, D; Nandakumar, and M; Nathaniel. 2015. Contactless Sleep Apnea Detection on Smartphones. The 13th Annual International Conference on Mobile Systems, Applications and Services, 45–57. http://doi.org/10.1145/2742647.2742674

18. Mark J. Rice, Nikolaus Gravenstein, and Timothy E. Morey. 2013. Noninvasive Hemoglobin Monitoring: How Accurate is Enough? Anesthesia and Analgesia 117, 4, 902–907. http://doi.org/10.1213/ANE.0b013e31829483fb

19. Js Ruckman. 2011. A Comparative Study of Total Hemoglobin Measurement Technology: Noninvasive Pulse Co-oximetry and Conventional Methods.

20. H. Von Schenck, M. Falkensson, and B. Lundberg. 1986. Evaluation of “HemoCue,” a new device for

603

SESSION: HEALTH I

determining hemoglobin. Clinical Chemistry 32, 3, 526–529.

21. Christopher G. Scully, Jinseok Lee, Joseph Meyer, et al. 2012. Physiological Parameter Monitoring from Optical Recordings with a Mobile Phone. IEEE Transactions on Biomedical Engineering 59, 2, 303–306. http://doi.org/10.1109/TBME.2011.2163157

22. Jarrel Seah and Jennifer Tang. Eyenaemia. Retrieved May 12, 2015 from www.eyenaemia.com

23. Jeff Thompson. 2015. Smartphone Attachment for Stethoscope. Methods in molecular biology 1256, 213–29. http://doi.org/10.1007/978-1-4939-2172-0

24. U. Timm, J. Kraitl, H. Gewiss, et al. 2015. Novel Multi Wavelength Sensor Concept to Detect Total Hemoglobin Concentration, Methemoglobin and Oxygen Saturation. Progress in Biomedical Optics and Imaging - Proceedings of SPIE 9332, Figure 1, 1–9. http://doi.org/10.1117/12.2080144

25. K Wac. 2012. Smartphone as a Personal, Pervasive Health Informatics Services Platform: Literature Review. Yearbook of medical informatics 7, 83–93. http://doi.org/arXiv:1310.7965v1 [cs.CY]

26. John Webster. 2009. Medical Instrumentation: Application and Design. Wiley and Sons.

27. B. Woodward, R. S H Istepanian, and C. I. Richards. 2001. Design of a Telemedicine System using a Mobile Telephone. IEEE Transactions on Information Technology in Biomedicine 5, 1, 13–15. http://doi.org/10.1109/4233.908361

28. Hongying Zhu and Aydogan Ozcan. 2015. Opto-Fluidics Based Microscopy and Flow Cytometry on a Cell Phone for Blood Analysis. Methods in molecular biology 1256, 213–29. http://doi.org/10.1007/978-1-4939-2172-0

29. George Zouridakis, Tarun Wadhawan, Ning Situ, et al. 2015. Melanoma and Other Skin Lesion Detection using Smart Handheld Devices. In Methods in molecular biology. Clifton, NJ, 213–29. http://doi.org/10.1007/978-1-4939-2172-0

30. Wireless Blood Pressure Wrist Monitor. Retrieved January 4, 2016 from https://ihealthlabs.com/blood-pressure-monitors/wireless-blood-pressure-wrist-monitor/

31. Wireless Smart Gluco-monitoring System. 53. http://doi.org/10.1017/CBO9781107415324.004

604


Documents

˘ ˇ ˆ˙ - University of Washington · narrowband light source; (2) a comparison of the hemoglobin estimation with a clinical blood draw and another noninvasive optical hemoglobin