Comparison of the Effectiveness of Retinal Cameras for

Malarial Retinopathy Detection in Malawi

Joshi, Vinayak S.1; Nemeth, Sheila C.1; Barriga, E. Simon1; Soliz, Peter 1; Harding, Simon P.2; Lewallen, Susan4; Taylor, Terrie E.5,6 ; MacCormick, Ian J.2,3

1. VisionQuest i-Rx Corp., Albuquerque, NM, United States. 2. Eye and Vision Science, Institute of Ageing and Chronic Disease, University of Liverpool, Liverpool, United Kingdom. 3. Malawi-Liverpool-Wellcome Trust Clinical Research Programme, Blantyre, Malawi. 4. Kilimanjaro Centre for Community Ophthalmology, University Cape Town Groot Schuur Hospital, Cape Town, South Africa. 5. Osteopathic Medical Specialties, Michigan State University, East Lansing, MI, United States. 6. Blantyre Malaria Project, Blantyre, MI, United States.

November 4, 2015

Final Report

Peter Soliz / CEO

505-508-1994 / psoliz@visionquest-.com

VisionQuest i-Rx, Corp. ©2015 Page 2

Background. Cerebral malaria (CM) is a life-threatening clinical syndrome associated with malarial

infection. The 2014 World Malaria report estimates that malaria affected over 198 million people in

2013, and claimed the lives of 584,000 people worldwide, about 75% of whom were African children

under 5 years of age1. Many of these mortalities were thought to be due to CM2. This observation points

to the need for a highly specific diagnostic to ensure detection of CM.

Making an accurate diagnosis of CM remains a clinical challenge in many parts of Africa. In areas where

malaria is endemic, the presence of the malaria parasite is common, even among children without

symptoms. Thus, when a child presents with symptoms of cerebral malaria and is in a comatose state

from any cause, simply detecting malaria parasites in the blood is not sufficient for a specific diagnosis of

CM. In fact, a large autopsy study revealed that 23% of children with clinically defined CM were found

to have non-malaria causes2. This study also showed that the presence of highly specific lesions in the

retina, known as malarial retinopathy (MR), which was detected in the comatose children before death,

was the best indicator of CM as verified by autopsy. MR occurs in a spectrum of severity. Among

children with clinically defined CM, the acuteness of MR correlates well with severity of various signs of

CM. In the past two decades it has been observed that approximately 60% of children in the Blantyre

(Malawi, Africa research project) diagnosed with clinical CM, were positive for MR, suggesting 40% likely

had the symptoms from something other than CM.

Dilated Binocular Indirect Ophthalmoscopy (BIO) examination by an ophthalmologist is the current

reference standard for detection of MR. This form of examination depends upon the availability of

qualified and highly-skilled ophthalmic expertise. Even among experienced ophthalmologists there will

be disagreements on findings. Most non-ophthalmologists do not practice enough ophthalmoscopy to

be proficient in the BIO examinations. In the regions of the African continent where CM is highly

prevalent there is a scarcity of ophthalmic expertise. Thus, in most settings where CM is treated in

Africa, looking for signs of MR is not considered and thus a misdiagnosis can result. Additionally, in this

resource limited region of the world, the solution to making this medical diagnosis must be accessible

and affordable to the affected population.

Digital retinal imaging technology has improved recently, and the retinal imaging process is now possible

at sites other than the malaria care clinics, thus making possible the diagnoses of CM using image

visualization by ophthalmic experts located remotely, through tele-retinal screening services. However,

a routine use of retinal fundus photography is still problematic. One issue is that table top retinal

cameras are cumbersome and expensive and are not suitable for clinical environment in Africa. New

technologies using handheld retinal cameras and smart phone cameras may allow retinal fundus

imaging to be done simply and inexpensively. Retinal imaging with hand held devices opens for the first

time the possibility of providing clinicians who treat children in malaria areas, with high quality fundus

photos of children with clinically defined CM. BIO and retinal photography technologies have been

tested separately, thus no direct comparisons in a clinical setting. In this study we compared BIO exam

findings with the clinical performance of three retinal cameras in capturing signs of malarial retinopathy.

1 World Malaria Report, World Health Organization Global Malaria program, Dec 2014. 2 Taylor TE, et al. “Differentiating the pathologies of cerebral malaria by postmortem parasite counts.” Nat Med. 2004; 10:143-145.

VisionQuest i-Rx, Corp. ©2015 Page 3

Figure 1. Retinal cameras utilized in the study: Left: Peek. Middle: Pictor-plus. Right: Topcon

Study objective. The objective of this study was to test and compare three retinal cameras: the Pictor-

Plus (Volk Optical), the Topcon 50EX (Topcon Corporation) and the Peek camera (Portable eye

examination kit). We quantified the performance of these cameras for MR detection by comparing the

interpretation of images from each against the clinically accepted reference standard, BIO, in terms of

sensitivity, specificity, and Kappa agreement. The current clinical requirement for the verification of CM

diagnosis is MR detection with high specificity. Examination of a patient for MR will identify the non-CM

patients who may be mistakenly diagnosed for CM. Therefore, one of the criteria for analyzing the

camera performance included the specificity for MR detection, which is the first step toward

accomplishing the ultimate goal of reducing the false positive rate for CM. The results of this study can

be used by clinicians implementing MR screening programs to select the most appropriate camera for

their intended application and clinical setting. Figure 1 shows the three retinal cameras used in this

study.

The aim of this study was achieved through primary and secondary objectives. The primary objective

was:

1. To determine the inter-device agreement as judged by a single trained reader between, 1)

handheld cameras (Pictor-Plus and Peek) and a table top camera (Topcon), 2) handheld cameras

(Pictor-Plus and Peek) and BIO, and 3) Topcon and BIO, in classifying a clinically diagnosed CM

patient into MR positive or MR negative (as determined by the BIO exam); when BIO and camera

imaging are performed by an experienced ophthalmologist.

The secondary objectives were:

2. To determine the inter-device agreement as judged by the same observer between, 1) handheld

cameras (Pictor-Plus and Peek) and table top camera (Topcon), 2) handheld cameras (Pictor-Plus

and Peek) and BIO, 3) Topcon and BIO; (both performed by an experienced ophthalmologist) in

identifying and grading the various features of malarial retinopathy: (a) hemorrhages in macula (b)

peripheral hemorrhages (c) macular whitening, (d) peripheral whitening, (e) abnormal vessels and

(f) papilledema.

3. To determine the inter-camera variation in image collection quality in relation to the classification

of a clinically diagnosed CM patient into MR positive or MR negative, presence of papilledema, and

presence of the 3 individual signs of MR (hemorrhage, retinal whitening and vessel abnormalities)

between photos taken by Peek, Pictor-Plus, and Topcon; when all the three image sets are graded

by an independent retinal grader.

VisionQuest i-Rx, Corp. ©2015 Page 4

Figure 2. Signs of malarial retinopathy in retinal images. Left: vessel discoloration. Middle: white-centered retinal hemorrhages. Right: retinal whitening.

Table 1: Camera specifications

Camera FOV Pixel format Mode

Topcon 50 96 DPI, 8 bits Single frame

Pictor-Plus 45 72 DPI, 8 bits Single frame

Peek 20 72 DPI, 8 bits Video (30 frames/sec)

Camera Specifications. Two of the factors affecting the results of the inter camera comparison are,

naturally, the resolution and field of view (FOV) of each (See Table 1). While the optimum set of images

would include the posterior pole as well as the periphery of the retina, the limits of the cameras’ FOV

required the ophthalmologist to attempt to acquire multiple images to maximize the area covered by

each camera. The Peek had a different

implementation of the image capture

process. Peek acquires a video rather

than a single image. This allows the user

to scan the retina and collecting multiple

frames of the area.

Study Design. The study consisted of prospective data collection in the context of an ongoing descriptive

study of pediatric CM at the Queen Elizabeth’s Hospital in Blantyre, Malawi. A single ophthalmologist

(IM) performed MR assessment through BIO as well as mydriatic retinal imaging using the three retinal

cameras (Peek, Pictor-Plus, and Topcon) on 23 comatose children (aged 6 months - 12 years) admitted in

Hospital, with a Blantyre coma score of ≤2 and peripheral P. falciparum. Parents of the children provided

consent according to international research standards and with IRB approvals from the College of

Medicine Research and Ethics Committee, University of Malawi.

The retinal images were graded by a certified retinal grader for the detection of specific signs of MR. A

statistical agreement coefficient (Kappa) and sensitivity/specificity metrics were obtained by comparing

the retinal image assessments and indirect ophthalmoscopy examination findings to evaluate the

performance of each camera and its usability in identifying MR. The ophthalmologist also evaluated

each camera in terms of factors such as image quality, usability, ergonomics, portability, cost, etc.

Data. Of the N=23 children enrolled in the study, N=8 were controls; that is, they did not present with

signs of MR, and N=15 were clinically diagnosed with CM and presented signs of MR. Using an

internationally recognized grading system developed by the MR grading consensus group at the

University of Liverpool, the retinal images were graded by a certified retinal grader for specific signs of

MR. Figure 2 presents examples of the three lesions: vessel discoloration (left), white-centered retinal

VisionQuest i-Rx, Corp. ©2015 Page 5

hemorrhages (middle), and retinal whitening (right). The results of the reader’s evaluation of the images

were compared to the BIO exam results, and sensitivity, specificity, and Kappa for the presence of MR

lesions were calculated.

The BIO exams found 12 cases with whitening, 8 cases with hemorrhages, and 4 cases with vessel

discoloration. Several cases had multiple MR lesions. Papilledema was present in 3 cases with 2 of them

presenting signs of MR. The reader evaluated the images from each of the cameras tested. The Peek

camera, which is cell phone-based, captured an average of four videos per subject (two per eye); the

Pictor-plus, a handheld retinal camera, eight images per subject (four per eye); and finally, six images

per subject were captured with the Topcon, a desktop camera. The reader’s grading was masked to

previous reads from other cameras as well as the BIO results.

Figure 3 shows sample images for the same subject. This patient had retinal hemorrhages, the majority

of which were white centered, and retinal whitening. A large hemorrhage is centered over the

fovea/macula. For the Peek, a sample of three still frames from the video captured is presented in Figure

3. However, for the reading of the cases using Peek, the entire set of four 20-30 second videos is used.

Patches of retinal whitening are visible temporally from the macula in both the Topcon and Pictor-plus

views. The Topcon has a 50○ field of view (FOV) whereas the Pictor has 45○ FOV. As seen see in the

images, the Pictor captures the complete hemorrhages in the far temporal region of the retina, while

the Topcon has cut them in half due to its smaller horizontal FOV. The large central macula hemorrhage

and several smaller ones can be seen in the Peek video. Peek also captures signs of whitening, but is

frequently masked by glare or reflection artifacts, which makes the whitening detection difficult and

Figure 3. Images captured for same subject: (Top) Sample frames from video captured with Peek.

(Bottom) Left: Pictor-plus, Right: Topcon

VisionQuest i-Rx, Corp. ©2015 Page 6

Figure 4. Images showing retinal whitening: (Top) Sample frames from Peek video. (Bottom) Left: Pictor-plus, Right: Topcon. Whitening was not evident in the Peek image which led to incorrect evaluation of the subject.

unreliable. In addition to masking true whitening, these artifacts can be mistakenly misidentified as

whitening. Retinal reflexes are natural occurrences due to the nature of the retina and can be

differentiated from the glare artifacts mentioned previously. The retinal reflexes were more common

with Peek and were at minimum a distracting, but often masked the presence of one of the MR

pathologies. The artifacts in Peek can be seen in Figure 3 and Figure 4. The Peek has a limited FOV of

approximately 20○, and the views captured will depend on examiner’s skill. Furthermore, the design of

Peek’s camera attachment to a cell phone makes it difficult to bring the camera-optics close to the

patient’s cornea, which affects the image quality. The flat, rectangular shape of the Peek makes it

difficult to align and requires an unnatural handling of the device. Its small FOV and its flat rectangular

shape make the Peek a challenge to align in order to capture the intended region of the retina.

Figure 4 shows an example of a patient with retinal whitening in the temporal region of the retina (near

the macula), which was clearly seen on both the Pictor-plus and Topcon images. However, using the

Peek video, the physician could only image the area surrounding the optic disc, but could not image the

temporal region where the retinal whitening was present. Additionally, when the macula was captured,

there was significant reflection from cornea of the camera illumination source. Focusing the Peek was

problematic. Being able to both maintain the correct focus on the retina while at the same time moving

the camera in the direction needed to image all retinal regions is one aspect that adds difficulty when

trying to image all the areas of interest in the retina. The Peek camera may perform adequately in the

hands of an ophthalmologist experienced with this device and when used for immediate diagnosis, but it

is more challenging to master than the Pictor-plus or Topcon for a non-ophthalmic technician.

VisionQuest i-Rx, Corp. ©2015 Page 7

Results. This section describes the results of the comparison of each camera against the clinical

reference standard (BIO examination) and clinical imaging standard (Topcon), as well as an inter-camera

comparison in terms of grading for overall signs of MR and then by lesion types: hemorrhages,

whitening, and vessel discoloration. We also include the detection performance for Papilledema (PE) as

it represents one of the life-threatening signs associated with CM and observed in as many as 20% of CM

patients. We determined sensitivity/specificity metrics for overall MR and lesion detection, as well as

the Kappa agreement in defining the grade of each lesion. Due to the small sample of cases, the 95%

confidence intervals for sensitivity, specificity, and Kappa values overlapped between the three cameras.

Camera performance against BIO. Figure 5 shows sensitivity/specificity of each camera against the BIO

in detecting MR and individual lesions. From figure 5 we see that the Pictor-plus produced images with

quality and resolution that allows the best overall performance, sensitivity of 100% in detecting MR at a

specificity of 87%. The Pictor-plus performance was comparable or superior to the other two cameras in

terms of sensitivity and specificity of detecting MR lesions (Figure 5). Although, sensitivity for Topcon

images was as high as Pictor-plus, Topcon showed lower specificity as the retinal reader overcalled the

MR lesions in Topcon images. The reason for lower specificity when interpreting the Topcon images was

that contrast between the retinal background and MR lesions (e.g. whitening) was poorer in Topcon

images; whereas the contrast was relatively higher in the Pictor-plus images.

As described previously, the specificity for MR detection is of clinical importance for reducing the over-

diagnosis of CM. In this study, Pictor-plus provided the highest specificity among the three cameras.

However, none of the cameras imaged the vessel discoloration seen in the BIO examination by the

Figure 5. Sensitivity and specificity of each camera against BIO in detecting MR and its lesions. BIO exams revealed 8 cases with hemorrhages, 12 cases with whitening and 4 cases with vessel discoloration. Several cases had multiple MR lesions. Papilledema was present in 3 cases with 2 of them presenting signs of MR.

0%

20%

40%

60%

80%

100%

Malaria Retinopathy Hemorrhages Whitening Papilledema

Sensitivity to BIO Peek Pictor Topcon

0%

20%

40%

60%

80%

100%

Malaria Retinopathy Hemorrhages Whitening Papilledema

Specificity to BIO

VisionQuest i-Rx, Corp. ©2015 Page 8

ophthalmologist. This is a result of the vessel discoloration being in the far-periphery beyond the field of

view (FOV) of a single image acquisition by these cameras. This can be overcome by the photographer

realigning the camera to the periphery or using a lens with a wider FOV. For the small sample of cases,

the Pictor-plus performance was consistently comparable or superior to the other two cameras (See

Figure 5.

The reader assigned levels between 1 and 4 of the severity of each MR lesion based on criteria described

in Table 2. Hemorrhages and whitening were graded separately in each of the clinically significant retinal

regions such as macular, foveal, out of macula, and overall retinal region. In case of hemorrhages, a

proportion of white-centered hemorrhages among the total was also graded. Papilledema was detected

for its presence/absence only and not graded for its severity level.

Figure 6. Kappa agreement of each camera with BIO in determining the grades of hemorrhages and whitening.

0

0.2

0.4

0.6

0.8

1

Macular Out of macula White-centered Overall

Kappa for grade of hemorrhagesPeek and BIO Pictor and BIO Topcon and BIO

0

0.2

0.4

0.6

0.8

1

Macular Foveal Out of macula Overall

Kappa for grade of whiteningPeek and BIO Pictor and BIO Topcon and BIO

Table 2: MR lesion grading criteria

MR lesion Grading criteria Grade

1 2 3 4

Hemorrhage Number of hemorrhages 0 1-5 5-20 20-50

Whitening Area occupied compared

to the optic disc (OD) area 0

< 1/3 OD

1/3-1 OD

> 1 OD

Vessel discoloration

Color of discolored vessels N OR W NA

N: No color change, OR: Orange, W: White, NA: Not applicable

VisionQuest i-Rx, Corp. ©2015 Page 9

A Cohen’s Kappa agreement was calculated between each camera and the BIO for hemorrhages and

retinal whitening (Cohen’s Kappa was linearly weighted), as shown in charts (Figure 6). The 95%

confidence intervals (CI) were large and overlapping due to the limited amount of cases. Nonetheless,

the Pictor-plus camera performed consistently comparable or better than Peek and Topcon. The Kappa

agreement for vessel discoloration was nearly zero as none of the cameras could capture far-peripheral

views of retina where the discoloration is commonly observed; which was also evident from the

sensitivity, specificity results as before.

Camera performance against Topcon. A similar set of statistics as above was also calculated for the

sensitivity, specificity of Peek and Pictor-plus cameras against the Topcon camera (clinical standard for

retinal imaging), as shown in Figure 7. Again, Pictor-plus presented a higher overall sensitivity and

specificity compared to Peek. Similar to the previous results, the confidence intervals for sensitivity,

specificity values were large and the values between cameras were not statistically significantly

different. Vessel discoloration results were not included here as none of the cameras including Topcon

imaged regions of the retina where vessel discoloration was seen in the BIO exam. Figure 8 shows the

Cohen’s Kappa agreement of each of the cameras (Peek and Pictor-plus) with Topcon in determining the

grades of hemorrhages and whitening. The Pictor-plus camera performed consistently better than Peek.

The Kappa for vessel discoloration between Peek/Pictor and Topcon could not be determined as none of

the cameras including Topcon imaged the regions of vessel discoloration.

Figure 7. Sensitivity and specificity of each camera against Topcon in detecting MR and its lesions.

0%

20%

40%

60%

80%

100%

Malaria Retinopathy Hemorrhages Whitening Papilledema

Sensitivity to Topcon Peek Pictor

0%

20%

40%

60%

80%

100%

Malaria Retinopathy Hemorrhages Whitening Papilledema

Specificity to Topcon

VisionQuest i-Rx, Corp. ©2015 Page 10

Table 3. Image quality assessment

Image quality Peek Pictor Topcon

Macular 72% 83% 87%

Out of macula (Peripheral)

3% 47% 69%

Inter-camera agreement. Apart from calculating the

performance of each camera in detecting MR lesions, the

cameras were also evaluated for image quality by

quantifying the fraction of the total retinal area that was

available for grading (captured by the camera) and

gradable (adequate image quality as determined by the

retinal grader). The results are shown in table 3. As presented in Table 3, all the three cameras could

capture only a partial view of the peripheral retina.

Discussion. Although BIO has been the reference standard for identifying MR, it is important to

recognize that it may not necessarily be optimum for detecting MR. Retinal fundus photography has the

advantage of allowing the grader to study an area more closely and can provide higher magnification

than BIO. Desktop cameras such as the Topcon pose a significant challenge to the imager when trying to

maneuver the camera and perform alignment when the subject is comatose and in a supine position.

This was the major disadvantage of the Topcon.

The handheld retinal imaging systems such as the Pictor-Plus and the Peek provide varying quality color

fundus images or videos. Pictor-plus produces images with good quality and resolution to allow for the

best performance in terms of overall MR and lesion detection and presents a substantial agreement

(Kappa > 0.6) with BIO. These results for the Pictor-plus were comparable or superior to the Peek and

Topcon. Hemorrhages and whitening were detected with high accuracy in images from Pictor-plus and

Figure 8. Kappa agreement of each camera with Topcon in determining the grades of hemorrhages and whitening.

0

0.2

0.4

0.6

0.8

1

Macular Out of macula White-centered Overall

Kappa for grade of hemorrhagesPeek and Topcon Pictor and Topcon

0

0.2

0.4

0.6

0.8

1

Macular Foveal Out of macula Overall

Kappa for grade of whiteningPeek and Topcon Pictor and Topcon

VisionQuest i-Rx, Corp. ©2015 Page 11

Table 4: Factors evaluating the camera usability

Factor PEEK Pictor-Plus Topcon

Set-up Cellphone Handheld Desktop

Field of view 20○ 45○ 50○

Ergonomic Handy Handy Bulky

Size, weight Small, light Portable Cumbersome

Imaging stability Highly unstable Stable Stable

Image quality Poor, artifacts Adequate Adequate

Cost $500 $10,000 >$25,000

Figure 9. Snapshots of the Imaging process: Left: Peek, Middle: Pictor-plus, Right: Topcon

Topcon, and with lower accuracy using the Peek images. Because of the aforementioned difficulty in

aligning the Peek, peripheral regions were almost absent in Peek’s field of view (Table 3).

Apart from the accuracy of each

camera in detecting MR lesions,

the cameras were also evaluated

by the user (ophthalmologist) in

terms of usability, ergonomics and

relevant factors, as described in

table 4. Figure 9 shows the

imaging procedure for each

camera. The handheld Pictor-plus

camera was found by the

ophthalmologist to be more

stable, user-friendly, handy, and

ergonomically suitable for imaging

comatose children, unlike the Topcon. Topcon camera, although provides high accuracy for MR

detection, its bulky and difficult to handle especially when imaging The Peek camera is inexpensive,

small and lightweight, but its structure makes it hard to position the camera optics close enough to the

patient’s cornea to get the maximum field of view while holding the eyelids open. Peek was found to be

unstable for imaging due to patient’s non-cooperation and involuntary eye movements (quite common

among comatose children) which can result in poor imaging performance and image quality.

Peek imaging also resulted in number of artifacts which can make the images challenging for manual

grading as well as for automated image analysis. Peek imaging process necessitates the presence of a

trained ophthalmologist making an immediate diagnosis, but can be more challenging for a non-

ophthalmic technician or for a remote grading of images. Furthermore, because of the alignment

difficulty, the photographer could only manage to capture a limited region of the retina thereby missing

the vessel discoloration in the periphery. Considering the factors discussed above, the Pictor-plus offers

the best overall performance compared to other cameras in terms of image quality, MR and lesion

detection accuracy, ergonomic, size/weight, and costs.

VisionQuest i-Rx, Corp. ©2015 Page 12

Nonetheless, there are design changes for each of the three cameras that could improve their

performance in this specific application. Any table top retinal camera, including the Topcon, is

cumbersome and difficult to use given the comatose state and position of the malaria patient. Reading

of the images will likely be done offsite from the hospital bed, therefore the best quality image must be

forwarded for reading. None of the three cameras that were tested offer real-time image quality

feedback to the user. In these resource limited application in Africa, the imager will likely be a minimally

trained healthcare worker who may not recognize an inadequate image prior to transmitting the image

to the specialist who will make the diagnosis. In a rapidly progressing disease like cerebral malaria, the

time taken to re-image the patient could significantly impact the outcome of the patient’s recovery.

A single shot wide field of view ( 120○) image would improve significantly the utility of the retinal

imaging device, given that some MR lesions occur in the periphery where multiple images are otherwise

required to capture the full area of the posterior pole. Finally, a device that can include or link to an

automatic MR detection system is needed in these regions where immediate care is required, but

accurate diagnosis by a specialist is not easily available.

Future Work. A hand-held, low-cost, smart camera is needed that encompasses many of the attributes

of the Pictor-plus, but has wider field of view, integrated software for real-time image quality

assessment, and on-site, automatic MR/no MR determination.

Acknowledgments. We would like to thank Dr. Terrie Taylor, Dr. Ian MacCormick, Dr. Susan Lewallen,

and Dr. Simon Harding for their valuable contribution to the management of the study and relevant

clinical procedures. We also thank Michigan State University, MI, USA, and Blantyre Malaria Project,

University of Malawi College of Medicine, Malawi, Africa; for their valuable assistance in providing the

clinical site and staff required for conducting this study. Furthermore, we appreciate the assistance from

Dr. Andrew Bastawrous and Dr. Matthew Burton from the London School of Hygiene and Tropical

Medicine, UK, in providing the Peek camera equipment. We appreciate the loan of the Pictor Plus

camera from Volk Optical, Inc. This study was funded using VisionQuest i-Rx’s internal research and

development funds. None of the authors have any financial interest in any of the three cameras.