10
Proceedings of Machine Learning Research 136:318–327, 2020 Machine Learning for Health (ML4H) 2020 CheXphoto: 10,000+ Photos and Transformations of Chest X-rays for Benchmarking Deep Learning Robustness Nick A. Phillips * [email protected] Pranav Rajpurkar * [email protected] Mark Sabini * [email protected] Rayan Krishnan [email protected] Sharon Zhou [email protected] Anuj Pareek [email protected] Nguyet Minh Phu [email protected] Chris Wang [email protected] Mudit Jain [email protected] Nguyen Duong Du [email protected] Steven QH Truong [email protected] Andrew Y. Ng [email protected] Matthew P. Lungren [email protected] Editors: Emily Alsentzer , Matthew B. A. McDermott , Fabian Falck, Suproteem K. Sarkar, Subhrajit Roy , Stephanie L. Hyland Abstract Clinical deployment of deep learning algorithms for chest x-ray interpreta- tion requires a solution that can inte- grate into the vast spectrum of clini- cal workflows across the world. An ap- pealing approach to scaled deployment is to leverage the ubiquity of smart- phones by capturing photos of x-rays to share with clinicians using messaging services like WhatsApp. However, the application of chest x-ray algorithms to photos of chest x-rays requires reliable classification in the presence of arti- facts not typically encountered in dig- ital x-rays used to train machine learn- ing models. We introduce CheXphoto, a dataset of smartphone photos and synthetic photographic transformations of chest x-rays sampled from the CheX- pert dataset. To generate CheXphoto we (1) automatically and manually cap- * These authors contributed equally to this work tured photos of digital x-rays under dif- ferent settings, and (2) generated syn- thetic transformations of digital x-rays targeted to make them look like pho- tos of digital x-rays and x-ray films. We release this dataset as a resource for testing and improving the robust- ness of deep learning algorithms for au- tomated chest x-ray interpretation on smartphone photos of chest x-rays. 1. Background & Summary Chest x-rays are the most common imag- ing exams, critical for diagnosis and man- agement of many diseases and medical pro- cedures. With over 2 billion chest x-rays performed globally each year, many clinics in both developing and developed countries have an insufficient number of radiologists to perform timely x-ray interpretation [1, 2]. © 2020 N. A. Phillips et al.

CheXphoto: 10,000+ Photos and Transformations of Chest X-rays …proceedings.mlr.press/v136/phillips20a/phillips20a.pdf · 2020. 12. 9. · to photographs [17], interpretation perfor-mance

  • Upload
    others

  • View
    10

  • Download
    0

Embed Size (px)

Citation preview

  • Proceedings of Machine Learning Research 136:318–327, 2020 Machine Learning for Health (ML4H) 2020

    CheXphoto: 10,000+ Photos and Transformations of ChestX-rays for Benchmarking Deep Learning Robustness

    Nick A. Phillips∗ [email protected] Rajpurkar∗ [email protected] Sabini∗ [email protected] Krishnan [email protected] Zhou [email protected] Pareek [email protected] Minh Phu [email protected] Wang [email protected] Jain [email protected] Duong Du [email protected] QH Truong [email protected] Y. Ng [email protected] P. Lungren [email protected]

    Editors: Emily Alsentzer⊗, Matthew B. A. McDermott⊗, Fabian Falck, Suproteem K. Sarkar,

    Subhrajit Roy‡, Stephanie L. Hyland‡

    Abstract

    Clinical deployment of deep learningalgorithms for chest x-ray interpreta-tion requires a solution that can inte-grate into the vast spectrum of clini-cal workflows across the world. An ap-pealing approach to scaled deploymentis to leverage the ubiquity of smart-phones by capturing photos of x-raysto share with clinicians using messagingservices like WhatsApp. However, theapplication of chest x-ray algorithms tophotos of chest x-rays requires reliableclassification in the presence of arti-facts not typically encountered in dig-ital x-rays used to train machine learn-ing models. We introduce CheXphoto,a dataset of smartphone photos andsynthetic photographic transformationsof chest x-rays sampled from the CheX-pert dataset. To generate CheXphotowe (1) automatically and manually cap-

    ∗ These authors contributed equally to this work

    tured photos of digital x-rays under dif-ferent settings, and (2) generated syn-thetic transformations of digital x-raystargeted to make them look like pho-tos of digital x-rays and x-ray films.We release this dataset as a resourcefor testing and improving the robust-ness of deep learning algorithms for au-tomated chest x-ray interpretation onsmartphone photos of chest x-rays.

    1. Background & Summary

    Chest x-rays are the most common imag-ing exams, critical for diagnosis and man-agement of many diseases and medical pro-cedures. With over 2 billion chest x-raysperformed globally each year, many clinicsin both developing and developed countrieshave an insufficient number of radiologiststo perform timely x-ray interpretation [1, 2].

    © 2020 N. A. Phillips et al.

  • CheXphoto

    Figure 1: Overview of the CheXphoto data generation process.

    Computer algorithms could help reduce theshortage for x-ray interpretation worldwide.

    Several recent advances in training deeplearning algorithms for automated chest x-ray interpretation have been made possibleby large datasets [3, 4]. In controlled set-tings, these deep learning algorithms canlearn from labeled data to automaticallydetect pathologies at an accuracy compa-rable to that of practicing radiologists [5].These developments have been fueled byboth improvements in deep learning algo-rithms for image classification tasks [6, 7],and by the release of large public datasets[8, 9, 10, 11]. Although these algorithms havedemonstrated the potential to provide accu-rate chest x-ray interpretation and increaseaccess to radiology expertise, major obsta-cles remain in their translation to the clinicalsetting [12, 4].

    One significant obstacle to the adoption ofchest x-ray algorithms is that deployment re-quires a solution that can integrate into thevast spectrum of clinical workflows aroundthe world. Most chest x-ray algorithms aredeveloped and validated on digital x-rays,while the majority of developing regions use

    films [13, 2]. An appealing approach toscaled deployment is to leverage the ubiq-uity of existing smartphones: automatedinterpretation of x-ray film through cellphone photography has emerged through a“store-and-forward telemedicine” approach,in which one or more digital photos of chestfilms are sent as email attachments or in-stant messages by practitioners to obtain sec-ond opinions from specialists as part of clin-ical care [14, 15]. Furthermore, studies haveshown that photographs of films using mod-ern phone cameras are of equivalent diag-nostic quality to the films themselves [13],indicating the feasibility of high-quality au-tomated algorithmic interpretation of photosof x-ray films.

    Automated interpretation of chest x-rayphotos at the same level of performance asdigital chest x-rays is challenging becausephotography introduces visual artifacts notcommonly found in digital x-rays, such asaltered viewing angles, variable ambient andbackground lighting conditions, glare, moiré,rotations, translations, and blur [16]. Imageclassification algorithms have been shown toexperience a significant drop in performance

    319

  • CheXphoto

    when input images are perceived througha camera [16]. Although recent work hasdemonstrated good generalizability of deeplearning algorithms trained on digital x-raysto photographs [17], interpretation perfor-mance could be improved through inclusionof x-ray photography in the training process[18, 19]. However, there are currently nolarge-scale public datasets of photos of chestx-rays.

    To meet this need, we developed CheX-photo, a dataset of photos of chest x-raysand synthetic transformations designed tomimic the effects of photography. We be-lieve that CheXphoto will enable researchersto improve and evaluate model performanceon photos of x-rays, reducing the barrier toclinical deployment.

    2. Methods

    We introduce CheXphoto, a dataset of pho-tos of chest x-rays and synthetic transforma-tions designed to mimic the effects of pho-tography. Specifically, CheXphoto includes aset of (1) Natural Photos: automatically andmanually captured photos of x-rays underdifferent settings, including various lightingconditions and locations, and (2) SyntheticTransformations: targeted transformationsof digital x-rays to simulate the appearanceof photos of digital x-rays and x-ray films.The x-rays used in CheXphoto are primarilysampled from CheXpert, a large dataset of224,316 chest x-rays of 65,240 patients, withassociated labels for 14 observations from ra-diology reports [8].

    CheXphoto comprises a training set of nat-ural photos and synthetic transformations of10,507 x-rays from 3,000 unique patients thatwere sampled at random from the CheXperttraining set, and validation and test sets ofnatural and synthetic transformations of all234 x-rays from 200 patients and 668 x-raysfrom 500 patients in the CheXpert valida-

    tion and test sets, respectively. In addition,the CheXphoto validation set includes 200natural photos of physical x-ray films sam-pled from external data sources, intended tomore closely simulate pictures taken by radi-ologists in developing world clinical settings.As much of the developing world performs x-ray interpretation on film, this distinct set ofimages enables users to perform additionalvalidation on a novel task that may be en-countered in clinical deployment.

    2.1. Acquiring Natural Photos ofChest X-Rays

    Natural photos consist of x-ray photographyusing cell phone cameras in various lightingconditions and environments. We developedtwo sets of natural photos: images capturedthrough an automated process using a Nokia6.1 cell phone, and images captured manu-ally with an iPhone 8.

    2.1.1. Automated Capture ofNokia10k dataset

    We developed the ‘Nokia10k’ dataset by cap-turing 10,507 images of digital chest x-raysusing a tripod-mounted Nokia 6.1 cell phone(16 megapixel camera with a Zeiss sensor)and a custom Android application termedCheXpeditor to fully automate the processesof photography and metadata management.The primary challenge in automation wassynchronizing picture-taking on the phonewith displaying the chest x-ray on the moni-tor, to maintain a 1-to-1 correspondence be-tween each chest x-ray and its photographedimage. Without a robust synchronizationmethod, photos of chest x-rays might beskipped or duplicated, jeopardizing the datacollection process. Thus, bidirectional com-munication over UDP was established be-tween the phone and the computer drivingthe monitor to exchange image metadata,

    320

  • CheXphoto

    Phone

    UDP Server App UI

    Storage

    Computer

    UDP Client

    Storage

    Monitor Camera

    1

    2

    3

    4

    6

    7

    5

    a.

    b. c.

    Figure 2: Acquiring Natural Photos of Chest X-Rays Using Automated Capture a. Visualrepresentation of the automated picture-taking process used for Nokia10k. Thesteps are described: 1. X-ray retrieved from computer storage, 2. X-ray displayedon monitor, 3. X-ray index and metadata sent to phone over UDP, 4. Indexverified by phone, and camera triggered, 5. Application UI updated with newpicture and filename, 6. Picture saved to phone storage with metadata in filename,7. Computer notified that imaging was successful. b. The physical setup used forNokia10k, set in an example environment. c. Phone application UI, displayingmost recent picture and saved filename.

    take photos, and advance the chest x-ray onthe monitor.

    The 10,507 x-rays in Nokia10k were in-dexed deterministically from 1 to N . Weselected disjoint subsets of 250 to 500 consec-utive indices to be photographed in constantenvironmental conditions. For each subset of

    indices, photography was conducted as fol-lows:

    1) The ith chest x-ray was retrieved fromcomputer storage. 2) The ith chest x-raywas displayed on the monitor. 3) The imagemetadata m was assembled by the computer,and (i,m) were sent to the phone via UDP.4) The phone verified that i was one greaterthan the previous index. If so, its camera

    321

  • CheXphoto

    Table 1: The distribution of labeled observations for the Nokia10k training dataset.

    Pathology Positive (%) Uncertain (%) Negative (%)

    No Finding 972 (9.25) 0 (0.00) 9535 (90.75)Enlarged Cardiomediastinum 518 (4.93) 600 (5.71) 9389 (89.36)Cardiomegaly 1313 (12.50) 370 (3.52) 8824 (83.98)Lung Opacity 5184 (49.34) 213 (2.03) 5110 (48.63)Lung Lesion 415 (3.95) 78 (0.74) 10014 (95.31)Edema 2553 (24.30) 634 (6.03) 7320 (69.67)Consolidation 671 (6.39) 1315 (12.52) 8521 (81.10)Pneumonia 263 (2.50) 885 (8.42) 9359 (89.07)Atelectasis 1577 (15.01) 1595 (15.18) 7335 (69.81)Pneumothorax 957 (9.11) 166 (1.58) 9384 (89.31)Pleural Effusion 4115 (39.16) 607 (5.78) 5785 (55.06)Pleural Other 170 (1.62) 127 (1.21) 10210 (97.17)Fracture 391 (3.72) 31 (0.30) 10085 (95.98)Support Devices 5591 (53.21) 48 (0.46) 4868 (46.33)

    was triggered. Else, the computer was noti-fied of an error, and the entire picture-takingprocess was aborted. 5) The phone appli-cation UI, responsible for displaying statusand current image, was updated to show thenew picture and filename. 6) The picturewas saved to phone storage with the meta-data m embedded in the filename. 7) Thephone notified the computer that the imag-ing was successful, and the entire process wasrepeated for the i + 1st chest x-ray.

    After all images for a Nokia10k subset weretaken, they were exported in one batch fromthe phone to storage. The metadata wasparsed from the image filenames and usedto automatically assign the correct CheX-pert label. Alterations made to the imagingconditions after every subset included mov-ing to a different room, switching the roomlight on/off, opening/closing the window-blinds, rotating the phone orientation be-tween portrait/landscape, adjusting the po-sition of the tripod, moving the mouse cur-sor, varying the monitor’s color temperature,and switching the monitor’s screen finish be-tween matte/glossy. In all conditions, the

    chest x-ray was centered in the camera view-finder and lung fields were contained withinthe field of view.

    2.1.2. Manual Capture of iPhone1kdataset

    We developed the ‘iPhone1k dataset’ bymanually capturing 1,000 images of digitalchest x-rays using an iPhone 8 (12 megapixelcamera with a Sony Exmor RS sensor).The digital x-rays selected for the iPhone1kdataset are a randomly sampled subset of thex-rays used in the Nokia10k dataset. To pro-duce the iPhone1k dataset, chest x-rays weredisplayed in full-screen on a computer moni-tor with 1920 x 1080 screen resolution and ablack background. A physician took photosof the chest x-rays with a handheld iPhone 8using the standard camera app. The physi-cian was advised to change angle and dis-tance from the computer monitor in-betweeneach picture within constraints.

    For all images, the chest x-ray was cen-tered in the viewfinder of the camera, andthe thoracic and lung fields were contained

    322

  • CheXphoto

    within the field of view. Conformant to ra-diological chest x-ray standards, both lung-apices and costodiaphragmatic recesses wereincluded craniocaudally, and the edges of theribcage were included laterally. Photos werecaptured in sets of 100 to 200 images at atime; between sets, ambient alterations weremade, such as switching the room-lightingon/off, opening or closing of the window-blinds, and physically moving the computermonitor to a different location in the room.

    2.2. Generating SyntheticPhotographic Transformations ofChest X-Rays

    Synthetic transformations consist of auto-mated changes to digital x-rays designed tosimulate the appearance of photos of digi-tal x-rays and x-ray films. We developedtwo sets of complementary synthetic trans-formations: digital transformations to altercontrast and brightness, and spatial trans-formations to add glare, moiré effects andperspective changes. To ensure that the levelof these transformations did not impact thequality of the image for physician diagnosis,the images were verified by a physician. Insome cases, the effects may be visually im-perceptible, but may still be adversarial forclassification models. For both sets, we applythe transformations to the same 10,507 digi-tal x-rays selected for the Nokia10k dataset.

    Digital transformations were produced bysuccessive random alterations of contrast andbrightness. First, the image was either en-hanced for greater or lesser contrast. Set-ting a contrast factor of 1 for the originalimage, the contrast up transformation in-creased contrast by a factor of 1.1 and thecontrast down transformation decreased con-trast by a factor of 0.83. For both these fac-tors, random noise between -0.01 and 0.01was applied. After the contrast modification,the brightness of the image was then trans-

    formed randomly up or down using the samenumeric factors. Both the brightness andcontrast transformations were applied usingthe Python PIL ImageEnhance class.

    Spatial transformations consisted of alter-ations to add glare, moiré effects and per-spective changes. First, we applied a glarematte transformation to simulate the effectof photographing a glossy film which reflectsambient light. This was produced by ran-domly generating a two-dimensional multi-variate normal distribution which describesthe location of a circular, white mask. Sec-ond, a moiré effect was added to simulatethe pattern of parallel lines seen by digi-tal cameras when taking pictures of com-puter screens. The effect is produced as aresult of the difference in rates of shutterspeed and LCD screen sweeping refresh rate.The moiré effect was simulated by generat-ing semi-transparent parallel lines, warpingthem and overlaying them onto each image.Finally, a tilt effect was added to simulaterandom distortions of perspective that mayarise in taking a photo at angle to the screen.The tilt effect was produced by randomlyscaling the x and y values of each of the cor-ners by a factor between 0 and 0.05 towardsthe center. This random movement of cor-ners is used to skew the entire photo.

    Both the digital and spatial transforma-tions are provided in CheXphoto. Eachtransformation may be reproduced individ-ually using the code provided. Additionaltransformations - glare glossy, blur, motion,rotation, translation - are also included.

    2.3. Validation and Test

    We developed a CheXphoto validation andtest set to be used for model validation andevaluation. The validation set comprises nat-ural photos and synthetic transformations ofall 234 x-rays in the CheXpert validation set,and is included in the public release, while

    323

  • CheXphoto

    Table 2: Natural Photos (a-b) and Synthetic Transformations (Digital (c-f) and Spatial(g-i)) included in CheXphoto.

    (a) iPhone (b) Nokia

    (c) Brightness Up (d) Brightness Down (e) Contrast Up (f) Contrast Down

    (g) Glare Matte (h) Moiré (i) Tilt

    the test set comprises natural photos of all668 x-rays in the CheXpert test set, and iswithheld for evaluation purposes.

    We generated the natural photos of thevalidation set by manually capturing imagesof x-rays displayed on a 2560 × 1080 monitorusing a OnePlus 6 cell phone (16 megapixelcamera with a Sony IMX 519 sensor), follow-ing a protocol that mirrored the iPhone1kdataset. Synthetic transformations of thevalidation images were produced using thesame protocol as the synthetic training set.The test set was captured using an iPhone 8,following the same protocol as the iPhone1kdataset.

    The validation set contains an additional200 cell phone photos of x-ray films for 200unique patients. As photos of physical x-

    ray films, this component of the validationset is distinct from the previously describednatural and synthetic transformations of dig-ital x-rays. Films for 119 patients weresampled from the MIMIC-CXR dataset [11],and films for 81 patients were provided byVinBrain, a subsidiary of Vingroup in Viet-nam, and originally collected through jointresearch projects with leading lung hospitalsin Vietnam. The film dataset spans 5 obser-vation labels (atelectasis, cardiomegaly, con-solidation, edema, pleural effusion), with 40images supporting each observation. Obser-vation labels for each image were manuallyverified by a physician. Images were cap-tured using a VinSmart phone with a 12MPcamera by positioning the physical x-ray filmvertically on a light box in typical clinical

    324

  • CheXphoto

    Table 3: The number of patients, studies,and images in CheXphoto.

    Dataset Patients Studies Images

    TrainingiPhone 295 829 1,000Nokia 3,000 8,931 10,507Synthetic 3,000 8,931 10,507

    ValidationNatural 200 200 234Synthetic 200 200 234Film 200 200 200

    Test 500 500 668

    Figure 3: CheXphoto directory structure

    lighting conditions, and images were auto-matically cropped and oriented.

    2.4. Technical Validation

    CheXphoto was developed using images andlabels from the CheXpert dataset [8]. Pho-tography of x-rays was conducted in a con-trolled setting in accordance with the pro-tocols documented in the Methods section,which were developed with physician consul-tation. Although CheXphoto contains mul-tiple images for some patients, either fromthe same or different studies, there is no pa-tient overlap between the training, valida-tion, and test sets. Code developed for syn-thetic transformations is version controlledand made available as an open source re-source for review and modification. All im-ages are uniquely identifiable by patient ID,study ID, and view, and the age and sex ofeach patient is provided in the data descrip-tion CSV file. The original, unaltered imagescan be obtained from the CheXpert datasetby the unique identifiers.

    The CheXphoto dataset is organized byby transformation; the training and valida-tion sets contain directories corresponding tothe method of data generation. Within each

    directory, the x-ray images are organized insubdirectories by a patient identifier, studyID, and one or more individual views. Im-ages are stored as JPEG files, and image di-mensions vary according to the method ofgeneration. Each transformation set has anassociated CSV file, which provides observa-tion labels from the CheXpert dataset andrelative paths to the corresponding images.

    2.5. Data Access

    The CheXphoto training and validation setsare available for download1. The CheXphototest set is withheld for official evaluation ofmodels. CheXphoto users may submit theirexecutable code, which is then run on the pri-vate test set, preserving the integrity of thetest results. The testing process is enabledby CodaLab [20], an online platform for col-laborative and reproducible computationalresearch. CodaLab Worksheets exposes asimple command-line interface, which en-ables users to submit a Docker image, depen-dencies, and the necessary commands to runtheir models. These features allow us to runarbitrary code submissions on the withheld

    1. https://stanfordmlgroup.github.io/competitions/chexphoto

    325

    https://stanfordmlgroup.github.io/competitions/chexphotohttps://stanfordmlgroup.github.io/competitions/chexphoto

  • CheXphoto

    test set. Once a user has successfully up-loaded their code to CodaLab, we will eval-uate their performance on the withheld testset and share results on a live leaderboard onthe web.

    In addition, the code used to preparethe synthetically generated dataset is pub-licly available2. The synthetic transforma-tions can be reproduced by running thesynthesize.py script with the appropriateCSV file containing the paths to the imagesfor which the perturbation is to be applied.Detailed instructions on flags and usage areincluded in the repository README.

    3. Conclusion

    We believe that CheXphoto will enablegreater access to automated chest x-rayinterpretation algorithms worldwide, prin-cipally in healthcare systems that arepresently excluded from the benefits of dig-ital medicine. By facilitating the develop-ment, validation, and testing of automatedchest x-ray interpretation algorithms with aubiquitous technology such as smartphonephotography, CheXphoto broadens access tointerpretation algorithms in underdevelopedregions, where this technology is poised tohave the greatest impact on the availabilityand quality of healthcare.

    References

    [1] Withey, S., Goh, V., Montana, G.,Pesce, E. & Bakewell, R. Auto-mated triaging of adult chest radio-graphs with deep artificial neural net-works. Radiology 2019, 1–7, 10.1148/radiol.2018180921 (2019).

    [2] Andronikou, S. et al. Paediatric radiol-ogy seen from africa. part i: providing

    2. https://github.com/stanfordmlgroup/cheXphoto

    diagnostic imaging to a young popula-tion. Pediatric Radiology 41, 811–825,10.1007/s00247-011-2081-8 (2011).

    [3] Hwang, E. J. et al. Developmentand Validation of a Deep Learn-ing–Based Automated DetectionAlgorithm for Major Thoracic Dis-eases on Chest Radiographs. JAMANetwork Open 2, e191095–e191095,10.1001/jamanetworkopen.2019.1095(2019). https://jamanetwork.com/journals/jamanetworkopen/articlepdf/2728630/hwang 2019 oi 190065.pdf.

    [4] Singh, R. et al. Deep learning inchest radiography: Detection of findingsand presence of change. PLOS ONE13, 1–12, 10.1371/journal.pone.0204155(2018).

    [5] Kallianos, K. et al. How far have wecome? artificial intelligence for chestradiograph interpretation. Clinical Ra-diology 74, 10.1016/j.crad.2018.12.015(2019).

    [6] Huang, G., Liu, Z. & Weinberger, K. Q.Densely connected convolutional net-works. CoRR abs/1608.06993 (2016).1608.06993.

    [7] Rajpurkar, P. et al. Deep learning forchest radiograph diagnosis: A retrospec-tive comparison of the chexnext algo-rithm to practicing radiologists. PLOSMedicine 15, 1–17, 10.1371/journal.pmed.1002686 (2018).

    [8] Irvin, J. et al. Chexpert: A large chestradiograph dataset with uncertainty la-bels and expert comparison. CoRRabs/1901.07031 (2019). 1901.07031.

    [9] Wang, X. et al. Chestx-ray8: Hospital-scale chest x-ray database and bench-marks on weakly-supervised classifica-tion and localization of common thorax

    326

    10.1148/radiol.201818092110.1148/radiol.2018180921https://github.com/stanfordmlgroup/cheXphotohttps://github.com/stanfordmlgroup/cheXphoto10.1007/s00247-011-2081-810.1001/jamanetworkopen.2019.1095https://jamanetwork.com/journals/jamanetworkopen/articlepdf/2728630/hwang_2019_oi_190065.pdfhttps://jamanetwork.com/journals/jamanetworkopen/articlepdf/2728630/hwang_2019_oi_190065.pdfhttps://jamanetwork.com/journals/jamanetworkopen/articlepdf/2728630/hwang_2019_oi_190065.pdf10.1371/journal.pone.020415510.1016/j.crad.2018.12.0151608.0699310.1371/journal.pmed.100268610.1371/journal.pmed.10026861901.07031

  • CheXphoto

    diseases. In 2017 IEEE Conference onComputer Vision and Pattern Recogni-tion (CVPR), 3462–3471 (2017).

    [10] Bustos, A., Pertusa, A., Salinas, J.-M.& de la Iglesia-Vayá, M. Padchest: Alarge chest x-ray image dataset withmulti-label annotated reports (2019).1901.07441.

    [11] Johnson, A. E. W. et al. Mimic-cxr, ade-identified publicly available databaseof chest radiographs with free-text re-ports. Scientific Data 6, 317, 10.1038/s41597-019-0322-0 (2019).

    [12] Nam, J. G. et al. Development andvalidation of deep learning–based au-tomatic detection algorithm for ma-lignant pulmonary nodules on chestradiographs. Radiology 290, 218–228, 10.1148/radiol.2018180237 (2019).PMID: 30251934, https://doi.org/10.1148/radiol.2018180237.

    [13] Schwartz, A. B. et al. The accu-racy of mobile teleradiology in theevaluation of chest x-rays. Journalof Telemedicine and Telecare 20,460–463, 10.1177/1357633X14555639(2014). PMID: 25322696, https://doi.org/10.1177/1357633X14555639.

    [14] Goost, H. et al. Image and diagno-sis quality of x-ray image transmissionvia cell phone camera: A project studyevaluating quality and reliability. PloSone 7, e43402, 10.1371/journal.pone.0043402 (2012).

    [15] Vassallo, D., Buxton, P., Kil-bey, J. & Trasler, M. The firsttelemedicine link for the britishforces. BMJ Military Health 144,125–130, 10.1136/jramc-144-03-02(1998). https://militaryhealth.bmj.com/content/144/3/125.full.pdf.

    [16] Kurakin, A., Goodfellow, I. & Bengio,S. Adversarial examples in the physicalworld. ICLR Workshop (2017).

    [17] Rajpurkar, P. et al. CheXpedition:Investigating Generalization Challengesfor Translation of Chest X-Ray Algo-rithms to the Clinical Setting. arXive-prints arXiv:2002.11379 (2020). 2002.11379.

    [18] Hendrycks, D. & Dietterich, T. G.Benchmarking neural network robust-ness to common corruptions and per-turbations. CoRR abs/1903.12261(2019). 1903.12261.

    [19] Goodfellow, I. J., Shlens, J. & Szegedy,C. Explaining and harnessing adversar-ial examples (2014). 1412.6572.

    [20] Liang, P. et al. CodaLab (2020, accessedJuly 12, 2020). https://codalab.org.

    327

    1901.0744110.1038/s41597-019-0322-010.1038/s41597-019-0322-010.1148/radiol.2018180237https://doi.org/10.1148/radiol.2018180237https://doi.org/10.1148/radiol.201818023710.1177/1357633X14555639https://doi.org/10.1177/1357633X14555639https://doi.org/10.1177/1357633X1455563910.1371/journal.pone.004340210.1371/journal.pone.004340210.1136/jramc-144-03-02https://militaryhealth.bmj.com/content/144/3/125.full.pdfhttps://militaryhealth.bmj.com/content/144/3/125.full.pdf2002.113792002.113791903.122611412.6572https://codalab.org

    Background & SummaryMethodsAcquiring Natural Photos of Chest X-RaysAutomated Capture of Nokia10k datasetManual Capture of iPhone1k dataset

    Generating Synthetic Photographic Transformations of Chest X-RaysValidation and TestTechnical ValidationData Access

    Conclusion