Roof Damage Assessment using Deep LearningMahshad Mahdavi Hezaveh

Chester F. Carlson Center for ImagingRochester Institute of Technology

54 Lomb Memorial DriveRochester, NY 14623-5604, USA

E-mail addresses: mxm7832@rit.edu

Christopher KananChester F. Carlson Center for Imaging

Rochester Institute of Technology54 Lomb Memorial Drive

Rochester, NY 14623-5604, USAE-mail addresses: kanan@rit.edu

Carl SalvaggioChester F. Carlson Center for Imaging

Rochester Institute of Technology54 Lomb Memorial Drive

Rochester, NY 14623-5604, USAE-mail addresses: salvaggio@cis.rit.edu

Abstract—Industrial procedures can be inefficient in terms oftime, money and consumer satisfaction. the rivalry among busi-nesses, gradually encourages them to exploit intelligent systems toachieve such goals as increasing profits, market share, and higherproductivity. The property casualty insurance industry is not anexception. The inspection of a roof’s condition is a preliminarystage of the damage claim processing performed by insuranceadjusters. When insurance adjusters inspect a roof, it is a timeconsuming and potentially dangerous endeavor. In this paper,we propose to automate this assessment using RGB imagery ofrooftops that have been inflicted with damage from hail impactcollected using small unmanned aircraft systems (sUAS) alongwith deep learning to infer the extent of roof damage (see Fig. I).We assess multiple convolutional neural networks on our uniquerooftop damage dataset that was gathered using a sUAS. Ourexperiments show that we can accurately identify hail damageautomatically using our techniques.

I. INTRODUCTION

One of the most damaging phenomena for rooftops thatresult in billions of dollars of property casualty insuranceclaims each year results from hail. To mitigate this problem,owners of homes in hail prone geographic locations often carryproperty insurance that can cover the repair costs to theirroof should it be severely damaged by a storm. Replacinga roof is expensive, but if damage is minor and does notundermine the integrity of the roof to seal out the elements,then the roof can be repaired for far less cost. Determiningthe severity of damage is therefore imperative to keepingpremium costs down, while at the same time, making surethat those rooftops that do need repaired are done so in afast and efficient manner, keeping the homeowner protectedand preventing further damage from occurring. After severestorms, insurance companies are deluged with storm damageclaims. Roofers will often prey on homeowners after stormsand claim that their roof is a total loss, even if the repairsare minor [12]. Detecting the true severity of the damage todetermine if the roof should be replaced, or merely repaired,requires sending inspectors to each home. This is expensive,time-consuming, and dangerous, especially after severe storms[2]. In this paper, we propose to automate this inspectionprocess using small unmanned aircraft systems (sUAS) anddeep learning techniques to classify this damage from RGBimagery.

To create a fully automated inspection system, robust defectdetection and classification algorithms are required. We dothis using deep convolutional neural networks (CNNs). Theseapproaches have achieved state-of-the-art performance on nu-merous computer vision problems, and now rival humans attheir efficacy at image classification. We assess their suitabilityfor roof damage detection.

Our Contribution: In this paper, we make three majorcontributions: 1) we propose an automatic system for roofdamage detection based on deep learning methods; 2) weintroduce the first dataset for assessing rooftop hail damage,which we will make publicly available (see Fig. 2); and 3)we recommend a collection methodology to provide imagerywith the geometric fidelity and spatial resolution necessary toprovide useful and accurate damage characterization.

Fig. 1. The proposed algorithm takes high-resolution RGB images acquiredwith a small unmanned aircraft system (sUAS) as input and processes throughthe network to detect and localize damaged spots

The remainder of this paper is organized as follows: Section2 describes the related work, Section 3 provides the details ofour work flow including the problem and proposed approach,Section 4 reviews the dataset and algorithms, and Section 5discusses the results obtained.

II. BACKGROUND

Little research has been devoted to automated roof condi-tion assessment using machine learning or computer visiontechniques. However, there has been a limited amount of978-1-5386-1235-4/17/$31.00 c© 2017 IEEE

Approach MethodStatistical 1. Histogram properties

2. Co-occurence matrix3. Local binary pattern

4. Autocorrelation5. Registration-based

Filter Based 1.Spatial domain filtering2. Frequency domain analysis

3. Joint spatial/spatial-frequency

TABLE ILIST OF TEXTURAL DEFECT DETECTION METHODS.

research in automated building damage assessment and surfaceinspection.

A. Building Damage Assessment

Building damage assessment is the closest work to whatwe propose in this study. Algorithms for building damageassessment falls into two categories: methods working withpre- and post-event data and cases in which only the post-event data is available. Although typically using data fromboth before and after an event yields more accurate results, inmany cases pre-event data is not available [3].

B. Visual surface inspection as a texture analysis problems

Surface defects are divided into local textural irregularitiesand global divergence of color or texture. According to [4],methods used for visual inspection addressing texture abnor-malities fall into four categories: statistical, structural, filter-based, and model-based methods. Statistical and filter basedapproaches are the most common ones in visual inspection.Table 1 indicates some of the texture analysis techniques thatfall into these two categories [1].

C. Visual inspection via supervised classification

Supervised classification has been extensively used in visualinspection. This method has been performed reasonably wellwhen both normal and defective samples are provided andwell-conditioned. However, if defective samples are unavail-able, novelty detection will be more advantageous. The k-nearest neighbor (kNN) method is a simple supervised learn-ing algorithm. In previous studies, ceramic tiles have beenclassified using kNN based on chromatic features extractedfrom individual channels. The feature extraction method usedis critical for classification of data, as the learned featurescan increase the separation between classes which leads tobetter classification. Linear feature extraction methods, suchas principal component analysis (PCA) or linear discriminantanalysis (LDA), are not able to characterize the nonlinear rela-tionships between inputs and outputs. Despite some methodslike manifold learning methods, which are supposed to modelthe non-linearity of data, deep learning methods are widelyutilized for this purpose. Deep learning methods are capableof processing large scale dataset and learning the nonlinearfeatures by constructing a representation in a hierarchicalmanner with increasing the order of abstraction from lowerto higher levels of neural network [5]. Methods exploiting

deep neural nets are frequently used for aerial image analysisas well. There are many works that attempt to use CNNsfor object detection and localization in large-scale remotesensing images [13],[14]. According to those convolutionalneural networks performs as powerful tools for most of theimage processing tasks on aerial images.

III. METHODS

In this section, we provide details of the algorithm andtraining process for our proposed method. We evaluate CNNmodels that are trained from scratch to assess rooftop damageas well as using CNNs pre-trained on ImageNet [6]. The pre-trained CNNs have a much larger amount of data used fortraining, enabling them to have rich feature hierarchies. Usingpre-trained CNNs is the standard approach that is now used forusing CNNs with small-scale datasets. However, it is possiblethat only the lower layers are useful for assessing rooftopdamage, because the lower layers detect texture features thatmay be more transferable to this task. This suggests that forthis task it may be possible to do equally as well by traininga CNN from scratch.

We trained a simple CNN from scratch on our dataset. Forthis purpose, we used a very small CNN having 3 convolu-tional layers, 2 fully-connected layers and few filters per layer,along with data augmentation and batch normalization.

Features are then extracted from the pre-trained networkto examine if the general features acquired with pre-traineddeep neural networks can outperform the features which arelearned specifically on our dataset via a shallower network.To study the effect of having more convolutional layers in thepre-trained CNNs, classification performances of different pre-trained CNNs (VGG-f, VGG19, and ResNet50) are comparedfor classifying the RGB dataset [7,8,9]. For further detailson the training process of the proposed damage detector, seeSection 4.3.

Being able to detect hail-damaged spots in small patches ofroof shingles depicts that neural networks can see and learn thedifference between uniform and hail-damaged samples. Thissuggests that the proposed method can be further improved byusing better quality, more realistic, and larger image trainingsets. In order to conduct an experiment which more closelyresembles a real, in-field inspection scenario, we decided todo damage detection and localization on wide-area views ofrooftops using standoff sUAS-collected and handheld imagery.It should be noted that only hail damaged samples wereexamined in this study due to the availability of data andbased on the recommendation of property casualty insurancespecialists as these the most relevant roof malady that theyencounter. Implementing our algorithm on wide-area rooftopimages, we are able to create heat-maps that indicate theimpact points and can be used as visualization aids to roofinspectors. It is possible to get the network response (heat-map) for the whole input image in one call to code byconverting fully-connected layers to convolutional layers [11].

The effect of resolution was also examined regarding its’ onthe performance of networks to determine, for future imagery

collections, the optimum distance at which one can fly a sUASto obtain the proper balance between spatial resolution andareal coverage. A collection methodology is suggested basedon this experiment.

IV. EXPERIMENTS

The methods applied for hail-damage detection uniquelydepend on the type of data available. Damage classificationwas carried out using two different groups of samples basedon the distinct information we could extract from each. Inthe following sections, datasets and detection methods corre-sponding to each data type are described.

Fig. 2. Hail damaged roof shingles cropped and ready to be analyzed.Different classes and colors of samples are shown: (left to right) ice, uniform,ice, steel and ice.

A. High-resolution RGB images

RGB images of damaged and intact roof shingles werecollected at the Insurance Institute for Business & HomeSafety (IBHS) Research Center in Richburg, SC, USA [10].There are different types of shingle samples in terms ofdamage types, colors, damage size, producing manufacturer,damage location, etc. These products were coded based on theproducing manufacturer and materials. These roof panels weredamaged by 4 different sizes of either hail or steel balls. In thefirst run, two out of four different ball sizes, the most severeand the least severe, were considered in the analysis of thehigh-resolution RGB imagery. Additionally, these images arecropped into small patches and labeled (locations classifiedas damaged or undamaged). This allowed for the creation ofa larger dataset to be used for training of a neural networkand fine-tuning a deep pre-trained network. If the croppedimage has some portion of damage in it, it is labeled asdamaged, and if not it is labeled as uniform. That being done,1678 samples were assembled that were randomly dividedinto 1222 training and 456 test samples. Before feedingthe samples to the network, two pre-processing steps werenecessary. First, removing the documenting chalk arrows fromthe shingles, which were there to indicate the impact pointson the roof panels, needed to be accomplished. This was doneusing standard image editing techniques (see Fig. 3). Second,cropping the images into small pieces in order to get biggerdataset. The cropping process, which is referred to as dataaugmentation, resulted in having more damaged samples thanundamaged ones. By adding more uniform images, we wereable to make sure that the dataset is balanced and the results

are not biased. Fig. 2 shows an illustrative portion of finaldataset.

Fig. 3. A damaged shingle before (left) and after (right) removing the chalkarrows

B. Wide rooftop imagery

Although using pre-trained networks for detecting damagedshingles on the small patches of high-resolution RGB imageswas quite successful, these images are not similar to realimages taken of damaged rooftops in the field. The spatialresolution, illumination consistency, and quality of focus ofthese experimental sample photographs were far superior toany data that we have received from real-field collections.To address this inconsistency, we decided to go even furtherand try to do damage detection with deep learning methodson real images that were gathered during insurance claimsinspections. These images have two major differences com-pared to our previous RGB dataset. First, there are multipledamaged spots on each image, whereas the patches had onlyone damaged spot per photograph. Second, these new dataare, in general, taken from a larger standoff distance than theprevious RGB images, resulting in a coarser spatial resolution.Some other challenges regarding classifying damage on widerooftop images are occlusions caused by trees, penetrationscasting shadows on the roof, or debris collected on the roof.Moreover, these images are very limited by nature since theyare taken from real damaged houses. Damaged tiles wereground truthed on this dataset, training samples were annotatedby defining a bounding box for each damage spot. We had fewof this examples to learn from, for a classification problemthat is far from simple. Although it could be challenging,it is also a realistic machine learning problem: in a lot ofreal-world applications, even small-scale data collection canbe extremely expensive or sometimes near-impossible (e.g. inmedical imaging). Therefore, being able to make the most outof very little data is of great value.

C. Training and Evaluation

To work with limited training set in deep learning fieldsthere are couple of options: training a small network fromscratch (as a baseline), using the features of a pre-trainednetwork, or fine-tuning the top layers of a pre-trained network.For the first group of our samples, very high-resolution RGBimages, we first trained a convolutional neural network todetect damages. Our damage detector consists of 6 layerswhere the first 3 layers are convolutional and the last 3 layersare fully-connected. Later, we attempted a transfer learning

approach to extract features knowing that deep learning modelsare by nature reusable for tasks other than the one they trainedspecifically for. We used both an off-the-shelf classifier and athree fully-connected layer network to do the classificationtask on these features. Furthermore, since we are assumingthat learned features, except for the top layers, are not specificto a single RGB dataset, we also explored fine-tuning a pre-trained CNN. Thus, we just removed the last layer and added anew output layer compatible with our new dataset and then re-trained the entire network. The pre-trained networks which areused in this study are VGG-f, VGG19 and ResNet50 (commonnetworks used by the computer vision community). Deepernetworks are utilized to see the effect of a more powerfulmodel on the results. We are fully aware that the high detectionaccuracy on this dataset extremely relies on the resolution ofinput images. To understand how far we can fly a UAS withoutsacrificing the accuracy, we studied the changes in accuracyversus the distance.

Afterwards, we implemented our trained algorithm on realsamples obtained from real damaged houses. On the assump-tion that these images have multiple damage spots, not onlyshould the proposed method be capable of detecting damagedspots, but it also should be able to do damage localization.Consequently, we would be able to map damage patterns onthe rooftop surfaces by locating the relative position of damagespots, which is one of the key factors that is desired to beunderstood from visual inspections. With this intention, weproduced heat-maps for wide-area images of rooftops to beused as a detection aid in roof inspections. To preserve thespatial information, we removed the average pooling layerfrom the network, and converted the fully connected layersinto the convolutional layers with (1,1) filters [11]. Damagelocalization can be further improved by using a bounding boxregression module.

V. RESULTS AND DISCUSSION

The spatially varying nature of these damaged spots arewhat allows our human visual system to detect them soeasily. Training a simple CNN from scratch on a small imagedataset will still yield reasonable results, without the need forany custom feature engineering (Table 2). A more improvedapproach would be to leverage a network which is pre-trainedon a large dataset. All the other methods on Table 2 exploita pre-trained CNN and they reach to a very good accuracyas we go deeper, since the spatial variability of the targets ofinterest are included in the identification process. At first, wetried a quite shallow pre-trained CNN (VGG-f) to see whetherdeeper and trained networks can actually make a difference.Getting reasonable mean-per-class accuracy motivated us to trydeeper networks. To this point, ResNet50 is the best algorithmto detect roof shingle damage. However, deeper networksprobably can do better on this dataset, so we plan to go evenfurther. It is interesting to notice that even though fine-tuningVGG-19 provides us with better result than the ImageNet pre-trained version, it could not perform as well as ResNet50which does not learn any weights based on our dataset yet

has more layers. This motivated us to visualize what deepneural networks learned after training. To this end, we triedto visualize the activation of networks during a forward pass.Fig. 4 is showing activations on the first CONV layer (left),and the 5th CONV layer (right) of a trained VGG-19 lookingat a damaged spot of a roof. Notice that some activation mapsmay be all zeros for many different inputs, which are basicallydead filters, and can happen as a result of high learning rates.

Fig. 4. Activations on the first CONV layer (left), and the 5th CONV layer(right) of a pre-trained VGG-19 looking at a damaged roof shingle. Everybox shows an activation map corresponding to some filter. The activations aresparse (in this visualization zero values are shown in black).

Based on our observation, pre-trained networks can benotably successful in detecting damages when they are fedwith very high-resolution images. This raises the questionwhether the images we got from sUAS meet this requirement.To investigate this, first we resized the high-resolution RGBimages we had to create images with coarser resolution.

Fig. 5. The effect of resolution on the performance of VGG-19 to figure outthe optimum distance

Fig. 5 shows the detection accuracy using VGG-19 on theresized datasets. The pixel size was changed to create imageswith lower resolutions and then resized to 224 × 224 andprovided to a CNN (VGG-19) for damage detection. Theoriginal image was blurred using spatial convolution with an × n rectangular filter, and the resulting response was thensubsampled on a n × n grid to simulate a coarser resolutionsensor. The accuracy significantly dropped as the resolution(pixel size) was became coarser. Surprisingly, we can observea slight increase in the accuracy before the major drop. Thismainly happens due to the noisy nature of the input images.As the image starts to get blurry, it becomes smoother andsome of the noises will be faded, so damage spots turn into

TABLE IIDAMAGE DETECTION RESULTS ON ROOF SHINGLES USING PRE-TRAINED

CNNS IN COMPARISON WITH TRADITIONAL CLASSIFICATION OFSPECTRAL BANDS. LR IS USED AS CLASSIFIER IN ALL CASES.

Methods Mean-per-class accuracy(%)Trained network 83.44

VGG-f 82.67VGG-19 86.18ResNet50 91.44

Fine-tuned VGG-19 87.71Fine-tuned ResNet50 91.92

TABLE IIIMEAN-PER-CLASS ACCURACY (%) USING DIFFERENT CLASSIFIERS.

Methods Fully Connected NN SVM LRVGG-19 80.04 83.55 86.18ResNet50 91.01 87.93 91.44

easier targets for detection. Thus, too much resolution is alsonot beneficial to the goal of this study.

Second, we implemented the algorithm on real imageswhich are gathered during insurance claims inspections. Anexample of a heat-map is shown in Fig. 6. Each point in theheat-map shows the CNN response, the probability of havinga damaged spot, for its corresponding 224 × 224 region in theoriginal image. Note that, we removed the classifier layer andfound that the network output are informative enough for thetask of damage detection and roof inspection. Comparing thebounding boxes in annotated dataset to the high probabilityareas in the corresponding heat-maps, we can see that thenetwork is capable of detecting and localizing damages. Spotswhich are more obvious in terms of the damaged size, contrastand more importantly resolution are more likely to be detectedby the system. In Fig. 6, damages on the lower part of the im-age which are closer to the camera are mostly detected whereasthe ones at the top which are quite far are completely ignoredby the network. That means if the network was providedwith a dataset in which the images have perpendicular angleview, resulting in uniform resolution throughout the image, wecould create more accurate heat-maps. Thus, we recommend acollection methodology to provide imagery with the geometricfidelity and spatial resolution necessary to provide usefuldamage characterization. Imagery should be collected with theplatform at a constant distance from the roof deck at all time,viewing the roof surface from a perpendicular vantage point.The resolution will then be constant, and perspective distortionwill be minimized.

In order to gather more imagery so that our training canbecome more robust, we are in the process of developingautomated flight planning algorithms to fly our UAS systemsat a constant standoff distance from a rooftop utilizing imageryor LiDAR derived roof models. This will provide us witha uniform spatial resolution dataset that it is believed willperform similarly to those experimentally gathered samplesat the IBHS Research Center (with classification accuracies at

the 90% level).

Fig. 6. left) an example image of a wide-view damaged roof. The annotationbounding boxes are shown for each damaged spot. right) heat-map for theresponse of fine-tuned VGG-19 on High-res RGB images

VI. CONCLUSION

In this paper, we proposed a damage detector based ondeep learning methods. This research is motivated by replacingmanual inspection and developing automatic systems for roofconditions assessment. The proposed method is capable ofdetecting damages with the accuracy as high as 91.92% onpatches of high-resolution samples which is a significantimprovement over baseline. We can also provide inspectorswith real-time heat-maps indicating the damaged spots to beused as visualization aids. This is of great value since it makesthe whole process much faster, especially for the case of largearea inspections, and it only requires one proper wide-viewimage of the roof. We took advantage of pre-trained networksand we compared three different approaches to prove thatroof damage assessment using deep CNNs is efficient, easyto implement and significantly more accurate. Finally, wesuggested a collection method to get appropriate images byflying UAS systems.

ACKNOWLEDGMENT

The authors would like to thank Nina Raqueno and TimothyBausch at the Rochester Institute of Technology for theirassistance with data collection, and Tanya Brown-Giammancoand Heather Estes at Insurance Institute for Business & HomeSafety for their cooperation in providing the experimental haildameged shingles.

REFERENCES

[1] Kumar, A. (2008). Computer-vision-based fabricdefect detection: A survey. IEEE transactions on industrialelectronics, 55(1), 348-363.

[2] Samsudin, S. H., Shafri, H. Z., & Hamedianfar, A.(2016). Development of spectral indices for roofing materialcondition status detection using field spectroscopy andWorldView-3 data. Journal of Applied Remote Sensing,10(2), 025021-025021.

[3] Dong, L., & Shan, J. (2013). A comprehensive reviewof earthquake-induced building damage detection with remotesensing techniques. ISPRS Journal of Photogrammetry andRemote Sensing, 84, 85-99.1

[4] Xie, X. (2008). A review of recent advances in surfacedefect detection using texture analysis techniques. ELCVIA:electronic letters on computer vision and image analysis,7(3), 1-22.

[5] Xing, C., Ma, L., & Yang, X. (2015). Stacked denoiseautoencoder based feature extraction and classification forhyperspectral images. Journal of Sensors, 2016.

[6] Pan, S. J., & Yang, Q. (2010). A survey on transferlearning. IEEE Transactions on knowledge and dataengineering, 22(10), 1345-1359.

[7] He, Kaiming, et al. ”Deep residual learning forimage recognition.” Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition. 2016.

[8] Simonyan, K., & Zisserman, A. (2014). Very deepconvolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556.

[9] Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman,A. (2014). Return of the devil in the details: Delving deepinto convolutional nets. arXiv preprintarXiv:1405.3531.

[10] The Insurance Institute for Business and Home SafetyResearch Center. Retrieved from http://disastersafety.org/ibhs-research-center

[11] Iandola, F., Moskewicz, M., Karayev, S., Girshick, R.,Darrell, T., & Keutzer, K. (2014). Densenet: Implementingefficient convnet descriptor pyramids. arXiv preprintarXiv:1404.1869.

[12] Petty, S. (2013) Synthetic storm damage (fraud) toroof surfaces. Forensic Engineering: Damage Assessment forResidential and Commercial Structures.

[13] evo, I., & Avramovi, A. (2016). Convolutional neuralnetwork based automatic object detection on aerial images.IEEE Geoscience and Remote Sensing Letters, 13(5), 740-744.

[14] Long, Y., Gong, Y., Xiao, Z., & Liu, Q. (2017).Accurate Object Localization in Remote Sensing ImagesBased on Convolutional Neural Networks. IEEE Transactionson Geoscience and Remote Sensing, 55(5), 2486-2498.