Linear Measurement Accuracy of DJI Drone Platforms and ... · determine the optimum flight settings for producing maps that can deliver accurate 2D ... , sidelap, and frontlap. Drone

LinearMeasurementAccuracyofDJIDronePlatformsandPhotogrammetry

BYANDYPUTCH,PROJECTMANAGER

-2-

DroneDeploy|1045BryantStreet,Suite300|SanFrancisco,CA94103|dronedeploy.com

TableofContents

Abstract......................................................................................................................3

Introduction................................................................................................................4GoalsofThisStudy..............................................................................................................4ExploringBestPracticesforPhotogrammetricDataCollection............................................4DroneModelsUsedinThisStudy........................................................................................4CameraSensors...................................................................................................................5

Background.................................................................................................................6StandardDeviationofLinearMeasurementsfromPhotogrammetry...................................6GuidelinesforProducingAccurateAerialMapswithDrones...............................................6

Methodology...............................................................................................................6EstablishingaGroundControlSystem.................................................................................6CollectingAerialData..........................................................................................................7CapturingGroundControlPointData..................................................................................7

KeyFindings................................................................................................................9OnAverage,LinearMeasurementsareAccuratetoWithin1.1%.........................................9HigherResolutionCamerasReducedAverageMeasurementErrorby0.33%.......................9FlyingLowImprovedMeasurementAccuracyby0.35%......................................................10Together,CameraResolutionandAltitudeHalvedAverageMeasurementError................11VisualizingMapResolutionbyAltitude..............................................................................12GroundControlPointsImprovedMappingAccuracy10x....................................................12

Conclusion.................................................................................................................13UseHighResolutionCamerasforIncreasedMeasurementAccuracy..................................13FlyMissionsatLowAltitudeforHigherMeasurementAccuracy.........................................13GroundControlPointsProducetheMostConsistent,AccurateResults..............................14AltitudeMattersWhenMappingwithGroundControlPoints............................................14

AdditionalDroneDeployResources...........................................................................15AGuidetoUsingGroundControlPointswithDroneMappingSoftware...................................15DecidingIfYourDroneMappingProjectNeedsGCPs..............................................................15CapturingGroundControlPoints(GCPs).............................................................................15GCPRequestChecklist........................................................................................................15DroneDeployIn-BrowserGCPTaggingWorkflow................................................................15

Appendix...................................................................................................................16

-3-


AbstractDronesarequicklybecomingthego-tomeansforthecollectionofon-demandaerialimageryacrossindustriessuchasconstruction,surveying,insurance,andmining.Photogrammetryallowsustodigitalizethephysicalworldandusethatdatatosolvesomeoftoday’stoughestchallenges.Intheprocess,iteliminatestheneedtomanuallycapturedataindangerousareassuchasindustrialjobsites,quarries,roofs,andotherelevatedstructures.Photogrammetryreliesoncamerastomeasurereal-worldobjectsandturn3Dspaceinto2Dmaps.Thisrequiresphotogrammetrysoftwaretoidentifythelocation,orientation,andmovementofthecameratocalculatethepositionofthree-dimensionalpoints.Withoutaphysicalrelationshipbetweenthecameraandthesubjectbeingmapped,itisdifficulttoquantifytheaccuracyofphotogrammetricoutputs.Therefore,toproperlyquantifyerrortheoutputsmustbeground-truthedagainstknownvalues.Camerasarephysicaldevicesthatintroduceerrorsintothedatacaptureprocess.Acamera’s2Dimageisn’tatruerepresentationofthephysicalworld.Thisisbecausecamerabodiesandlensescannotbemanufacturedperfectly,whichcreateerrors—suchasdistortedlinesinaphoto—thatphotogrammetrysoftwaremustcompensatefor.Theseerrors,evenwhencompensatedfor,cancreateinaccuraciesinlinearmeasurementsmadeonaprocessedmap.InthisstudyDroneDeployhasinvestigatedwaystoimprovemappingaccuracyandputtogetherasetofbestpracticestobeusedwhenmakinglinearmeasurements.Totesttheaccuracyofthemeasurementsmadeusingmapsgeneratedfromdatacapturedwithindustry-standardDJIdroneplatforms,DroneDeployestablishedagroundcontrolsystemontheroofofitsoffice.Then,morethan80flightswerelogged—eachexploringdifferentflightaltitudes,cameras,andphotooverlapsettings.TheimagescollectedwerethenprocessedinthecloudusingtheDroneDeploymapengine.Thedatasetswereanalyzedandusedtocalculatetheaveragemarginoferrorformeasurementsofknowncontrollengths.Usingtheresults,DroneDeploydeterminedthatusingaperformancecameraandflyingatlowaltitudewithhighimageoverlapproducedmapswiththebestlinearmeasurementaccuracy.

-4-


IntroductionGoalsofThisStudyWhenitcomestodronedata,thereisonemetricthatmattersmost:accuracy.Overthelastfewyears,asdronetechnologyhasadvanced,sotoohastheabilityfordronemapstobehighlyaccurate.Butphotogrammetricaccuracyispoorlyunderstood—particularlyamongthosewhoarenewtoaerialmappingorjustenteringthecommercialdroneindustry.Tohelpshedlightonthistopicandprovidebusinesseswithaccesstoreliabledata,DroneDeployconductedanexperimenttoassesstheoveralllinearmeasurementaccuracyofdronemapscapturedbystandardDJIdroneplatformsandprocessedinthecloudusingtheDroneDeploymapengine.Thegoalofthisstudywastoprovidequantitativedataandguidelinesforwhatuserscanexpectfromlinearmeasurementsonmodelscreatedwithcloud-basedphotogrammetrysoftware.Thisstudyalsoincludedtherelativeaccuracyofdatacollectedandprocessedwithandwithouttheuseofgroundcontrolpoints(GCPs).TheresultsprovideinsightintotheaccuracyonecanexpectfromeachmethodanddatatoaidindecidingwhetheraDroneDeploymappingmissionrequirestheuseofgroundcontrolpoints.ExploringBestPracticesforPhotogrammetricDataCollectionThispaperprovidesguidanceonhowtoimprovethequalityofinputdatacollectedusingDJIdronemodelsandcameraswhenproducingmapsontheDroneDeployplatform.Thisstudy,comparesimageresultsfromawiderangeofhardwareoptionstodeterminetheoptimumflightsettingsforproducingmapsthatcandeliveraccurate2Dmeasurements.Theresultscanserveasguidelineswhenselectingflightparametersformappingmissionsincluding:altitude,sidelap,andfrontlap.DroneModelsUsedinThisStudyTheDJIdronemodelschosenforthisexerciseincludedtheMavicPro,Phantom3Pro(P3P),Phantom4Pro(P4P),Inspire1(withDJIX3Camera),Inspire1Pro(withDJIX5Camera),andInspire2(withDJIX4sCamera).TheotherDJIPhantom3modelsandtheDJIPhantom4wereexcludedbecausethesemodelssharethesame12MPcamerasensorincludedinthePhantom3Pro.DroneDeploychosetoconductthisstudywithDJIdronesbecausetheyarethenumberonecameraandhardwareplatformusedbyprofessionalsinthecommercialdroneindustry.AsthesedronesaretheprimarychoiceofDroneDeploycustomersacrossindustriesincludingconstruction,agriculture,inspection,surveying,andmining,thisdatasetprovidesDroneDeploy’slargestcustomerbaseinsightintotheaccuracythatiscapablewiththeirhardwaresolutionofchoice.

-5-


CameraSensorsTherewereavarietyofsensorsusedtocollectdatainthisstudyrangingfrom12MP,1/2.3”to20MP,1”.ThisselectionrepresentsthefullrangeofDJIcamerasavailableonthemarkettodayandincludesbothmechanicalandelectroniccamerashutters.ThesensorspecificationsandcharacteristicscanbefoundinFigure1andTable1.

Figure1.CameraSensorSizes

Table1.CameraSensorSpecifications

Drone Megapixels SensorSize FieldofView ShutterType

P3P&I1(X3) 12MP 1/2.3” 94° Rolling

MavicPro 12MP 1/2.3” 78° Rolling

P4P&I2(X4s) 20MP 1” 84° Mechanical

I1Pro(X5) 16MP 4/3” 72° Rolling

-6-


BackgroundStandardDeviationofLinearMeasurementsfromPhotogrammetryPubliclyavailabledatarelatedtothestandarddeviationoflinearmeasurementsfromphotogrammetrymodelsislimited.Existingstudiestypicallylookattheaccuracyofstockpilevolumes,anddoesnotincludedataspecifictolinear,point-to-pointmeasurements.Thismakesitincrediblydifficulttounderstandthemarginoferroronecanexpectfromdistanceandareameasurements.Inaddition,canonicalresearchonphotogrammetricaccuracyfocusesontheuseofGCPcheckpoints.ThismeanstheaccuracyismeasuredasadifferencebetweenthegeographicpositionofacheckpointasrecordedbyprecisionGPS,andthetheoreticalpositionofthatcheckpointascalculatedbythephotogrammetrysoftware.Thismethodhasbeenfavoredbecauseitisthebestwaytoisolateexperimentalvariables,andconsideredtobethemostscientificallyrigorous.However,becauseacheckpoint’saccuracycanonlyapplytoasinglepointinspace,thedatadoesapoorjobofevaluatingtheaccuracyoflinearmeasurements.GuidelinesforProducingAccurateAerialMapswithDronesTodate,therearefewresourcesthatprovideguidelinesregardingthecaptureandprocessingofaerialdataforcommercialuse.Thosethatdoexistdonotprovideharddatatobackupsuggestions,whichleavesmappingprofessionalswithlittleevidencetoguidedatacapturemethods.

MethodologyEstablishingaGroundControlSystemTotesttheaccuracyoftwo-dimensionalmeasurements,DroneDeployestablishedagroundcontrolsystemontheroofofits1045BryantStreetoffices.ThisisshowninFigure2.Inadditiontogroundcontrolpoints,thiscontrolsystemdefinedcontroldistanceswithknownlengths.Thesecontroldistancescouldthenbecomparedwiththeoreticalmeasurementsmadewithphotogrammetrysoftware.

-7-


Figure2.GroundControlSystem

Thegroundcontrolsystemusedthree1’x1’checkeredmarkersthatwereplacedinalargerightangled“L”shapealongthebuilding'sprimaryaxis,withafourthmarkerwasaddedasalengththatbisectsthe“L”axis.ThecontroldistancesbetweeneachmarkerweremeasuredmanuallyusingametaltapemeasureandcheckedwithaLeicaLaserDistanceMeter.CollectingAerialDataAerialdatawasthencollectedovermorethan80individualflightsusingDJIdronesandtheDroneDeploymobileapplication.Atotalof4flightswereflownat66ft.,100ft.,200ft.,and400ft.abovethesurfaceoftheroofusingeachDJIdronemodel,andthenidenticallyprocessedwiththeDroneDeploymapengine.Slightvariationsinlighting,flightpath,andaltitudewereincludedtosimulaterealworldvariationincollectedmappingdata.CapturingGroundControlPointDataTocollectthegroundcontrolpointdata,5additionaltargetsmeasuring3”x3”wereplacedontheroof.ThegeographiclocationsofthecenteroftheselargertargetswerethenrecordedwithanEmlidReachRSandTrimbleCatalyst.Theseglobalnavigationsatellitesystem(GNSS)receiverscanbeseeninFigures3and4.AprocessedaerialmapofthegroundcontrolsystemsetupcanbeseeninFigure5.Oncethegroundstationwassetup,aDJIPhantom4Prowasusedtomaptheroofoncemoreatelevationsof66ft.,100ft.,200ft.,and400ft.

-8-


Tobetterunderstandthestandarddeviationandaveragemarginoferror,astatisticalsamplewastakenusingthePhantom4Pro.FlyingidenticalmissionplanswithGCPsatanelevationof66ft.,atotalof6flightswerecompleted.Similarly,afinaldatasetwascreatedwithoutGCPsbyflyingthePhantom4Proat66ft.anadditional10times.Thismethodwasadoptedtoapproximatethevariationpresentinlinearmeasurementaccuracywhenmappingonelocationunderthesameconditions.

Figure3.EmilReachRS Figure4.TrimbleCatalyst

Figure5.AerialViewofGroundControlSystem

-9-


KeyFindingsOnAverage,LinearMeasurementsareAccuratetoWithin1.1%WhencomparingthedatacollectedacrossalltheDJIdronemodelsincludedinthestudy,theaveragemarginoferrorforallflightswasfoundtobe1.1%.Bythiswemeanthevalueoftheaveragemeasurementerrorwas1.1%ofthevalueofthecontrollength.Thus,whenmeasuringa100ftlengththeaverageerrorwouldbe1.1ft.TheseresultscanbefoundinTable2.

FlightAltitude(ft.) Measurement1(40ft)Error(ft.)

Measurement3(50ft)Error(ft.)


66 0.25 0.27 0.54

100 0.35 0.39 0.85

200 0.56 0.79 1.34

400 0.67 0.88 1.44Avg.MeasurementError(ft.) 0.45 0.57 1.04

Avg.MeasurementError(%) 1.12% 1.14% 1.04%

Table2.MarginofErrorWithoutGCPs

HigherResolutionCamerasReducedAverageMeasurementErrorby0.33%Thedatashowsaclearcorrelationbetweenhigherresolutionimageryandhighlyaccuratemaps.Forexample,themapsproducedwiththe12MPP3Pcamerayieldedanaveragemarginoferrorof0.72ft.acrossmeasurementstaken,whereasthemapsproducedwiththe20MPP4Pcamerayieldedanaveragemarginoferrorof0.46ft.Bycomparingtheaveragemarginsoferrorforeachcontrollengthasapercentage,itwasfoundthattheaveragemeasurementerrorfor20MPcamerawas0.33%lessthanthe12MPcamera.ThisdatacanbeseeninTable3.While0.33%mynotseemsignificant,rememberthatwouldbea0.33ft.(~4in.)reductionoferrorfora100ft.measurement.Theaveragemeasurementerrorsbetweenthe12MPmapsand20MPmapsshowedthaterrorwouldincreaseproportionallytocameraresolutionwithacorrelationcoefficientof0.33.Between12MPand16MPmaps,thiscorrelationcoefficientwascalculatedat0.56.

-10-


Measurement 1

(40ft) Error (ft.) Measurement 2

(100ft) Error (ft.) Measurement 3 (50ft) Error (ft.)

Flight Altitude (ft.) P3P (12MP)

P4P (20MP)

P3P (12MP)

P4P (20MP)

P3P (12MP)

P4P (20MP)

66 0.34 0.19 0.73 0.46 0.34 0.245

100 0.39 0.30 0.88 0.84 0.41 0.31

150 0.47 0.60 1.09 1.52 0.38 0.64

200 0.32 0.89 0.64 2.18 0.52 1.04

300 1.55 0.12 3.58 0.36 1.94 0.06

400 0.92 0.33 1.99 0.68 1.13 0.41

Avg.MeasurementError(ft.) 0.49 0.40 1.06 0.49 0.60 0.49

Avg.MeasurementError(%) 1.22% 1.00% 1.06% 0.49% 1.20% 0.98%

Table3.CorrelationBetweenResolutionandMeasurementAccuracy

Theaveragemeasurementerrorsbetweenthe12MPmapsand20MPmapsshowedthaterrorwouldincreaseproportionallytocameraresolutionwithacorrelationcoefficientof0.33.Between12MPand16MPmaps,thiscorrelationcoefficientwascalculatedat0.56.FlyingLowImprovedMeasurementAccuracyby0.35%Thedatasuggeststhatflightplansconductedwithloweraltitudeswillproducemapswithhigheraccuracythanthoseflyingathigherelevations.Theaveragemeasurementerrorsshowedthaterrorwouldincreaseproportionallytoflightaltitudewithanaveragecorrelationcoefficientof0.42.ThiscorrelationisvisualizedbelowinChart1andTable4.Therewasa0.35%improvementinmeasurementaccuracyformapsflownat66ft.elevationcomparedwiththoseflownat100,200,or400feetofelevation.HeightCorrelationFactor

Measurement1(40ft.)

Measurement2(100ft.)

Measurement3(50ft.) Average

Phantom4Pro 0.466 0.453 0.419 0.45

Inspire1Pro(X5) 0.355 0.334 0.564 0.42

Inspire1(X3) 0.548 0.512 0.807 0.62

Phantom3Pro 0.355 0.295 0.378 0.34

Total 0.46

Table4.FlightAltitudetoMeasurementErrorCorrelationCoefficients

-11-


Chart1.Phantom4ProMeasurementErrorvs.FlightAltitudeTogether,CameraResolutionandAltitudeHalvedAverageMeasurementErrorWhenflyingatloweraltitudesandwithahigh-resolutioncamera–theaveragemarginoferrorwasreducedto0.64%.Thisisnearlyhalftheaveragemarginoferrorforalldronesatallaltitudes.TheseresultscanbefoundinTable5.

P4PFlightNumber Measurement1(40ft)Error(ft.)



1 0.20 0.47 0.14

2 0.34 0.90 0.35

3 0.42 1.15 0.45

4 0.49 1.24 0.50

5 0.07 0.21 0.02

6 0.13 0.32 0.26

7 0.14 0.24 0.22

8 0.52 1.32 0.56

9 0.04 0.09 0.07

10 0.25 0.63 0.22

Average(ft.) 0.26 0.28 0.66

Average(%) 0.65% 0.56% 0.66%

Table5.MarginofErrorWithoutGCPs

0

1

2

3

4

5

0 50 100 150 200 250 300 350 400

MeasurementError(ft)

FlightAltitude

P4PMeasurementErrorvsFlightAltitude

M1Error

M2Error

M3Error

-12-


VisualizingMapResolutionbyAltitudeTherewasasignificantdifferenceinmapresolutionwhenflyingathigheraltitudes.ThiseffectofaltitudeonmapresolutionmadeitdifficulttomarkthecenterofeachgroundcontrolpointwhenprocessingmapdatainDroneDeploy.ForanexampleoftheeffectaltitudehasonimageresolutionseeFigure7.

Figure7.MapResolutionVisualizedbyAltitudeGroundControlPointsImprovedMappingAccuracy10xAcrossthe9mapsprocessedwithGCPs,theaveragemeasurementerrorwasreducedto0.04ft.(0.5inches).Unlikethenon-GCPmaps,theaverageerrorwasnotdependentonthemagnitudeofthecontrollength.The100ft.controllength,whichalwaysproducedthelargestmarginoferrorwhenmeasuredonthenon-GCPmaps,wasthemostaccuratewithanaverageerrorof0.01ft.(0.12inches.Thesesmallerrorswerealsoincrediblyconsistentandhadverylittlevariance.Theaveragestandarddeviation(𝜎")ofmeasurementerrorsonGCPmapswas𝜎" = 0.02,whichwas10xsmallerthannon-GCPmapsasshowninChart2.

-13-


Chart2.Phantom4ProMapMeasurementVariance

Conclusion UseHighResolutionCamerasforIncreasedMeasurementAccuracyDroneDeploy’sdatashowsthatusingacamerawithatleast20MPreducestheaveragemarginoferrorinlinearmeasurementsto0.64%.ThedatacollectedusingtheDJIPhantom4Proand20MPcamerawithmechanicalshutterprovidedthemostaccuratedataandmeasurementresultscomparedwithlowerresolutioncamerassuchasthoseusedwiththeDJIMavicProorInspireSeries.DroneDeploysuggestscustomersseekingtomakethemostaccuratemapsformeasuringdistancespoint-to-pointusethePhantom4Pro. FlyMissionsatLowAltitudeforHigherMeasurementAccuracyDroneDeploy’sdatasuggeststhatcreatingflightplanswithlowerelevationscanreducethemarginoferrorsignificantly.Exactreductionsinerroraredependentuponthecameraused.SeeTable4foracomparisonofthisreductioninboth12MPand20MPDJIcameras.DroneDeployrecommendsthatpilotscreateflightplansthatbalancetheneedsofthemissionagainsttheflightsettings.Flylower,withhigheroverlapandsidelap,ifoneintendstomakeaccuratelinearmeasurementsusingthedrone-generatedorthomosaicmap.ThiswillofcourseresultinalargerimagepayloadtobeprocessedwithintheDroneDeploymapengine.Thisincreaseinimageswillalsohaveanincreaseonuploadandprocessingtime.

-14-


GroundControlPointsProducetheMostConsistent,AccurateResultsDroneDeploy’sstudyshowsthattherecanbeawiderangeoferrorpresentacrossdatasetscollectedusingDJIdronemodelsandcameras.Theaveragelinearmeasurementmarginoferrorwithouttheuseofgroundcontrolpointswas1.1%ofthecontrollength,however,individualerrorvaluesfellanywhereintherangeof0.01ft.to6.99ft.Ifaprojectrequireshighlyconsistentratesofaccuracy,orshouldmission-criticaldeterminationsneedtobemadefromthedataset,thengroundcontrolpointsshouldbeused.Thisstudyfoundthatprocessingwithgroundcontrolpointsreducedtheaveragemeasurementerrorto0.5inches,anear10ximprovement,forallthecontrollengths.ThemeasurementerrorsforGCPmapsalsoshowedasimilar10xreductioninthestandarddeviationoftheerror,meaningnotonlyisGCPdatamoreaccurate,itismoreconsistentandreliableforapplicationsthatrequirehigheraccuracy.AltitudeMattersWhenMappingwithGroundControlPointsAspreviouslymentioned,imageresolutionwillbeaffectedbyaltitude.ThiswillmakeitdifficulttomarkthecenterofyourgroundcontrolpointmarkerwhenprocessingthemapinDroneDeploy.Accuracywillnotmatterifyouareunabletopreciselyidentifythecenterofyourtarget.PilotsshouldbesuretotakethisintoconsiderationandadjustflightaltitudetoachieveadesiredresolutionthatwillallowthemtoproperlyandconfidentlyidentifytheGCPtargetwithintheirprocesseddronemap.

-15-


AdditionalDroneDeployResourcesAGuidetoUsingGroundControlPointswithDroneMappingSoftware

https://blog.dronedeploy.com/what-are-ground-control-points-gcps-and-how-do-i-use-them-4f4c3771fd0bDecidingIfYourDroneMappingProjectNeedsGCPs

https://blog.dronedeploy.com/when-to-use-ground-control-points-2d404d9f5b15CapturingGroundControlPoints(GCPs)

https://support.dronedeploy.com/docs/working-gcp-step-by-stepGCPRequestChecklist

https://support.dronedeploy.com/docs/gcp-request-checklistDroneDeployIn-BrowserGCPTaggingWorkflow

https://support.dronedeploy.com/docs/in-browser-gcp-taggingAcknowledgementsDroneDeploywouldliketothankAlliantUAVforsupportingthisstudybylendingustheEmlidReachRS.ForUSordersoftheEmlidReachRSpleasevisit:https://alliantuav.com/product/reach-rs/DroneDeploywouldalsoliketothankTrimbleforincludingusintheCatalystbetaprogram.FormoreinformationabouttheTrimbleCatalystvisit:https://catalyst.trimble.com/

-16-


AppendixViewCompleteDatasetandExploreCompletedMapsandMeasurements:https://www.dropbox.com/s/wqqxhe6od798u4j/Linear%20Measurement%20Accuracy%20of%20DJI%20Drone%20Platforms%20and%20Cloud-Based%20Photogrammetry%20Appendix%20A%2C%20Dataset.xlsx?dl=0

-17-


©2017DroneDeploy.AllRightsReserved.

Documents

Linear Measurement Accuracy of DJI Drone Platforms and ... · determine the optimum flight settings for producing maps that can deliver accurate 2D ... , sidelap, and frontlap. Drone