Embed Size (px)
Citation preview
October 2020
Fully Autonomous Asset Inspection with UAVs
Hayk MartirosHead of Autonomy @ Skydio
Founded in 2014150+ person world-class organization spanning hardware, software, manufacturing, sales, and customer success from top companies, research labs, and universities.
Designed and Built in USHeadquartered in Redwood City, CA with all major functions from design through manufacturing & support.
Raised $170MFrom world-class investors and strategic partners.
Investors:
Company Heritage:Skydio is the leading U.S. drone manufacturer and world leader in autonomous flight.
Drones are 10x More Effective and 100x Cheaper and Safer than Heavy Machinery
Drones vs. tower climbs
● 33% accelerated inspection time
● Full digitization of physical assets for detailed tracking over time
● Save lives and injury by reducing tower climbs
TOWER INSPECTION
Drones vs. helicopters● 90%+ cost reduction (fixed and
operating)
● Multiple order of magnitude reduction in consequences of crashes
● Far less socially disruptive
PUBLIC SAFETY BRIDGE INSPECTION
Drones vs. snooper trucks
● 75% cost reduction. Smaller teams, cheaper equipment
● 90% reduction in social disruption cost
● Reduced accident risk
MANUAL DRONESSTATUS QUO
Up to 80% of an average drone program’s budget can be consumed by pilot training and salary
First responders, federal agencies, and enterprises are starting to see the value small drones can provide…
But there are still major challenges:
● Easy to crash: hand-flown and complex to fly, requiring expert pilots
● Not scalable: drone availability tied to that of an expert pilot
● Expensive: to buy and replace upon frequent crashes
● Made in China
UTILITY INSPECTIONSSTATUS QUO - MANNED AVIATION
Power utilities use helicopters for line inspections, but such operations come with big challenges:
● Extreme danger to employees and infrastructure
● High operating costs - fuel, pilots, and maintenance
● High capital expense to purchase aircraft and hangar space - a 4-passenger Airbus AS350B2 helicopter costs $2.4m
UTILITY INSPECTIONSSTATUS QUO - MANUAL DRONES
Power utilities are starting to look at drones, but today’s technology is still causing pain:
● Parts failing unexpectedly
● Waiting for hours during an outage or issue for visual inspection
● Inspecting manually with dangerous tower climbs, or flying a drone manually and hoping not to crash
BRIDGE INSPECTIONSSTATUS QUO - SNOOPER TRUCKS
Bridge inspections by snooper truck are an expensive liability, causing:
● Lengthy and inefficient inspections
● High equipment and labor costs - a snooper truck can cost between $200K and $500K
● Extensive lane closures and social disruption
● Real danger to operators & bystanders
BRIDGE INSPECTIONSSTATUS QUO - MANUAL DRONES
Bridge inspections by manual drones are limited by quality and safety risks:
● GPS-denied environments below bridges disable manual drones’ positioning systems
● Prohibitively high pilot skill required to navigate bridge trusses to see inside
● Risk of crashing causes danger to people under bridge, or risk of water damage
Risk of crashes is the #1 concern among enterprises customers
● Training can take up to 80% of the budget in a drone program
● Even the best trained pilots crash manual drones
The true solution is Autonomy.
Top Concerns with Current Drone Products
*Based on poll of companies with or exploring enterprise drone fleets (greater than 100 responses)
12%
55%
14%
18%
Difficulty sourcing & training pilots
Risk of crashes
Foreign components & manufacturers
Lack of autonomy
Unmatched autonomy in an incredible form factor.
SKYDIO 2
● 360° obstacle avoidance based on computer vision and deep learning
● Autonomous flight patterns for navigation, cinematography and inspection
● Unprecedented ease of use with flight apps designed with camera-like simplicity
● Skydio Beacon™, a first-of-its-kind controller for a drone
LAUNCHED OCTOBER 2019
Unmatched autonomy meets enterprise performance.
SKYDIO X2™
● Field-tested AI building upon Skydio 2’s groundbreaking technology foundation
● Ultimate data capture via 12MP color camera and 320x256 thermal camera
● Fly anytime, anywhere with GPS night flight capability, 35+min endurance, 6 km range, and rucksack portability
● Enterprise (X2E) and Defense (X2D) configurations available
INTRODUCING
COMING SOON
SKYDIO AUTONOMY™ENTERPRISE FOUNDATION
● 360 Superzoom™ (right) enables a full omnidirectional view with 100x zoom
● Precision Mode sharp tuning on flight controls with close-proximity (0.5m.) obstacle avoidance
● 180 Vertical View allows the gimbal-mounted user cam to look straight up for overhang inspections
INTRODUCING
Advanced AI-powered flight assistance to turn anyone into an expert pilot.
COMING SOON
COMING SOON
SKYDIO 3D SCAN™INTRODUCING
● One-tap operation so all pilots have to do is set a bounding box
● 3D autonomous capture as opposed to staying in one plane
● Consistent coverage autonomously at operator’s desired resolution
● Pure software upgrade that does not require expensive new camera sensor
Adaptive scanning solution for autonomous photogrammetry capture.
Skydio Enterprise Architecture
Skydio Autonomy™ CoreReal-Time 3D Mapping | Object Recognition | Obstacle Avoidance | Motion Prediction
SKYDIO AUTOMATES THE SKILLS OF AN EXPERT PILOT
CORE AUTONOMYFUNCTIONS
Skydio 2™ & Skydio X2™
HARDWARE
ADVANCED AI SKILLS
Skydio Autonomy™ Enterprise Foundation360 Superzoom | Close Proximity Obstacle Avoidance | 180 Vertical View
Human Operator
Controller
USERINTERFACE
Skydio Enterprise App
3D Scan™Complex structure inspection
Dock™
House Scan™Residential roof inspection
More skills coming soon
Skydio 2™ & Skydio X2™
Skydio Autonomy Core
Skydio Autonomy™ CoreReal-Time 3D Mapping | Object Recognition | Obstacle Avoidance | Motion Prediction
CORE AUTONOMYFUNCTIONS
HARDWARE
ADVANCED AI SKILLS
Human Operator
Controller
USERINTERFACE
Dock™
Visual perception and planning stack for truly trustworthy autonomous flight.
Skydio Autonomy EngineSkydio Autonomy Core
Skydio Autonomy Core
Extreme Visual ConditionsThe challenge of an autonomous drone: Customer pays for the robot to fly itself in a place we’ve never seen and quickly gains unconditional trust. Every limit is pushed.
• Few semantic priors - fly in a jungle, in a factory, on a mountain, in a city center
• Missed obstacles = robot will crash or fly away
• Imagined obstacles = drone will be erratic and unpredictable
• Time is ticking! Limited battery life, limited connectivity, high speed motion
19
Daunting set of challenges to visual navigation:
•Thin objects, extreme glare, shadows, camera artifacts, motion blur, dirt, smudges
•Waves, snow, rain, fog, desert dunes, salt flats, reflective surfaces, textureless walls
•Calibration errors, rolling shutter errors, time-sync errors, vibrations, EMI
20
a) thin branches, b) sun glare with dirty lens, c) severe motion blur,d) reflections, e) textureless sky or ceiling, f) water droplets
Rolling Shutter
Problem: Skydio 2 uses rolling shutter navigation cameras (higher quality, lighter, cheaper). Need to do geometric processing during high speed, high rotation motions close to objects!
Solution
- Careful modeling of rolling shutter effects, accounting for translation and rotation
- Accurate time synchronization between camera triggers, exposure periods, IMUs
21
Global Shutter Rolling Shutter
pixel shift through exposure period
Lens Intrinsics
Problem: Environmental conditions significantly change intrinsic lens properties. Need to estimate online to maintain high quality mapping.
Solution: Visual-inertial odometry system jointly estimates intrinsics, extrinsics, IMU biases, and world features during flight, accounting for rolling shutter.
22
stationary lens in temperature chamber
Moving Objects
23
Problem: All of the close visual features in this scene are moving and malicious for state estimation purposes.
Solutions:
+ Tight integration with our IMU
+ Joint consideration of visual and GPS uncertainty
+ Semantic information
Real World Problems: Dirty Camera Detection
Problem: Dirt, dust, fingerprints, water on the lenses can ruin photometric consistency and cause false matches.
Solution: Estimate dirty regions of cameras in flight.
24
25
Idea: Regions with poor photometric consistency over a variety of background content are likely from dirty cameras.
Aggregate from our matching algorithms across time and vehicle rotations, use as an invalid mask.
Continue flying, applying the mask, but warn the user to land and clean lenses!
If the mask is a large portion of a camera, go into a high-level error state and tell the pilot (if possible) to land quickly.
Real World Problems: Dirty Camera Detection
Challenges for Learning Depth
No ground truth - infeasible to get human labels
Billions of images captured by S2 to learn from —> use unsupervised learning!
• However, computer vision is not good enough yet. Photometric consistency fails.
26
Need to find ways to use leverage large amounts of in-domain unlabeled data for extreme visual conditions.
More acausal reasoning? Better features? Cycle consistency?
Deep-Learned Depth Estimation
27
Deep-Learned Depth Estimation
28
Intelligent Error Handing + Graceful Degradation
30
Comprehensive error handling is extremely difficult and requires complex integration of the whole stack. This what separates demos from real products – there is no substitute.
Skydio 3D Scan
Skydio Autonomy™ CoreReal-Time 3D Mapping | Object Recognition | Obstacle Avoidance | Motion Prediction
CORE AUTONOMYFUNCTIONS
Skydio 2™ & Skydio X2™
HARDWARE
ADVANCED AI SKILLS
Skydio Autonomy™ Enterprise Foundation360 Superzoom | Close Proximity Obstacle Avoidance | 180 Vertical View
Human Operator
Controller
USERINTERFACE
3D Scan™Complex structure inspection
Dock™
Adaptive scanning solution for autonomous photogrammetry capture.
COMING SOON
SKYDIO 3D SCAN™INTRODUCING
● One-tap operation so all pilots have to do is set a bounding box
● 3D autonomous capture as opposed to staying in one plane
● Consistent coverage autonomously at operator’s desired resolution
● Pure software upgrade that does not require expensive new camera sensor
Adaptive scanning solution for autonomous photogrammetry capture.
Data Capture StrategiesExpert Pilot Manual Flight
● Pilot attempts to keep track of what has been imaged● Ends up taking too much data to be safe● Stressful to fly and avoid obstacles continuously
Blind Automated Capture Pattern
● Fly a planar lawnmower pattern, cross hatch, or perimeter orbit● Works well on planar scenes, not on complex structures● Does not handle obstructions like power lines, trees, or overhangs● Requires GPS
3D Scan Flight
● See and understand the 3D structure to compute best path● Track coverage with superhuman precision● Go down as low as handheld DSLR imagery● Operator’s job becomes easier and less stressful
Other
● laser scanners● handheld cameras● manned aircraft● satellite
3D Scan Workflow● Choose an operating scan volume and
set a visual geofence
● Choose autonomous capture settings -scan mode, desired GSD, overlap %
● Hit GO, sit back and watch the UAV capture the imagery you need
3D Scan Algorithms
Real-time structure from motion to maintain visually aligned poses
Create live 3D model to understand the scene instead of flying blind
Scan planner iteratively computes a path to paint all surfaces with imagery
What imagery is needed for good reconstruction?
● Most photogrammetry packages need at least three consistent viewpoints to accept a 3D point
● The views must be from similar enough angles to successfully match
● The views must be from different enough angles to provide information for triangulation
● The views should have significant image space overlap
● The entire graph of viewpoints should be tightly connected without disjoint sets
QUESTIONS?
Use or disclosure of data contained on this sheet is subject to the restriction on the second page of this proposal
