Embed Size (px)
Citation preview
SENIOR THESIS
S I N G L E - H A N D G E S T U R E C O N T R O L L E R F O R Q U A D C O P T E R S
Yucan Liu (Advised by Cherrice Traver)
Union College | CLASS OF 2017, COMPUTER ENGINEERING
2
1. Report Summary
This project focuses on building a single-hand gesture controller. In the recent years, quadcopter
style drones are becoming more and more popular since they are small, convenient, and low-cost.
Many innovative applications in different realms, including photography, architecture, and rescue,
are invented with drone technology. However, the traditional controller of drones requires both
hands and is bulky, which is inconvenient in most situations. A single-hand controller will be helpful
for those pilots who have only one hand available to control a drone. At the same time, the gesture
control system offers a simpler way to drive a drone. This control system will enhance both the
pilots’ and others’ safety. Also, driving a drone will become friendlier to newcomers because of
this intuitive controller.
This project includes two parts: a gesture controller, and a test environment. The test environment
for this project will be a real quadcopter, which is bought from the market. Then, the gesture
controller will be built and tested on this real drone. The gesture controller includes a 9 Degrees
of Freedom Inertial Measurement Unit (9 DOF IMU), a square force-sensitive resistor (force
touchpad), a microcontroller, and a wireless transmitter. Both the 9 DOF IMU and the force
touchpad will collect the user's hand gesture. Then, the microcontroller will store all those data as
input, and then generate correct PWM output to the wireless transmitter. Finally, the transmitter
will send the commands to the drone and control it.
3
Table of Contents
1. Report Summary ................................................................................................................ 2
Tables of Figures and Tables .................................................................................................. 4
2. Introduction ...................................................................................................................... 5
3. Background ....................................................................................................................... 7
4. Design Requirement ........................................................................................................ 15 4.1 Test Environment ................................................................................................................... 15 4.2 Single-Hand Gesture Controller: .............................................................................................. 18
5. Design Alternatives .......................................................................................................... 21
6. Preliminary Propose Design ............................................................................................. 26
7. Final Design and Implementation: .................................................................................... 30 7.1 Wireless Connection: .............................................................................................................. 31 7.2 Hand Controller: ..................................................................................................................... 33
7.2.1 Force Touch Pad ......................................................................................................................... 33 7.2.2 9 DOF IMU .................................................................................................................................. 34 7.2.3 Arduino Pro Mini (Microcontroller of the Hand Controller) ...................................................... 35
7.3 Modified Board: ..................................................................................................................... 35 7.3.1 Control Board ............................................................................................................................. 36 7.3.2 Arduino Pro Mini (Microcontroller of the Modified Board) ...................................................... 36
7.4 Data Transmission and Code ................................................................................................... 36
8. Performance Estimates and Results .................................................................................. 40
9.Production Schedule ......................................................................................................... 43
10. Cost Analysis .................................................................................................................. 45 10.1 Hand Controller Cost ............................................................................................................ 45 10.2 Modified Board Cost ............................................................................................................. 45 10.3 Test Drone Cost .................................................................................................................... 46
11. User’s Manual ................................................................................................................ 47
12. Discussion, Conclusion, and Recommendations .............................................................. 50 12.1 Discussion ............................................................................................................................ 50 12.2 Conclusion and Recommendations ........................................................................................ 51
References: ......................................................................................................................... 53
Appendix A .......................................................................................................................... 54
Appendix B .......................................................................................................................... 56
Appendix C .......................................................................................................................... 59
Appendix D .......................................................................................................................... 60
4
Tables of Figures and Tables Figure 1. Traditional Bulky Controller for Drones ........................................................................... 5 Figure 2. Quadcopter in Unmanned Cargo System ........................................................................ 7 Figure 3. Movements of Drone ....................................................................................................... 9 Figure 4. Typhoon Wizard [3]........................................................................................................ 10 Figure 5. "Shift" Controller [4] ...................................................................................................... 10 Figure 6. Signal Transfer Path ....................................................................................................... 15 Figure 7. Functional Decomposition of Single-Hand Gesture Controller ..................................... 18 Figure 8. Simple State Diagram for Gestures and Modes............................................................. 22 Figure 9. Standalone 5-Pad Capacitive Touch Sensor (Left) ......................................................... 27 Figure 10. Pinout Configuration (Right) ........................................................................................ 27 Figure 11. Square Force-Sensitive Resistor ................................................................................... 27 Figure 12. Square Force-Sensitive Resistor ................................................................................... 28 Figure 13. 9-DOF IMU ................................................................................................................... 29 Figure 14. Final Block Diagram ...................................................................................................... 30 Figure 15. Hand Controller (Left) .................................................................................................. 30 Figure 16. Modified Board (Middle) ............................................................................................. 30 Figure 17. Test Drone (SYMA X5C-1) (Right) ................................................................................. 30 Figure 18. XBee Connection .......................................................................................................... 31 Figure 19. Code Screenshot .......................................................................................................... 37 Figure 20. Software Flow Chart .................................................................................................... 39 Figure 21. Wear the Controller ..................................................................................................... 47 Figure 22. Turn on the Modified Board and Connect to the Drone ............................................. 48 Figure 23. Fully Calibrate the IMU ................................................................................................ 48 Figure 24. Fly the Drone! .............................................................................................................. 49 Table 1. Corresponding Movements between Drone and Controller ............................................ 8 Table 2. Quadcopter Detailed Specifications and Selection Criteria ............................................ 17 Table 3. Detailed Information of Drones ...................................................................................... 17 Table 4. Controller Parts Selection Criteria................................................................................... 20 Table 5. Design Alternatives for Single-Hand Gesture Controller ................................................ 23 Table 6. Parts for Gesture Controller ............................................................................................ 26 Table 7. Relationship between PWM Value and Sensor Value .................................................... 38 Table 8. Hand Gestures and Correlated Drone Movements ........................................................ 49
5
2. Introduction
For those pilots who have only one hand available to control a drone, a single-hand remote
controller will be more useful and convenient than the traditional bulky controller. At the same
time, adding gesture controls for pilots allows more intuitive manipulations, which improves the
preciseness and ease of controlling a drone.
Figure 1. Traditional Bulky Controller for Drones
In this situation, a single-hand gesture controller of drones, especially quadcopters, will be made
for this senior project. This controller can be used by whoever requires single-hand control for a
drone or a more intuitive controller. So, it should accomplish both single-hand controlling and
gesture controlling. The controller will be built as glove-like or case-like to attach on the pilot’s
hand. All functionalities to fly a drone should be accomplished by using the manipulating hand
without any help from the other hand. At the same time, instead of using joysticks and buttons,
the movement of the pilot’s hand, which will be detected by the controller, should be able to
control the movement of the drone.
6
This report begins with an introduction of the context and application of drones and a discussion
of previous work and former designs. Also, both ethical issues and social effects will be discussed
in this paper to introduce the general design idea and the possible impacts of this project. Detailed
design requirements that serve as the basis of the controller will be described. Functional
decomposition, design criteria, and decision matrices will be included in this part. Design
alternatives for each sub-function, including options and justifications of the overall approach and
each functional part, will be analyzed. The design choice of this project will be fully explained by
using diagrams, tables, figures, and text. After then, the final design and implementation will
describe the finalized project. Results, schedule, cost analysis, and user's manual follow the final
design and explain the whole project in detail. Finally, a discussion, conclusion, and
recommendations section will summarize this thesis again.
7
3. Background In recent years, due to the rapid development of technology, quadcopter style drones become
more popular, familiar, and convenient in not only the professional realm but also daily life (Figure
2). Those drones are widely used for different tasks such as search and rescue, inspections,
security, surveillance, science and research, aerial photography and video, surveying and GIS
(Geology Information System), and unmanned cargo systems, etc. [1]
Figure 2. Quadcopter in Unmanned Cargo System
The traditional controllers for those drones normally require both hands and are bulky. At the
same time, the feedback [2] of the commands is ambiguous. The pilot will be easily confused by
the relationship between the controller commands and the corresponding drone’s movements.
These old-style controllers are insufficient in some new applications. Most importantly, in many
previous situations, one hand of the pilot might be occupied by other equipment. In that situations,
8
the traditional controller won’t satisfy the pilot. For example, one hand of the pilot may need to
hold an iPad for viewing live video and pictures which are streaming from the drone in aerial
photography and video. In search and rescue applications, the pilot may have to hold the first-aid
case in one hand and manipulate drone with the other hand. In cargo delivery system, one hand
of the pilot may be occupied by other cargoes.
Another problem with traditional controllers is that the joysticks on the traditional controller are
not intuitive enough for pilots to drive the drone. As shown in Figure 1, most traditional controllers
contain two joysticks: a left joystick and a right joystick, which are controlled by thumbs. Most
commonly, the corresponding movements between the drone and the controller follow Table 1
and Figure 3.
Joystick Movement of Joystick Movement of Drone
Left Joystick Up and down Fly up and down
Left and right Yaw
Right Joystick Up and down Roll
Left and right Pitch
Table 1. Corresponding Movements between Drone and Controller
The functionalities of the two joysticks are entirely different and may cause confusion to
newcomers. The interface of the traditional controller avoids affordance and mapping [2], which
will also increase the confusion of users. At the same time, according to Hick’s law, more choices
will cause longer time to make the decision. However, the longer decision time on the controller
will cause less precision of the drone’s movements. Importantly, time is the most valuable thing in
9
inspections, security, GIS, search, and rescue, etc. Also, most of the controllers on the market are
simple and don’t have any built-in functions or programs. For instance, flying around a circle with
100 meters’ radius can be a daily task in the security application. In this case, the drone will be
more helpful if some programmed movements can be recorded and people can call them
whenever they want.
Figure 3. Movements of Drone
With all those problems and considerations, this project will be focusing on building a single-hand
gesture controller for quadcopter style drones. Currently, several other companies are also
developing single hand controller for drones. For example, “Yuneec” has built a single-hand, TV-
controller like a remote controller for drones (Figure 4), called “Typhoon Wizard.” However, most
of them are not supporting gesture control or only compatible with some specific drone models.
There are also some fresh ideas from startup companies. For example, the “Shift” is a new
generation of drone and controller, which allows the user to control a drone by only using a thumb
(Figure 5). But all those projects, the new generation of the drone’s controller, are still in progress
10
or canceled because of some unknown reasons. So, this project will focus on building a prototype
of a single-hand gesture controller within a $250 budget, including the drone, and compatible with
most of the drones with a 2.4Ghz controller. The $250 budget is an acceptable amount of money
for an experiment. And the 2.4Ghz frequency is used by most of the drones.
Figure 4. Typhoon Wizard [3]
Figure 5. "Shift" Controller [4]
11
Today, unmanned aerial vehicles, which are typically known as drones, have come into the public
eye not only because of their applications but also their ethical problems. The use of drones in
warfare, such as to attack members of ISIS and AL-Qaeda, has been the subject of considerable
debate [5]. Supporters argue that an unmanned weapons system can eliminate enemies without
bringing any risk to the lives of American soldiers. The drones reduce the possibility of face-to-face
conflicts which cause most of the deaths and injuries of soldiers today. Additionally, these
weapons are far more precise than other weapon systems, and in consequence, do far less
collateral damages to the battle field’s environment. However, the critics insist that the unmanned
systems make the kills too easy from a distance which is utterly not morally correct. They also
accuse the government of reporting far less civilian casualties than the actual statistic because the
drone’s algorithm is not smart enough to clearly distinguish terrorists and civilians. At last, the
critics conclude that the terrorists’ anger over the threat from a distance and the deaths of civilians
fuels them and causes further extreme actions of terrorism.
The debate has been argued for a long time. Indeed, new weapons, which allow military personnel
kills from a safe distance, are always debatable. Plenty of examples can be found within history,
such as the English longbow against French Knight, the European cannons against Vatican, and the
submarines in the First World War. Similarly, drones are also deemed as “immoral” by some and
“necessary” by others. Weapons in warfare is always a complicated and controversial topic. On
one hand, terrorists, who threaten the country, are dangerous and condemnable. They won’t stop
attacks even that the drone program ceased. On the other hand, humanistic points of view should
also be considered. Although precise data on the death and destruction of the unmanned aerial
12
vehicles is difficult to get, the improvement in drone’s algorithm to distinguish terrorists and
civilians should be helpful to minimize the deaths and harm. The person who uses the new “evil”
technology should instead be blamed. The technology itself is never evil. However, the debate of
the drone for warfare usage should remind us of the importance of the ethical problem in the
drone’s development.
Other than the battlefield, drones are widely used in Geology Information System (GIS) as a
valuable tool to gather geological data [6]. Because drones can gather information cheaply from
an inaccessible or remote area, the drone has been considered as the first choice in assessing
archeological sites, and mapping threatened plant species in Michigan. Competing with traditional
methods, drones have advantages in cheapness and quickness of gathering high-resolution
geographical information. However, other than the benefits, ethical and legal problems have also
been addressed. Caitlin Dempsey [6] states that the ethical and legal problems in gathering
geographical data using drones can be divided into four aspects: regulations, privacy, safety, and
noise.
Drone regulation is the first and the hardest problem. Although the available fly areas and height
are clearly limited by the Federal Aviation Administration (FAA), surveillance is hard to execute
comparing the size of a drone to the size of the supervised area. Even though the drones are
required to registered on FAA’s website, pilots may break the rule either on purpose or due too
loose surveillance. Reported by American Civil Liberties Union (ACLU) in 2014, thirty-six states had
introduced legislation intending to restrict drone’s usage in particular data collections. However,
13
illegal drone activities still exist. While most individuals are free to fly drones as a hobby, most
commercial activities based on drones are illegal in America. According to Gakstatter [6],
companies can skirt the rules by only charging for image processing instead of drone flying time.
However, this this is still categorized as an illegal activity by the FAA. In addition, Privacy issues
related to drone use is not a fresh topic in newspaper articles. There are plenty of articles about
the private drone’s intrusion into private properties. According to the FAA, drones are only allowed
to fly at a low altitude, which will cause safety problems. Limited to the size, the drone may not be
harmful to the human with clothes However, the propellers are still dangerous to eyes, face, and
bare skin. At the same time, the low altitude means increased noise. With restriction, drones are
not allowed to fly near populated areas or over people for safety reasons. But, the noise may still
effect residents and wildlife. Therefore, drones can be defined as not environmental friendly. In
fact, because drones disturbed the Big Horn Sheep and interrupted recreational visitors at Grand
Canyon National Park, the National Park Service (NPS) banned flying drones.
Even though the drone has plenty of unresolved concerns, the global drone industry is fast-
growing. The drone vendors are developing the ecosystem, including both software and hardware,
to cater to a long list of clients in agriculture, land management, GIS, warfare, energy, and
architecture [7, 8, 9]. In the current market, most of these drone companies are private companies
or startups. At the same time, large defense-focused companies and government-controlled
industries are beginning to invest in drone technology because of the unmanned aerial vehicle’s
advantages. In a report from PwC on the commercial application of drone technology, the
emerging drone’s market values more than 127 billion dollars. Worldwide, the drone with high-
14
resolution camera and multiple sensors provide various opportunities within infrastructure,
transport, media, and entertainment. Other than services, the drone itself will lead development
in other realms. For example, the booming drone market may stimulate the improvement of
power supply, motor, and materials. Limited by the drone’s size, smaller battery with larger power
supply ability will improve the drone’s performance. Similar to the battery, drones also require
little motors with massive torque and less power. At last, a lighter material with better resistance
to wind, sand, and water will surely enhance the drone’s capabilities.
15
4. Design Requirement The hand gestures are detected by the sensors on the controller. Then, the microcontroller will
take those signals as input and generate correct output to the wireless module on the controller.
The wireless transmitter will transmit the signal to the receiver on the drone. Finally, the
microcontroller on the drone will control the drone’s motors. The signal transferring path is shown
in Figure 6.
Figure 6. Signal Transfer Path
This project is only focused on the single-hand gesture controller instead of the whole drone
system. However, a drone is still required for testing the controller. In this situation, the whole
project is divided into two parts: the controller part, which will be made and tested; and the test
environment, which includes wireless modules and the drone for testing. Since the price of most
of the quadcopters is affordable, the quadcopter for testing will be bought from the market and
only be used for experiment and testing. The gesture controller is the focus of this project.
4.1 Test Environment First, the test environment should be clarified. Since the gesture controller is designed to be able
to use on any 2.4Ghz quadcopter, the tested drone should be commonly used and easy to get
from the market. Any special model or high-level drone should not be considered for this project’s
test environment. Most importantly, the drone should be safe enough for testing and flying. So,
16
only guaranteed online websites or professional drone website with quality assurance, like
Amazon and “buythebestdrone,” are in consideration. Since the project is to design the controller
part and the drone is only used as essential testing equipment, the price of the drone should be
in a reasonable range, which is $30-$80. With the consideration of saving time, the drone should
not involve assembling parts. For indoor experiments, the length and width of the drone should
be acceptable, which is less than 15 inches. The flight time should be as large as possible, but the
battery charging time should be as small as possible. In fact, there is no such a battery that has
both short charging time and long duration. So, the tradeoff between the charging time and flight
time should be carefully considered. Most of the drones have 90-100 minutes’ charging time and
5-10 minutes’ flight time. This controller project will only focus on 2.4GHz frequency because it is
the most commonly used and it offers reasonable range, which is essential for drones’ controlling,
and acceptable data rates. At last, the control axis should be 6, to do all the basic movements,
which are pitch, yaw, and roll (Figure 3). The camera is not required for this project. However, it’s
better to have one for further experiments. All detailed specifications for quadcopter are listed in
Table 2.
Aspect Criteria
Control frequency 2.4Ghz
Battery charging time Less than 120 minutes
Flight time More than 8 minutes
Assemble parts No assemble parts involved
Axis 6
17
Size Less than 15 inches * 15 inches
Price range $30- $80
Table 2. Quadcopter Detailed Specifications and Selection Criteria
Basic movements of a drone are described below and illustrated in Figure 5:
1) Pitch: rotate about Y-axis
2) Yaw: rotate about Z-axis
3) Roll: rotate about X-axis
4) Move up and down: change in altitude
Using the drone’s selection criteria, Table 3 was made after searching drones online. All four of
these drones had a reasonable price range. Their qualities were specified by the website as follows.
Criteria Syma X5C
DBPOWER MJX X400W
UDI U818A
SYMA X5C-1
Price ($) 43.5 75.99 54.99 65.99
Flight time (min) 7 8-9 7-9 8
Charge time (min) 100 120 120 100
Axis (axis) 6 6 6 6
Controller Included included included included
Camera yes yes yes yes Frequency (GHz) 2.4 2.4 2.4 2.4
Size (inches) 16.5 x 12.2 x
3.8
11.8 x 11.8 x 3
13 x 2.2 x 13.3
12.40x12.40x2.95
Remote distance (m)
30-50 100 30 100
Note: the words in red represents better performance than others Table 3. Detailed Information of Drones
18
Base on the criteria, the SYMA X5C-1 (shaded as gray in Table 3), which is an updated version of
SYMA X5C, is generally better than others. It has a reasonable price with a replacements bundle,
such as free batteries, SD card, and plastic parts. It has 8 minutes’ flight time and 100 minutes’
charging time, which is a better performance than the average. At the same time, it has 6 axis,
controller and camera, and 2.4Ghz frequency. The size is smaller than the criteria and should be
able to fly both indoor and outdoor. Finally, the 100 meters’ remote distance is also able to support
outdoor experiments.
The whole test environment includes a wireless receiver and a drone. Commonly, the drone from
market is already embedded with 2.4GHz wireless receiver. In this case, the test environment
won’t need any further improvement. All in all, this test environment assures the availability of
testing the controller. The commands from the controller are able to drive the drone for better
experiment observation.
4.2 Single-Hand Gesture Controller: The functional decomposition for the single-hand gesture controller is shown in Figure 7.
Figure 7. Functional Decomposition of Single-Hand Gesture Controller
19
According to the design requirement, the controller should be able to detect three types of hand
gestures: finger movements, joint actions, and whole hand movements. Since the whole controller
should be able to attach on hand, the total weight of the controller should be as low as possible.
At the same time, the size of the controller should also be comfortable for users. In this case, the
size of the microcontroller should be as small as possible. Finally, wires are always a big problem
for wearable devices. More wires cause more opportunities for bad connections and more
difficulties in debugging and maintenance. Since the device is going to be moved around a lot,
connection problems are always an issue. So, the controller will be better if less pins in total are
required.
In this project, the gesture controller will be built as a custom embedded system. Based on Figure
7, there should be at least 3 types of sensors attached on the hand, which are touch sensors for
finger actions, touch sensors for joint movements, and movement sensors for the whole hand
movements. The sensors to detect finger and joint movements should not require too much force
in order to assure comfort of users. At the same time, they are not required to give feedback of
the precise force. But they must be sensitive enough to detect every single movement.
A microcontroller should be used in order to receive data from the sensor and generate correct
data output to the wireless module. The size of the microcontroller should be as small as possible
because it will be attached on hand. With the same consideration of user’s comfort, the battery
package for microcontroller should be small and convenient also. In this situation, the required
power supply of the microcontroller is better to be low. The power requirement of the
20
microcontroller is better to be same with other sensors for convenient reason. In addition, the
frequency of the microcontroller should be at least 8MHz because the generated signal should be
at least same with the drone’s frequency, which is 8MHz in common. The processing speed of the
microcontroller should be as fast as possible. Finally, the wireless module should be able to
transmit and receive 2.4GHz signal to match the design requirement. Most of the wireless modules
in the market are using UART for input data. So, the microcontroller should also support UART.
Sensor or part Criteria
Finger action Precisely detect finger bends; less pin required
Joint movement Precisely detect changes in different directions; less pin required
Hand direction Precisely detect acceleration, direction, and rotation; less pin required
microcontroller Small and able to support all other parts; At least 8MHz; At least 3.3V
Wireless module 2.4Ghz
Table 4. Controller Parts Selection Criteria
21
5. Design Alternatives This single-hand gesture controller is only designed for 2.4Ghz drones, the SYMA X5C-1. For the
controller part, two important gestures are designed to indicate different input conditions for
the drone:
“Fist” represents the pilot’s fingers contract and press the sensor at the center of hand.
“Fist” is positive (true).
“Palm”, opposes to “fist”, is not making fist in a certain amount of time (0.5 or 1 second).
“Palm” is negative (“fist” = false).
In this design, “fist” and “palm” are the two input conditions: “fist” represents positive;
and “palm” represents negative.
The following list defines each gesture input and drone state on the left side of the “->”, and
correlated drone movements on the right side of “->”.
1) Make fist then palm within a certain amount of time -> turn on (fly) and hold in air
2) Make fist twice within a certain amount of time when the drone is in the air -> land and
turn off
When the drone is in the air (fly):
3) Make and keep fist -> drive mode
4) Make and keep palm -> static mode
In drive (follow hand gestures with corresponding response until another gesture is detected)
or static (follow hand gestures with short duration responses) mode:
5) Turn hand on three axes -> pitch, yaw, and roll
22
6) Move hand up and down -> move drone up and down
In static mode:
7) Move hand forward and backward -> drive drone forward and backward
8) Touch fingers -> different build-in functions
Figure 8 is a simple state diagram that describes the drone’s modes and gesture commands.
Figure 8. Simple State Diagram for Gestures and Modes
Following the controller’s selection criteria, the sensors on hand should include at least:
1) A sensor (or other plans) at the center of palm to detect fist and palm (joint movements)
2) Axis sensor and accelerometer to detect hand’s 3D motions and speed (for drive mode and
move forward and backward commands)
3) At least 4 sensors for finger touch functions (finger movements).
23
Alternative parts for gesture controller are listed in Table 5:
Finger movements Joint movements Hand directions Hand movements
Touch sensor Touch sensor Gyroscope Accelerometer
Force sensor Force sensor Leap sensor
Muscle sensor Muscle sensor 3D motion sensor
Bend sensor Leap sensor
Table 5. Design Alternatives for Single-Hand Gesture Controller
All sensors’ information and technical data were reviewed. The touch sensor commonly has a
digital output, which indicates whether a touch is detected or not. The force sensor works like a
resistor. The resistance will change depending on the force applied to it. The muscle sensor
requires additional preparation steps: electrodes need be placed on the skin to detect muscle
movements every time before using. Similar to force sensor, the bend sensor also works as a
resistor. The resistance will change according to where to bend and the degree of bending. The
leap sensor is a small USB peripheral device which includes two monochromatic cameras and three
infrared LEDs [10]. The gyroscope sense angular velocity from the Coriolis force applied to a
vibrating object. Piezoelectric, piezoresistive and capacitive components are commonly used in
accelerometer to convert the mechanical motion into an electrical signal.
Then, some options with distinct disadvantages were rejected. For the finger movement detection,
the muscle sensor and bend sensor were not considered for detecting finger movements because
they required too much force from the user, which meant the users would be uncomfortable while
24
doing the actions. For example, the force sensor required at least 1N to be added to the sensor,
which is unnecessary for finger movement detections. For muscle sensor, the preparation was
complicated, and the electrodes were unable to reuse, which wasn’t resources efficient.
For palm or fist (joint) detection, leap sensor looked fancy, but it’s not accurate enough. This
device’s valid detect area is a hemispherical area with roughly 1-meter radius. The USB cable must
be connected to the computer, and the Leap Motion software is required. This sensor can detect
3D motions and manipulate complex 3D data visualizations. However, the precision of this
technology is low. For example, this sensor has more than 50% error rate when detecting the
difference between a fist or single finger. For this gesture controller, preciseness is one of the most
important features. So, leap sensor was rejected.
The required sensor for movements (speed) of hand was evident. The accelerometer was the only
sensor that can measure acceleration forces. A 3-axis accelerometer would be helpful in this
scenario since the controller was based on 3D space with all 3D movements. On the other hand,
the direction (rotation) of hand can be detected by several methods. With the same accuracy
reason, the leap sensor has been rejected. The 3D motion sensor has higher accuracy comparing
to the leap sensor. The static (RMS) measurement for roll and pitch could be precise to 0.75
degrees, and the dynamic (RMS) measurement could be precise to 1 degree. Both static and
dynamic measurement for yaw could be accurate to 2 degrees. However, it was much more
expensive. For example, a starter level 3D motion sensor from “Xsens” [11] would cost $145. At
the same time, most of its applications were built with the computer. The output from the 3D
25
motion sensor is hard to communicate with microcontroller without an application as the interface.
So, the best option for detecting the hand’s movements would be the gyroscope.
26
6. Preliminary Propose Design Using the gesture controller’s criteria, Table 6 was compiled of possible parts. All these parts were
found online. Detailed information of each part is given in Appendix A. The test methods and plan
are also given in Appendix B For the axis sensor and accelerometer, there was available integration
of sensors called IMU (Inertial Measurement Unit). In this project, 9 DOF (Degree of Freedom) IMU
was sufficient for detecting hand gestures. The horizontal and vertical moves can be detected by
3-axis of accelerometer. The turning of hand can be detected by 3-axis of gyroscope. The direction
of the hand can be detected by the other 3-axis of magnetic sensor.
Palm (joint) touch sensor
Axis sensor Accelerometer Finger touch sensor
Round Force-Sensitive Resistor (FSR) – Interlink 402
Adafruit 9-DOF Absolute Orientation IMU Fusion Breakout BNO055
Round Force-Sensitive Resistor (FSR) – Interlink 402
Short Flex Sensor Adafruit 10-DOF IMU Breakout – L3GD20H+LSM303+BMP180
Short Flex Sensor
Square Force-Sensitive Resistor (FSR) – Interlink 406
L3GD20H Triple Axis Gyro Breakout Board – L3GD20/L3G4200 Upgrade – L3GD20H
ADXL326 – 5V ready triple-axis accelerometer (+-16g analog out)
Adafruit 12-Key Capacitive Touch Sensor Breakout – MPR121
Muscle Senor Surface EMG electrodes – H124SG + MyoWare Muscle Sensor
FLORA 9-DOF Accelerometer/Gyroscope/Magnetome – LSM9DS0 – v1.0
Standalone 5-Pad Capacitive Touch Sensor Breakout
Note: the words in red represents better performance than others
Table 6. Parts for Gesture Controller
The 5-pad capacitive touch sensor will be used to detect finger movements. The 5 pads will be
used to represent 5 fingers. Each finger’s touch will active a different build-in function or program.
Since the sensor is capacitive, no extra force is needed. Simply touches from fingers can be
27
precisely detected. The power is 1.8V to 5.5V. As shown in Figure 10, two modes can be used,
either the I2C protocol or 5-bit digital data. In order to decrease the required number of pins for
the microcontroller, the I2C protocol will be used because only two wires are needed.
Figure 9. Standalone 5-Pad Capacitive Touch Sensor (Left) Figure 10. Pinout Configuration (Right)
For joint detection, a square force-sensitive resistor (FSR) is selected. It works like a resistor (Figure
11) with analog or digital output. In this project, this force-sensitive resistor will be used as digital
output because it will be attached at the center of palm and detect “fist” or “palm” as defined
before. The touch area of the square resistor (88mm*43.7mm) is larger than the area of the round
resistor (56.77mm*18.48mm). So, the larger touch-area sensor is selected in order to enhance the
interface’s forgiveness [2], which is the tolerant level of mistakes.
Figure 11. Square Force-Sensitive Resistor
28
Figure 12. Square Force-Sensitive Resistor
Finally, the 9-DOF Absolute Orientation IMU is selected to detect the whole hand’s movements.
Both I2C and UART interface are available for data transmission. The sensor integrates an
advanced triaxial 16bit gyroscope, a versatile, leading edge triaxial 14bit accelerometer, and a full
performance geomagnetic sensor. The supply voltage range is from 2.4V to 3.6V. For the
accelerometer, the acceleration ranges ±2g/±4g/±8g/±16g. For the magnetometer, the magnetic
field range is typical ±1300μT for x and y-axis, and ±2500μT for z-axis. The magnetic field resolution
is ~0.3μT. It is able to precisely transfer hand’s movements into actual "3D space orientation". At
the same time, it is small enough (1.05 inches* 0.8 inches) to fit in a single-hand project.
29
Figure 13. 9-DOF IMU
The output pin of the microcontroller should be no less than 2.4V in order to generate enough
power for the sensor. At the same time, the required smallest frequency is 8MHz to match the
drone’s frequency. So, the microcontroller should at least support 8MHz frequency. Since I2C,
UART, and digital inputs are used, the microcontroller should support all of these protocols. There
should be at least 5 pins for input: 2 pins for I2C, 2pins for ground and Vin, and 1 pin for digital
input. Finally, the microcontroller should also have enough pins for UART. Because of familiarity,
the microcontroller should be an Arduino series product. Finally, the Arduino Pro Mini 3.3V 8MHz
is selected. The decision matrix is recorded in Appendix C.
30
7. Final Design and Implementation:
Figure 14. Final Block Diagram
Figure 15. Hand Controller (Left)
Figure 16. Modified Board (Middle) Figure 17. Test Drone (SYMA X5C-1) (Right)
The final design of the single-hand gesture controller (Figure 14) included two sections: a
controller on hand (hand controller, Figure 15) and a modified control board (modified board,
Figure 16). The hand controller was made from scratch, and the modified board was disassembled
and rebuilt from the original controller that came with the SYMA X5C-1 drone (Figure 17).
-The hand controller included a 9 DOF IMU, a force touch pad, an Arduino Pro Mini, and
an XBee (wireless serial transmitter).
-The modified board included an XBee (wireless serial receiver), an Arduino Pro Mini, and
a modified board from the SMYA drone’s controller.
31
The 9 DOF IMU and force touch pad would detect user’s hand gestures (step 1). Then, the first
Arduino Pro Mini would store the coming data (step2) and send to the XBee transmitter in serial
(step 3). When the XBee receiver received enough data (step 4), the second Arduino Pro Mini
would store the data into local variables, and then translate to PWM values (step 5). Then, the
second Arduino Pro Mini would output the PWM signals (step 6) based on their values to the
modified board to mimic the two joysticks’ movements. Finally, the modified controller board
would transmit the signal to the SYMA drone and control the four motors on it (step 7).
7.1 Wireless Connection:
Figure 18. XBee Connection
In general, a controller had to wirelessly connect to the drone to control the it’s movements. The
drone and controller were configured as a pair. Since this project focused on the drone’s controller,
a SYMA X5C-1 quadcopter was bought from the market instead of building one. Unfortunately,
the flight control chip on the SYMA drone was encoded. The drone was only able to be controlled
by the SYMA series controller. In this case, the preliminary design had to be changed because a
third-party controller did not have access to the drone. Instead of controlling the drone directly,
the self-built controller should send signals through the SYMA controller’s transmitter, and then
controlled the SYMA drone. After disassembled the SYMA controller, the transmitter was
32
impossible to be desoldered. Also, the whole control board was too big to be attached on hand.
To limit the size of this project within single hand, the project was divided into two sections: a
hand controller and a modified control board. Then, two XBee 802.15.4 wireless modules were in
charge of building communication between the two sections.
XBees are tiny blue chips that can communicate wirelessly with each other. They can do simple
things like replacing a couple of wires in a serial communication, which was exactly what this
design needed. XBee had different modules. Since they were used to replace wires in this project
to minimize the size of the hand controller, two intro-level XBee 802.15.4 modules were enough.
According to Digi, the manufacture, “XBee modules are embedded solutions providing wireless
end-point connectivity to devices. These modules use the IEEE 802.15.4 networking protocol for
fast point-to-multipoint or peer-to-peer networking. They are designed for high-throughput
applications requiring low latency and predictable communication timing.”
In order to differ with other XBees with default settings, the channel (CH), PAN ID (ID), destination
address high (DH), destination address low (DL), and a 16-bit source address (MY) of the two XBees
were set as a pair by using XCTU. XCTU was a free multi-platform application designed to enable
developers to interact with XBee modules through a simple-to-use graphical interface. Finally,
experiments proved the two XBees connected: when the first XBee (defined as the transmitter)
sent a sentence, the second XBee (defined as the receiver) received the same sentence. Whenever
the XBees were configured, the setting was stored inside the chips. At this time, the wireless
33
connection between the hand controller and the modified board (step 3) was established. The
data from the hand controller would transmit through the XBees, which worked like wires.
7.2 Hand Controller: The hand controller required to collect hand gestures and movements by using a force touch pad
and a 9 DOF IMU. An Arduino Pro Mini would collect all these data (step 1), encode in the correct
format, and send to the XBee transmitter (step 2). Then, the XBee transmitter would send the data
through the wireless connection (step 3).
7.2.1 Force Touch Pad
The hand controller had two states:
- “Fist” represented the touch pad was pressed.
- “Palm” represented the touch pad was not pressed.
Different axes changed according to different state. The force touch pad worked like a resistor: its
resistance depended on the applied force. The force touch pad’s resistance would scale with the
applied force. To gather the force data, the hand controller included a voltage divider circuit to
measure the applied force as analog input. The circuit contained a 10k Ohm resistor and the force
touch pad. The analog value of the voltage across the force touch pad would be varied from 0 to
1023. When no force was applied to the touch pad, the value would be 1023, and when enough
force was applied, the value would decrease to 0. According to the design requirement, the force
touch pad only needed to be sensitive enough to a force that the user could comfortably apply.
Whether enough force was applied to the pad was more important than the precise value of the
force. So, the force touch pad was defined as “touched” when the analog reading was less than
34
800. If enough force applied on the touch pad, a “1” would be sent through the wire less
connection. On the opposite, a “0” would send if less force applied to the touch pad.
7.2.2 9 DOF IMU
The 9 DOF IMU had 3 axes, which were X, Y, and Z. But to mimic two joysticks, four axes were
required because each joystick had two axes. In this case, the Z axis value had two meanings, which
depended on the force touch pad’s state.
The 9 DOF IMU used I2C protocol, which required two wires to send data. At the same time, it
required 3.3V power input. The Arduino Pro Mini would supply correct power to the sensor. The
IMU had a well-design library, which was compatible with Arduino IDE. The accelerometer,
magnetometer and gyroscope data were combined and calculated to provide X, Y, and Z axis data
in Euler. In this project:
- X-axis represented yaw.
- Y-axis represented roll.
- Z-axis represented pitch and power.
According to the datasheet, the 9 DOF IMU had to be fully calibrated before used. The user had to
shake the sensor to initialize it, especially the magnetometer. After the calibration, the sensor’s
coordinate axes would build and settle based on the geological North direction.
35
7.2.3 Arduino Pro Mini (Microcontroller of the Hand Controller)
The 3.3V power of Arduino Pro Mini fitted all sensors, including the 9 DOF IMU. The 28 pins assured
enough inputs and outputs for this project. With previous experience, its support to Arduino IDE
provided better coding experience. A 9V battery was used as the power supply to the
microcontroller. A voltage regulator circuit was built on for the microcontroller to regulate the
voltage down to 3.3V. Then, the Arduino would supply power to other parts.
7.3 Modified Board: This section was not in the preliminary design. It was added to communicate the SYMA drone and
minimize the hand controller’s size. This extra section was built on bread board instead of soldered
on a perf board. The modified board contained:
- A control board (disassembled from a SYMA controller that came with the drone).
- Another Arduino Pro Mini.
- A wireless serial receiver (another XBee).
- A 6V battery pack.
The battery pack supplied power to both the SYMA control board and the microcontroller. The
SYMA control board required 6V voltage. When the sensors’ data were received (step 3), the
Arduino would store the data into local variables (step 4). After calculation, the Arduino would
output PWM signals to the SYMA control board (step 5). Finally, the SYMA control board would
send signal to the drone (step 6) and control its movements.
36
7.3.1 Control Board
The control board was disassembled from the controller that came with the SYMA X5C-1 drone.
Originally, the board had two joysticks. Each joystick had two potentiometers. When the
potentiometer changed, the voltage value of the corresponding pin would change. The voltage
range was 0-3.3V. Then, the chip on the controller would send command signals to the drone. In
order to mimic the joysticks, PWM signals would be directly sent to the board instead of changing
the voltage by potentiometers. Both of two joysticks were desoldered from the SYMA board. Then,
pins were soldered instead. After the modification, the control board had eight pins (four pairs)
for receiving the PWM signals.
7.3.2 Arduino Pro Mini (Microcontroller of the Modified Board)
This Arduino Pro Mini used its on-board regulator to regulate the voltage from 6V to 3.3V. Four
pins, pin 3, pin 11, pin 9, and pin 10, generated PWM signals to the modified control board. The
PWM signal from pin 3 and pin 11 were controlled by time 1, and the PWM signals from pin9 and
pin ten were controlled by time 2. All these PWM pins were set to fast PWM mode and highest
frequency, which was 62500Hz, in order to decrease the PWM period because the modified board
should not sense each pulse comparing to a steady voltage.
7.4 Data Transmission and Code The program for the hand controller would only gather data and send through the XBee
transmitter (step 1-3), and the program for the modified board should receive and store the data
(step 4). The sensor’s data were designed to transmit as quick as possible with necessary delay. A
code’s screenshot was shown in Figure 18. The code was recorded in Appendix D. Every three-axis
37
and the force touch pad data combined a group. In order to distinguish these data, the transmitter
added a special header, which was “XYZ” in a word, to each group of data. Also, the data always
transmitted in the order of X-axis data, Y-axis data, Z-axis data, and touch pad data right after the
header. When the modified board received the data (step 3), the X, Y, Z, and touch pad data should
store into corresponding variables (step 4). Each X, Y, Z axis value was divided into two bytes to
transmit since the value was possible to be larger than 155 (27). So, for each complete data group,
there are 3 + 2*3 + 1 = 10 bytes.
Figure 19. Code Screenshot
38
When the XBee on the modified board received enough bytes, which was a group of 10 bytes. The
program on the receiver side would check if the header was correct, which should be “xyz” (Figure
18, line 5). The data would discard if the header was incorrect. If the header matched, every two
bytes from the following 6 bytes would combine as an integer and store into the local X, Y, and Z
value. Finally, the last byte would be store into another variable that hold the touch pad’s state.
After all transmitted values were stored into local variables, the program would calculate the PWM
value for each axis according to these variables. For Arduino Pro Mini, a PWM value was in range
of 0-255, which gave a voltage of 𝑣𝑎𝑙𝑢𝑒
255×3.3𝑉. Then, an algorithm was designed to translate the
sensor’s value to PWM value (Table 7).
PWM Data Min
PWM Min
Data Max
PWM Max
Initial state
Touch Pad State
Note
X -- 0 -- 255 127 Pressed +1 if 0<valX<50; -1 if 120<valX< 170
Y -50 0 50 255 127 No Matter Scale on valY
Z -50 0 50 255 127 Not
pressed Scale on valZ
Power (Altitude)
-- 0 -- 255 0 Pressed +5 if valZ > 20; -10 if valZ < -20
Table 7. Relationship between PWM Value and Sensor Value
The drone should stay balance and on the land at the initial state. So, the initial values of the X, Y,
Z axis were 127, which equaled 255
2. No power should be supplied to the drone, which assigned
power as 0. Depended on the touch pad’s state, the data of Z axis would either change the pitch
action or the altitude (power) of the drone. The roll and pitch had direct response to the data’s
change because they didn’t evolve any change to the drone’s coordinate system. On the opposite,
the yaw would turn the drone coordinate, which should smoothly change. For safety consideration,
39
the altitude (power) should also change smoothly. All of these changes would response to any
valid received data (Figure 20).
Figure 20. Software Flow Chart
40
8. Performance Estimates and Results Different from the preliminary design, the actual implementation had an extra modified board and
didn’t implement the built-in functions. The project needed an extra modified board since the
SYMA drone was only able to connect with a SYMA controller. The built-in functions, with was
initially designed by using the five pins capacitive touch sensor, was not included in this project
because the time run out. However, it was still able to implement in the future development.
To control a drone, a controller had to control four basic movements were required: pitch, yaw,
roll, and altitude (power). This project should be able to fully control a drone, which controlled the
four movements in reasonable reaction time. The pitch and roll should respond to correlated
gestures in real time. And the yaw and altitude should be changed smoothly. The performance
criteria for this project was defined as:
- The controller should attach to a single hand and detect hand movements.
- The drone should be able to stay balanced in the air.
- The drone should be able to respond to Roll and Pitch commands in real time.
- The drone should be able to change in Altitude and Yaw smoothly.
Since this project focused on the control part, the drone’s performance would determine the
controller’s performance. In general, the controller was able to control the drone by controlling
the four basic movements fully. According to the designed performance criteria, each single test
was examined.
41
First, the controller fitted single hand. The size of the controller was reasonable as a single-hand
attachment. The force touch pad was comfortable and could sense applied force in an acceptable
range, which won’t require any uncomfortable presses from the pilot. The 9 DOF IMU was able to
detect user’s gestures, which were divided into X, Y, Z axis and recorded as numeric values. At the
same time, the force touch pad would record the fingers’ movements, whether making a “fist”
(force applied on the touchpad) or staying in “palm” (no force applied on the touchpad). However,
the weight of the controller was a little bit heavy for a single hand since the 9V battery was heavy.
In the future development, a lighter power supply would be better. Also, the two perf-boards that
contacted hand were not comfortable. They also made hand sweating a lot. In ideal, a softer and
better ventilation material would improve the hand controller’s performance.
Second, the drone was not able to stay perfectly balanced in the air. But the balance was good
enough comparing to the drone’s balance that controlled by a traditional controller. Because the
controller was a little bit heavy for single hand usage, the drone would be hard to stay in balance
after a time of use.
Third, the drone was able to response to pitch and roll commands in real time. In fact, a small
delay existed between the hand gestures and the drone’s movements. But the delay was short,
less than 0.3s, and could be accepted. In this project, since the self-built controller didn’t have
direct control access to the drone, a modified board section was added as a joint section between
the hand controller and the drone. This extra part might cause the short delay because the signals
had to transmit through the modified board, which took more time.
42
Then, the drone was able to change in altitude and yaw smoothly. For yaw, the PWM would
increase or decrease one each time. And for altitude, the PWM would increase 5 or decrease ten
each time. For the test purpose, these two movements worked. However, it was hard to find one
exact value while driving the drone. For example, whenever the user increased the altitude, the
drone was impossible to stay balanced again because the altitude’s balance value (F=mg, where
‘F’ was determined by the speed of the drone’s motors, ‘m’ was the drone’s weight, and ‘g’ was
the gravity) was impossible to get in a short time.
All in all, the drone’s performance was reasonable and acceptable. A single hand was able to use
the hand controller. The controller was able to connect the drone. And all four basic drone’s
movements worked. At the same time, the criteria were mostly met. In the future development,
a lighter battery and material would improve the hand controller’s weight and comfortableness.
The direct access to the drone would eliminate the short delay between the controller and the
drone and the extra modified board section. Also, a better algorithm to translate sensor data into
PWM signals would give a more precise changing process for the yaw and altitude.
43
9.Production Schedule This project was finished based on weekly meeting and efforts. The whole project was divided into
smaller experiments and missions to each week:
Week 1 (1/9):
Experiment 1, combine sensors and write demo to control LEDs
Test parts and sensors.
Week 2 (1/16):
Research on microcontroller and wireless modules, order parts
Connect XBees to test wireless transmission
Week 3 (1/23):
Experiment 2, combine sensors with microcontrollers and transmit data through wireless
modules
Test data transmission
Week 4 (1/30):
Continue Experiment 2, test prototype with drone
Write program to control the drone
Week 5 (2/6):
Analyze test result, make further plan
Improve algorithm, test by controlling the drone
Week 6 (2/13):
Finish the first prototype, wire-wrap on perf board, test
Week 7 (2/20):
44
More tests and changes on the prototype
Week 8 (2/27):
Solder the parts to perf board
Make PPT for presentation
Week 9 (3/6):
Finish the project, finalize everything, including hand controller and modified board
Demo & Final Report
Week 10 (3/17):
Final Report & Website
45
10. Cost Analysis This project built a single-hand gesture controller for quadcopters. There were three sections: a
hand controller, a modified board, and a test drone. The costs of each section were recorded
below.
10.1 Hand Controller Cost
Part Cost ($)
9 DOF IMU 34.95
Square Force TouchPad 7.95
XBee 802.51.4 22.95
Arduino Mini Pro 3.3V 8MHz 9.95
Resistors, Capacitors, Regulators, Wires,
Boards…
2
Total 77.8
10.2 Modified Board Cost
Part Cost ($)
Arduino Mini Pro 3.3V 8MHz 9.95
SYMA Controller 15.29
XBee 802.51.4 22.95
Resistors, Capacitors, Regulators, Wires,
Boards…
3
Total 51.19
46
10.3 Test Drone Cost
Part Cost ($)
Drone 40
Total 40
The total cost of the project is 77.8+51.19+40 = 169$, without tax. Within this cost, a SYMA drone
cost 55$. Comparing to other controllers, which are around 20$ from the market, this controller
cost more. However, the total cost is reasonable and acceptable because this controller is only a
test version. In the future development, the modified board can eliminate if the controller has
direct access to the test drone. The 9 DOF IMU, force touch pad, and XBee will cost less if they are
buy in amount. Finally, the estimated cost for a future single hand gesture controller will be in the
range of 50~60$. Although the price is still higher than the normal controller, the single-hand
gesture controller is worth since it requires single hand and is controlled by the gestures.
47
11. User’s Manual (Video available on https://muse.union.edu/2017Capstone-liuy/) 1. Wear the hand gesture controller. The larger section with chips should be on the hand back.
The force touch pad should be on the palm.
2. Adjust the two perf-boards and the rope to a comfortable position.
3. Tighten the rope on the force touch pad side. Connect the battery. If the Arduino works, the
LEDs on the Arduino will blink.
Figure 21. Wear the Controller
4. Connect and turn on the power supply to the modified board. If the Arduino works, the LEDs on
the Arduino will blink.
5. Turn on the SYMA control board. The blue LED on the board will blink. At the same time, a sound,
beep, can be heard.
48
Figure 22. Turn on the Modified Board and Connect to the Drone
6. Turn on the drone. The LEDs on the drone should blink rapidly.
7. Reset the two Arduinos. Fully calibrate the IMU by shaking the hand.
Figure 23. Fully Calibrate the IMU
49
8. Press the force touch pad and pitch hand up until hearing a beep. Then, continually press the
force touch pad and pitch hand down until hearing another beep. The drone should already
connect with the controller if the drone’s and the modified control board’s LEDs are not blinking.
Figure 24. Fly the Drone!
9. Control the drone by gestures.
Hand Gesture Drone’s movement
Roll Roll
Pitch without pressing the touchpad Pitch
Pitch with pressing the touchpad Change in altitude (Power)
Yaw with pressing the touchpad Yaw
Table 8. Hand Gestures and Correlated Drone Movements
50
12. Discussion, Conclusion, and Recommendations
12.1 Discussion Drones technology was relatively new and rapidly developing. Even though the ethical concerns
about the drone were still debatable, drones were widely used in agriculture, land management,
GIS, warfare, energy, and architecture. In most of the drone technology applications, only one
hand of the pilot would be available. For example, when an architect was using a drone to inspect
the high-level structure of a building, he always needed an assistant to hold the building’s blueprint
for him while he was using the controller. In this case, a single-hand controller would be helpful
and more convenient. At the same time, the traditional controller used two joysticks to control
the drone’s movement. In America, the left joystick would control the altitude and yaw. And the
right joystick would control the pitch and roll. However, this two-joystick control mode was not
intuitive enough to most of the intro-level pilots. Most newbies needed several experimental
flights to understand how to control the drone. After then, the pilots still needed more practices
to be able to fly a drone safely. With all of these considerations, this project focused on building a
single gesture controller for drones, especially quadcopters, as a replacement of the traditional
two-hand controllers.
Since this project only focused on the controller part, a test drone, SYMA X5C-1, was bought from
the market. Unfortunately, the drone’s flight chip was encoded and only able to connect with
SYMA controller. To minimize the size of the part on hand, the hand controller would communicate
to an extra modified board, which was disassembled and redesigned from a SYMA controller. Then,
the modified board was able to control the drone. A 9 DOF IMU and a force touch pad would
51
detect the user’s hand gesture. Two XBee wireless modules established the wireless
communication between the hand controller and the modified board. Finally, two Arduino Pro
Mini collected, transmitted, and calculated the correlated PWM signals to mimic the two joysticks’
movements.
One of the most important features of a drone was its balance. This controller should be able to
manipulate a drone’s movement, including pitch, yaw, roll, and altitude, by using the gesture. At
the same time, the size of the controller should be reasonable as a single-hand attachment.
12.2 Conclusion and Recommendations The controller’s overall performance met the designed criteria, which were:
- The controller should attach to a single hand and detect hand movements.
- The drone should be able to stay balanced in the air.
- The drone should be able to respond to Roll and Pitch commands in real time.
- The drone should be able to change in Altitude and Yaw smoothly.
Even though there were some minor flaws, the hand controller was able to fully control the test
drone. The size of hand controller was reasonable. The weight was a little bit heavy as a single-
hand equipment. But a lighter power supply and material would solve the problem. The pitch and
roll movement could trigger response to the correlated hand gestures in real time with a short
delay (less than 0.3s). The delay didn’t have a negative effect on the controlling in the tests. But if
the hand controller had direct access to the drone, the extra modified board section could be
eliminated. Then, the delay might be shorter or undetectable by human eyes. The altitude and
52
yaw changes were smooth. However, it was hard to find one exact value while driving the drone.
For example, whenever the user increased the altitude, the drone was impossible to stay balanced
again because the altitude’s balance value (F=mg, where ‘F’ was determined by the speed of the
drone’s motors, ‘m’ was the drone’s weight, and ‘g’ was the gravity) was impossible to get in a
short time. In the future development, a better algorithm to change the altitude and yaw would
improve the performance and preciseness.
53
References: 1. "UAV / Drone Solutions for Mapping, Aerial Inspection, Unmanned Cargo." UAV / Drone
Solutions for Mapping, Aerial Inspection, Unmanned Cargo. N.p., n.d. Web. 23 Nov. 2016.
2. Lidwell, William, Kritina Holden, Jill Butler, and Kimberly Elam. Universal Principles of
Design: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better
Design Decisions, and Teach through Design. Beverly, MA: Rockport, 2010. Print.
3. "Quadcopters & Aerial Drones - Yuneec." YUNEEC. N.p., n.d. Web. 23 Nov. 2016.
4. "What You Need to Know Before Flying a Drone (To Stay Out of Trouble)." HowTo Geek
RSS. N.p., n.d. Web. 23 Nov. 2016.
5. Mockaitis, Tom. "Drones and the Ethics of War." The Huffington Post.
TheHuffingtonPost.com, 12 Jan. 2016. Web. 13 Mar. 2017.
6. Dempsey, Caitlin. "Drones and GIS: A Look at the Legal and Ethical Issues ~ GIS
Lounge." GIS Lounge. N.p., 18 Dec. 2016. Web. 13 Mar. 2017.
7. "Global market for commercial applications of drone technology valued at over
$127bn." Press room. N.p., n.d. Web. 13 Mar. 2017.
8. Dr. Helmuth Ludwig, Executive Vice President - Siemens, and Pamela Adams. "37 Critical
Problems That Need to Be Solved for Drone Delivery to Become Viable." DaVinci Institute
– Futurist Speaker. N.p., 2016. Web. 23 Nov. 2016.
9. Culver, Kathleen Bartzen. "From Battlefield to Newsroom: Ethical Implications of Drone
Technology in Journalism." Journal of Mass Media Ethics 29.1 (2014): 52-64. Web.
10. Motion, Leap. "Leap Motion." Leap Motion. N.p., n.d. Web. 23 Nov. 2016.
11. "MTi 1-Series." Xsens : 3D Motion Tracking - Xsens Online Shop. N.p., n.d. Web. 23 Nov.
2016.
54
Appendix A
Sensor Descriptions Interface voltage
Adafryut 9-DOF Absolute Orientation
IMU Fusion Breakout BNO055
sensor data from an
accelerometer,
gyroscope and
magnetometer into
actual "3D space
orientation"
I2C 3.3V
Adafruit 10-DOFIMU Breakout –
L3GD20H+LSM303+BMP180
11 axes of data: 3
axes of accelerometer
data, 3 axes
gyroscopic, 3 axes
magnetic (compass),
barometric
pressure/altitude and
temperature
I2C 3V or
5V
L3GD20H Triple Axis Gyro Breakout
Board – L3GD20/L3G4200 Upgrade –
L3GD20H
A gyroscope which
can sense twisting and
turning motions.
I2C or
SPI
3.3V
or 5V
ADXL326 – 5V ready triple-axis
accelerometer (+-16g analog out) 3 axis accelerometer analog
3.3V
or 5V
FLORA 9-DOF
Accelerometer/Gyroscope/Magnetome –
LSM9DS0 – v1.0
3 axes of
accelerometer data, 3
axes gyroscopic, 3
axes magnetic
(compass)
I2C
Round Force-Sensitive Resistor (FSR) –
Interlink 402
-Detect physical
pressure, squeezing
and weight
-Not accurate
Resistor
(Digital
or
Analog)
Square Force-Sensitive Resistor (FSR) –
Interlink 406
-Detect physical
pressure, squeezing
and weight
-Not accurate
Resistor
(Digital
or
Analog)
Short Flex Sensor
Detect flexing or
bending in one
direction
Resistor
(Digital
or
Analog)
Muscle Senor Surface EMG electrodes –
H124SG + MyoWare Muscle Sensor
Detect muscle
movement of the arm
(in order to detect fist)
Analog 2.9-
5.7V
55
Adafruit 12-Key Capacitive Touch
Sensor Breakout – MPR121
- easy-to-use 12-
channel capacitive
touch sensor
-can handle up to 12
individual touch pads
I2C 3V or
5V
Standalone 5-Pad Capacitive Touch
Sensor Breakout
easy-to-use 5-channel
capacitive touch
sensor
digital 1.8-
5.5V
56
Appendix B Sensors：
The sensors are able to work correctly and transmit correct data to the microcontroller. Microcontroller: The microcontroller should collect data from sensors correctly. At the same time, it should be able to convert the inputs in to proper signal. Finally, the output should send to the wireless module. Wireless Module: The wireless module on the remote controller should get the data from the microcontroller and transmit the data to the other receiver on the drone. The wireless module on the drone should be able to take the data and sent the control signals to the drone. Drone: The drone should be able to work accordingly by using the default controller. Then, the receiver of the wireless module should be able to generate correct signal to control the drone with following basic movements. Basic Movements: 1) Fly up and land
a) The drone is about to lift off and land as operation indicated. b) The drone is can detect gestures and figure out hold mode and the drive mode after
successful lift off. c) The land process is able to automatically finish by the drone itself without driver’s helps.
2) Rotate, yaw, and pitch a) Rotate with proper angle. b) Yaw with proper angle. c) Pitch with proper angle. d) Hold in air while doing the actions. e) Do correct movements while in drive mode.
Test Sequence: 1) Test the single sensor -> to find out their signal patterns and how they transmit the data
through I2C. 2) Test the connection of the sensors to the microcontroller -> test all sensors’ connections
status when they are all added to the microcontroller. 3) Test the connection between the microcontroller and the wireless module -> whether the
transmitter and receiver are able to work correctly -> find out the maximum or optimum work range.
4) Test the receiver’s controlling to the drone -> the receiver should be able to send correct signal to control the drone’s movements -> find out whether a new controller should be used
57
to control the drone or not. 5) Test the basic movements of the drone -> to prove the glove controller is able to control the
drone correctly and efficiently. Detailed Test Methods: Single sensor tests: 1) Touch sensor:
a) Connect the sensor with multimeter and power b) Touch the sensor and read the corresponding result c) Touch the sensor one by one to check whether the output is representing the same
touch situation d) Rapidly touch a single sensor 5 times, the output should have exactly 5 high peaks e) Touch a single 3 times, wait for some time, then touch 2 times more. The output should
have exactly 5 high peaks f) Touch whichever two sensors, the result should have two high voltages
correspondingly g) Touch all of the five sensors, the output should all be high at this point
2) Palm sensor：
a) Connect the sensor with multimeter and power b) Touch the sensor and read the corresponding result c) Put the sensor with palm (or maybe stick it to palm), touch and read the result d) Do the same part d) and e) from touch sensor tests e) Use two or more figure to touch the pad. The output should have only one output
3) 9DOF IMU a) Connect the sensor with microcontroller and computer. The sensor will be placed on
the back of the hand b) Wave the hand with the sensor, “random” data that describe the movement should be
read from the screen c) Do rotation along x, y, and z axis with 45, -45, 90, -90, 180, and -180 degree to check
whether the data from the sensor is correct or not d) Rotate hand to get the maximum output e) Rotate hand along x, y, and z axis and quickly go back the opposite direction. The output
should be able to detect the movement by output negative outputs. f) Move hand (but not rotate) in different directions to check the accelerometer is
working g) Move hand forward quickly and then move backward. The output of the accelerometer
should be positive while moving forward and negative while moving backward h) Same test g) should be done along different axis. i) Do the translation from the output to the real-world data by using the manual book to
check whether the data are making sense or not. Wireless module tests: 1) Connect the transmitter and receiver 2) Send from number 1 to 10 one by one. Check at the receiver side to assure correct
58
transmission 3) Send string “Hello World”. Check at the receiver side to assure correct transmission 4) Send a long data, such as 128 bits. Check at the receiver side to assure correct transmission 5) Move the receiver to another room and check the connection 6) Check the connection outdoor to assure it works well in outdoor environment Microcontroller tests: 1) Connect each single sensor to the microcontroller and do the previous tests. The received
output by the microcontroller should be as same as the previous tests 2) Connect wireless module to the microcontroller and program to send previous test data. The
transmitter and receiver should work as same as they work in the previous tests (assume the wireless module passed the previous tests)
3) Connect all sensors to the microcontroller and do multiple test at once. For example, c) from 9DOF IMU test and e) from touch sensor test. The output should be the combination of those tests.
59
Appendix C
Arduino Pro Mini 3.3V 8MHz
Teensy 3.2 PRO Micro 3.3V 8MHz
Price $9.95 $19.95 $19.95 Voltage 3.3V 3.3V 3.3V
Frequency 8MHz 72MHz 8MHz Dimension 0.7x1.3" (18x33mm)
1.4 x 0.7" (~35 x 18 mm)
1.3x0.7"
Chip ATmega328
32 bit ARM Cortex-M4
ATMega 32U4 Pins Analog Pins: 8
Digital I/Os: 14
21 High Resolution Analog Inputs (13 bits usable, 16 bit hardware)
34 Digital I/O Pins (5V tolerance on Digital Inputs)
12 PWM outputs
9 x 10-bit ADC pins
12 x Digital I/Os (5 are PWM capable)
UART and I2C Support Yes Yes Yes
Comments Excellent! Support Arduino IDE
Need extra library to use Arduino IDE
Poor quantity USB port
60
Appendix D
Transmitter Side (Hand Controller) #include <Wire.h> #include <SoftwareSerial.h> #include <Adafruit_Sensor.h> #include <Adafruit_BNO055.h> #include <utility/imumaths.h> int touchPin = A0; SoftwareSerial mySerial(7, 8); /* This driver reads raw data from the BNO055 Connections =========== Connect SCL to analog 5 Connect SDA to analog 4 Connect VDD to 3.3V DC Connect GROUND to common ground */ /* Set the delay between fresh samples */ #define BNO055_SAMPLERATE_DELAY_MS (100) Adafruit_BNO055 bno = Adafruit_BNO055(); /**************************************************************************/ /* Arduino setup function (automatically called at startup) */ /**************************************************************************/ void setup(void) { Serial.begin(9600); Serial.println("Orientation Sensor Raw Data Test"); Serial.println(""); mySerial.begin(9600); /* Initialise the sensor */ if(!bno.begin()) { /* There was a problem detecting the BNO055 ... check your connections */
61
Serial.print("Ooops, no BNO055 detected ... Check your wiring or I2C ADDR!"); while(1); } delay(1000); bno.setExtCrystalUse(true); Serial.println("Calibration status values: 0=uncalibrated, 3=fully calibrated"); } /**************************************************************************/ /* Arduino loop function, called once 'setup' is complete (your own code should go here) */ /**************************************************************************/ void loop(void) { // Possible vector values can be: // - VECTOR_ACCELEROMETER - m/s^2 // - VECTOR_MAGNETOMETER - uT // - VECTOR_GYROSCOPE - rad/s // - VECTOR_EULER - degrees // - VECTOR_LINEARACCEL - m/s^2 // - VECTOR_GRAVITY - m/s^2 imu::Vector<3> euler = bno.getVector(Adafruit_BNO055::VECTOR_EULER); char x = 'x'; char y = 'y'; char z = 'z'; int varX = euler.x(); int varY = euler.y(); int varZ = euler.z(); // Serial.println(varX); byte bX[2]; bX[0] = highByte(varX); bX[1] = lowByte(varX); byte bY[2]; bY[0] = highByte(varY); bY[1] = lowByte(varY); byte bZ[2]; bZ[0] = highByte(varZ); bZ[1] = lowByte(varZ); mySerial.write(x);
62
mySerial.write(y); mySerial.write(z); mySerial.write(bX, 2); mySerial.write(bY, 2); mySerial.write(bZ, 2); Serial.print("X: "); Serial.print(varX); Serial.print("; "); Serial.print("Y: "); Serial.print(varY); Serial.print("; "); Serial.print("Z: "); Serial.print(varZ); Serial.println("; "); char touch; if (analogRead(touchPin)<800){ touch ='1'; }else {touch = '0';} mySerial.write(touch); delay(BNO055_SAMPLERATE_DELAY_MS); }
Receiver Side (Modified Board) #include <Wire.h> #include <SoftwareSerial.h> SoftwareSerial mySerial(7, 8); // the communication wire char touch; int previousPower = 0; String readString; //timer 1 int pwmXPin = 3; int pwmYPin = 11; //timer 2 int pwmZPin = 10; int pwmPower = 9; int pwmXVal = 127; int pwmYVal;
63
int pwmZVal; int pwmPowerVal; long valX = 0; long valY = 0; long valZ = 0; long valPower = 0; void setup(void) { // baud rate Serial.begin(9600); mySerial.begin(9600); // print Serial.println("Getting Data"); Serial.println(""); // set pwm type pinMode(pwmXPin, OUTPUT); pinMode(pwmYPin, OUTPUT); pinMode(pwmZPin, OUTPUT); pinMode(pwmPower, OUTPUT); // set timer 1 to fast pwm with prescale 1 TCCR1A = _BV(COM1A1) | _BV(WGM10); TCCR1B = _BV(CS10) | _BV(WGM12); // set timer 2 to fast pwm with prescale 1 TCCR2A = _BV(COM2A1) | _BV(COM2B1) | _BV(WGM21) | _BV(WGM20); TCCR2B = TCCR2B & 0b11111000 | 0x01; // initialize the pwm outputs analogWrite(pwmPower, 0); analogWrite(pwmXPin, 127); analogWrite(pwmYPin, 127); analogWrite(pwmZPin, 127); } void loop(void){ while(mySerial.available() < 10); char x = mySerial.read(); char y = mySerial.read(); char z = mySerial.read(); if (x == 'x' && y == 'y' && z == 'z'){
64
valX = mySerial.read() << 8; valX |= mySerial.read(); valY = mySerial.read() << 8; valY |= mySerial.read(); valZ = mySerial.read() << 8; valZ |= mySerial.read(); touch = mySerial.read(); Serial.print("X: "); Serial.print(valX); Serial.print("; "); Serial.print("Y: "); Serial.print(valY); Serial.print("; "); Serial.print("Z: "); Serial.print(valZ); Serial.print("; "); Serial.print("Touch: "); Serial.print(touch); Serial.print("; "); if (touch == '1'){ Serial.print("Pressed!"); } testX(); } } void testX(){ if (valY >-50 && valY < 0){ pwmYVal = ((0-valY)*255)/100+127; }else if(valY < 50 && valY >=0){ pwmYVal = ((50-valY)*255)/100; }else if(valY < -50){ pwmYVal = 255; }else if (valY > 50){ pwmYVal = 0; } analogWrite(pwmYPin, pwmYVal); Serial.print("Ypwm:"); Serial.println(pwmYVal); if (valX > 120 && valX < 200 && touch == '1'){ if(pwmXVal >= 1){ pwmXVal -=1; }
65
}else if (valX > 0 && valX < 50 && touch == '1' ){ if(pwmXVal <=254){ pwmXVal +=1; } } analogWrite(pwmXPin, pwmXVal); Serial.print("Xpwm:"); Serial.println(pwmXVal); if (touch == '1'){ //touch pad is pressed if (valZ >20 && previousPower < 255){ previousPower+=5; }else if (valZ<-20 && previousPower >0){ if (previousPower >=10){ previousPower-=10; }else{ previousPower =0; } } analogWrite(pwmPower, previousPower); Serial.print("Power pwm:"); Serial.println(previousPower); }else if (touch == '0'){ if (valZ >-50 && valZ < 0){ pwmZVal = ((0-valZ)*255)/100+127; }else if(valZ < 50 && valZ >=0){ pwmZVal = ((50-valZ)*255)/100; }else if(valZ < -50){ pwmZVal = 255; }else if (valZ > 50){ pwmZVal = 1; } analogWrite(pwmZPin, pwmZVal); Serial.print("Zpwm:"); Serial.println(pwmZVal); analogWrite(pwmPower, previousPower); } }
