Upload
others
View
21
Download
0
Embed Size (px)
Citation preview
Computer Vision in FRC
By Max DeVos
What is Computer Vision?
▸ Process of taking an image (often using a camera) and manipulating it into usable data.
2
Vision Processor RoboRIOCamera
Target
What isn’t Computer Vision?
▸ Process of taking an image and displaying it to a human for interpretation.
3
RoboRIOCamera
Target
Driver
How Vision Processing WorksAcquiring, Transferring, and Using Data
Transferring Vision DataThe process of moving the data from the processor to a place where it can be used by the central robot controller to adjust actuators
The 3 Parts of Computer Vision Tracking
Acquiring Vision Data
The process of using some form of processor for collecting and manipulating camera input to get meaningful data and information from it
Using Vision DataThe use of the Vision data to move actuators and solve engineering problems using this new sensing ability.
5
Camera Vision Processor RoboRIOVision
Processor RoboRIOMotor
Piston
Acquiring Vision DataThe Process of Turning Camera Input into Usable Data
Cameras - Acquiring Vision Data
▸ The camera is the sensor for Vision Tracking.
▸ Things to look for:▸ USB Real-Time Transfer ▸ Aperture Priority▸ High frame rate▸ High Field of View (FOV)▸ Mountability (can I get this on my robot)▸ Affordability (You should have at least 2)▸ Durability
7
Camera
Choosing a Camera Type - Acquiring Vision Data
Standard RGBPros:
▸ Easier to find▸ More variety▸ Often higher FOV▸ Higher resolution
Cons:▸ Doesn’t work as well with retro-reflective▸ Can interfere with other teams’ wavelengths▸ Harder to filter and use for vision tracking▸ May have to be turned off for light rules
Example:Microsoft Lifecam HD-3000
8
Infrared (IR)Pros:
▸ Works far better for Vision tracking▸ Doesn’t interfere with other teams’ light▸ Doesn’t have to be turned off for light rules▸ Drastically reduces misdetections
Cons:▸ Not as many of them (hard to find)▸ Sometimes have smaller FOV▸ Often aren’t USB
Example:Infrared Camera Module (NoIR) V2
The Vision Processor- Acquiring Vision Data
▸ The Vision Processor is the part of the system that runs the program that creates the usable data from the camera images
▸ Runs the program on startup▸ Program written in C++, Python, or Java
9
Vision Processor
▸ Recommended ▸ NVidia Jetson Tegra K1▸ NVidia Jetson Tegra X1▸ NVidia Jetson Tegra X2
▸ Many available offboard processors ▸ Almost any Linux-powered development board that has either 3
USB ports or 2 USB ports and an ethernet port will work as a vision processor.
▸ Processor needs to be powerful enough
Vision Processor Options - Acquiring Vision Data10
▸ Other Options▸ Raspberry Pi▸ BeagleBoard/BeagleBone▸ Intel Galileo▸ A10 Linux Dev Board
The Vision Processing ProgramThe Software Itself
The Program Pipeline - Acquiring Vision Data
▸ The program runs a script on loop to handle new images.
12
Input
The image from the camera is loaded into the program as a 2D
array of colored pixel data called a Mat
Image Manipulation
The image is blurred and converted into a format that can
be easily filtered.
Calculations
The pixel points of the target received from the filtering step are used to run calculations and compared to the last image to verify sensibility.
1 3
4
2
Filtering
The program filters the image using both color and geometry to achieve a set of pixel coordinates of the points of the target
Step 1 - “Input”
▸ The latest still frame is taken from the camera and inputted into the program
▸ A VideoManager object will handle this for you
13
Python
C++
Step 2 - “Image Manipulation”
The image is blurred and converted into a format (HSV) that can be easily filtered.
14
Blur Color Conversion
Blur - Step 2 “Image Manipulation”
Blurring the image may seem counter-intuitive, but it serves the purpose of eliminating imperfections and oddities from an image to get more accurate filtering and detection.
Python
C++
15
Color Conversion - Step 2 “Image Manipulation”
Converting the type of the image is done to allow for a more intuitive filtering process. It is important to understand the different types of image color storage.
16
Color Storage Formats - Step 2 “Image Manipulation”
BGR (RGB)Classic images, contain a different amount of blue, green, or red to create any color. Terrible for filtering because of 3 different hue values.Note: For some reason OpenCV refers to this format by default as BGR. Your IDE may suggest an BGR enum, but do not use it.
HSV:A coloring format optimized for filtering.Like BGR, has 3 data points, but instead of different amounts of base colors includes one value (H) for the hue range, (S) for saturation (difference from white), and V (difference from black).
17
Mask Image (Boolean Image)Pixels are activated or inactivated.Used for contour detection and math. Often referred to as a 2-channel image/mask.
Color Conversion- Step 2 “Image Manipulation”
In this step, you will convert your image from the unfriendly BGR format to the filter-friendly HSV format. You will use an OpenCV function called inRange.
Python
C++
18
Geometric Filtering
Step 3 - “Filtering”
The program filters the image using both color and geometry to achieve a set of pixel coordinates of the points of the target
19
Color Filtering ContourDetect
Color Conversion- Step 3 “Filtering”
In this step, you will convert your image from HSV to an Boolean Mask using upper and lower color filters that you tune to the field.
Python
C++
20
Contour Detection - Step 3 “Filtering”
Now we need to tell the computer to convert those white “active” pixels into blobs of coordinate data. OpenCV has a function to do this for you called FindContours.
Python
C++
21
Geometric Filtering - Step 3 “Filtering”
The final step in the filtering process is called Geometric filtering. The point of geometric filtering is to remove any misdetections based on their shape or size.
22
Some things that can be filtered for:▸ Area▸ Perimeter▸ Aspect Ratio▸ Location on screen▸ Location relative to
other shapes
Geometric Filtering (cont.) - Step 3 “Filtering”
An example of using geometric filtering would be the peg rectangles in Steamworks.
23
How it’s done:▸ For loops, filtering out
contours that are too small or oddly shaped
▸ Sorting functions to remove small,rectangular misdetections.
▸ Final, large scale sorting to determine final target(s) of out of remaining contours
Geometric Filtering (cont.) - Step 3 “Filtering”
Example Pipeline:
24
All Contours Array
Large Contours Array
For: Larger than 100px2
For: Aspect Ratio ~ 2/5
Correct Shape
Sorting: Largest Two
Final Contour(s)
Step 4 - Calculations
The pixel points are used to run calculations and compared to the last image to verify sensibility.
25
Geometric Math Comparison
Absolute Tuning:Using inverse homology and other complex math to find exact pose of target, and then tuning robot mechanisms to that data.
Relative Tuning:Using simple math and geometric properties to find relative position of target, and then tuning robot mechanisms to that data.
Relative Tuning - Geometric Calculations
This step is where the coordinate point data is used to interpret information about your target. However, the use of relative tuning can make the math aspect of this simple.
26
Solving Math Problems - Geometric Calculations
▸ Finding Actual Center of a Target▹ Use the X system
▹ Draw lines from opposing corners and find point of intersection▹ Solves perspective problem by cancelling corners
27
Note:
To determine which corner is which, sort them by slope from the origin, take most extreme two and least extreme two and sort those by their X or Y value to ensure correct corners.
Important Math Functions - Geometric Calculations
Yaw
28
Pitch
▸ Finding Yaw/Pitch Offset to Target▹ Used for alignment to target around an axis
TargetX = The X value of the center of your targetTotalX = Total Width of screenThis will return a value -1 to 1 of how much the robot needs to move/rotate-1 is left, 1 is right
TargetY = The Y value of the center of your targetTotalY = Total Height of screenThis will return a value -1 to 1 of how much the robot needs to move1 is up, -1 is down
Comparison- Step 4 “Calculations”
Due to the nature of misdetections, it’s important to verify that data makes sense. This can be done through comparing data to previous data.
29
Store
If the data is good, store in in the place of the previous data for future comparisons.
Timestamp
Use your language’s time system to assign a variable to the time of the frame for latency comparison
Compare
Compare your current data to your most recent data to verify that it is possible. If not, disregard that data and wait for next data.
Transferring Vision DataMoving Vision Data from the Processor To Your Robot Controller
Methods of Communication - Moving Vision Data
Serial Communication
▸ Pros▹ Faster
▸ Cons▹ More difficult▹ Third Party Serial
Manager
31
Network Communication
▸ Pros▹ Easier▹ Built in support for
multiple values▸ Cons
▹ Slower▹ Requires a radio port
▸ NetworkTables Library▹ Built into WPILib▹ Can be compiled
independently▹ PyNetworkTables for
Python▹ System of sending
multiple values over radio
Network Communication - Moving Vision Data32
▸ Network Configuration▹ Use RoboRIO as Server▹ Use Coprocessor as Client▹ Send data over radio
Vision Processor RoboRIORadio
Using Vision DataMoving Vision Data from the Processor to the Robot Controller
Using Vision Data to Actuate Actuators
▸ This is where your Vision data will be used to move the robot.
▸ The values you have available:▹ Yaw Offset (-1 to 1)▹ Pitch Offset (-1 to 1)▹ Average height of contour for distance
34
Relative Tuning:Using simple math and geometric properties to find relative position of target, and then tuning robot mechanisms to that data.
We can use relative tuning to find a function that tells the robot specifically how to behave based on sensor data, in this case, Vision data.
Example:Sensor Data: Height of contour for distance Feedback: RPM of flywheelGoal: Adjust RPM of flywheel to match distance
Relative Tuning - Using Vision Data35
Vision Processor
RoboRIO PID Loop
Motor
Piston
1. Create a data table2. Use a dynamic variable as your X
axis (height of target contour)3. Use your robot adjustment for
your Y axis (flywheel RPM to get to goal)
4. Repeat a lot of times at different X values
5. Plot the data & create a line of best fit
6. Use that equation in your programming for interpreting vision data.
Relative Tuning - How To36
Thanks!Any questions?You can email us at [email protected] me personally at [email protected] see me after the presentation :)Presentation can be found on https://kingtec2169.com/resources/documents/