Embed Size (px)
Citation preview
Overview
• Introduction
• Installation
• OpenCV basic types
• Image representation
• Image processing
• Video processing
OpenCV -History
• OpenCV (Open Source Computer Vision) is a library of programming functions mainly aimed at real-time computer vision.
• Intel team from Nizhny Novgorod created the library• Vadim Pisarevsky – the largest single contributor to the library
• Gary Bradski – in 1999. launched OpenCV with the hopes of accelerating computer vision and artificial intelligence by providing a solid infrastructure for everyone working in the field.
• Support: Intel (IPP), Google, Itseez, Arraiy
Nizhny Novgorod
OpenCV licensing
• “OpenCV is released under a BSD license,it is free for both academic and
commercial use.”
“The BSD License allows proprietary use, and for the software released under the license to be incorporated into proprietary products. Works based on the material may be released under a proprietary license or as closed source software. This is the reason for widespread use of the BSD code in proprietary products, ranging from Juniper Networks routers to Mac OS X.
Why OpenCV?
• Speed• designed for computational efficiency and with a strong focus on real-time applications• written in optimized C++ and takes advantage of multicore processors
• Portability• C/C++ library• runs under Linux, Windows, and Mac OS X• interfaces for Python, Java, MATLAB, and other languages• portable to Android and iOS for mobile applications
• Diversity of usage and applications• provides a simple-to-use computer vision infrastructure that enables building fairly sophisticated vision
applications quickly.• OCV library contains over 500 functions that span many areas in vision, including factory product inspection,
medical imaging, security, user interface, camera calibration, stereo vision, and robotics.
• Functions are well documented, but they lack in example diversity.
• OpenCV is not great if you just want to manipulate images.
• OpenCV is great for teaching the computer how to see something!
According to one well-known story, in 1966, Marvin Minsky at MIT asked his undergraduate student Gerald Jay Sussman to “spend the summer linking a camera to a computer and getting the computer to describe what it saw” – That is how computer/machine vision started!
OpenCV Overview:
General Image Processing Functions
Machine
Learning:•Detection,
•Recognition
Segmentation
Tracking
Matrix Math
Utilities and
Data Structures
Fitting
Image Pyramids
Camera
Calibration,
Stereo, 3D
Transforms
Features
Geometric
Descriptors
Robot support
opencv.willowgarage.com
> 500 functions
7
Books - https://opencv.org/books.html
tnt.etf.rs/~mv/book1
tnt.etf.rs/~mv/book2
OpenCV versions
• Current stable versions• 2.4.13 – still active since 3.x OpenCV is not compatible with 2.x
• 3.3.1 – introduces DNN and advanced Machine Learning algorithms, C++11, IPP support updated
• Newest• 3.4.1 – Additional improvement for DNNs, Matlab interface
• We chose OpenCV 3.2 since it supports all of our needs, but feel free to install newer versions
OpenCV Block Diagram
• OpenCV is built in layers. At the top is the OS under which OpenCV operates.
• Next comes the language bindings and sample applications.
• Below that is the contributed code in opencv_contrib, which contains mostly higher-level functionality.
• After that is the core of OpenCV, and at the bottom are the various hardware optimizations in the hardware acceleration layer (HAL).
OpenCV contrib
• Face Recognition
• Object Tracking
• DNN – Deep Neural Networks – Using Caffe framework models
• SFM – Structure From Motion
• Structured Light
• Multi-camera Calibration
• Stereo Matching
• ...
C++ vs. Python
• Python is slower compared to C++ or C. Python is built for its simplicity, portability and moreover, creativity where users need to worry only about their algorithm, not programming troubles.
• Python-OpenCV is just a wrapper around the original C/C++ code. It is normally used for combining best features of both the languages, Performance of C/C++ & Simplicity of Python.
• So when you call a function in OpenCV from Python, what actually run is underlying C/C++ source.
• Performance penalty is <4%
• Python 3 has some difficulties running in Windows
Documentation
• https://docs.opencv.org/3.2.0/
• tutorials• introduction: installation and hello world• core module: matrix manipulation, file I/O, random generator…• imgproc: smoothing, sharpening, edge detection, common image processing• highgui: manipulate windows displays and add trackbars and other UI• …
• cheatsheet (OCV 2.4!) http://docs.opencv.org/3.0-last-rst/opencv_cheatsheet.pdf
• High level overview of the library
• OpenCV 3.x Modules ppt
Installing OpenCV 3.2
• Using prebuilt OpenCV binaries [Win]• tutorial from docs.opencv.org - it is made for v2.4 but works for all newer• libraries “as is”, if we want some additional function we need to rebuild sources
• NuGet• nuget.org• package manager for the Microsoft development platform• works only for VS14 and older• very simple and fast OpenCV installation, but not flexible • recommendation: install opencv3.1 package
• Building OpenCV from source• same tutorial from above• rather complicated on Windows, Linux provides much easier interface
Installing OpenCV 3.2
• Using prebuilt OpenCV binaries [Win]• Prebuilt binaries are located at C:\opencv\ and environment variable is set
• If you want to download them go to Sourceforge page -> 3.2.0 -> -vc14.exe
• run vc14.exe and place opencv in C:\
• set the environment variable: OPENCV_DIR C:\opencv\build\x64\vc14
• add bin folder to system path: %OPENCV_DIR%\bin
• lib – static libraries
• dll – dynamic link libraries – loaded only on demand during runtime
Building OpenCV apps in Visual Studio
• To build an application with OpenCV you need to do two things:• Tell the compiler how the OpenCV library looks. You do this by showing it the
header files.
• Tell the linker where to find the functions or data structures of OpenCV, when they are needed.
• VS -> New Project - > Win32 Console Application -> give name -> OK -> Next -> Empty Project -> Finish
• View -> Other Windows -> Property Manager
• open Property Manager view and select Debug|x64 -> Add New Project Property Sheet
Building OpenCV apps in Visual Studio
• created Property Sheet -> Properties
• OpenCV include directory (*.h and *.hpp files)• C++ -> General -> Additional Include Directories = $(OPENCV_DIR)\..\..\include
(make sure to press Enter and Apply)
• Libraries – we want to link them• static (*.lib)
• Linker -> General -> Additional Library Directories -> $(OPENCV_DIR)\lib• Linker -> Input -> Additional Dependencies -> we need to write down all debug libraries
(opencv_world320d) (when creating property page for Release x64, then release libraries -> opencv_world320).
• dynamic (*.dll)• program will look for the .dll’s during runtime and if we have correctly added their path to
path environment variable it should successfully find them
Display a picture! -> read_image.cpp
• Add read_image.cpp to your source files
#include <opencv2/highgui.hpp> //image I/O, window display and many other functions
int main(int argc, char** argv){
// init image and window variablescv::Mat image;cv::namedWindow("figure", cv::WINDOW_AUTOSIZE);
// YOUR CODE COMES HERE// read the image and display it
cv::destroyWindow("figure");return 0;
}
SW
HighGUI
• Image I/O (cv:: imread, cv:: imwrite)
• Rendering (cv:: namedWindow, cv:: imshow, cv::destroyWindow)
• Processing keyboard and other events, timeouts (cv::waitKey(n);)
• Trackbars
• Mouse callbacks
• Video I/O (cv::VideoCapture )
Display a picture!
#include <opencv2/highgui.hpp> //image I/O, window display and many other functions
int main(int argc, char** argv) {// init image and window variablescv::Mat image;cv::namedWindow("figure", cv::WINDOW_AUTOSIZE);
// YOUR CODE COMES HERE// read the image and display itimage = cv::imread("../../lena.jpg", cv::IMREAD_UNCHANGED); // image is read “as is”, color or grayscaleif (image.empty()) return -1; // always check if the image was successfully read!!!
cv::imshow("figure", image); // show image; the window is selected through is unique namecv::waitKey(0); // to keep the window open we use waitKey(ms delay), 0 is forever
cv::destroyWindow("figure"); // after any button is pressed close the window and finishreturn 0;
}
HighGUI: OpenCV functions
• void cv::namedWindow(const string& winname, int flags=WINDOW_AUTOSIZE);• Creates window accessed by its name. Window handles repaint, resize events.
Its position is remembered in registry.
• void cv::destroyWindow(const string& winname);
• void cv::imshow(const string& winname, cv::Mat& mat);• Copies the image to window buffer, then repaints it when necessary.
{8u|16s|32s|32f}{C1|3|4} are supported.
• Only the whole window contents can be modified. Dynamic updates of parts of the window are done using operations on images, drawing functions etc.
HighGUI: OpenCV functions
• Mat imread(const string& filename, int flags=1);• loads image from file, converts to color or grayscale, if need, and returns it (or
returns empty cv::Mat()).
• image format is determined by the file contents.
• bool imwrite(const string& filename, Mat& image);• saves image to file, image format is determined from extension.
• Example: convert JPEG to PNG• cv::Mat img = cv::imread(“picture.jpeg”);
• if(!img.empty()) cv::imwrite( “picture.png”, img );
cv::Mat
• considered to be the epicenter of the C++ implementation of OpenCV
• majority of functions in the OpenCV library• are members of the cv::Mat class
• take a cv::Mat as an argument, or
• return cv::Mat as a return value
• images are specialized uses of the cv::Mat class, but such specific use does not require a different class or type.
• Most of the used types have their own Mat constructor. -> open docs
cv::Mat
• dims - number of dimensions
• rows & cols - number of rows and columns. If dims>2, (rows, cols) =(-1,-1)
• size - size(cols, rows)
• *data - pointer to the data
• step - defines data layout of the matrix (regarded as an array)
• cv::Mat behaves very much like a smart pointer for the data contained in data. The address of an element whose indices are (for 2D array) i and j:
&(mat_i,j) = mat.data + mat.step[0]*i + mat.step[1]*j
• This means that 2-dimensional matrices are stored row-by-row, 3-dimensional matrices are stored plane-by-plane, and so on.
• Each element of the data in a cv::Mat can itself be either a single number or multiple numbers.
Creating array/matrix
• cv::Mat mymatrix; //matrix(array) mymatrix has no size and no data type.
• mymatrix.create(vector sizes, type); // later allocation and type definition
• mymatrix.create(rows, cols, type); // later allocation and type definition
• mymatrix.create(cv::Size sizes, type); // later allocation and type definition
• Data type = fundamental type of element + channels• CV_{8U,16S,16U,32S,32F,64F}C{1,2,3}
• For example, CV_32FC3 would imply a 32-bit floating-point three-channel array.
THE MOST IMPORTANT PARAGRAPH IN THE BOOK
• It is critical to understand that the data in an array is not attached rigidly to the array object.
• The cv::Mat object is really a header for a data area, which — in principle — is an entirely separate thing.
• For example, it is possible to assign one matrix n to another matrix m (i.e., m=n). In this case several things happen:• the data pointer inside of m will be changed to point to the same data as n. • the data pointed to previously by the data element of m (if any) will be deallocated. • the reference counter for the data area that they both now share will be incremented.• the members of m that characterize its data (such as rows, cols, and flags) will be updated to
accurately describe the data now pointed to by data in m.
• This all results in a very convenient behavior, in which arrays can be assigned to one another, and the work necessary to do this takes place automatically behind the scenes to give the correct result.
Images (opencv/imgproc.hpp)
• Images in OpenCV are stored in a Mat object
• Consists of a matrix header and a pointer to the matrix containing the pixel values
• Header contains information such as the size of the matrix, the number of color channels in the image, etc.
• Access this information through functions:• Mat.rows() : Returns the number of rows
• Mat.cols() : Returns the number of columns
• Mat.channels() : Returns the number of channels
Image Types
• The type of a Mat object specifies how to interpret the underlying binary data
• Represented as CV_<Datatype>C<#Channels>
• Examples: CV_8UC3 represents an image with three channels each represented by an 8-bit unsigned integer• Mat.type() : Returns an identifier that specifies the underlying
arrangement of the data (enum type, 0 is CV_8UC1)
• Mat.depth() : Returns the number of bytes for a matrix element which depends on the datatype
Pixel Types
• Also need to know the underlying ordering of the channels
• OpenCV defaults to BGR
• Can also handle other types of channels/ordering such ad RGB, HSV, GRAY, YCbCr …
• Can access the value for a particular channel at a specific pixel once we have know how to interpret the underlying data• Mat.at<datatype>(row, col)[channel]: Returns a pointer to the
image data at the specified location
• Convert between color spaces using• cv::cvtColor(foo, bar, CV_BGR2GRAY)
Copying Images
• In order to avoid making unnecessary copies of images, OpenCV uses a reference counting system to manage the underlying matrix data and performs shallow copies by default
• The copy constructor and assignment operator only copy the headers and the pointer to the underlying data
• Use Mat::clone() and Mat::copyTo() to perform a deep copy• Mat::clone() – clones the image as is
• Mat::copyTo() – accepts additional argument mask which enables selective copying
Display grayscale version of the image
• cv::cvtColor(src, dst, code)• cvtColor() is not a part of highgui -> look for appropriate module and
include it
• which color systems are supported?
• in OpenCV input images are in BGR color space!!!
SW
Simple transformation
• Perform image blurring with Gaussian filter 5x5, sigma = 5.
• cv::GaussianBlur(src, dst, ksize, sigmaX, sigmaY=0.0, borderType = cv::BORDER_DEFAULT)
• Sobel edge detection on a grayscale image -> Save image (imwrite)• where are the edges?
• what’s the data type?
• how does imshow()work?
SW
Documentation
• https://docs.opencv.org/3.2.0/
• tutorials• introduction: installation and hello world• core module: matrix manipulation, file I/O, random generator…• imgproc: smoothing, sharpening, edge detection, common image processing• highgui: manipulate windows displays and add trackbars and other UI• …
• cheatsheet (OCV 2.4!) http://docs.opencv.org/3.0-last-rst/opencv_cheatsheet.pdf
• High level overview of the library
• OpenCV 3.x Modules ppt
Input from Camera -> cv::VideoCapture
• We need a cv::VideoCapture object.
• Depending on the source, we use one of three calls to create a cv::VideoCapture object: • cv::VideoCapture::VideoCapture(const string& filename); // video filename• cv::VideoCapture::VideoCapture(int device); // camera (device) ID• cv::VideoCapture::VideoCapture(); // default constructor
• If the open is successful and we are able to start reading frames, the member function cv::VideoCapture::isOpened() will return true.
• The returned value of cv::VideoCapture::isOpened() will be false if for some reason the file could not be opened (e.g., if the file does not exist), but that is not the only possible cause. The constructed object will also not be ready to be used if the codec with which the video is compressed is not known. Because of the many issues surrounding codecs (legal as well as technical), this is not as rare of an occurrence as one might hope.
Reading the camera
• cv::VideoCapture::read(cv::OutputArray image);
• cv::VideoCapture::operator>>(cv::Mat& image);
• cv::VideoCapture::grab(void) -> copies raw data from camera quickly to computer, and then when we want to use it, we must decode it• cv::VideoCapture::retrieve(cv::OutputArray image, int channel=0)
• cv::VideoCapture::read() preforms grabbing and retrieving together
• Where is grab-retrieve useful?• The most common situation arises when there are multiple cameras (e.g., with stereo
imaging). In this case, it is important to have frames that are separated in time by the minimum amount possible (ideally they would be simultaneous for stereo imaging). Therefore, it makes the most sense to first grab all the frames and then come back and decode them after you have them all safely in your buffers.
Video metadata
• cv::VideoCapture::get() and cv::VideoCapture::set()
• Video files contain not only the video frames themselves, but also important metadata, which can be essential for handling the files correctly. When a video file is opened, that information is copied into the cv::VideoCaptureobject’s internal data area. It is very common to want to read that information from the cv::VideoCapture object, and sometimes also useful to write to that data area ourselves.
• Some metadata:• video duration• frame size• bit rate mode• color space• chroma subsampling• …
Input from a Camera
• Open file read_camera.cpp
• Function expects camera filename as input from command line, but…
• If there isn’t one, it tries to access a camera with ID 0.
• Run the code, see what happens!
• Instead of simple streaming, show edges!
SW
Back to Sobel
• Perform Sobel edge detection step by step by accessing each individual pixel and its neighborhood.
• S = [-1, -2, -1; 0, 0, 0;1, 2, 1]; //horizontal edges
SW
Machine vision – what else do we need?
• points
• lines
• arrays
• feature vectors
• …
Getting to know OpenCV data types
• OpenCV has many data types, which are designed to make the representation and handling of important computer vision concepts relatively easy and intuitive.
• At the same time, many algorithm developers require a set of relatively powerful primitives that can be generalized or extended for their particular needs.
• This library attempts to address both of these needs through the use of templates for fundamental data types, and specializations of those templates that make everyday operations easier.
OpenCV data types
1. Basic data types - are those that are assembled directly from C++ primitives (int, float, etc.). These types include simple vectors and matrices, as well as representations of simple geometric concepts like points, rectangles, sizes, and the like. cv::Matx
2. Helper objects - These objects represent more abstract concepts such as the range objects used for slicing, e.g. cv::Range
3. Large array types - These are objects whose fundamental purpose is to contain arrays or other assemblies of primitives or, more often, the basic data types. The star example of this category is the cv::Mat class, which is used to represent arbitrary-dimensional arrays containing arbitrary basic elements.
In addition to these types, OpenCV also makes heavy use of the Standard Template Library (STL). OpenCV particularly relies on the vector class, and many OpenCV library functions now have vector template objects in their argument lists.
cv::Matx – fixed matrix class
• matrix dimension known at compile time
• all memory for their data is allocated on the stack, which means that they allocate and clean up quickly
• Operations on them are fast, and there are specially optimizedimplementations for small matrices (2 × 2, 3 × 3, etc.)
cv::Vec<> - also fixed
• Used through aliases (typedefs) for common instantiations of the cv::Vec<> template.
• They have names like cv::Vec2i, cv::Vec3i, and cv::Vec4d (for a two-element integer vector, a three element integer vector, or a four-element double-precision floating-point vector, respectively).
• In general, anything of the form cv::Vec{2,3,4,6}{b,w,s,i,f,d} is valid for any combination of two to four dimensions and the six data types.
• Can be accessed by a vector index -> myvec[0], myvec[1],…• In the proper sense of C++ inheritance, it is correct to say that the fixed
vector template cv::Vec<> is a cv::Matx<> whose number of columns is one.
cv::Point<> - also fixed
• The point classes, which are containers for two or three values of one of the primitive types.
• cv::Point2i, cv::Point2f
• members are accessed by named variables• mypoint.x
• mypoint.y
• Similar to cv::Point<> is cv::Size which allows storing 2D object size and also has two members: width and height.
cv::Range – helper object
• two elements: start and end
• Ranges are inclusive of their start value, but not inclusive of their end value
• cv::Range(int start, int end) -> [start, end)
How to explicitly initialize the matrix• Use C/C++ arrays and initialize via constructor
• MATLAB style initializers: zeros(),ones(),eye().Specify the size and data type to use:
• For small matrices you may use comma separated initializers:
Accessing Array Elements Individually
• By location -> member function at<data_type>(row,col)• The way this function works is that you specialize the at<>() template to
the type of data that the matrix contains, then access that element using the row and column locations of the data you want.
• cv::DataType<>
Accessing Array Elements by Block
• member functions of the cv::Mat class • row()
• col()
• m(cv::Range(i0,i1),cv::Range(j0,j1)); //Array corresponding to the subrectangle of matrix m with one corner at i0,j0 and the opposite corner at (i1-1,j1-1) (i indexes rows, and j columns)
• m(cv::Rect(j0,i0,w,h)); //Array corresponding to the subrectangle of matrix m with one corner at i0,j0 and the opposite corner at (i0+h-1,j0+w-1)
Algebra and cv::Mat
• Operations available for matrix expressions• m0 + m1, m0 – m1;
• m0 + s; m0 – s; s + m0, s – m1;
• -m0;
• s * m0; m0 * s;
• m0.mul( m1 ); m0/m1; // per element
• m0 * m1; // matrix multiplication
• m0>m1; m0>=m1; m0==m1; // per element comparison
• m0&m1; m0|m1; m0^m1; ~m0;m0&s; s&m0; m0|s; s|m0; m0^s; s^m0 ; //bitwise logical operations
More Things an Array Can Do
• m1 = m0.clone();
• m0.copyTo( m1 );
• m0.copyTo( m1, mask ); //only entries indicated in the array mask are copied
• m0.convertTo(m1, type, scale, offset); // Convert elements of m0 to type (e.g., CV_32F) and write to m1 after scaling by scale (default 1.0) and adding offset (default 0.0)
• m0.reshape(chan, rows);
• m0.empty();
Back to Sobel
• Perform Sobel edge detection step by step by accessing each individual pixel and its neighborhood.
• S = [-1, -2, -1; 0, 0, 0;1, 2, 1]; //horizontal edges
SW
Image processing all over again - Dices
• Detect number of dots seen on the top of the dices• images are in /dices folder
• open dices.cpp
• solve the problem!
Final goal – act like cats
1. Detect red circle in the image
2. Extract it from the background
3. extract its center position
4. Display the path of the circle through the given image set
Images and template code given:• track_red_circle.cpp
• \fotke
Hints and advices
• Use only one image for start!• decide upon color space you want to use (RGB, Lab, HSV)• debugging is not so simple, show your image between each image
processing step, use multiple windows• want to extract a range of values -> use cv::inRange(src, lowerb, upperb, dst)
• Image is of bad quality? Maybe use some blurring to make objects uniform?
• Object has holes or not-so-round shape? Morphological operations!• I detect several objects? bwlabel (connectedComponents)• Thresholding is boring? Try using Hough circle transform!
Time performance of accessing single elements• DEBUG RELEASE
Reference
[1] http://docs.opencv.org/master/– OpenCV 3.2.0-dev Documentation
[2] Adrian Kaehler and Gary Bradski. 2016. Learning OpenCV 3: Computer Vision in C++ with the OpenCV Library (1st ed.). O'Reilly Media, Inc..
[2a] Book examples: https://github.com/oreillymedia/Learning-OpenCV-3_examples
[3] Kenneth Dawson-Howe. 2014. A Practical Introduction to Computer Vision with OpenCV (1st ed.). Wiley Publishing.
