Embed Size (px)
Learn Embedded Linux by Building a Video Surveillance Device
Timesys University
1
2 Timesys Staff
Answering your questions today are:
Maciej HalaszProduct [email protected] 325 6313
Theresa [email protected] 325 6362
Al FeczkoTechnical Pre-Sales/[email protected] 325 6390
2
3 We Are Building a … Video Surveillance Device!
More than one third of all shops are under video surveillance
3
4 Building a Video Surveillance Devicewith LinuxLink — Agenda
Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development
environment with the matching SDK
Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using
Qt Embedded for Linux
Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording
under Linux
Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot
and quick deployment
4
5 Module 1
Your Development Environment Embedded Linux Primer Where do I start my Linux design — Product
Requirements Assemble a custom BSP using the
LinuxLink FREE Edition Web Interface• Reflect product requirements• Produce complete Linux design• Learn how to find information
Application/System Development environment setup with an SDK
Deploy the images on the Phytec Vybrid system
5
6
Your Development Environment:
Session Hardware/Software Requirements
6
7 What We Need to Build Our Product
A Host Machine• Linux is the best – any recent version is ok• Windows is fine, but you’ll need a VirtualBox with a Linux
installation
Cross-development environment
Linux source code for the product• Bootloader • Linux kernel• APIs
Various Linux utilities e.g. mkimage
IDE with WYSIWYG for faster development
A Development Kit
7
8 Your Workstation Setup
Development Environment
• Cross tools• IDE
Exercises
USB
Ethernet/Serial
8
9 Having Fun Yet?
Question bucket
• We will ask some review questions at the end of each Module.
• In return for the first correct answer for each question asked, we will award a giveaway item.
9
10
Embedded Linux Primer
10
11
Repository Docs Support
Mul
ti O
S C
onfi
gura
tion
& B
uild
Too
l
DS
5 —
Dev
elop
men
t &
Deb
ug T
ools
Multi OS Infrastructure
Bootloader
Tower Modules
Cortex A5 Cortex M4
App on MQX
Linux Kernel
App on Linux
Message passing
MQX
App on Linux
JTA
G
Sup
port
Complete Software Solution for Vybrid
Qt Application PEG Application
Device Drivers
Device Drivers
Demos
11
12
U-Boot bootloader
Embedded Linux Reference Model
Middleware
shell
Vybrid
USB
TouchScreenLCD
Linux kernel
Security Surveillance
CameraUser Interface, Buttons, Stream sel, etc
SDIOXIP
Button
Network control interfaceAuto-launch
Qt Embedded
Driver Driver
networking
Wireless tools
openssh
Driver DriverShared MemoryDriver
sysfs
SplashScreen
setup scripts
Ethernet
Driver Driver
Serial
Sensor Control
LCD calibration
Memory
setup script MCC
API
1
2
3
12
13
Bootloader
MQX Reference Model
FunctionalBlocks
Vybrid
USB
TouchScreenLCD
MQX kernel
Application (s)
SDIOXIP
Button
Network control interface
Driver Driver
networking
Driver DriverShared MemoryDriver
File system
Services
setup scripts
Ethernet
Driver Driver
Serial
Sensor Control
Memory
setup script MCC
API
1
2
13
14 Linux and MQX strengthsLinux MQX
Core footprint 1.5M <100k
Scheduler Priority, preemptive, latency not guaranteed
Priority, preemptive, latency guaranteed
Kernel Configurability Modular, fully customizable
Modular, fully customizable
APIs 40,000+ Limited
System protection Separate address spaces Kernel/Application separated
Single address space; no protectionKernel/Application
14
15Vybrid Solution Approach: LinuxLink – Tower – MQX
Linuxlink cloud development allows customers to rapidly deploy pre-built or custom BSP/SDKs for Tower or other hardware platforms.
15
16
Development Ready — When?
Linux kernel
1000s of Linux
packages
Custom software
gstreamer.com
Toolchain
Private plugins
GStreamerVer. 0.22?
Open Source plugins
Ver.3.10?
Additional drivers?
GCC4.1?
uClibc or glibc?
Debug APIs/libs
optimizations
Embedded? Version?
Which modules?
Need HW acceleration?
Third party
mainline
Embedded Linux Challenges
For 2010, 57% of all projects finished late or were cancelledSource: EE Times Embedded Market Study, 2010
16
17 Use Timesys If You…
Have an aggressive schedule. Have an understaffed engineering team. Need proven middleware, configured for your product. Need help from a real person when you’re stuck.
Expert Support
Development Ready
drivers
Dependencies
Linux kernel
Open Source plugins
Toolchain1000s of packages
GCC4.1? uClibc or glibc?
Debug
APIs/libs
Optimizations
Third partyversions
Your custom embedded Linux platform
Professional Services (optional)
Highly customizable embedded distribution
17
18 RTOS Challenges
Future Proofing new/future protocols (Stacks) • Bluetooth Light
Peripheral Support across product families Prototyping and Evaluation support Creating drivers to use with a scheduler Would like to spend your time developing application code and not
creating or maintaining a scheduling system Multi-thread support with synchronization Portability of application code to other CPUs Resource handling Adding new features without affecting higher priority functions Support for upper layer protocols such as:
• TCP/IP, USB, flash systems, web servers• CAN protocols, embedded GUI, SSL, SNMP
Determinism
18
19 MQX Solutions
Established community of users and code examples• MQX is a 15-year-old RTOS
Solid RTOS with extensive code examples and community support
Standardized Driver creation Support across all Freescale MCU Portfolio Extensive Stack Library Extensive third-party community development
support End User empowered to create custom BSP Non Royalty RTOS Ported Tower Systems for easy system lever
evaluation
19
20
Product Requirements — Where Do I Start My Linux Development?
20
21 Requirements – First Glance
GUI with the following features:
• Live feed screen• Recorded feed management screen• Control panel screen
Capture video feed from USB connected cameras
• Store on a local file system
Manage recordings
Utilize a video overlay• See video feed and controls on a
single screen• Display text on a video feed
Detect movement via light sensor
Use Multi-core communication (MCC) layer
• Trigger video recording
Video Surveillance Features:
21
22
U-Boot bootloader
Security Surveillance Device (Blueprint)
Middleware
shell
Vybrid
USB
TouchscreenLCD
Linux kernel
Security Surveillance DeviceVideo
playbackUser Interface, Buttons, Stream sel, etc
SDIONAND
VPU
Network control
interfaceAuto-launch
Qt Embedded
Driver Driver
networking
Wireless tools
openssh
DriverDriverDriver
GStreamer
SplashScreen
setup scripts
Ethernet
Driver Driver
Serial
Camera Capture
LCD calibration
IPU
setup script
Codecs
Shared Memory
Light Sensor
Memory
MCC API
22
23
Exercise 1:
Assemble a Custom BSP Using the LinuxLink FREE Edition Web Interface
23
24 LinuxLink Architecture
24
25 Exercise Details
Identify the initial set of requirements:• All needed software to simply boot a reference board• Build a BSP for development
Build a custom BSP and a matching SDK• Use LinuxLink FREE Edition Web Interface
Start with a small Linux design, and build it up per your requirements • Provides you with a small embedded design
with only the features you need
25
26 Requirements Helper
Feature What I need Where to selectLinux kernel Latest, 3.x kernel Linux kernelCross-Toolchain GCC, Binutils, GDB ToolchainsC Library glibc ToolchainsGStreamer gst-plugins-good PackagesFont freetype PackagesQt qt-embedded-linux PackagesNetworking openssh PackagesSystem initialization busybox Packages
26
27 Account Login info
We have accounts setup specifically for Timesys University• https://linuxlink.timesys.com • Login with the following credentials:
U: labuserX where X is 1-20P: lab123
You can register for a free personal account• http://www.timesys.com/register
27
28
Application/System Development Environment Setup with an SDK
28
29 Embedded Linux Reference Model
Middleware and Application Services
UI NetworkingNetworking
Security Reliability
LCD HW accel.
SD USB more
Linux kernel Driver Driver Driver Driver more
Application 1 Application 2 Third-Party Application
Application (e.g. Base Station Control) App Features
SOCBoard
Target Host
Hundreds of APIs from
Open Source and
Proprietary Reliability
Development Tools• Cross-Development Environment
(SDK)• IDE
• Application Debugging• Build System (e.g. Factory)
• RFS, Kernel, App• Collaborative Development• Profiling• Tracing• System Management
• Target Images• Binaries Deployment• Target Management
App
licat
ion
Sys
tem
Har
dwar
eS
oftw
are
Virtualization
• KGDB/ printk Debugging• JTAG• SMP support, RT
enhancements
29
30 Embedded Linux Reference Model
Middleware and Application Services
UI NetworkingNetworking
Security Reliability
LCD HW accel.
SD USB more
Linux kernel Driver Driver Driver Driver more
Application 1 Application 2 Third-Party Application
Application (e.g. Base Station Control) App Features
SOCBoard
Target
Hundreds of APIs from
Open Source and
Proprietary Reliability
App
licat
ion
Sys
tem
Har
dwar
eS
oftw
are
Virtualization
Host
Development Tools• Cross-Development Environment (SDK)• IDE
• Application Debugging• Build System (e.g. Factory)
• RFS, Kernel, App• Collaborative Development• Profiling• Tracing• System Management
• Target Images• Binaries Deployment• Target Management
• KGDB/ printk Debugging• JTAG• SMP support, RT enhancements
30
31 Development Environment Setup
Can find instructions on how to setup the board
BSP/SDK Summary
Software Development Kit (SDK)• Install on your host PC
Desktop Factory tools• Install on your host PC
31
32
Exercise 2:
Setup Your Application Development Environment
32
33 Exercise Details
Setup your own development environment• Cross-toolchain • IDE• Libraries and header files
The goal• Easily develop your value-add software
33
33 Exercise Details
Setup your own development environment• Cross-toolchain • IDE• Libraries and header files
The goal• Easily develop your value-add software
Answer:
33
33 Exercise Details
Setup your own development environment• Cross-toolchain • IDE• Libraries and header files
The goal• Easily develop your value-add software
+
Answer:
33
33 Exercise Details
Setup your own development environment• Cross-toolchain • IDE• Libraries and header files
The goal• Easily develop your value-add software
+
Answer:
TimeStorm IDE
33
34 Exercise Details
Install an SDK from a build output
Installation command (optional):• sh <SDK name.sh>
For your convenience the BSP/SDK has been downloaded for you and preinstalled under:• /home/tsu-lab/LAB-5/BSP
Look at a directory structure
Verify toolchain with command:• <fully qualified path to toolchain’s gcc> -###
34
35
Deploying Linux Using NFS
35
36 Is NFS useful?
Run the custom BSP on the target• U-Boot bootloader • Linux kernel• Network mounted RFS
The goal• Easily modify all aspects of your RFS• Easily deploy and test your application
36
37 Network File System setup
Host Target
• NFS server• DHCP server (local subnet)• TFTP server (/tftpboot)• Console
• U-Boot• Console
Host RFS
Target RFS Target RFS
• Ethernet• Serial
37
38 NFS Setup Details
Services already preset for you• NFS server• DHCP server (local subnet)• TFTP server (/tftpboot)• Timesys offers getting started guides to walk you through the process
Setup U-Boot environment to load/run a Linux kernel> setenv bootfile uImage //image to load> setenv ipaddr 10.0.0.10 //your local IP> setenv serverip 10.0.0.1 //server IP> saveenv //store variables in NAND> Setenv rootpath <rfs full path> //specify path to nfs exported rfs> run nfs boot //boot the image using pre-set command
Full directions can be found in the Getting Started Guide for the Phytec Vybrid board. Access it through a LinuxLink dashboard.
38
39 Module 1 — Summary
Learned about the Vybrid LinuxLink – needed for all exercises
Reflected initial product requirements in Linux BSP and SDK
Built a custom BSP with LinuxLink FREE Edition• Experiment with a pre-built starting point
Setup a development environment• System level development and optimizations• Development of a value-add software (applications)
Discussed NFS deployment• Image transfer• Bootloader configurations
39
40 Module 1 — Quiz
1. What do you need to know before you start your initial design?
2. Why would you be interested in Linux kernel updates?
3. What are the benefits of using LinuxLink?(name at least 1)
4. How soon can you start your application development?
40
41 Building a Video Surveillance Devicewith LinuxLink — Agenda
Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development
environment with the matching SDK
Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using
Qt Embedded for Linux
Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording
under Linux
Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot
and quick deployment
41
42 Module 2
Modify the underlying Linux image to support Qt
Qt development tools installation/setup
Coffee Vending Machine – GUI development• Create a QMainWindow application• Use a number of Qt widgets including
• QPushButton• QLabel• Layouts and more
• Add a custom widget for the clock • Create a resource file• Test your application locally• Cross-compile for the Vybrid Phytec target• Debug on the target
42
43
U-Boot bootloader
Security Surveillance Device (Blueprint)
Middleware
shell
Vybrid
USB
TouchscreenLCD
Linux kernel
Security Surveillance DeviceVideo
playbackUser Interface, Buttons, Stream sel, etc
SDIONAND
VPU
Network control
interfaceAuto-launch
Qt Embedded
Driver Driver
networking
Wireless tools
openssh
DriverDriverDriver
GStreamer
SplashScreen
setup scripts
Ethernet
Driver Driver
Serial
Camera Capture
LCD calibration
IPU
setup script
Freescale Codecs
Shared Memory
Light Sensor
Memory
MCC API
43
44
Adding APIs/Libraries Using the LinuxLink PRO Desktop Interface
44
45 LinuxLink Value Proposition
Advanced Customization (all aspects of your Linux platform)
‘Factory’ – Only easy to use, innovative Build System with: • Advice engine — recommends software based on your hardware
and application requirements• Update engine — automatic updates for your custom configuration
‘Build up’ vs. strip downto small distribution
Affordable – costs ~40% less
Build RepeatabilityEasy Extensibility
45
Desktop Factory
46
©2013 Timesys Corp.
47 Factory Architecture
The Desktop Factory is the Timesys build engine with the menuconfig UI.
The UI creates/edits the Factory “work-order” in a .config file.
The Factory fetches source code for components required by the work order, and follows recipes to cross build those components.
Each component has its own recipe, embody in a pair of files which drive the UI and the “make” build.
47
©2013 Timesys Corp.
48 Exercise 3: Basic UI
Basic navigation, Searching, Help, String entry Source code locations Running parallel make jobs Package options Templates A peek at the .config file
48
©2013 Timesys Corp.
49 Factory: High-Level FlowAt a high level, the Factory flows through these five stages. However, due to dependencies that require components to be built “out of order,” the Factory operates in a recursive manner.
1. All sources, patches, and other files (e.g., startup scripts) are fetched from the designated source (typically Timesys Repository).
2. The host tools are built including ccache, and the gcc cross compiler suite (glibc or uclibc, binutils, etc.). Any additional host utilities are built as well. Simply stated, everything that will execute on the x86 host are built here.
3. All target software is built. The kernel image and RFS image are created here as well.
4. As requested (e.g., RPM, deb), binaries are packaged.
5. Installers (e.g. SDK) are created.
1. Fetch Sources, Patches, Additional files
2. Build Host Tools and Libraries (toolchain)
3. Build Target Binaries (Root File System)
4. Package binaries
5. Create Installers
49
©2013 Timesys Corp.
50 Factory: Directory Tree
./bin/ - contains scripts useful for extending Factory. ./doc/- contains license and version information. ./include/ - contains common recipes along with the KConfig library. ./src /– contains the downloaded sources. ./target/ - contains sub-directories containing recipes for the kernel, boot-loaders, and
RFS. ./toolchain/ - contains the recipes for each toolchain component as well as the host
utilities that are built with the toolchain. ./build/ – the ouput directory tree. Makefile includes all other make rule files within the Factory tree Config.in - includes all other Config.in files and sets global and default options. .config - the generated work order used by the make rules.
bin doc include src target toolchain
Makefile
factory
build
Config.in
.config
50
©2013 Timesys Corp.
51 Factory: Build Output Tree
NOTE: “{ }” in names imply architecture or other build specific attribute. toolchain-initial-{ }.tgz - This is a bare toolchain (cross compiler, etc.) toolchain-final-{ }.tgz - Bare toolchain plus all selected package libraries. { }.sh - SDK installer ./images/ - contains kernel, bootloader, and RFS images ./packages/ – contains binary packages in RPM, DEB, IPKG, or TGZ. ./<package name>/ - each package has its own working directory. ./toolchain/ – toolchain working directory ./rfs/ - RFS working directory. This can be modified by hand and then the RFS
images can be recreated with the 'rfs-images' target. NOTE: this directory cannot be directly NFS exported as it does not have valid device files (/dev). The device files are populated during the RFS image creation process.
images packages rfstoolchaintoolchain-initial{ }tgz
Build-{ }
toolchain-final-{ }tgz
{ }.sh
<pkg name>
51
©2013 Timesys Corp.
52 make TargetsMake Targets Functionality
checksystem, checktool-oprofile, checktool-ltp, checktool-mpatrol, checktool-gdb
System requirements and compatibility checks.
advice, showpackages, showoutput Information commands
checkupdates, update, update-noprompt, oldconfig
Updating
Clean, download-clean, distclean, rfs-distclean, packages-distclean, software-distclean, reset-toolchain
Clean commands
-clean, -distclean, -fetch, -unpack, -patch, -configure, -build, -package, -rfs-install, -host-install, -show
Per-application targets (all preceded by application name without version e.g. “busybox-clean)
menuconfig, config “workorder” (.config) editors
52
53
Exercise 4:
Enhance Your Design Locally by Adding Required Packages
53
54 Exercise Details
Find Desktop Factory under: • /home/tsu-lab/LAB-5/factory/factory-<string>
Look at the help screen by running make help
Bring up a configuration screen• make menuconfig
Locate and select missing packages
54
55 Requirements Helper
Feature What I need Where to selectLinux kernel Latest, 3.x kernel Linux kernelCross-Toolchain GCC, Binutils, GDB ToolchainsC Library glibc ToolchainsGStreamer gstreamer Packages
gst-plugins-good PackagesMCC mcc-kmod, libmcc PackagesVideo 4 Linux v4l-utils PackagesFont freefont PackagesText display pango PackagesQt qt-embedded-linux PackagesNetworking openssh PackagesSystem initialization busybox Packages
55
56
Qt Development Tools Installation/Setup
56
57 Qt Development Tools
Options• DS5 with TimeStorm & Qt plugins http://linuxlink.timesys.com
– Qt Creator http://www.qt.nokia.com
Installation• DS5 can be downloaded from ARM• Download TimeStorm and Qt plugins from LinuxLink• Preinstalled in your UBUNTU
Run• Desktop icon for ARM DS5
57
58
Host PC/Linux
IDE and Cross-Environment
IDE
LinuxLink Desktop Interface
RecognizesAutomatically
58
59
Security Surveillance Device –GUI Development
59
60 The Qt Story
The Qt toolkit is a multi-platform C++ GUI toolkit (class library)
Developed by Troll Tech AS, later acquired by Nokia Qt is licensed under dual license:
• LGPL• Commercial (mostly if you plan to extend Qt and sell it)
Around 1997, Qt was chosen as the code basis for the KDE Linux desktop environment
Latest version — Qt 5.0 is available for Windows, Unix, Linux, Embedded Linux, Mac OS X libraries and Symbian OS
Recent additions include:• Qt Quick (programming with QML)• Qt Mobility• Target form factor simulator (cell phones)
60
61 Qt/Embedded (cont’d)
Modular approach
Each additional functionality can be enabled by specifying module to be added• Core classes provide the
baseline• GUI building tools• Can run with or without X• Can be easily ported from
other OSs
61
62 Qt — Flexible and Powerful UI API
Used in many applications in various markets WYSIWYG approach Can take advantage of Freescale hardware:
• OpenGL ES API• Multiple frambuffers
62
63 Widgets
Base class for all UI widgets• QPushButton, QLineEdit, QTabView, …
Properties• width, height, backgroundColor, font, mouseTracking,
backgroundPixmap, etc.
Slots• repaint, show, hide, move, setGeometry, setMainWidget, etc.
Signals• mouseMoveEvent, keyPressEvent, resizeEvent, paintEvent,
enterEvent, leaveEvent, etc.
63
64 Event Handling
Qt approach to IPC: signals and slots• A widget sends out various signals• Object methods can be declared as slots• Compatible signals and slots can be connected or plugged together
(parameter types must match!)
Clear UI/Logic separation • This separation between UI components and program elements
lends itself to component-based programming
64
65 Signals and Slots
http://doc.qt.nokia.com/latest/signalsandslots.html
65
66 Signals and Slots (cont.)
#include <QObject>
class Counter : public QObject { Q_OBJECT
public: Counter() { m_value = 0; } int value() const { return m_value; }
public slots: void setValue(int value);
signals: void valueChanged(int newValue);
private: int m_value;
};
66
67 Events – Triggered by Signals
Signals: emit events• declare as signals• You don't implement them, you send them with the keyword “emit”
– e.g. emit(counterAtZero(100))
Slots: receive and handle events• Normal member functions declared as slots
Connect: must connect signals to slots• QObject::connect( Object 1, SIGNAL(activated(int)), Object 2,
SLOT(handlingMethod(int)) );
67
68 qmake
The qmake utility is typically invoked with the following commands:
• qmake – project• qmake
IDE tools invoke qmake automatically• You can invoke it as well (Project->qmake)
Rules:• Be sure to place code in its own directory.• qmake scans all subdirectories for dependencies. • Looks at the .pro file and generates Makefile structure
68
69 Qt Application Structure
.pro file• Holds information on all files that will be used to build a predefined
executable(s)• Used by qmake to generate Makefiles for your project
.ui file• User Interface file• Can open in a designer view• Has canvas on which you can drag and drop widgets• Its XML format is translated into C file (moc_) at compile time
.qrc file• Holds information on all resources (non C++ code) used• Referenced throughout the project e.g. QPushButton icons
.qss file• Used to define a style sheet for a custom look and feel• Loaded at runtime by your application
69
70
Exercise 5:
Create a GUI application with DS5 &TimeStorm/Qt plugins
70
71 Video Surveillance Application
Start DS5• Desktop icon
Open a Workspace• /home/tsu-lab/DS5-workspace-clean
Create a new Qt GUI Project• UI Type – QMainWindow• Don’t need to select any additional Qt frameworks (initially)
Create the following components• Remove menu bar and status bar• Use layouts• Use a QTabView
• Preset the size of all UIs to selected LCD size (will help you see actual design)
• Design UIs on all three screens
Example can be found in Exercise 5 folder
in downloads.
71
72
Exercise 6:
Develop Logic Behind the GUI
72
73 Exercise Details: Video Surveillance Qt Application
Define/Connect signals (Hide Button)• Add code for Hide Button• Capture the signal in a service class• Implement the service routine
– Set alpha blending to 0
Switch the building layout graphic• Utilize the border-image property of widgets’ style sheet
Compile and run your application on your host
Cross compile and debug your application on the target
73
74 Module 2 — Summary
Used LinuxLink PRO Edition Desktop Factory interface• Added needed packages and rebuilt the runtime BSP and
SDK images
Used an IDE to design/develop Qt based UI
Developed a GUI with the following:• Layers, several Widgets and Labels• Implemented signals and handlers
Cross-compiled for Phytec’s Vybrid target
74
75 Module 2 — Quiz
1. Name at least one graphics package (other than QT)?
2. What tools can you use to develop with Qt?
3. How do you trigger cooperation between objects in Qt?
4. What is the purpose behind using Layers?
75
76 Building a Video Surveillance Devicewith LinuxLink — Agenda
Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development
environment with the matching SDK
Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using
Qt Embedded for Linux
Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording
under Linux
Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot
and quick deployment
76
77 Module 3
GStreamer Primer
MQX overview• Pingpong example
Multi-Core Communication (MCC) Layer• Purpose• Setup• Add MCC communication in Security Surveillance Application
Add appropriate code to the UI application
77
78
U-Boot bootloader
Security Surveillance Device (Blueprint)
Middleware
shell
Vybrid
USB
TouchscreenLCD
Linux kernel
Security Surveillance DeviceVideo
playbackUser Interface, Buttons, Stream sel, etc
SDIONAND
Framebuffer
Network control
interfaceAuto-launch
Qt Embedded
Driver Driver
networking
Wireless tools
openssh
DriverDriverDriver
GStreamer
SplashScreen
setup scripts
Ethernet
Driver Driver
Serial
Camera Capture
LCD calibration
IPU
setup script
Codecs
Shared Memory
Light Sensor
Memory
MCC API
78
79
GStreamer Primer
79
80 GStreamer: Element
Is the most important class of objects in GStreamer. You will usually create a chain of elements linked together and let data flow through this chain of elements.
It has one specific function, which can be the reading of data from a file, decoding of this data or outputting this data to your sound card (or anything else).
By chaining together several elements, you create a pipeline that can do a specific task, for example media playback or capture.
GStreamer ships with a large collection of elements by default, making the development of a large variety of media applications possible.
If needed, you can also write new elements!!!
Examples:• v4l2src• filesrc • oggdemux• alsasink • v4lsink
80
81 GStreamer: Pads
Pads are element's input and output, where you can connect other elements.
They are used to negotiate links and data flow between elements in GStreamer.
A pad can be viewed as a "plug" or "port" on an element where links may be made with other elements, and through which data can flow to or from those elements.
Pads have specific data handling capabilities: a pad can restrict the type of data that flows through it.
Links are only allowed between two pads when the allowed data types of the two pads are compatible.
81
82 GStreamer: Pads
Data types are negotiated between pads using a process called caps negotiation.
For the most part, all data in GStreamer flows one way through a link between elements. Data flows out of one element through one or more source pads, and elements accept incoming data through one or more sink pads.
Source and sink elements have only source and sink pads, respectively.
Dynamic vs. on Request pads• On Request pad examples: multiplexers, aggregators, tee elements.• Aggregators (Mix) are elements that merge the content of several input streams together
into one output stream. • Tee elements are the reverse: they are elements that have one input stream and copy this
stream to each of their output pads, which are created on request. Whenever an application needs another copy of the stream, it can simply request a new output pad from the tee element.
82
83 GStreamer: Pipelines
A pipeline is a special subtype of a bin that allows execution of all of its contained child elements.
As you set a pipeline to PAUSED or PLAYING state, data flow will start and media processing will take place. Once started, pipelines will run in a separate thread until you stop them or the end of the data stream is reached.
83
84 GStreamer: Communication
buffers — objects for passing streaming data between elements in the pipeline. Buffers always travel from sources to sinks (downstream).
events — objects sent between elements or from the application to elements. Events can travel upstream and downstream. Downstream events can be synchronized to the data flow.
messages — objects posted by elements on the pipeline's message bus, where they will be held for collection by the application. Messages can be intercepted synchronously from the streaming thread context of the element posting the message, but are usually handled asynchronously by the application from the application's main thread. Messages are used to transmit information such as errors, tags, state changes, buffering state, redirects etc. from elements to the application in a thread-safe way.
Queries — allow applications to request information such as duration or current playback position from the pipeline. Queries are always answered synchronously. Elements can also use queries to request information from their peer elements (such as the file size or duration). They can be used both ways within a pipeline, but upstream queries are more common.
84
85 GStreamer: Architecture
Divided into packages• Baseline• Plugins • Third-party
Third-party plugins can take advantage of hardware- specific acceleration
• Developed by Freescale • Integrated and available
in LinuxLink
85
86 GStreamer Command Line Tools
Embedded player• gplay
– Creates its own pipeline based on file extension
Execute a pipeline• gst-launch
Learn more about an element capability• gst-inspect
– Provides information on all available gstreamer elements
• Helps you find out specific element capabilities
# gst-inspect | grep enc
# gst-inspect ffdec_mpeg4
86
87 GStreamer Playground
Video playback# gst-launch videotestsrc ! fbdevsink device=/dev/fb2
Webcam live stream# gst-launch v4l2src ! fbdevsink device=/dev/fb2
# gst-launch v4l2src device=/dev/video0 ! fbdevsink device=/dev/fb2
87
88 GStreamer Playground
Webcam capture# gst-launch v4l2src ! fbdevsink device=/dev/fb2# gst-launch v4l2src device=/dev/video0 ! ffenc_mpeg4 ! matroskamux ! filesink location=test.file
Text overlay# gst-launch v4l2src device=/dev/video0 ! textoverlay text=“Hello” ! fbdevsink device=/dev/fb2
Clock overlay# gst-launch v4l2src device=/dev/video0 ! clockoverlay halign=right valign=bottom shaded-background=true time-format="%Y.%m.%D" ! fbdevsink device=/dev/fb2
Playing and recording at the same time# gst-launch v4l2src device=/dev/video0 ! clockoverlay halign=left valign=top time-format="%Y/%m/%d %H:%M:%S" ! tee name="splitter" ! queue ! fbdevsink device=/dev/fb2 sync=false splitter. ! queue ! ffenc_mpeg4 ! matroskamux ! filesink location=recording.file
88
89
Exercise:
GStreamer Pipeline
89
90 Exercise Details: Capture Camera Feed with GStreamer
Verify available codecs• gst-inspect • gst-inspect | grep mux
Play camera feed on HDMI output
Components to be used• gst-launch• v4l2src (camera on /dev/video0)• fbdevsink device=/dev/fb2
90
91 GStreamer Language Bindings
Language StatusPython Released. See the gst-python module.
Perl Released as part of the gtk2-perl project.
.NET The bindings are still under development. See the gstreamer-sharp module.
C++ Released. See the gstreamermm GNOME module.
Qt Released. See the qt-gstreamer module.
Android Under development. See the gstreamer android module.
Guile Released under the aegis of the guile-gnome project.
Haskell See Haskell.org.Java Released as part of the gstreamer-java project.
Ruby The Ruby bindings are released as part of Ruby-GNOME2.
Vala Released as part of Vala.
91
92
MQX & MCC Overview
92
93
Bootloader
MQX Reference Model
FunctionalBlocks
Vybrid
USB
TouchScreenLCD
MQX kernel
Application (s)
SDIOXIP
Button
Network control interface
Driver Driver
networking
Driver DriverShared MemoryDriver
File system
Services
setup scripts
Ethernet
Driver Driver
Serial
Sensor Control
Memory
setup script MCC
API
1
2
93
©2013 Timesys Corp.
94 OS Comparison (for the purpose of training) MQX
• O/S components are libraries linked with your application to create a single bare-metal execution image.
• Source distributed as DS-5 (Eclipse) projects in a single source tree.
• Built with ARM bare-metal compiler.
(advanced)• A single address space – no
memory protection, mapping, or transformations between hardware I/O, O/S components, application.
• Programs are tasks each in their own thread.
Linux• Kernel is a bare-metal executable,
but isn’t enough.• Your application (except possibly device
drivers) is built separately and runs (launched) from a file system.
• Therefore a root file system (RFS) is a required component of Linux.
• Timesys Factory (on web or desktop) builds the RFS. It will build the Kernel and your Application, but they can be also built “outside” as well.
• Built with (Timesys) gcc compiler.
(advanced)• Linux utilizes the memory management
unit (MMU) of the A5 core. All processes are hardware protected from stepping on each other’s memory.
• Programs run as a process – which can have many threads.
94
©2013 Timesys Corp.
95 Overview of MCC Multi-Core Communication Communication layer between MQX and Linux Lightweight, fast
• API calls are simple send / receive between endpoints • Endpoints are [core, node, port] triplets
– Sort of IP host, address, port – but not really.– Core is fixed (A5=0, M4=1]– Linux can have multiple nodes, MQX only 1– Each can have multiple endpoints, arbitrary numbering
• Uses shared SRAM and interrupts• Received data can be passed by pointer or copied• Variable timeouts
Configurable:• Buffer sizes• # of buffers• Max # of endpoints
95
©2013 Timesys Corp.
96 MCC ArchitectureA5 - Linux M4 - MQX
API shared library
Kernel Module Driver
User App (tasks)
API library
Shared SRAM
Endpoint 0,0,1
User App process(es)
Endpoint 0,2,3 Endpoint 1,0,1 Endpoint 1,0,2
Free buffer pool
Interrupts
HardwareSemaphore
96
©2013 Timesys Corp.
97 Simple ExampleA5 - Linux
1. “I will receive on endpoint [0,0,1]”
2. mcc_initialize(0)
3. mcc_create_endpoint([0,0,1], 1)
4. mcc_send([1,0,2], “hello”, 5, 5000) // <= 5 secs for buffer
5. mcc_recv_copy([0,0,1], &buf, sizeof(buf), length, 0xffffffff)
M4- MQX1. “I will receive on endpoint [1,0,2]”
2. mcc_initialize(0)
3. mcc_create_endpoint([1,0,2], 2)
4. mcc_recv_nocopy([1,0,2], &buf_p, length, 0xffffffff)
5. mcc_send([0,0,1], “hello”, 5, 5000) // <= 5 secs for buffer
97
©2013 Timesys Corp.
98 Resource Sharing
X
Y
Z
Photosensor
MCC
A5 – Linux
• boa webserver• Sensor demo• Security
Surveillance App• theater-vybrid
M4 – MQX• pingpong/
Photosensor example
Vybrid SOC
Phytec-LCDTouchscreen
I2C Bus
98
©2013 Timesys Corp.
99 Protecting I2C SequencesMQX Application
1. Open the I2C drivera) Disable and clear I2Cb) Clear out all I2C
eventsc) Set the station
addressd) Set the frequencye) Enable I2C
2. Set master mode3. Set the device
address4. Do read/write
sequence5. Send out stop6. Close the driver.Must be protected in application.
Linux Driver1. Enable I2C controller2. Set master mode,
Interrupts 3. Set the device address4. Wait for interrupt5. Do read/write
sequence including wait for interrupt
6. Send out stop7. Disable I2C controller
Protected in driver. Transparent to application.
Hardware semaphore protection.
99
100 A5/M4 Boot Process
U-Boot• Reset, copy U-Boot to SDRAM and jump to start address
• Basic System Init. (IMPORTANT)
• Copy the Linux kernel Image to SDRAM from SD
• Decompress the kernel if needed
• Jump to upload address and start the kernel
Kernel• Run Kernel Init code
• Init kernel subsystems (device drivers)
• Init SD card
• Mount SD card partition with RFS
• Execute init script
MQX• Load to specific address in memory with mqxboot utility
Application• Depends on your specific requirements
Power On
U-bootHardware Init
U-bootCopies kernel to memory/
uncompress
Kernel BootHardware/Subsystems
initialization
RFSFetch/Mount/Boot into
User Application
Boot Process SequenceTI
ME
MQXLoad to addr & run
100
101
Exercise 8:
MQX & MCC
101
102 Exercise Details: MQX & MCC Components
Open DS5-Workspace-clean-preset1. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mqx/build/ds5/bsp_twrvf65gs10_m42. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mqx/build/ds5/psp_twrvf65gs10_m43. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mcc/build/ds5/mcc_twrvf65gs10_m44. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mcc/examples/pingpong
Import the Linux side of the light sensor application• Sensor demo or• MCCDemo (already in the project)
Start pingpong/MCC/MQX component on M4 core• run startmqx script on the target• location: /home/ Note: Serial Console
will behave strange as it is now shared
between M4 and A5
102
103 Integrate Application Components
Added a QThread based class• Can handle independently MCC traffic
Leverage MCCDemo application
Create a custom Signal-Slot pair to trigger camera feed when desired light threshold is reached• Leverage start_camera() method and onStopButton() slot
Compile and deploy on Vybrid• Place a breakpoint at start_camera method and debug on target
103
104 Module 3 — Quiz
1. How does GStreamer know which codec to use?
2. How can you construct a pipeline?
3. Name one major difference between Linux and an RTOS
4. What are endpoints?
104
105 Building a Video Surveillance Devicewith LinuxLink — Agenda
Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development
environment with the matching SDK
Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using
Qt Embedded for Linux
Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording
under Linux
Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot
and quick deployment
105
106 Module 4
How to measure boot time in our dev environment
Fast boot optimizations• Boot process overview• Bootloader optimizations• Kernel level optimizations• Filesystem optimizations• Other options
106
107
Boot time Measurement
107
108 Measuring the boot time
Each embedded design is different
Instrumentation is available• Helps measure boot time at all stages
• Most are open source based
• Can be easily removed when you are done
The trick behind boot time optimization is to know where your system is spending time.
You may end up losing a lot of time with little results!
108
109 Available Instrumentation
Bootloader• Logic analyzer (using GPIO pins)
Kernel Measurement• Printk
– Compile kernel with: CONFIG_PRINTK_TIMES=y– Switch on dynamically: “echo Y >/sys/module/printk/parameters/time”
• initcall_debug– On boot command line, use: initcall_debug=1
User space measurement• Bootchart (www.bootchart.org)• Strace
– Run strace –cT 2>/tmp/strace.log bluetoothd• Linux Trace Toolkit
System• uptime
– Add (echo -n "uptime:" ; cat /proc/uptime) to an RC script
109
110
Fast boot Optimizations
110
111 Bootloader Optimizations
Low hanging fruit • Set the bootdelay variable to 0 (time savings 4s)• Preset the bootcmd; do not use setenv (time savings 0.5s)• Disable console (time savings 2s)
– CFG_CONSOLE_INFO_QUIET– CONFIG_SILENT_CONSOLE– In our case- silent=yes
• Disable other tests (time savings 2-6s)
Additional modification/enhancements • If possible, use uncompressed Linux kernel• Optimize the NAND read operation, to shorten image copy time• Rewrite/disable CRC32 checksum code• Load the image directly to Entry point
– Set CONFIG_LOADADDR• If NOR Flash is used, leverage XIP• For large kernel image, use different compression algorithms
Boot Time<1s 75s
111
112 Linux Kernel Optimizations
Low hanging fruit (time savings: 2-15+s)• Use uncompressed kernel
– Uncompressing takes time• Remove unused kernel options
– Not used networking i.e. IPV6, multiple file systems– Debug features and symbols (for final deployment)
• Build not initially used device drivers as Loadable Kernel Modules– Keep the features needed at boot time built into the kernel– Remaining drivers built as LKMs will make kernel smaller
• Consider various approaches for your RFS deployment– JFFS2 with appended journal summary (skip flash scan)– CRAMFS, UBIFS
• Suppress the console output– Use “quiet” with your kernel command line
Boot Time<1s 75s
112
113 Making the Linux Kernel Small and Fast
Linux kernel options we will look at todayKernel Option CommentCONFIG_EMBEDDED Disables or tweaks a number of kernel options and
settings. Think uClinux
CONFIG_IKCONFIG Saves complete kernel configuration in the kernel
CONFIG_KALLSYMS Prints our symbolic crash information and backtraces
CONFIG_BUG Disables BUG and WARN functions
CONFIG_HOTPLUG Can be disabled if no external devices will be attached and if you use static device files
CONFIG_DNOTIFY File change notification to user space
CONFIG_EXT2 Disable if using jffs2 file system
CONFIG_PRINTK Makes kernel silent when disabled
CONFIG_PRINTK_TIME A way to track where time is spent at boot time
CONFIG_CC_OPTIMIZE_FOR_SIZE Will select –Os instead of –O2 resulting in a smaller kernel
113
114 Userspace Optimizations
Filesystem modifications• Optimize the RC Scripts (remove the ones you don’t use)• If possible replace RC scripts with a single init• Pre-link• Link applications statically• Avoid udev – create needed device nodes ahead of time• Use static IP address
Filesystem modifications• Staged booting• Code level modifications• Add a splash screen
Boot Time<1s 75s
114
115 Fast Boot Offering from Timesys
If your requirements include fast booting, Timesys can help you save time
Implement added levels of optimizations with open source techniques
• Some of the deep-level optimizations techniques, described earlier
Use non open source technologies available to Timesys to further speed up the boot-time
• Typically needed when many services have to be up and running asap
This engineering experience is available as a service
115
116 Module 4 — Quiz
1. Why is fast boot useful?
2. What knowledge is required to get under 10s?
3. Can one get to sub 2s boot time?
116
117 How to Reach Us
Maciej Halasz — for questions related to this workshop• +1-412-325-6313• [email protected]
Al Feczko — for questions related to LinuxLink subscriptions• +1-412-325-6390 (o) • [email protected]
If you have a LinuxLink userid and password, you can work with our technical support engineers by submitting support requests directly• https://linuxlink.timesys.com/support
117
118
Q&A
Thanks for attending!
118
