Upload
vuongdieu
View
228
Download
4
Embed Size (px)
Citation preview
1
Renesas Electronics America Inc.
© 2010 Renesas Electronics America Inc. All rights reserved.
ID B32L: Graphical Application Developmentunder Linux/Implementing VideoSolutions using G-Streamer
Chris Brandt
13 October 2010
Version: 1.0
Staff Application Engineer
2
2 © 2010 Renesas Electronics America Inc. All rights reserved.
Chris Brandt
� Applications Engineering
� Currently supporting SH4A and SH-Mobile
Linux applications (among other things)
� Worked for Renesas/Mitsubishi 12 years (I’ve
done a little of everything in that time)
3
3 © 2010 Renesas Electronics America Inc. All rights reserved.
Renesas Technology and Solution Portfolio
Microcontrollers
& Microprocessors#1 Market share
worldwide *
Analog and
Power Devices#1 Market share
in low-voltage
MOSFET**
Solutions
for
Innovation
Solutions
for
InnovationASIC, ASSP
& MemoryAdvanced and
proven technologies
* MCU: 31% revenue basis from Gartner
"Semiconductor
Applications Worldwide
Annual Market Share:
Database" 25
March 2010
** Power MOSFET: 17.1%
on unit basis from
Marketing Eye 2009
(17.1% on unit basis).
In the session 110C, Renesas Next Generation Microcontroller and Microprocessor Technology
Roadmap, Ritesh Tyagi introduces this high level image of where the Renesas Products fit. The big picture.
4
4 © 2010 Renesas Electronics America Inc. All rights reserved.
4
Renesas Technology and Solution Portfolio
Microcontrollers
& Microprocessors#1 Market share
worldwide *
Analog and
Power Devices#1 Market share
in low-voltage
MOSFET**
ASIC, ASSP
& MemoryAdvanced and
proven technologies
* MCU: 31% revenue basis from Gartner
"Semiconductor
Applications Worldwide
Annual Market Share:
Database" 25
March 2010
** Power MOSFET: 17.1%
on unit basis from
Marketing Eye 2009
(17.1% on unit basis).
Solutions
for
Innovation
Solutions
for
Innovation
This is where our session, B32L is focused within the ‘Big picture of Renesas Products’, Microcontroller
and Microprocessors.
5
5 © 2010 Renesas Electronics America Inc. All rights reserved.
5
Microcontroller and Microprocessor Line-up
Superscalar, MMU, Multimedia� Up to 1200 DMIPS, 45, 65 & 90nm process
� Video and audio processing on Linux
� Server, Industrial & Automotive
� Up to 500 DMIPS, 150 & 90nm process
� 600uA/MHz, 1.5 uA standby
� Medical, Automotive & Industrial
� Legacy Cores
� Next-generation migration to RX
High Performance CPU, FPU, DSC
Embedded Security
� Up to 10 DMIPS, 130nm process
� 350 uA/MHz, 1uA standby
� Capacitive touch
� Up to 25 DMIPS, 150nm process
� 190 uA/MHz, 0.3uA standby
� Application-specific integration
� Up to 25 DMIPS, 180, 90nm process
� 1mA/MHz, 100uA standby
� Crypto engine, Hardware security
� Up to 165 DMIPS, 90nm process
� 500uA/MHz, 2.5 uA standby
� Ethernet, CAN, USB, Motor Control, TFT Display
High Performance CPU, Low Power
Ultra Low PowerGeneral Purpose
Here are the MCU and MPU Product Lines, I am not going to cover any specific information on these families, but rather I want to show you where this session is focused
6
6 © 2010 Renesas Electronics America Inc. All rights reserved.
6
Microcontroller and Microprocessor Line-up
Superscalar, MMU, Multimedia� Up to 1200 DMIPS, 45, 65 & 90nm process
� Video and audio processing on Linux
� Server, Industrial & Automotive
� Up to 500 DMIPS, 150 & 90nm process
� 600uA/MHz, 1.5 uA standby
� Medical, Automotive & Industrial
� Legacy Cores
� Next-generation migration to RX
High Performance CPU, FPU, DSC
Embedded Security
� Up to 10 DMIPS, 130nm process
� 350 uA/MHz, 1uA standby
� Capacitive touch
� Up to 25 DMIPS, 150nm process
� 190 uA/MHz, 0.3uA standby
� Application-specific integration
� Up to 25 DMIPS, 180, 90nm process
� 1mA/MHz, 100uA standby
� Crypto engine, Hardware security
� Up to 165 DMIPS, 90nm process
� 500uA/MHz, 2.5 uA standby
� Ethernet, CAN, USB, Motor Control, TFT Display
High Performance CPU, Low Power
Ultra Low PowerGeneral Purpose
SuperH
Notes for Devcon Positioning Slide: There’s a lot of vital information on this slide, which spotlights
the Renesas MCU/MPU product lines recommended for new designs. Perhaps the best way to discuss this material is to cover it from a very high level.
Since the merger, we have scrutinized the needs of our global markets, reassessed our strengths, and
implemented a business strategy focusing on supporting the ‘ubiquitous computing’ paradigm. This insightful concept — often abbreviated as ‘ubicomp’, and sometimes termed ‘pervasive computing’ or ‘ambient intelligence’ — was introduced by Mark Weiser of Xerox in 1988.
Ubiquitous computing refers to a new genre of computing, a worldwide electronic environment in which computer-controlled products completely permeate the life of end users around the globe. Obviously,
many types of products and an enormous range of applications are encompassed by this paradigm, all driven by human ingenuity, engineering creativity and marketing expertise. To one extent or another, people everywhere are already beginning to enjoy the first wave of benefits of the concept’s reality.
7
7 © 2010 Renesas Electronics America Inc. All rights reserved.
Innovation
Mobile Media is everywhere…and it’s not going away any time soon.
The perception in the market place is that embedded Ethernet is hard to achieve. The cost of TCP/IP stacks, development tools, and the time to learn the technology all make it very difficult to design
products that talk to each other over Ethernet. However, with a low cost Ethernet enabled MCU and the software freely available in the Open Source community embedded Ethernet is very easy to achieve. Renesas has a complete set of solution for H8S/2472 Ethernet connectivity.
8
8 © 2010 Renesas Electronics America Inc. All rights reserved.
Renesas Multimedia solutions
Renesas has hardware & software solutions to increase
performance and decrease development time.
The perception in the market place is that embedded Ethernet is hard to achieve. The cost of TCP/IP stacks, development tools, and the time to learn the technology all make it very difficult to design
products that talk to each other over Ethernet. However, with a low cost Ethernet enabled MCU and the software freely available in the Open Source community embedded Ethernet is very easy to achieve. Renesas has a complete set of solution for H8S/2472 Ethernet connectivity.
9
9 © 2010 Renesas Electronics America Inc. All rights reserved.
Agenda
� Introduction of GStreamer software
� Features / Usage / Pros / Cons
� Lab Overview
� Q&A
10
10 © 2010 Renesas Electronics America Inc. All rights reserved.
What is GStreamer?
� What is says on their web site:“GStreamer is a library for constructing graphs of media-handling
components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.”
“Applications can take advantage of advances in codec and filter technology transparently. Developers can add new codecs and filters by writing a simple plugin with a clean, generic interface. Read more ...”
� My summary:
“A quick and easy way to add audio & Video support to your product for all kinds of media formats without you ever having to dig up and digest a bunch of media specification……and it’s free.
The purpose of this slide is to have a relevant summary of what GStreamer is that can’t be found on
any webpage
11
11 © 2010 Renesas Electronics America Inc. All rights reserved.
GStreamer Basics
� Website http://gstreamer.freedesktop.org/
� GNU LGPL (GNU Lesser General Public License)� May used (or be used by) proprietary software without source
disclosure
� Example Applications� VLC Media Player
� Mplayer
� Many more…
With the GNU LGPL license, even though the code that you get is free and open source, you can still
use it as part of a project that will not be free/open source. Compared to GPL where any software that uses GPL code will automatically be required to also be open source. The importance of this LGPL license is that you get to use free open source code, but it doesn’t hinder you from making a closed
source product.
VLC and MPlayer are very widely used. Those applications run in both Linux and Windows. Compared to Windows Media Player, VLC and MPlayer can play many more media formats and have more advanced
features.
One thing to note is that while GSTreamer can play all type of media formats, it’s not really the GStreamer software doing the decoding. It’s mostly made up of other open source (and closed source)
codecs. GStreamer is essentially a ‘wrapper’ around all those different codecs and video/sound manipulators making it easy to integrate into a product and switch elements in and out as you needed.
12
12 © 2010 Renesas Electronics America Inc. All rights reserved.
How does it work?
� The framework is a ‘pipeline’ of plugins
Example of a typical pipleline
Looking at this image, the idea of a pipeline is pretty apparent. Data flows from stage to stage where
each stage has a relevant task and prepares the data for the next stage.
13
13 © 2010 Renesas Electronics America Inc. All rights reserved.
How does it work?
� The framework is a ‘pipeline’ of plug-ins
� Essentially you combine plug-ins to get the desired result
� Plug-in outputs are referred as ‘Source’, and inputs are referred to as ‘Sink’
� Plug-ins have unique options that can be set
Element
(Input)
Source
Element
(Decoder)
Source
Sink
Element
(Output)
Sink
Example of a typical pipeline
The terms ‘plug-in’ and ‘element’ mean the same thing.
Knowing which plug-ins are available and which need to be ‘piped’ together and how is the trick about learning GStreamer.
Filters are used to change the default format of data as they flow from element to element. You can think of them as like a type cast in programming language.
In Linux, the term “Sink” is also used to define the final stage of an output. A screen sink for example would be a video frame buffer. An audio sink could be ALSA (a common Linux audio player)
Each plug-in can define as many options as they want. Some plug-ins (like the one Renesas made) even take a configuration file location as an argument because the number of options are greater than
you really want to specify as individual parameters.
14
14 © 2010 Renesas Electronics America Inc. All rights reserved.
How the pieces fit together
http://www.gstreamer.net
Provided by
GStreamer
End
Application
The boxes here in blue represent items that are distributed with gstreamer. The orange one are boxes
that represent applications or utilities that can be created using gstreamer.
15
15 © 2010 Renesas Electronics America Inc. All rights reserved.
GStreamer Tools
� gst-inspect
� Displays all properties, inputs, output types, parameters.
� Example: gst-inspect fakesrc
� gst-launch
� Command line application that can be used to test pipelines.
� Example:
� gst-edit
� A set of widgets to display a
Graphical representation of
a pipeline.
gst-launch audiotestsrc ! audioconvert ! audio/x-raw-int,channels=2 ! Alsasink
GStreamer comes with a default set of command-line utilities that can help in application development.
We will discuss only gst-launch and gst-inspect here.
gst-inspect
gst-inspect can be used to inspect all properties, signals, dynamic parameters and the object hierarchy
of
an element. This can be very useful to see which GObject properties or which signals (and using what
arguments) an element supports. Run gst-inspect fakesrc to get an idea of what it does.
gst-launch
gst-launch is a simple script-like commandline application that can be used to test pipelines. For
example, the command gst-launch audiotestsrc ! audioconvert ! audio/x-raw-int,channels=2 !
alsasink will run a pipeline which generates a sine-wave audio stream and plays it to your ALSA audio
card.
The command line of the application uses exclamation points ( ‘!’ seen in red ) to separate the plug-ins, kind of like the pipe symbol ( | ) on a shell command line.
Plug-ins can also take arguments (seen in blue).
16
16 © 2010 Renesas Electronics America Inc. All rights reserved.
Packages
� GStreamer� The core include some essential elements,
documentation
� Base� Well-maintained collection of GStreamer plug-ins and elements
� The Good (collection of plug-ins)� Well supported, good license, good documentation� demuxers, ffmpegcolorspace, flac
� The Bad (collection of plug-ins)� less supported, needs testing/fixes� swfdec, xvid, faad, celt, jack, fbdevsink
� The Ugly (collection of plug-ins)� Well supported, good code, but…watch out for license issues� mad, lame, dvdread, realmedia
� Others…� ffmpeg plugin, access to most ffmpeg library� Python bindings
The ‘GStreamer’ and ‘Base’ are essentially your starting point for the overall framework.
There is a master list on the gstreamer web site that show what media support is located in which package so you know what to download. By putting the software in these categories, it at least give you a general idea of what type of software you’re getting. For the ‘Ugly’ for example, you may not have
been aware of the licensing issues for some of the media formats, so this gives you a good heads up in that sense.
The ‘Others…’ is referring to the fact that there are about 15 download packages in total. Some codes
(like FFmpeg) or other specialty utilities have a module package completely for themselves.
17
17 © 2010 Renesas Electronics America Inc. All rights reserved.
Pros and Cons
� Pros� Lots of media formats, codecs, parsers, protocols and filters
� Simple projects to large audio/video editing software� Bindings in multiple languages
– C++, Python, Perl, .NET, Guile, Java, Ruby, Vala� Command line tool for pipeline prototyping� Features like auto-detection of pipelines� Standard Linux API support (V4L2)� Pluggable design allows proprietary plugins
� Cons� Trading convince for performance� Indirect use of media libraries
Cons: Since at it’s core, gstreamer is essentially a set of wrappers around codecs and filters. So, we could probably hypothesize that the software flow is not as optimized as it could be. But to that effect, you would really have to know the codecs inside and out to really get that optimization anyway, so maybe even this point is moot.
18
18 © 2010 Renesas Electronics America Inc. All rights reserved.
SH7724 Ports & Custom Plug-ins
� gst-sh-mobile
� Hardware decoder and framebuffer display
– Much faster than using ffmpeg
� Camera capture
� Encode arbitrary video format to h.264 using hardware
� Combine with existing gstreamer demuxers to play h.264 from containers
� https://oss.renesas.com/modules/document/?GStreamer
Camera I/F
Blending Scaling
SoundJPEG
LCDC
NTSC
H.264/MP4 D1@60fps
2D Accel
SH-Mobile SH7724
SH-4A500MHz900MIPs
The SH7724 is a SH4A superscalar SH-Mobile device with FPU.
There is hardware supporting 2 camera capturing devices.
The LCD controller output (while can be used to drive an LCD) can also be used to drive an DVI or HDMI display (given the appropriate interface chip)
The reference to NTSC output (and input by the way) also means it supports PAL.
The sound is a 24-bit Sound processing unit
The major importance of the SH7724 is the hardware video processing units. The blending and scaling units can be tied together so that the data flows through them without software intervention. The 2-D
acceleration is for quickly drawing shapes in a frame buffer. The JPEG engine can be used for decode or encode, and could also be used to make a motion JPEG video file (with additional appropriate software of course).
The H.264 block is the most important. It can encode/decode 60 frames per second at 640x480. It will also do 1280x720 at 30 fps (also knows as HD 720p).
Besides H.264 (also known as AVC or MPEG-4 part 10), the block also supports MPEG-4 (MPEG-4 part 2) and VC-1 (SMPTE 421M-2006).
The oss.renesas.com site contains everything from Linux BSPs, drivers and utilities to presentations by Renesas people.
19
19 © 2010 Renesas Electronics America Inc. All rights reserved.
SH7724 Evaluation Board Features
USB mini-AB
(USB1)
USB Host
(USB0) LAN (PoE)
RMII
USB mini-B for
debug console
(SCIF0)
DC Power
Power Switch
POR Reset
Manual Reset
NMI
MFI Bus
(BSC)
DVI
(LCDC)
LCD Board
(LCDC)
DIP
Switches
Push Button
(KEYSC)
IrDAMic Jack
(FSBI)HP Jack
(FSBI)NTSC/PAL
OUT (DV)
NTSC/PAL
IN (DV)
Camera
1 (VIO0)
Camera
2 (VIO1)
JTAG
SD/MMC
(SHDI1)
SD
(SHDI0)
20
20 © 2010 Renesas Electronics America Inc. All rights reserved.
Lab Overview
� Look at the GStreamer ‘gst-launch’ tool.
� Examine how ‘plug-ins’ are chosen and used
� Container de-muxing
� Camera h.264 encoding
� Streaming h.264 video
� HW vs SW decoding comparison (SH7724)
� Build an application from source
� Modify an application
21
21 © 2010 Renesas Electronics America Inc. All rights reserved.
Checking Progress
� We are using the die to keep track of where everyone is in the lab. Make sure to update it as you change sections.
� When done with the lab, your die will have the 6 pointing up as shown here.
23
23 © 2010 Renesas Electronics America Inc. All rights reserved.
Lab Question Answers
1. Since the custom output plug-in ‘gst-sh-mobile-sink’ does not work with other general GStreamer plug-ins, what you could do for adding effect?
2. Besides available CPU bandwidth, what would also affect flawless playback/recording.
3. Building this application is simple because all the GStreamer libraries and headers were already installed on your board. What do you think it would take rebuild the GStreamer libraries?
24
24 © 2010 Renesas Electronics America Inc. All rights reserved.
Innovation
Mobile Media is everywhere…and it’s not going away any time soon.
The perception in the market place is that embedded Ethernet is hard to achieve. The cost of TCP/IP
stacks, development tools, and the time to learn the technology all make it very difficult to design products that talk to each other over Ethernet. However, with a low cost Ethernet enabled MCU and the software freely available in the Open Source community embedded Ethernet is very easy to achieve. Renesas has a complete set of solution for H8S/2472 Ethernet connectivity.