Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Bob FengDirector, Consumer Business SegmentPrincipal System Architect, ProAV & Broadcast Business Segment03/19/2019
Streaming Media & Entertainment
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Professional Streaming Endpoints
˃ DistributionHigh quality & Low latency
‒ Pleasant user experience
Low bitrate‒ Network bandwidth efficiency
Low power‒ Cost efficiency
˃ ArchiveLossless quality‒ Post editing & production
Low storage size‒ Cost efficiency
Simultaneously satisfy distribution & archiving need
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
RTP
/RTS
P –
UD
P C
ompr
esse
d Fl
ows
Unc
ompr
esse
d Fl
ows
Unc
ompr
esse
d Fl
ows
Professional Streaming Network
Sense + Acquire
Distribute Watch + Interact
Deliver (Stream)
Compress + Package
Capture + Process
Content Delivery Network
Scal
e / C
onve
rt / M
ix
Manageable End-to-End Latency
LightCompression
HLS
>> 3
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Shared Media Content at Home
˃ One set up box per broadband connectionInadequate bandwidth for more anyway
˃ Leverage in-house home network WiFi or Gigabit Ethernet
˃ Transcoded stream and multicastMultiple watchers in multiple rooms
Set Up Box
Broadband Internet
Shared Watch: No Compromised Immersion
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Shared Game Play at Home
Shared Play: Equalized Smoothness
Geforce GTX 1080tiCoffee Lake i9-9900K64GB DDR4
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Broadband Modem
Office
RemoteContent
Game Console
STB
1 GbE / WIFI Switcher/Router1 GbE
HDMI HDMIHDMI HDMI
HDMI HDMI
HDMI
1 GbE 1 GbE
1 GbE
1 GbE1 GbE
Broadband
STB STB
STB
A lot of setup boxes: Broadband Bandwidth ???
Conventional Content Watching Network at Home
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Retaining the best quality & Minimizing the end-to-end Latency
Latency
Conventional Game Play Network at Home
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Challenge #1 - Picture Quality: A Bandwidth Trade-off
˃ Original ˃ 55Mbps
˃ Average PSNR: 33.12 dB
˃ 6.65Mbps
˃ Average PSNR: 23.98 dB
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Video vs. Home Broadband Bandwidth
High resolution video: The root cause of Internet Speed Demand
Deeply H.264/265 compressed; If lower, picture quality suffers
Raw Bandwidth 0.4 Gbps 3 Gbps 12 Gbps 48 Gbps
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Challenge #2: Latency - Game Play Crucial Factor
Smooth game play: Motion (joystick/mouse move) to Photon (Display) < 20 ms
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Challenge #3: Codec Mystery - Lossless vs. Lossy
Type Codec Choices Compression Ratio
Picture Quality (PQ) 4K Bandwidth
Latency Network
Lossless(Visually or Mathematically)
TICO/JPEG-XSVC-2JPEG2000DNxHDProRes
4:1 to 10:1 Best PQ (close to original) and low-latency (<< 1 frame *)4:2:2/4:4:4 - suitable for both video and graphics content
1 ~ 3 Gbps 1 ~ 10 ms Wired:Gigabit EthernetLAN / Home
Lossy H.264/AVCH.265/HEVC
100:1 to 300:1 Decent PQ and latency (>= 1 frame *)4:2:0 typical –suitable for video content
25 ~ 50 Mbps >= 16ms Wireless:802.11 A/CInternet / WAN
Proper choose between codec type and network
* Per 60-Hz frame = 16ms
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Stream Throughput > Network BandwidthToo many setup boxes
Too many concurrent streams
Timing critical AV packet Late arrivalsIrrelevant competing packets
Challenge #4: Packet Losses & Out of order
Appropriate traffic shaping and prioritization desired
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Unbearable Experience #1: Crappy Image Quality
Streaming Master: Game PC Streaming Slave: Mac Air
Likely Cause: Improper codec + packet loss
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Unbearable Experience #2: Obese Latency
Streaming Master: Game PC Streaming Slave: Mac Air – delayed 100ms
Likely Cause: Improper codec + Packet out of order
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Req
uire
men
tsTr
ends
Other Challenges
Multi-Stream & Multi-Codec
Several formats, codecs, and bit rates
Performance
Portable, low power, flexibility
Need to reduce end-to-end latency
Low LatencyLive IP Streaming
YouTube, Facebook, othersOTT streaming
Synchronization
Simultaneous capture & playback
HD and UltraHD
1080p and 2160p growth
Connectivity
Higher data rates and different standards
Storage
Several types
>> 15
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Solution Offers from Xilinx
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Zynq UltraScale+ MPSoC Single-Chip Solution
Real-Time Processors32-bit Dual-CoreUp to 600MHz
Platform & Power ManagementGranular Power ControlFunctional Safety
Configuration & Security UnitAnti-Tamper & TrustIndustry Standards
FPGA Acceleration16nm UltraScale+™ LogicCustomizable Engines
Video Codec8K4K (15fps)4K2K (60fps)
High-speedPeripheralsPCIe® Gen2, USB 3.0, DisplayPort
Graphics ProcessorARM Mali-400MP22D/3D Visualization
MemorySubsystemDDR3/4LPDDR3/4
Application Processors64-bit ARM®v8Up to 1.5GHz
* ZU11EG, ZU15EG, ZU19EG Only
CG DevicesDual-Core A53s
EG DevicesQuad-Core A53sGPU
EV DevicesQuad-Core A53sGPU and Video Codec
>> 17
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Xilinx VCU FeaturesEncoder Decoder
Feature H.265 (HEVC) H.264 (AVC) H.265 (HEVC) H.264 (AVC)General • Embedded Processors
• Separate Encoder/Decoder Engines -> Simultaneous Encode and Decode• Up to 8 streams of encode and 8 streams of decode simultaneously; 1 stream at max. resolution & rate
Profiles MainMain IntraMain10Main10 IntraMain 4:2:2 10Main 4:2:2 10 Intra
BaselineMainHighHigh10High 4:2:2
MainMain IntraMain10Main10 IntraMain 4:2:2 10Main 4:2:2 10 Intra
Baseline MainHighHigh10High 4:2:2
Level Up to 5.1 High Tier Up to 5.2 Up to 5.1 High Tier Up to 5.2Maximum Resolution/rate 3840x2160p60 (4KUHD), 7680x4320p15 (8KUHD), 4096x2160p30 (DCI4K)Sample Bit Depth 8 or 10 bpcChroma Format YCbCr 4:2:0, 4:2:2, 4:0:0Slice Type I, P, BVideo Format ProgressivePixel Format Semi Planar (YCbCr)Entropy Encoding CABAC CABAC, CAVLC CABAC CABAC, CAVLCSlice Support Supported Supported Supported Supported Tile Support Supported Not Applicable Supported Not ApplicableIn-loop Deblocking Filter Supported Supported Supported SupportedConfigurable Bit Rate Up to 533Mb/s Up to 960Mb/s Up to 533Mb/s Up to 960Mb/s
Rate Control CBR, VBR, Constant QP
>> 18
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Xilinx VCU Latency˃ Xilinx has control over stages 2, 3, 5, and 6
Stages 1, 4, 7 are user dependentAdditional latencies in jitter buffers (few ms) and input/output sync (up to 1 frame)*
˃ The calculations assume 60 frames per second
˃ Low latency assumes 8 slices per frame
˃ Low latency will have restrictions on GOP structure and bit rates
Date Use caseCapture Latency
(ms)
Encode Latency
(ms)
Decode Latency
(ms)
DisplayLatency
(ms)
Total(ms)
2017.1 Normal latency 16.6 16.6 83.33 16.6 133
2017.3 Reduced latency 16.6 16.6 50 16.6 99.8
2018.1 Low latency VCU Only 16.6 4 22.6 16.6 59.8
2019.1* Low Latency Full Pipeline* 5 5 12.5 12.5 35
Video Processing Stages Adding Latency
*Numbers and plans subject to change
PS DDR Memory (Encoder, Capture,
and Display)
PL DDR Memory (Decoder)
Live Video Source
HDMI / SDI / MIPI /
Ethernet / DP
Processing Pipeline (VPSS)
Frame Buffer DMA
Encode
Ethernet
Video Sink
PS DDR Memory (Encoder, Capture,
and Display)
PL DDR Memory (Decoder)
HDMI / SDI / MIPI /
Ethernet / DP
Video Mixer
Frame Buffer DMA
Decode
Ethernet
1 Camera
2 Capture
6 Display
3 Encode
5 Decode4
IP Network
7 Monitor
2 3 5 6
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
MPSoC Development Tools & Target Reference Designs
˃ Demonstrates capabilities of EV device’s VCU integrated block
˃ HDMI Live video capture to evaluate the compression quality
˃ Video encode and decode
˃ Encoder parameter configuration
Vivado® HLx– Targets the FPGA fabric– Primarily used by FPGA designers– Supports novice to expert level flows– Used to design hardware and
generate hardware definitions
SDSoC™– Targets the APU and FPGA logic– Builds on world-class Vivado tool and
Eclipse– Primarily used by software designers
and system architects– Allows rapid SoC-level
hardware/software iterations
PetaLinux– Yocto-based embedded software
development– Integrates several other tools/scripts
to provide unified development flow– Primarily used by software
developers
VCU TRD (Target Reference Design)
MPSoC Base TRD
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Lossless Codec IP from Xilinx PartnersJPEG 2000 JPEG-XS VC2 ProRes
Standard Body ISO 15444-1 ISO 21122 SMPTE 2042-1 AppleMaximum Resolution/rate SD, HD, 3840x2160p60 (4KUHD), 4096x2160p30 (DCI4K), 7680x4320p60 (8KUHD)Sample Bit Depth 8, 10, 12, 14, 16 bpc 10, 12 bpcChroma Format 4:2:2, 4:4:4, 4:2:2:4, 4:4:4:4 4:2:2, 4:4:4, 4:4:4:4Color Space RGB, YUV, XYZ, YCbCr RGB, YCbCrVideo Format Interlaced or ProgressiveLatency Encode: 1~2 frames
Decode: 0.5~1 frames Near zero: a few lines Near zero: a few lines Encode: 1+ framesDecode: 1+ frames
External Memory DDR3/4 N/A N/A DDR3/4Configurable Bit Rate or Compression Ratio CBR: 0.25 ~ 8 Gbps
Lossless: variable & unlimitedNear lossless: 2:1Visually lossless: 10:1 Visually lossless: 4:1
1080p50: 245 Mbps3840x2160p60: 1 Gbps7680x4320p60: 4 Gbps
Application Streaming & Archiving Archiving & NLEFPGA Fitting 2K HD: 7K70T
4K UHD: 7K325T8K UHD: KU060
2K HD: Aritx/Spartan-74K UHD: Aritx/Spartan-78K UHD: K7 / KU / KUP
2K HD: Aritx/Spartan-74K UHD: Aritx/Spartan-78K UHD: K7 / KU / KUP
2K HD: Kintex-74K UHD: KU8K UHD: KUP
Vendor
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
SMPTE standards & protocolsIS06/NMOS: media network discovery and enumeration
SMPTE 2110: packetization & traffic shaping
STMPTE 2022-5: SDI packetization & FEC
SMPTE 2022-7: Dual stream redundancy
SMPTE 2059: Multi-endpoint synchronization
IEEE 1588: Precision time stamps
……
Improved Network Protocols
A pile of broadcast standards developed over 10+ years
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Partner Video over IP PortfolioTr
ansp
ort SMPTE 2022-1/5
Forward Error Correction (FEC) for 1G and 10G networks respectivelySMPTE 2022-2/6Transport protocol for SDI video on IP networks (1G and 10G)SMPTE 2022-7Seamless protection switching to allow network redundancySMPTE 2110Transport protocol for separate audio, video and ANC dataSony NMI Sony’s proprietary transport protocol
Tim
ing SMPTE 2059-1/2
Precision Time Protocol to allow nano-second level synchronization of media streams
Soft
war
e NMOS/IS-04, IS-05, IS-06 Network Media Open Specifications Control SW used to configure and manage media networksSony Live System Manager (LSM) Sony’s control SW
Com
pres
sion
Integrated Broadcast CodecsintoPIX TICO JPEG XS (pending)intoPIX JPEG2000intoPIX JPEG2000 ULL Sony LLVCAll lightweight compression codecs for transporting 4K over 10GbE
http://www.macnicatech.com/
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Conceptual FPGA design for Professional Streaming
Processing System
Programmable Logic
DDR Controller
PlatformManagement
UnitSecurity System
Control
Application Processing UnitQuad-Core
ARM® Cortex™-A53
Graphics Processing UnitARM Mali-400 MP2
High-Speed Connectivity
General ConnectivityReal-Time Processing UnitDual-Core
ARM Cortex-R5
Video Processing
Multi-Channel VCUH.264/H.265
“Lossless ” Video CodecProRes, DNxHD, TICO, JPEG-2000, JPEG-XS
DDR Memory
DIsplayPortUHD-SDIHDMIDIsplayPortUHD-SDI
DDR Memory
Cable DriversCable EqualizersRetimers
Ethernet PHYsEthernet MACEthernet MAC
USB3.0SATA3.1
GbESD/eMMC
AppsMedia Software Stack
Audio ProcessingSystem Management
OS
Capture Process Compress Deliver
Package(Record)
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Ultra low latency, lossless & synchronous transport
HDMI RX
TICO Encoder
TICO Packetizer
AES67 Packetizer
Audio Extract
ST2110Scheduler
ST2059 Master L2+L3
FramerEthernet
MACEthernet
PHY
IEEE1588Master
L2+L3Framer
EthernetMAC
EthernetPHY
EthernetMAC
EthernetPHY
EthernetMAC
EthernetPHY
L2+L3Deframer
L2+L3Deframer
Packet Filter + Host Queue
Packet Filter + Host Queue
ST2022-7PacketBuffer
TICO Depacketizer
AES67 Depacketizer
ST2059Slave
TICO Decoder
VideoTiming
HDMI TX
Conceptual drawing – A 7z030 based reference design coming soon from Macnica
1/10 GbE
1/10 GbE
1/10 GbE
1/10 GbE
Leveraging a pile of SMPTE standards developed over 10+ years
Ultra low latency codec
Ultra low latency codec
Dual channel redundancy
Fully synchronous timing
Fully synchronous timing
Dual channel redundancy
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Flexible Capture with Any-to-Any Connectivity
Sense + Acquire
Distribute Watch + Interact
Deliver (Stream)
Compress + Package
Capture + Process
˃ Xilinx supports UHD-4K interfaces with transceivers and IP cores NOWHDMI 1.4/2.0/2.0a, DisplayPort 1.2a, 12/6/3G-SDI, 10/25/100GbEST2110 support with partner IP
˃ Short Xilinx roadmap to UHD-8K supportDisplayPort 1.3/1.4HDMI 2.1
˃ V4L2 device driver support
˃ Flexibility to use only the I/O you need
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Sense + Acquire
Distribute Watch + Interact
Deliver (Stream)
Compress + Package
Capture + Process
Custom Video Processing Engines for Your Application
˃ Video processing involves transforming the uncompressed data in some way
Examples: Color Space Conversion, De-interlacing, Scaling, MixingVideo processing ASSP roadmaps are weak for UHD-4K and beyond
˃ FPGA logic-based processing allows highly customized processing pipelines before the compression stage
˃ Real-time video processing IPXilinx VPSS (Video Processing Subsystem) and VMIX (Video Mixer) ‒ Supports UHD4k60p
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Sense + Acquire
Distribute Watch + Interact
Deliver (Stream)
Compress + Package
Capture + Process
Integrated VCU for Low-Power Multi-Channel Compression
˃ VCU: Integrated Multi-Channel Encode/Decode
˃ EncodeH.264 and/or H.265, up to UHD-4kp60Configurable GOP, Rate Control (CBR, VBR, Constant QP)
˃ DecodeH.264 and/or H.265, up to UHD-4kp60
˃ Power EstimateMany, many variables involved; exact figure depends on your use caseApproximate power for encode core: ‒ HEVC UHD4k 4:2:2 10-bit: ~0.9W (30 fps) and ~1.8W (60 fps)
Please use XPE (Xilinx Power Estimate) tool for detailed estimate
• ProRes• DNxHD• TICO• VC-2
• JPEG2000• And more…
˃ Visually lossless
˃ Fabric-based solutions include:
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Sense + Acquire
Distribute Watch + Interact
Deliver (Stream)
Compress + Package
Capture + Process
Flexible Encapsulation, Delivery, and Storage Options
˃ HardwareEthernet: Embedded quad-gigabit or soft IP options (10GbE)Integrated interfaces: SATA 3.1, USB 3.0, SD/eMMCLogic can be used for additional storage interfaces if needed
˃ Ethernet: Streaming OutputVideo bitstream is encapsulated in many formats using the APUHTTP (HLS, HDS, MSS, DASH), UDP (RTP, RTSP, RTCP)
˃ Package and Record: File OutputVideo bitstream is encapsulated in any file format requiredMP4, MOV, AVI, MXF, etc…
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Improved Home Content and Game Sharing
HDMI IP
1/10/25 GbE / WIFI Switcher/Router1 GbE
Office
HDMI IP
HDM
I
HDM
I
HDMI IP
Game Console
STB
HDMIHDM
I HDMI
HDMI
HDMI IP
HDMI
HDMI IPHDMI
HDMI
1 GbE
1 GbE1 GbE
1 GbE
1 GbE
Broadband Modem
RemoteContent
Broadband
1 GbE
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
The Best Offer for Capture, Compress and Deliver
˃ Capture, Compress, and DeliverZynq Ultra Scale+ MPSoC’s EV device provides a solution for multiple capture interfaces, compression options, and reliable deliveryFPGA logic integrated with SoC provides high degree of flexibility and differentiation
˃ Solves the Key Requirements for Streaming & RecordingCapture: HDMI, DisplayPort, SDI, and moreProcess: Logic-based programmable video processing pipelineCompress: Embedded multi-channel multi-codec compression and decompression built-inPackage: Software-based packaging of streams in any transport or file formatDeliver: Ethernet streaming (soft or hard IP)Record: Multiple built-in interfaces for storage media; augment with additional interfaces if needed
˃ Open-Source Media Framework
˃ Industry-leading Tools (Vivado, PetaLinux, SDSoC) & TRD
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
For More Information:
Pro AV & Broadcast Solutions: xilinx.com/broadcast Zynq UltraScale+ MPSoC: xilinx.com/products/silicon-devices/soc/zynq-ultrascale-mpsoc.htmlMPSoC Reference Designs: wiki.xilinx.com/Technical+Articles#TRDXilinx Software Zone: xilinx.com/products/design-tools/software-zone.html
Contact your local Xilinx sales representative
© Copyright 2019 Xilinx赛 灵 思 技 术 日XILINX TECHNOLOGY DAY
Adaptable.Intelligent.
赛 灵 思 技 术 日XILINX TECHNOLOGY DAY