Embed Size (px)
Citation preview
SUSE® Linux EnterpriseReal Time Extension
Kai DupkeSenior Product Manager
Libor PecháčekProject Manager
2
TopicsSUSE® Linux Enterprise Real Time Extension
Overview
Recent Updates
Customers
Roadmap
Feature Details
3
Challenge
• Do you lose money if the system timing is wrong?
• Do you need
– Deterministic command execution
– Deterministic timing
– Low latency response on events
– Low latency communication
Real Time
Can you afford delayed execution?
Real Time ExtensionOverview
5
OverviewSUSE Linux Enterprise Real Time Extension
• An extension to SUSE Linux Enterprise Server
• Provides deterministic low latency performance for time-critical applications
• Industry-standard real-time version‒ Kernel preemption
‒ CPU shielding
‒ Task prioritization
‒ Priority inheritance
‒ Interrupt threads
‒ Open Fabrics Enterprise Distribution
6
OverviewSUSE Linux Enterprise Real Time Extension
• Latency sensitive Workloads
‒ Guaranteed process response
‒ Sub-millisecond latency
‒ Immediate resource access for time critical processes
• Low jitter‒ Repeated execution in the
same time
‒ Precision Time Protocol
• Separate Processes‒ Hierarchical priority scheme
‒ CPU shielding
‒ Task prioritization
• Low Latency Communication
‒ 10G ethernet / Infiniband
‒ TCP offload
• LTTng 2.1‒ Application tracing
8
Target CustomersSUSE Linux Enterprise Real Time Extension
Manufacturing, Telecommunication, Finance
Using SUSE for standard workloads
Customized applications
Replace proprietary and embedded Real Time
Test latency impact on workloads
Recent Updates
10
Service Pack 3SUSE Linux Enterprise Real Time Extension
• Update to same 3.0 kernel as SUSE Linux Enterprise Server
‒ Small patchset
‒ Focus on RT capabilities
‒ Share hardware enablement
• Precision Time Protocol (PTP)
Customers
12
CustomersSUSE Linux Enterprise Real Time Extension
Jülich Supercomputer Center
Thyssen Krupp Electric Steel
UMB Financial Corp.
NASA JPL Telescope
13
NASA JPLSUSE Linux Enterprise Real Time Extension
• 200-inch telescope with adaptive optics on Mount Palomar
• Avoid atmospheric blurring in Real Time
• Control more than 3000 mirror segments with a latency <250 msec
Roadmap
15
RoadmapSUSE® Linux Enterprise Real Time Extension
SLE RT 11 SP2• Same Kernel
as SLES• HW enablement
shared with SLES• Tracing tools:
- LTTng- Eclipse Plug-in
• NVIDIA drivers
SLE RT 11 SP2• Same Kernel
as SLES• HW enablement
shared with SLES• Tracing tools:
- LTTng- Eclipse Plug-in
• NVIDIA drivers
SLE RT 11 SP1• PREEMPT_RT Kernel• Per device IRQ threads• Enhanced scheduler• Tuning tools:
cset, rt-test,perf
SLE RT 11 SP1• PREEMPT_RT Kernel• Per device IRQ threads• Enhanced scheduler• Tuning tools:
cset, rt-test,perf
SLE RT 12• Based on SLES 12• GPU computing• Match throughput
of non-RT
SLE RT 12• Based on SLES 12• GPU computing• Match throughput
of non-RT
2009 2010 2011 2012 2013 2014
SLE RT 10
SLE RT 11
SLE RT 12
GA SP1 SP2
GA
SP3
SP3
SLE RT 11 SP3• LTTng update• Precision Time
Protocol
SLE RT 11 SP3• LTTng update• Precision Time
Protocol
16
Customer Timer BenchmarkSUSE Linux Enterprise Real Time Extension
Improvement: Factor 1,800!
Feature Details
18
Kernel PreemptionSUSE Linux Enterprise Real Time Extension
• Time critical tasks get immediate access to the CPU
• Real-Time Kernel improvements‒ Priority Inheritance Mutexes substitute non-premptive
spinlocks (enables preemption for critical Kernel sections)
‒ adaptive locking
‒ read-write locks converted
‒ preemptive interrupt handlers in thread context
19
Priority Inversion – Problem SUSE Linux Enterprise Real Time Extension
• High priority task wait for lock release of low priority task initiated by medium priority task
Lock
Low Priority Task
High Priority Task
High Priority Task
keep
wait
Effective priority goes down
Med. Priority Task
preempt
20
Priority Inversion – SolutionSUSE Linux Enterprise Real Time Extension
• Priority Inheritance‒ Boost priority of low priority process until the lock is released
Lock
Low Priority Task
High Priority Task Low Priority Task
wait
inheritpriorityrelease
Med. Priority Task
22
Interrupt ThreadsSUSE Linux Enterprise Real Time Extension
• Convert interrupts into threads‒ Prioritization possible
‒ Preemption possible
‒ Leverage CPUSets
preemptive
non preemptive
Top Half
Bottom Half
Non Real TimeReal Time
Wake up Thread
Bottom Half
Top Half
23
PrioritizationSUSE Linux Enterprise Real Time Extension
• Tasks and interrupts can be prioritized
• Optimized schedulers‒ FIFO & ROUND_ROBIN for real time tasks
‒ OTHER for non-real time tasks
‒ Manipulated by chrt(1)
• Threads can be prioritized above interrupts
Profiling, Tracing
26
Profiling, Tracing
• Profiling‒ Statistics, interested in big picture
• Tracing‒ Like logging
‒ Interested in detailed sequence of events
28
Oprofile and perf
• Stochastic profilers
• Leverage hardware support
• Transparent background operation while collecting information
• Suitable for finding bottlenecks in real-world systems
• perf top
29
perf top
Kernel compilation running
Tracing
32
ftrace
• In-depth view of kernel function calls
• Function tracer
• Function graph tracer
• Deepest stack, preemption disabled, latency threshold
• Capture trace upon oops or panic
• All these features available with kernel-trace
33
trace-cmd – ftrace frontend
• Ftrace is controlled through debugfs by default
• trace-cmd simplifies capturing traces
• … and also viewing them
34
SystemTap
• Scripted tool for examining live Linux systems
• SystemTap script lists events and associated handlers which execute an action
‒ Probe can be inserted inside a function
• Script compiled into kernel modules
• Can access and alter arbitrary structures
36
LTTng
• Kernel and userspace tracing
• Minimal performance impact
• Kernel instrumentated in key subsystems‒ Scheduler, timers, signal, IRQ, block
‒ Dynamic trace points
• Application instrumentation points‒ May be added into signal handlers and libraries
37
LTTng 2.1 - New features
• Network streaming‒ A new component: lttng-relayd
• Userspace tracing (UST)
• Flush pending records to storage upon lttng stop
‒ Can be overriden on command line
38
LTTng Eclipse Viewer
• Eclipse plugin available for graphical viewing
• Following kernel-oriented analysers are offered:‒ Control Flow - to visualize processes state transition
‒ Resources - to visualize system resources state transitions
‒ Statistics - to provide simple statistics on event occurences
• SUSE specific‒ CPU Flow – to visualize what each CPU has been doing
39
LTTng Eclipse Viewer
Precision Time Protocol
41
Precise Timekeeping
• Across a group of hosts (c.f. NTP)
• Leverages hardware support‒ Dedicated network interface card oscillators
• Master – slave mode of operation‒ Master is elected dynamically within the group
• Sub-microsecond synchronization
42
Master-Slave Design
Slave Clock
Master Clock
LANSymmetric paths,
Low jitter
• Master sends out time information into network‒ Often in two steps - “time signal” and time information
• Slave nodes calculate offset and adjust their clocks
Slave Clock
Slave Clock
Slave Clock
Slave Clock
Slave Clock
Slave Clock
43
Hardware-assisted PTP in Linux
• PTP Hardware Clock (PHC)
PHYMAC
PTPClock
Control Logic
PTPTime
Stamping
LAN
NICDriver
phc2sys
ptp4l
SystemClock
Network interface card
Corporate HeadquartersMaxfeldstrasse 590409 NurembergGermany
+49 911 740 53 0 (Worldwide)www.suse.com
Join us on:www.opensuse.org
44
Unpublished Work of SUSE. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.
