Upload
michal-sojka
View
1.843
Download
2
Tags:
Embed Size (px)
DESCRIPTION
This presentation reports the results of thorough analysis of timing properties of CAN-to-CAN gateway built with Linux kernel CAN subsystem. The latencies induced by this gateway are evaluated under many combinations of conditions, such as when traffic filtering is used, when the gateway is configured to modify the routed frames, when various types of load are imposed on the gateway or when the gateway is run on different kernels (both rt-preempt and vanilla are included). From the detailed results, we derive the general characteristics of the gateway. Some of the results apply not only for the special case of CAN-to-CAN routing, but also for the whole Linux networking subsystem because many mechanisms in the Linux networking stack are shared by all protocols.
Citation preview
Timing Analysis of a Linux-BasedCAN-to-CAN Gateway
Michal Sojka1, Pavel Píša1, Ondrej Špinka1,Oliver Hartkopp2, Zdenek Hanzálek1
1Czech Technical University in PragueFaculty of Electrical Engineering
2Volkswagen Group Research
13th Real-Time Linux WorkshopOctober 22, Prague
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 1 / 26
Outline
Motivation
Testbed Description
Analysis Results
Conclusions
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 2 / 26
Motivation
CAN Bus
I Intended to interconnect microcontrollers for shortdistances.
I Network with deterministic medium access ⇒ real-timeapplications.
I Heavily used in automotive industry.I Bit rates up to 1 Mbit/s.I Every frame can carry up to 8 bytes of data.I Frames are identified by frame IDs (11 or 29 bits).
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 3 / 26
Motivation
Rapid prototyping with CAN
I Goal: Quickly interconnect devices/subsystems frommultiple vendors.
I Frame ID clashes cause problems – devices cannot beconnected together.
I A need for CAN-to-CAN gateways to route (and modify)messages between multiple buses.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 4 / 26
Motivation
SocketCAN
I Open-source project, official CAN-bus support for Linuxkernel
I Initiated by Volkswagen research – first open sourceproject where Volkswagen participates
I Core developers also from a few other companiesI Contributors from many other peopleI In mainline Linux since 2.6.25I Based on standard Linux networking infrastructureI http://socketcan.berlios.de
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 5 / 26
Motivation
SocketCAN-based Gateway
I Routes CAN frames between multiple CAN buses.I Frames can be modified (ID, length, data)I Modifications: SET, AND, OR, XOR, CRCI Frames can be filtered based on ID/mask pairs.I Implemented as a Linux kernel moduleI Easy to use:
cangw -A -s can0 -d can1
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 6 / 26
Motivation
Motivation of our Work
I Adding gateways to real-time network can violate real-timerequirements (deadlines).
I How are the routed frames delayed by the gateway?I What influences the delays (latencies)?
I Traffic patterns,I Additional load,I Gateway configuration,I Linux kernel version.
I Where are the limits of the SocketCAN-based gateway?
I Nice way to measure temporal properties of the Linuxkernel.
I Some of our results apply not only for CAN, but also forLinux networking subsystem in general.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 7 / 26
Motivation
Motivation of our Work
I Adding gateways to real-time network can violate real-timerequirements (deadlines).
I How are the routed frames delayed by the gateway?I What influences the delays (latencies)?
I Traffic patterns,I Additional load,I Gateway configuration,I Linux kernel version.
I Where are the limits of the SocketCAN-based gateway?
I Nice way to measure temporal properties of the Linuxkernel.
I Some of our results apply not only for CAN, but also forLinux networking subsystem in general.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 7 / 26
Testbed Description
Testbed Setup
I Gateway – PowerPC MPC5200 (Busybox)I PC – Pentium 4, Kvaser PCI quad-CAN SJA1000-based
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 8 / 26
Testbed Description
Testbed Setup
I Gateway – PowerPC MPC5200 (Busybox)I PC – Pentium 4, Kvaser PCI quad-CAN SJA1000-based
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 8 / 26
Testbed Description
Measured Latencies
CAN bus 0
CAN bus 1
time
msg 1
Duration
msg 1'
CAN gateway(Linux)
GW latency
RX timestamp 1 RX timestamp 2
Total latency
I Same clock for bothtimestamps
I SO_TIMESTAMPNSsocket option
I Clock source = TSC
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 9 / 26
Testbed Description
Tested Scenarios
I Traffic patterns:I one message at a time,I 50% bus load,I flood (almost 100% bus load)
I Additional loads:I CPU load – hackbenchI Ethernet load – ping -f -s 6000
I Versions:I Kernel: 2.6.33, 2.6.33-rt29, 2.6.36, 3.0.4, 3.0.4-rt14I Latest SocketCAN from SVN (rev 1199 + fixes)
I All combinations of the above.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 10 / 26
Analysis Results
The Simplest GW Configuration
I Experiment:I Frames with payload of 2 bytesI Message transmission time: 60µs
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 11 / 26
Analysis Results
Presentation of Results – Histogram
0
1000
2000
3000
4000
0 20 40 60 80 100
Fram
es
GW latency [µs]
Histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Cumulative histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Backward cumulative histogram
1
10
100
1000
10000
0 50 100 150 200
Fram
es
Latency profile
I What is theworst-case latency?
I How many frameshad latency greaterthen 60µs?
I Backward-cumulativehistogram.
I Logarithmic scale.I Answer: 10 frames
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26
Analysis Results
Presentation of Results – Cumulative Histogram
0
1000
2000
3000
4000
0 20 40 60 80 100
Fram
es
GW latency [µs]
Histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Cumulative histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Backward cumulative histogram
1
10
100
1000
10000
0 50 100 150 200
Fram
es
Latency profile
I What is theworst-case latency?
I How many frameshad latency greaterthen 60µs?
I Backward-cumulativehistogram.
I Logarithmic scale.I Answer: 10 frames
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26
Analysis Results
Presentation of Results – Backward-Cumulative H.
0
1000
2000
3000
4000
0 20 40 60 80 100
Fram
es
GW latency [µs]
Histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Cumulative histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Backward cumulative histogram
1
10
100
1000
10000
0 50 100 150 200
Fram
es
Latency profile
I What is theworst-case latency?
I How many frameshad latency greaterthen 60µs?
I Backward-cumulativehistogram.
I Logarithmic scale.I Answer: 10 frames
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26
Analysis Results
Presentation of Results – Latency Profile
0
1000
2000
3000
4000
0 20 40 60 80 100
Fram
es
GW latency [µs]
Histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Cumulative histogram
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
Backward cumulative histogram
1
10
100
1000
10000
0 50 100 150 200
Fram
es
Latency profile
I What is theworst-case latency?
I How many frameshad latency greaterthen 60µs?
I Backward-cumulativehistogram.
I Logarithmic scale.I Answer: 10 frames
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26
Analysis Results
Presentation of Results – Multiple plots
0
1000
2000
3000
4000
0 20 40 60 80 100
Fram
es
GW latency [µs]
No loadCPU load
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
No loadCPU load
0 2000 4000 6000 8000
10000
0 50 100 150 200
Fram
es
No loadCPU load
1
10
100
1000
10000
0 50 100 150 200
Fram
es
No loadCPU load
I What is theworst-case latency?
I How many frameshad latency greaterthen 60µs?
I Backward-cumulativehistogram.
I Logarithmic scale.I Answer: 10 frames
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26
Analysis Results
Latency Profile vs. Time Plot
1
10
100
1000
10000
0 20 40 60 80 100 120 140
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
0
20
40
60
80
100
120
140
2.5 3 3.5 4 4.5 5 5.5G
W late
ncy
[µ
s]Experiment time [s]
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 13 / 26
Analysis Results
Measurement PrecisionOur test-bed (PC) versus CANalyzer.
1
10
100
1000
TX-duration=84
0 50 100 150 200 250
Late
ncy
pro
file
[fra
mes]
Total latency [µs]
PCCANalyzer
I Difference 10µs is sufficient for us.I Systematic error?I CANanalyzer has only resolution of 10µs.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 14 / 26
Analysis Results
Measurement PrecisionInfluence of Ethernet load generator on measurement precision. No GW involved!
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 15 / 26
Analysis Results
Measurement PrecisionInfluence of Ethernet load generator on measurement precision. No GW involved!
1
10
100
1000
10000
31 0 50 100 150 200 250 300
Late
ncy
pro
file
[fra
mes]
TX+RX overhead [µs]i.e. tkernel RX - tbefore send - tTX duration
No loadEthernet load
I TX+RX overhead (interfaces can0 and can1 on the PC)I Desired result: vertical lines at the same positionI The error is much smaller than latencies observed for the
gateway.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 15 / 26
Analysis Results
Batched Processing of Frames
1
10
100
1000
10000
0 20 40 60 80 100 120 140
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
One frame at a timeFlood
I Soft-IRQ processes packets in batches (if possible).I Decreases average latency.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 16 / 26
Analysis Results
Effect of Loading the Gateway – CPU
1
10
100
1000
10000
0 50 100 150 200 250 300
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
No loadCPU load
I Hackbench
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 17 / 26
Analysis Results
Effect of Loading the Gateway – Ethernet
1
10
100
1000
10000
0 250 500 750 1000 1250 1500 1750 2000 2250 2500 2750 3000
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
No loadCPU load
Ethernet load
I ping -f -s 60000
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 18 / 26
Analysis Results
Frame Filtering
1
10
100
1000
10000
0 200 400 600 800 1000
Late
ncy
pro
file
[fra
mes]
Different number of filters, only the last one matches
List length 1List length 128List length 256List length 384List length 512List length 768
List length 1024List length 1536List length 2048
1
10
100
1000
10000
0 200 400 600 800 1000
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
2048 SFF filters, different frame IDs
Frame ID 1Frame ID 256Frame ID 512Frame ID 768
Frame ID 1024
Frame ID 1536Frame ID 2048
I Two filter implementations:list traversal, array indexing
I More than 80 list entries ⇒the delay increases faster(cache trashing)
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 19 / 26
Analysis Results
Frame Modifications
1
10
100
1000
10000
0 50 100 150 200
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
No modifications2 modifications4 modifications
4 modifications + XOR checksum4 modifications + CRC8 checksum
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 20 / 26
Analysis Results
Differences between Kernels – non-rt
1
10
100
1000
10000
0 20 40 60 80 100 120 140
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
2.6.33.72.6.36.23.0.4
I The overhead increases over timeI make oldconfig
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 21 / 26
Analysis Results
Differences between Kernels – rt-preempt
1
10
100
1000
10000
0 50 100 150 200 250 300 350
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
2.6.33.72.6.36.23.0.42.6.33.7-rt29
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 22 / 26
Analysis Results
Differences between Kernels – rt-preempt
1
10
100
1000
10000
0 50 100 150 200 250 300 350
Late
ncy
pro
file
[fra
mes]
GW latency [µs]
2.6.33.72.6.36.23.0.42.6.33.7-rt293.0.4-rt143.0.7-rt20
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 22 / 26
Analysis Results
Problem (bug?) in rt-preempt kernelsGW kernel 3.0.4-rt14, Traffic oneatatime, Load none
1
10
100
1000
10000
0.5 1 2 5 10 20 50
Late
ncy
pro
file
[fra
mes]
GW latency [ms]
Frame id 0Frame id 255Frame id 511Frame id 767
I Huge latencies in rt-preempt kernel. Conditions: 2048 EFFfilters, GW kernel: 3.0.4-rt14, traffic: one frame at a time,load: none, payload: 2 bytes.
I Appears also in 2.6.33.7-rt29I High latency repeats periodically every 1 second
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 23 / 26
Analysis Results
Problem (bug?) in rt-preempt kernelsGW kernel 3.0.4-rt14, Traffic oneatatime, Load none
0.1
1
10
100
0 1 2 3 4 5 6 7 8
GW
late
ncy
[m
s]
Experiment time [s]
Frame ID 511
I Huge latencies in rt-preempt kernel. Conditions: 2048 EFFfilters, GW kernel: 3.0.4-rt14, traffic: one frame at a time,load: none, payload: 2 bytes.
I Appears also in 2.6.33.7-rt29I High latency repeats periodically every 1 second
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 23 / 26
Analysis Results
User-Space Gateway
Traffic: one at a time
1
10
100
1000
10000
0 100 200 300 400 500
Late
ncy
pro
file
[fra
mes]
GW Latency [µs]
Kernel GWUserspace GW
Traffic: flood
1
10
100
1000
10000
0.01 0.1 1 10 100 1000 10000 100000
Late
ncy
pro
file
[fra
mes]
GW latency [ms]
Kernel GWUser GWKernel GW, 33-rtUser GW, 33-rt
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 24 / 26
Analysis Results
Results processing
I More than 600 experimentsI HTML-based navigation on results
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 25 / 26
Analysis Results
Results processing
I More than 600 experimentsI HTML-based navigation on results
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 25 / 26
Conclusions
Conclusion
I SocketCAN-based gateway is a reliable solution for CANbus routing.
I Kernel-space solution performs better than a user-spaceone.
I There are things to avoid:I Combining CAN traffic with Ethernet traffic.I Use of too many filters.
I The gateway will be merged in 3.2 kernel.
I Nice way of measuring Linux kernel temporal propertiesfrom outside.
I https://rtime.felk.cvut.cz/can/benchmark/2.1/
I https://rtime.felk.cvut.cz/gitweb/can-benchmark.git
Questions?
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 26 / 26
Conclusions
Conclusion
I SocketCAN-based gateway is a reliable solution for CANbus routing.
I Kernel-space solution performs better than a user-spaceone.
I There are things to avoid:I Combining CAN traffic with Ethernet traffic.I Use of too many filters.
I The gateway will be merged in 3.2 kernel.
I Nice way of measuring Linux kernel temporal propertiesfrom outside.
I https://rtime.felk.cvut.cz/can/benchmark/2.1/
I https://rtime.felk.cvut.cz/gitweb/can-benchmark.git
Questions?
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 26 / 26
Conclusions
Conclusion
I SocketCAN-based gateway is a reliable solution for CANbus routing.
I Kernel-space solution performs better than a user-spaceone.
I There are things to avoid:I Combining CAN traffic with Ethernet traffic.I Use of too many filters.
I The gateway will be merged in 3.2 kernel.
I Nice way of measuring Linux kernel temporal propertiesfrom outside.
I https://rtime.felk.cvut.cz/can/benchmark/2.1/
I https://rtime.felk.cvut.cz/gitweb/can-benchmark.git
Questions?Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 26 / 26