View
17
Download
0
Category
Preview:
Citation preview
Building, Deploying and Testing
an Industrial Linux Platform
SZ Lin (林上智)
MXcore, Software Supervisor
06/01 2017
Open Source Summit Japan 2017
Confidential
About Me
SZ LIN (林上智)
Software Engineer at Moxa
• Industrial Grade Linux Distribution
Debian Developer
Blog - https://szlin.me
2
Confidential
Industrial/ Harsh Environments Including smart rail, smart grid, intelligent transportation,
factory automation, oil & gas, marine, and more
Confidential 4
Confidential
Smart Grid
Smart Rail
Smart Oil Field
Smart Factory
Smart Transportation
Smart Marine
Application
Device
Industrial Routers
Industrial Ethernet
Industrial Wireless
LAN
Network Infrastructure
Confidential
Embedded Computers
Industrial Computing
Industrial Linux Platform
Serial Connectivity
I/O Connectivity
Video Connectivity
Edge Connectivity
5
Confidential
Smart Grid
Smart Rail
Smart Oil Field
Smart Factory
Smart Transportation
Smart Marine
Application
Device
Industrial Routers
Industrial Ethernet
Industrial Wireless
LAN
Network Infrastructure
Confidential
Embedded Computers
Industrial Computing
Industrial Linux Platform
Serial Connectivity
I/O Connectivity
Video Connectivity
Edge Connectivity
6
Hardware
Software
Confidential
Bug fixes
Security
fixes
New userspace
program backports
New kernel
features backports
Industrial Linux Distribution
Over 10+ years
Maintenance release
7
Confidential
Industrial Linux
Distribution
CI
CD LT
8
Ecosystems for Industrial Linux Distribution
CI : Continuous Integration
CD : Continuous Delivery/ Deployment
LT : Long-term Test
Confidential
Industrial Linux
Distribution
CI
CD LT
10
Ecosystems for Industrial Linux Distribution
CI : Continuous Integration
CD : Continuous Delivery/ Deployment
LT : Long-term Test
Confidential
Master
Develop
Feature ..
Merge
Pull Request
Fork
Local
Branch
Patches
Notification
Internal/ External
Developers
CI/ CD Automatic
Release Pipeline
Maintainer
Approval
Pass
Pass
Y
Y N
N
11
Confidential
Code
Continuous
Integration
CI/ CD Automatic Release Pipeline
Building Testing
Deploying
4 3 2 1
Release
13
Confidential
Server
Slave
node
Webhooks
Patches
Internal/ External
Developers
Local
Branch
Pull Request
Slave
node Slave
node
Static Program
Analysis Coding style
OWASP [2]
Infer [3]
Sonarqube [4]
…
14
Confidential
Test Cases Management - Jenkins
15
Static analysis
#1
Static analysis
#2 …
Static analysis
#n
Confidential
Distributed Compiler
• Software
– Icecream/ IceCC was created by SUSE based on distcc
[5][6]
• Improve performance of compile jobs in parallel
• Add dynamic scheduler of the compilation jobs
• Support multiple platform
• Support cross compiling
• Hardware - for each node
– SSD
– Large capacity memory
– Gigabit LAN
17
Confidential
Client Client Client
ICECC - Distributed Compiler Server
Patches
Internal/ External Developers
Local
Branch
Pull Request
3. Send
log & image
18
Webhooks
1. Send command
2. Get source
Confidential
CI/ CD Automatic Release Pipeline
Building Testing
Deploying
4 3 2 1
Release
Code
Continuous
Integration
Continuous Delivery
19
21
Server
Master
Worker Worker Worker
1. Send job file via XML-RPC
5. Deployment via TFTP
2. Dispatch job via ZMQ
3. Download image
via curl or wget
4. Boot up via Ethernet remote I/O
DUT Clusters
Dynamic Program
Analysis ● gcov [9]
● valgrind [10]
● profiling tools [11]
● ...
Platform Test ● LTP [12]
● Security testing
[13]
● Kselftest [17]
● ...
6. Send test command
Confidential
CI/ CD Automatic Release Pipeline
Building Testing
Deploying
4 3 2 1
Release
Continuous Delivery Continuous
Deployment Code
Continuous
Integration
22
Confidential
Master
Develop
Merge
Notification
CI/ CD Automatic
Release Pipeline
Maintainer
Approval
Pass
Pass
Y
Y N
N
23
Image
Deployment
24/ 7 Long-term
Platform Test
Confidential
Industrial Linux
Distribution
CI
CD LT
25
Ecosystems for Industrial Linux Distribution
CI : Continuous Integration
CD : Continuous Delivery/ Deployment
LT : Long-term Test
Confidential
24/ 7 Long-term Platform Test
Robustness
Robustness is the ability
of a computer system to
cope with errors during
execution and cope with
erroneous input [18]
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [1]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
Longevity
Long-term support at
least 10 years life cycle
with bug fixes, new
features and new
hardware components
27
Endurance test
Compatibility test
…
Confidential
24/ 7 Long-term Platform Test
Robustness
Robustness is the ability
of a computer system to
cope with errors during
execution and cope with
erroneous input [18]
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [1]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
Longevity
Long-term support at
least 10 years life cycle
with bug fixes, new
features and new
hardware components
28
Fuzz testing
[14][15][16]
…
Confidential
24/ 7 Long-term Platform Test
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [1]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
29
Power failure test
Reboot test
…
Confidential
24/ 7 Long-term Platform Test
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
30
Daily test for CVE
…
Confidential 32
Collaboration with
upstream developer Bootloader test integration
Performance/ Unit test
integration Test cases framework
enhancement Keep eyes on Fuego
Future Work
Confidential
References
[1]https://en.wikipedia.org/wiki/Reliability,_availability_and_servic
eability
[2] https://wiki.jenkins-ci.org/display/JENKINS/Plugins
[3] http://fbinfer.com/
[4] https://www.sonarqube.org/
[5] https://github.com/icecc
[6] https://www.slideshare.net/szlin/distributed-compiler-icecc
[7] https://www.linaro.org/initiatives/lava/
[8] http://elinux.org/images/3/35/LAVA_Project_Update.pdf
[9] https://gcc.gnu.org/onlinedocs/gcc/Gcov.html
[10] http://valgrind.org/
[11] https://perf.wiki.kernel.org/index.php/Main_Page
35
Confidential
References
[12] http://linux-test-project.github.io/
[13] http://www.openvas.org/
[14] https://github.com/google/syzkaller
[15] http://codemonkey.org.uk/projects/trinity/
[16] https://github.com/google/oss-fuzz
[17] https://kselftest.wiki.kernel.org
[18] https://en.wikipedia.org/wiki/Robustness_(computer_science)
36
Recommended