Upload
docker-inc
View
1.441
Download
0
Embed Size (px)
Citation preview
Docker RPi2 challenge@damienduportal @ndeloof @YoannDubreuil @Quintus23M
Challenge Introduction
DockerCon 15, June 22-23
Live Demo: starting 100x Docker Containers on a Raspberry Pi 2 B
showed a screen shot with 250x running Containers
and plenty of resources/memory left, let’s improve it
OK, let’s start the Challenge
Challenge: How many Containers could be really started on a Pi 2?
my wish: I’d like to see 1000x Containers
personal record to this time: 613x Containers
--- but honestly, I was not sure, if this is possible at all!
let’s break the challenge
First tries to the recordTry, Learn and Share !
1 - Measure and automate all the things
Measures :
sysstat for post mortem
node-collector from Prometheus.io for “real time”
Provisionning :
Basic shell script published on my Github
Let’s try Ansible later : easier to maintain, need to learn :)
2 - Lower the container footprint
Tried with custom compiled nginx for ARM with few extensions ➢ ~ 80 containers
Footprint too big per container. Reading carefully Hypriot Blog : "rpi-nano-httpd" : 16 Kb footprint, 88 Kb image
➢ ~150 containers
3 - Docker tuning
Wanted to disable tcp-proxy process : no use here
Docker logs says “iptables” errors
Footprint with iptable + net. namespace : around 1 Mb !
Disable network / port forwarding + --net=host (Client side)
--bridge=none --iptables=false --ipv6=false --ip-forward=false --ip-masq=false -tls=false --userland-proxy=false (Engine side)
4 - Moaaar Docker tuning
No debug : --debug=true
No filesystem writing : --read-only=true
No logging : --log-driver=none
Following Docker upgrades :
--ipc=host
--utc=hostReached ~ 950 containers
5 - Linux tuning : start sharing
Tune sysctl : limits ! (MaxOpenFiles, MaxProcs, etc.)
Still stuck, footprint / container still big (~ 750 Kb)
Help from @booya and Hypriot guys + eading the /etc/init.d/docker script : Stack size (8 Mb -> 2 Mb in Hypriot:Jack)
Playing around doing abacus : <32 Kb breaks docker pull, <16Kb breaks docker run
~ 1700 containers
6 - To the record : Learning and sharing
GOGC to 10
~ 2 200 containers
Docker Hack days + Twitter :KSM : KO (need to recompile Kernel)
Learning the Linux memory management : Freeing caches + disabling OOM
~ 2498 containers !!!
Share and challenge !Community, discussions, challenge and Whales
BreizhCamp Hack (night) Party
Brainstorm for ideas
… then test everything in arbitrary order
help each other
drink some beer
learn a lot
Our Methodology
where to start?
first naïve tryonly 38 containers :-\ !
but 70 on a RPi1
figure out RPi2 limits without Dockerweb server footprint
network namespace footprint
get some help !let’s collaborate with @DamienDuportal (aka “French mafia”)
web server
from hypriot/rpi-nano-httpd image
written in ARM assembly code
already highly optimized1 page for code
1 page for data
1 page for stack
1 page for vsdo
=> 16kb memory footprint per process !
launched 27.000 on a RPi2
network namespace RPi2 limit
launched web server in a dedicated network namespace
ip netns exec <NS_NUMBER> httpd
RPi2 limit is ~ 1.100 network namespace
=> To break the challenge, we needed to run without network isolation
network configuration
disabled everything we could on Docker daemon
--bridge=none --userland-proxy=false --sig-proxy=false --ipv6=false --ip-masq=false --iptables=false
started containers with --net=host
disabled IPv6 on Linux kernel boot with ipv6.disable=1
Systemd
Docker daemon run as root
… but still has some limits set by systemd (so the 38 containers...)
LimitSIGPENDING=infinityLimitNOFILE=infinityLimitAS=infinityLimitNPROC=infinityLimitSTACK=?
stack size
Damien pro-tip : adjust LimitSTACK parameter to your needs
Default stack size is 8Mba stack consume 8Mb of process VM space (8 * 4 * 38 = 1,2 Gb)
=> ~ 1800 / 2000 containers
process memory layout
RPi2 processor is 32 bits !32 bit process address space
only 3Gb of virtual memory per process
collision between stack and heap memory areas
reduce thread stack sizesmallest working thread stack size: 24kb
reduce Golang memory consumptionlaunched docker with GODEBUG=gctrace=1 GOGC=1
speed up tests
launching thousands of containers on a RPi2 takes hours if not days!
everything in memory with zram devicesswap (ratio 5:1)
/var/lib/docker on ext4 FS (ratio 10:1)
swap as early as possible to keep free memory (vm.swappiness = 100)
USB external disk vs low perf, I/O limited SD card
more CPU for GO with GOMAXPROCS=4
reduce kernel perf event slowdown kernel.perf_cpu_time_max_percent = 1
system tuning
limit memory consumptionreduce GPU memory to 16Mb (can’t do less)
blacklisted non required Linux modules
remove some Linux limitsvm.overcommit = 1
kernel.pid_max = 32768
kernel.threads-max = 14812
Challenge completed!
We started 2499 containers !
RAM on RPi2 was not exhausted but Docker daemon crashed
docker[307]: runtime: program exceeds 10000-thread limit
Why is there a limit?
4 threads per container
10.000 threads for a Go application => 2500 containers max
Unlocked this with runtime.debug.SetMaxThread(12000)
hack not eligible for RpiDocker challenge, was just to confirm
can run ~2740 webserver containers, before actual OOM
what did not work
Btrfs
not working properly : strange web server 404 failures after ~20 successful launchs
stick with overlayfs
LXC driver
way sloooooooower
4 threads per container anyway
Go 1.5
compiled Docker with Go 1.5 for “better GC”, had no significant impact
What’s next
Understand why Docker do need 4 threads per container
(hey, lot’s of Docker core contributors here, time to ask !)
Collaboration (and beer) were the keys
to break this challenge !
links
Nicolas’ blog
http://blog.loof.fr/2015/10/how-to-run-2500-webservers-on-raspberry.html
code
https://github.com/ndeloof/RPiDockerKillerScore