Embed Size (px)
DESCRIPTION
HighLoad++ 2014
Citation preview
scaling compiled applications Joe Damato
@joedamato
about me • Joe Damato
• San Francisco, CA
• i like:
• systems
programming
• debugging
• performance
analysis
@joedamato
timetobleed.com
sassanosystems.c
om
Repeatable builds • the most important thing for compiled software
• set up a build server
• with jenkins (or another type of software)
• backup your built objects
• copying them to s3 is not a horrible idea
• regardless of source control system or branching strategy,
ensure you can always rebuild any version of your
software
Jenkins
problems • git plugin didn’t work on windows (maybe
fixed?)
• branch building is painful, jenkins API can
help
• getting windows build agent working is
painful
why?
почему?
tools for... • RPMs
• DEBs
• Everything else
• windows installers (MSIs)
• other linux/unix like OS’s
• etc
chroot ??
chroot: • an operation that changes the apparent root directory
for the current running process [...].
• A program that is run in such a modified environment
cannot name (and therefore normally not access) files
outside the designated directory tree.
• (from wikipedia)
Everything else
• KVM
• Amazon EC2
• other virtualization
KVM EC2
KVM
• Create a base image on disk
• Clone base image
• Boot the cloned image
• Do the build and copy built object out.
• Delete cloned image when done
• Base image is still pristine and can be reused.
Create builds in cleanroom • Avoid contaminating builds with artifacts from previous
builds.
• chroots help
• use mock or pbuilder for RPMs and DEBs
• KVM, EC2, or equivalent for everything else
• Always create pristine base images and do builds in a
copy.
• Use SSDs
Tool problems • git-buildpackage can’t set changelog distribution field
• signing key setup is really painful (especially for
RPMs)
• deb naming scheme for packages is quite painful
• all tools are undocumented and very hard to actually
use
• recent versions of KVM provided by ubuntu fail to
boot VMs sometimes
two types of
linking....
• dynamic linking
• static linking
static linking
• calls to functions in library are resolved at compile
time.
• code from the library is copied into the resulting
binary.
dynamic
linking • calls to functions are resolved at
runtime.
• code for a library lives in it’s own
object:
• libblah.so.4
• libblah.dll
http://www.akkadia.org/drepper/no_static_linking.html
why?
почему?
static
linking • figure out which libraries your app needs
• pick a supported major release, if possible
• build and package this library
• link it statically against your binary during build
• you now have fewer stones to turn over when
debugging
static linking • you will need to track your upstream deps
• you will probably want to package your upstream
deps
• you can then point your chroots and build envs at
private deb, rpm, etc repos in your internal
infrastructure
• merging upstream changes in becomes its own
project
why?
почему?
Use files • src/
•redhat5/
•some_internal_thing.c
•ubuntu1204/
•some_internal_thing.c
•debian6/
•some_internal_thing.c
Use the build system
Determine which file
to build at compile
time.
Use
modules
• break up ifdef soup into separate files
• use the build system to compile the right file at
build time
• this seems obvious but many C libraries and
programs are full of crazy ifdef soup.
Use modules
• very easy to fall down a rabbit hole breaking things
apart
• can make the build process more complicated and
harder to debug
why?
почему?
Capture debug
symbols
• DEB and RPM can both output debug packages
that contain debug symbols.
• output these packages.
• store these and make backups.
• (or just don’t strip your binary)
Use google-
coredumper • you can use google-coredumper to catch
segfaults, bus errors, and other bad things.
• you can output a coredump when this
happens.
• you can use this coredump and your debug
symbols to figure out what is going on.
Plan for
failure • Have a backup plan
• Capture debug symbols during your automated
build process.
• Store them somewhere safe (and make
backups).
• Capture coredumps (if possible).
• Use coredumps and debug symbols to figure
out what happened.
Plan for
failure • can significantly increase complexity
• google coredumper can’t help if your kernel is
buggy
• some linux distributions don’t allow ptrace
• google coredumper only supports linux
Check things
like... • Is the binary actually statically linked?
• Does it get copied to the right path?
• Are the right config files autogenerated?
• Does the version string the program outputs match
the package version?
• ....
You also need... correctness
testing.
RSpec is useful for
this.
Automated
Testing • It will be impossible to build and test every
change on every supported platform.
• Use your build server to do this for you.
• Test things like:
• installing/uninstalling the object
• object is actually statically linked
• correctness testing (RSpec can be useful)
Automated
testing • Fine line between not enough and too much
• Windows can be painful, but cygwin can help with
scripting
• Easy to forget about testing the actual package
install/removal
• Can be difficult to get working with branch builds
To
summarize...
sassanosystems.c
om
@joedamato
спасиб
о
