View
661
Download
3
Category
Tags:
Preview:
DESCRIPTION
The talk introduces the need of live kernel patching. Further, it explains what is kGraft, how it works, what are its limitations, and our plans with the implementation in the future. The presentation includes also a live demo if stars constellation allows. Jiri Slaby, SUSE
Citation preview
kGraftLive patching of the Linux kernel
Jirí Kosina, Petr Mládek, Vojtech Pavlík, Jiri Slaby
SUSE Labs
September 25th 2014Paris, France
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 1 / 23
Why Live Patching?
1000 machines & severe security problemNeeds fixing now!
Rebooting the machinesIs not a quick way to fix an issueHas a risk of not coming up
Live patchingAllows quick responseLeaves an actual update to a scheduled downtime window
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 2 / 23
Where is Live Patching Useful?
Common tiers of change management1 Incident response – we are exploited2 Emergency change – we could be exploited (we are vulnerable)3 Scheduled change – time is not critical, we are safe
Live patching fits in with 1 and 2
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 3 / 23
Presentation Outline
1 KGRAFT
2 KGRAFT Internals
3 Live Demo
4 Conclusion
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 4 / 23
Section 1
KGRAFT
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 5 / 23
KGRAFT
Research projectLive patching technologyDeveloped by SUSE LabsSpecifically for the Linux kernelBased on modern Linux technologies
INT3/IPI-NMI self-modifying codeLazy update mechanismfentry-based NOP space allocationStandard kernel module loading/linking mechanisms
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 6 / 23
Advantages of KGRAFT
Does not require stopping the kernelEver!Not even for short time periodsUnlike competing technologies
Allows code review on KGRAFT patch sourcesPatches can be built from C source directlyNo need for object code manipulation
Only an alternative: object code based automated patch generation
kGraft is leanSmall amount of codeLeveraging other Linux technologiesNo complex instruction en/decoders or such
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 7 / 23
How does KGRAFT work?
A kGraft patch is a .ko kernel moduleThe .ko is inserted into the kernel using insmod
All linking (incl. the fix) is done by kernelKGRAFT replaces whole functions in the kernel
Even while those functions may be executed
An updated KGRAFT module can replace an existing patch
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 8 / 23
Limitations
kGraft is designed for fixing critical bugsPrimarily for simple changes
Changes in kernel data structure layout require special careDepending on the size of the change, reboot may be neededSame as with other live patching techniques
KGRAFT depends on a stable build environmentHaving history of built kernelsBest suited for
Linux distributionsTheir customersAnyone who builds their own kernels
Not good for 3rd party support
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 9 / 23
Section 2
KGRAFT Internals
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 10 / 23
KGRAFT and fentry
KGRAFT needs some space at the start of a functionTo insert a jump to a patched function
The space can be provided by GCC profiling-pg -mfentryKGRAFT uses this
fentry call instructionsPatched out at bootReplaced with 5-byte NOPs
kernel_func
CALL fentry
kernel_func
5-byte NOP
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 11 / 23
Using 5-byte NOPs Space
KGRAFT uses the ftrace infrastructure to perform patchingINT3 handler is installed with a JMP to the destination address
1 First byte of NOP is replaced by INT32 Remaining bytes are replaced by address3 First byte is replaced by JMP4 NMI IPIs are used to flush instruction decoders on other CPUs
kernel_func
5-byte NOP
kernel_func
INT3 | xxxx
INT3 handler
JMP addr
kernel_func
INT3 | addr
kernel_func
JMP addr
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 12 / 23
Patching a Function
Patching during runtime, no stop_kernel();Callers are never patched
Rather, callee’s NOPs are replaced by a JMP to the new functionJMP remains forever
But this takes care of function pointers, including in structuresLike indirect calls (handler->function())
Does not require saving any old data in case we want to un-patch
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 13 / 23
Patching a Function in Pictures
kernel_func
buggy_func();
buggy_func
JMP fixed
fixed_func
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 14 / 23
Issue: Non-consistency
What happens whenReplaced function changes semantics and subsequent calls relyon each other?It is called recursively?
kernel_func
buggy_func();
buggy_func();
BOOM!
buggy_func
fixed_func
KGRAFT patch comes in
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 15 / 23
Cure: RCU-like Replacement
We need to provide a consistent “world-view” to each threadUser processesKernel processesInterrupts
Solution: “reality check” trampolinePer-thread flag set on each kernel entry/exit
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 16 / 23
RCU-like Replacement
kernel_func
heavy work
buggy_func();
reality_check
which universeare you
coming from?
buggy_func
fixed_func
Userspace
Kernelspace
Welcome tothe new universe!
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 17 / 23
Lazy Replacement
All processes must wake up or execute a syscallSometimes this requires a signal to be sent (like for getty’s)
Once all processes have the "new universe" flag setPatching is completeTrampolines can be removed
Files to check/proc/<pid>/kgr_in_progress/sys/kernel/kgraft/kgr_in_progress
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 18 / 23
Lazy Replacement
kernel_func
heavy work
buggy_func();
buggy_func
fixed_func
Userspace
Kernelspace
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 19 / 23
Get It
UpstreamingKGRAFT was submitted and reviewed upstream
There are other groups working on competing technologiesKPATCH, KSPLICE, criu-based aproach, . . .SUSE will work together with themExpectations: common standard kernel live patching
PublishingPart of SLE12 kernel treeGIT repository upstream
http://git.kernel.org/pub/scm/linux/kernel/git/jirislaby/kgraft.git
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 20 / 23
Maintenance
KGRAFT patch is an RPM packageOnce installed, always protected
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 21 / 23
Live Demo
Kernel with a security vulnerabilityExploit programkGraft patch
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 22 / 23
Conclusion
SUSE providesDemanded live Linux kernel patchingDubbed KGRAFT
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 23 / 23
Recommended