Working with the Linux Community - University of Texas at ... · Working with the Linux community ... API should be simple ... APIs with future official versions of the kernel and

Linux TechnologyCenter

Working with the Linux Community

A Guest Lecture for CS 378 classat the University of Texas at Austin

Khoa Huynh, Ph.D.IBM Linux Technology Center

Austin, TX


TopicsTopicsOpen source software development

Open source licences

Common risks

Working with the Linux communityGeneral advices & tips

Submitting your work to Linux kernel communitySource code control

Bug tracking

Submitting your work to Linux distributors

Linux debug tools


Freely available software in source code formatOften distributed via Internet or CD (e.g., Red Hat)

Redistribution cannot be prohibited or royalty imposed

Often developed in a collaborative community effort

OSS community very sensitive about meaning & use of term "open source"

For the current definition, see: http://www.opensource.org/osd/

Not "Public Domain" Copyright law gives author rights

author determines appropriate uses of a work

e.g. you can't reproduce or modify a work without author's permission

author may grant permission via a license which can impose obligations

Open Source Software (OSS)Open Source Software (OSS)


There are many open source licensese.g. GPL, LGPL, CPL, IPL, BSD, Apache, etc.

Some common featuresGoverns use, modification and distribution of code

Grant of rights (e.g. modification, redistribution, etc.)

Obligations (e.g. attribution, redistribution, etc.)

Different licenses have different objectivesGPL = "General Public License"

LGPL = "Lesser/Library Public License"

CPL/IPL = "Common/IBM Public License"

BSD = "Berkeley Software Delivery"

Others

Some licenses make source code available but should not be confused with OSS licenses (e.g. Sun Community Source License)

Open Source LicensesOpen Source Licenses


Open Source Licenses (cont'd)Open Source Licenses (cont'd)Many projects are under GPL (e.g., Linux kernel, GNU tools, Gnome)

GPL often referred to as "Copyleft"

Author: Free Software Foundation

Software available freely

Free availability of all source code and modifications

No restriction on what others may do with the code

Common risksNo warranty (provided "as is")

Viral effect (or contamination) between proprietary and open source softwareIntermingling of objects

Cutting & pasting source code

Linking to libraries (static or dynamic)

Possible relinquishing of patent rights

More info on OS licenseshttp://www.opensource.org/licenses/


Working with Linux CommunityWorking with Linux CommunitySubmit your work to the community

Match the patch to the code

Use multiple patches

Submit early, submit often

Stakeholders and Decision Makers


Match Patch to the CodeMatch Patch to the CodeIf the patch is to existing code, make sure the patch matches the surrounding code

From a style perspectivematch indentation styles

match function/variable naming

From a design perspectiveuse a similar functional breakdown.

your code should be at least as portable as the surrounding code

make sure your changes don't interfere with the maintainer/author's long-term vision for the code


Use Multiple PatchesUse Multiple PatchesSmall patches are accepted more easily than big patches

Each patch should make a single functional changeThat way the maintainer can accept a subset of your patches, if he/she wishes

Patches can be submitted as a set or separately

Document what they do & pre-requisites


Submit Early, Submit OftenSubmit Early, Submit OftenIn general, there is no penalty if you submit a patch and it is rejected

The rejection often contains good constructive criticism

Even if the patch is not ready to be merged into the mainline, post the patch for comments

Open source developmentImprovements suggested by the community

Extra testing by the community


Submission CriteriaSubmission CriteriaAs soon as minimal functionality is present

Doesn't have to be feature-complete

Doesn't have to be tuned for performance

Passes the smoke test --- obvious bugs have been fixed

Overall design completed (recommended)


Stakeholders and Decision MakersStakeholders and Decision MakersIdentify who the stake holders and who the decision makers are

They are not necessarily the same!

Consult with the stake holders

Satisfy the decision maker

Avoid falling into the trap of spending too much time trying to achieve 100% approval from stake holders

For the Linux kernel:Stake holders: LKML, individual developers, subsystem maintainers, etc.

Decision maker: Linus Torvalds


Submitting to the Linux KernelSubmitting to the Linux Kernel

Key considerationsSource control: BitKeeper

Bug tracking: Kernel Bug Tracker (new)

Trivial Patch Monkey (for small patches)

Types of kernel patches


Source Control: BitKeeperSource Control: BitKeeperPreferred form for large submissions to Linus

Send e-mail to Linus with URL to BK tree, and description of the change

Make sure the change sets (patches) will apply cleanly to the current BK tree

BitKeeperhttp://www.bitkeeper.com

Scalable through replication

Change sets == Patches

Powerful merging tools

Repositories as branches allows controlled development Each person has his/her own personal repositories

Repositories are compressed to reduce disk space and increase performance

Multi-protocol (FS/RSH/SSH/HTTP/BKD/SMTP) connectivity


Using BitKeeperUsing BitKeeperFile names

revisioned and propagated just like contents

Graphical interfacesprovided for merging, browsing, and creating changes

Change setsOne or more changes to one or more files (i.e., patches)

Built-in configuration managementCreation of a change set saves the entire state of the repository (both what changed and what didn't)

Open LoggingRequired for BitKeeper Open License (free use)

Not required for paying customers

Centralized change logs (everyone can see)

Trading privacy loss for money


Using Trivial Patch MonkeyUsing Trivial Patch MonkeyPreferred form for small patches

Rusty Russell reviews & bundles small patches into a large submission to the mainline kernel source tree

E-mail address of this patch system: [email protected]

More details on how to submit trivial patches to this system

http://www.uwsg.iu.edu/hypermail/linux/kernel/0205.3/1019.html.


Types of Kernel PatchesTypes of Kernel PatchesNew subsystem

Infrastructural patches

Replacing / rewriting existing code


New SubsystemNew SubsystemExamples:

New filesystem (JFS, CIFS)

New subsystem (EVMS, SCTP)

New device driver

Most changes are in a self-contained file or directory

Minimal changes needed to the rest of the kernel (config.in, Configure.help, makefiles)

Small changes to kernel code are acceptable, but should be minimized


Submitting A New SubsystemSubmitting A New SubsystemSince it represents new functionality, Linus is much more forgiving

Stylistic guidelines not as important

If any infrastructure changes are required:Package them separately

Submit at the same time as the new subsystem

Create a Bitkeeper tree, and send e-mail to Linus with the Bitkeeper URL

Include description/diffs of code changes to the core kernel in the e-mail


Infrastructure PatchesInfrastructure PatchesExample: Read-Copy-Update

Patch should include at least one or two uses of the new interfaces

Should demonstrate why new interface is needed

API should be simpleUnderlying implementation can be complicated (as needed)

Community support is much more important for infrastructure patches

Because the community will be using the interface



Replacing / Rewriting Existing CodeReplacing / Rewriting Existing CodeAcceptance from existing maintainer is critical

Must provide noticeable improvements to community


Bug Tracking: Kernel Bug TrackerBug Tracking: Kernel Bug TrackerBug tracking system for Linux kernel community

http://bugme.osdl.org

Currently for 2.5 kernel bug tracking only (will likely extend for 2.6)

Introduced by IBM Linux Technology Center (Khoa Huynh, Martin Bligh, Jon Tollefson)

Why?Discussions, bugs, patches, announcements, etc., were all communicated via the Linux Kernel Mailing List (LKML)

BitMover's Bug Manager is still under development

Key characteristicsBased on Mozilla.org's Bugzilla bug tracking system

Web-based user interface

Accessible via many popular network protocols (HTTP, SMTP, etc.)

Automatic e-mail change notification

MySQL database technology

File attachments (testcases, patches, etc.)


Life of A Bug in Kernel Bug TrackerLife of A Bug in Kernel Bug Tracker

NEW

ASSIGNED

RESOLVED

VERIFIED

CLOSED

REJECTED

DEFERRED

Bug Owner Assigned

Fix (patch) Available

Fix (patch) NOT OK

Patch Reviewed & Accepted By KernelCommunity

Patch Included In An Official Kernel Release

Patch Already Available

Bug Reopened

Not A Valid BugDuplicateUnreproducibleDocumentedInsufficient DataWill Not Fix

Will Fix Later


Patches should be against current, official codebase of the project

Patches should not lead to potential binary incompatible APIs with future official versions of the kernel and libraries

Coding style should follow the coding style of the project

Quality of patches should be in such a way that the officiamaintainers could accept them

Tests are always welcome

Information on importance and relevance of the patchesIs this a show stopper?

Whom will the patch effect - positive and negative?

Are there any binary incompatible issues ?

Submitting Code to Linux DistributorsSubmitting Code to Linux Distributors


Linux Debug ToolsLinux Debug ToolsLinux RAS Package

IBM's Linux Event Logging

Opersys' Linux Trace Toolkit

IBM's Dynamic Probes

SGI's Linux Kernel Crash Dump (LKCD)

SGI's Kernel Debugger (KDB)

For Linux kernel, most useful (in my opinion):Kernel Crash Dump (LKCD) for system hangs (especially at customer sites)

Kernel Debugger (KDB) for kernel oops (crashes), panics, etc.


Linux Kernel Crash Dumps (LKCD)Linux Kernel Crash Dumps (LKCD)What is it?

Set of utilities and kernel patches that allow a crash dump to be captured

Tools that allow developers to analyze a crash dump

When to install?Must be installed before the failure occurs

A crash dump will be generated whenA kernel oops occurs

A kernel panic occurs

A system administrator types Alt-SysRq-c on the console

Crash dump processStage 1

Contents of system memory is copied to temporary disk location (dump device)

System is rebooted

Stage 2Linux is booted

Memory image is moved from dump device to /var/log/dump directory


LKCD (cont'd)LKCD (cont'd)Key files (in /var/log/dump)

dump.n (crash dump file)

kerntypes.n (info about kernel data structures, created during kernel build)

map.n (info about kernel symbols, a copy of System.map file)

lcrash.n (crash dump analyzer tool)

Key optionsNon-disruptive dumps

System is not rebooted after crash dump

Dedicated dump device must be used

Not available for kernel panics

Location of dump device Default is swap partition

Dedicated dump device is recommended (for faster recovery time)

Must be at least as large as physical memory

Location of /var/log/dump directory

More info: http://lkcd.sourceforge.net


Linux Kernel Debugger (KDB)Linux Kernel Debugger (KDB)What is it?

Set of kernel patches

A way to examine kernel memory & data structures "live"

When to installMust install patches and rebuild the kernel

Key funtionsDisplay essential system data structures given an identifier or address

Single-step a processor

Instruction disassembly

Set breakpointsUpon execution of a specific instruction

Upon access or modification of a specific virtual memory address

Upon access to a register in the I/O address space

Stack traceback for current active task or other tasks (via PID)

More info: http://oss.sgi.com/projects/kdb/


SummarySummaryOpen source software

Beware of different licenses

Avoid common risks (no warranty, contaminination, patent rights, etc.)

Working with the Linux communityMatch your patch to the surrounding code

Use multiple patches


Consult stake holders, satisfy decision maker

Working with the Linux kernel communityMain communications medium: Kernel Mailing List (LKML) & sub-lists

Source code control: BitKeeper

Bug tracking: Kernel Bug Tracker

Small patches --> Trivial Patch Monkey

Submitting patches to Linux distributors


SummarySummary (continued) (continued)Patches must be against current, official code base

Avoid potential binary incompatibilities with APIs in future versions of the kernel and libraries

Code quality must be high (must pass rigorous testing)

Follow coding standards of affected projects

Debug tools are introducedLKCD

KDB

Documents

Working with the Linux Community - University of Texas at ... · Working with the Linux community ... API should be simple ... APIs with future official versions of the kernel and