Project-Builder.org presentation

Bruno Cornec HP EMEA Open Source Profession Lead

Project Builder:A GPL continuous packaging solution

V2.0 April 2013

HP / OSSI / Bruno Cornec2

Introducing Myself● Software engineering and Unices since 1988

● Mostly Configuration Management Systems (CMS), Build systems, quality tools, on multiple commercial Unix systems

● Discover Open Source & Linux (OSL) & first contributions in 1993

● Full time on OSL since 1995, first as HP reseller then @HP

● Currently:● Master Technology Architect on OSL for the HP/Intel Solution Center, Grenoble

● OSL HP Advocate

● EMEA OSL HP Profession Lead

● Solutions Linux Conference and OWF board member

● MondoRescue, Dploy.org, Project-Builder.org project lead

● LinuxCOE, mrepo, tellico, rinse, fossology, collectl contributor

● FOSSBazaar and OSL Governance enthusiast

● Mandriva, Mageia, Fedora packager


What is the problem ?

Users / Sysadmins environment:

< 1%

< 10%

>= 90%

Users: easyness (GUI, CLI)Admins: distribution compliance, smooth integration – reduce admin load


Project

Flat Files

CMS/VCSCVSSVNGIT, ...


Developers environment:

autotools ?Install proc ?Soft. Eng. ?Fear of distro ?

Sourceforgeftp, web


Package

Flat Files

CVSSVNGIT, ...


Packagers environment:

rpmlint ?lintian ?

Distroftp, web

Does s/he only exists ?On time ? Lag behind ?Distro coverage (ver, #)Beta avail ?

CMS/VCS


I'm a sysadmin so I want to deal with packages only (deployment server integration)

I'm a developer so I want to ease my tests without screwing up my system and testing as a std user/admin

I'm a software engineer and want no duplication of code nor information (metadata)

I'm a a packager so I want to streamline that work for new projects I'm interested in packaging

I'm the marketing department of my own project and use packages as a “competitive advantage”

Founding no tool existing to help me doing all that (2005), so I started writing one to scratch my own itch

My answer with my various hats


Packaging should be a project concern as well as coding, testing, installing, .... especially for smaller projects

New mantra: “Package early, package always”

Use packaging as your only way of delivery (not a dream)Overhead minimal (not much longer than providing a tgz), maximum benefit (consistancy, reproduceability, team work improvement, distribution integration, improved deployment)

Packaging is also a marketing activity based on a technical content. It's an easy way to extend your user base, and improve your community. Makes tests easy.

THE SOLUTION IS CONTINUOUS PACKAGING

What is the solution ?


Project

Build +

metadata

Cloud based Continuous Packaging

VM or VE Build

Local build

RM Build Farm (may host VMs) Remote build

RepositoryServer

Local Build Server (may host VMs and/or VEs)

Packagers

Developers


Project

Build +

metadata

Provisioning the Continuous Packaging Cloud

VM or VE Build

Local build

RM Build Farm (may host VMs)

Remote build

RepositoryServer

Local Build Server (may host VMs and/or VEs)


Project-builder main goal is to help you package continuously being agnostic:

CMS agnostic: no CMS but guys it's 21st century now, SVN, CVS, Mercurial, GIT, SVK....

OS agnostic: Linux: RPM, deb, ebuild, slack based, ... 120+ distro tuples at the moment – repositories for yum, urpmi, apt. Solaris pkg. HP-UX sd in progress

Build environment agnostic: local, VM (QEMU, KVM...), VE (rpmbootstrap, rinse, mock, debootstrap...), RM (external build farm)

No project impact: preserves the md5sum of the delivered upstream sources. Can be completely external to the upstream project.

Goals


Project-builder provides additional goodies:

Easy VMs/VEs/RMs management

Macro system with perl variables to avoid duplication

Skeleton generation to help starting

Manages delivery up to your repository (ssh based)

Manages announces on mailing lists

Integration of tests in the process

Manages patches/additional sources when not upstream

Checks validity of packages built (lintian, rpmlint)

Easy creation of new versions for upstream management

Goodies


VM, VE & RM ManagementVE Creation:

debootstrap for .deb based ones.

rpmbootstrap (modeled after debootstrap): create chroot for multiple RPM based distributions (Mageia, Fedora, Mandriva, CentOS, OpenSuSE)

VM|RM Creation:

KVM or QEMU iso install possibilities.

LinuxCOE/OpenStack as provisioning tool for your VM|RM

VM|VE|RM setup: adds pb tools and build account

VM|VE Snapshot feature

Building in VM|VE|RM == building locally (build2vm|ve|rm vs build2pkg).

ssh communication for VM/RM


Metadata Management

Packaging process independent of the project and external

Metadata isolated - No duplication


./rpmbootstrap/pbfilter/all.pbf:filter PBSUMMARY = rpmbootstrap is a tool similar to debootstrap for RPM based distributions

./rpmbootstrap/rpm/rpmbootstrap.spec:Summary: PBSUMMARY

./rpmbootstrap/deb/control:Description: PBSUMMARY

./rpmbootstrap/pkg/pkginfo:NAME="PBSUMMARY"

Macro example:

Instantiation possible from distro-ver-arch, distro-ver, distro, distro-family, distro-type, os

Similar approach for all other parameters (Cf: man pb.conf)

Configuration file usage


[cms|sbx]2build: Create tar files for the project under your CMS. Parameters are packages to build if not using default list

build2pkg: Create packages for your running distribution

build2ssh: Send the tar files to a SSH host

pkg2ssh: Send the packages built to a SSH host

build2vm|ve|rm: Create packages in VMs/VEs/RMs, launching them if needed and send those packages to a SSH host once built

launchvm: Launch one virtual machine

script2vm|ve|rm: Launch a VM/VE/RM if needed and executes a script on it

test2vm|ve|rm: Test inside a VM/VE/RM

newvm|ve: Create a new VM/VE/RM

setupvm|ve|rm: Setup a VM/VE/RM for pb usage

newver: Create a new version of the project derived from the current one

newproj: Create a new project and a template set of configuration files under pbconf

Commands


Project

Packaging

cms|sbx2build

build2vm|ve|rm

build2pkg

newvm|ve / setupvm|ve|rmnewver / newproj

The Big picture

build2sshDevelopers

Packagers

PackagesBuilding

pkg2ssh


Multi platform life cycle

User Requirements

FunctionalSpecification

DesignSpecification

Coding

Unit tests

Integration tests

Validation


HP+Intel Solution Brief


Packages

Multi platform packaging cycle

Developers

Packagers

VMs/VEs/RMs

VM/VE/RM

UUWL


Q & A up to now ?


DEMONSTRATION


LSB chroot support. (or not)

LXC support

Support for libvirt, virsh, openstack API

Other CMS (Bazaar, …) only when/if needed

Other VMs (VMWare, Xen, ...) only when/if needed

Multiple delivery means

Look at interactions with Buildbot

Add non-interactive signature support for .deb

Config-Model for configuration file management

REST API and Web Interface

To be done


Learn Project-Builder.org

Start with the Lab (63 pages)

Use man (pb, pb.conf + 8 ProjectBuilder::* man pages)

Use the mailing-list pb-announce and pb-devel at http://www.mondorescue.org/sympa

Use examples from http://trac.project-builder.org/browser/projects/


Project-Builder Web site / Trac / Wiki:

● http://www.project-builder.org

● http://trac.project-builder.org

Projects using project-builder.org:

● http://www.project-builder.org (of course :-)

● http://www.mondorescue.org

● Http://www.linuxcoe.org

● http://www.fossology.org

● http://kde-apps.org/content/show.php/Meganizer?content=129907

Web Resources


Project-Builder is mostly suited for upstream projects wanting to package their applications

Distributions provide each their build tools● SuSE: Open Build Service (Multi distro, Web based, BaaS)● Fedora (Koji)● Mandriva/Mageia (Youri, mdvsys, mgarepo...)

Other complementary tools:● Buildbot● KVM/QEMU● rpmbootstrap, rinse, mock● Parallel::ForkManager, DBI, DB::SQLite, File::MimeInfo, Mail::Sendmail

Project-Builder.org is one of the tool of the vcs-pkg.org initiative.

Related tools


”Changes are never easy to make. There is comfort and safety in tradition, but change must come, no matter how painful or expensive it may be.”

Bill Hewlett

Contact

ThanksOpen Source at HP

http://opensource.hp.com

[email protected]

(Linux Solution Consultant in the HP/Intel Solution Center)

http://www.hp.com/linux

Linus Torvalds, Richard Stallman, Eric Raymond, Nat Makarevitch, René

Cougnenc, Eric Dumas, Rémy Card, Bdale Garbee, Bryan Gartner, Gallig Renaud,

Joachim Langenbach among others, for their work and devotion to the Open

Source Software cause... and my family for his patience :-)