Using containers and Continuous Packaging to Build native FOSSology packages

© 2017 HPE, Siemens - CC-BY-SA 4.0Open Source Summit Europe 2017

Talk Title HereAuthor Name, Company

Using Containers and Continuous Packaging to Build Native Fossology PackagesSpeakersBruno Cornec ([email protected]), Michael C. Jaeger ([email protected])


Overview: Contents

1. Introduction FOSSologyWhat is FOSSology

2. MotivationWhat FOSSology needs

3. Introduction Project BuilderThe ProjectBuilder Project

4. Build Native Fossology PackagesTo get container running in the continuous build

5. ConclusionWhere to see it

2 Page 2


Introduction FOSSology


The Problem Actually

Distributing open source software requires to

∙ Provide licenses of involved software

∙ Provide copyright statements of involved authors

∙ Provide disclaimers

∙ … and much more

You know these examples

Page 4


It is about finding licenses

∙ License texts

∙ References to licenses

∙ Written texts explaining licensing

∙ License relevant statements

Finding Licenses

Page 5


What is FOSSology?

A Web server application for license and copyright compliance of software components.

FOSSology Projecthttps://www.fossology.org/

∙ Published first in 2008, GPL-2.0

∙ 2015: Linux Foundation collaboration project

∙ Web server based and command line interfaces

∙ Scanning agents searching for license and copyright relevant hits (and more …)

∙ A multi-user / multi-tenant Web UI for review organizing clearing job

FOSSology Developmenthttps://www.github.com/fossology/fossology

▪ Standard Web application stack:▪ Linux, Apache 2, PostgreSQL, PHP,

▪ Web-based UI in PHP, but scannerswritten in C / C++

▪ Two ways to interact: ▪ Web user interface▪ Command line utilities

Page 6


How does FOSSology work?

See more details the Basic Workflow Description: https://www.fossology.org/get-started/basic-workflow

▪ Upload an open source package to the server▪ Select scan agents that analyze the software

▪ Review what scanners have found▪ Review license occurrences and correct findings if necessary

▪ Generate report output▪ For example list of licenses or SPDX

Upload OSS Package

Review and Adjust (“Clearing”)

Generate

Page 7

https://www.fossology.org/get-started/basic-workflow


What is the point of FOSSology?

See more details the Basic Workflow Description: https://www.fossology.org/get-started/basic-workflow

▪ Upload an open source package to the server▪ Select scan agents that analyze the software

▪ Review what scanners have found▪ Review license occurrences and correct findings if necessary

▪ Generate report output▪ For example list of licenses or SPDX

Upload OSS Package

Review and Adjust (“Clearing”)

Generate

Page 8

https://www.fossology.org/get-started/basic-workflow

Page 9© 2016-2017 Siemens AG, Linux Foundation - CC-BY-SA 4.0Open Source Summit Europe 2017

Using FOSSology with this Example

∙ It is natural that an OSS project reuses available https://github.com/fossology/fossology

∙ Likely OSS from other projects is found

∙ For example, FOSSology will find 25 other licensing relevant text occurrences in Apache thrift

Open Source and Reuse

9 Page 9

Page 10© 2016-2017 Siemens AG, Linux Foundation - CC-BY-SA 4.0Open Source Summit Europe 2017


Motivation


The Problem Actually

∙ ~ “creating binaries for linux is difficult” (starts at 5:40)

∙ https://www.youtube.com/watch?v=qHGTs1NSB1s

∙ Many linux distros with own package universe

∙ Different distros and different versions of these

∙ E.g. Packages dependencies on debian 8 change with debian 9

∙ Even within Debian 8 postgresql changes ...

See Linus Torvalds

Page 12


FOSSology Demand

∙ Debian, Ubuntu, CentOS and Fedora

∙ To efficiently build packages for these

∙ = efficiently means not to have manual step for each distro

∙ also means dealing with specificities of each distro/version(dependencies, availability of packages, …)

Support (at least) a basic set of Linux Distros

Page 13


Technically

∙ Different Distros required (and their versions)

∙ Integration in the CI

∙ State-of-the-art: Docker

∙ Support of two main package building formats: RPM and Deb

It is about building Linux packages

Page 14


Introduction to

Project-Builder.org


Project-Builder.org goal

“Make upstream projects life easier with regards to packaging their software”

Page 16


Project-Builder.org goal

“Make mylife easier with regards to packaging my software”

Page 17


Benefits from Continuous Packaging

● Packaging should be a project concern as well as coding, testing, installing, .... especially for smaller projects

● Packaging as your only way of delivery (not a dream)● Minimal overhead, maximum benefit:● Consistancy and reproduceability for devs and users● Distribution & deployment server integration,● Consistency with distribution and avoids dependecy hell for consumers● Packaging as a marketing activity for the upstream project. Easy way to extend

your user base, and improve your community relationship and is a “competitive advantage”.

● New mantra: “Package early, package always”● THE SOLUTION IS INDEED CONTINUOUS PACKAGING (whatever the tool)

Page 18


Project-Builder.org goals

● VCS agnostic: no VCS but guys it's 21st century now, SVN, CVS, Mercurial, GIT and GIT/SVN, SVK....

● OS agnostic: Linux: RPM, deb, ebuild, slack based, ... 150+ distro tuples made and counting – repositories for yum, urpmi, apt. Solaris pkg.

● Build environment agnostic: local, VM (QEMU, KVM...), VE (Docker, chroot, rpmbootstrap, rinse, mock, debootstrap...), RM (build farm)

● No project impact: preserves the md5sum of the delivered upstream sources. Can be completely external to the upstream project.

● Avoids duplication of code and metadata● THE SOLUTION IS INDEED CONTINUOUS PACKAGING (with project-builder.org !)

Page 19


Project-Builder.org architecture

Page 20


Build Native Fossology Packages

Demonstration !!


Problem encountered

● fossology build issues● project-builder.org bugs● composer phar !● build infrastructure● introduction in CI toolset

Page 22


Conclusion


Technology

Using containers and Continuous Packaging to Build native FOSSology packages