Embed Size (px)
Citation preview
Two Years of Work ofPaid Contributors in the
Debian Long Term Support Project
By Raphaël Hertzog <[email protected]>
DebConf 16 / Cape Town / 2016-07-08
Plan of the talk
● Presentation of the LTS project/team● Workflow of the team: how to contribute● Statistics about the team● Changes since last year● How do we (try to) avoid money-related
problems● Questions
● Feel free to ask questions at any time
Presentation of the LTS project
What is LTS about?
What were the challenges?
Choices made: at the technical level, at the organizational level
What is LTS about ?
● Providing 5 years of security support
● Thus allowing users to skip a release
Initial challenges
● Keeping a distribution secure for 5 years is hard work that is not very rewarding
● The security team● has limited resources● aims to support all Debian packages on all
release architectures
Technical choices:restrict the perimeter
● Restrict architecture support to amd64, i386, armel and armhf (two more arches than squeeze).
● Exclude some “problematic” packages from security support (much less than squeeze so far):● chromium-browser, openstack, iceape, libv8, mantis,
mediawiki, movabletype-opensource, openjdk-6 (7.x is supported), openswan, redmine, rails 2.x (3.x is supported), sogo, swift, tomcat6 (end of 2016), typo3-src, virtualbox, vlc
● http://anonscm.debian.org/cgit/collab-maint/debian-security-support.git/tree/security-support-ended.deb7
Organizational choice #1:creation of a new team
● Security team ≠ Debian LTS team● But members of the security team helped to
bootstrap the LTS team
● Different policies● Different infrastructure
● Mailing list : [email protected]://lists.debian.org/debian-lts/
● IRC channel: #debian-lts on irc.debian.org (OFTC)
Organizational choice #2:seeking help of companies
● Try to pool the work of companies which were doing in-house long term security support already➔ Press release to invite companies to join
● Let other organizations fund the project so that Debian contributors can be paid to do the work➔ https://wiki.debian.org/LTS/Funding lists all ways to
help with money● In practice, most of the (wanting to be) paid
contributors joined forces behind a single offer managed by Freexian SARL : https://www.freexian.com/services/debian-lts.html
Freexian's intermediary role
Workflow of the team
Triage of security issues
Preparation of security update
Test of security update
Upload and announce of update
Triage of security issues
● Done in the security tracker (common to Debian Security and Debian LTS)https://security-tracker.debian.org/ http://security-team.debian.org/security_tracker.html
1.New issues added to data/CVE/list
2.Issues dispatched on source packages
3.Issues reviewed for each release
4.Classification according to analysis
Ways to classify security issues
● Depending on analysis:➔ Package added to data/dla-needed.txt so that
someone will take care of preparing the update (currently <unfixed>)
➔ Issue does not apply (<not-affected>)➔ Issue ignored because package is not supported
(<end-of-life>)➔ Issue not important enough (<no-dsa>)➔ Issue already fixed in a former version
● Keep the maintainers in the loop, they can always fix issues (even the non-important ones)
Extract of data/CVE/list
CVE-2015-2317 (The utils.http.is_safe_url function in Django…) {DSA-3204-1} - python-django 1.7.7-1 (bug #780873) [squeeze] - python-django <no-dsa> (Minor issue, can wait next security upload) NOTE: https://github.com/django/django/commit/… (1.4.x)
CVE-2015-2189 (Off-by-one error in the pcapng_read…) {DSA-3210-1} - wireshark 1.12.1+g01b65bf-4 (bug #780372) [squeeze] - wireshark <not-affected> (Vulnerable code not present) NOTE: https://bugs.wireshark.org/bugzilla/…
CVE-2014-9701 [XSS issue in MantisBT permalink_page.php] - mantis <removed> (bug #780875) [wheezy] - mantis <no-dsa> (Minor issue) [squeeze] - mantis <end-of-life> (Unsupported in squeeze-lts) NOTE: Fixed by https://github.com/mantisbt/… (1.2.x)
Preparation of the security update
● Find a patch● Backport it if required● Prepare an upload with a “+deb7uX” suffix,
applying the patch as appropriate● Document fixed CVE
in the changelog andin patch headers
Test the update and upload
● Build and test the result to ensure that● the package still works● the fix works as expected● there's no obvious regression
● If unsure of your update, get in touch:● Ask others to test● Seek reviews of your debdiff
● If everything is ok, upload to wheezy-security
Announce the security update
● Prepare a “DLA” (Debian LTS Advisory)$ ./bin/gen-DLA --save expat CVE-2012-6702 CVE-2016-5300Enter wheezy's version [unset]: 2.1.0-1+deb7u4DLA text written to ./DLA-508-1$ svn commit
● Send it to [email protected]$ mutt -H DLA-508-1
● This process updates data/DLA/list which is used by the security tracker to know the CVE fixed by the update
Statistics about the team
Who uploaded packages?
How did it evolve since the beginning?
How is the funding evolving?
Data between 2014-06-01 and 2015-07-31
Stats: 549 LTS uploads
● By affiliation:● Freexian: 380● None (maintainers): 88● Security team: 32● EDF: 14● credativ: 12● Individuals: 11● Toshiba: 7● Univention: 4● Catalyst: 1
● By contributor:● Thorsten Alteholz: 125 (in 24 months)● Santiago Ruano Rincon: 51 (13 months)● Raphaël Hertzog: 40 (16 months)● Chris Lamb: 29 (8 months)● Ben Hutchings: 28 (15 months)● Holger Levsen: 28 (8 months)● Markus Koschany: 25 (8 months)● Mike Gabriel: 23 (11 months)● Thijs Kinkhorst: 17 (9 months)● Guido Günther: 14 (11 months)● Kurt Roeck: 13 (12 months)● Raphaël Geissert: 13 (6 months)● Scott Kitterman: 12 (8 months)● Christoph Berg: 9 (8 months)● …
LTS uploads over time
2014-062014-07
2014-082014-09
2014-102014-11
2014-122015-01
2015-022015-03
2015-042015-05
2015-062015-07
2015-082015-09
2015-102015-11
2015-122016-01
2016-022016-03
2016-042016-05
0
10
20
30
40
50
60
Debian LTS uploads
UniventionToshibaNoneFreexianEDFDebian SecurityDebian LTScredativCatalyst
Nu
mb
er
of u
plo
ad
s
Statistics about sponsored hoursmanaged by Freexian
● Hours sponsored● 135 h/month currently
dispatched to 11 contributors● 1854h since the start (740h
already paid to be dispatched over the next year)
● Sponsors: 46● Platinum (>= 24h/month): 1● Gold (>= 8 h/month): 5● Silver (>= 4 h/month): 10● Bronze (>= 1h/month): 22● Iron (< 1 h/month): 8
● Average: 2.94 h/month/sponsor
2014-072014-08
2014-092014-10
2014-112014-12
2015-012015-02
2015-032015-04
2015-052015-06
2015-072015-08
2015-092015-10
2015-112015-12
2016-012016-02
2016-032016-04
2016-052016-06
0
20
40
60
80
100
120
140
160
0
2
4
6
8
10
12
Hours sponsored and number of paid contributors (by month)
Hours Nb Paid Contributors
Ho
urs
Nu
mb
er
of p
aid
co
ntr
ibu
tors
Changes since last year
Switch from Squeeze to Wheezy LTS
New architectures
Working with external partners to support some packages without upstream support
Switch from Squeeze LTSto Wheezy LTS
● No wheezy-lts repository● We keep using wheezy-security● No changes for the user● Changes for the contributors
● More packages supported● Xen, qemu/qemu-kvm, firefox, icedove, libav,
libvirt, zabbix, …● Made possible by larger amount of sponsorship
New architectures in Wheezy LTS
● armel and armhf● Requested by new Japanese sponsor:
● Accepted by ftpmasters and buildd maintainers in the last days before the start of Wheezy LTS
Working with external partners
● To support important packages that do not benefit from upstream support (for the version we use in Wheezy)
● External partners: upstream developers that can be contracted, or consultants/companies with expertise on that specific software
● Two such cases currently:● Xen with credativ (Bastian Blank so far)● Libav with Diego Biurrun
How do we (try to) avoidmoney related problems
● Transparency● External● Internal
● Open rules to join the set of paid contributors● Hours allocation rules● Communication rules● Point of contact for complaints
External transparency withpublic monthly reports
● From Freexian:● How many hours were assigned to contributors● Links to their respective reports● Some high level analysis on what happened● List of sponsors● Syndicated on Planet Debian
● From paid contributors:● How many hours they worked and what they did● On their blog or on the debian-lts mailing list
Internal transparency with legder for hours allocation (1/3)
● Payments from sponsors transformed in work hours assigned to future months (split over all months from the payment period):
2016-06-13 Invoice 201606-063 (Offensive Security) Funded Available:2016:07 2h Available:2016:08 2h Available:2016:09 2h [...]
Internal transparency with legder for hours allocation (2/3)
● Work hours dispatched to contributors, once per month (respecting the limit they set for themselves):
2016-06-01 Distribute work hours for June 2016 Available:2016:06 Contributors:Available:AntoineBeaupre 4.00h Contributors:Available:BalintReczey 16.00h Contributors:Available:BenHutchings 15.00h Contributors:Available:BrianMay 15.00h Contributors:Available:ChrisLamb 18.00h Contributors:Available:EmilioPozueloMonfort 16.00h Contributors:Available:GuidoGuenther 8.00h Contributors:Available:MarkusKoschany 18.75h Contributors:Available:OlaLundqvist 10.00h Contributors:Available:SantiagoRuanoRincon 18.75h Contributors:Available:ThorstenAlteholz 18.75h
Internal transparency with legder for hours allocation (3/3)
● Contributors report back work hours completed over last month
2016-06-07 May hours of Thorsten Alteholz ; Report: http://blog.alteholz.eu/2016/06/my-debian-activities-in-may-2016/ Contributors:Available:ThorstenAlteholz Contributors:Invoiced:ThorstenAlteholz 31.00h2016-06-10 May hours of Guido Guenther ; Report: http://honk.sigxcpu.org/con/Debian_Fun_in_May_2016.html Contributors:Available:GuidoGuenther Contributors:Invoiced:GuidoGuenther 17.25h
Open rules to jointhe set of paid contributors
● Documented on Freexian's website (from the start), no arbitrary selection:● Debian Developer or Debian Maintainer● Prior experience with security updates● Good programming skills (multiple languages)● Can emit invoices to Freexian● Accept the rules
– Privacy of customer data– Public monthly report– Debian code of conduct, obligation to respond to queries– Best effort to meet high quality standards of security team
Hours allocation rules (1/2)
● Available hours split evenly across all contributors (but each contributor can define a maximal number of hours that he wants to be assigned, hence the differences)
● If we have too many active contributors (never happened so far), then we would organize a rotation so that we don't assign less than 8 hours per contributor.
Hours allocation rules (2/2)
● Freexian recruited contributors regularly:● No contributor dependant solely on paid LTS
work for their life● More redundancy (if contributor goes missing)
0
2
4
6
8
10
12
14
16
18
Average number of paid hours per contributor
Communication rules
● Contributors have to respect the Debian Code of Conduct
● Always offer the current package maintainer to handle LTS updates● Temptation to do everything immediately (in particular
when it's easy) because the contributor wants to spend his work hours
● Obligation to respond to queries from other Debian contributors● Lack of communication is hardly acceptable for
volunteers, it's clearly intolerable for paid contributors
Point of contact for complaints
● Clear objective of high-quality work:● The website invites everybody to raise their concerns when it's
not the case.
● I am the point of contact currently:● Raphaël Hertzog [email protected]
● No official complaints so far.● But I complain privately (and politely) to some of the paid
contributors from time to time.● About the quality of the work, of the report, about the fact that
they did not use their assigned work hours, etc.● Some have stepped down when they realized that they did not
deliver what was expected.
Lessons learned
● It's possible to pay Debian contributors without disrupting the entire community
● Care must be taken at many levels:● To work transparently and in an inclusive way● To avoid someone getting locked in a paid position● To have fair criteria to use the money or at least a
fair chance of being paid
● You must be aware that it will have consequences ● Change of priorities for some volunteers
Questions ?Come to the BoF at 14:00
Using Debian Money to Fund Debian Projects
Credits & License
● Content by Raphaël Hertzoghttp://raphaelhertzog.comLicense: GPL-2+
● Cliparts from https://openclipart.orgLicense: Public domain
● OpenOffice.org template by Raphaël Hertzoghttp://raphaelhertzog.com/go/ooo-templateLicense: GPL-2+
● Background image by Alexis Younes “ayo”http://www.73lab.comLicense: GPL-2+
