View
220
Download
0
Category
Preview:
Citation preview
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
SVN Pilot: CVS Replacement
Manuel Guijarro
Jonatan Hugo Hugosson
Artur Wiecek
David Horat
Jonathan Brugge
Michel Manent
September 2008
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
2
Outline
• Introduction• Motivation• Subversion• Objectives• Performance Tests• Security• Implementation• Questions
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Version Control Systems
• Maintain current and historical versions of files and data (source code)
• There are many commercial and Open Source VC Systems:– (Centralised) CVS/SVN
– (Distributed) GIT, Bazaar, Darcs, GNU arch, Mercurial, Monotone, etc
– But subversion seems to be the most popular one (used by GCC, Phyton, PuTTY, Apache, GNOME, KDE, etc)
• Physics User Community: (IN2P3, ROOT, Totem..)
3
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
CERN Central CVS Service
• Hosts over 330 Software Projects– 29 for Atlas– 46 for CMS– 8 for LHCb,…..
• Over 3000 developers registered• Over 90 GBytes of source code• Creates 250 Remedy tickets per year• Over 100000 commits per month
4
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
CERN Central CVS Service
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Central CVS service features
• High Availability and Load Balancing• Web interface to repositories• Usage Statistics• Repository Remote Replication + Mirroring• Daily archive of Repositories and DR• Developers Mailing list• Pre/Post Commit Actions (such us e-mail
notification, etc)• Various access method (ssh/kerberos)• Role split (CVS Admin/Librarian/Developer)
6
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Motivation for SVN Pilot
• Originally designed to host less than 100 projects
• Requests to provide a central SVN service:– From CMS– From ATLAS (case study in 2006)– And from many others
• CVS is over 20 years old while SVN is this millennium technology
• Requests for Read Access control
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
SVN vs. CVS
8
Feature SVN CVS
Speed Faster Slower
Permission Full Limited
File types All Limited
Off line operations Yes No
Repository format Database File system
Locks No Yes
Atomic commits Yes No
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
New Features (SVN 1.5)
• Automatic update of working copy• Merge tracking
– Subversion keeps track of what changes have been merged where
• Sparse checkouts• Interactive conflict resolution
9
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Pilot Objectives
• Provide current CVS service features• Add new features (available with SVN)
– Control Read access per path (module)– Authenticated Web access– Binary files handling
• Ease CVS to SVN migration• Improved usage statistics (SVN Stats)• Handling of first line support via the Help Desk• Delegate administrative tasks to Software
Librarians of each project• Prevent uncontrolled setup of SVN servers• Manpower: 1.2 FTE project
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Timetable
11
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
SVN Pilot study
• Access methods– https
– ssh
• Shared storage– NFS 3/4
– AFS
• Securing service– Restricted Shell
– Chrooted hooks (commit scripts)
• Infrastructure: – Librarian tools, Statistics, Web Interface,…
12
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Performance Tests
• SVN check out of a 110 Mb project• Parameters
– AFS/NFS3/NFS4– HTTPS/SSH
13
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
AFS vs NFS3 (1 server)
14
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
AFS vs NFS4 (1 server)
15
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
AFS vs NFS4 (3 servers)
16
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Preliminary Conclusions
• AFS much faster than NFS• SSH much faster than https• SSH scales very well with high load• … • New tests ongoing (with mixture of read and
write operations)
17
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Security
• Project Isolation• Windows/Linux clients • Worldwide access• Shared file system independent• Hooks executed on servers
• Librarians may put any script into the hooks
• Librarians might need file system level access to repository – being studied
18
Security risk!!
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Hooks (scripts)
Client Server
19
Svn commit
Pre-commit hook is executed
Post commit hookis executed
SVN: Commit OKEmail notificationrecieved
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Hook scripts chrooted:
Server
20
svnserer
hooks/post-commit hook
RepositoriesSystem files
Usr-hooks/post-commit hook
Librarian hooks: jailed
Repository (1)
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Architecture
• svn.cern.ch (rw)– Secured subversion
server (only ssh)– Read and write
access to repository
• svnweb.cern.ch (ro)– User documentation– Project request– SVN web interface– Usage statistics
21
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Pilot Implementation Summary
• SSH access for SVN clients– Restricted shell for all SVN clients– Hooks chrooted
• SVN web (ro)– Web interfaces: websvn, trac– SSO Authenticated access
• Administration delegated to librarian– Access rights– Hooks– Admin tools
22
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Conclusions
• Secure service• This will replace CVS by end of 2009• The service is supported (pre-production)• Pilot setup may differ from final setup
– Access method, Web interface, shared file system, etc.
– Changes will be transparent to the users
23
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Support
http://cern.ch/svn
–Try the pilot
–Documentation
Svn.support@cern.ch
24
CERN - IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
Questions?
Thanks For Listening….
M. Guijarro, A. Wiecek, David Horat, Jonathan Bugge, M. Manent, H. Hugosson
25
Recommended