Upload
toby-roberts
View
365
Download
3
Embed Size (px)
DESCRIPTION
Citation preview
Scaling Source Control for NextGen Game Development 1
2007 Perforce User’s Conference
Scaling Source Control for the Next Generation of Game Development
Toby RobertsMike Sundy
Scaling Source Control for NextGen Game Development 2
Overview• Current Gen vs. NextGen
• Test Methodology
• Test Results
• Lessons Learned
• Future of P4 at EA
Scaling Source Control for NextGen Game Development 3
Challenges in Game Industry• CurrentGen vs. NextGen
• 9 GB for DVD's for CurrentGen. (Xbox, PS2)
• 30-50 GB for NextGen (Blu-Ray, HD-DVD – XBox 360, PS3) - 3-5x increase in storage capacity
• Asset Volume Explosion
• 50k clientspec grew to 500k clientspec
• Build Farms
• 20-30 machines grew to 100+
• Team Sizes
• Outsourcing
Scaling Source Control for NextGen Game Development 4
EA Redwood Shores (EARS)P4 Installation Metrics
• 11 Perforce Servers, 1200 users (90+ servers, 4,000+ users at EA)
• 7 million files on one server, 80 GB db.have
• Store all source code and source data in P4
• 1.5 TB on one server, 7 TB total P4 storage
• Avg. client workspace – 80-120GB and 250k-500k files
• 2,500 – 66,000 syncs per day
• Titles include Sims, Godfather, Simpsons, and Spore.
Scaling Source Control for NextGen Game Development 5
Problems with Windows• 2 GB per process memory limit
• Filehandle leak (crashed up to 6 times per day)
• Performance during large concurrent syncs
• 40 GB sync went from 40 min. to 3+ hours
• NTFS
• Threads and troubleshooting
• Patching
Scaling Source Control for NextGen Game Development 6
Goals of Project
• Improve stability
• Improve scalability
• Solution that works for our dev environment, infrastructure, and team cultures
Scaling Source Control for NextGen Game Development 7
Parameters of Project• Dell and STK hardware
• Intel
• RedHat
• Windows Desktops
• 95% of user machines
• Gig network
Scaling Source Control for NextGen Game Development 8
Test Methodology
• Single operation vs. load
• P4 benchmarking tools
• fstst and locktest
• Homegrown scripts
• Force sync of 39 GB data, 180k files
• p4.exe on Windows
Scaling Source Control for NextGen Game Development 9
Configuration Overview
• Tested 21 different server configurations.
• Windows vs. Linux
• 32-bit vs. 64-bit
• NTFS v. ext3 v. ReiserFS v. XFS
• 10k vs. 15k drives
Scaling Source Control for NextGen Game Development 10
db locktest
0
1
2
3
4
5
6
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
config
Seco
nds
Scaling Source Control for NextGen Game Development 11
db 8k read-write
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
config
MB
\s
Scaling Source Control for NextGen Game Development 12
rcs 64k write
0
100
200
300
400
500
600
700
800
900
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
config
MB
\s
Scaling Source Control for NextGen Game Development 13
rcs 64k read
0
500
1000
1500
2000
2500
3000
3500
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
config
MB
\s
Scaling Source Control for NextGen Game Development 14
1 force sync avg.
0 5 10 15 20 25 30 35 40 45 50
1
3
5
7
9
11
13
15
17
19
21
conf
ig
minutes
Scaling Source Control for NextGen Game Development 15
3 force syncs avg.
0 10 20 30 40 50 60 70 80
1
3
5
7
9
11
13
15
17
19
21
conf
ig
minutes
Scaling Source Control for NextGen Game Development 16
5 force syncs avg.
0 10 20 30 40 50 60 70 80
1
3
5
7
9
11
13
15
17
19
21
conf
ig
minutes
Scaling Source Control for NextGen Game Development 17
% degradation from 1 to 3 syncs
0 0.1 0.2 0.3 0.4 0.5 0.6
1
3
5
7
9
11
13
15
17
19
21
conf
ig
percent
Scaling Source Control for NextGen Game Development 18
edit loop avg. (176 files)
0 0.2 0.4 0.6 0.8 1 1.2
1
3
5
7
9
11
13
15
17
19
21
conf
ig
seconds
Scaling Source Control for NextGen Game Development 19
Test Results Summary• DB operations 5-10x faster on 64-bit Linux on XFS than
Windows 32-bit.
• Linux sync performance under load degraded only 5% instead of the 450% we saw with Windows.
• XFS outperforms ext3 and even edges ReiserFS for data syncs from the repository volume.
Scaling Source Control for NextGen Game Development 20
dm-CommitSubmit
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
win-gf1_1666
win-gf1_1777
linux-gf1_1666co
nfig
seconds
Scaling Source Control for NextGen Game Development 21
avg. edit
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
win-gf1_1666
win-gf1_1777
linux-gf1_1666co
nfig
seconds
Scaling Source Control for NextGen Game Development 22
avg. fstat
0 0.1 0.2 0.3 0.4 0.5 0.6
win-gf1_1666
win-gf1_1777
linux-gf1_1666co
nfig
seconds
Scaling Source Control for NextGen Game Development 23
avg. sync compute phase
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
win-gf1_1666
win-gf1_1777
linux-gf1_1666co
nfig
seconds
Scaling Source Control for NextGen Game Development 24
Configuration Lessons Learned• OS
• 2.6 kernel was 2 – 4x faster than 2.4 in some tests.
• RAID
• RAID 5 best for RCS due to the high number of syncs/reads.
• DB benefited from RAID 1 or RAID 10.
• Filesystem
• XFS proved to be significantly faster than ext3 and beat Reiser by 30% in some tests.
• Drive Speed
• 20% less degradation with 15k drives during multiple concurrent forced syncs due to higher I/O capacity.
Scaling Source Control for NextGen Game Development 25
Performance Principles
• Keep your db’s on their own disks
• Proxy heavy sync users such as build farms or remote locations
• Best place to improve hardware performance is memory and disk I/O for db’s
• There is a big difference in filesystem speed
Scaling Source Control for NextGen Game Development 26
Success Stories• New standard has had a huge impact on our local team
productivity and P4 performance across EA.
• New platform has scaled well for 1,000 user team and could scale to larger teams.
• Stability is great!
• No more memory or file descriptor limits
• Can identify and kill individual threads.
• $5 million saved per year in productivity across EA.
Scaling Source Control for NextGen Game Development 27
NextGen EA P4 Architecture
• P4 Server setup
• Distributed Development
• Proxy Build Farm (30% gain)
• Case-insensitivity
• Disk to disk mirroring
Scaling Source Control for NextGen Game Development 28
The Future of Perforce at EA• RAID 50 (2x faster locking than RAID 10)
• SAS SFF drives
• AD2P4
• Multi-Node Cluster
• Perforce + binary delta copy technology
• Workspace mirroring
• Single View of Perforce Servers
Scaling Source Control for NextGen Game Development 29
Thanks
• Michael Shields
• Brett Taylor
• Kenny Wu
Scaling Source Control for NextGen Game Development 30
Trivia for Schwag
Questions?