A High Performance, High Reliability Perforce Server

Shiv Sikand, IC ManageMarc Lewert, Angela Thomas, TiVo

Advanced Perforce SolutionsSemiconductor design companies

Design tool plug-ins Cadence, Synopsys, Mentor, Magma

Enterprise SolutionsHigh Performance ServersSync EngineCaching Accelerator

Loss of interactive performanceUser complaintsBuild system slowdown

Long checkpoint timesLong verify times

Server demandsLarge number of build clients running ‘sync’ every 15 minutes

Complex build systemLarge file counts, complex mappings

Server EnvironmentSun 420RHitachi 9200 Storage ArrayVeritas VXFSRAID5Storage Area Network

Uptime issuesServer hardware failuresVXFS problems

SAN AdvantagesProtect against hardware failures

Decoupled storageEasy storage scalabilityConsolidated backup and disaster recovery for all data

SAN DisadvantagesIncreased latency

Hardware wiresKernel & I/O subsystem distance to disk drives

Software wiresNetwork and switching protocols

Performance ConsiderationsLarge DB’s are disk seek limitedRandom and Sequential I/O performance is key

SAN’s can turn sequential I/O into random I/O

IC Manage SolutionHigh performance, high availability serverSync Engine

IC Manage ServersLinux

Low cost x86 hardwareDirect attached disksHigh MTBFHigh performance kernelsAsynchronous directory updatesNext generation XFS filesystem

Journalled, RAID optimized, scalable

Direct Attach ArraysPros

PerformanceEase of use

Monitoring utilities, configurable RAID levelsLogical volume managementWeb based management interfaces

Direct Attach ArraysCons

System failures can result in long downtimes

Motherboard, controllers, network interfaces

Linux High AvailabilityDual hosts with failover

“heartbeat” packageSerial pulseEthernet pulseIP address takeoverService stop/start

Linux High AvailabilityDual active-active controllers

RAID10 volumeRedundant power

Linux High AvailabilityPerformance and reliability are always tradeoffs

Active-active controllers offer less performance than single controllers

Sync EngineDesigned for Automated Syncing

Avoid interval based syncsEliminate continual db lookupsPrevent excessive read locksPrevent write starvation

Use 2003.2 sync optimisation automaticallyp4 sync @lastchange,@current

No need to modify build system

Method of operationMySQL based

QT GUI front endRegister whole clients or viewsTwo operating modes

ActiveAutomated scheduling

ManagedQuery based

Method of operationMonitor change counter

Parse change descriptions and match against registered mappings

Multi-threaded schedulerRSH or SSH based execution

Hosts or groupsStore change number on success

‘p4 sync [spec] @c1,@c2’

TiVo Data AnalysisAngela Thomas

FigureOldServer

Command execution time

0

2

4

6

8

10

12

14

16

describe dirs filelog fstat have changes

time Old

New

Command execution time

0

1

2

3

4

5

6

edit open opened reopen add delete

time Pre

Post

Command execution times

0

1

2

3

4

5

6

7

8

diff2 integ revert review reviews

time Pre

Post

Command execution times

0

2000

4000

6000

8000

10000

12000

14000

Pre Post

time

verify

Enterprise Storage Requirements

Data consolidation from multiple sitesNetwork storage

99.999%, 24x7 AvailabilityBackup, archiving, disaster recoveryScalability

Caching AcceleratorBlock diagram

Enterprise Storage

Disk Cache

Server

Fader

Replicator

Replicatorp4jrep based

Change drivenNear real time replication of both db and archive files

Supports scp, rcp or rsync

FaderFades “old” data

Two modesRemoves from local storageUses link replacement to replica

Custom p4 client/api classTraps librarian errors and signals fader for recoveryRetries failed operation

ConclusionIC Manage high performance servers

Demonstrate significant performance advantagesOffer high availability

ConclusionSync Engine

Scalable alternative to interval based syncingSuit complex build environments

ConclusionCaching Accelerator

Best of both worlds for enterprisesPerformance and Reliability

Thank you !