Mathias Westerdahl - System Architect, Tools The Asset pipeline for Just Cause 2: Lessons Learned

Mathias Westerdahl - System Architect, ToolsThe Asset pipeline for Just Cause 2: Lessons Learned

2

The Asset pipeline for Just Cause 2: Lessons Learned

Overview• Background

• Just Cause 1

• Just Cause 2

• Current pipeline

• Build system and compilers

• Central code

• Take aways

• Q & A

3


What is Just Cause?• Large open world

• Sand box game play

• Dynamic game play

• No Levels, continuous streaming

• Multiplatform

• Very flexible editing and testing of content

• Needs a fast and flexible pipeline

4


JustEdit• Our main editor is called JustEdit

• Edits the terrain, missions, locations, roads, etc.

• Each mission/location contains entities

• An entity contains game objects or other entities

• This highly flexible system led to very many assets to export (~1500 locations and ~4000 entities)

• Runs the game

• No turnaround times for testing

5


JustEditJustEdit MayaMaya

PerforcePerforce Damage Editor

Damage Editor

TerrainLocationsMissionsParticles

ModelsPhysicsAnimationsSkeletons

Physics

Workflow Overview for Just Cause 1

GameGame

6




ModelsPhysicsAnimationsSkeletons

Physics


Many manual stepsGame ready data in version control

Game ready data in version control

Editing of game ready data!!!

PerforcePerforce Damage Editor

Damage Editor

GameGame

7


Lessons learned from Just Cause 1• Remove as many manual steps as possible

• Put export steps in separate tools (i.e. compilers)

• Decrease code complexity

• Increase robustness

• Easier to read and debug

• Introduce intermediate data formats

• Tools code broke easily

8


Actions Taken after Just Cause 1• We decided on creating our own content build system

• Compilers in C++, derived from previous export steps

• Took existing code and put into use directly

• The Project Compiler was the heart of the system

• Triggered all other compilers

• Dependency checking was implemented separately by each compiler (time stamps)

• Users can right click on any file and compile that dependency tree

• The editor and other tools use this functionality

9


Actions Taken after Just Cause 1• Auto Builders were setup

• Code

• All compilers, exporters, plug-ins, editor

• Content

• Full nightly builds for all platforms

• Builds ISO’s

• Cruise Control + NAnt was used to create the auto builders

• CCTray is a system tray icon tool

• Alerts users when a build fails

10


Just Cause 2 – New Tools• A lot of new tools were ordered by the content creators

• Detailed design docs were written

• Each tool was assigned a “customer”

• The programmers and the customer kept close communication

• The customer was responsible to test the tool and approve it

• Tools as plug-ins to JustEdit

• WTL was used for GUI in editors/plug-ins

11



PerforcePerforce

CompilersCompilers


ModelsPhysicsSkeletons

CompileGameGame


MotionBuilderMotionBuilder

AnimationsCutscenesConstraints

Plug-insPlug-ins

VehiclesCharactersCutscenes

Auto BuilderAuto Builder

Sync

12


Lessons Learned from Just Cause 2 – New Tools• 10 new tools at start of project!!

• Code not up in the 90%-100% complete where it should be

• Flaws in workflows

• Not all tools were used in the extent that was intended

• Content that was easy to edit was not very suitable in game with respect to performance and memory

• Design docs were accepted “as-is”

• They were essentially a wish list from content creators

13


Lessons Learned from Just Cause 2 – New Tools• Customers

• They were not part of the dev-team, communication suffered

• Often planned full time on other tasks

• Unclear responsibilities

• Plug-ins

• Unstable C++ interfaces

• Should have gone with network communications

• Too complex

• GUI: WTL and C++

• WTL is too bare bone and C++ slows down iterations

14


Lessons Learned from Just Cause 2 – Compilers• Compilers in C++

• Using C++ for scripting isn’t always very convenient

• Long build times, making it awkward to tweak

• Some compilers included all game code

• Very platform specific

• Win32 heavy with DirectX amongst other things

• Couldn’t run on auto builders with no desktop

15


Lessons Learned from Just Cause 2 – Compilers• Programmers implemented things slightly differently

• Causing anomalies in overall behavior

• Breaking dependency checking

• Hard coded parameters in the compilers

• File formats weren’t stable enough

• If something went wrong, compiler probably crashed

• Or worse, the broken data made it into the game

16


• Reluctance to work with compilers

• The pipeline was considered somewhat magic

• Lack of documentation

• The overall flow wasn’t easily overviewed

• Debugging broken data often done in game code

• Programmers already had all code & data set up

• This sometimes led to runtime fixes being done rather than fixing the bugs in the compilers

Lessons Learned from Just Cause 2 – Compilers

17


• No central code

• Not using central code meant problems when integrating

• Fixes went missing, you had to find them again

• Built compilers were version controlled in Perforce

• Not automatically rebuilt and used when needed

• You could still use old buggy code


18


• Data formats

• A lot of different file types (~30)

• Many of the formats were Xml based

• Mapping “property name” to “value”

• Formats were more optimal for the editor than for the game

• Read/write errors weren’t detected

• No version numbers

• No proper error messages could be logged

• Lot of things to improve!


19


• Content build system

• A bit naïve solution

• Slow dependency checking

• Incomplete dependency tree

• Time stamps are not good enough

• Due to bugs & anomalies in the behavior, users didn’t trust it to 100%

• “It went wrong! Did I do it in the wrong order?”

• “I’d better resort to my old ways without asking anyone…”

• No trust meant full clean and full rebuild


20


Statistics from Just Cause 2• 3 platforms: Win32, PS3, Xenon

• ~110000 source assets in version control

• ~80000 of those files pass through our compiler pipeline

• Source data amount to ~13GB

• Compiled data amount to

• ~7GB uncompressed

• ~4GB compressed

• A full rebuild took about 4h45m for all 3 platforms

• Building all tools code took ~1h

21


Current pipeline• What was done to address the retrospective from JC2?

• Fresh goals

• Rebuild the trust in the pipeline

• Incremental builds, no magic workflows, easy to configure

• 100% accuracy using the dependency checking

• Simplify the development

• Move common code to a central repository

• Decrease turn around times when creating tools

22


Current pipeline• Big decisions

• We needed a build/deployment system for central code

• The content build system also needed a major push

• Needed a simple to use and yet powerful generalized data format

• Development had to be done alongside JC2

• Needed to commit to the task to actually get anywhere soon

23


Current pipeline - Needy• Dependency resolver and deployment system

• Written in house using python, very light weight

• Handles “packages”

• Built libraries (.h + .lib/.so)

• Built executables

• Bundles of python scripts (or whatever you want)

• 3rd party libraries / executables

• Deploys packages to a central repository

• Fetches packages to local system

Dependencies.txt

Python-2.5.150.1013.1;buildtool

Needy-4.4.0.204986;buildtool

Buildsystem-6.1.*;buildtool

Platform-7.*;buildtool

Debug-8.*;buildtool

Adf-8.*

havok-55x.277059

Product.txt

PRODUCT=animcompiler

VERSION=1.0.0

PLATFORMS="Win32 Linux"

CONFIGURATIONS="Debug Release"

CATEGORY=compilers

24


Current pipeline - Needy• Big change in the coding process

• Code base evolves towards smaller libraries

• Really fast compile times

• Helps keeping track of version conflicts

• Allows for very rapid development and testing

25


Current pipeline – WAF• Light weight build system for code and data

• Open source (BSD), source is ~80kb

• Dependency checking

• Multi core support

• Easily to maintain and extend

• Replaces our in house Project Compiler system

• Python

A small snippet from a task definition using waf:

cls = Task.simple_task_type(‘.anim’, ‘${PYTHONC} ${ANIMC} –i ${SRC} –o ${TGT} –p ${PLATFORM}’)cls.ext_in = (‘.anim’, )cls.ext_out = (‘.ban’, )Task.TaskBase.classes[‘.anim’].scan = FindDependenciesFromAnimation

26


Current pipeline - ADF• Avalanche Data Format

• Serialization framework

• C/C++, Python support

• Xml <-> Binary support

• Hash based versioning for binary files

• Error handling

• The ability to access data both from Python and C/C++ enables seamless cross language interchange

• Cross platform support

• E.g. pass data from tools to/from consoles

27


Current pipeline - ADF

TM2VertexData = DefStruct( "TM2VertexData", ( "PositionIndex", Array( uint32 ) ), ( "TangentSpaceIndex", Array( uint32 ) ), ( "ColorIndex", Array( uint32 ) ), ( "UVIndices", Array( uint32 ) ) ) TM2Model = DefStruct( "TM2Model", ( "Name", String ), ( "Triangles", Array( uint32 ) ), ( "Vertices", TM2VertexData ), ( "MorphTargets", Array( TM2MorphTarget ) ), ( "Positions", TM2PositionData ), ( "TangentSpaces", Array( float ) ) )

Type definitions:

28


Current pipeline - ADF

from Adf import Contextctx = Context()ctx.ReadTypeLibrary('mytype.adf')objs = ctx.ReadInstancesFromXml(‘infile.xml')objs[0].Name = 'newvalue'ctx.WriteInstancesToFile( ‘outfile.bin', objs, BIG_ENDIAN )

Read / Write

mytype.pymytype.py AdfPack.exeAdfPack.exe mytype.adfmytype.adf

mytype.hmytype.h

compiler.pycompiler.py

Infile.xmlInfile.xml

Outfile.binOutfile.bin

29


Current pipeline - Compilers• C++

• Shared libraries, loaded from Python

• Removed dependencies on old game code

• Replaced in-house code with standard code

• fopen() rather than CFile()

• Decrease complexity, increase portability

• Scripting and logic in python

• Data processing in Python or C++

• E.g. read/write/process Havok files using C++

30


Current pipeline - Compilers• Python

• Extreme turn around times

• Vast collection of built in modules

• We use these a lot:

• optparse, ctypes, md5, numpy, cStringIO…

• Many of the modules have the backend implemented in C++

• Most compilers were completely rewritten

• Cut a lot of dependencies

• Code size down to a 1/10th

31


Current pipeline – Dependency Checking• Waf handles the mechanics

• Scanners find the dependencies

• Waf caches the result

• MD5 checksums

• Command lines arguments

• Source/Dependency file contents

• A new compiler path will trigger a compilation

• Very handy when updating packages with Needy

32


Current pipeline – Dependency Checking• A small example of scanning a file for other file references

# Reading a text file containing file referencesdef FindDependencies(path): f = open(path, "rb" ) lines = f.readlines() f.close() return [ x.strip() for x in lines ]

33


Current pipeline• Platform independence

• Possible to run on win32, linux, ps3, xenon etc.

• Helps keeping the code clean

• Running on linux

• We wanted to run on linux for the file system performance

• Run win32 exe’s through WINE

• Wine is a Win32 implementation in linux. It’s not an emulator.

• Intermediate solution before all code was rewritten

34


Current pipeline• Incremental auto builder

• Syncs from version control

• Only builds the things necessary

• Full auto builder

• Compares incremental build with full build

• Finds dependency problems

• Find bugs in data formats

35


Current pipeline• Change of mindset

• Keep it simple!

• Each component should do one thing only

• …and do it well!

36


Current pipeline• To sum things up so far

• We rewrote ~20 compilers (out of ~30) in 6 man weeks

• Using old code as blue print

• Remaining compilers took longer due to more complex work flows

• Code size decreased to 1/10th

• Performance increased in all cases

• 10% to 1000%, 100% was quite common

• Possible to use compilers in the previous pipeline with little effort

• Early benefits of the new pipeline

37


Future improvements• Reloading of assets in run time (in progress)

• Virtual File System

• Can load data from hard drive, archives or web server

• Distributed builds

• Looking into using AFS, a distributed file system

• ssh builder01 modelcompiler.exe -i mymesh1.tm …

• ssh builder02 modelcompiler.exe -i mymesh2.tm …

38


Take Aways• A few thing we’ve learned so far

• Some are quite trivial but also easy to overlook

39


Take Aways• Hardware

• Defrag the hard drives

• Try using other file systems, ext3, ext4, afs.

• Our pipeline is a lot faster on linux than on win32

• Two separate discs that are designated source/target

• Ram discs/SSD are getting cheaper

40


Take Aways• How to implement the compilers

• No assumptions on working directory or any other variables

• Keep the compiler data driven, it’s quite easy

• Project specific options could be read from a file or command line

• Create a simple plugin system

• Extremely easy in python

41


• Example of loading python files. I.e. creating a plug-in system:

for root, dirs, files in os.walk(‘c:/myapp/plugins’): if not root in sys.path: sys.path.append(root) for filename in files: if not filename.endswith('.py'): continue # We do an import() instead of an execfile() so that # the loaded code can act as a regular python file module = __import__(filename.replace('.py', '') )

Take Aways

42


Take Aways• How to implement the compilers

• Write portable code

• E.g. remove DirectX, Win32 API

• ‘-e’ for endianess rather than ‘-p’ for platform

• Beware of case sensitiveness from the start

• Also, beware of ‘\\’ versus ‘/’

43


Take Aways• How to organize the data

• Keep source data separate from compiled data

• When creating new file formats

• Avoid doing it “by hand”. Create a generalized file type.

• Include a version number

• Store file dependencies easily accessible

• Write the game ready format with efficiency in mind

• Memory, performance

44


Take Aways• How to optimize the compilers

• Write compilers with parallelism in mind

• No “global variables” such as reused temp files on disc

• Disc cache thrashing

• It comes down to the access patterns

• Do as few reads/writes as possible

• Use tools like Process Explorer or vmstat

45


Take Aways• How to optimize the compilers

• Do reads and writes in memory

• Lowers the disc usage

• E.g allocate large buffer and write into it

• Avoid temp files altogether

import cStringIOmemfile = cStringIO.StringIO()MultipleWritesToAFile( mydata, memfile)f = open(options.output, "wb" )f.write( memfile.getvalue() )f.close()memfile.close()

46


Take Aways• Dare to do stuff!

• If the compiler code is unreadable/inefficient

• Will it be faster/more productive to rewrite it?

• Rewriting will probably introduce new bugs

• But you will also…

• Probably throw away many old bugs as well

• Increase your understanding of the work flow

• Get a chance to improve design/performance more directly

47


Take Aways• Dare to do stuff!

• Rewriting doesn’t mean you throw the code away

• You will use the code as a blue print of what, or what not, to do

• It’s easy to rewrite something that does the same thing, only better!

48


Take Aways• Separate code

• Use one code base to read/write data in compilers

• Use another code base for loading the data in game

• Keeps down code dependencies

• Keeps the compilers small and simple

• Less code to maintain in the compiler

• Decreased code size down to 1/10th

• Less code to compile, faster turnaround times

• Suitable for file formats that rarely change

49


Take Aways• Keep the build working!

• Only replace old code with new working code

• Down time is both costly and stressful

• Keep it simple!

50


Links• Python – http://www.python.org/

• waf – http://code.google.com/p/waf

• wine – http://winehq.org

• jqplot – http://www.jqplot.com/

• protobuf – http://code.google.com/p/protobuf/

• Download slides and source code on our site!

http://www.python.org/

http://code.google.com/p/waf

http://winehq.org/

http://www.jqplot.com/

http://code.google.com/p/protobuf/

51


Q & A

www.avalanchestudios.se/presentations

[email protected]

Download slides and source code on our site!

http://www.avalanchestudios.se/presentations

mailto:[email protected]

mailto:[email protected]

52


Documents

Mathias Westerdahl - System Architect, Tools The Asset pipeline for Just Cause 2: Lessons Learned