Mathias Westerdahl - System Architect, Tools The Asset pipeline for Just Cause 2: Lessons Learned

  • View
    213

  • Download
    0

Embed Size (px)

Transcript

  • Slide 1
  • Mathias Westerdahl - System Architect, Tools The Asset pipeline for Just Cause 2: Lessons Learned
  • Slide 2
  • 2 Overview Background Just Cause 1 Just Cause 2 Current pipeline Build system and compilers Central code Take aways Q & A
  • Slide 3
  • 3 The Asset pipeline for Just Cause 2: Lessons Learned What is Just Cause? Large open world Sand box game play Dynamic game play No Levels, continuous streaming Multiplatform Very flexible editing and testing of content Needs a fast and flexible pipeline
  • Slide 4
  • 4 The Asset pipeline for Just Cause 2: Lessons Learned JustEdit Our main editor is called JustEdit Edits the terrain, missions, locations, roads, etc. Each mission/location contains entities An entity contains game objects or other entities This highly flexible system led to very many assets to export (~1500 locations and ~4000 entities) Runs the game No turnaround times for testing
  • Slide 5
  • 5 The Asset pipeline for Just Cause 2: Lessons Learned JustEdit Maya Perforce Damage Editor Terrain Locations Missions Particles Models Physics Animations Skeletons Physics Workflow Overview for Just Cause 1 Game
  • Slide 6
  • 6 The Asset pipeline for Just Cause 2: Lessons Learned JustEdit Maya Terrain Locations Missions Particles Models Physics Animations Skeletons Physics Workflow Overview for Just Cause 1 Many manual steps Game ready data in version control Editing of game ready data!!! Perforce Damage Editor Game
  • Slide 7
  • 7 The Asset pipeline for Just Cause 2: Lessons Learned Lessons learned from Just Cause 1 Remove as many manual steps as possible Put export steps in separate tools (i.e. compilers) Decrease code complexity Increase robustness Easier to read and debug Introduce intermediate data formats Tools code broke easily
  • Slide 8
  • 8 The Asset pipeline for Just Cause 2: Lessons Learned Actions Taken after Just Cause 1 We decided on creating our own content build system Compilers in C++, derived from previous export steps Took existing code and put into use directly The Project Compiler was the heart of the system Triggered all other compilers Dependency checking was implemented separately by each compiler (time stamps) Users can right click on any file and compile that dependency tree The editor and other tools use this functionality
  • Slide 9
  • 9 The Asset pipeline for Just Cause 2: Lessons Learned Actions Taken after Just Cause 1 Auto Builders were setup Code All compilers, exporters, plug-ins, editor Content Full nightly builds for all platforms Builds ISOs Cruise Control + NAnt was used to create the auto builders CCTray is a system tray icon tool Alerts users when a build fails
  • Slide 10
  • 10 The Asset pipeline for Just Cause 2: Lessons Learned Just Cause 2 New Tools A lot of new tools were ordered by the content creators Detailed design docs were written Each tool was assigned a customer The programmers and the customer kept close communication The customer was responsible to test the tool and approve it Tools as plug-ins to JustEdit WTL was used for GUI in editors/plug-ins
  • Slide 11
  • 11 The Asset pipeline for Just Cause 2: Lessons Learned JustEdit Maya Perforce Compilers Terrain Locations Missions Particles Models Physics Skeleto ns Compile Game Workflow Overview for Just Cause 2 MotionBuilder Animations Cutscenes Constraints Plug-ins Vehicles Characters Cutscenes Auto Builder Sync
  • Slide 12
  • 12 The Asset pipeline for Just Cause 2: Lessons Learned Lessons Learned from Just Cause 2 New Tools 10 new tools at start of project!! Code not up in the 90%-100% complete where it should be Flaws in workflows Not all tools were used in the extent that was intended Content that was easy to edit was not very suitable in game with respect to performance and memory Design docs were accepted as-is They were essentially a wish list from content creators
  • Slide 13
  • 13 The Asset pipeline for Just Cause 2: Lessons Learned Lessons Learned from Just Cause 2 New Tools Customers They were not part of the dev-team, communication suffered Often planned full time on other tasks Unclear responsibilities Plug-ins Unstable C++ interfaces Should have gone with network communications Too complex GUI: WTL and C++ WTL is too bare bone and C++ slows down iterations
  • Slide 14
  • 14 The Asset pipeline for Just Cause 2: Lessons Learned Lessons Learned from Just Cause 2 Compilers Compilers in C++ Using C++ for scripting isnt always very convenient Long build times, making it awkward to tweak Some compilers included all game code Very platform specific Win32 heavy with DirectX amongst other things Couldnt run on auto builders with no desktop
  • Slide 15
  • 15 The Asset pipeline for Just Cause 2: Lessons Learned Lessons Learned from Just Cause 2 Compilers Programmers implemented things slightly differently Causing anomalies in overall behavior Breaking dependency checking Hard coded parameters in the compilers File formats werent stable enough If something went wrong, compiler probably crashed Or worse, the broken data made it into the game
  • Slide 16
  • 16 The Asset pipeline for Just Cause 2: Lessons Learned Reluctance to work with compilers The pipeline was considered somewhat magic Lack of documentation The overall flow wasnt easily overviewed Debugging broken data often done in game code Programmers already had all code & data set up This sometimes led to runtime fixes being done rather than fixing the bugs in the compilers Lessons Learned from Just Cause 2 Compilers
  • Slide 17
  • 17 The Asset pipeline for Just Cause 2: Lessons Learned No central code Not using central code meant problems when integrating Fixes went missing, you had to find them again Built compilers were version controlled in Perforce Not automatically rebuilt and used when needed You could still use old buggy code Lessons Learned from Just Cause 2 Compilers
  • Slide 18
  • 18 The Asset pipeline for Just Cause 2: Lessons Learned Data formats A lot of different file types (~30) Many of the formats were Xml based Mapping property name to value Formats were more optimal for the editor than for the game Read/write errors werent detected No version numbers No proper error messages could be logged Lot of things to improve! Lessons Learned from Just Cause 2 Compilers
  • Slide 19
  • 19 The Asset pipeline for Just Cause 2: Lessons Learned Content build system A bit nave solution Slow dependency checking Incomplete dependency tree Time stamps are not good enough Due to bugs & anomalies in the behavior, users didnt trust it to 100% It went wrong! Did I do it in the wrong order? Id better resort to my old ways without asking anyone No trust meant full clean and full rebuild Lessons Learned from Just Cause 2 Compilers
  • Slide 20
  • 20 The Asset pipeline for Just Cause 2: Lessons Learned Statistics from Just Cause 2 3 platforms: Win32, PS3, Xenon ~110000 source assets in version control ~80000 of those files pass through our compiler pipeline Source data amount to ~13GB Compiled data amount to ~7GB uncompressed ~4GB compressed A full rebuild took about 4h45m for all 3 platforms Building all tools code took ~1h
  • Slide 21
  • 21 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline What was done to address the retrospective from JC2? Fresh goals Rebuild the trust in the pipeline Incremental builds, no magic workflows, easy to configure 100% accuracy using the dependency checking Simplify the development Move common code to a central repository Decrease turn around times when creating tools
  • Slide 22
  • 22 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline Big decisions We needed a build/deployment system for central code The content build system also needed a major push Needed a simple to use and yet powerful generalized data format Development had to be done alongside JC2 Needed to commit to the task to actually get anywhere soon
  • Slide 23
  • 23 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline - Needy Dependency resolver and deployment system Written in house using python, very light weight Handles packages Built libraries (.h +.lib/.so) Built executables Bundles of python scripts (or whatever you want) 3rd party libraries / executables Deploys packages to a central repository Fetches packages to local system Dependencies.txt Python-2.5.150.1013.1;buildtool Needy-4.4.0.204986;buildtool Buildsystem-6.1.*;buildtool Platform-7.*;buildtool Debug-8.*;buildtool Adf-8.* havok-55x.277059 Product.txt PRODUCT=animcompiler VERSION=1.0.0 PLATFORMS="Win32 Linux" CONFIGURATIONS="Debug Release" CATEGORY=compilers
  • Slide 24
  • 24 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline - Needy Big change in the coding process Code base evolves towards smaller libraries Really fast compile times Helps keeping track of version conflicts Allows for very rapid development and testing
  • Slide 25
  • 25 The Asset pipeline for Just Cause 2: Lessons Learned Current pipeline WAF Light weight build system for code and data Open source (BSD), source is ~80kb Dependency checking Multi core support Easily to maintain and extend Replaces our in house Project Compiler system Python A small snippet from a task definition using waf: cls = Task.simple_task_type(.a