Embed Size (px)
DESCRIPTION
By Kristoffer Benjaminsson, CTO, Easy.This talk presents the telemetry system used in Battlefield Heroes and how it helps the team make technical decisions in order to provide the best service possible. We will show real life examples of how telemetry helped improve matchmaking, reduce latency for players and help find false alarms from the cheat detection system. We will also discuss how telemetry can be used in development for catching bugs and support game designers in their work.
Citation preview
How data rule the worldKristoffer Benjaminsson CTO
Easy?• Easy is a studio within EA Games that do free to
play games• Originates from DICE• 3 live titles– Battlefield Heroes– Battleforge– Lord of Ultima
Trailer
Telemetry - Wikipedia” Telemetry (synonymous with telematics) is a
technology that allows remote measurement and reporting of information. The word is derived from Greek roots tele = remote, and metron = measure.”
Telemetry - Easy• Data that helps us make decisions to provide a better service for
our players• Provides answers to specific questions• Multiple sources
– Financial data– Game statistics– Web behaviour data– Game telemetry
• Vertical slice of the player population
Why bother?• We prefer to know what is going on instead of
guessing• We can measure results from actions• Data do not (usually) lie• Forums not representative – Less than 20% read forum posts– Less than 5% post in forums
Game telemetry examples• Latency data• Punkbuster kicks• Matchmaking• Time to level• Hardware profiles• Play time
Architecture
Game Client
Game Server
Magma DB Reporting
Implementation• TelemetryManager sends data to our backend for
database storage• Hooks in the game code collects data• Hooks are records packed as name value pairs• Each record will create its own table in the database• All telemtry includes a common identifer so we can
cross reference data between telemetry hooks
Example recordclass PingRecord : public ITelemetryRecord{public:
PingRecord(const std::string& clientIp, const std::string& serverIp, int ping, const std::string& dataCenter, const std::string& community);
public:virtual bool hasType( int typeId ) const { return typeId == RTPing; }virtual bool assimilate( const ITelemetryRecord* other ) { return false; }virtual IDataCollector::DataRef buildData() const;
private:std::string m_clientIp;std::string m_serverIp;int m_ping;std::string m_dataCenter;std::string m_community;
};
Example usage
player->addTelemetry(new backend::telemetry::PingRecord(clientIp, serverAddressAndPort, conn->getAveragePing(), serverSettings->getProdDataCenter(), serverSettings->getServerCommunity()));
Real life examples
High latency - before
Why were people in Europe being matchmade to Australia and North America?
High latency - after
How did we do it?• Generated an HTML page– GeoIP City– A bit of C#– Google Maps API– Telemetry data– A bit of SQL
Example page<html> <head> <meta name="viewport" content="initial-scale=1.0, user-scalable=no" /> <script type="text/javascript" src="http://maps.google.com/maps/api/js?sensor=false"></script> <script type="text/javascript"> function initialize() { var latlng = new google.maps.LatLng(45.5735931396484, 12.1060943603516); var myOptions = { zoom: 2, center: latlng, mapTypeId: google.maps.MapTypeId.ROADMAP };
var map = new google.maps.Map(document.getElementById("map_canvas"), myOptions);
var marker = null; var coords = null; var path = null; marker = new google.maps.Marker({ position: new google.maps.LatLng(59.433, 24.728), map: map, title:"gva, 62.65.243.167" , icon: "client.png" }); marker = new google.maps.Marker({ position: new google.maps.LatLng(52.350, 4.916), map: map, title:"gva" , icon: "server.png" }); coords = [new google.maps.LatLng(59.433, 24.7283), new google.maps.LatLng(52.350, 4.916)]; path = new google.maps.Polyline({ path: coords, strokeColor: "#ff0000", strokeOpcaity: 0.5, strokeWeight: 1}); path.setMap(map); } </script> </head> <body onload="initialize()"> <div id="map_canvas" style="width: 100%; height: 100%"></div> </body></html>
Google Maps
Punkbuster - before
Punkbuster - after
How did we do it?• Microsoft SQL Server 2008: Reporting Services– Standard BI tool– Easy to create charts and tables– Updates on a daily basis
During development
Asserts• We collect all asserts from both client and server• We differentiate between content asserts and code
asserts• We have leaderboards and summaries– Top 10 asserts– Total content/code asserts per branch
• Provides guidance to what we need to fix
Fragalyzer• Level design helper tool• Collects information about how a map is played– Kill position– Killed from position– Player/Vehicle movement– Etc
• Currently collected locally on a game server
Summary• Telemetry is awesome!• Collect data to answer specific queries• Build a flexible system to make it super easy to
add new telemetry hooks• Chose the right reporting tools to mine and
report on the data
Questions?
