Boundary Front end tech talk: how it works

Tech Talk - FrontendStephen Boak (steve@boundary.com)Mark Mahoney (mark@boundary.com)Matt King (m@boundary.com)

When we say real-time we mean it: JSON being pushed to the browser once a second and visualizations built to constantly move and adapt to streaming data

Boundary | Architecture

Meters Lightweight and highly scalable, these sit on the cloud as well as private data centers across virtual or physical servers.

!

Collectors Intercept Meter data via Transport Layer Security (TLS) Authentication.

Data Store Collect all of the data at high resolution to gain rich insight into complex environments & problems before they can impact critical business services.

Streaming Real-time continuous data streaming at high resolution, low latency intervals.

Streaming UI Stream data to your dashboard with sub-second latency,providing you with intuitive, powerful dashboard.

How We Get Data

• CometD server (streaker)!

• Pulls in data from multiple backend services and streams it out to clients!

• Can also aggregate and filter data on demand

Subscriptions

54321

Receiveadd/remove

messages

Receivestate-dump

(inserts & schema)

Subscribe tonew query

(unique channel)

Get back a subscription ID

(Query ID)

Make asubscription request

(query and filters)

!

!

!

! !!

Data Structure

• Schema and key(which fields are concatenated for each record)

• State dump with schema for requested time window(up to 100kb state dumps)

• Save bandwidth with N-tuples indexed by the schema

• Adds/removes update the state with keys (subset of schema)

JSON

{channel:""/query/ac3941b8924a2f73/custom9query92f536bbc93f159418b98f1a973b5dc4e78379filter979564d6c94c9394f479abae906ff95a3411e9aggregation9b5cc18d49098d949c29a08497037b08a30e6"data:"{insert:"[[1331234598000,"80:6,"100,"8463,"93,"9672],"[1331234925000,"8080:6,"242,"39657,"243,"40865],"…],keys:"[epochMillis,"portProtocol],schema:"[epochMillis,"portProtocol,"ingressPackets,"ingressOctets,"egressPackets,"egressOctets],timestamp:"1331235119001

}}

{"1331234598000:80:6":"{"epochMillis":=1331234598000,"portProtocol":="80:6","ingressPackets":"100,"ingressOctets":"8463,"egressPackets":"93,"egressOctets":"9672

},"1331234925000:8080:6":"{"epochMillis":=1331234925000,"portProtocol":="8080:6","ingressPackets":"242,"ingressOctets":"39657,"egressPackets":"243,"egressOctets":"40865

}…

}

Example port:protocol subscription State Object

DataSource.js

• Negotiates between multiple data sources and subscribers on a page

• One data source for N subscribers

• Abstracts CometD subscription process for JS development

• Smart enough to resubscribe and notifies subscribers(so that state dumps aren’t doubled up)

DataSources

Subscribers

Subscriptions

bndry.datasource repo

//"Receives"object"with"properties"for"state,"inserts"and"removes."function"updateData(data)"{"" console.log(data.added);"}"

//"Create"a"handle"to"a"new"data"source"var"volumeDataSource"="bndry.dataSource('volume_1s');"

//"Add"a"subscriber"and"start"getting"updates."volumeDataSource.addSubscriber(updateData);

Demo Time

Data and Subcription Problems

• Monolithic, Multi-Purpose Queries• Large State Dumps cause CometD timeout

(5-10MBs per second)• No Resubscription• Aggregated all data on front-end

Data and Subcription Solutions

• Stratified, Filterable Queries and server-side aggregation• Top-N Limitations

(constrained output)• Resubscription & filtering• Web workers haven’t helped much• Time smoothing (1-second ticks)

Visualization

DOM manipulation is expensive ...so is tweening

CanvasSVG

What’s Next

• WebSockets

• Historical Data and long term data storage

• HTML5 local storage (store data)

• Machine-learning for real-time network visualization

Thank You