Visualizing large-scale IP traffic flows

Florian Mansmann, Fabian Fischer, Daniel A. Keim, Stephen C. North

University of Konstanz, GermanyAT&T Research, USA

Email: {mansmann,fischerf,keim}@inf.uni-konstanz.de, north@research.att.com

Abstract

Hierarchical Network Maps are a scalable approachto the presentation of IP-related measurements onthe global Internet. This study focuses on how toextend them for emphasizing the source destina-tion relationship of network traffic aggregated on IPprefix, autonomous system, country, or continent.Edge bundles consisting of several spline curves vi-sually group traffic that shares common ancestornodes along the IP/AS hierarchy.

1 Introduction

Today, signature-based and anomaly-based intru-sion detection is considered the state-of-the-art fornetwork security. However, fine-tuning parametersand analyzing the output of these methods can becomplex, tedious, and even impossible when donemanually. If this situation was not challengingenough, current malware trends suggest an increasein security incidents for the foreseeable future. Thehealth of the network infrastructure clearly dependson the effectiveness of both manual and automatedmethods to analyze, comprehend, and disseminateunderstanding of large network data sets.

Hierarchical Network Maps (HNMaps) are anapproach to the presentation of IP-related measure-ments on the global Internet. They are based ona hierarchy (prefix → AS → country →continent) on top of all Internet subnet prefixesand displayed using a space-filling visualizationtechnique. This pixel-conservative approach is ap-propriate as display space is a scarce resource whendisplaying about 200,000 IP prefixes at once.

In previous work, we considered network statis-tics observed at a single vantage point (arriving at orleaving from a particular gateway) and displayed iteither by its source or destination in the IP addressspace. In this study, we consider traffic being trans-

ferred through multiple routers, such as in serviceprovider networks. The use of edge bundles enablesvisually displaying and detecting patterns throughaccumulative effects within nodes of the AS/IP hi-erarchy. The novelty of this approach lies in its ca-pability to support the formation of a mental modelthat places each autonomous system or IP prefix ona map while linking nodes according to the trafficunder consideration, and at the same time limitingvisual clutter.

The rest of this paper is structured as follows:we briefly discuss the employed database technol-ogy to speed up the backend of our applicationand review related work. Our HNMap appraochis then discussed and extended through so-callededge-bundles. Afterwards, we examine generationof random data and apply the presented methods be-fore assessing the overall contribution.

2 Efficient querying of large IP-related data sets

To support visualization that is fast enough for in-teractive data exploration, we need not only to con-sider efficient rendering techniques, but also beaware of database technology as precondition forflexibility in querying different data sets as well asfor speed. We therefore briefly regard the multi-dimensional data model ([13]) which stores largeamounts of facts with associated numerical mea-sure in data cubes that are particularly well-suitedfor data analysis (in contrast to storing of trans-actional data). Queries aggregate measure valuesover a range of dimensional values to provide re-sults such as the number of security events aggre-gated on each AS in a given country (dimension IPhierarchy) at a certain day (dimension time). Figure1 shows the user interface to specify the databasequery and visualization parameters.

VMV 2007 H. P. A. Lensch, B. Rosenhahn, H.-P. Seidel, P. Slusallek, J. Weickert (Editors)

Figure 1: Specifying the database query and visual-ization parameters.

Using the snowflake schema, a separate table iscreated for each level of the dimension IP addressand the tables’ entries are linked accordingly. Forexample, each IP address is linked to the tuple ofits advertised IP prefix, which in turn is linked to itsAS, etc. Pre-joining the lowest level of the IP hier-archy with all its upper level entries avoids expen-sive join operations during interactive analysis andthus considerably speeds up queries. IP addressesare grouped by IP prefix→ autonomous system→country→ continent and we thus obtain the hierar-chy as shown in Table 1.

Table 1: IP/AS hierarchy

Level Name Entries1 continents 72 countries 1903 autonomous systems 230544 prefixes 197427

The snowflake schema is not limited to the di-mension IP hierarchy, but can be - depending on theanalysis field - easily extended to various dimen-sions, such as time, protocol, ports, or type of event.

3 Related Work

A common way of displaying hierarchical data isin layouts where child nodes are placed inside par-ent node boundaries. Such displays provide spatiallocality for nodes under the same parent, and visu-ally emphasize the sizes of sets at all levels in thehierarchy. Usually, leaf nodes may have labels oradditional statistical attributes that may be encodedgraphically as relative object size or color.

The most important layouts of this type areTreemaps – space-filling layouts of nested rectan-

gles, of which there are several main variants. Theearliest variant was the slice-and-dice Treemap [8].Here, display space is partitioned into slices whosesizes are proportional to to the sum of the nodesthey contain. At each hierarchy level this proce-dure is repeated recursively, rendering child nodesinside parent rectangles while alternating betweenhorizontal and vertical layouts. This is not diffi-cult to program and can run efficiently, but long,thin rectangles arise, which are hard to perceive andcompare visually.

Squarified Treemaps [3], remedy this deficiencyby using rectangles with controlled aspect ratios.Rectangles are prioritized by size, so large ones aretreated as the most critical ones for layout. Thisimproves the appearance of Treemaps, but does notpreserve the input node order, which is also a prob-lem in some applications. This drawback was no-ticed, and Ordered Treemaps [2] were introduced toaddress it.

An alternative, non-Treemap layout algorithmwhich is not space filling was applied in [7] to thevisualization of computer security data. Their pro-posed method maps hosts to rectangles, and subnetsto larger enclosing rectangles. In contrast to thismethod, the approach proposed here has the goalof integrating both geographic and abstract layoutsin the same view, and scaling up to the entire IPv4address space.

Another related area is recent work on rectangu-lar layouts in cartography, such as rectangular car-tograms [5] which optimize the layout of rectangleswith respect to area, shape, topology, relative po-sition, and display space utilization. A genetic al-gorithm has been applied to find a good compro-mise between the objective functions describing theabove mentioned properties. This technique renderslayouts offline, not interactively and does not yet ex-ploit hierarchical structures.

HNMap [11] supports the formation of a mentalmodel of measurements reflecting the global Inter-net. It combines several layout techniques to copewith a large, multilevel AS/IP hierarchy in a toolthat runs fast enough for interactive response. Inanother study [10], we evaluated alternative layoutsand reported a case study with network data froma web server, a university network gateway, andan intrusion detection system (IDS) from a serviceprovider to gain deeper insight into these large datasets.

Figure 2: Multi-resolution HNMap approach to display aggregated IP-referenced measurements of prefixes(middle), ASes (left), countries (right), and continents (bottom) using a bi-color scale.

Drawing network traffic as lines on top of mapsis well-known and has been studied in the carto-graphic community. Unfortunately, highly inter-connected nodes lead to a lot of visual clutter andmake it difficult to recognize any structure. Beckeret al. [1] attack the problem by inventing line short-ening and visualizing an adjacency matrix insteadof geographic objects. Brushing has also been rec-ommended [15].

A further approach to show the movement of ob-jects from one location to another are so-called flowmaps. Traditionally, these flow maps were handdrawn to reduce visual clutter introduced by over-lapping flows. Phan et al. presented a method forgenerating well-drawn maps, which allow users tosee the differences in magnitude among the flowswhile minimizing the amount of clutter [14]. How-ever, interpretation of flow maps with multiple van-tage points stays challenging.

A recent study [6] handles the problem of vi-sual clutter elegantly: a hierarchical classificationof nodes is exploited for bundling lines that connectleaf nodes, to visually emphasize correlations. In-

spired by this work, we draw edge bundles on top ofHNMaps in this paper. This enables us to view end-to-end relationships in network data sets, instead oflimiting our analysis to the outgoing or incomingtraffic from a single vantage point.

4 Hierarchical Network Map

In visualizing times series of network statistics, set-ting node (rectangle) sizes proportional to a timeseries variable leads to confusing displays, due torepositioning of nodes between HNMap frames.Figure 2 shows the multi-resolution HNMap ap-proach addressing spatial memory by fixing nodepositions to facilitate tracing through time. A rea-sonable approach to this is to make node sizes pro-portional to the number of IP addresses contained,which is static in our experiments.

The general visualization paradigm of our tech-nique places child nodes within the bounds of theirparent node’s rectangle. This results in a groupingoperation which adds semantic meaning to the oth-erwise unstructured data. The benefit is obvious for

the parent child relationship among the continentand country as well as the AS and prefix nodes.However, the relationship between countries andASes is not always clear. For this, we rely on statis-tics for each IP within an AS, which we obtainedfrom a commercial GeoIP database [12]. Unfortu-nately, the country information within the registra-tion services of ARIN, RIPE, AFRINIC, APNIC,LACNIC as obtained from [16] were uncompleteand sometimes misleading (a lot of ASes are regis-tered to EU rather than a country).

The upper two levels of the AS/IP hierarchy aregeographic entities. In general, geographic visual-ization is often very compelling– two-dimensionalmaps are familiar to most people as a conventionfor representing three-dimensional reality. Remark-ably enough, mental models derived from mapsare effective for many tasks even when extremescales and nonlinear transformations are involved.Many approaches have been investigated for show-ing geographically-related as well as more abstractinformation on maps [4].

As a similarity measure for the AS level, we cal-culated the middle IP address of each AS by averag-ing the weighted middle IP address of all its adver-tised IP prefixes. This information is only meaning-ful to a certain extent as ASes sharing similar pre-fixes are positioned next to each other, but it doesnot take connectivity among ASes into account.

The lowest level of the hierarchy consists of theprefixes which have a clearly defined order. Oursystem therefore offers the capability to display thislevel using the Ordered Strip Treemap [2] with aline-wise sorting order. However, the HistoMap 1Dmethod scales better to the large hierarchy data athand with respect to visibility of small prefixes, lay-out preservations, and rendering performance (cf.[10]). The hierarchy levels can be interactivelydrilled-down or rolled-up. The mouse cursor high-lights the measure for one particular hierarchy nodeand labels for the current hierarchy level as well asall parent nodes. When loading a data set, prun-ing nodes with little or no traffic frees space for thecurrent analysis and retaining layout stability at thesame time becomes an important aspect [10]. Sincethis paper is a continuation of our previous work,we discard nodes with no traffic in the HNMaps pre-sented in this study.

Large differences in sizes between IP prefixes,ASes, countries, and even continents turn visual

comparison of the respective rectangles into a chal-lenge, especially when dealing with ordinary com-puter displays as opposed to wall-sized displays.We opted for a compromise by scaling the IP prefixsizes (number of contained IP addresses) and there-fore indirectly also the upper level aggregates usingsquare-root or, alternatively, logarithmic scaling:

fsqrt(x) =√

x (1)

flog(x) = log x + 1 (2)

The effects of node size scaling are illustrated inFigure 3. Note that the size of the upper level rect-angles is determined through the sum of the scaledchild nodes.

(a) linear

(b) square-root

(c) logarithmic

Figure 3: Effects of scaling on the space-filling lay-out demonstrated on some IP prefixes in Germany.

In the proposed visualization, the value of ameasure of interest is encoded as color, using afixed color scale and logarithmic normalization:colorindex(v) = log(v + 1)/log(vmax + 1). Theuser is free to move the transition point (white) inthe bi-color scale (blue to red) to focus his analysison a range of values. When drawing edge-bundles,we substitute the blue-red color scale with a white-black color scale for the HNMap in the backgroundto improve visibility of the bundles.

(a) Straight connection lines (b) Exploiting hierarchical structure (c) Compromise through edge bundles

Figure 4: Comparison of different strategies to draw adjacency relationships among nodes of the IP/AShierarchy on top of the HNMap.

In some cases, analyzing an absolute measureof network traffic (e.g., number of connections orbytes transferred) provides hardly any insight in thetime-varying dynamics of the data. Therefore, wecalculate and display the change over time in someanalysis scenarios.

5 Drawing routed traffic for multiplesources and destinations

Figure 4 shows three different approaches to drawadjacency relationships among nodes of the IP/AShierarchy. (a) The problem of visual clutter is obvi-ous when straight lines are drawn. (b) Simply con-necting with the upper level nodes of the hierarchyremoves visual clutter, but adds a lot of ambiguity,e.g. the lines connecting the left star (U.S.) and thestars in Europe (middle) – one for every country –are overdrawn several hundred times. (c) An inter-esting compromise is to use so-called edge bundlesand transparency effects to combine the advantagesof the latter approaches.

5.1 Edge Bundles

A recenty study proposed a way of using splinecurves to draw adjacency relationships amongnodes organized in a hierarchy [6]. Figure 5 il-lustrates how we use hierarchy structure to drawa spline curve between two leaf nodes. Pstart

(green) is the center of the source rectangle rep-resenting a node in the IP/AS hierarchy and Pend

(red) the center of the destination rectangle. Allpoints Pi (green, blue, and red) from Pstart overLCA(Pstart, Pend) (least common ancestor) to

Pend form the cubic B-spline’s control polygon.Because many splines share the same LCA (in thiscase the center point of the world rectangle), we donot use them for the control path.

EuropeNorth AmericaUS

DEAS123

AS553

Figure 5: Spline (red) with control polygon (gray)

The degree of the B-spline used to control thebundling strength. In our experiments, a degree of 6has proven to be a good choice for the cases wherewe have enough control points.

5.2 Edge Coloring

In general, we considered two options for coloringthe edges, namely (a) use of color to convey theamount of traffic transferred and (b) use of colorto distinguish edges. For the first case (see Fig. 6),we employed a heat map color scale from yellowto red using 50 to 0 percent alpha blending to vi-sually weight the high traffic links. Naturally, theless important splines were drawn first. For the sec-ond case (see Fig. 7), we chose a HSI color mapwith constant saturation and intensity which is sil-houetted against the background. The largely vary-ing colors make tracing of single splines easier, butwe noticed that users were strongly distracted by thecolorful display.

Figure 6: HNMap with edge bundles showing the 500 most important connections of one-day incoming andoutgoing traffic of our university gateway (the anonymized destination/source is semi-randomly chosen).Color combined with transparency and line width communicate the amount of traffic of each spline.

5.3 Interaction design

In our opinion, the task of tracing splines is notsolvable by means of coloring as soon as their num-ber excels about one hundred nodes. We thereforetried to tackle the problem through interaction. Ba-sically, there are two possible interaction scenarios.The first scenario is that a particular spline or re-gion is selected, refined, and visually highlighted.Selecting a spline or a region by the start and endpoints of the splines is relatively intuitive to imple-ment using the spatial data structure of the HNMapapplication.

However, the second scenario comprises markinga whole bundle of splines and was discarded sincethe implementation becomes tedious at this place(probably a point in polygon test for each pixel ofall splines).

After specifying the start or end points of thesplines using mouse interaction, we redraw allsplines using their previous RGB color values – the

not selected splines with higher alpha blending andthe selected ones without transparency effects ontop. This allows the user to easily trance the fewhighlighted splines to their end.

5.4 Data Simulation

As it was impossible to obtain real traffic data fromservice providers for publishing, we settled on us-ing real netflow data from our university gatewayand substituted the internal source IP prefix with arandomized one according to algorithm 1.

This randomization schema is based upon theassumption that nodes with more traffic are morelikely to communicate with several other nodes,whereas the opposite applies to low traffic nodes.

Figures 6 and 7 demonstrate the outcome of ourrandomization schema based upon a one-day trafficload of our university gateway. The images high-light the high-level connectivity information whilestill being able to recognize the low-level relations

Figure 7: HNMap with randomly colored edge bundles makes splines more distinguishable. The amount oftraffic is expressed only through spline width.

beginSortedList← all unprocessed nodes andweightswhile |SortedList| > 0 do

(nmin, tmin)←SortedList.removeMin()(nrand, trand)←SortedList.RandomSample()drawSpline(nmin, nrand, tmin)SortedList.update(nrand, trand −tmin)

end

Algorithm 1: Randomization schema to simulatebackbone provider traffic.

to a large extent. The strong bundling effects inNorth America (center left) are explainable throughthe proximity of two control points to each other(North America, United States).

6 Findings and Evaluation

Edge bundles offer the possibility to convey sourcedestination relationships on top of the HNMap andthus leverage the application from a purely mea-surement based analytical approach to a more com-plete view on large-scale network traffic. Since it isvery challenging to trace single splines as soon asa certain volume of traffic links are placed on themap, we experimented with the RGB and alpha val-ues of the spline colors. Mouse interaction is usedto select splines at their start or end points in orderto silhouet them from the remaining splines.

When monitoring larger networks, focusing theanalysis on a particular type of traffic, for examplecommunication of hijacked computers of a botnet,helps to significantly reduce the number of nodesand connections to be displayed.

During the analysis, further detailed informationof other attributes of the data set at hand can be dis-played using bar charts or the Radial Traffic Ana-lyzer [9]. One major drawback of our approach is

that conveying significance of splines or to distin-guishing them through color disqualifies the visualvariable color for being used to indicate direction oftraffic flows.

7 Conclusion

HNMaps visually represent network traffic aggre-gated on IP prefixes, ASes, countries, or conti-nents and can be used for exploration of largeIP-related data sets. In this study, we extendedHNMaps through edge bundles to emphasize thesource destination relationship of network traffic.Rather than representing new visualization or anal-ysis techniques, we combined two existing methodsand applied them to large-scale network traffic togain deeper insight.

In order to facilitate tracing of splines, which rep-resent source destination relationships among net-works or abstract nodes of the IP hierarchy, wecompared two alternative coloring schemes. More-over, mouse interaction was proposed to visuallyenhance network traffic links of interest.

References

[1] R. A. Becker, Stephen G. Eick, and A. R.Wilks. Visualizing network data. IEEETransactions on Visualization and ComputerGraphics, 1(1):16–21, March 1995.

[2] Benjamin B. Bederson, Ben Shneiderman,and Martin Wattenberg. Ordered and quantumtreemaps: Making effective use of 2d spaceto display hierarchies. ACM Trans. Graph.,21(4):833–854, 2002.

[3] M. Bruls, K. Huizing, and Jarke J. Van Wijk.Squarified treemaps. In Proceedings of theJoint Eurographics and IEEE TCVG Sympo-sium on Visualization, pages 33–42, 2000.

[4] Martin Dodge and Rob Kitchin. Atlas of Cy-berspace. Addison-Wesley, 2001.

[5] Roland Heilmann, Daniel A. Keim, ChristianPanse, and Mike Sips. RecMap: Rectangu-lar Map Approximations. In InfoVis 2004,IEEE Symposium on Information Visualiza-tion, Austin, Texas, pages 33–40, October2004.

[6] Danny Holten. Hierarchical Edge Bundles:Visualization of Adjacency Relations in Hi-erarchical Data. IEEE Transactions on Visu-

alization and Computer Graphics, 12(5):741–748, 2006.

[7] Takayuki Itoh, Hiroki Takakura, AtsushiSawada, and Koji Koyamada. Hierarchicalvisualization of network intrusion detectiondata. IEEE Computer Graphics and Applica-tions, 26(02):40–47, 2006.

[8] B. Johnson and Ben Shneiderman. Tree-maps:A space filling approach to the visualizationof hierarchical information structures. In VIS’91: Proceedings of the 2nd IEEE Conferenceon Visualization, pages 284–291, 1991.

[9] Daniel A. Keim, Florian Mansmann, JornSchneidewind, and Tobias Schreck. Monitor-ing network traffic with radial traffic analyzer.In Proc. of IEEE Symposium on Visual An-alytics Science and Technology 2006 (VAST2006), pages 123–128, 2006.

[10] Florian Mansmann, Daniel A. Keim,Stephen C. North, Brian Rexroad, andDaniel Shelehedal. Visual analysis of networktraffic for resource planning, interactivemonitoring, and interpretation of securitythreats. IEEE Transactions on Visualizationand Computer Graphics, 13(6), 2007.

[11] Florian Mansmann and Svetlana Vinnik. In-teractive Exploration of Data Traffic with Hi-erarchical Network Maps. IEEE Transac-tions on Visualization and Computer Graph-ics, 12(6):1440–1449, 2006.

[12] Maxmind, LLC. Geoip database, 2007.http://www.maxmind.com.

[13] T. B. Pedersen and C. S. Jensen. Multidimen-sional database technology. IEEE Computer,34(12):40–46, 2001.

[14] Doantam Phan, Ling Xiao, Ron Yeh, Pat Han-rahan, and Terry Winograd. Flow map layout.In INFOVIS ’05: Proceedings of the Proceed-ings of the 2005 IEEE Symposium on Informa-tion Visualization, page 29, Washington, DC,USA, 2005. IEEE Computer Society.

[15] Ben Shneiderman and Aleks Aris. Networkvisualization by semantic substrates. IEEETrans. Vis. Comput. Graph., 12(5):733–740,2006.

[16] Team Cymru. IP to ASN Lookup Page, April2007. http://www.cymru.com/BGP/asnlookup.html.