Swimming In Sensors, Drowning In Data— Big Data Analytics For Military …€¦ · · 2016-05-18INTRODUCTION In the integrated operating environment of the near-future, intelligence

INTRODUCTION

In the integrated operating environment of

the near-future, intelligence data will come from a

multitude of sources—from closed sources and open

sources; from air defence radars, weapon-locating

radars and maritime surveillance systems; from print

media, radio, blogs and Twitter; from individual ships,

aircraft and ground troopers. Data will come in the

form of video feeds, ground photographs, aerial

imagery, electronic signatures, radar returns, sound

clips, conversation transcripts, tactical reports,

digital text and paper printouts. Twitter generates

an average of 5,700 tweets per second.1 Given the

flight endurance of the Heron 1 UAV, a single mission

could produce a continuous video feed of more than

24 hours.2 If every soldier were a sensor (see Figure

1), the SAF could have more than 300,000 sensors

on the ground during wartime. If each registered

just one data point a week, there would be one data

point every two seconds.3 Yet, today’s methods of

exploitation and analysis still rely on a human analyst

to access, interpret and evaluate each piece of data,

sift out the valuable information from the noise, then

understand how the information fits into a whole. As

we build an SAF that “[sees] first and [sees] more,” we

need to adopt methods of exploitation and analysis

that will enable us to make sense of this swarming,

seething mass of data.4

In the last decade, the 3rd Generation (3rd Gen)

SAF has operationalised its vision of a networked

force with integrated strike capabilities that will give

us a decisive edge in war.5 The early attrition of a

potential adversary’s key infrastructure and assets—

many of which would be highly mobile—would

defang the adversary, boost the success of the SAF’s

operations and minimise the loss of Singaporean lives.

tech edge 51

POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1

Swimming In Sensors, Drowning In Data—Big Data Analytics For Military Intelligence

by ME4 Toh Bao En

Abstract:

With the Singapore Armed Forces (SAF) investing heavily in integrated strike and Command, Control,

Communications, Computers and Intelligence (C4I) capabilities, intelligence analysts are now faced with the

need to produce ever-more precise intelligence in the face of information overload—a deluge of data beyond

the ability of humans to process and understand. Big data analytics provide the ability to quantitatively

deal with the masses of information, as well as to qualitatively improve intelligence assessments by drawing

out patterns and insights from data. In this essay, the author briefly examines how defence and intelligence

agencies in other countries deal with big data and then outlines a model of what big data architecture would

entail and a vision of how data analytics will change the way intelligence analysis is performed. Finally, the

author proposes two approaches to seeding and implementing big data for intelligence in the SAF.

Keywords: Data; Intelligence; Analysis; Information; Integrate

51-65_.SwimmingInSensors.indd 51 1/3/16 3:51 PM

Figure 1: SAF’s Advanced Combat Man System (ACMS).6

Implicit in this vision is the information that fl ows

through the various networks. Precision Manoeuvre

and Precision Fire capabilities have an insatiable

appetite for an exponential volume of Precision

Information on targets and areas of operation—

intelligence that is accurate, geographically-precise

and real-time.7 Recognising this, the Ministry of

Defence (MINDEF) has also invested heavily in

Intelligence/Command, Control, Communications,

Computers and Intelligence (C4I) structures,

capabilities and human resources. Most visible of

these investments was the setting up of the C4I

Community in 2012, with a two-star general at the

helm.8 In the same year, the RSAF inaugurated its

Heron 1 UAVs, which will signifi cantly increase the

amount of imagery intelligence available to the

SAF.9

Yet, all our investments in strike and Command,

Control, Communications, Computers, Intelligence,

Surveillance and Reconnaissance (C4ISR) capabilities

will go to waste if the SAF is not able to translate its

volumes of perishable and time-sensitive data into

actionable intelligence for the shooter. Analysis is

the bottleneck; the numbers of human analysts are

fundamentally limited. Without a paradigm shift, the

SAF cannot keep producing more intelligence, not to

mention better intelligence. We need a radical change

in the way we exploit intelligence data—we need big

data analytics.

tech edge 52



Minister for Defence, Dr Ng Eng Hen at the Inauguration of the Heron 1 UAV.10

In the integrated operating environment of the near-future, intelligence data will come from a multitude of sources— from closed sources and open sources; from air defence radars, weapon-locating radars and maritime surveillance systems; from print media, radio, blogs and Twitter; from individual ships, aircraft and ground troopers.

WHAT IS BIG DATA?

In 2001, analyst Doug Laney of the META Group

identifi ed a new trend, a shift in “consciousness”

about “how data is managed.”11 With the surge

in e-commerce activity and in collaboration, he

predicted that traditional data management

methods would no longer be able to keep up with

the new ‘3Vs’ of data: Volume, Velocity and Variety.12

Data was and is being generated in unprecedented

quantities, at unprecedented speeds. Some of this

data, especially in organisations, is structured in

databases for easy access and management by a

computer. However, 95% of the digital universe is

unstructured and comes in a staggering variety of

formats including documents, images, audio, video,

transaction records and sensor data.13

The ‘3Vs’ apply not only to the supply of data but

also to the demand for information. Consumers—

whether an ‘Amazon’ customer, a strategic decision-

maker or a ground commander—are demanding

for more information at a faster rate, and are

asking questions that traditional databases are not

MIN

DEF

tech edge 53



designed to answer. McKinsey Global Institute

avoided defining ‘big data’ as having a specific

size, reasoning that the ability of an organisation

to handle volumes of data using “typical database

software tools” would vary by sector and time.14

In other words, the defining characteristic of big

data is the organisation’s discomfort with handling

it, using human-centric methods. Big data is, by

definition, beyond human capacity.15

DEFENDING WITH DATA

Although ‘Big Data’ has been the trend for the last 15

years, both commercial and government organisations

are still struggling to overcome its challenges and

make the most of its potential. Governments, and in

particular, defence and intelligence agencies, have

access to an enormous repository of data, but generally

face compounded issues in implementation—privacy

and security concerns, stove piped structures, lengthy

acquisition procedures, a lack of skilled people and

industry solutions geared toward business rather than

government service.16 In spite of this, governments

of countries such as Israel, the United States (US)

and the United Kingdom (UK) are deeply convinced of

the opportunities afforded and have taken big steps

toward building up their big data capabilities.

In 2014, several senior Hamas leaders were

eliminated in a series of targeted attacks. Separately,

the hunt for two terrorists responsible for the lives

of three Israeli teenagers took just three months.

"I am telling you with certainty that quite a few

terrorists are looking at us from the sky owing to

big data capabilities," said Ronen Horowitz, former

head of the Israel Security Agency (ISA).17 According

to him, the leadership of General Ayalon as early as

1996 catalysed the growth of the ISA’s information

capabilities, as critical resources were diverted

from elsewhere and invested in technology. The ISA

maintains close partnerships with industry experts

and research institutes while proactively reaching

out to operational forces, even sending consultants

to work with brigades deployed in the field. Horowitz

describes ISA analysts as the crème de la crème, with

a “highly methodological yet outside-the-box” way of

thinking.18 This is Israel’s recipe for big data success:

visionary leadership, investments, partnerships,

mission-focus and people.

The 2010 report on Networking and Information

Technology produced by the US President’s Council of

Advisors on Science and Technology (PCAST) boldly

advocated, “Every federal agency needs to have a

‘big data’ strategy.”19 Two years later, the Obama

administration unveiled a slew of ongoing Federal

big data programmes. Under the Defense Advanced

Research Projects Agency (DARPA) alone were

nine initiatives. The Anomaly Detection at

Multiple Scales (ADAMS) Programme could detect

anomalies in massive datasets, while the XDATA

programme aimed to develop computational

techniques and tools for handling and visualising

volumes of imperfect and unstructured data. The

Cyber-Insider Threat (CINDER) programme sought to

expose potential attacks on networks by comparison

with predicted models of ‘adversary missions’.

A number of projects worked on developing the

artificial intelligence to understand nuances in text

and activities in images or videos.20 Meanwhile,

in a remote corner of Utah, the National Security

Agency (NSA) was building a controversial US$1.9

billion data centre to house ‘yottabytes’—a trillion

terabytes, multiplied—of personal data.21 The

National Geospatial-Intelligence Agency (NGA) in

2014 reiterated its belief in the power of human-

machine analysis in its ambitious and inspiring “2020

Analysis Technology Plan”.22 Even the US Marine Corps

is now equipping its field intelligence specialists not

just with rifles, but with ‘Wikipedia-like’ interfaces

to query and connect data.23

tech edge 54



Observing these developments from across the

pond, the UK government in 2013 commissioned

think-tank Royal United Services Institute (RUSI)

to produce a report on ‘Big Data for Defence

and Security’. Ironically, given the US’ apparent

emphasis on big data technologies, the report took

aim at the US military’s incompetence in handling

information overload with an estimate that 95% of

its battlefield video data was never viewed, let alone

assessed.24 The report stated the need for change in

no uncertain terms—“The consequences of ignoring

Big Data… could be profound, including the loss of

life and operational failure.”25 The recommendations

were circumspect—to cultivate security culture in

tandem with the drive toward greater data sharing;

to develop institutional competence in big data

technologies by tapping on industry partners and

expertise in the reserve forces; to balance growth in

sensor capabilities with the ability to exploit; to have

dedicated senior leaders drive the efforts to harness

big data; and to kickstart those efforts with focused

and independent pilot projects.26

This brief survey of how big data has been used

or implemented in various defence and intelligence

agencies shows the promise of the technology,

as well as a reassuring spread of capabilities and

comfort levels. Common to all of them is a deep

conviction of the need to deal with big data—not

just as an ‘opportunity’ or an ‘enabler’ but as an

imperative, without which the mission and survival

of the organisation would be compromised. Their

motivation to harness big data resonates strongly

with the SAF, while their best practices, strategies

and recommendations are strikingly relevant to our

context. Looking outwards will enable us to fast-

forward our progress through learning from others

and encourage creative, cross-disciplinary methods

to unlock the value of our data.

A MODEL FOR BIG DATA ARCHITECTURE

The characteristics of big data described earlier

demand a robust architecture that can handle massive

end-to-end flows. We have also seen how other

defence agencies have dealt with and benefited from

setting up such big data systems and architectures.

What would it take for Intelligence in the SAF to

exploit its big data?

The first element of a big data architecture is

storage, access and computation capacity—the

hardware. The storage and computation capacity must

be able to accommodate the volumetric demands of

big data, while the access networks must be able

to handle rapid and massive flows of traffic. Cloud

computing is the widely-accepted solution, providing

centralised access to powerful processors and massive

storage servers while minimising data duplication.

Of course, in an intelligence ecosystem, the desire

to centralise must be weighed against the need to

segregate classified systems and information.

The second element is to set up the algorithms

and models to conduct data integration and

data analytics. Data integration refers to the

transformation of the mass of unstructured data into

structured data which can be used and analysed,

through machine-automated capture of the content

and provenance of each piece of data. This process

in itself may require sophisticated algorithms

that can parse non-textual content such as audio

and video, which segues into data analytics. Data

analytics is essentially artificial intelligence that

can derive meaning from massive datasets. This

artificial intelligence is powered by models, which

are sequences of analytical algorithms that attempt

to replicate human reasoning. Data analytics is the

real generator of value—I will deal more thoroughly

with its applications in the next section.

tech edge 55



The third element is the user interface, which

mediates the interactions between human and

machine. The software interface and even physical

infrastructure must cater to two groups of people

with different needs: the analysts and the end-users.

Analysts demand full access to raw data and fast

processing speeds, preferring to have complex and

rich data that can be manipulated. A user interface

for analysts should provide multiple views of data

and a comprehensive analytical toolbox. End-users,

especially in a military context, require very specifi c

and relevant information for rapid decision-making.

An end-user interface for a diverse audience must

be able to query and display critical information in

an intuitive and responsive manner.27 Ultimately,

if intelligence is not delivered to the end-user in a

timely, useful and actionable form, it is wasted.

The fi nal element is a body of skilled people.

Statisticians, systems experts and data scientists are

needed to manage, maintain and enhance the data

infrastructure, while human analysts, programmers

and information engineers would develop the

algorithms and animate the analytical models with

their insights and experience. Data analytics is not a

substitute for experience, insight and deep expertise.

What it does is codify and systematise that analytical

expertise so that it can be propagated across the

organisation—so that the furthest-fl ung department

and most junior operators are able to benefi t from it,

so that human effort can be applied with maximum

effi ciency, and so that institutional knowledge will

outlast the individual.28 Deep expertise, captured in

analytical models and algorithms, multiplies in value

when it is used to draw meaning from and is in turn

enriched by massive datasets.

DATA ANALYTICS FOR INTELLIGENCE ANALYSIS

In my introduction I painted a picture of an

information-saturated battlespace with numerous

data sources, producing data, in multiple formats at

overwhelming speeds. The fi rst challenge is therefore

data integration, or converting unstructured big

data to structured data. Data types like audio, video,

imagery and handwriting are easy for humans to

understand but challenging for computers. Machine

learning refers to the set of algorithms that enables

computers to evolve behavior in response to data

and feedback, to approach a ‘human’ understanding

of such data types.29 One important application is

natural language processing, which allows a computer

to make sense of spoken and written words, and

even their nuances. This technology could be used

to transcribe communication intercepts, audio/ video

clips, news reports and handwritten or hardcopy

documents; data with key words could be fl agged out

for a human analyst’s attention. Another application

is automatic target detection in imagery or video

feeds, where the computer is trained to recognise and

track target signatures, or through anomaly detection,

where the computer identifi es pixel clusters that are

signifi cantly different from their surroundings. Again,

possible targets and anomalies would be highlighted

for verifi cation by a human analyst. As the report

“Designing a Digital Future” put it, “Automation can

track the mundane, and cue the human to attend to

what is interesting, suspicious, and relevant.”30

A deep understanding of a target, be it a

country, organisation, person or military unit,

typically requires a sustained, long-term research

effort. With or without prior study, data analytics

can help a human analyst to quickly understand a

target or subject—its characteristics, interests and

perspectives, or doctrines and patterns of life—as

long as a large historical dataset is available. Data

mining and statistics are basic analytical tools that

allow analysts to fi nd data relevant to their target

and to gain a sense of the spread and content of

tech edge 56



that data. Data mining could be used to sieve out

activities or information associated with events of

concern, while statistics would provide some baseline

norms for target behaviour—future occurrences of

those activity or information indicators as well as

deviations from the baseline would cue analysts to

pay closer attention.31

Visualisation is a valuable technique for

displaying information in a summarised manner

that allows human analysts to quickly spot patterns

and anomalies. It is a simple but effective way of

combining what computers are good at—computation

and processing—with what humans are good at—

pattern-recognition, creative thinking and making

cognitive connections.32 A word tree (Figure 2) or tag

cloud (Figure 3) derived from the speeches made by

an individual could highlight the issues of greatest

concern to him or her. Hotspot mapping (Figure 4)

could illustrate the concentration of target sightings

in an area, while a spatial information fl ow (Figure 5)

would provide a sense of a region’s network activity.

More advanced analytical algorithms can be used to

distill meaning from data. Network analysis maps out

the relationships and key nodes between entities—

applying it to a set of recorded transmissions could

suggest which callsigns were associated with key

decision-makers, while a similar analysis on a

social network could identify relationship clusters

and the major infl uencers in a group (Figure 6).

Time series analysis relates data points in time to

Figure 2: Word Tree.33 Figure 3: Tag Cloud.34

Figure 4: Hotspot Map.35 Figure 5: Spatial Information Flow.36

tech edge 57



extract correlations and identify cyclical patterns.

Examining a particular unit or military asset could

reveal peaks of activity occurring over specific

weeks of the year, painting a picture of the target’s

pattern of life, and perhaps even correlating

certain peaks with specific exercise schedules.

These analytics tools are not just quantitative;

they also allow us to manage and sense-make

masses of data. They are also qualitative, helping

to overcome analyst bias with system logic, making

previously unseen associations; allowing analysts to

derive more rigorous, data-supported assessments

and insights.

The same data mining, statistical and analytical

tools used to extract research insights are even more

critical in short-cycle operations support. Rapid

sense-making for strike requires analysts to generate

target assessments within minutes so as to provide

precise and actionable intelligence to shooters. To

automate and speed up as much of the analysis process

as possible, a human analyst can arrange analytical

tools in a sequence known as a model, which can then

be automatically run to answer the same question

repeatedly, with modifi ed parameters and a different

set of data.

Figure 6: Link Analysis.37

Big data is changing the way intelligence

operates. Today, the SAF’s Intelligence Cycle is

Direction, Collection, Exploitation, Assessment and

Dissemination. Other organisations’ intelligence

cycles may vary in semantics but not in spirit. This

intelligence cycle begins with a question, a user’s

need for specifi c information. The analysts would

decide what data was needed and the collectors

would gather that data. The analysts would analyse

the collected data and deliver the answer to the

user. With really big data, the data is there before

it is demanded for. As a result, the approach many

organisations take is to “collect everything and

then search for signifi cant patterns in the data.”38

This is starting to be described as the “Query, Mine,

Assemble, Disseminate and Integrate” (QMADI) cycle.39

Without planning, data might not always tell us what

we want to know, but it could also tell us what we

never knew we needed to know.

Before big data analytics, such patterns, insights

and assessments would have required countless hours

to generate, or might even have been missed. Data

analytics cannot replace human intuition, curiosity,

tech edge 58



creativity and expertise.

What it can do, however,

is free the human analyst

to focus on the quality of

analysis and to keep an eye

on the big picture—Are

my sources reliable and

my assumptions sound?

How can my analytical

processes and models be improved? Are we missing

the forest for the trees? These are the questions that

only a human analyst can ask and answer.40

FROM CONCEPT TO REALITY

Implementing big data architecture in the SAF

is certainly a daunting enterprise—as seen in the

overview of other defence and intelligence agencies. I

see two concurrent inroads to seeding and ultimately

establishing big data capabilities in the SAF: bottom-

up and top-down.

To see immediate returns, we should begin where

the value is—with data analytics.41 The advantage of

data analytic algorithms is that they are accessible

and useful to individual units or departments.

While developed to handle quantitative big data,

most data analytics techniques and algorithms are

highly scalable and can easily be used to provide

qualitative improvements in how existing ‘small’

data is handled. The algorithms are also the easiest

part of the ecosystem to set up—with software

available commercially ‘off-the-shelf ’, a single

revolutionary algorithm could be used to solve a

problem “a thousand times faster than conventional

computational methods.”42 This approach allows

individual departments to see the value of data

analytics for themselves and for analysts to become

adept at big data approaches.

Today, the SAF has

begun using software

to bring Geospatial

Information Systems

(GIS) capabilities to

units and departments

across the three services.

GIS has been especially

useful in supporting

Humanitarian Assistance and Disaster Relief (HADR)

operations, when analysts need to help deployment

teams quickly familiarise themselves with new and

unstable areas of operations, and where information

from a multitude of agencies fl ow in rapidly and must

be quickly assimilated. I see GIS as a more location-

focused precursor of big data—inbuilt in GIS systems

are data management structures for a set of large data

formats, machine learning and analytical algorithms,

tools for building analytical models, and tailored

interfaces to help both analysts and end-users make

sense of information. We should build on our existing

capabilities in this area to seed the concept of a big

data ecosystem across the SAF and help stakeholders

see what it can do for them. Our experience with GIS

also provides invaluable lessons about the structural

and systemic obstacles that need to be overcome,

such as the diffi culty of information sharing and the

limited bandwidth and storage of current IT systems,

so that we can clear the path for big data.

To overcome systemic obstacles, strategic

leadership is key. A visionary leader at the highest

levels is needed to provide direction and emphasis

on developing the capability across the organisation.

The leader must oversee the end-to-end development

of the sensor-exploiter-shooter pipeline, to ensure

that resources invested in all parts are commensurate

and that a single bottleneck or point of failure does

not bring all efforts to naught. The leader is in a

Data analytics cannot replace human intuition, curiosity, creativity and expertise. What it can do, however, is free the human analyst to focus on the quality of analysis and to keep an eye on the big picture

tech edge 59



position to coordinate and create overarching systems

and structures, both organisational and physical. A

dedicated position or office could be set up or tasked

to create the C4, security and information-sharing

infrastructure, processes and policies to accommodate

and enable big data solutions. “Skill sets remain

the biggest challenge” for organisations like the

NGA.43 A dedicated office would have the mandate

to proactively bring in the human expertise we lack;

these data and information systems professionals can

then build and drive big data efforts centrally and

develop organisation-wide systems and standards in

ways far beyond the scope of individual units.

GIS has been especially useful in supporting Humanitarian Assistance and Disaster Relief operations, when analysts need to help deployment teams quickly familiarise themselves with new and unstable areas of operations, and where information from a multitude of agencies flow in rapidly and must be quickly assimilated.

CONCLUSION

Big data has already changed the game. Either

we learn to ride the wave early and maintain that

headstart, or we allow ourselves to drown in the

data deluge while others surge ahead. It will be an

immense challenge but, like many other challenges,

promises to be rewarding. Big data will compel us to

build the architecture, systems and capabilities to

handle it. Big data will demand more highly skilled

people even as it provides an avenue to relieve our

manpower constraints. Intelligence in the age of big

data will reveal different and surprising insights, and

would force us to consider new perspectives and ways

of thinking. Willing or not, big data will evict us from

our comfort zones, and open our eyes to what this

brave new world has to show us.

As an intelligence practitioner in the SAF, it is my

firm conviction that what I do every day contributes

directly to the defence and security of Singapore.

While those who serve in other arms are prepared

to give their lives for this nation when called upon,

it is my duty to ensure that they never have to—

by detecting threats before they materialise, by

understanding how to deal with them through

physical as well as through deterrent and diplomatic

means, and ultimately by guiding our forces to swiftly

and decisively neutralise them before they can take

out a single one of our own. Big data could allow us

to do all this faster and better. With all the sensors

and sources that we have, big data could potentially

provide us with critical, life-saving information, if

we could only recognise and capitalise on it. Can

we stomach a tragedy arising out of an intelligence

“failure to connect the dots”?44 Do we not owe it to

our soldiers and citizens to process, exploit, integrate

and make sense of all information that could impact

their life and death? Harnessing big data is not just

a luxury—it is an obligation and an imperative.

BIBLIOGRAPHY

Ackerman, Robert K. “Multiple Thrusts Define Geospatial Agency Big Data Efforts.”

SIGNAL. 1 August 2014. http://www.afcea.org/content/?q=multiple-thrusts-define-geospatialagency-big-data-efforts.

Bamford, James. “The NSA Is Building the Country’s Biggest

Spy Center (Watch What You Say).” 15 March 2012. http://

www.wired.com/2012/03/ff_nsadatacenter/all/.

tech edge 60



Bertolucci, Jeff. “Military Uses Big Data As Spy

Tech.” InformationWeek. 22 April 2013. http://www.

informat ionweek.com/big-data/big-data-analy t ics/military-uses-big-data-as-spytech/d/d-id/1109641?.

Blasch, Erik P., Stephen Russell and Guna Seetharaman. “Joint Data Management for MOVINT: Data-to-Decision Making.” Proceedings of the 14th International Conference on Information Fusion (2011). http://www.nrl.navy.mil/itd/imda/sites/www.nrl.navy.mil.itd.imda/files/pdfs/Fusion11_JDM_110126.pdf.

“Challenges and Opportunities with Big Data.” November 2012 to February 2013.

http://www.purdue.edu/discoverypark/cyber/assets/pdfs/BigDataWhitePaper.pdf.

Couch, Neil and Bill Robins. “Big Data for Defence and Security.” Royal United Services Institute. September 2013. https://www.rusi.org/downloads/assets/RUSI_BIGDATAReport_2013.pdf.

Davenport, Thomas H. and Jill Dyché. “Big Data in Big Companies.” SAS Institute. May 2013. http://www.sas.com/reg/gen/corp/2266746.

Davenport, Thomas H., Paul Barth and Randy Bean. “How ‘Big Data’ Is Different.” MIT

Sloan Management Review. 30 July 2012. http://sloanreview.mit.edu/article/how-big-data-is different/.

Edwards, John. “Big Data Takes a Strategic Turn at DoD.” C4ISR and Networks. 21 November 2014. http://archive.c4isrnet.com/article/20141118/C4ISRNET14/311180008/Big data-takes-strategic-turn-DoD.

Edwards, John. “Military, Intel Turn to Big Data for Better Situational Awareness.” Federal

Times. 2 June 2014. http://archive.federaltimes.com/article/20140602/FEDIT /306020009/Military-intel-turn-big-data-better-situational-awareness.

Erwin, Sandra I. “As Defense, Intelligence Agencies Drown in Data, Technology Comes to

the Rescue.” National Defense Industrial Association. November 2014. http://www.nationaldefensemagazine.org/archive/2014/November/Pages/AsDefenseIntelligence AgenciesDrowninDataTechnologyComestotheRescue.aspx.

Executive Office of the President. “Big Data Across the Federal Government.” 29 March 2012. http://www.whitehouse.gov/sites/default/files/microsites/ostp/big_data_fact_sheet_final.pdf.

Fahey, Sean. “Big Data and Analytics for National Security” (presentation, Workshop on

Algorithms for Modern Massive Data Sets, Stanford, CA, 11 July 2012). http://web.stanford.edu/group/mmds/

slides2012/s-fahey.pdf.

Fargnoli, Joseph D. “Big Data Enables Activity & Location Based Predictive Analytics

Applications.” Ritre Corporation (presentation, Government Big Data Symposium, Arlington, VA, 30 November 2013).

http://semanticommunity.info/@api/deki/f iles/27182/FARGNOLITTC_Gov_Bigdata_Conf_fl1.pdf.

George, Roger Z. and James B. Bruce. “Intelligence Analysis—The Emergence of a

Discipline.” Analysing Intelligence: Origins, Obstacles, and Innovations. Ed. Roger Z. George and James B. Bruce (Washington DC: Georgetown University Press, 2008).

Gorg, Carsten, Youn-ah Kang, Zhicheng Liu and John Stasko. “Visual Analytics Support for

Intelligence Analysis.” http://www.cc.gatech.edu/~stasko/papers/computer13-intell.pdf.

Hoffman, Mike. “‘Big data’ Poses Big Problem for Pentagon.” DefenseTech. 20 February 2013. http://defensetech.org/2013/02/20/big-data-poses-big-problem-for-pentagon/.

Howard, Courtney. “Dealing with Big Data.” Military and Aerospace Electronics. 1 December 2012. http://www.militaryaerospace.com/articles/print/volume-23/issue12/technology-focus/dealing-with-big-data.html.

Hunt, Ira. “The CIA’s ‘Grand Challenges’ with Big Data.” Central Intelligence Agency (presentation, GigaOM Structure 2013, San Francisco, CA, 16-17 Oct 2013). http://www.slideshare.net/morellimarc/central-intelligence-agency-gigaom2013?next_slideshow=1.

“IBM i2 Enterprise Insight Analysis for Defense Intelligence.” IBM Corporation. December2013. http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=BR&infotype=

PM&appname=SWGE_ZZ_ZZ_USEN&htmlfid=ZZB03045USEN&attachment=ZZB03045U EN.PDF#loaded.

Laney, Doug. “3D Data Management: Controlling Data Volume, Velocity, and Variety.”

META Group. 6 February 2001. http://blogs.gartner.com/doug-laney/f iles/2012/01/ad949-3D-DataManagement-Controlling Data-Volume-Velocity-and-Variety.pdf.

LaValle, Steve, Eric Lesser, Rebecca Shockley, Michael S. Hopkins and Nina Kruschwitz.

“Big Data, Analytics and the Path from Insights to Value.” MIT Sloan Management Review 52, no. 2 (Winter 2011): 21-31. http://www.ibm.com/smarterplanet/global/files/in_idea_smarter_computing_to_big-data-analytics_and_path_from_insights-to-value.pdf.

Lohr, Steve. “The Age of Big Data.” The New York Times. 11 February 2012.http://www.nytimes.com/2012/02/12/sunday-rev iew/big-datas-impact-in-theworld.html?_r=2&scp=1&sq=Big%20Data&st=cse&.

tech edge 61



Manyika, James, Michael Chui, Brad Brown, Jacques Bughin, Richard Dobbs, Charles

Roxburgh and Angela Hung Byers. “Big Data: The Next Frontier for Innovation, Competition,and Productivity,” McKinsey Global Institute. May 2011. http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovaion.

Marshall, Patrick. “Battlespace awareness through big data.” GCN. 18 November 2014.http://gcn.com/Blogs/Emerging-Tech/2014/11/Battlespace-awareness.aspx.

McNulty, Eileen. “US Military Using Big Data to Improve Situational Awareness.”

Dataconomy. 12 June 2014. http://dataconomy.com/us-military-using-big-data-improvesituational-awareness/.

National Geospatial-Intelligence Agency (NGA). “2020 Analysis Technology Plan.” August2014.https://www1.nga.mil/MediaRoom/PressReleases/Documents/NGA_Analysis_Tech_Plan.pdf.

Dr. Ng Eng Hen. “Speech by Minister for Defence Dr. Ng Eng Hen at the C4I Community Inauguration Parade.” Ministry of Defence, Singapore. 2 April 2012. http://www.mindef.gov.sg/imindef/press_room/official_releases/sp/2012/02apr12_speech.html#.VO4c4PmUfhu.

President’s Council of Advisors on Science and Technology (PCAST). “Report to the

President and Congress: Designing a Digital Future: Federally Funded Research and

Development in Networking and Information Technology.” December 2010.http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-nitrd-report-2010.pdf.

“‘Quite a Few Terrorists Lost Their Lives Owing to Big Data.’” Israel Defense. 3 Jan 2015.http://www.israeldefense.com/?CategoryID=484&ArticleID=3288.

Rooney, Ben. “U.K. Military Failing to Tackle Big Data.” The Wall Street Journal. 23

September 2013. http://blogs.wsj.com/tech-europe/2013/09/23/u-k-military-failing-to-tackle-bigdata/.

Savas, Onur, Yalin Sagduyu, Julia Deng and Jason Li. “Tactical Big Data Analytics:

Challenges,Use Cases, and Solutions.” Intelligent Automation Inc. 21 June 2013.http://www.sigmetrics.org/sigmetr ics2013/bigdataanalyt ics/abstracts2013/bdaw2013_submission_12.pdf.

Shaw, Jonathan. “Why ‘Big Data’ Is a Big Deal.” Harvard Magazine. March – April 2014.http://harvardmagazine.com/2014/03/why-big-data-is-a-big-deal.

Wood, Colin. “How Does the Military Use Big Data?” 6 Jan 2014.http://www.emergencymgmt.com/safety/Military-Use-Big-Data.html.

Woodie, Alex. “How Analytics is Driving Military Intelligence.” Datanami.com. 3 Feb 2014. http://www.datanami.com/2014/02/03/how_analytics_is_dr iving_military_intelligence/.

Young, Chris. “Military Intelligence Redefined: Big Data in the Battlefield.” Forbes. 12 March 2012. http://www.forbes.com/sites/techonomy/2012/03/12/military-intelligence redefined-big-data-in-the-battlefield/.

ENDNOTES

1. Raffi Krikorian, “New Tweets per second record, and how!” (blog), Twitter, 16 August 2013, https://blog.twitter.com/2013/new-tweets-per-second-record-and-how.

2. Tan Guan Wei, “RSAF welcomes inauguration of Heron 1 UAV,” Ministry of Defence, Singapore, 13 May 2012, http://www.mindef.gov.sg/imindef/resourcelibrary/c y b e r p i o n e e r/ t o p i c s / a r t i c l e s / n e w s / 2 0 1 2 /may/23may12_news.html#.VO4YKfmUfht.

3. Gordon Arthur, “‘Sense & Sensorbility’ - Singapore Leads Asian Future Soldier Systems,” Defence Review Asia, 3 June 2012, http://www.defencereviewasia.com/articles/167/SENSE-SENSORBILITY-SINGAPORE-LEADS-ASIAN-FUTURE-SOLDIER-SYSTEMS.

Estimated from the statement, “‘If only 50% of active and reserve strength of the SAF were to visit the museum only once a year,’ the study group maintained, ‘the museum would have a visitorship in excess of 150 000.’”

Dr. Albert Lau, “Towards an SAF Museum,” Ministry of Defence, Singapore, 25 April 2011, http://www.mindef.gov.sg/imindef/mindef_websites/atozlistings/army/microsites/armymuseum/army_museum_singapore/About_Us/towards_an_saf_museum.html.

4. Dr. Ng Eng Hen, “Speech by Minister for Defence Dr Ng Eng Hen at the C4I Community Inauguration Parade,” Ministry of Defence, Singapore, 2 April 2012, ht tp://www.mindef.gov.sg/imindef/press_room/of f icial_releases/sp/2012/02apr12_speech.html#.VO4c4PmUfhu.

5. “Advanced Combat Man System”, ST Electronics, accessed 27 Feb 2015, http://www.stee.stengg.com/pdf/ ACMS.pdf.

6. Ministry of Defence, Singapore, “3rd Generation SAF,” 20 August 2013, Ministry of Defence, Singapore, http://www.mindef.gov.sg/imindef/key_topics/3rd_generation_saf.html.

tech edge 62



7. “Overview of SAF,” Ministry of Defence, Singapore, 28 July 2011, http://www.mindef.gov.sg/imindef/mindef_websites/atozlist ings/army/microsites/paccpams/abt_spore/saf.html. Although this frame of Precision Warfare being Precision Manoeuvre, Precision Fires and Precision Information is an Army concept, I find it useful for describing the Joint context as well.

8. Dr. Ng Eng Hen, “Speech at the C4I Community Inauguration Parade.”

Jermyn Chow, “Military chief to get second star in rank,” The Straits Times, 30 June 2013, http://news.asiaone.com/print/News/Latest%2BNews/Singapore/Story/A1Story20130629-433414.html.

9. Tan, “RSAF welcomes inauguration of Heron 1 UAV.”

10. “Photo Gallery: RSAF Inaugurates the Heron 1 UAV into 119 Squadron”, Ministry of Defence, Singapore, 23 May 2012, http://www.mindef.gov.sg/imindef/press_room/official_releases/nr/2012/may/23may12_nr/23may12_ photos.html#.VPFh3vmUfhs.

11. Doug Laney, “3D Data Management: Controlling Data Volume, Velocity, and Variety,” META Group, 6 February 2001, 1-2, http://blogs.gartner.com/doug-laney/f iles/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf.

12. Ibid.

13. Erik P. Blasch, Stephen Russell and Guna Seetharaman, “Joint Data Management for MOVINT: Data-to-Decision Making,” Proceedings of the 14th International Conference on Information Fusion (2011): 3, http://www.nrl.navy.mil/itd/imda/sites/www.nrl.navy.mil.itd.imda/files/pdfs/Fusion11_JDM_110126.pdf.

14. James Manyika, Michael Chui, Brad Brown, Jacques Bughin, Richard Dobbs, Charles Roxburgh and Angela Hung Byers, “Big Data: The Next Frontier for Innovation, Competition, and Productivity,” McKinsey Global Institute, May 2011, 1, http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation.

15. This may not be a very enlightening definition, but it certainly forces us to discard traditional mental models of data management and analysis.

16. Sean Fahey, “Big Data and Analytics for National Security.”

Ira Hunt, “The CIA’s ‘Grand Challenges’ with Big Data,” Central Intelligence Agency, (presentation, GigaOM Structure 2013, San Francisco, CA, 16-17 Oct 2013), ht tp://www.slideshare.net/morel l imarc/central-intelligence-agency-gigaom-2013?next_slideshow=1.

17. “‘Quite a Few Terrorists Lost Their Lives Owing to Big Data,’” IsraelDefense, 3 Jan 2015, http://www.israeldefense.com/?CategoryID=484&ArticleID=3288.

18. Ibid.

19. President’s Council of Advisors on Science and Technology (PCAST), “Report to the President and Congress: Designing a Digital Future: Federally Funded Research and Development in Networking and Information Technology,” December 2010, xvii, http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-nitrd-report-2010.pdf.

20. Executive Office of the President, “Big Data Across the Federal Government,” 29 March 2012, 1-2, http://www.whitehouse.gov/sites/default/files/microsites/ostp/big_data_fact_sheet_final.pdf.

21. James Bamford, “The NSA Is Building the Country’s Biggest Spy Center (Watch What You Say),” Wired.com, 15 March 2012, http://www.wired.com/2012/03/ff_nsadatacenter/all/.

22. National Geospatial-Intelligence Agency (NGA), “2020 Analysis Technology Plan,” August 2014, https://www1.nga.mil/MediaRoom/PressReleases/Documents/NGA_Analysis_Tech_Plan.pdf.

23. Alex Woodie, “How Analytics is Driving Military Intelligence,” Datanami.com, 3 Feb 2014, http://www.datanami.com/2014/02/03/how_analytics_is_driving_military_intelligence/.

24. Ibid.

25. Neil Couch and Bill Robins, “Big Data for Defence and Security,” Royal United Services Institute, September 2013, 26, https://www.rusi.org/downloads/assets/RUSI_BIGDATA_Report_2013.pdf.

26. Ibid., 26-29.

27. The end-users I have in mind are operational planners in networked command posts. Designing a user interface for military end-users in theatre and on the ground will be far more complex. Secured networks must be in place; hardware and software must be able to function in inhospitable and low-bandwidth environments; information displays may need to be scalable for different devices. If the soldier is a “sensor” or data source as well as a user, the system will need to account for this added dimension.

28. Colin Wood, “How Does the Military Use Big Data?”, 6 Jan 2014, http://www.emergencymgmt.com/safety/Military-Use-Big-Data.html.

29. McKinsey Global Institute, “Big Data: The Next Frontier,” 29.

tech edge 63



30. PCAST, “Designing a Digital Future,” 27.

31. NGA, “2020 Analysis Technology Plan,” 7.

32. Jonathan Shaw, “Why ‘Big Data’ Is a Big Deal,” Harvard Magazine, March – April 2014, http://harvardmagazine.com/2014/03/why-big-data-is-a-big-deal.

33. Carsten Gorg, Youn-ah Kang, Zhicheng Liu and John Stasko, “Visual Analytics Support for Intelligence Analysis,” 5, http://www.cc.gatech.edu/~stasko/papers/computer13-intell.pdf.


35. “Creating Heat Maps with Bing Maps and Dynamics CRM,” Microsoft Corporation, (blog, 29 October 2012), http://blogs.msdn.com/b/crm/archive/2012/10/29/creating-heat-maps-with-bing-maps-and-dynamics-crm.aspx


37. “The Next Generation of Big Data Visualization,” Sentinel Visualizer, accessed 25 February 2015, http://www.fmsasg.com/Products/SentinelVisualizer.

38. Shaw, “Why ‘Big Data’ Is a Big Deal.”

39. Joseph D. Fargnoli, “Big Data Enables Activity & Location Based Predictive Analytics Applications,” Ritre Corporation, (presentation, Government Big Data Symposium, Arlington, VA, 30 November 2013), http://semanticommunity.info/@api/deki/files/27182/FARGNOLI-TTC_Gov_Bigdata_Conf_fl13.pdf.

40. NGA, “2020 Analysis Technology Plan,” 6; John Edwards, “Military, Intel Turn to Big Data for Better Situational Awareness,” Federal Times, 2 June 2014,

http://archive.federaltimes.com/article/20140602/FEDIT/306020009/Mil i t ar y-inte l-turn-big-data-better-situational-awareness.

41. Steve LaValle, Eric Lesser, Rebecca Shockley, Michael S. Hopkins and Nina Kruschwitz, “Big Data, Analytics and the Path from Insights to Value,” MIT Sloan Management Review 52, no. 2 (Winter 2011): 21-31, http://www.ibm .com/smarterplanet/global/files/in_idea_smarter_computing_to_big-data-analytics_and_path_from_insights-to-value.pdf.

42. Jonathan Shaw, “Why ‘Big Data’ Is a Big Deal,” Harvard Magazine, March – April 2014, http://harvardmagazine .com/2014/03/why-big-data-is-a-big-deal.

43. Robert K. Ackerman, “Multiple Thrusts Define Geospatial Agency Big Data Efforts,” SIGNAL, 1 August 2014, http://www.afcea.org/content/?q=multiple-thrusts-define-geospatial-agency-big-data-efforts.

44. The phrase was used specifically with reference to intelligence failure prior to 9/11. I hesitate to use 9/11 as a throwaway example and thereby oversimplify the events leading up to it or implicitly suggest that big data could have prevented it, but the point remains that reasons like “failure to connect the dots” seem like frail excuses following a tragedy. Roger Z. George and James B. Bruce, “Intelligence Analysis—The Emergence of a Discipline,” in Analyzing Intelligence: Origins, Obstacles, and Innovations, ed. Roger Z. George and James B. Bruce (Washington DC: Georgetown University Press, 2008), 4.

tech edge 64



ME4 Toh Bao En is currently a Staff Offi cer in Joint Intelligence Department. A recipient of the SAF Academic Scholarship, she graduated from the University of Michigan in 2011 with a Bachelor of Arts in English (Highest Honours). She has also served in the 30th Battalion, Singapore Combat Engineers and the Imagery Support Group.

tech edge 65

