INTRODUCTION
In the integrated operating environment of
the near-future, intelligence data will come from a
multitude of sources—from closed sources and open
sources; from air defence radars, weapon-locating
radars and maritime surveillance systems; from print
media, radio, blogs and Twitter; from individual ships,
aircraft and ground troopers. Data will come in the
form of video feeds, ground photographs, aerial
imagery, electronic signatures, radar returns, sound
clips, conversation transcripts, tactical reports,
digital text and paper printouts. Twitter generates
an average of 5,700 tweets per second.1 Given the
flight endurance of the Heron 1 UAV, a single mission
could produce a continuous video feed of more than
24 hours.2 If every soldier were a sensor (see Figure
1), the SAF could have more than 300,000 sensors
on the ground during wartime. If each registered
just one data point a week, there would be one data
point every two seconds.3 Yet, today’s methods of
exploitation and analysis still rely on a human analyst
to access, interpret and evaluate each piece of data,
sift out the valuable information from the noise, then
understand how the information fits into a whole. As
we build an SAF that “[sees] first and [sees] more,” we
need to adopt methods of exploitation and analysis
that will enable us to make sense of this swarming,
seething mass of data.4
In the last decade, the 3rd Generation (3rd Gen)
SAF has operationalised its vision of a networked
force with integrated strike capabilities that will give
us a decisive edge in war.5 The early attrition of a
potential adversary’s key infrastructure and assets—
many of which would be highly mobile—would
defang the adversary, boost the success of the SAF’s
operations and minimise the loss of Singaporean lives.
tech edge 51
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
Swimming In Sensors, Drowning In Data—Big Data Analytics For Military Intelligence
by ME4 Toh Bao En
Abstract:
With the Singapore Armed Forces (SAF) investing heavily in integrated strike and Command, Control,
Communications, Computers and Intelligence (C4I) capabilities, intelligence analysts are now faced with the
need to produce ever-more precise intelligence in the face of information overload—a deluge of data beyond
the ability of humans to process and understand. Big data analytics provide the ability to quantitatively
deal with the masses of information, as well as to qualitatively improve intelligence assessments by drawing
out patterns and insights from data. In this essay, the author briefly examines how defence and intelligence
agencies in other countries deal with big data and then outlines a model of what big data architecture would
entail and a vision of how data analytics will change the way intelligence analysis is performed. Finally, the
author proposes two approaches to seeding and implementing big data for intelligence in the SAF.
Keywords: Data; Intelligence; Analysis; Information; Integrate
51-65_.SwimmingInSensors.indd 51 1/3/16 3:51 PM
Figure 1: SAF’s Advanced Combat Man System (ACMS).6
Implicit in this vision is the information that fl ows
through the various networks. Precision Manoeuvre
and Precision Fire capabilities have an insatiable
appetite for an exponential volume of Precision
Information on targets and areas of operation—
intelligence that is accurate, geographically-precise
and real-time.7 Recognising this, the Ministry of
Defence (MINDEF) has also invested heavily in
Intelligence/Command, Control, Communications,
Computers and Intelligence (C4I) structures,
capabilities and human resources. Most visible of
these investments was the setting up of the C4I
Community in 2012, with a two-star general at the
helm.8 In the same year, the RSAF inaugurated its
Heron 1 UAVs, which will signifi cantly increase the
amount of imagery intelligence available to the
SAF.9
Yet, all our investments in strike and Command,
Control, Communications, Computers, Intelligence,
Surveillance and Reconnaissance (C4ISR) capabilities
will go to waste if the SAF is not able to translate its
volumes of perishable and time-sensitive data into
actionable intelligence for the shooter. Analysis is
the bottleneck; the numbers of human analysts are
fundamentally limited. Without a paradigm shift, the
SAF cannot keep producing more intelligence, not to
mention better intelligence. We need a radical change
in the way we exploit intelligence data—we need big
data analytics.
tech edge 52
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 52 1/3/16 3:51 PM
Minister for Defence, Dr Ng Eng Hen at the Inauguration of the Heron 1 UAV.10
In the integrated operating environment of the near-future, intelligence data will come from a multitude of sources— from closed sources and open sources; from air defence radars, weapon-locating radars and maritime surveillance systems; from print media, radio, blogs and Twitter; from individual ships, aircraft and ground troopers.
WHAT IS BIG DATA?
In 2001, analyst Doug Laney of the META Group
identifi ed a new trend, a shift in “consciousness”
about “how data is managed.”11 With the surge
in e-commerce activity and in collaboration, he
predicted that traditional data management
methods would no longer be able to keep up with
the new ‘3Vs’ of data: Volume, Velocity and Variety.12
Data was and is being generated in unprecedented
quantities, at unprecedented speeds. Some of this
data, especially in organisations, is structured in
databases for easy access and management by a
computer. However, 95% of the digital universe is
unstructured and comes in a staggering variety of
formats including documents, images, audio, video,
transaction records and sensor data.13
The ‘3Vs’ apply not only to the supply of data but
also to the demand for information. Consumers—
whether an ‘Amazon’ customer, a strategic decision-
maker or a ground commander—are demanding
for more information at a faster rate, and are
asking questions that traditional databases are not
MIN
DEF
tech edge 53
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 53 1/3/16 3:51 PM
designed to answer. McKinsey Global Institute
avoided defining ‘big data’ as having a specific
size, reasoning that the ability of an organisation
to handle volumes of data using “typical database
software tools” would vary by sector and time.14
In other words, the defining characteristic of big
data is the organisation’s discomfort with handling
it, using human-centric methods. Big data is, by
definition, beyond human capacity.15
DEFENDING WITH DATA
Although ‘Big Data’ has been the trend for the last 15
years, both commercial and government organisations
are still struggling to overcome its challenges and
make the most of its potential. Governments, and in
particular, defence and intelligence agencies, have
access to an enormous repository of data, but generally
face compounded issues in implementation—privacy
and security concerns, stove piped structures, lengthy
acquisition procedures, a lack of skilled people and
industry solutions geared toward business rather than
government service.16 In spite of this, governments
of countries such as Israel, the United States (US)
and the United Kingdom (UK) are deeply convinced of
the opportunities afforded and have taken big steps
toward building up their big data capabilities.
In 2014, several senior Hamas leaders were
eliminated in a series of targeted attacks. Separately,
the hunt for two terrorists responsible for the lives
of three Israeli teenagers took just three months.
"I am telling you with certainty that quite a few
terrorists are looking at us from the sky owing to
big data capabilities," said Ronen Horowitz, former
head of the Israel Security Agency (ISA).17 According
to him, the leadership of General Ayalon as early as
1996 catalysed the growth of the ISA’s information
capabilities, as critical resources were diverted
from elsewhere and invested in technology. The ISA
maintains close partnerships with industry experts
and research institutes while proactively reaching
out to operational forces, even sending consultants
to work with brigades deployed in the field. Horowitz
describes ISA analysts as the crème de la crème, with
a “highly methodological yet outside-the-box” way of
thinking.18 This is Israel’s recipe for big data success:
visionary leadership, investments, partnerships,
mission-focus and people.
The 2010 report on Networking and Information
Technology produced by the US President’s Council of
Advisors on Science and Technology (PCAST) boldly
advocated, “Every federal agency needs to have a
‘big data’ strategy.”19 Two years later, the Obama
administration unveiled a slew of ongoing Federal
big data programmes. Under the Defense Advanced
Research Projects Agency (DARPA) alone were
nine initiatives. The Anomaly Detection at
Multiple Scales (ADAMS) Programme could detect
anomalies in massive datasets, while the XDATA
programme aimed to develop computational
techniques and tools for handling and visualising
volumes of imperfect and unstructured data. The
Cyber-Insider Threat (CINDER) programme sought to
expose potential attacks on networks by comparison
with predicted models of ‘adversary missions’.
A number of projects worked on developing the
artificial intelligence to understand nuances in text
and activities in images or videos.20 Meanwhile,
in a remote corner of Utah, the National Security
Agency (NSA) was building a controversial US$1.9
billion data centre to house ‘yottabytes’—a trillion
terabytes, multiplied—of personal data.21 The
National Geospatial-Intelligence Agency (NGA) in
2014 reiterated its belief in the power of human-
machine analysis in its ambitious and inspiring “2020
Analysis Technology Plan”.22 Even the US Marine Corps
is now equipping its field intelligence specialists not
just with rifles, but with ‘Wikipedia-like’ interfaces
to query and connect data.23
tech edge 54
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 54 1/3/16 3:51 PM
Observing these developments from across the
pond, the UK government in 2013 commissioned
think-tank Royal United Services Institute (RUSI)
to produce a report on ‘Big Data for Defence
and Security’. Ironically, given the US’ apparent
emphasis on big data technologies, the report took
aim at the US military’s incompetence in handling
information overload with an estimate that 95% of
its battlefield video data was never viewed, let alone
assessed.24 The report stated the need for change in
no uncertain terms—“The consequences of ignoring
Big Data… could be profound, including the loss of
life and operational failure.”25 The recommendations
were circumspect—to cultivate security culture in
tandem with the drive toward greater data sharing;
to develop institutional competence in big data
technologies by tapping on industry partners and
expertise in the reserve forces; to balance growth in
sensor capabilities with the ability to exploit; to have
dedicated senior leaders drive the efforts to harness
big data; and to kickstart those efforts with focused
and independent pilot projects.26
This brief survey of how big data has been used
or implemented in various defence and intelligence
agencies shows the promise of the technology,
as well as a reassuring spread of capabilities and
comfort levels. Common to all of them is a deep
conviction of the need to deal with big data—not
just as an ‘opportunity’ or an ‘enabler’ but as an
imperative, without which the mission and survival
of the organisation would be compromised. Their
motivation to harness big data resonates strongly
with the SAF, while their best practices, strategies
and recommendations are strikingly relevant to our
context. Looking outwards will enable us to fast-
forward our progress through learning from others
and encourage creative, cross-disciplinary methods
to unlock the value of our data.
A MODEL FOR BIG DATA ARCHITECTURE
The characteristics of big data described earlier
demand a robust architecture that can handle massive
end-to-end flows. We have also seen how other
defence agencies have dealt with and benefited from
setting up such big data systems and architectures.
What would it take for Intelligence in the SAF to
exploit its big data?
The first element of a big data architecture is
storage, access and computation capacity—the
hardware. The storage and computation capacity must
be able to accommodate the volumetric demands of
big data, while the access networks must be able
to handle rapid and massive flows of traffic. Cloud
computing is the widely-accepted solution, providing
centralised access to powerful processors and massive
storage servers while minimising data duplication.
Of course, in an intelligence ecosystem, the desire
to centralise must be weighed against the need to
segregate classified systems and information.
The second element is to set up the algorithms
and models to conduct data integration and
data analytics. Data integration refers to the
transformation of the mass of unstructured data into
structured data which can be used and analysed,
through machine-automated capture of the content
and provenance of each piece of data. This process
in itself may require sophisticated algorithms
that can parse non-textual content such as audio
and video, which segues into data analytics. Data
analytics is essentially artificial intelligence that
can derive meaning from massive datasets. This
artificial intelligence is powered by models, which
are sequences of analytical algorithms that attempt
to replicate human reasoning. Data analytics is the
real generator of value—I will deal more thoroughly
with its applications in the next section.
tech edge 55
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 55 1/3/16 3:51 PM
The third element is the user interface, which
mediates the interactions between human and
machine. The software interface and even physical
infrastructure must cater to two groups of people
with different needs: the analysts and the end-users.
Analysts demand full access to raw data and fast
processing speeds, preferring to have complex and
rich data that can be manipulated. A user interface
for analysts should provide multiple views of data
and a comprehensive analytical toolbox. End-users,
especially in a military context, require very specifi c
and relevant information for rapid decision-making.
An end-user interface for a diverse audience must
be able to query and display critical information in
an intuitive and responsive manner.27 Ultimately,
if intelligence is not delivered to the end-user in a
timely, useful and actionable form, it is wasted.
The fi nal element is a body of skilled people.
Statisticians, systems experts and data scientists are
needed to manage, maintain and enhance the data
infrastructure, while human analysts, programmers
and information engineers would develop the
algorithms and animate the analytical models with
their insights and experience. Data analytics is not a
substitute for experience, insight and deep expertise.
What it does is codify and systematise that analytical
expertise so that it can be propagated across the
organisation—so that the furthest-fl ung department
and most junior operators are able to benefi t from it,
so that human effort can be applied with maximum
effi ciency, and so that institutional knowledge will
outlast the individual.28 Deep expertise, captured in
analytical models and algorithms, multiplies in value
when it is used to draw meaning from and is in turn
enriched by massive datasets.
DATA ANALYTICS FOR INTELLIGENCE ANALYSIS
In my introduction I painted a picture of an
information-saturated battlespace with numerous
data sources, producing data, in multiple formats at
overwhelming speeds. The fi rst challenge is therefore
data integration, or converting unstructured big
data to structured data. Data types like audio, video,
imagery and handwriting are easy for humans to
understand but challenging for computers. Machine
learning refers to the set of algorithms that enables
computers to evolve behavior in response to data
and feedback, to approach a ‘human’ understanding
of such data types.29 One important application is
natural language processing, which allows a computer
to make sense of spoken and written words, and
even their nuances. This technology could be used
to transcribe communication intercepts, audio/ video
clips, news reports and handwritten or hardcopy
documents; data with key words could be fl agged out
for a human analyst’s attention. Another application
is automatic target detection in imagery or video
feeds, where the computer is trained to recognise and
track target signatures, or through anomaly detection,
where the computer identifi es pixel clusters that are
signifi cantly different from their surroundings. Again,
possible targets and anomalies would be highlighted
for verifi cation by a human analyst. As the report
“Designing a Digital Future” put it, “Automation can
track the mundane, and cue the human to attend to
what is interesting, suspicious, and relevant.”30
A deep understanding of a target, be it a
country, organisation, person or military unit,
typically requires a sustained, long-term research
effort. With or without prior study, data analytics
can help a human analyst to quickly understand a
target or subject—its characteristics, interests and
perspectives, or doctrines and patterns of life—as
long as a large historical dataset is available. Data
mining and statistics are basic analytical tools that
allow analysts to fi nd data relevant to their target
and to gain a sense of the spread and content of
tech edge 56
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 56 1/3/16 3:51 PM
that data. Data mining could be used to sieve out
activities or information associated with events of
concern, while statistics would provide some baseline
norms for target behaviour—future occurrences of
those activity or information indicators as well as
deviations from the baseline would cue analysts to
pay closer attention.31
Visualisation is a valuable technique for
displaying information in a summarised manner
that allows human analysts to quickly spot patterns
and anomalies. It is a simple but effective way of
combining what computers are good at—computation
and processing—with what humans are good at—
pattern-recognition, creative thinking and making
cognitive connections.32 A word tree (Figure 2) or tag
cloud (Figure 3) derived from the speeches made by
an individual could highlight the issues of greatest
concern to him or her. Hotspot mapping (Figure 4)
could illustrate the concentration of target sightings
in an area, while a spatial information fl ow (Figure 5)
would provide a sense of a region’s network activity.
More advanced analytical algorithms can be used to
distill meaning from data. Network analysis maps out
the relationships and key nodes between entities—
applying it to a set of recorded transmissions could
suggest which callsigns were associated with key
decision-makers, while a similar analysis on a
social network could identify relationship clusters
and the major infl uencers in a group (Figure 6).
Time series analysis relates data points in time to
Figure 2: Word Tree.33 Figure 3: Tag Cloud.34
Figure 4: Hotspot Map.35 Figure 5: Spatial Information Flow.36
tech edge 57
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 57 1/3/16 3:51 PM
extract correlations and identify cyclical patterns.
Examining a particular unit or military asset could
reveal peaks of activity occurring over specific
weeks of the year, painting a picture of the target’s
pattern of life, and perhaps even correlating
certain peaks with specific exercise schedules.
These analytics tools are not just quantitative;
they also allow us to manage and sense-make
masses of data. They are also qualitative, helping
to overcome analyst bias with system logic, making
previously unseen associations; allowing analysts to
derive more rigorous, data-supported assessments
and insights.
The same data mining, statistical and analytical
tools used to extract research insights are even more
critical in short-cycle operations support. Rapid
sense-making for strike requires analysts to generate
target assessments within minutes so as to provide
precise and actionable intelligence to shooters. To
automate and speed up as much of the analysis process
as possible, a human analyst can arrange analytical
tools in a sequence known as a model, which can then
be automatically run to answer the same question
repeatedly, with modifi ed parameters and a different
set of data.
Figure 6: Link Analysis.37
Big data is changing the way intelligence
operates. Today, the SAF’s Intelligence Cycle is
Direction, Collection, Exploitation, Assessment and
Dissemination. Other organisations’ intelligence
cycles may vary in semantics but not in spirit. This
intelligence cycle begins with a question, a user’s
need for specifi c information. The analysts would
decide what data was needed and the collectors
would gather that data. The analysts would analyse
the collected data and deliver the answer to the
user. With really big data, the data is there before
it is demanded for. As a result, the approach many
organisations take is to “collect everything and
then search for signifi cant patterns in the data.”38
This is starting to be described as the “Query, Mine,
Assemble, Disseminate and Integrate” (QMADI) cycle.39
Without planning, data might not always tell us what
we want to know, but it could also tell us what we
never knew we needed to know.
Before big data analytics, such patterns, insights
and assessments would have required countless hours
to generate, or might even have been missed. Data
analytics cannot replace human intuition, curiosity,
tech edge 58
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 58 1/3/16 3:51 PM
creativity and expertise.
What it can do, however,
is free the human analyst
to focus on the quality of
analysis and to keep an eye
on the big picture—Are
my sources reliable and
my assumptions sound?
How can my analytical
processes and models be improved? Are we missing
the forest for the trees? These are the questions that
only a human analyst can ask and answer.40
FROM CONCEPT TO REALITY
Implementing big data architecture in the SAF
is certainly a daunting enterprise—as seen in the
overview of other defence and intelligence agencies. I
see two concurrent inroads to seeding and ultimately
establishing big data capabilities in the SAF: bottom-
up and top-down.
To see immediate returns, we should begin where
the value is—with data analytics.41 The advantage of
data analytic algorithms is that they are accessible
and useful to individual units or departments.
While developed to handle quantitative big data,
most data analytics techniques and algorithms are
highly scalable and can easily be used to provide
qualitative improvements in how existing ‘small’
data is handled. The algorithms are also the easiest
part of the ecosystem to set up—with software
available commercially ‘off-the-shelf ’, a single
revolutionary algorithm could be used to solve a
problem “a thousand times faster than conventional
computational methods.”42 This approach allows
individual departments to see the value of data
analytics for themselves and for analysts to become
adept at big data approaches.
Today, the SAF has
begun using software
to bring Geospatial
Information Systems
(GIS) capabilities to
units and departments
across the three services.
GIS has been especially
useful in supporting
Humanitarian Assistance and Disaster Relief (HADR)
operations, when analysts need to help deployment
teams quickly familiarise themselves with new and
unstable areas of operations, and where information
from a multitude of agencies fl ow in rapidly and must
be quickly assimilated. I see GIS as a more location-
focused precursor of big data—inbuilt in GIS systems
are data management structures for a set of large data
formats, machine learning and analytical algorithms,
tools for building analytical models, and tailored
interfaces to help both analysts and end-users make
sense of information. We should build on our existing
capabilities in this area to seed the concept of a big
data ecosystem across the SAF and help stakeholders
see what it can do for them. Our experience with GIS
also provides invaluable lessons about the structural
and systemic obstacles that need to be overcome,
such as the diffi culty of information sharing and the
limited bandwidth and storage of current IT systems,
so that we can clear the path for big data.
To overcome systemic obstacles, strategic
leadership is key. A visionary leader at the highest
levels is needed to provide direction and emphasis
on developing the capability across the organisation.
The leader must oversee the end-to-end development
of the sensor-exploiter-shooter pipeline, to ensure
that resources invested in all parts are commensurate
and that a single bottleneck or point of failure does
not bring all efforts to naught. The leader is in a
Data analytics cannot replace human intuition, curiosity, creativity and expertise. What it can do, however, is free the human analyst to focus on the quality of analysis and to keep an eye on the big picture
tech edge 59
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 59 1/3/16 3:51 PM
position to coordinate and create overarching systems
and structures, both organisational and physical. A
dedicated position or office could be set up or tasked
to create the C4, security and information-sharing
infrastructure, processes and policies to accommodate
and enable big data solutions. “Skill sets remain
the biggest challenge” for organisations like the
NGA.43 A dedicated office would have the mandate
to proactively bring in the human expertise we lack;
these data and information systems professionals can
then build and drive big data efforts centrally and
develop organisation-wide systems and standards in
ways far beyond the scope of individual units.
GIS has been especially useful in supporting Humanitarian Assistance and Disaster Relief operations, when analysts need to help deployment teams quickly familiarise themselves with new and unstable areas of operations, and where information from a multitude of agencies flow in rapidly and must be quickly assimilated.
CONCLUSION
Big data has already changed the game. Either
we learn to ride the wave early and maintain that
headstart, or we allow ourselves to drown in the
data deluge while others surge ahead. It will be an
immense challenge but, like many other challenges,
promises to be rewarding. Big data will compel us to
build the architecture, systems and capabilities to
handle it. Big data will demand more highly skilled
people even as it provides an avenue to relieve our
manpower constraints. Intelligence in the age of big
data will reveal different and surprising insights, and
would force us to consider new perspectives and ways
of thinking. Willing or not, big data will evict us from
our comfort zones, and open our eyes to what this
brave new world has to show us.
As an intelligence practitioner in the SAF, it is my
firm conviction that what I do every day contributes
directly to the defence and security of Singapore.
While those who serve in other arms are prepared
to give their lives for this nation when called upon,
it is my duty to ensure that they never have to—
by detecting threats before they materialise, by
understanding how to deal with them through
physical as well as through deterrent and diplomatic
means, and ultimately by guiding our forces to swiftly
and decisively neutralise them before they can take
out a single one of our own. Big data could allow us
to do all this faster and better. With all the sensors
and sources that we have, big data could potentially
provide us with critical, life-saving information, if
we could only recognise and capitalise on it. Can
we stomach a tragedy arising out of an intelligence
“failure to connect the dots”?44 Do we not owe it to
our soldiers and citizens to process, exploit, integrate
and make sense of all information that could impact
their life and death? Harnessing big data is not just
a luxury—it is an obligation and an imperative.
BIBLIOGRAPHY
Ackerman, Robert K. “Multiple Thrusts Define Geospatial Agency Big Data Efforts.”
SIGNAL. 1 August 2014. http://www.afcea.org/content/?q=multiple-thrusts-define-geospatialagency-big-data-efforts.
Bamford, James. “The NSA Is Building the Country’s Biggest
Spy Center (Watch What You Say).” 15 March 2012. http://
www.wired.com/2012/03/ff_nsadatacenter/all/.
tech edge 60
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 60 1/3/16 3:51 PM
Bertolucci, Jeff. “Military Uses Big Data As Spy
Tech.” InformationWeek. 22 April 2013. http://www.
informat ionweek.com/big-data/big-data-analy t ics/military-uses-big-data-as-spytech/d/d-id/1109641?.
Blasch, Erik P., Stephen Russell and Guna Seetharaman. “Joint Data Management for MOVINT: Data-to-Decision Making.” Proceedings of the 14th International Conference on Information Fusion (2011). http://www.nrl.navy.mil/itd/imda/sites/www.nrl.navy.mil.itd.imda/files/pdfs/Fusion11_JDM_110126.pdf.
“Challenges and Opportunities with Big Data.” November 2012 to February 2013.
http://www.purdue.edu/discoverypark/cyber/assets/pdfs/BigDataWhitePaper.pdf.
Couch, Neil and Bill Robins. “Big Data for Defence and Security.” Royal United Services Institute. September 2013. https://www.rusi.org/downloads/assets/RUSI_BIGDATAReport_2013.pdf.
Davenport, Thomas H. and Jill Dyché. “Big Data in Big Companies.” SAS Institute. May 2013. http://www.sas.com/reg/gen/corp/2266746.
Davenport, Thomas H., Paul Barth and Randy Bean. “How ‘Big Data’ Is Different.” MIT
Sloan Management Review. 30 July 2012. http://sloanreview.mit.edu/article/how-big-data-is different/.
Edwards, John. “Big Data Takes a Strategic Turn at DoD.” C4ISR and Networks. 21 November 2014. http://archive.c4isrnet.com/article/20141118/C4ISRNET14/311180008/Big data-takes-strategic-turn-DoD.
Edwards, John. “Military, Intel Turn to Big Data for Better Situational Awareness.” Federal
Times. 2 June 2014. http://archive.federaltimes.com/article/20140602/FEDIT /306020009/Military-intel-turn-big-data-better-situational-awareness.
Erwin, Sandra I. “As Defense, Intelligence Agencies Drown in Data, Technology Comes to
the Rescue.” National Defense Industrial Association. November 2014. http://www.nationaldefensemagazine.org/archive/2014/November/Pages/AsDefenseIntelligence AgenciesDrowninDataTechnologyComestotheRescue.aspx.
Executive Office of the President. “Big Data Across the Federal Government.” 29 March 2012. http://www.whitehouse.gov/sites/default/files/microsites/ostp/big_data_fact_sheet_final.pdf.
Fahey, Sean. “Big Data and Analytics for National Security” (presentation, Workshop on
Algorithms for Modern Massive Data Sets, Stanford, CA, 11 July 2012). http://web.stanford.edu/group/mmds/
slides2012/s-fahey.pdf.
Fargnoli, Joseph D. “Big Data Enables Activity & Location Based Predictive Analytics
Applications.” Ritre Corporation (presentation, Government Big Data Symposium, Arlington, VA, 30 November 2013).
http://semanticommunity.info/@api/deki/f iles/27182/FARGNOLITTC_Gov_Bigdata_Conf_fl1.pdf.
George, Roger Z. and James B. Bruce. “Intelligence Analysis—The Emergence of a
Discipline.” Analysing Intelligence: Origins, Obstacles, and Innovations. Ed. Roger Z. George and James B. Bruce (Washington DC: Georgetown University Press, 2008).
Gorg, Carsten, Youn-ah Kang, Zhicheng Liu and John Stasko. “Visual Analytics Support for
Intelligence Analysis.” http://www.cc.gatech.edu/~stasko/papers/computer13-intell.pdf.
Hoffman, Mike. “‘Big data’ Poses Big Problem for Pentagon.” DefenseTech. 20 February 2013. http://defensetech.org/2013/02/20/big-data-poses-big-problem-for-pentagon/.
Howard, Courtney. “Dealing with Big Data.” Military and Aerospace Electronics. 1 December 2012. http://www.militaryaerospace.com/articles/print/volume-23/issue12/technology-focus/dealing-with-big-data.html.
Hunt, Ira. “The CIA’s ‘Grand Challenges’ with Big Data.” Central Intelligence Agency (presentation, GigaOM Structure 2013, San Francisco, CA, 16-17 Oct 2013). http://www.slideshare.net/morellimarc/central-intelligence-agency-gigaom2013?next_slideshow=1.
“IBM i2 Enterprise Insight Analysis for Defense Intelligence.” IBM Corporation. December2013. http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=BR&infotype=
PM&appname=SWGE_ZZ_ZZ_USEN&htmlfid=ZZB03045USEN&attachment=ZZB03045U EN.PDF#loaded.
Laney, Doug. “3D Data Management: Controlling Data Volume, Velocity, and Variety.”
META Group. 6 February 2001. http://blogs.gartner.com/doug-laney/f iles/2012/01/ad949-3D-DataManagement-Controlling Data-Volume-Velocity-and-Variety.pdf.
LaValle, Steve, Eric Lesser, Rebecca Shockley, Michael S. Hopkins and Nina Kruschwitz.
“Big Data, Analytics and the Path from Insights to Value.” MIT Sloan Management Review 52, no. 2 (Winter 2011): 21-31. http://www.ibm.com/smarterplanet/global/files/in_idea_smarter_computing_to_big-data-analytics_and_path_from_insights-to-value.pdf.
Lohr, Steve. “The Age of Big Data.” The New York Times. 11 February 2012.http://www.nytimes.com/2012/02/12/sunday-rev iew/big-datas-impact-in-theworld.html?_r=2&scp=1&sq=Big%20Data&st=cse&.
tech edge 61
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 61 1/3/16 3:51 PM
Manyika, James, Michael Chui, Brad Brown, Jacques Bughin, Richard Dobbs, Charles
Roxburgh and Angela Hung Byers. “Big Data: The Next Frontier for Innovation, Competition,and Productivity,” McKinsey Global Institute. May 2011. http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovaion.
Marshall, Patrick. “Battlespace awareness through big data.” GCN. 18 November 2014.http://gcn.com/Blogs/Emerging-Tech/2014/11/Battlespace-awareness.aspx.
McNulty, Eileen. “US Military Using Big Data to Improve Situational Awareness.”
Dataconomy. 12 June 2014. http://dataconomy.com/us-military-using-big-data-improvesituational-awareness/.
National Geospatial-Intelligence Agency (NGA). “2020 Analysis Technology Plan.” August2014.https://www1.nga.mil/MediaRoom/PressReleases/Documents/NGA_Analysis_Tech_Plan.pdf.
Dr. Ng Eng Hen. “Speech by Minister for Defence Dr. Ng Eng Hen at the C4I Community Inauguration Parade.” Ministry of Defence, Singapore. 2 April 2012. http://www.mindef.gov.sg/imindef/press_room/official_releases/sp/2012/02apr12_speech.html#.VO4c4PmUfhu.
President’s Council of Advisors on Science and Technology (PCAST). “Report to the
President and Congress: Designing a Digital Future: Federally Funded Research and
Development in Networking and Information Technology.” December 2010.http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-nitrd-report-2010.pdf.
“‘Quite a Few Terrorists Lost Their Lives Owing to Big Data.’” Israel Defense. 3 Jan 2015.http://www.israeldefense.com/?CategoryID=484&ArticleID=3288.
Rooney, Ben. “U.K. Military Failing to Tackle Big Data.” The Wall Street Journal. 23
September 2013. http://blogs.wsj.com/tech-europe/2013/09/23/u-k-military-failing-to-tackle-bigdata/.
Savas, Onur, Yalin Sagduyu, Julia Deng and Jason Li. “Tactical Big Data Analytics:
Challenges,Use Cases, and Solutions.” Intelligent Automation Inc. 21 June 2013.http://www.sigmetrics.org/sigmetr ics2013/bigdataanalyt ics/abstracts2013/bdaw2013_submission_12.pdf.
Shaw, Jonathan. “Why ‘Big Data’ Is a Big Deal.” Harvard Magazine. March – April 2014.http://harvardmagazine.com/2014/03/why-big-data-is-a-big-deal.
Wood, Colin. “How Does the Military Use Big Data?” 6 Jan 2014.http://www.emergencymgmt.com/safety/Military-Use-Big-Data.html.
Woodie, Alex. “How Analytics is Driving Military Intelligence.” Datanami.com. 3 Feb 2014. http://www.datanami.com/2014/02/03/how_analytics_is_dr iving_military_intelligence/.
Young, Chris. “Military Intelligence Redefined: Big Data in the Battlefield.” Forbes. 12 March 2012. http://www.forbes.com/sites/techonomy/2012/03/12/military-intelligence redefined-big-data-in-the-battlefield/.
ENDNOTES
1. Raffi Krikorian, “New Tweets per second record, and how!” (blog), Twitter, 16 August 2013, https://blog.twitter.com/2013/new-tweets-per-second-record-and-how.
2. Tan Guan Wei, “RSAF welcomes inauguration of Heron 1 UAV,” Ministry of Defence, Singapore, 13 May 2012, http://www.mindef.gov.sg/imindef/resourcelibrary/c y b e r p i o n e e r/ t o p i c s / a r t i c l e s / n e w s / 2 0 1 2 /may/23may12_news.html#.VO4YKfmUfht.
3. Gordon Arthur, “‘Sense & Sensorbility’ - Singapore Leads Asian Future Soldier Systems,” Defence Review Asia, 3 June 2012, http://www.defencereviewasia.com/articles/167/SENSE-SENSORBILITY-SINGAPORE-LEADS-ASIAN-FUTURE-SOLDIER-SYSTEMS.
Estimated from the statement, “‘If only 50% of active and reserve strength of the SAF were to visit the museum only once a year,’ the study group maintained, ‘the museum would have a visitorship in excess of 150 000.’”
Dr. Albert Lau, “Towards an SAF Museum,” Ministry of Defence, Singapore, 25 April 2011, http://www.mindef.gov.sg/imindef/mindef_websites/atozlistings/army/microsites/armymuseum/army_museum_singapore/About_Us/towards_an_saf_museum.html.
4. Dr. Ng Eng Hen, “Speech by Minister for Defence Dr Ng Eng Hen at the C4I Community Inauguration Parade,” Ministry of Defence, Singapore, 2 April 2012, ht tp://www.mindef.gov.sg/imindef/press_room/of f icial_releases/sp/2012/02apr12_speech.html#.VO4c4PmUfhu.
5. “Advanced Combat Man System”, ST Electronics, accessed 27 Feb 2015, http://www.stee.stengg.com/pdf/ ACMS.pdf.
6. Ministry of Defence, Singapore, “3rd Generation SAF,” 20 August 2013, Ministry of Defence, Singapore, http://www.mindef.gov.sg/imindef/key_topics/3rd_generation_saf.html.
tech edge 62
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 62 1/3/16 3:51 PM
7. “Overview of SAF,” Ministry of Defence, Singapore, 28 July 2011, http://www.mindef.gov.sg/imindef/mindef_websites/atozlist ings/army/microsites/paccpams/abt_spore/saf.html. Although this frame of Precision Warfare being Precision Manoeuvre, Precision Fires and Precision Information is an Army concept, I find it useful for describing the Joint context as well.
8. Dr. Ng Eng Hen, “Speech at the C4I Community Inauguration Parade.”
Jermyn Chow, “Military chief to get second star in rank,” The Straits Times, 30 June 2013, http://news.asiaone.com/print/News/Latest%2BNews/Singapore/Story/A1Story20130629-433414.html.
9. Tan, “RSAF welcomes inauguration of Heron 1 UAV.”
10. “Photo Gallery: RSAF Inaugurates the Heron 1 UAV into 119 Squadron”, Ministry of Defence, Singapore, 23 May 2012, http://www.mindef.gov.sg/imindef/press_room/official_releases/nr/2012/may/23may12_nr/23may12_ photos.html#.VPFh3vmUfhs.
11. Doug Laney, “3D Data Management: Controlling Data Volume, Velocity, and Variety,” META Group, 6 February 2001, 1-2, http://blogs.gartner.com/doug-laney/f iles/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf.
12. Ibid.
13. Erik P. Blasch, Stephen Russell and Guna Seetharaman, “Joint Data Management for MOVINT: Data-to-Decision Making,” Proceedings of the 14th International Conference on Information Fusion (2011): 3, http://www.nrl.navy.mil/itd/imda/sites/www.nrl.navy.mil.itd.imda/files/pdfs/Fusion11_JDM_110126.pdf.
14. James Manyika, Michael Chui, Brad Brown, Jacques Bughin, Richard Dobbs, Charles Roxburgh and Angela Hung Byers, “Big Data: The Next Frontier for Innovation, Competition, and Productivity,” McKinsey Global Institute, May 2011, 1, http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation.
15. This may not be a very enlightening definition, but it certainly forces us to discard traditional mental models of data management and analysis.
16. Sean Fahey, “Big Data and Analytics for National Security.”
Ira Hunt, “The CIA’s ‘Grand Challenges’ with Big Data,” Central Intelligence Agency, (presentation, GigaOM Structure 2013, San Francisco, CA, 16-17 Oct 2013), ht tp://www.slideshare.net/morel l imarc/central-intelligence-agency-gigaom-2013?next_slideshow=1.
17. “‘Quite a Few Terrorists Lost Their Lives Owing to Big Data,’” IsraelDefense, 3 Jan 2015, http://www.israeldefense.com/?CategoryID=484&ArticleID=3288.
18. Ibid.
19. President’s Council of Advisors on Science and Technology (PCAST), “Report to the President and Congress: Designing a Digital Future: Federally Funded Research and Development in Networking and Information Technology,” December 2010, xvii, http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-nitrd-report-2010.pdf.
20. Executive Office of the President, “Big Data Across the Federal Government,” 29 March 2012, 1-2, http://www.whitehouse.gov/sites/default/files/microsites/ostp/big_data_fact_sheet_final.pdf.
21. James Bamford, “The NSA Is Building the Country’s Biggest Spy Center (Watch What You Say),” Wired.com, 15 March 2012, http://www.wired.com/2012/03/ff_nsadatacenter/all/.
22. National Geospatial-Intelligence Agency (NGA), “2020 Analysis Technology Plan,” August 2014, https://www1.nga.mil/MediaRoom/PressReleases/Documents/NGA_Analysis_Tech_Plan.pdf.
23. Alex Woodie, “How Analytics is Driving Military Intelligence,” Datanami.com, 3 Feb 2014, http://www.datanami.com/2014/02/03/how_analytics_is_driving_military_intelligence/.
24. Ibid.
25. Neil Couch and Bill Robins, “Big Data for Defence and Security,” Royal United Services Institute, September 2013, 26, https://www.rusi.org/downloads/assets/RUSI_BIGDATA_Report_2013.pdf.
26. Ibid., 26-29.
27. The end-users I have in mind are operational planners in networked command posts. Designing a user interface for military end-users in theatre and on the ground will be far more complex. Secured networks must be in place; hardware and software must be able to function in inhospitable and low-bandwidth environments; information displays may need to be scalable for different devices. If the soldier is a “sensor” or data source as well as a user, the system will need to account for this added dimension.
28. Colin Wood, “How Does the Military Use Big Data?”, 6 Jan 2014, http://www.emergencymgmt.com/safety/Military-Use-Big-Data.html.
29. McKinsey Global Institute, “Big Data: The Next Frontier,” 29.
tech edge 63
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 63 1/3/16 3:51 PM
30. PCAST, “Designing a Digital Future,” 27.
31. NGA, “2020 Analysis Technology Plan,” 7.
32. Jonathan Shaw, “Why ‘Big Data’ Is a Big Deal,” Harvard Magazine, March – April 2014, http://harvardmagazine.com/2014/03/why-big-data-is-a-big-deal.
33. Carsten Gorg, Youn-ah Kang, Zhicheng Liu and John Stasko, “Visual Analytics Support for Intelligence Analysis,” 5, http://www.cc.gatech.edu/~stasko/papers/computer13-intell.pdf.
34. McKinsey Global Institute, “Big Data: The Next Frontier,” 34.
35. “Creating Heat Maps with Bing Maps and Dynamics CRM,” Microsoft Corporation, (blog, 29 October 2012), http://blogs.msdn.com/b/crm/archive/2012/10/29/creating-heat-maps-with-bing-maps-and-dynamics-crm.aspx
36. McKinsey Global Institute, “Big Data: The Next Frontier,” 36.
37. “The Next Generation of Big Data Visualization,” Sentinel Visualizer, accessed 25 February 2015, http://www.fmsasg.com/Products/SentinelVisualizer.
38. Shaw, “Why ‘Big Data’ Is a Big Deal.”
39. Joseph D. Fargnoli, “Big Data Enables Activity & Location Based Predictive Analytics Applications,” Ritre Corporation, (presentation, Government Big Data Symposium, Arlington, VA, 30 November 2013), http://semanticommunity.info/@api/deki/files/27182/FARGNOLI-TTC_Gov_Bigdata_Conf_fl13.pdf.
40. NGA, “2020 Analysis Technology Plan,” 6; John Edwards, “Military, Intel Turn to Big Data for Better Situational Awareness,” Federal Times, 2 June 2014,
http://archive.federaltimes.com/article/20140602/FEDIT/306020009/Mil i t ar y-inte l-turn-big-data-better-situational-awareness.
41. Steve LaValle, Eric Lesser, Rebecca Shockley, Michael S. Hopkins and Nina Kruschwitz, “Big Data, Analytics and the Path from Insights to Value,” MIT Sloan Management Review 52, no. 2 (Winter 2011): 21-31, http://www.ibm .com/smarterplanet/global/files/in_idea_smarter_computing_to_big-data-analytics_and_path_from_insights-to-value.pdf.
42. Jonathan Shaw, “Why ‘Big Data’ Is a Big Deal,” Harvard Magazine, March – April 2014, http://harvardmagazine .com/2014/03/why-big-data-is-a-big-deal.
43. Robert K. Ackerman, “Multiple Thrusts Define Geospatial Agency Big Data Efforts,” SIGNAL, 1 August 2014, http://www.afcea.org/content/?q=multiple-thrusts-define-geospatial-agency-big-data-efforts.
44. The phrase was used specifically with reference to intelligence failure prior to 9/11. I hesitate to use 9/11 as a throwaway example and thereby oversimplify the events leading up to it or implicitly suggest that big data could have prevented it, but the point remains that reasons like “failure to connect the dots” seem like frail excuses following a tragedy. Roger Z. George and James B. Bruce, “Intelligence Analysis—The Emergence of a Discipline,” in Analyzing Intelligence: Origins, Obstacles, and Innovations, ed. Roger Z. George and James B. Bruce (Washington DC: Georgetown University Press, 2008), 4.
tech edge 64
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 64 1/3/16 3:51 PM
ME4 Toh Bao En is currently a Staff Offi cer in Joint Intelligence Department. A recipient of the SAF Academic Scholarship, she graduated from the University of Michigan in 2011 with a Bachelor of Arts in English (Highest Honours). She has also served in the 30th Battalion, Singapore Combat Engineers and the Imagery Support Group.
tech edge 65
POINTER, JOURNAL OF THE SINGAPORE ARMED FORCES VOL.42 NO.1
51-65_.SwimmingInSensors.indd 65 1/3/16 3:51 PM