1

Human Interpretation of Text and Video Analytics

Elizabeth K. Bowman a, Marie-Eve Jobidonb, Alexandre Bergeron-Guyardb, Sue E. Kase a

a Army Research Laboratory, 321 Johnson St., Aberdeen Proving Ground, MD 21005 b Defence Research and Development Canada, 2459 de la Bravoure Road, Quebec, Quebec G3J

1X5

Abstract

New information technologies (mobile Internet, cloud storage, social networking, Internet of Things) are transforming how individuals and social groups create new knowledge and understand the world. Military analysts cannot keep pace with the volume/variety/velocity of data collected from global sensors and multimedia sources (e.g., text, video, and images). Impressive gains in advanced analytics are emerging and are needed, in combination with new visualization technologies and approaches, to enable analysts full control over, and understanding of, the data relations. These technologies will help humans to derive meaningful and actionable conclusions from the massive amount of data available to them. This is particularly critical in a context where nations face an increasing threat of the global spread of terrorism, humanitarian crises/disaster, and public health emergencies. These threats are informed and/or influenced by the unprecedented rise of information sharing technologies and practices, even in the most underprivileged nations. In this new information environment, agile data algorithms, machine learning software, and alert mechanisms must exploit available data to support timely and effective decision making. Human users of these systems must understand the underlying context of the computational space to accurately interpret machine-processed indicators and warnings and recommendations. In this paper, we will explore human dimension considerations (e.g., opportunities, challenges, constraints) of multi-media analytics, in the context of advanced C2 environments that rely on a wide variety of machine learning and artificial intelligence systems to support commanders’ decision making. We briefly review text and video processing before turning to how analysis occurs and the cognitive challenges associated therein. We consider four aspects: User Interface, Collaboration, Adaptive Automation, and Trust. We conclude with a discussion of future research needed to ensure effective human factors integration into the design and execution of text and video analytics.

DRDC-RDDC-2017-P087

2

Introduction Military decision makers require access to intelligence analysis products derived from multi-source data to achieve mission success. Because these data have variable formats and provenance/uncertainty signatures, complex machine processing algorithms are needed to transform data into meaningful information. Understanding those computational algorithms is critical for human analysts charged with making sense of outputs in the context of the military mission. This is especially true in coalition campaigns against adaptive adversaries in unfamiliar terrain. In these scenarios, military staff engage in intelligence preparation of the operational environment to evaluate how adversary land, air, maritime, space, and other forces might threaten coalition forces and mission success (Joint Staff, 2014). A holistic view of this environment covers a wide range of factors spanning traditional political, military, economic, social, information, and infrastructure (PMESII) variables and also sociocultural factors such as the impact of ethnic groups and religions, leadership and worship traditions, demographic patterns, and the general cultural landscape with regard to political, economic, and social dynamics (Joint Staff). These information sources can be expected to include a mix of structured and unstructured data that require preprocessing transformations prior to loading into the analysis knowledge base. As the range and complexity of data sources continues to increase, analysts are faced with more data than can be practically interpreted by a human. This increases the risk that potentially critical information will be missed. The combination of disparate “hard” (video and imagery) and “soft” (text) data sources into a common format of discrete data components, and subsequent exploitation by automated agents, is expected to alert analysts to intelligence cues that may otherwise be missed. Such a fusion process could allow the analyst to append more time interpreting information and less time searching for it (Bowman et al., 2016b). In the next section we review text and video processing before turning to how analysis occurs and the human factors challenges associated therein. We consider four aspects: User Interface, Collaboration, Adaptive Automation, and Trust. We conclude with a discussion of future research needed to ensure effective human factors integration into multi-source analytics.

Single Source Analytics of Text and Video Text Analytics Text and language play an important part in understanding the complex operational environment by providing key inputs for understanding cultures, attitudes, events, and relationships that drive behavior and coordinated actions. With advances and increased use of Internet and mobile communications, text information is available in unprecedented amounts and formats, presenting a unique opportunity to gain understanding through a wide variety of Natural Language Processing (NLP) technologies. These support a range of linguistic, statistical, and machine learning techniques to model and structure information for exploratory data analysis, enrichment, and visualization (Bowman et al., 2016a). New applications of text analytics for threat alerts are increasingly demonstrated in the following ways: 1) Classification of individuals associated with

3

target adversary or criminal groups, 2) tracking topics/themes in social media or news platforms to identify emerging narratives, 3) identifying social groups associated with topics/themes in the larger social narrative, and 4) forecasting near-term (e.g., within six months) violent actions in a region based upon discourse analysis from political rhetoric, news, and social media. These types of text analytics products are critical to intelligence analysis because they provide supporting context to signals obtained from sensors that produce imagery and video observations.

Video Analytics Machine indexing algorithms are critical to intelligence analysis to filter extremely large video files for frames relevant to a specific threat model.1 Scarce human analyst resources can then be used for analyzing those frames to detect, for example, signs of human activity or vehicle(s) that may pose a military threat. This requires algorithms to analyze video data (e.g., to detect and track people and vehicles and to identify where a particular human activity is taking place).

In recent years, military forces have fielded several wide-area motion imagery (WAMI) sensor systems, which persistently monitor fixed geographic locations for long periods of time using electro-optic sensors. As part of the overall persistent surveillance mission, the military services are expanding the deployment and development of airborne WAMI sensors. These sensors operate in predominantly urban settings, are capable of constantly monitoring many square kilometers for many hours, and generate terabytes of data per mission. Operationally, WAMI data are exploited for events of interest, both forensically and in real time. Events of interest can include starting points and destinations of tracks and nodes for related entities within the persistent field of view. They can also include activity and event-based normalcy and anomaly detections such as unique or unusual driving patterns that could signal a threat. Other types of events can be used to discover or highlight patterns-of-life associated with a variety of network types, including social, political, regional, and economic or military networks.

The challenge is to identify potential threats based on the accumulation and correlation of multiple events and anomalies, and issue timely alerts with a minimal number of false alarms. WAMI data are having an increasing impact on operational outcomes, but current efforts to exploit these data are mostly manual and require hours to days of painstaking analysis to produce results. The tedious nature of current exploitation capabilities limits the ability to fully utilize the available data. Consequently, critical battlefield questions go unanswered and timely threat cues are missed.

The primary information elements in WAMI data are entities in the context of roads, buildings, and other scene features. Exploitation of these entities yields tracks but in a complex urban environment, these tracks are severely fragmented due to occlusions, stops, and other factors. Within these localized events, algorithms that discover relationships and anomalies that are indicative of suspicious behavior, match previously learned threat activity, or match user-defined 1 In that sense, and for the purposes of this paper, video analytics also includes the analysis of still images (either stand-alone – e.g., pictures published online – or still frames extracted from videos.

4

threat activity should also be incorporated. While the localized events may each occur over a small spatiotemporal window, the overall threat activity sequence may span a much larger spatiotemporal window. Integrating these signals with results of text analytics products can serve to add context to the WAMI observations, for example, indicating a sports game is scheduled for a location where high traffic patterns are detected (Bowman et al., 2017).

Combining Text and Video Analytics Irregular warfare, non-state terrorism threats, and uncertain environmental patterns that trigger major weather disasters make it difficult for military and government leaders to rely on traditional physics-based sensors alone to plan current and future actions. While WAMI imagery is valuable for anomaly detection, vehicle movement and related indications and warnings, these sensors are affected by weather and environmental occlusions and are not capable of providing clues to human context that is critical to monitor goals, functions, and data needs (Rogova, 2009). Strategies for achieving contextual understanding can include observational data, a priori knowledge models, and inductive knowledge (Gomez-Perez & Moreno, 2010). Contextual understanding is generally achieved through a combination of human and computer processing techniques that take advantage of a person’s cognitive ability to fuse and assimilate multiple sources and types of information for new insights (Secker & Vachon, 2007). In military environments, it is critical to incorporate both hard and soft data to gain an understanding of the delicate balance between individuals and groups in society and the environments (geopolitical, social, agricultural, etc.) upon which they depend. Analytic methods that correlate and fuse hard and soft data are challenging due to the nature of the data types and the metrics for determining outcomes (Hall et al., 2010). Unlike much physical sensor information, however, data sources for this new type of problem are not classified or difficult to obtain; open source data are available and plentiful. The challenge becomes correlating and fusing data of many different types that represent various aspects of a region of interest. This approach is similar to the signal processing approach of weak signal detection, which is used to extract received signals (Kang et al., 2008), identify images in noisy backgrounds (Guangzhi et al., 2009), and conduct remote sensing of land and water resources for sustainable development of natural resources (Rao, 2001). A novel approach to correlating a variety of data sources to understand problems in an area and forecast conflict is described by (West, 2005). In this example, the potential conflict is the weak signal that is detected through the correlation of diverse datasets describing many features of the region, to include demographic, political, social, economic, educational, agricultural, weather, etc.

6

text and image-based data. This is especially the case with information collected from the internet, including social media. As part of the analysis process, the analyst must assess the veracity of the data (e.g., are there indicators of deception or manipulation?), its representativeness (e.g., the demographics of the source of data, the extent to which it represents the region or target of interest), the context in which it is collected (e.g., sociocultural nuances can change the meaning and interpretation of the information, and so can temporal factors such as when data is collected and over what period of time), and the “ground truth” of the data (i.e., to what extent does it represent or translate to the offline world and what are the operational impacts) (e.g., Jobidon & Forrester, 2016).

The predictive analytics stage of the workflow provides the analyst with the meaningful results of the computational exploitation of the multi-source data. These can include attack warnings, a changing geospatial picture, identification of emerging threat entities and threat network discoveries, and suggested risks to the mission. Each of these must be visualized in an intuitive manner, such as the use of maps, network diagrams, trend lines, baseball cards for key actors or groups, and risk dashboards.

The analysis workflow shown in Figure 1 captures the familiar human factors challenge for intelligence analysis; to deliver the right information, at the right time, in the right place, in the right way, to the right person (Fischer, 2015). Technologies that take advantage of computational machine learning algorithms need an intuitive means of interacting and collaborating with humans in a transparent way. This enables the humans to understand the computational strategy (e.g., assumptions the algorithm makes, how it can err) and outputs in a sufficient manner that they recognize and know the meaning of outputs, and can trust system components.

Following are some identified challenges, desired capabilities, and research opportunities to support human decision making in this area. We discuss four specific challenges: user interaction, collaboration, adaptive automation, and trust.

There are several human factors and cognitive challenges with intelligence analysis overall (McNeese, Buchanan, & Cooke, 2015; McNeese, Cooke, & Buchanan, 2015; Trent, Patterson, & Woods, 2007) and these can be compounded with multi-source analysis. Multi-source data is “complex, heterogeneous, dynamic, distributed, and very large” (Uselton et al., 1998). Therefore, even more than with single-source analysis, the onus is on the analyst to integrate, comprehend, and assess ever-evolving data of different natures coming in different formats and modalities. This creates a context in which some of the challenges identified are of particular interest: complexity, data integration and visualization, time pressure, workflow, and the need for user-centric adaptable tools.

With respect to complexity, data/information overload is already an issue. With multiple sources of information to be analyzed, the process will be complicated with the necessity to integrate and comprehend large quantities of heterogeneous data in a dynamic manner (which is akin to understanding a complex system). However, the human cognitive constraints remain the same.

7

Making sense of complex systems is challenging, even for experts. The literature shows that even the judgment of highly skilled individuals is unreliable when they try and anticipate the behaviour of a dynamic system that has more than four variables (Halford, Baker, McCredden, & Bain, 2005). What this shows is that even for experts, considering the evolution and interrelationships of about five variables is very difficult (e.g., Halford et al., 2005).

Considering the challenge of data integration and visualization, we note the following issues. Customizable and adaptable visualization is a critical component of text and video analysis to allow information to be displayed in a meaningful way for the analyst (McNeese, Cooke, & Buchanan, 2015). Not only must the tools reflect the heterogeneity and dynamism of the data and allow the analyst to visualize it from different perspectives and for different purposes (Antony, 2016; Uselton et al., 1998), the display should be consistent with the analyst’s mental model of the analysis problem. Analysts have been shown to generally follow a process model that is divided into three phases: developing context (building a mental model), data gathering and information processing, and understanding/knowledge creation (Hendrickson, Fendley & Kuperman, 2013). Tools must help the analyst integrate and “grasp” the information in order to support sensemaking and analysis. One of the challenges is to support the cognitive constraint mentioned previously and to allow the analyst to comprehend and make sense of as much relevant data as possible without cognitive overload or biased interpretations. Cognitive bias can impact the data analysis process in several ways: anchoring (being influenced by one piece of information), projection (thinking that others implicitly know what you know), and representativeness (judging the correctness of a hypothesis by how much the available data resembles it) (Clegg et al., 2015). Game-based training strategies were found to be effective for helping analysts understand, recognize, and mitigate sources of and tendencies toward bias in the sense-making process (Clegg et al.). Computational mitigation strategies can also be employed to reduce bias, such as in recommender strategies that encourage humans to consider more one answer to a query.

Time pressures appears to be one of the biggest cognitive challenges for intelligence analysis (e.g., Johnston, 2005; McNeese, Buchanan, & Cooke, 2015; Trent, Patterson & Woods, 2007). Text and video data analysis can be either an added stressor or an aid in relieving some of the pressure of off the analysts, depending on how it is implemented, conducted, and managed. New approaches for video indexing have potential to reduce analyst cognitive burden if large repositories can be searched for video frames that contain behaviors of interest. Adding Automatic Character Recognition (ACR) capabilities to video analysis can supplement scene recognition tasks such as identifying the presence of a foreign flag in a video where the presence of that object is not mentioned in audio (and therefore not captured via audio transcription). Automatic detection and tagging of behaviors in video with text can also aid analyst cognitive burden, for example, by determining whether a detected throwing action is non-threatening (such as a person throwing a baseball) or threatening (such as a rioter throwing a Molotov cocktail or similar weapon).

Related to the previous points, moving more towards analysis of multiple types of data may have impact on current workflows than those more focused on single-source analysis. From a human

8

factors perspective, understanding the workflow progression (what tasks, for what purpose, and when, both for the individual and collaborative aspects of the work) (McNeese, Cooke, & Buchanan, 2015) is critical to design and implement adequate processes and support tools. Changes to workflow needed to account for the conduct of multi-source analysis must be captured, assessed, and implemented.

Multi-source analysis may require specific training, with regards to the tasks to be performed (e.g., new tradecraft or processes), collaborative requirements, and potential new tools to be used (McNeese, Buchanan, & Cooke, 2015). With the rush to exploit the social media domain, new tools are being developed and proposed as new modules to existing analysis pipelines. As workflows are expanded or modified to take advantage of new analytic capabilities, training for analysts is important from a minimum of two standpoints. The first is to correctly understand the tool’s inherent functionality and relation to the larger workflow. The second is to ensure that the analyst’s mental model of the tool(s) correlates to the larger ecosystem of the analysis human-machine system (Trent Patterson & Woods, 2007).

Tool selection for complex analysis workflows should consider human factors elements in addition to the modular contribution the tool makes to the entire information processing pipeline. One of the most critical is that the tool support the human analysts and the way they work (McNeese, Buchanan, & Cooke, 2015; McNeese, Cooke, & Buchanan, 2015; Trent Patterson & Woods, 2007). A tool will be discarded by analysts if they do not feel like it adds value to their work and/or if they feel that it increases their workload and time pressure burden. In a multi-source analysis context, it is therefore critical to determine what tasks or cognitive processes need to be supported and when (both for individual and collaborative aspects of the work) and to design and develop tools based on that knowledge.

User Interaction Graphical User Interfaces (GUIs) have become fairly sophisticated as mobile computing continues to develop and expand, with the more successful applications being those that understand the goals of the user community. This can be difficult to achieve with a diverse user population in terms of computational expertise and task goals. Cognitive work analysis (Rasmussen, Petjersen & Goodstein, 1994) is a discipline that studies complex sociotechnical systems for analysis, design, and evaluation. A highlight of effective GUI design is natural features for task and content interaction, speech, behavior, and gesture recognition/control, data visualization and manipulation tools, and cueing.

For multi-source analytics, dashboards have become commonplace for the range of displays required to represent relevant features of the operating environment. Such dashboards can rapidly become overwhelming to analysts with a finite amount of attention and mental workload capacities. A recent experiment compared two social network analysis dashboards with a manual text analysis approach (Kase, Roy, & Cassenti, 2015). Participants completed a knowledge

9

discovery task with a dashboard that visualized data with a sparkline graph, scatter plot, or messages representing the social network (manual method). Results showed that, on average, participants were more accurate performing sentiment analysis when using a scatter plot (73% correct answers) compared to a sparkline graph (50%) or reading text messages (45%). Further, participants more rapidly provided the correct response when using the scatter plot (Mdn = 27.77 seconds) compared to the sparkline graph (Mdn = 29.41seconds) or reading text messages (Mdn = 68.58 seconds) (Kase, Roy, & Cassenti). In a more recent study, Ren et al. (submitted), investigated human understanding of different network visualizations in a large-scale online experiment. These were node-link and matrix representations of either 20 or 50 nodes. Results indicated that participant understanding was best for the node-link visualization, with higher accuracy and task times (Ren, et al.).

Integrating multi-modal data analysis products in a dashboard visual display requires careful consideration of user knowledge requirements and expected interactions between the various types of data. Identifying, exploring, and capturing relationship linkages that could signal important features in data can be accomplished in several ways. Various types of diagrammatic knowledge maps that link data relationships include Mind-Maps (Buzan & Buzan, 1996), Concept Maps (Novak, 1984), Semantic Desktops (Sauermann et al., 2009), and Zooming user interfaces (Bederson et al., 1996). Haller and Abecker (2010) have built on the latter type to create iMapping, an approach that brings together strengths of several mapping approaches. This approach is built on general requirements for knowledge mapping approaches, including free placing (user decides where to place information), free relationship linking (linking information with different degrees of formalization), annotations (notes that can be hidden and easily retrieved), overview/abstraction (clustering and hierarchical relations), and scalability (for visually dealing with large data). Developing UIs that provide this type of flexibility for users with increasingly small screens (such as mobile phones) will continue to be a challenge in the multi-modal knowledge creation domain.

Collaboration Intelligence analysis involves several aspects of collaboration throughout the workflow process from collection, to analysis, to production of intelligence outputs. Kang and Stasko (2011) suggest three types of collaboration inherent in this analysis workflow; sharing, content, and functional. Sharing information in all phases of the task facilitates shared awareness of the team. Content collaboration involves team members working together to produce an intelligence product. Functional collaboration occurs during the production phase where team members work together to edit documents and polish final deliverables (Kang & Stasko). In a more recent study of cybersecurity analysts, Ahrend, Jirotka, and Jones (2016) found that informal forms of collaboration and coordination build tacit knowledge of threats for actionable threat intelligence. However, the authors note that a lack of accessibility to knowledge about new threats and actors can reduce analysts’ ability to detect cyber threats in advance, contain damage, and obtain situational awareness.

11

Information Information fusion/integration of multi-source data Knowledge graphs with embedded threat models

Approaches for transparency of computational solutions to analysts (humans can

understand and explain computer solutions)

People Computational modeling of social dynamics

Social behavior considered at scale (local, regional, national, global) Social informatics, influence, learning and norms Dynamics of collective action Social contagion and diffusion of ideas

Technology Visualization approaches to allow rapid sensemaking of complex patterns in data

High Performance Computing processing for big data problems Natural Language Processing methods for extracting meaningful entities, events,

relations, etc. Enterprise architectures to allow modularity of emerging tools and workflows

Table 1 A Golden Triangle Depiction of Exemplary Research Topics for an Interdisciplinary Portfolio

Adaptive Automation As information displays grow more complicated, adaptive automation may provide assistance to analysts in the form of increased situational awareness and reduced cognitive workload. Adaptive automation encompasses a class of computer aids that are meant to increase a user’s performance and triggered only when the user encounters difficulty with a task. Timing adaptive aids to assist only when the user needs them prevents the aids from becoming distracting or creating over reliance on them. In Cassenti, Gamble, and Bakdash (2016), the authors propose that cognition during complex task completion can be analyzed into four different stages: perception, information processing, decision making, and motor control. Cassenti and Veksler (in press) discuss different adaptive aids that may help each of these processes (Kase, et al., 2017). Challenges in adaptive automation aids include determining when automation is helpful, how much automation is needed, and which tasks have optimal gain from automation. Feigh et al. (2012) provides a framework for investigating adaptive automation triggering conditions. While these authors outline a paradigm for empirical investigation of adaptive automation, they do not execute the paradigm in a human subjects study. Kaber et al. (2005) tested adaptive automation at various cognitive stages in an air traffic control task, including information acquisition, information analysis, decision making, and

12

action. Results of this study indicated that humans were better able to adapt to adaptive automation in action implementation compared with two of the other three cognitive stages; that is, information analysis and decision making.

Trust An advanced analytics system should behave and perform in such a way that it generates a high level of trust in the user. Trust increases the likelihood that newly acquired knowledge is usefully absorbed (Atkinson & Clark, 2013) and used efficiently. This is important in the Intelligence and Counterterrorism domains as: the problems addressed are often complex and involve critical consequences; the analysts are highly skilled people; and they are imputable for their decisions. As with any collaboration with another person, the level of trust in the analytics system will need to develop with time, supported by various strategies. This notion of trust is addressed in (Bergeron-Guyard et al., 2014).

The first obvious step to raise the level of trust would be to provide the users with the appropriate level of training before they use the system. Having a better understanding of the system and how it works will possibly lead to a lower uncertainty about the obtained results, and potentially to higher trust.

Another strategy could be to provide access to a traceability of the rationale behind any analysis or recommendation made. This will allow the user to build a level of understanding and confidence that may lead to greater trust in the system’s results. The system should allow the user to associate confidence indicators for different types of tasks (e.g., the user may develop trust more rapidly in certain identification types). As the system gets a higher level of trust for a given task, it could adapt its behavior in providing a lower level of feedback and, eventually, gain a certain level of automation (pending user validation).

The use of conversational agents is another possible avenue to build trust. Brickmore & Cassell (2001) propose a conversational agent model meant to increase user’s trust in a system. Moon (1998) demonstrated that a computer which uses a strategy of reciprocal, deepening self-disclosure in its (text-based) conversation with the user will cause the user to rate it as more attractive, divulge more intimate information, and become more likely to buy a product from the computer.

In considering how to establish trust between human and system, one must also consider the types of tasks that are better suited for automation. The Automation Principle stipulates that if any of the tasks undertaken by the user in communicating to a computer are mechanical, tedious, error prone or prevalent, then they should be automated (Lambert, 2009).

13

Conclusions In summary, advanced text and video analytics need to be developed in ways that 1) Adapt to information needs based on mission objectives, context, and command decision maker needs; 2) Enable understanding of the situation to allow analysts to track machine algorithms and offered solutions for accuracy and relevance to the mission; and 3) Ensure timely decision making despite imperfect information/connectivity. Human factors research that can support these capabilities would include the following. These suggestions will be used to inform strategic research investments across the US and Allied laboratories in the near future.

Iterative human subject experimentation is needed to empirically establish the utility of various components of multi-source analytics and to ensure that humans can understand the computational mechanisms underlying the analysis process and accurately interpret results. Advanced touch, gestures, and voice technologies should continue to be explored to allow analysts to naturally interact with information objects and analysis results. Working toward a naturalistic human-machine interface with structured and unstructured data sources and information objects will enhance the human’s understanding of the computational transformations and outputs and make interpretation of results more accurate and timely. Supporting this goal would be a taxonomy of naturalistic control behaviors for interacting with information across any information display, especially 3D displays in immersive settings. As these technologies mature and spread in use, interacting with complex data with adaptive automation displays should increase the knowledge discovery process by intuitive exploration of fused information. These naturalistic environments will be expected to include multi-modal (audio-visual) information presentation systems, to reduce analyst cognitive workload and support the discovery of patterns in the database related to the threat model currently exercised by the analysis process. Collaboration will continue to be a driving force, both with human-human and human-machine. Developing synchronous and asynchronous collaboration methods for both types is a common goal. Finally, benchmarks should be established for current human-machine interaction metrics (speed, accuracy, completeness) in order to quantify the value obtained by the computational methods employed for information exploitation and to specifically quantify the mission effectiveness quotient gained by these processes.

14

References

Ahrend, J.M.; Jirotka, M.; & Jones, K. (2016). "On the collaborative practices of cyber threat intelligence analysts to develop and utilize tacit Threat and Defence Knowledge," 2016 International Conference On Cyber Situational Awareness, Data Analytics And Assessment (CyberSA), London, 2016, pp. 1-10.

Antony, R., T. (2016). Data fusion support to activity-based intelligence. Boston, MA: Artech House.

Atkinson, D. J. & Clark, M. H. (2013). Autonomous Agents and Human Interpersonal Trust: Can We Engineer a Human-Machine Social Interface for Trust? 2013 AAAI Spring Symposium.

Bederson, B.B.; Hollan, J.D.; Perlin, J.; Meyer, K.; Bacon, D.; & Furnas, G. (1996). Pad++: A Zoomable Graphical Scetchpad For Exploring Alternate Interface Physics. Journal of Visual Languages in Computing, 7(1):3-31, 1996.

Bergeron-Guyard, A., Lavigne, V., Poussart, D., Gouin, D., Roy, J. (2014). Intelligence Virtual Analyst Capability – Governing Concepts and Science and Technology Roadmap, DRDC-RDDC-2014-R156.

Boury-Brisset, A.C., Bowman, E.K., Broome, B.D., Burghouts, G., Duselis, J., Forrester, B., Holland, V.M., Howe, J., van Huis, J., Kwantes, P., Madahar, B.K., Mulayim, A.Y., Rao, R.M., Summers-Stay, D. (2015). ARL TR 7705: Complex Event Processing for Content-Based Text, Image, and Video Retrieval. Aberdeen Proving Ground, MD: US Army Research Laboratory.

Bowman, E. K., Lin, J., Rangwala, H., Madahar, B. & Mittu, R. (2016a). Content-Based Multimedia Analytics: Rethinking the Speed and Accuracy of Information Retrieval for Threat Detection. NATO ACT Operations Research and Analysis Conference, Norfolk, VA.

Bowman, E. K., Saulwick, A., Trentelman, K., Power, T., & Kwantes, P. (2016b). Hard and Soft Fusion for Proactive Decision Making: Coalition Capabilities and Challenges (TTCP TR-C3I-1-2016).

Bowman, E. K., Turek, M., Tunison, P., Porter, R., Thomas, S., Gintautas, V., Shargo, P., Lin, J., Li, Q., Gao, Y., Li, X., Mittu, R. (2017). Advanced Text and Video Analytics for Proactive Decision Making. Proceedings from SPIE: Next-Generation Analyst V Conference. Anaheim, CA.

15

Brickmore, T., & Cassell J. (2001), Relational agents: a model and implementation of building user trust, Proceedings of the SIGCHI conference on Human factors in computing systems.

Buzan, T. & Buzan, B. The Mind Map Book: How to Use Radiant Thinking to Maximize Your Brain's Untapped Potential. Plume, 1996.

Cassenti, D.N., & Veksler, V.D. (in press). Using cognitive modeling for adaptive automation triggering. In D.N. Cassenti (Ed.): Advances in human factors and simulation. New York: Springer.

Cassenti, D.N., Gamble, K.R., & Bakdash, J.Z. (2016). Multi-level cognitive cybernetics in human factors. In S.K. Hale & K. Stanney (Eds.): Advances in neuroergonomics and cognitive computing (pp. 315-326). New York: Springer.

Clegg, B.A., McKernan, V. Martey, R.M., Taylor, S.M., Stromer-Galley, J., Kenski, K., Saulnier, E.T., Rhodes, M.G., Folkestad, J.E., McLaren, E., Shaw, A., and Strzalkowski, T. (2015). “Effective mitigation of anchoring bias, projection bias, and representativeness bias from serious game-based training,” 6th International Conference on Applied Human Factors and Ergonomics and the Affiliated Conferences, pp. 1558-1565.

Feigh, K. M., Dorneich, M. C., & Hayes, C. C. (2012). Toward a characterization of adaptive systems: A framework for researchers and system designers. Human Factors: The Journal of Human Factors and Ergonomics Society, 54(6), 1008-1024.

Fischer, G. (2012). Context-Aware Systems: The ‘Right’ Information, at the ‘Right’ Time, in the ‘Right’ Place, in the ‘Right’ Way, to the ‘Right’ Person. AVI, May 21-25, Capri Island, Italy. Available on the Internet: https://www.researchgate.net/publication/241623010_Context-Aware_Systems_The_%27Right%27_Information_at_the_%27Right%27_Time_in_the_%27Right%27_Place_in_the_%27Right%27_Way_to_the_%27Right%27_Person. Accessed 2 June 2017.

Gomez-Perez, J.M.; Moreno, C.R.. (2010). "Overcoming Information Overload in the Enterprise: The Active Approach," Internet Computing, IEEE , vol.14, no.6, pp.39-46, Nov.-Dec.

Guangzhi Cao; Bouman, C.A.; Theiler, J. (2009). "Weak signal detection in hyperspectral imagery using sparse matrix transform (smt) covariance estimation," Hyperspectral Image and Signal Processing: Evolution in Remote Sensing. WHISPERS '09. First Workshop pp.1-4, 26-28 Aug.

Halford, G., Baker, R., McCredden, J., & Bain, J. (2005). How many variables can humans process? Psychological Science, 16(1), 70-76.

16

Hall, D.L.; More, L.D.; Graham, J. & Rimland, J.C. , "Test and evaluation of soft/hard information fusion systems: A test environment, methodology and initial data sets," Information Fusion (FUSION), 2010 13th Conference on , pp.1-7, 26-29 July 2010.

Haller, Heiko & Abecker, Andreas. (2010). iMapping - A zooming user interface approach for personal and semantic knowledge management. HT'10 - Proceedings of the 21st ACM Conference on Hypertext and Hypermedia. 119-128. Available on the Internet: https://www.researchgate.net/publication/221267427_iMapping_-_A_zooming_user_interface_approach_for_personal_and_semantic_knowledge_management. Accessed August 21, 2017.

Hendrickson, K.K., Fendley, M. E., and Kuperman, G.G. (2013). “Modeling Analyst Process of Transforming Data into Understanding,” 12th IFAC Symposium on Analysis, Design, and Evaluation of Human-Machine Systems, pp. 431-434, 11-15 August 2014.

Jobidon, M.-E. & Forrester, B. Understanding Social Media Netoworks through Trending Analysis. (2016). In Proceedings, International Command and Control Research Technology Symposium, London, England, 6-8 September.

Johnston, R. (2005). Analytic culture in the US intelligence community: An ethnographic study. CENTRAL INTELLIGENCE AGENCY WASHINGTON DC CENTER FOR STUDY OF INTELLIGENCE.

Joint Staff. (2014). Joint Intelligence Preparation of the Operational Military Environment. Joint Publication 2-01.3. US Chairman of the Joint Chiefs of Staff: Washington, D.C.

Kaber, D. B., Wright, M. C., Prinzel, L. J., & Clamann, M. P. (2005). Adaptive automation of human-machine system information-processing functions. Human Factors: The Journal of the Human Factors and Ergonomics Society, 47(4), 730-741.

Kang Lei; Mi Wujun; Lv Chao; Wang Shujuan; , "Research on weak signal detection technique for electromagnetic ultrasonic inspection system," Industrial Electronics and Applications, 2008. ICIEA 2008. 3rd IEEE Conference on, pp.2394-2399, 3-5 June 2008.

Kang, Y.a. and Stasko,J. "Characterizing the intelligence analysis process: Informing visual analytics design through a longitudinal field study," 2011 IEEE Conference on Visual Analytics Science and Technology (VAST), Providence, RI, 2011, pp. 21-30.

Kase, S. E., Roy, H., & Cassenti, D. (2015, April 17-21). Visualization approaches for displaying measures of sentiment. Proceedings of the 2015 SPIE Sensing Technology + Applications Conference. Baltimore, Maryland.

Lambert, D. A. (2009). A blueprint for higher-level fusion systems. Information Fusion, 10(1), 6-24.

17

McNeese, N. J., Buchanan, V., & Cooke, N. J. (2015). The cognitive science of intelligence analysis. In 2015 International Annual Meeting of the Human Factors and Ergonomics Society, HFES 2015 (Vol. 2015-January, pp. 826-830). Human Factors an Ergonomics Society Inc.

McNeese, N. J., Cooke, N. J., & Buchanan, V. (2015). Human factors guidelines for developing collaborative intelligence analysis technologies! In 2015 International Annual Meeting of the Human Factors and Ergonomics Society, HFES 2015 (Vol. 2015-January, pp. 821-825). Human Factors an Ergonomics Society Inc.

Moon, Y. (1998). Intimate self-disclosure exhanges: Using computers to build reciprocal relationships with consumers. Division of Research, Harvard Business School.

Novak, J.D. & Gowin, D.B. (1984). Learning how to learn. Cambridge University Press, New York.

Rao, D.P.; , "A remote sensing-based integrated approach for sustainable development of land water resources," Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on , vol.31, no.2, pp.207-215, May 2001.

Rasmussen, J., Petjersen, A.M. & Goodstein, L.P. (1994). Cognitive systems engineering. Wiley: New York

Ren, D., Marusich, L., O’Donovan, J., Bakdash, J., Cassenti, D., Kase, S., Roy, H., Schaffer, J., Lin, W., and Hollerer, T. (submitted). Understanding Node-Link and Matrix Visualizations of Networks: A Large-scale Online Experiment. Network Science.

Rogova, G.L.; , "Context-awareness in crisis management," Military Communications Conference, 2009. MILCOM 2009. IEEE , pp.1-7, 18-21 Oct. 2009.

Sauermann, L.; Kiesel, M.; Schumacher, K.; & Bernardi, A. (2009). Semantic Desktop. In A. Blumauer and T. Pellegrini, editors, Social Semantic Web, pages 337-362. Springer-Verlag, X.media.press.

Savas, O. Final Report for Contract No. W911NF-14-C-0013. Scalable Distributed and Modular Toolkit for Mining Text-Rich Heterogeneous Information Networks to Enhance Contextual Understanding: iTextNetMiner. 2016.

Schneier, B. (1999). Schneier on Security: People, Process, and Technology. Available on the Internet: https://www.schneier.com/blog/archives/2013/01/people_process.html. Accessed September 11, 2017.

Secker, J.; Vachon, P.W.; , "Exploitation of multi-temporal SAR and EO satellite imagery for geospatial intelligence," Information Fusion, 2007 10th International Conference on , vol., no., pp.1-8, 9-12 July 2007

18

Selva, P.J. (2017). Joint Concept for Operating in the Information Environment (Draft Version 0.50). US DOD: Washington, D.C.

Trent, S. A., Patterson, E. S., & Woods, D. D. (2007). Challenges for cognition in intelligence analysis. Journal of Cognitive Engineering and Decision Making, 1(1), 75-97.

Uselton, S., Ahrens, J., Bethel, W., Treinish, L., & State, A. (1998, October). Multi-source data analysis challenges. In Visualization'98. Proceedings (pp. 501-504). IEEE.

West, K. “Scoping Out the Planet,”, Scientific American. October 24, 2005. Available on the Internet at: http://www.scientificamerican.com/article.cfm?id=scoping-out-the-planet, downloaded June 4, 2011.

AbstractIntroductionSingle Source Analytics of Text and VideoText AnalyticsVideo AnalyticsCombining Text and Video Analytics

Gaining Knowledge from Text and Video Analysis ProductsUser InteractionCollaborationAdaptive AutomationTrustConclusionsReferencesAtkinson, D. J. & Clark, M. H. (2013). Autonomous Agents and Human Interpersonal Trust: Can We Engineer a Human-Machine Social Interface for Trust? 2013 AAAI Spring Symposium.Bederson, B.B.; Hollan, J.D.; Perlin, J.; Meyer, K.; Bacon, D.; & Furnas, G. (1996). Pad++: A Zoomable Graphical Scetchpad For Exploring Alternate Interface Physics. Journal of Visual Languages in Computing, 7(1):3-31, 1996.Bergeron-Guyard, A., Lavigne, V., Poussart, D., Gouin, D., Roy, J. (2014). Intelligence Virtual Analyst Capability – Governing Concepts and Science and Technology Roadmap, DRDC-RDDC-2014-R156.