vis IT - Fraunhofer IOSB · vis IT Defense 2 Topics Hardening of Optical and Electro-Optical Systems ... the UAV LUNA drone) and have proven themselves on numerous occasions, inclu-

[ Defense ]

IOSB

vis IT

Frau

nh

ofe

r

www.iosb.fraunhofer.de

ISSN 1616-8240

Eye and Sensor Protection

Interoperable Video Exploitation

CSD Plus

Incoming Fire Warning

Image-based Infrastructure Analysis

Navigation and Mapping

Identification of Persons

System-of-Systems Engineering

Information Management

Deployment of Laser Radar

3D Environmental Models

CONTENT

vis ITDefense

2

Topics

Hardening of Optical and Electro-Optical Systems against Blinding and Destruction by Laser RadiationBernd Eberle

ABUL: Interoperable Video Exploitation for Aerial Surveillance and ReconnaissanceNorbert Heinze

The CSD PLUSBarbara Essendorfer

Protection against AmbushJürgen Krieg, Dirk Seiffer

SiteAnalystAssistance for Interactive Image-Based Infrastructure AnalysisSusanne Angele, Elisabeth Peinsipp-Byma

Navigation and Mapping for Mobile Task ForcesJochen Meidow

Perception of PersonsMichael Arens, Dieter Willersinn

Architecture-Based Methodology for System-of-Systems EngineeringRainer Schönbein

Information Management with Semantic ModelsReinhard Herzog

Use of Laser Radar for Unmanned Convoy VehiclesWalter Armbruster

3D Environmental Models from Mini-Drone VideosPeter Solbrig

4

6

8

10

12

14

16

18

20

22

23

Page

Page

Page

Page

Page

Page

Page

Page

Page

Page

Page

PublisherProf. Dr. Maurus TackeProf. Dr.-Ing. Jürgen Beyerer

EditorSibylle Wirth

LayoutChristine Spalek

English Editiondb Sprachenservice GbRDresden

PrintingE&B engelhardt und bauerKarlsruhe

Editoral AdressFraunhofer Institute of Optronics, SystemTechnologies and Image Exploitation IOSB

Fraunhoferstr. 176131 KarlsruhePhone +49 721 6091-300Fax +49 721 [email protected]

© Fraunhofer IOSB Karlsruhe 2012

Institute of the Fraunhofer-Gesellschaft,Munich

13. YearISSN 1616-8240

Photo acknowledgements

People / product Photos / Key visuals:indigo Werbefotografi e

People: Volker Steger

Page 10/11: U.S. Army

Page 14: Volker Steger

All other illustrations:© Fraunhofer IOSB

Reproduction of any materialis subject to editorial authorization.

EditorialNotes

vis IT3 Defense

Dear Friends of IOSB,

The German armed forces are currently reorganized. As this is written, the reorganization plans are known in outlines, a framework has been established, and the next step will be the final adjustment of the organisation.

This is the time for our partners to shift their attention to internal processes. The outcome will be of essential importance to us.

IOSB itself just went through such a process. The initial phase had been defined by the analysis of our scientific base and building blocks, the definition of strategic objectives and the search for an appropriate internal structure. The last step, implementation, is done.

The business area »Defense« is the synthesis of the departments contributing to its goals. In many respects, it acts like sort of a company within the institute. The department heads form a management circle, which annually elects a speaker. Dr. Reinhard Ebert currently holds this position. Embedded in the institute manage-ment this results in a structure that is both dynamic with respect to changing requirements and stable in a strategic sense, which is ideal for our partners – also in times when they are called by internal issues.

We continue to focus on topics that promise to deliver innovations for defense. We hope the following examples such as ABUL and 3D reconstruction of MUAV video data will be an interesting reading for you, and give you an impression of IOSB defense activities.

Karlsruhe, July 2012

Prof. Dr. Maurus Tacke Prof. Dr.-Ing. Jürgen Beyerer

Prof. Dr.-Ing. Jürgen Beyerer

Prof. Dr. Maurus Tacke

Editorial

Themes

CO

NT

AC

T

vis IT Defense

4

Figure 1: Even at day time a relatively low laser power (class 3B laser pointer) can significantly dazzle a Cessna from a distance of 600 m.

HARDENING OF OPTICAL AND ELE CTRO-OPTICAL SYSTEMS AGAINST BLINDING AND DESTRUCTION BY LASER RAD IATION

Dr. Bernd Eberle

Optronics (OPT)Fraunhofer IOSB Ettlingen

Phone +49 7243 [email protected]


which are currently available with powers of up to one watt within the visible spectral range. They are easy to acquire and are therefore ideal for dazzling aircraft or vehicle operators; however, dazzling of electro-optical systems is also imaginable. (b) By means of pulsed lasers, which are capable of achieving their peak intensity within nanoseconds and have a substantial potential for destruction. Laser weapons must also be mentioned in this respect, which were developed by different coun-tries with the goal of blinding and dam-aging sensors as well as the human eye.

While the dazzling effect of electro-optical systems stops as soon as the laser radiation subsides, the human eye may continue to suffer from impaired vision (e.g. flash-blindness, afterimages or an impaired ability to discern color contrast) for a prolonged period of time. Figures 1 and 2 depict the dazzling of aircraft using the example of a Cessna (by day) and a CH-53 transport helicopter (at night). Under such circumstances, safe continuation of a night flight would no longer be possible.

The availability of compact, high-performance laser sources is constantly increasing around the world. This implies a potential threat to the human eye and also to sensor systems, operating in the visible (VIS) and near infrared (NIR) spec-tral range. Fraunhofer IOSB has developed methods that considerably reduce the risk of dazzling or permanent damage.

RISK OF LASER-INDUCED DAZZLING

Current measures introduced by the German Armed Forces for the protec-tion of the human eye or electro-optical sensors employ conventional protection technologies. Laser protection filters are used, which only block specifically defined wavelengths while allowing all other wavelengths to transmit. Due to the rapid advancement of laser technologies in recent years, laser sources now exist that cover the entire visual and near-infrared spectral range.

One can differentiate between two types of laser threats: (a) Compact, inexpensive continuous wave lasers,

vis IT 5 Defense

Figure 2: Dazzling of the cockpit of a CH-53 trans-port helicopter at night, recorded with a TV camera.

Figure 3: Demonstration of the IOSB protection strategy to prevent dazzling induced by continuous wave lasers. a) without dazzling. b) dazzling from a distance of 400 m. c) suppression of dazzling.

Figure 4: Filter with broadband spectral effect for protecting against laser radiation of any wavelength.

HARDENING OF OPTICAL AND ELE CTRO-OPTICAL SYSTEMS AGAINST BLINDING AND DESTRUCTION BY LASER RAD IATION

PROTECTION STRATEGIES

Fraunhofer IOSB is working on develop-ing strategies for hardening optical and electro-optical systems against dazz-ling and damaging by laser radiation. The current state-of-the-art offers no measures to protect against dazzling and damaging simultaneously. There-fore, these topics are being treated individually.

SENSOR PROTECTION

Hardening of electro-optical sensors against dazzling inflicted by contin-uous wave laser radiationAn active protection strategy based on a spatial light modulator combined with wavelength multiplexing was developed at IOSB to protect electro-optical sensors against dazzling induced by laser radi-ation. The protection strategy enables color selective filtering of laser radiation within a freely definable part of the sensor’s field of view and nonetheless

offers the possibility of determining the location of the dazzling source. Figure 3 depicts the effect of such a protective measure.

Hardening of optical and electro-optical systems against damaging by pulsed laser radiationIn order to harden optical and electro- optical sensors against pulsed laser radiation (Fig. 4), passive protection filters based on nanoparticles are being developed which can be deployed over a broad spectral range from the visual to the near infrared. They serve as so-called sacrificial elements, which are placed in the intermediate focal plane of an optical system and attenuate nanosecond laser pulses to such an extent that the residual energy is just slightly beyond the destructive threshold for the eye or optical sensor.

Introducing such a hardening measure would be a highly advisable step as it efficiently diminishes the laser threat and particularly offers the human eye significantly more safety.

Themes

CO

NT

AC

T

vis IT Defense

6

Figure 1: ADS3 95 Ranger drone system of the Swiss Air Force.

ABUL: INTEROPERABLE VIDEO EXP LOITATION FOR AERIAL SURVEILLANCE AND RECONNAISSANCE

Dipl.-Inform. Norbert Heinze

Video Exploitation Systems (VID)Fraunhofer IOSB Karlsruhe



OVERVIEW OF FUNCTIONS

For many years, Fraunhofer IOSB has developed robust, high-performance methods for video exploitation, which are capable of running in real time and which were integrated in the »ABUL« video exploitation system (Automated Image Exploitation using the example of the UAV LUNA drone) and have proven themselves on numerous occasions, inclu-ding international applications. The sys-tem supports the interpreter with follow-ing functions and real-time methods:• Image stabilization (disruptive move-ments of the airborne sensor carrier are compensated)• Mosaicing: Individual consecutive frames from a video image sequence are merged into a high-quality overview in order to obtain a large quantity of geo-referenced image information• Stereo image generation: Stereo images are generated from a video stream and enable 3-D visualization• Multi-resolution images: Images taken at the same time in different resolutions are merged into overviews• Super-resolution images: Multiple images with poor resolution or which are noisy are overlaid to generate a superior image with greater contrast

OBTAINING AN OVERVIEW

UAS (Uninhabited Aerial Systems) are becoming increasingly important when it comes to reconnaissance and surveil-lance missions. Depending on the type of task, many different sizes of UAS are deployed, which range from systems weighing a few grams to flying plat-forms with a weight of several tons. Despite substantial differences in weight and flight duration, the typical sensor arrays used by reconnaissance systems are similar in many respects: They are generally equipped with VIS1 and IR2 video sensors with a large zoom range and transfer video signals to a ground control station, where they are analyzed by specially trained interpreter, via a secured radio channel. This exploita-tion is an exceptionally demanding task, which must be performed in a concen-trated, precise and conscientious man-ner as subsequent mission planning is based on this information among other things. Automatic procedures facilitate processing in order to support the inter-preter – even during long flight times of e.g. 6 h or 24 h – and eliminate possible error sources.

vis IT 7 Defense

Figure 2: ABUL video exploitation system comprising 2 monitors.

Figure 4: Interoperable aerial image exploitation at the Common Shield for Condor and Smaragd aircraft with 4 ABUL stations and on the right: MiSAR6 processing at the LUNA ground control station.

Figure 3: Mobile object detection: Vehicles and walking persons are detected and marked in red to alert the interpreter.

ABUL: INTEROPERABLE VIDEO EXP LOITATION FOR AERIAL SURVEILLANCE AND RECONNAISSANCE

• Real-time super-resolution video processing using the computing capa-city of a GPU4. This function particularly improves quality deficiencies in the image material caused by digital video compression for radio transmission.• Movement detection (objects in motion in the video are automatically detected and marked) – (Fig. 3)

INTEROPERABLE APPLICABILITY

The methods can be applied universally for the processing of sensor equipments of various UAS. This ensures that the exploitation methods developed and tested can be applied for data from

different aircrafts. On one hand, specific interfaces were implemented for the interface to different systems and on the other hand, the NATO-uniform STANAG5 4609 video interface was realized so that in ABUL video streams together with their geo data can be processed from all interoperable systems.Interoperability was also created with respect to results so that mosaic products, super-resolution images etc. can be created as STANAG-compliant products and also be stored on CSD7/NSD8 deve-loped at Fraunhofer IOSB (interoperable database system for storing and distri-buting exploitation products), where they can be distributed to various end customers. An OGC9-compliant geo-

server has also been realized, which also permits real-time distribution of geo-products via other standard interfaces.

SAMPLE APPLICATIONS

ABUL systems are integrated in the ground control station of the German UAS LUNA and are deployed by the German Armed Forces in arenas such as Afghanistan. Various departments of the German Armed Forces and the Swiss Air Force also use ABUL systems. In 2010, the processing and exploitation capabilities of the compact image series SAR10 system, MiSAR, were demonstra-ted as a further potential application.

1 VIS: Visual-optical2 IR: Infrared 3 ADS: Reconnaissance drone system4 GPU: Graphics Processing Unit5 STANAG: Standardization agreement (of NATO states)6 MiSAR: Miniature SAR7 CSD: Coalition Shared Database8 NSD: National Shared Database9 OGC: Open Geospatial Consortium10 SAR: Synthetic Aperture RADAR

Themes

CO

NT

AC

T

vis IT Defense

8

Figure 1: Data sharing with the CSD.

THE CSD PLUS

Barbara Essendorfer M.A.

Interoperability and Assistance Systems (IAS)Fraunhofer IOSB Karlsruhe



EXCHANGE OF INFORMATION

In many cases, reconnaissance information is theoretically available, but cannot be accessed on a practical level. Not only technical aspects like different data formats, but also the »Need to Know« paradigm complicate the exchange of information in the process. This is why the project MAJIIC (Multi- sensor Aerospace-ground Joint ISR Interoperability Coalition), funded by the BMVg, focused on how to efficiently and promptly exchange reconnaissance information from different sensor sources (radar, moving and individual images ...) during joint, international missions of armed forces.

FROM »NEED TO KNOW« TO »NEED TO SHARE«

The success of military reconnaissance is determined by intelligence gained through the analysis of existing data and by being a few steps ahead in terms of time and content. A common problem is that reconnaissance data exists in a proprietary format and is only available on an evaluation system assigned to the sensor. Information can only be passed on to an interested party at request and / or only if informa-tion channels have been clearly defined in advance.

vis IT 9 Defense

Figure 2: Components of the CSD PLUS server (PLUS components are marked in green).

THE CSD PLUS

METADATA CONCEPT The concept of the CSD (Coalition Shared Data) was developed further, implemented and established to this end. In the process, tasking, sensor and exploitation systems save their information in a standardized format on a CSD server. The data is annotated and organized using metadata. The data can then be filtered and accessed by any exploitation or analytical system. CSD servers organize their metadata beyond network limits and localize pro-duct data as necessary (Fig. 1). In this way, information can be retrieved across system and organizational limits. The »Need to Know« becomes a »Need to Share« (need to distribute information). Due to the considerable success of the concept, which has proven itself in practical exercises, various nations and NATO have already deployed CSD servers and are using them for sharing information.

Fraunhofer IOSB is involved in the fur-ther development of the concept and has implemented its own servers and clients. However, an operational environ-ment poses increased requirements on software. For which reason, the study result »CSD Server IOSB« had to be revised and redesigned.

EXPANSIONS AND HARDENING In addition to the need to subject the system to detailed security and system tests, aspects of user friendliness and IT security were to be taken into account. System access operations had to be logged and user administration had to be integrated. Additional data types or metadata should also be made available on a national basis. This requires an extension of the metadata model.

IOSB has developed the concept of the CSD PLUS for this reason. Therefore the CSD is supplemented with a logging component as well as interfaces for user administration. The CSD data model can also be extended by specific attributes and consequently access options. In the process, the core of the data model is maintained and the server continues to be compatible with other CSDs. Other entities and attributes can also be added.

The data model extensions concern, for example, the filing and description of additional product types, refined geo- coordinate retrieval or supplemental specifications (e.g. with respect to security classification).

The adjustment of the extended data model in the respective community of interest (COI) and comprehensive documentation are decisive. Likewise, possibilities for conversion and linking to the conventional CSD data model must be defined. The CSD PLUS is already being shipped to the German Armed Forces in connection with an immediate mission-based need (ESB) and will be deployed in the near future.

Literatur: [1] Bürkle, A.; Essendorfer, B.: Feldlagerschutz. Strategie und Technik. Report Verlag. März 2010[2] Bürkle, A.; Essendorfer, B.; Partmann, T.: Vernetzte Aufklärung und Überwachung. Sicher-heitstechnischer Report. Report Verlag. Frühjahr 2010[3] Essendorfer, B.; Schneider, G.; Kerth, C.: A multilevel approach to interoperability in surveillance and reconnaissance. Proceedings of the annual conference on Data Analysis, Data Quality & Metadata Management 2010[4] Essendorfer, B.; Monari, E.; Wanning, H.: An Integrated System for Border Surveillance. The Fourth International Conference on Systems (ICONS 2009). Gosier, Guadeloupe / France

Themes

CO

NT

AC

T

vis ITDefense

10

Figure 1: Sniper in Afghanistan.

1 John F. Kennedy, U.S. President, 1963† Martin Luther King, U.S. civil rights, 1968† Zoran Djindjic, Serbian Prime Minister, 2003†

PROTECTION AGAINST AMBUSH

Dr. Jürgen Krieg

Signatorics (SIG)Fraunhofer IOSB Ettlingen



aircraft flights. Helicopters in action are often fired upon with barrel weapons. If none of the critical components or passengers is hit, the attack only beco-mes apparent after landing in many cases. Although active protection against such attacks is currently nothing more than a pipe dream, immediate detection of such attacks combined with specifi-cation of direction would be a signifi-cant component for increasing security – risks could be bypassed, areas could be flown around, cover could be taken in the right place.

GUNFIRE DETECTION

For nearly two decades now, many countries have been working diligently on developing corresponding warning devices. Generally speaking, there are several ways to detect a shot or projectile. If rifles are used, the explosion that pro-pels the projectile produces a blast wave. Microphones can be used to measure such blast waves. Radar technology also permits detection of the projectile if it is large enough. The trajectory can be

John F. Kennedy, Martin Luther King and Zoran Djindjic1 are only three well known sniper victims. However, snipers firing from remote locations have cost a much larger number of people their lives since the invention of the rifle. Individuals are not always the target. Sniper fire from covert locations is often employed in order to achieve questio-nable goals. For example, in the Bosnian War, Bosnian-Serb paramilitary deployed sharp shooters to demoralize residents of Sarajevo and to force the surrender of the city. Gaddafi‘s snipers in Bengasi and Misrata likely had similar interests in mind.

EASY TARGETS

In addition to such attacks, where victims are generally located outdoors and are unprotected, people traveling by vehicle are also fired upon. A single hit could, under certain circumstances, prove fatal for all passengers. The same applies for

CO

NT

AC

T Dirk Seiffer

Signatorics (SIG)Fraunhofer IOSB Ettlingen



vis IT 11 Defense

Figure 2: Unnoticed attack on a US helicopter in Afghanistan. The hole is located on the bottom left in the image within the red circle.

PROTECTION AGAINST AMBUSH

determined. The flash produced by the explosion can, however, also be detected with electro-optical detectors. If ade-quate optical resolution is available, the projectile can also be tracked along its trajectory.

URGENT NEED

The increase in asymmetrical or terrorist threats in the last decade and the resul-ting urgent need has led to the early deployment of devices for detecting rifle fire on an operational level, which have only just completed the test phase. The United States and Great Britain in particular often deployed such techno-logy in Iraq and Afghanistan. These warning systems are also being used by US communities to an increasing extent. Cities like Pittsburgh, Newark or Dallas have also set up warning systems in city districts particularly at threat, or they are in planning.Unfortunately, the devices are still not functioning in a satisfactory manner. For most of the systems in use the detection principle is based on acoustic positioning. Particularly in an urban envi-ronment, this can be difficult or even impossible as, among other things, reflections of the sound wave against buildings may cause misinterpretation and thus errors in determining the firing position.

APPROACHES

The expansion of a warning system that functions on a purely acoustic basis with an extra electro-optical component may be the solution needed for exact deter-mination of the direction of fire. In an initial step, a basic implementation for a warning system is being worked on at IOSB, which reliably detects the flash with electro-optical sensors. Studies to date have returned positive results with respect to the use of sensors that function in the infrared spectral range. A high sampling rate could also contri-bute to the reduction of false alarm rates as the temporal signature of the momentary moment of firing can be analyzed. Another problem, which has not yet been resolved, is the realization of the panorama view that is required on an operational level. While a micro-phone picks up blast waves from all directions, the visual field of an optical sensor is in most cases very restricted.

For this reason, more than one sensor must be used to guarantee spacious coverage. Various disadvantages are associated with this, such as increased energy consumption and an increase in size and weight. IOSB is currently working on and testing approaches and ideas for solving this problem. The next logical step involves work to merge acoustics and optronics in order to offer civil and military users a warning system that functions reliably.

PARTNERS

Bundeswehr Technical Center for Weapons and Ammunition (WTD 91) in Meppen

Themes

CO

NT

AC

T

vis IT Defense

12

SiteAnalystASSISTANCE FOR INTERACTIVE IMAGE-BASED INFRASTRUCTURE ANALYSIS

Dr. Elisabeth Peinsipp-Byma

Interactive Analysis and Diagnosis (IAD)Fraunhofer IOSB Karlsruhe



This is accompanied by the fact that the image analyst must be in command of between 100 and 300 predefined terms for describing a specific infrastructure type. Shorter training and working hours make it increasingly difficult for image analysts of the German Armed Forces to accumulate enough experience and knowledge for high-quality evaluation of infrastructure facilities. Despite advancements in the field of automatic image processing, methods for auto-matic object recognition to date have only been able to marginally support the infrastructure analysis. SiteAnalyst is an assistant developed for supporting the image analyst when performing image-based infrastructure analysis.

TASK: IMAGE EXPLOITATION

An important task of image-based reconnaissance is to describe an infrastructure with its objects, sub-areas and functions. Aerial and satellite images serve as a basis for this description. Infrastructures of interest are e.g. ports, airports and industrial plants. Such facilities comprise a complex arrangement of individual objects (e.g. buildings, hardened areas, tanks, pipes and roads), which can be identified readily on the basis of their image signature. Their function (e.g. workshop, administrative building or warehouse) can, however, often only be clearly determined by taking account of their environment and / or objects which are present there. To manage this challenging task the image analyst must acquire exten-sive knowledge from technical literature.

CO

NT

AC

T Dipl.-Inform. Susanne Angele

Interactive Analysis and Diagnosis (IAD)Fraunhofer IOSB Karlsruhe



vis IT13 Defense

SiteAnalystASSISTANCE FOR INTERACTIVE IMAGE-BASED INFRASTRUCTURE ANALYSIS

INTELLIGENT ASSISTANT AND ELECTRONIC MANUAL

SiteAnalyst provides support in three different configuration levels. As a manual, it offers the image analyst a hierarchically structured overview of the terms for all objects, sub-areas and functions, which may occur in an infra-structure type. Interaction is carried out via a tree structure, which offers typical methods for navigation or via the graphical representation of the terms. For each term a profile function is available. It provides as well a text-based description as example images of various sensors and sketches for each term.

On the second configuration level, SiteAnalyst offers suggestions on the type of infrastructure as well as on yet unidentified objects. To use the suggestion functions, the image analyst describes the objects recognized on the image in a step-by-step process, in which he utilizes the

terms provided by SiteAnalyst. The probability of a certain arrangement of object types is modeled in a probabilistic scene model in order to implement the suggestion functions depending on the function of the respective infrastructure. Not just mandatory existing object types (e.g. the presence of dispersal areas in the case of military fighter airports) can be modeled, but also optionally present objects can be included in the model with an assigned probability where the occurrence 0 is integrated in the interval. Using the Bayes theorem, conclusions about the infrastructure’s functions and objects can be made with respect to the objects identified in the image to be analyzed. It is therefore possible to calculate the above suggestions.

The spatial relations between the objects are taken into account on a third con-figuration level in order to improve the suggestions. For this purpose a display

area for the image to be analyzed is added to the user interface where the image can be annotated. The annota-tions are evaluated by the SiteAnalyst assistant and considered in the sug-gestion function. A usability test with image analysts indicated that all three configuration levels are considered to be helpful with the third configuration level providing the best support.

Literature:[1] Bauer, A.: »Probabilistic reasoning on object occurrence in complex scenes«, Image and Signal Processing for Remote Sensing, Proc. SPIE 7477A, 74770A (2009)[2] Bauer, A.: »Assisted Interpretation of Infra-structure Facilities from Aerial Imagery«, Proc. SPIE 7481, 748105 (2009)[3] Bauer, A.: »Probabilistic Scene Models for Image Interpretation, In: Communcations in Computer and Information Science, vol 81, Springer, (2010)[4] Bauer, A.; Herschelmann, O.: »Exploiting Context for Assisted Aerial Image Interpretation«,Proc. SPIE 7835, 78350K (2010)

Themes

CO

NT

AC

T

vis IT Defense

14

Figure 1: Lab model of the mobile sensor system for task force members.

NAVIGATION AND MAPPING FOR MOBILE TASK FORCES

Dr.-Ing. Jochen Meidow

Scene Analysis (SZA)Fraunhofer IOSB Ettlingen



nation. In most cases, satellite naviga-tion is used for positioning and inertial sensors are used for determining direc-tion changes and accelerations.

ORIENTATION IN AN URBAN AREA

Indoors or in urban canyons, radio and satellite signals may be difficult or even impossible to receive due to obstacles along the signal path (shadowing) and interreflections. Radio interferences – also unintentional – and enemy counter measures (jamming, spoofing) are further risks associated with this technology. Dead reckoning as an alternative solution is subject to an unavoidable drift; inertial sensors in the required accuracy class are bulky, expensive and heavy.

Cameras offer attractive possibilities for solving these tasks: They are affordable, light and small and are also plausible for use on highly agile platforms, such as soldiers. As imaging, passive sensors, they enable inconspicuous mapping of the environment at the same time.

INFORMATION SUPERIORITY THANKS TO INTELLIGENT IMAGE ANALYSIS

The success of task forces is particularly determined by a precise assessment of the situation as a basis for mission planning and guidance in addition to factors such as training and experience. Necessary prerequisites for successful action are abilities to reliably orient oneself at the place of action and quickly survey the mission environment including, for example, unknown building rooms.

Navigation and exploration tasks must therefore be solved for the coordination and leadership of task forces. Conventi-onal navigation methods use measure-ments of position, velocity and direction in order to determine the paths covered and in order to reach the desired desti-

vis IT 15 Defense

Figure 2: Task force coordination at the base station. The geo-referenced paths covered by persons are displayed.

NAVIGATION AND MAPPING FOR MOBILE TASK FORCES

SYSTEM CONCEPT

As commissioned by the Technical Center for Information Technology and Electronics of the German Armed Forces (WTD 81, “Navigation” business unit), a concept for a high-performance navigation procedure, which requires minimal interaction, was developed at IOSB in consideration of operational boundary conditions. This provides for the use of light and affordable sensors and takes account of the different types of measurements, their situtation-based availability and the required synchroni-zation due to different acquisition rates.

SOLUTION A system demonstrator was constructed which utilizes modern computer archi-tectures. It consists of a light, mobile mission pack for one person and a stationary base station for mission sur-veillance and command. The mobile mission pack contains sensor equipment and a commercially-available laptop. The integrated sensors comprise a barometric altimeter, a magnetometer, inertial sensors, a GPS receiver and a daylight camera (Fig. 1).

The evaluation of the measurement data occurs in real time and provides geo-referenced sensor trajectories (green) as well as spatial points (blue) determined by means of triangulation. These analytical results are transmitted to the base station via a radio link, where they are superimposed on an aerial photograph (Fig. 2).

The use of imaging sensors for navigation and exploration tasks is an example of innovative technologies, which go beyond the purely visual exploitation of image data.

Literature: [1] Kleinert, M.; Schleith, S.: (2010) Inertial Aided Monocular SLAM for GPS-Denied Navigation. IEEE Conf. on Multisensor Fusion and Integration[2] Kleinert, M.; Schleith, S.; Ascher, C.; Trommer, G. F.; Stilla, U.: (2011) A New Pedestrian Navigation Algorithm based on the Stochastic Clo-ning Technique for Kalman Filtering. International Technical Meeting of the Institute of Navigation

Themes

CO

NT

AC

T

vis IT Defense

16

Figure 1: Tracking of persons using the example of infrared image sequences. The methods utilized are also suitable for images within the visible spectrum. The characteristics extracted during tracking are used for further analytical steps.

PERCEPTION OF PERSONS

tion / MOUT and checkpoint control. In order to be capable of making decisi-ons in a technological respect and also to account for specific military require-ments in particular, comprehensive research has been performed in the field of the perception of persons at IOSB. On the basis of the aforementioned areas of application select work will be presented below.

APPLICATIONS FOR SPECIFIC SITUATIONS

The scope of perception tasks addressed at IOSB with respect to convoy protec-tion / MOUT ranges from the detection of individual persons [3] to pedestrian detection, which is also relevant for civil applications in order to prevent accidents, up to and including the assessment of the behavior of individual persons and groups. For this purpose, approaches for the extraction of corresponding movement verbs and behavior

PERCEPTION OF PERSONS OFFERS CONTROL AND PROTECTION

The perception of persons through the utilization of image exploitation methods has a long tradition in the area of civil applications. There are, for example, already established pro-ducts in the area of person identification, which analyze images of faces, eyes and finger prints. Recently, the Kinect system distributed by Microsoft has proven to be a sensation in the area of human-machine-interaction as it delivers a detailed movement model of a person and her limbs in indoor areas by utilizing an active sensor. Another field where marketable products are expected in coming years is that of automobile safety systems, which is primarily focusing on the prevention of collisions with pedestrians.

The rise of asymmetric warfare scenarios lead to an increased importance of the perception of persons also for military purposes, for example in the areas of field camp protection, convoy protec-

Dr. Michael Arens

Object Recognition (OBJ)Fraunhofer IOSB Ettlingen



Dr. Dieter Willersinn

Video Exploitation Systems (VID)Fraunhofer IOSB Karlsruhe


www.iosb.fraunhofer.deCO

NT

AC

T

vis IT 17 Defense

Figure 2: Automatic detection of a handover situation by means of robust tracking of persons in a crowd of people.

Figure 3: Dual use – commercial application for analyzing soccer games (generation of movement data and situation recognition).

PERCEPTION OF PERSONS

descriptions from video recordings based on the appearance of persons and their posture are investigated [1] (Fig. 1).

In the case of a field camp, where multiple stationary cameras have been installed for monitoring, reidentification of persons from one camera to the next for a prolonged period is necessary in order to completely record and analyze the movement pattern of individual persons [2]. Local characteristics, implicit shape models and manifolds are investi-gated in terms of their appropriateness at IOSB.

Compared to field camp protection, checkpoint control applications provide the particular challenge of detecting and tracking of individual persons or all persons within a crowd. For this purpose, vertical recordings were made and tracking methods were applied to the resulting video sequences. Following further optimization, automatic recognition of a handover situa-tion, which takes place within the protection by the crowd, was realized using the method that returned the best evaluation results [4] (Fig. 2).

STATE OF RESEARCH AND SOLUTIONS THAT HAVE REACHED PRODUCT MATURITY

At IOSB, the perception of persons has a broad scale and covers all spectral channels, camera perspectives and exploitation tasks of military relevance. Perception is achieved for both mobile and stationary camera platforms.

While some topics such as recognition and the identification of behaviors and situations are still the subject matter of research work, the competencies and methods for the detection of persons and their tracking in place at IOSB are available for application developments, both in military operational systems and civil applications in the sense of dual use. An initial commercial application is currently being developed in the field of sporting event analysis (Fig. 3).

Literatur: [1] K. Jüngling & M. Arens: Local Feature Based Person Detection and Tracking Beyond the Visible Spectrum. In: R. Hammoud et al. (eds.): Machine Vision Beyond Visible Spectrum, Springer 2011, pp. 3-32[2] K. Jüngling and M. Arens: View-invariant Person Re-identification with an Implicit Shape Model. In: Proc. 8th IEEE AVSS 2011[3] J. Metzler & D. Willersinn: Toward a sensor-based threat warning system for patrols in MOUT scenarios. In: Proc. SPIE, Vol. 6566, 2007[4] U. Jäger & D. Willersinn: Video Tracker Trajectory Analysis - who meets whom, when and where. In: Proc. SPIE, Vol. 7709, 2010

Themes

CO

NT

AC

T

vis ITDefense

18

1 command and control, communications, compu- ters, intelligence, surveillance, and reconnaissance

ARCHITECTURE-BASED METHODOL OGY FOR SYSTEM-OF-SYSTEMS ENGINEERING

Dr.-Ing. Rainer Schönbein

Interoperability and Assistance Systems (IAS)Fraunhofer IOSB Karlsruhe


www.iosb.fraunhofer.de/?NEC

FUNCTION LISTS

The cross-cutting capability for net centric operations is the improved effi-ciency of systems and organizational units (collectively referred to as »nodes« in this case) which must be effectively and efficiently combined with other nodes on a temporary basis. The descrip-tion of this cooperation is a function list in the form of a sequence of functions performed by various nodes. A function list should also be performed for net centric operations through the coope-ration of various different systems and organizational units (for example from various branches of the armed forces for different nations). The goal is the compilation of function lists in a more flexible and rapid manner than in the past. A network of nodes, which may potentially work together (nodes enabled for NEC) is needed. All nodes available for a mission from the categories of

In addition to the technology-oriented integration of systems the cross-cutting ability for network enabled capabilities (NEC) also requires general willingness and the organizational ability to inte-grate the operational units for networ-king. The service-oriented approach goes beyond technology-focused sys-tem integration and expressly includes the organizational ability to integrate operational units. For this purpose, enterprise architecture frameworks are an important instrument (in the C4ISR1 field of application, the NATO Architectural Framework (NAF)) which enable system-of-systems engineering (SOSE) in a structured process both on an operational and system level [1].

vis IT 19 Defense

ARCHITECTURE-BASED METHODOL OGY FOR SYSTEM-OF-SYSTEMS ENGINEERING

reconnaissance, management (command and control) and effect are present in the network and are available for forming function lists from their capabilities.

NETWORKS

SOSE’s goal goes beyond system inte-gration technology [2]: A system-of-systems is in place if the elements of a network are self-contained systems, which have an independent »owner« and are available for individual orga-nizational regulations. This is also relevant on a technical level because, among other things, individual user administrations and data models may exist. With respect to integration with SOSE means, one does not seek to break up the individual systems on a network, but to temporarily integrate them in a network »as they are« to the greatest extent possible in order to achieve a superordinate function. Merely temporary integration is neces-sary because a system must be capable of participating in multiple networks and also has to fulfill its original task in the process.

APPROACH

SOSE with NAF (as of version 3) means to initially identify the organizational units involved, which are required for the superordinate function, as opera-

tional nodes with their activities, infor-mation elements and communication relationships. The operational activities are compared to potential services, the services are specified as interfaces bet-ween operational nodes, and the per-formance of services is assigned to technical systems. At the end of theNAF procedure, the network architecture and system interfaces are described on a technical level; however, the involved operational nodes on the organizational level had to be analyzed with their ability to be organizationally integrated.

GOVERNANCE

SOA (service-oriented architecture) is a paradigm, with which the further goal of system integration for NEC can be fulfilled. In contrast to other integration paradigms, the SOA understanding also includes consideration of superordinate provision processes, known as SOA governance, which describes require-ments of goals, processes and organi-zation in order to make IT-based abilities available to a business or operational user in a suitable quality. The use of an enterprise architecture framework like NAF directly supports SOA governance processes by means of the required explicit modeling of all dependencies between nodes, capabilities, services, systems, and functions.

Literature: [1] Leuchter, S.; Schönbein, R. (2006): Die Verwen-dung von Architectural Frameworks als Vorgehens-modell für die System-of-Systems-Entwicklung. In: C. Hochberger, R. Liskowsky (Hrsg.), INFOR-MATIK 2006. Informatik für Menschen (Band 1, S. 669-675). Bonn: Gesellschaft für Informatik[2] Jamshidi, M. (2009): Introduction to System of Systems. In: Jamshidi, M. (Hrsg.), System Of Systems Engineering. Innovations for the 21st Century (S. 1-20). Hoboken, NJ: Wiley

Themes

CO

NT

AC

T

vis IT Defense

20

Figure 1: Visualization of the information repository.

INFORMATION MANAGEMENT WITH SEMANTIC MODELS

Dipl.-Inform. Reinhard Herzog

Information Management and Production Control (ILT)Fraunhofer IOSB Karlsruhe



KNOWLEDGE EXPLOITATION

The question is initially raised as to how the pool of information described above can be exploited. As the content is often available in machine-readable documents, auto-matic tapping of content using the methods of »data mining« or »natural language processing« (NLP) come into question. However, they may not provide any entirely reliable classification statement in accordance with the present state of the art and should therefore only be considered supplemental methods. The approach of explicit manual classi-fication was therefore selected for the system described here.

Initially, a clear definition of terms and structures of the information repository are needed for this explicit classification. The questions, which have to be an-

Today, managing an ever increasing quantity of information in increasingly complex organizations has become a serious problem in many areas. This also applies for the German Armed Forces, perhaps to a special extent with respect to studies and analyses. There, valuable and often very specific findings, which must be evaluated in a defined context, result in connection with, in some cases, very large, com-prehensive projects. These findings are in many cases generally documented in extensive reports. At central decision- making bodies, where all information converges, this fortunately large amount of information may, however, become a problem.

In order to establish efficient and reliable access to this extensive pool of results and findings, an information manage-ment system is designed at Fraunhofer IOSB, which is based on semantic model-ing of significant information elements.

vis IT 21 Defense

Figure 2: Semantic query: The interface with a formal appearance is easily recognizable and can therefore be used in a quick, intuitive and effective manner.

INFORMATION MANAGEMENT WITH SEMANTIC MODELS

swered for the analytical tasks in the context described in the introduction, are determined in an initial step using the method of competency questions. The competency questions are typical questions regarding structures, terms and correlations, which may be raised by potential users of the system. These naturally-linguistically formulated questions are a very suitable form for defin-ing the requirements of an information model in a manner that closely relates to users of the system.

MODELING

Afterwards, the formal model of the information repository is derived from these questions in the form of a semantic model. This formalization, also referred to as ontology, is necessary in order to make it possible to interpret the infor-mation model for the information sys-tem software. The Fraunhofer IOSB product WebGenesis, which is capable of generating a functional, web-enabled information management system from a standardized ontology description, is used.

With this approach, it is possible to place the data pool in a semantic context. This supports the user when researching information and can be used for com-puter-based processing and analysis of content.

The result of cooperation with Fraunhofer IGD will be resorted to in this respect for visualizing the ontology. In the process, connection of the IGD SemaVis visualization to the ISOB WebGenesis information management system is realized, which enables the presentation of the semantic network of the WebGenesis entries in an inter-active visualization.

In addition to visualization and navigation, the modeled semantic correlations are also used for formulating semantic queries. Specific questions such as the assessed competency questions are for-mulated in a semi-formal notation in

this respect. A web-based graphic editor in WebGenesis is provided for this purpose. It can be used to retrieve structural information such as the question shown in Figure 2: »which project goals deal with a certain speci-alized topic with the following title?«.

The system is currently in the evaluation phase to investigate the fitness for use in practical applications with specific tasks. Experiences to date appear to be very promising.

The activities described are being funded by the German federal ministry of defense (funding number CE 41F A 227 X).

Themes

CO

NT

AC

T

vis ITDefense

22

Figure 2: Partial sequence of target tracking with possibilities of target confusion, partial concealments, target rotation, loss of sight and automatic reassignment.

Figure 1: Experimental vehicle with laser scanner, signal processor, monitor.

laser radar data and tracking results. Target designation at the beginning of a test drive is performed by marking one of the image points of the current range image on the target vehicle. A 3-D model of the target vehicle is automatically generated from the range image. It is used to differentiate the target vehicle from other vehicles and to recognize it after total occlusions.

All test drives with different target vehicles were performed without track loss. Typical tracking problems such as confusion with similar objects, partial occlusions, target rotations and changes in the target environment, which often result in the loss of tracking when using passive sensor data, were effectively resolved (Fig. 2). Automatic target re-identification after occlusion was suc-cessful, provided the surface geometry of the target vehicle was distinguishable (for the given range accuracy) from other vehicles within the field of view.

The use of laser radar sensors offers the possibility of developing and deploying largely autonomous, unmanned vehi-cles for military and civil applications. One of the basic capabilities of such vehicles is autonomous tracking of a reference object, for example a leading vehicle in a convoy.

The main advantage of laser radar sensors, which assign a distance to each image point, is the ability to effectively separate the background and foreground from the target, to determine the 3-D target position and target orientation and to perform target identification based on models, which are independent of radiometric variables. This enables both error-free target tracking and reliable re-identification of the target after total occlusions.

A scanning laser radar was mounted on a Mercedes Sprinter to demonstrate these capabilities in urban traffic (Fig. 1). A monitor in the vehicle visualizes the

USE OF LASER RADAR FOR UNMANNED CONVOY VEHICLES

Dr. Walter Armbruster

Object Recognition (OBJ)Fraunhofer IOSB Ettlingen



vis IT 23 Defense

A depth map was calculated from a video sequence (extract on the top left), which serves as a basis for surface modeling with area splines (bottom left).

Mini-drones are sensor carriers that can be “always brought along”. They are easy to transport, quickly to deploy and easy to use and thus promise to be the preferred means for quickly satisfying reconnaissance needs directly at the place of deployment. However, due to their miniaturized engineering, they are, however, subject to a number of restrictions, which put them at a disadvantage in various respects in comparison to larger systems. Innova-tive image and video exploitation methods can, however, considerably enhance their performance.

A particularly interesting application field is the acquisition of 3-D informa-tion from mini drone videos, which are, for example, suitable for filling gaps in the stock of 3-D maps, for updating geo-information or, in particular, also for quickly deriving highly up-to-date 3-D models for preparing specific missions.

In order to complete this task, ap-proaches for generating depth maps from monocular image sequences were combined with a method that has proven itself when deriving environ-

mental models from laser scan data. Depth maps from mini-drone videos, however, offer much less precise data than laser scanner point clouds. A fusion method was developed in order to increase the quality of depth maps, which uses the depth maps of individual overlapping image mosaics. In the depth maps merged in this manner, objects can now be located and classified as vegetation or buildings using the method developed for processing laser scanner data.

A description of the surface as area splines is possible as an alternative to this object classification. This may be the only abstract form of representation in the event of demolished buildings, which still retains the object form to the extent possible.

In conjunction with this procedure, mini-drones can be considered an ideal means when it comes to discretely acquiring 3-D data, for example for training in combat simulation systems and for predicting the dissemination of radio waves or visibility analyses with the fewest possible resources.

Dipl.-Ing. Peter Solbrig

Scene Analysis (SZA)Fraunhofer IOSB Ettlingen



3D ENVIRONMENTAL MODELS FROM MINI-DRONE VIDEOS

The same depth map also permits object clas-sification (e.g. buildings and trees), whereupon a 3-D environmental model can be generated with semantic information.

CO

NT

AC

T

KarlsruheFraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSBFraunhoferstraße 176131 KarlsruhePhone +49 721 6091-0Fax +49 721 [email protected]

Ettlingen Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSBGutleuthausstr. 176275 EttlingenPhone +49 7243 992-130Fax +49 7243 992-299www.iosb.fraunhofer.de

IlmenauFraunhofer Application Center System TechnologyAm Vogelherd 5098693 IlmenauPhone +49 3677 4610Fax +49 3677 [email protected]

Lemgo Fraunhofer Competence Center Industrial AutomationLangenbruch 6 32657 LemgoPhone +49 5261 702-572Fax +49 5261 702-5969juergen.jasperneite@iosb-ina.fraunhofer.dewww.iosb-ina.fraunhofer.de

Beijing Representative for Production and Information Technologies Unit 0610, Landmark Tower II 8 North Dongsanhuan Road Chaoyang District 100004 Beijing, PR China Phone +86 10 6590 0621 Fax +86 10 6590 0619 [email protected]

Documents

vis IT - Fraunhofer IOSB · vis IT Defense 2 Topics Hardening of Optical and Electro-Optical Systems ... the UAV LUNA drone) and have proven themselves on numerous occasions, inclu-