Meaghan_Brockhoff_Dissertation

Autonomous Weapons, are they Legal? a1630695

1

Table of Contents Abstract 2 Introduction 3 Chapter 1. What is an Autonomous Weapon? 6

Research Relevance 6 What is an Autonomous Weapon? 10

The Military Decision Making Process 10 Autonomy Incorrectly Defined 11

Automation 11 Autonomy Correctly Defined 13

Chapter 2. Human’s v’s Autonomous Machines 16 The Balance Between War and Humanity 16 Human Weakness and Proponents of Autonomous Weapons 17 Humanity – an Inherent Aspect of IHL 19 Combatant Human Emotion 19 Public Emotion 22 Additional Concerns 23 Robotic Behaviour in Humans that lead to IHL violations 24 Chapter 3. Targeting Law 27

Distinction 27 Customary Law 27 The Codification of Distinction 27 Combatants and Civilians 28 Direct Part in Hostilities 29 Duel Use Objects 31 Cultural Property 33 Hors de Combat 34

Proportionality 35 Customary Law 35 Proportionality Codified 37 The Challenge of Proportionality 37 Why will Autonomous Weapons Struggle with

Proportionality? 40 Proposed Solutions 42

Chapter 4. Accountability 44 Autonomous Weapons and State Responsibility 45 Autonomous Weapons and Joint Criminal Enterprise (JCE) 47 Private Corporations and Accountability 50 Where to From Here? 54 Conclusion 56 Bibliography 58


2

Abstract

Autonomous weapons are the future and as part of Australia’s treaty obligations and

customary law new weapons must be reviewed in order to determine their compliance

with law.

Weapons are considered to be autonomous when they are capable of independent

decision-making and operate with humans outside the OODA loop. As a result

autonomous weapons will struggle to comply with International Humanitarian Law (IHL).

The purpose of IHL is to humanise warfare by balancing military necessity with

requirements of humanity. One of the greatest restraints of war has been the reluctance of

humans to engage in mass killing. Autonomous weaponry dangerously removes this

fundamental restraint.

The principle of distinction protects civilians and combatants. Compliance requires

subjective elements such as intent and mercy, which are inherently human.

Proportionality restricts incidental loss of human life. Attacks must not be excessive in

relation to direct military advantage. The principle requires subjective human input to

adequately minimise the number of acceptable civilian casualties.

Additionally the independent decision making of autonomous weapons makes

accountability problematic; however there are various modes of liability applicable to

those who intentionally use autonomous weapons to violate IHL.


3

Introduction

Governments around the world are pouring millions of dollars into the research and

development of Autonomous weapons. It is imperative that the legality of such weapons

is discussed before these weapons are released onto the battlefield simply because

International Humanitarian Law (IHL) will be violated by their use and whilst individuals

and States can be held to account for such violations the purpose of IHL is to balance

humanitarian concerns with the needs of State militaries during times of armed conflict,

autonomous weapons will distort this balance.

Autonomous weapons are distinguished from other weaponry because they can think for

themselves and do not require humans to help them preform various missions. Proponents

argue they can process more information more quickly and are not subject to human bias

and error. This may well be the case but what autonomous robots cannot do, and what

IHL inherently requires, is the provision of humanity.

The MacQuarie dictionary1 defines humanity as ‘the condition or quality of being human;

human nature, the quality of being humane; kindness; benevolence’. Humanity is a

restraining factor on war, our compassion, kind-heartedness and empathy for our fellow

man makes not only killing the enemy difficult for soldiers it also makes going to war an

unpopular choice for democratic governments. Interestingly IHL is most likely to be

violated when the enemy has been dehumanised, to an autonomous weapon the enemy

cannot be anything but dehumanised, these issues have raised warning bells amongst the

international community.

1 (Macquarie Dictionary Publishers Pty Ltd, 5th ed, 2009).


4

Decisions must be made during war and autonomous weapons will process the data

required to make decisions much faster than a human ever could. However some

decisions inherently require a subjective human element. The principle of distinction at its

simplest demands civilians are distinguished from combatants, in the unlikely scenario

where a robot can make the requisite distinction, by its very nature it lacks the humanity

to make the best choice. The principle of proportionality requires that incidental loss of

civilian life is to be proportionate to the direct and concrete military advantage. This is not

a quantitative formula that can be pre programmed, again a subjective human element is

required, the cost of human life cannot be pre programmed, the advantage is unknown as

are the alternative choices of action. War is not pre choreographed.

IHL is not equipped to cope with the removal of humans from the battlefield; it is too

dependent on the human elements. Removal of humans would mean the law would

require additional law.

In the event that governments ignore warnings such as this dissertation and autonomous

weaponry finds it’s way into the military States will not escape accountability. State

responsibility could be invoked for violations of IHL by autonomous weapons.

Additionally individual combatants working as a group could also be held accountable via

the modes of liability. The modes are used to describe the way in which perpetrators may

be held to account for various types of roles play in in the commission of substantive

crimes. The mens rea of the principle perpetrator is not required for modes such as Joint

Criminal Enterprise ensuring that the use of an autonomous weapon in the commission of

a crime will not prevent the orchestrators or masterminds being held to account.


5

Unfortunately an accountability gap exists for corporations who create and sell such

weaponry.


6

Chapter One What is an Autonomous Weapon?

This chapter will begin with an overview of the international obligations States have in

developing autonomous weapons and why research such as this is important. The chapter

will continue with an explanation of what constitutes an autonomous weapon. To

understand autonomy firstly the human decision making process used in military

operations must be understood as some engineers are trying to replicate the process in

robots. This chapter will then move on to discuss the autonomy spectrum, distinguishing

between automated system and autonomous systems.2

Research Relevance

The scientific community is split on the possibility of fully autonomous weapons with

human like cognitive function.3 Despite this uncertainty the United States is investing

heavily in autonomous weapon development and during the middle of 2000 the United

States Senate Armed Services Committee added $246.3 million to speed development of

unmanned weapons systems,4 some of this research undoubtedly includes autonomous

weapons. The United States is not alone, many other nations, for example the United

Kingdom, China, Germany and others are investing significant amounts into autonomous

weapons research.5

2 A word of thanks must be extended to Ms Fiorina Piantedosi for guiding me through the technical complexities and enabling me to have a sufficient understanding of autonomy to write this dissertation. 3 Human Rights Watch, ‘Losing Humanity The Case against Killer Robots’ (2012) International Human Rights Clinic 1,29. 4 Thomas K Adams ‘Warfare and the Decline of Human Decision making’ (2001) 31(4) Parameters: US Army War College 57,57. 5 Michael C Horowitz The looming Robotics Gap Why America’s Global Dominance in Military Technology is Starting to Crumble (5th May 2014) Foreign Policy The Magazine <http://www.foreignpolicy.com/articles/2014/05/05/the_looming_robotics_gap_us_military_technology_dominance>.


7

Due to information barriers such as classified information it is impossible to know with

any certainty the current level of autonomous weapon development, although Honda has

revealed its new helper robot6 which has independent thought capacity suggesting that

autonomous weapon development may be further advanced than has been disclosed. This

dissertation is not about the feasibility of autonomous weapons development. Instead this

paper takes the stance that such development is possible and focuses on the notion that,

‘[I]f we wish to allow systems to make independent decisions without human

intervention, some considerable work will be required to show how such systems

will operate legally’.7

Additionally,

‘[T]he international community urgently needs to address the legal, political,

ethical and moral implications of the development of lethal robotic technologies’.8

The United Nations has recently headed this call and is to debate the use of killer robots.9

States that are a party to the Protocol Additional to the Geneva Conventions of 12 August

1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol

I)opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December

6 Tim Moynihan Watch the Astounding Dexterity of Honda’s Dancing Humanoid Robot (19th of April 2014) Wired <http://www.wired.com/2014/04/honda-asimo/>. 7 Ministry of Defense ‘The UK Approach to Unmanned Aircraft Systems’ (Joint Doctrine Note 2/11 The Development, Concepts and Doctrine Centre, 30th of March 2011) 5-2. 8 Patrick Worsnip, ‘UN Official Calls for Study of Ethics, Legality of Unmanned Weapons’ Washington Post, (Washington USA), October 24, 2010. 9 BBC, ‘Killer Robots to be Debated at UN’, BBC News Technology, 9 May 2014. <http://www.bbc.com/news/technology-27343076>.


8

1979), Additional Protocol I, and are working on the development of autonomous

weapons have an international legal obligation to review the legality of their progress.

Article 36 states,

In the study, development, acquisition or adoption of a new weapon, means or

method of war, a high Contracting Party is under an obligation to determine

whether its employment would, in some or all circumstances, be prohibited by this

Protocol or by any other rule of international law applicable to the High

Contracting Party.10

Incidentally States who are not a party to Additional Protocol I11 either view the review

process as customary law whilst others deem it to be best practice. 12

Autonomous robots are certainly in the conception stage. This is the point the

International Committee of the Red Cross (ICRC) believes the obligation outlined in

article 3613 begins and continues as the weapon moves into development.14

Component review is insufficient, The Judge Advocate General’s Office argued a new

Predator drone did not require because when the drone was used as a surveillance tool it

10 Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I) opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December 1979). 11 Ibid. 12 Human Rights Watch, above n, 3 21. 13 Article 36 Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I) opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December 1979) 14 Kathleen Lawand and Robin Coupland and Peter Herby, A Guide to the Legal Review of New Weapons, Means and Methods of Warfare. Measures to Implement Articles 36 of API of 1977 November 2006 International Committee of the Red Cross 1,23. < http://www.icrc.org/eng/assets/files/other/icrc_002_0902.pdf >.


9

underwent and passed the review. The Hellfire missile the Predator drone was armed now

armed with had also passed review. However,

‘an existing weapon that is modified in a way that alters its function, or a weapon

that has already passed a legal review but that is subsequently modified..’

must be reviewed again.15 When reviewing robotics for military use, which are not armed,

for example a surveillance drone, the review should consider the potential of the robot for

carry weaponry.16

The purpose of the review is to determine if the weapon is prohibited under international

law.17 To date there are no treaties which ban autonomous weapons. However some

components may be banned, for example a drone could not carry cluster munitions18 nor

could a robot lay mines as that may offend treaties.19 Customary law should also be

considered as it has the capacity to bind States to the obligation of review even if they are

not parties to the Additional Protocol I20 which is largely considered to be customary.

15 Ibid. 16 Human Rights Watch, above n 3, 23. 17Ibid 24. 18 Convention on Cluster Munitions, opened for signature 3rd December 2008, 2688 UNTS (entered into force 1st August 2010). Providing the State has either ratified this convention or it is deemed customary which it does not appear to be. 19 Ibid. 20 Article 36 Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I) opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December 1979).


10

What is an Autonomous Weapon?

The Military Decision Making Process.

The military continues to use the ‘Observe, Orient, Decide, Act’ loop (OODA loop) to

describe human decision making in the combat operations process. 21 Admittedly it is

oversimplified and much of the process is preformed subconsciously by humans. When

humans make decisions the first task is the observation of the world around them, so data

is gathered pertaining to their environment. The next step is orientation, the data is

interpreted, following orientation is the decision, working out how to act by weighing up

the potential causes of action based on the collected and interpreted data, then finally the

action. 22

Technological advances have meant that in some instances a machine may contribute or

replace parts of the OODA loop, indeed machines create and develop more information

than humans are truly capable of absorbing. Discussions about autonomous weapons often

refer to humans in, on or out of the loop. It is the OODA loop to which referral is made.

The OODA decision-making loop provides the basis for leading engineers working on

autonomy such as Thomas Sheridan.23 Sheridan proposes a model that follows the OODA

loop in the development of autonomous weapons. The loop would consist of Information

Acquisition, Information Analysis, Decision Selection and Action Implementation. This

effort by develops is clearly designed to emulate the human decision making process.24

The discussion must now move to precisely what constitutes an autonomous weapon. 21 William C Marra and Sonia K McNeil ‘Understanding “The Loop” Regulating the Next Generation of War Machines’ (2013) 36(3) Harvard Journal of Law and Public Policy 1139, 1146. 22Ibid 1144. 23Ibid 1145. 24Ibid 1146.


11

Autonomy Incorrectly Defined

The United States Department of Defence has defined an autonomous weapon system as:

A weapon system that, once activated, can select and engage targets without

further intervention by a human operator. This includes human-supervised

autonomous weapon systems that are designed to allow human operators to

override operation of the weapon system, but can select and engage target without

further human input after activation.25

This definition is fundamentally flawed as it fails to identify the way in which the weapon

selects and engages targets. Weapons which are both autonomous and run via automation

have the ability to select and engage targets without human in the loop.26 The process or

the degree of autonomy afforded to a weapon is of greatest concern to IHL rather than the

ability of a weapon to preform various tasks with humans out of the loop.

Automation

Understanding autonomy begins with automation. Automated systems fundamentally

differ from an autonomous systems because they are unable to make independent

decisions, instead they work on an ‘if/then’ sequence. A simplified example is a robot

than can be programmed that ‘if’ the green cube27 falls on the floor ‘then’ I pick it up. In

context Israel’s Iron Drome operating near the Gaza border works on such a program. ‘If’

the Iron Drome’s radar identifies a short range rocket and 155mm artillery shells ‘then’ it

25 Michael Schmitt and Jeffrey S Thurnher ‘Out of the Loop: Autonomous Weapon Systems and the Law of Armed Conflict’ (2012-2013) 4 Harvard National Security 213, 235. 26 Marra, above n 21, 1150. 27 Famous programmer Karl Sims used the possession of a green cube to enable his evolving cubes to compete. See Karl Sims, Evolved Virtual Creatures You Tube <http://www.youtube.com/watch?v=F0OHycypSG8>.


12

fires the 20 Tamir interceptor missiles in response.28 Similarly ‘if’ Sea Whiz detects an

incoming anti ship missile ‘then’ it will engage and terminate the anti ship missile.29

There is no reason for IHL to be concerned by weaponry working via automated

programmes. Such weaponry can process information more quickly and accurately than

the human brain, indeed such weaponry should be embraced.

On the 12th of April 1999 North Atlantic Treaty Organisation (NATO) forces legally

identified a bridge in the Grdelica Gorge as a legitimate military objective.30 NATO pilots

engaged the target and fired, upon firing the radar used by the pilots identified a civilian

passenger train to be emerging onto the bridge.31 The pilot was not able to stop the bomb

at this stage. Had the bomb be fitted with a hypothetical ‘if’ vehicle movement is detected

‘then’ do not detonate function civilian lives would have been saved enabling greater

adherence to IHL by NATO. In this particular incident the civilian passenger train was hit

by two bombs due to human limitation that automated systems or ‘if/then’ programing

could overcome.

Automated systems are limited by the data capability, a weapon system may encounter an

‘if’ but may take to long or jam up trying to find the ‘then’. Increasingly systems are

expected to preform tasks with greater complexity and the limitations of automation have

led developers towards autonomous weapons. It is this development that should be of

great concern to IHL.

28 Human Rights Watch, above n 3, 10. 29 Alton Parrish, Gee Whiz! Sea Whiz 4500 Rounds Per Minute For Close In Naval Fighting (10th September 2012) Before it’s News <http://beforeitsnews.com/science-and-technology/2012/09/gee-whiz-sea-whiz-4500-rounds-per-minute-for-close-in-naval-fighting-2467644.html>?>. 30 ITCY, Final report to the Prosecutor by the Committee Established to Review the NATO Bombing Campaign Against the Federal republic of Yugoslavia. (Initiated, 14th May 1999) International Criminal Tribunal for the for the former Yugoslavia) <http://www.icty.org/sid/10052#pagetop>. 31 Ibid.


13

Autonomy Correctly Defined

What then is an autonomous weapon? It is important to note that the term autonomy

begins at one end of a scale, at the other end is artificial intelligence. Perhaps the most

defining factor illustrating autonomy is the capacity of the machine to establish and

peruse its own goals thus attempting to emulate human process rather than simply

replicating them via automation.32 In short an autonomous robot has independent

decision-making capabilities rather than the simple ‘if/then’ sequence. Determining where

a particular weapon sits on the autonomous scale begins with asking three questions.

The first question is how often does the weapon require human intervention to complete

the given mission?33 The higher the level of independence the machine has the greater the

level of autonomy. Consider the robot picking up the green cube, if the cube is knocked

under the table rather than falling straight down as normal can the robot can move

independently without human intervention to retrieve the green cube? If so it is displaying

a relatively high level of autonomy in this respect.

The second question asks how much tolerance the machine has for environmental

uncertainty or how adaptable is the machine to different environments?34 How would our

robot respond if we placed the green cube in the next room? Our robot may be higher up

the autonomous scale and work out that it can’t pass through walls and it must locate a

open door to retrieve the green cube, other less autonomous robots may simply lose the

green cube at this stage and without human intervention the mission would fail. Those

with a high level of autonomy may learn that door enables passage and the next time the 32 Marra above n 21, 1150. 33 Ibid 1152. 34 Ibid 1153.


14

green cube is lost the robot will seek a door as it has learnt that is the best way to pursue

the goal of picking up the green cube.

The third question is the how assertive is the robot?35 Does the machine have the capacity

to change its operating plan to complete the mission without human intervention? Will the

robot use the means it deems necessary to achieve the end result designated by humans?

Returning to our robot, we have told it that keeping the green cube on the table is the

mission. Our robot identifies that another robot knocks the green cube off the table and

moves the green cube from room to room. A particularly assertive robot will attack and

destroy the robot moving the cube thus completing the mission of maintaining the green

cube on the table. A more chilling example is the robot could identify it was a human, not

another robot and use lethal force against that human without emotion or care, to a robot

there is not emotional distinction between a living being or another robot and the green

cube is just as important as the life of a child, a robot does not have the emotion to

differentiate between a green cube and a life.

Robots may become a threat to civilians as they increase in mobility and range.36 South

Korea employs a sentry robot that has heat and motion sensors. The robot from up to two

miles away can detect people and the machine can act without human intervention.37 This

weapon contains a 12.7mm or .50 calibre machine gun with a kill zone of about 1 to 1.5

kilometres.38 Despite the lack of human intervention this is still advanced automated

technology, ‘if’ I detect a human in this range, ‘then’ I fire.

35 Ibid 1154. 36 Human Rights Watch, above n 3, 13. 37 Ibid 14. 38 Ibid 15.


15

Clearly, a large aspect of robotic development centers on robots learning from their

experiences,39 for example our robot may have learnt that the door is the best way to pass

between rooms. Such leaning ability enables a robot to respond to novel situations not

previously envisaged by its developer. 40 The concern for IHL is the level of predictability

regarding what the robot will learn. A fully predictable robot is simply an automated

robot. Autonomous weapons are not at a point where they are ready to hunt and kill the

enemy with humans out of the loop.41 Commentators such as Michael Schmitt and

Thurnher suggest that whilst malfunction is certainly feasible as with any technology the

idea that robots could go rogue should be left to Hollywood,42 other academics are not so

optimistic suggesting that ‘Unfortunately, a full awareness of the risks from autonomous

robots may be impossible,’43 because;

‘While some systems might merely enact pre-programmed moral rules or

principles, autonomous robotic agents might be capable of formulating their own

moral principles, duties, and reasons, and thus make their own moral choices in

the fullest sense of moral autonomy’.44

The capacity of autonomous weapons to make the decision to execute lethal force is

certainly fraught with legal and ethical issues and as will be evidenced in the next chapter

IHL was not designed for the use of autonomous weapons as the law relies on distinctly

human traits.

39 Gary E Marchant, et al, ‘International Governance of Autonomous Military Robots’ (2011) 12 The Columbia Science and Technology Law Review 272, 275-284. 40 Ibid. 41 Schmitt, above n 25, 241. 42 Ibid 242. 43 Marchant et al, above n 39. 44 Peter Asaro ‘How Just Could a Robot War Be?’ (2008) Proceeding of the 2008 Conference on Current Issues in Computing and Philosophy, 50, 52.


16

Chapter Two. Human’s v’s Autonomous Machines

Proponents of autonomous weapons argue the precision and capacity of robots greatly

exceeds that of humans, there is not doubt this is the case however the law requires so

much more, this chapter will explore the inherently human characteristic components of

IHL which cannot be emulated by a machine.

The Balance Between War and Humanity

The purpose of IHL is to establish a balance between humanitarian concerns and the

military needs of states.45 Article 1 Vienna Convention on the Law of Treaties, opened for

signature 23rd May 1969, 1155 UNTS 331 (entered into force 27th January 1980) informs

us that when interpreting a treaty it should be done inter alia ‘in light of [the Treaties]

object and purpose’.46 Thus it is possible to infer that human’s must be involved in the

decision making process when decisions controlled by IHL are being made as they

provide a unique human factor which cannot be emulated by a machine. The removal of

humans from the decision making process is the problem rather than the introduction of

robotics.47

Henri Meyrowitz identified two senses of humanity the first understood humans to be

‘defining characteristic of the human race’48 and the second, more on point, suggested

45 International Committee of the Red Cross What is International Humanitarian Law? (July 2004) Advisory Service on International Humanitarian Law <http://www.icrc.org/eng/assets/files/other/what_is_ihl.pdf>. 46Article 1 Vienna Convention on the Law of Treaties, opened for signature 23rd May 1969, 1155 UNTS 331 (entered into force 27th January 1980). Also see art 2. 47 Vik Kanwar, ‘Post-‐Human Humanitarian Law: The Law of War in the Age of Robotic Weapons’ (2011) 2 Harvard National Security Journal Vol 616, 618. 48Ibid.


17

humanity meant ‘a feeling of compassion towards other human beings’.49 Decreasing the

human role in decision making and replacing it with robots will see respect for humanity

decrease.50

There are however proponents of autonomous weapons, Gordon Johnson of the Joint

Forces Command at the Pentagon is one, and recently stated ‘The lawyers tell me there

are no prohibitions against robots making life or death decisions.’51 There may not be any

international instruments prohibiting such use, nor is it spelled out in customary law.

However the law inherently requires humans to make life and death decisions either at the

automation programing level or during the OODA loop process in the heat of battle.

Human Weakness and Proponents of Autonomous Weapons

The Asian Experiment52 suggests that when a question or choice is framed with two

possible outcomes, one which is risk free 200 people will definitely be saved opposed to

the other containing risk where by there is one third of a chance that 600 people will be

saved and a two thirds chance no one will be saved. People will generally chose the risk

free option as humans are risk adverse. When a decisional outcome is framed in a way in

which losses are the result humans will prefer a preference for risk. When the scenario

changes to 400 people will die for sure or there is one in three chance no one will die and

two in three chance 600 people will die then human decision makers will chose the risk

whereby no one will die. Machines do not suffer such limitations and they not influenced

by how a question is framed.

49Ibid. 50Ibid. 51 Noel Sharkey, ‘Cassandra or False Prophet of Doom: AI Robots and War’ (2008) 23(4) IEEE Intelligent Systems 14,15. 52 A Tversky and D Kahneman ‘The framing of decisions and the psychology of choice’ (1981) 211(4481) Science 453, 453-‐458.


18

Significant bias also exists in human decision-making, humans use a system of cognitive

processing which ignores some information enabling us to cope with the massive amount

of information surrounding us, this is known as heuristics. 53 Additionally humans

encounter certain bias in decision-making such as availability, hindsight and fundamental

attribution biases. 54 These shortcuts and bias experienced by humans can lead to

decisional errors. Machines are not susceptible to such bias and can process information

more quickly and often more accurately avoiding human psychological problems.55

Technology continues to develop rapidly, information-based systems are producing data

overload making it more difficult if not impossible for humans to directly intervene in

decision making.56 For example giving humans the capacity to veto a machine decision

can be problematic due to bias. Often decisions must be made in a split second and

through bias humans will usually trust a machine.57 In such a scenario humans don’t

really have the capacity to override the machine’s decision due to bias.

Autonomous weapons undoubtedly have additional advantages, as Gordon Johnson

pointed out,

‘They don’t get hungry. They’re not afraid. They don’t forget their orders. They

don’t care if the guy next to them has just been shot.’58

53 Gerd Gigerenzar and Wolfgang Gaissmaier ‘Heuristic Decision Making’ (2011) 62 Annual Review of Psychology 451. 54 Stephen Choi and AC Pritchard, ‘Behavioural Economics and the SEC’ (2003) 56(1) Stanford Law Review 1, 27-‐33. Other bias include overconfidence and Groupthink. 55 Marchant et al above n 39, 280. 56 Adams, above n 4, 2. 57 Human Rights Watch, above n 3, 13. 58 Sharkey, above n 51.


19

Unless development is restricted autonomous weapons may take over the more dangerous

jobs saving military lives.59 Autonomous robots can be self-sacrificing and proponents

argue they will act without negative emotion such as fear or anger.60 At first glance

autonomous weapons do appear to offer valuable assistance to military operations.

Unfortunately the disadvantages far outweigh the advantages discussed above.

Humanity – an Inherent Aspect of IHL.

Combatant Human Emotion.

Autonomous weapons advocates suggest human emotion leads to error and violation of

the law but in actual fact the opposite appears to be the case.

‘One of the greatest restraints for the cruelty in war has always been the natural

inhibition of humans not to kill or hurt fellow human beings. The natural

inhibition is, in fact, so strong that most people would rather die than kill

somebody…Taking away the inhibition to kill by using robots for the job could

weaken the most powerful psychological and ethical restrain in war. War would be

inhumanely efficient and would no long be constrained by the natural urge of

soldiers not to kill.’61

Human emotion acts as a safeguard to the protection of life during times of war as humans

find it difficult to kill their fellow humans. Psychological studies reveal that

59Marchant et al, above n 39, 275-‐276. 60 Marchant et al, above n 39, 280. 61 Krishnan, Armin ‘Killer Robots Legality and Ethicality of Autonomous Weapons’ (Ashgate Publishing Limited, 2009) 130.


20

‘the average and healthy individual…..has such an inner and usually unrealized

resistance towards killing a fellow man that he will not of his own volition take

life if it is possible to turn away from that responsibility’.62

An apt example was provided by the United States Army Air Corps during the Second

World War. It was discovered that only one precent of enemy aircraft destroyed could be

attributed to being shot down by the United States fighter pilots. Some believe this was

due to fear, others more accurately identify that when the time to kill eventuated the

pilots,

‘looked into the cockpit at another man, a pilot, a flier, one of the brotherhood of

the air, a man frighteningly like themselves; and when faced with such a

man……..could not kill him’.63

War veteran Lieutenant Colonel Dave Grossman believed that identifying the ‘resistance

to kill one’s fellow man is…..strong, and it gives us cause to believe that there just may be

hope for mankind after all’. 64 The introduction of autonomous weapons onto the

battlefield will rapidly erode such hope.

A killer robot does not feel restraining emotions and it will have no problem killing

civilians or combatants. Killing to an autonomous weapon is no different to navigating its

way around a building it feels nothing. Human emotion enables a combatant to show

characteristics such as compassion, which in turn protects life. For example a robot for

62 Dave Grossman, ‘On Killing The psychological Cost of Learning to Kill in War and Society’ (Hachette Book Group, revised ed, 2009) 30. 63 Ibid 31. 64 Ibid 40.


21

may shoot a child and the attack may be lawful. A combatant on the other hand may find

a more compassionate or merciful solution because of the emotion felt. Alternatives may

include the capture of the child or advancement by the combatant in a different direction

keeping the child alive.65 The law inherently relies on this restraining emotion without

which additional law would be required to ensure the safeguards that are currently

provided by human combatants.

Commentators have suggested that war via autonomous weapons would become no

different to a video game,66 where killing is not refrained because human emotion is

lacking. Presently humans are in the OODA loop and operate automated systems such as

drones, from a distance. A United States Air force Pilot who operates the predator drones

says,

‘People think we’re sitting here with joysticks playing a video game, but that’s

simply not true….These are real situations and real-life weapons systems. Once

you launch a weapon, you can’t hit a replay button to bring people back to life’.67

Indeed psychologist and professor of cognitive science suggests that the detail and level

of monitoring prior to a drone attack means combatants are more emotionally impacted as

they see the before and after the detonation of the weapon clearly on a computer screen.68

The removal of the emotional restraint by replacing pilots with autonomous weapons, so

that only the weapon sees the screen before and after detonation would be emotionless

65 Human Rights Watch, above n 3, 38. 66 Ibid 40. 67 Denise Chow, Drone Wars: Pilots Reveal Debilitating Stress Beyond Virtual Battlefield (5th November 2013) Live Science <http://www.livescience.com/40959-‐military-‐drone-‐war-‐psychology.html>. 68 Ibid.


22

and certainly mimic a video game where the loss of human life is of no consequence. IHL

relies on these emotional restraints.

Public Emotion.

The development of autonomous weapons may also decrease the political barriers of entry

into war. The citizens of most nations are usually against going to war when it can be

avoided and almost always against any type of war that is deemed unjust.69 Due to these

attitudes when democratic nations opt to partake in armed conflict certain amounts of war

propaganda is required.70 Technologies, which decrease the risk of nations military

personnel, are popular propaganda choices, which often lead to significant funding of

such technologies. Autonomous weapons are deemed to be a ‘safe’ method of fighting.71

An example of response to political pressure and war was evident when NATO

responding to political pressure and utilised the idea of ‘safe’ fighting during the Kosovo

conflict in 1999 implementing the zero casualties policy.72 NATO jets flew at heights

which were out of enemy missile range ensuring pilots were not shot down.73 Arguably

this policy led to increased numbers of civilian casualties74 because the pilots were too

high to see targets adequately. NATO attacks involving civilian casualties were not found

to violate the principle of distinction or proportionality.75 But it should be noted that

during this NATO campaign humans were clearly in the OODA loop so emotional

restraint was present. The situation may become very different if humans are outside the

69 Asaro, above n 44, 54. 70 Ibid. 71 Ibid. 72 ICTY, above n 30. 73Asaro, above n 44. 74 Ibid. 75 ICTY Above n 30.


23

loop and autonomous weapons were utilised to make a war more palatable to a nations

population.

Furthermore decreasing combatant casualties may be domestically popular but for how

long? At what point does a war become one-sided killing with no risk to one side and only

economic cost?76 There is a difference between decreasing risk and the removal of

combatants being placed in any danger.77 Political barriers must remain in tact to ensure

wars are only ever fought when there is no other feasible option.

Additional Concerns

Autonomous weapons wielded by a repressive autocrat could be catastrophic for civilians

because there would be no emotional constraint on killing. Combatants find it

exceptionally difficult to fire against their own people for illegitimate reasons, in such

cases the most hardened combatant can turn on their leader. Such is not the case with

autonomous robots again the lack of human emotion endangers more life that it can ever

protect.78

The battle area could also be expanded as robots can travel and strike further.79 When the

autonomous weapon is deployed as part of a hybrid machine human mix the robot can

independently and objectively monitor behaviour on the battlefield,80 and report any

perceived violations of IHL to superiors. Some advocates for autonomous weapons go as

76 Christof Heyns Report of the Special Rapporteur on Extrajudicial Summary or Arbitrary Executions Agenda Item 3, A/HRC/23?47/Add.4, (23rd of April 2013), Agenda Item 3, 12. 77 Ibid. 78 Human Rights Watch, above n 3, 38. 79 Marchant, et al, Above n 39, 275-‐276. 80 Ibid 280.


24

far to suggest that if a robot is so good it may be able to refuse an order,81 such a

suggestion offends humanity as it elevates the status of a robot to higher level than that of

a combatant.

Robotic Behaviors in Humans that lead to IHL Violations.

Certain robotic type emotions can be identified in people who violate IHL. Violations of

IHL have been found to occur when a combatant experiences a moral disengagement

from the enemy. This often happens when the combatant identity’s themselves as the

victim of the enemy. 82 A robot will not experience feelings of victimisation and it will

never experience a moral disengagement because it is purely amoral.

Combatants have also been found to violate the law when they become part of a group

and their self-identity is within that group rather than as an individual. The group

membership serves to limit their feelings of personal responsibility and the value

judgements they make about their actions.83 The notion of personal responsibility or the

need to assess whether a particular decision was right or wrong without humanity is not

possible, autonomous weapons could not preform this task as they do not have a moral

conviction.

The enemy is often dehumanised in training and the combatant accepts this as

dehumanisation as legitimate,84 killer robots by their very nature do not feel anything, any

target by a robot is dehumanised. On the basis that these characteristics within humans

81Ibid 281. 82Daniel Munoz-‐Rojas and Jean Jacques Fresard, ‘Roots of Behavior in War’ (2004) 86(853) IRRC 189, 198. 83 Ibid 194. 84 Ibid.


25

have been shown to lead to violations of IHL robots on the battle field are even more

concerning.

These human traits or lack there of, moral disengagement, personal responsibility, lack of

value judgement and a dehumanised enemy allegedly lead to violations of IHL.

Programming an autonomous weapon with these uniquely human traits is impossible and

it is incomprehensible that autonomous weapons, with the capacity to think for

themselves to target and fire are lacking in such fundamental human restraints. Violation

of IHL by autonomous robots appears inevitable and the idea they will one day be on the

battlefield is simply inconceivable.

Many experts share these concerns. Last year over 270 artificial intelligence experts,

robotics experts and scientists from 37 countries submitted an open letter requesting the

development and deployment of inter alia, autonomous weapons to be prohibited.85 Neol

Sharkey robotics expert states, ‘We don’t want [unmanned aerial vehicle’s] selecting

targets and working out how best to carry out an attack’.86 The ICRC warns ‘[A]ll

predictions agree that if man does not master technology, but allows it to master him, he

will be destroyed by technology’.87 The Special Rapporteur reiterates this warning

suggesting some action is required immediately by way of transparency, accountability

and the rule of law and these items must be placed on the agenda.88

85 Noel Sharkey, Computing Experts from 37 Countries Call for Ban on Killer Robots (16th October 2013) International Committee for Robot Arms Control <http://icrac.net/2013/10/computing-‐experts-‐from-‐37-‐countries-‐call-‐for-‐ban-‐on-‐killer-‐robots/> 86 Ibid. 87 International Committee of the Red Cross, Commentary on the Additional Protocols of 8 June 1977 to the Geneva Conventions of 12 August 1949 (Geneva: Martinus Nijhoff Publishers, 1987). 88 Heyns, above n 74.


26

This chapter has illustrated the benefits of uniquely human emotions and the role these

sentiments plays in armed conflict. The story is not complete, another reason for the

violation of law is that whilst combatants may have knowledge of the law when it comes

to the application in real life situations, respect for the law and understanding of the law

may become blurred.89 Autonomous weapons are likely to suffer similar issues, as the

application of IHL often requires a subjective analysis of which a robot is incapable, this

is discussed further in the next chapter.

89 Munoz-‐Rojas, above n 82, 197.


27

Chapter 3 Targeting law

Targeting law, an aspect of IHL consists of two fundamental principles, distinction and

proportionality. Both require a subjective human element to ensure they are not breached

this is why autonomous weapons will not be able to adhere to IHL to the same degree

combatants can.

Distinction

Customary Law

The ICRC considers the principle of distinction to be customary law90 because many

military manuals identify the principle, various states have passed legislation ensuring

adherence and several official statements have been made acknowledging and adopting

this rule.91 The International Court of Justice identified distinction as one of the ‘cardinal’

and ‘intransgressible principles of international customary law’.92 Once a principle or law

is identified as customary it means that all States are bound to comply, thus States must

comply with the principle of distinction.

The Codification of Distinction.

The principle of distinction was codified in 1977 with the ratification of Additional

Protocol I.93 Distinction is a fundamental rule which states civilians and civilian objects

must be distinguished from military objectives and civilians or their objects must not be

90Jean-‐Marie Henckaerts and Lousie Doswald-‐Beck, Customary International Humanitarian Law Volume I: Rules (Cambridge University Press, 2005) 25. 91 Ibid. 92Nuclear Weapons (Advisory Opinion) [ 1996] ICJ Rep 225 No 93 [79]. 93 Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I) opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December 1979).


28

the object of attack.94 This concept is stated in Article 51(2)95 and Article 52(2)96

reiterating the above points. In practice this means that the aiming point of the attack must

be a military target although some collateral damage is acceptable97 the Rome Statute

holds violation of the principle of distinction to be a war crime.98

Combatants and Civilians

The International Criminal Tribunal for the former Yugoslavia (ICTY) asserts the

principle of distinction, stating that a ‘person shall not be made the object of attack when

it is not reasonable to believe……that the potential target is a combatant’.99

Autonomous expert Dr Peter Asaro suggests it would be extremely difficult for an

autonomous weapon to adhere to the principle of distinction.

‘The relevance of the civilian-combatant distinction to robotic soldiers is that if

they are to be autonomous in choosing their targets, they will have to be able to

reliably distinguish enemy combatants from civilians……it is extremely difficult

to identify particular people, or even types of people, much less to categorize them

reliably into groups such as “friend” or “foe,” the boundaries of which are often

poorly defined and heavily value-laden.’100

94 Ibid Article 48. 95 Ibid. 96 Ibid. 97 Krishnan, above n 61, 93. 98 Art 8(2)(b)(ii) Rome Statute of the International Criminal Court, opened for signature 17th July 1998, 2187 UNTS 90 (entered into force 1 July 2002). 99 Prosecutor v Galic, (Trial Judgment) (International Criminal Tribunal for the Former Yugoslavia, Trial Chamber, IT-‐98-‐29-‐A 5th of December 2003)[56]. 100 Asaro, above n 44, 63.


29

The following scenario illustrates the types of failures that would be made by autonomous

weapons trying to apply the principle of distinction. There may be children playing with

toy or even real guns near a soldier. A robot would easily mistake the children as

combatants with disastrous consequence. The intent of the children is easily recognised by

the human soldier because humans understand each others emotional states in ways robots

cannot. Indeed in this situation the combatant may empathise with the child remembering

their youth or children he or she is close to at the present time. The ICTY asks what is

‘reasonable’101 to believe. A person can only answer this question. There is no such thing

as a reasonable or unreasonable robot, autonomous weapons acting outside acceptable

parameters set by humans are broken and personification should not be attempted.

Direct Part in Hostilities

Article 51(3) of Additional Protocol I102 provides that civilians who take ‘direct part in

hostilities’ are not protected by the principle of distinction.103 The ICTY endorsed the

exception,

‘The protection of civilian and civilian objects provided by modern international

law may cease entirely to be reduced or suspended….if a group of civilians takes

up arms….and engage in fighting against the enemy belligerent, they may be

legitimately attacked by the enemy belligerent…’104

101 Prosecutor v Galic, (Trial Judgment) (International Criminal Tribunal for the Former Yugoslavia, Trial Chamber, IT-‐98-‐29-‐A 5th of December 2003)[56]. 102 Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I) opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December 1979). 103 Ibid Art 51(3). 104 Prosecutor v Kupreskic (Trial Judgment) (International Criminal Tribunal for the Former Yugoslavia, Trial Chamber IT-‐95-‐16-‐T, 14th of January 2000) 205[522-‐523].


30

Modern warfare has taken place in urban surrounds amongst the civilian population.105

‘Direct participation in hostilities’ rather than a distinguishing uniform often identifies

combatants. 106 The meaning of ‘direct participation in hostilities’ is uncertain and

presently there is no consensus,107 despite the ICRC drafting of a set of controversial

guidelines, a summary of which suggests it is to engage in or directly support military

operations.108

Despite this uncertainty ascertaining the intent would play a key role in complying with

the direct participation in hostilities aspect of distinction. An autonomous weapon would

struggle to determine direct participation in hostilities due to its inability to determine the

intent of an individual. Paul Bello suggests that,

‘[i]n context where we cannot assume that everyone present is a combatant, then

we have to figure out who is a combatant and who is not. This frequently requires

the attribution of intention’.109

Humans use an emotional state to interpret intention but

‘a system without emotion….could not predict the emotions or action of others

based on its own states because it has no emotional states’.110

105 Consider the wars in Iraq, Afghanistan and the current Ukraine crisis, all of these wars have seen fighting in urban settings. 106 Human Rights Watch, above n 3, 30. 107 Ibid. 108Nils Melzer, Interpretive Guidance on the Notion of Direct Participation in Hostilities under International Humanitarian Law Page International Committee of the Red Cross 22-‐24 <http://www.icrc.org/eng/assets/files/other/icrc-‐002-‐0990.pdf >. 109 Human Rights Watch, above n 3, 31. 110 Ibid.


31

Clearly autonomous weapons will struggle with the most basic forms of distinction. The

principle itself becomes more complex further supporting the fact that autonomous

weapons have no role in modern day warfare because they cannot adhere to basic IHL

principles.

Dual Use Objects

The principle of distinction also applies to dual use objects, that is objects which are used

by both civilians and the military. Subjective human interpretation is required to ensure

respect for the law and the following example illustrates just how inept autonomous

weapons would be at the application of a slightly more complex decision requiring

consideration of the principle of distinction.

The NATO bombing of the Serbian TV and Radio Station in Belgrade on the 23rd of

April 1999 saw NATO claim that the bombing of the target was a legitimate military

target because it ‘disrupted and degraded the C3 (Command, Control and

Communications) network’. The report found the station was a legitimate military target

on the C3 grounds.111

However NATO military spokesperson added that they needed to stop the propaganda of

the Milosovic regime, and this was done through the bombing of the civilian television

and radio network,112 British Prime Minister Tony Blair endorsed the bombing stating that

the civilian network,

111 ICTY, above n 28, [75]. 112 Ibid [74].


32

‘is the apparatus that keeps him [Milosevic] in power and we are entirely justified

as NATO allies in damaging and taking on those targets’.113

The report into NATO’s actions stated that disrupting propaganda in this instance was

insufficient to constitute military advantage, had there been no C3 justification the attack

would have been illegal. The report continued informing us that the propaganda being

aired in Rwanda facilitating genocide may have made the radio station a legitimate

military target.114

An autonomous weapon will struggle to understand these arguments and distinctions

because they are inherently human and require a thought process with various shades of

grey rather than black and white programing of a machine. Determining the meaning of

radio broadcasts in a foreign language and then ascertaining the difference between

potentially attacking civilian moral or attacking a broadcast designed to incite violence

would be extremely challenging and very easy for an autonomous weapon to make an

error. What the law inherently requires is a human’s understanding of the language and

the messages conveyed, a human would have to remain in the loop to ensure compliance,

as it would be reckless to let a killer robot make such a decisions.

113 ICTY, above n 28, [74]. 114 Ibid [76].


33

Cultural Property

Customary law protects cultural property, ICRC Customary IHL Rule 39115 says

‘The use of property of great importance to the cultural heritage of every people

for purposes which are likely to expose it to destruction or damage is prohibited,

unless imperatively required by military necessity.’116

The Convention for the Protection of Cultural Property in the Event of Armed Conflict

with Regulation for the Execution of the Convention , opened for signature 14th of May

1954 UNESCO (entered into force 7th of August 1956) provides a special protection regime

that has been initiated. The regime enables property ‘of very great importance’ to be

placed under special protection.117 Such property must be marked with a special emblem

during hostiles.118 States have obligations under this regime to ensure potential military

objectives are not placed within range of cultural property.119 Few States have signed up

for the scheme.120

Provided States were to adhere to the obligation to ensure military obligations were not

stored near the protected site this concept may be workable concept for autonomous

weapons. Autonomous weapons may actually be able to comply with IHL in this instance.

Indeed initiative could take this concept even further and such sites may be shelter for

115 Henckaerts, above n 88, 131. 116 Ibid. 117 Article 8 Convention for the Protection of Cultural Property in the Event of Armed Conflict with Regulation for the Execution of the Convention, opened for signature 14th of May 1954 UNESCO (entered into force 7th of August 1956). 118 Ibid Article 16. 119 Ibid Article 9 and Article 8(5). 120 CJS Forrest, ‘The Doctrine of Military Necessity and the Protection of Cultural Property During Armed Conflicts’ (2007) 37(2) California Western International Law Journal 177, 208.


34

civilians. One possible means of compliance however does not negate a multitude of

failures.

Hors De Combat

ICRC Customary Rule 47121 prohibits the attacking of persons who are hors de combat or

out of the fight and this includes ‘anyone who is in the power of an adverse party; anyone

who is defenceless because of unconsciousness, shipwreck, wounds or sickness; or

anyone who clearly expresses an intention to surrender, provided he or she abstains from

any hostile act and does not attempt to escape’.

Article 41122 codified hors de combat. A person is hors de combat if they are ‘in the power

of an adverse Party’,123 intent to surrender has been plainly expressed124 or a person is

‘rendered unconscious or is otherwise incapacitated by wounds or sickness, and therefore

is incapable of defending [themselves]’.125

Intent has been discussed above and its application to the intent to surrender is clearly

fraught with difficulties for an autonomous weapon. The robot could either be tricked or

not understand the intent to surrender and instead use inappropriate lethal force.

Autonomous weapons will struggle with incapacitated combatants. Sufficient funding

could add medical equipment to the robot enabling it to easily scan the heart rate, monitor

breathing and other vital signs to determine whether the combatant is out of the fight or 121 Henckaerts, above n 88, 164. 122 Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I) opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December 1979). 123 Ibid Article 41(2)(a). 124 Ibid Article 41(2)(b). 125 Ibid Article 41(2)(c).


35

just pretending. What the robot cannot do is account for shear will power, or mind over

matter that is unique to the human psyche. The robot will always be vulnerable, it will

either have to use lethal force to ensure an injured combatant does not find strength when

medics say there is no strength to find or place itself in a vulnerable position. Furthermore

medical equipment can be sensitive to certain devices such as mobile phones, a relatively

easy way for the enemy to confuse the robot.

At a more simplistic level where a autonomous weapon does not carry a medical triage a

robot may have great difficulty determining whether or not a person it shot has been killed

or mealy fallen to the ground faking the injury.126 How can a robot make decisions when

it is incapable of instinct, fear and survival and as such unable to emulate human emotion?

Hors de combat like all the other aspects of the principle of distinction relies on these

instincts.

Proportionality

Customary Law

Initially proportionality developed as customary law contributing to the just war theory,

requiring the overall evil of the war to be proportionate to the overall good. A just war

was a proportional war.127 Current customary law pertaining to proportionality can be

defined as

‘Launching an attack which may be expected to cause incidental loss of civilian

life, injury to civilians, damage to civilian objects, or a combination thereof, which

126 Human Rights Watch, above n 3,35. 127 Judith Gardam, Proportionality as a Restraint on the Use of Force (1999) 20 Australian Year Book of International Law, 161, 163.


36

would be excessive in relation to the concrete and direct military advantage

anticipated, is prohibited.’ 128

Combatants received customary law protections via limitations on the weaponry an

opposing party could use. Methods of harm that would impose unnecessary suffering or

superfluous injury were not endorsed.129 In other words combatants are not able to inflict

whatever damage they like on each other.130

Many military manuals incorporate the principle of proportionality131 so does some State

legislation that makes violations of the principle of proportionality an offence, this

legislation is supported by official statements who were not party to Additional Protocol

I132 at the time. The International Court of Justice received State submissions regarding

the Nuclear Advisory Opinion,133 in some of those submissions States who were not a

party to Additional Protocol I134 endorsed proportionality.135 The International Court of

Justice identified the principle of proportionality and deemed ‘respect for the

environment’ as one of the elements that go to assessing whether an action is in

conformity with the principles of necessity and proportionality’.136

128 Henckaerts, above n 88, 46. 129 Judith Gardam, above n127, 165. 130 Henckaerts, above n 88, 237. 131 Ibid 46. 132 Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I) opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December 1979). 133 [ 1996] ICJ Rep 225 No 93. 134 Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I) opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December 1979). 135 Nuclear Weapons (Advisory Opinion) [ 1996] ICJ Rep 225 No 93. 136 Nuclear Weapons (Advisory Opinion) [ 1996] ICJ Rep 225 No 93 [41].


37

Proportionality Codified

Additional Protocol I137 codified proportionality,

‘[A]n attack which may be expected to cause incidental loss of civilian life, injury

to civilians, damage to civilian objects, or a combination thereof, which would be

excessive in relation to the concrete and direct military advantage anticipated.’138

The Rome Statute139 provides that failure to comply with the principle of proportionality

is a war crime.

The principle of proportionality is entrenched as a part of International Humanitarian Law

but it must be noted that it ‘is easily stated, [but]there is no question that proportionality is

among the most difficult of [the Law of Armed Conflict] norms to apply’.140 ‘The military

says [calculating proportionality] is one of the most difficult decisions that a commander

has to make.’141

The Challenge of Proportionality

The proportionality test is not about balancing quantitative data. The Prosecution v

Gotovina (Trail Judgment) (International Criminal Tribunal for the Former Yugoslavia,

Trial Chamber IT-06-90-T 15th of April 2011) case provides an excellent example of how

trying to balance quantitative data fails to adequately measure proportionality. Through

137 Article 51(5)(b) Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I) opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December 1979). 138 Ibid, Also see Article 57(2)(iii) Precautions in attack. 139 Article 8(b)(iv) Rome Statute of the International Criminal Court, opened for signature 17th July 1998, 2187 UNTS 90 (entered into force 1 July 2002). 140 Michael N. Schmitt, ‘Essays on Law and War at the Fault Lines’ (Asser Press, 2012) 190. 141 Noel Sharkey, ‘Killing Made Easy: from Joysticks to Politics’ in Patrick Lin and George Bekey and Keith Abney (eds) The Ethical and Social Implications of Robotics (Cambridge: MIT Press, 2012) 123.


38

expert witness’s the Trial Chamber examined all the quantitative data pertaining to

artillery fire. Elements such as the type of guns, potential firing distances as well as

external factors such as muzzle velocity, wind speed, air temp and density were all

accounted for by the formula.

Using this data the Trial Chamber concluded that if a shell landed further than 200 meters

from a lawful target the attack was an intentional or indiscriminate attack on civilians.142

This assumption was the keystone in the guilty verdict for Gotovina and others.143

Prior to the appeal an Amicus Curiae,144 comprising of experts in the practical application

of the law of war in military operations found the 200 meter Trial Chamber finding to be

‘inconsistent with the realities of operational employment of artillery and other indirect

fire assets’.145 The decision was subsequently overturned by the Appeal Chamber on the

basis that the 200-meter standard was unanimously held to be an invalid legal standard146

and once removed the attacks found not to be in violation of IHL.147

Purely quantitative measurements such as the 200 meter standard are inadequate, a unique

psychological input is also required. The ICRC suggests the proportionality tests allow for

a ‘fairly broad margin of judgement…[and]….must above all be a question of common

142 Prosecution v Gotovina (Trail Judgment) (International Criminal Tribunal for the Former Yugoslavia, Trial Chamber IT-‐06-‐90-‐T 15th of April 2011)[1898]. 143 Gary D Solis, ‘The Gotovina Acquittal: A sound Appellate Course Correction’ (2013) 215 Military law Review 78, 86. 144 Amicus curiae is Latin for ‘a friend of the court. A person, usually a barrister who, with the court’s permission may advise the court on a point of law or on a matter of practice’. They have no personal interest in the case. Peter E Nygh and Peter Butt (eds) Butterworths Australian Legal Dictionary (Butterworths 1997) 52. 145The Prosecutor v Gotovina and Markac, (Ante Gotovina’s Response to ‘Application and Proposed Amicus Curiae Brief’)(International Criminal Court for former Yugoslavia) IT-‐06-‐90-‐A (13th January 2012) [9(4)]. 146 Solis, above n 141, 94. 147Ibid, 87. The joint criminal enterprise charge was also overturned.


39

sense and good faith for military commanders’.148 The Red Cross has accurately identified

that subjective human input is required.

Additionally the ICTY advised that,

‘In determining whether an attack was proportionate it is necessary to examine

whether a reasonably well informed person in the circumstances of the actual

perpetrator, making reasonable use of the information available to him or her,

could have expected excessive civilian casualties to result form the attack’.149

Again, reasonableness is a uniquely human quality not something that can be assigned to

an autonomous weapon. The difficulties humans have with proportionality in practice

were highlighted in the NATO bombing campaign against the Federal Republic of

Yugoslavia.

Allegations were made during the NATO campaign in Kosovo that the rule of

proportionality had been breached by NATO in the form of environmental damage.150 The

report suggests,

‘It is difficult to assess the relative values to be assigned to the military advantage

gained and harm to the natural environment, and the application of the principle of

proportionality is more easily stated than applied in practice’.151

148 International Committee of the Red Cross, Commentary – Precautions in attack International Committee of the Red Cross <http://www.icrc.org/ihl.nsf/COM/470-‐750073?OpenDocument> 149 Prosecutor v Galic, (Trial Judgment) (International Criminal Tribunal for the Former Yugoslavia, Trial Chamber, IT-98-29-A 5th of December 2003) [58]. 150Article 8(b)(iv) Rome Statute of the International Criminal Court, opened for signature 17th July 1998, 2187 UNTS 90 (entered into force 1 July 2002).


40

The report found that whilst a commander has the obligation to use weaponry least likely

to minimize incidental damage, operational reality is recognized by the Rome Statute.152

Long term and severe damage to the natural environment was not found to have

occurred.153 Again this is not a simple concept that can easily be emulated, indeed it is

‘abstract, not easily quantified, and highly relative to specific contexts and subjective

estimates of value’.154

Why will an Autonomous Weapon struggle with Proportionality?

Human’s clearly struggle with the application of proportionality and the notion that a

autonomous weapon would be more able is somewhat ludicrous. Even the most war

seasoned commander has a sense of compassion for the taking of a human life however

justified and this acts as a natural constraint when determining the proportionate number

of civilian casualties. A robot cannot emulate compassion.

Targeting decision also require value judgements heavily based on context,155 Constant

change is a predominant factor of battle and requires the good faith and common sense

elements and experts question the ability of autonomous robots to contend with these

factors.156 By way of example an enemy leader may be detected by an autonomous

weapon in a city, the robot has to contend with constant movement, busses, cars, trains,

151 ICTY, above n, 28[19]. 152 Rome Statute Article 8(b)(iv) Rome Statute of the International Criminal Court, opened for signature 17th July 1998, 2187 UNTS 90 (entered into force 1 July 2002). 153 ICTY, above n, 28 [21]. 154 Peter Asaro, Modeling the Moral User (16th March 2009) IEEE Explore Digital Library 21. <http://peterasaro.org/writing/Asaro%20Modeling%20Moral%20User.pdf> 155 Human Rights Watch, above n 3, 32. 156 Noel Sharkey, “Automated Killers and the Computing Profession’ (2007) 40(11) Computer, 122-‐124.


41

planes and pedestrians as well as the fact that the value of the detected leader could

change as the conflict changes a value judgement at this point is required, by a human.157

Experts in the field of autonomy question the ability of an autonomous weapon to comply

with proportionality test.158

A practical example illustrates that programing an autonomous weapon to determine

proportionality becomes more complicated when the limited arsenal the robot can carry is

taken into consideration. One of the big decisions that must to be made when analysing

proportionality is determining the appropriate level of force.159 Autonomous weapons will

struggle to apply the correct level of force needed on the battlefield. Judgements of this

nature need humans because the decision relies on reason, reflection and the interpretation

of information available to formulate an opinion. 160

Returning to our simple autonomous robot at the beginning of this dissertation. It may

decide a door must be opened due to ‘military necessity’ if it is only armed with a heavy

artillery fire there can clearly be no proportionality. The robot’s taking down the door will

engage in a disproportionate act. Alternatively if the robot does not fire the ‘military

necessity’ will not be achieved. Robots will generally have to be limited by what they are

able to carry and such limitation will make adhering to proportionality all the more

difficult.

157 Human Rights Watch, above n 3, 34. 158 Human Rights Watch, above n 3, 33. 159 Human Rights Watch ‘Shaking the Foundations, The Human Rights Implications of Killer Robots’ (12th May 2014) Human Rights Watch 12. <http://www.hrw.org/reports/2014/05/12/shaking-‐foundations>. 160Mary-‐Ann Russon, ‘Should Killer Robots be Banned in Policing?’ International Business Times May 12th 2014. Published in the ‘International Business Times’ (United Kingdom) May 12th 2014. <http://www.ibtimes.co.uk/should-‐killer-‐robots-‐be-‐banned-‐policing-‐1448189>.


42

Thus it is clearly evident that an autonomous weapon is not able to conform to the IHL

principle of proportionality due to its incapacity to emulate human decision making by

way of value judgements and reasonableness in context. Furthermore the weaponry it is

able to carry will limit the choice of the autonomous weapon pertaining to the level of

force applied.

Proposed Solutions

Krishnan161 suggests prior to releasing autonomous weapons onto the battle field they

undergo a reliability rate in the development stage such as those used by the United States

in relation to cluster weapons. According to the moratorium on the export of cluster

munitions a 99 precent reliability rate is required that the bomb lets will explode in the

blast is required, anything less is unacceptable.162 Understandably this is appealing for

proponents of autonomous weapons but the controls determining the reliability of cluster

munitions are relatively narrow compared to the immense number of scenarios that would

need to be tested prior to ascertaining the 99 precent reliability. It is questionable that

testing would even be effective, this is a weapon with an independent though capacity not

an automated bomb that is programed that ‘if’ I am detonated ‘then’ 100 precent of my

bomb lets must explode.

Autonomous weapons will violate IHL, development should be banned however the

amount of money being poured into this technology is staggering. In the event that it is

not stopped States who use weaponry that breaches their international obligations, one of

which is compliance with IHL, can be held to account. Individuals will not escape

accountability for war crimes where the autonomous weapon is the principle perpetrator

161 Krishnan, above n 61, 98. 162 Ibid.


43

in the commission of a crime. Corporations may escape accountability, which is discussed

in the next chapter.


44

Chapter 4 Accountability

Accountability is an important aspect of law as it seeks to deter future violations and

provides victims with a sense of retribution.163 When autonomous robots violation the

law, and they will, who will we hold to account for the violation? There are three potential

means of accountability. The first is State Responsibility, provided the State has an

obligation and the breach can be attributed to the State the State using the autonomous

weapon may be held to account. Deliberate interference with an autonomous weapon by

individuals leading to the robot being the principle perpetrator of crimes may invoke

individual accountability through Joint Criminal Enterprise. These areas of law are not

clear cut, issues certainly exist but the law can accommodate autonomous weapons. An

accountability gap exists when there is malfunction or the machine acts unpredictably

causing a violation due to the fault of the corporation who manufactured the machine,

accountability in this instance is a matter of luck.

Autonomous weapons will violate IHL they will not necessarily act the way they have

been programed to or the way we want them to. Unpredicted robotic behaviors have

arisen simply via the complexity of programs interacting within the robot.164 Additionally

‘Complex systems are prone to component failures and malfunctions, and to intermodal

inconsistencies and misunderstandings’.165 Such complexity means it is not possible for any

one person to accurately predict with 100 percent certainty the way the robot will respond

to a given command.166

163 Human Rights Watch, above n 3, 42. 164 Marchant et al, above n 39, 275-‐284. 165 Roger Clarke Asimov's Laws of Robotics Implications for Information Technology-‐Part II, (1994). 27(1) Computer 57, 65. 166 Ibid.


45

Autonomous Weapons and State Responsibility

States are the principle players in international law167 and as such should be held

accountable for any violations. The following fictitious scenario illustrates State

Responsibility. The United States purchases an autonomous weapons from the United

Kingdom and some months later becomes involved in a war in which the autonomous

weapons is deployed by two combatants despite the United States rules of engagement

stating that deployment of the weapon requires Commander approval. The deployment

violated the principle of distinction, which the United States had an obligation to uphold

furthermore the violation can be attributed to the United States.

Article 1 of the Draft Articles on Responsibility of States for Internationally Wrongful

Acts GA Res 56/83, UN GAOR, 53rd Sess , Agenda Item 162, Supp No 10, UN Doc

A/RES/56/83 (adopted November 2001) states that ‘any violation by a State of any

obligation, of whatever origin, gives rise to State responsibility’. The elements consist of

firstly that the violation or conduct must be attributable to the State and secondly an

international obligation must be breached.168

The fictitious scenario illustrates a breach of an international obligation to distinguish

between civilians and combatants. Attribution elements may give rise to holding the

United Kingdom who sold the weapon and the United States to utilised it to account.

167 Stephen Hall, Principles of International Law (LexisNexis, 2nd ed, 2011) 238 [5.1]. 168 Article 2 of the Draft Articles on Responsibility of States for Internationally Wrongful Acts GA Res 56/83, UN GAOR, 53rd Sess , Agenda Item 162, Supp No 10, UN Doc A/RES/56/83 (adopted November 2001).


46

Article 4 of the Draft Articles169 ensures that ‘the conduct of any State organs shall be

considered an act of that State…..’ Public military personal are considered organs of the

state 170 and violations of domestic military policy should not inhibit attribution. Article

7171 states that a person exercising government authority continues to do so even if they

exceed their authority. The Caire Claim (France v Mexico) (Arbitration Tribunal) (1929)

5 RIAA 516 is the case on point where the International Court of Justice stated that the

officers ‘acted in their capacity as officers and used the means placed at their disposition

by virtue of that capacity’. 172 The officers in this case were Mexican and the wrong was

attributed to Mexico. The wrong in our scenario should be attributed to the United States

with relative ease. Reparations are payable by the violating State173 and can take the form

of restitution174 compensation175 and satisfaction176 provided the victim State has not met

any of the exceptions which permit the failure to meet the obligation by the purported

offending State.

Attribution to the United Kingdom for supply of the weapon that caused the harm would

not be feasible. Article 6177 asserts that the conduct of organs placed at the full disposal of

a State by another State negates attribution provided the state using the organ is exercising

all the elements of governmental authority.178 Thus the nation who funds the research and

169 Draft Articles on Responsibility of States for Internationally Wrongful Acts GA Res 56/83, UN GAOR, 53rd Sess , Agenda Item 162, Supp No 10, UN Doc A/RES/56/83 (adopted November 2001). 170 Hall, Above n 167, 246 [5.19]. 171 Draft Articles on Responsibility of States for Internationally Wrongful Acts GA Res 56/83, UN GAOR, 53rd Sess , Agenda Item 162, Supp No 10, UN Doc A/RES/56/83 (adopted November 2001). 172 Caire Claim (France v Mexico) (Arbitration Tribunal) (1929) 5 RIAA 516. 173 Article 31 Draft Articles on Responsibility of States for Internationally Wrongful Acts GA Res 56/83, UN GAOR, 53rd Sess , Agenda Item 162, Supp No 10, UN Doc A/RES/56/83 (adopted November 2001). 174 Ibid Article 35. 175 Ibid Article 36. 176 Ibid Article 37. 177 Ibid. 178 Report of the Commission to the General Assembly on the Work of its fifty-‐third session [2001] II(2) Yearbook of the International Law Commission 1, 44[1].


47

development and capitalises from autonomous weapons would not be accountable under

State Responsibility.

The use of autonomous weapons which violate IHL should not prevent accountability via

State Responsibility.

Autonomous Weapons and Joint Criminal Enterprise (JCE).

International criminal law and human rights law are unique exceptions in international

law because they are concerned with individuals rather than States.179 Modes of liability

are used to hold individual perpetrators to account for various roles played in the

commission of substantive crimes. They are recognised as customary law180 and despite

some differences are also contained in the Rome Statue.181

The autonomous weapon was the principle perpetrator in our story but the two combatants

have deployed the weapon against regulations. Furthermore the over ride of the built in

safety was preformed by a robotics engineer in a private company in the United Kingdom

and the change in programing to enable the weapon to move over different terrain was

carried out by a disgruntled employee from a company in the United States. All

participants were aware that the weapon would be utilised in a crime to further their

beliefs. All of these individuals can be held to account, the autonomous weapon as the

principle perpetrator should not be significant.

179Hall, above n 167, 448 [10.1]. 180 International Criminal Law Services, International Criminal Law & Practice Training Materials Modes of Liability: Commission & Participation ICL Practice and Training Materials United Nations Interregional Crime and Justice Research Institute 3[9.2.1] <http://wcjp.unicri.it/deliverables/docs/Module_9_Modes_of_liability.pdf> 181Article 25(3) Rome Statute of the International Criminal Court, opened for signature 17th July 1998, 2187 UNTS 90 (entered into force 1 July 2002).


48

The complexity of autonomous weapons suggests that Joint Criminal Enterprise (JCE)

should ensure individual accountability. Many years ago computer programs could be

written and interpreted by individuals this is no longer the case and many teams of

programmers work on writing codes for different functions of the one robot.182 There is a

strong possibility that an autonomous weapon used in the commission of a crime will

require several perpetrators.

IHL ensures there are mechanisms in place to hold orchestrators to account. Criminal

masterminds often deliberately distance themselves geographically from the actual

crime,183 and autonomous weaponry acting as the principle perpetrator further facilitates

the ability of one to distance themselves. JCE, derived from the term ‘committed’ in the

ICTY statute,184 establishes an accountability mechanism. The ICTY utilised JCE which

has its foundations in customary law185 and The Rome Statue has incorporated the

concept in co-perpetration,186 indirect co perpetration187 and other types of common

purpose liability.188

182 Marchant, Above n 39, 275-‐284. 183 Katrina Gustafson ‘The Requirement of an ‘Express Agreement’ for Joint Criminal Enterprise Liability’ (2007) 5(1) Journal International Criminal Justice 134,135. 184 Prosecutor v Tadic (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-‐94-‐1-‐A, 15th July 1999) [190] referring to Art 7(1) Statute of the International Criminal Tribunal for the Former Yugoslavia SC Res 808, UN SCOR, 3175th mtg, (22nd February 1993). The term committed was interpreted to mean JCE. 185 Prosecutor v Tadic (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-‐94-‐1-‐A, 15th July 1999) [190] JCE first emerged here and was subsequently used in ICTY decisions. 186 Article 25(3)(a) Rome Statute of the International Criminal Court, opened for signature 17th July 1998, 2187 UNTS 90 (entered into force 1 July 2002). 187 Ibid, Article 25(3)(a). 188 Ibid, Article 25(3)(d).


49

Three types of joint criminal enterprise189 are recognised and all three require the same

actus reas elements.190 Firstly the ‘plurality of persons’,191 secondly the ‘existence of a

common plan, design or purpose which amounts to or involves the commission of a

crime’,192 and thirdly ‘[p]articpation of the accused in the common design’.193 The

accused does not have to be present during the commission of the crime ensuring

accountability exists when the crime is carried out by an autonomous weapon.

The mens rea of the crime varies between the three categories;194 at the most basic level,

JCEI criminal intent to commit crimes that form part of a common plan must exist195 and

the accused must intend to partake in the plan.196 The second JCE requires that the

accused ‘have personal knowledge of an organized criminal system and intent to further

the criminal purpose of the system’.197 The final JCE;

‘[T]he accused can only be held responsible for a crime outside the common

purpose if, under the circumstances of the case: (i) it was foreseeable that such a

crime might be perpetrated by one or other members of the group and (ii) the

189 Prosecutor v Tadic (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-‐94-‐1-‐A, 15th July 1999) [195-‐226]. 190 Ibid 227. 191 Ibid. 192 Ibid. 193 Ibid. 194 Prosecutor v Vasiljevic (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-‐98-‐32, 23rd February 2004) IT-‐98-‐32 [101]. 195 Ibid [97]. 196 Prosecutor v Kvocka and others (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-‐98-‐30/1-‐A, 28th February 2005) [82]. 197Prosecutor v Tadic (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-‐94-‐1-‐A, 15th July 1999) [202-‐203 and 228].


50

accused willingly took that risk’.198 The accused in particular must also be

evidenced to have seen that the crime was foreseeable.199

The fact that an autonomous weapon was the principle perpetrator and not a member of

the JCE should be of no consequence. The issue turns on whether or not the crime itself

was found to be within the common purpose of the JCE and that the crime could be

attributed to at least one member of the JCE.200 Furthermore the principle perpetrator of

the crime, in our case the autonomous weapon, does not have to share the mens rea of the

of the JCE member,201 this is advantageous to a prosecutor as establishing mens rea of an

autonomous weapon would be difficult if not impossible because as discussed above killer

robots do emulate the human emotion of intent. Despite not being tested before a legal

authority there is no obvious reason why an autonomous weapon could not act as the

principle perpetrator and the JCE members be held liable for those crimes.

Other modes of liability are available when JCE elements cannot be met.202

Private Corporations and Accountability

A prominent accountability gap exists with the private companies who develop and

promote autonomous weapons. Experts predict autonomous weapons will have a level of

unpredictability,203 Robert Sparrow a professor of philosophy states

198 Ibid [228]. 199 Ibid [220]. 200 Prosecutor v Martic (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-‐95-‐11-‐A, 8th October 2008) [168]. 201 Prosecutor v Bradanin (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-‐99-‐36-‐A, 3rd April 2007)[410]. 202 See for example aiding and abetting, planning, instigating or inciting and others. 203 Clarke, above n 165.


51

‘[T]he possibility that an autonomous system will make choices other then those

predicted and encouraged by its programmers is inherent in the claim that it is

autonomous’.204

The use of private firms in the design and manufacture of weapons has significantly

expanded following the second World War.205 There appears to be a distorted notion that

holding a programmer or developer to account is some how analogous to the idea that

parents can be held responsible for the actions of their adult children simply because a

weapon may have the capacity to think for itself. Weapons don’t have rights as children

do, they can be locked in a shed for all eternity. Obviously there is a reasonably

foreseeable risk that an unpredictable product, designed to kill will not act in accordance

with programing or law. It would be nothing short of negligent to release such

unpredictability.

Philosopher Peter Asaro argues,

Corporations can be held legally responsible for their practices and products,

through liability laws and lawsuits. If their products harm people through poor

design, substandard manufacturing, or unintended interactions or side effects that

corporation can be compelled to pay damages to those who have been harmed, as

well as punitive damages.206

Civil actions attracting large amounts in damages would certainly induce a high level care

from private companies.207 There would be additional benefits for the victims who would

204 Peter Sparrow, ‘Killer Robots’ (2007) 24(1) Journal of Applied Philosophy 62, 69-‐70. 205 Marcus Sumner, ‘Studies of the Defense Contracting Process’ (1964) Law and Contemporary Problems 19, 20. 206 Krishnan, above n 59, 104. 207 Eric Mongelard ‘Corporate Civil Liability for Violations of International Humanitarian Law’ (2006)88(863) International Review of the Red Cross 665, 666.


52

receive financial compensation. International law can be applied to non-state actors as

seen with criminal responsibility, it can also extend to corporations and international law

can also determine what constitutes a civil wrong by a company.208 In other words an

obligation can be imposed on corporations by international law. Accountability becomes

problematic because enforcement of these obligations is left to individual States.

The United States provides an excellent example of States ensuring their private

corporations are accountable. It has enacted the Alien Tort Claims Act 28 USC § 1350.

This part of the code enables,

‘The district courts [to] have original jurisdiction of any civil action by alien for a

tort only, committed in violation of the law of nations or a treaty of the United

States’.

This means a non-foreign national can sue an American corporation. Doe v Unocal Corp

110F Supp2d 1294 (CD Cal 2000) US Federal District Court, judgement of 31 August

2000, is an excellent example of how a private corporation can be held to account. This

case was held within the United States judicial system. A group of Myanmar citizens sued

Unocal Corp, an American based company, for aiding and abetting the Myanmar military

in human rights violations including rape, torture and forced displacement, all of which

would be violations of IHL had they occurred during war rather than peace. A pipeline

was being constructed in partnership between the Myanmar government and Unocal in

which the Myanmar government was to supply security and labour.

The court found in the first instance that responsibility could be attached to Unocal under

joint action theory, which is not dissimilar to JCE above. One of the requirements was the

208 Ibid 670.


53

working towards a common design, the court identified this common design to be a

profitable project. The court also applied the ‘proximate cause’ test that meant the

plaintiff had to establish that the company exercised control over the government, the

court held that whilst Unocal was aware of the slave labour and utilised it to achieve the

common design it was not enough to hold the company liable.

The decision was overturned on appeal, the Court of Appeal found aiding and abetting,

borrowed from the criminal jurisdiction, should have been applied. The Appeals Court

said that in international cases such as this international law either United States treaty or

customary law was preferable to domestic.209 The court provided sound reasoning for this

decision,

‘First, the needs of the international system are better served by an international

standard.210 Second, the relevant policies of the forum cannot be ascertained by

disregarding decision that has favoured international law.211 Third, reliance on

international standards of secondary liability promote consistency, uniformity,

predictability, and protection of justified expectations. 212 And fourth, the

overarching purpose of the ATCA is to provide a civil remedy for violations of

international law, a goal furthered by application of international standards.213’.

Unocal was held to be liable under the Alien Tort Claim Act.

209 Doe v Unocal, 395 F.3d 932, 42 (9th Circuit) (2002). 210 Ibid. 211 Ibid. 212 Ibid. 213 Ibid [42-‐43].


54

Unfortunately this policy by the United States is not mimicked globally. Reasons for this

include the fact that some of the corporate giants wield more economic power than some

individual states.214 Better economic yields area often achieved by corporations than the

nation states in which they operate.215 The new millennium mantra to ‘deregulate’ has

seen States vying for investment by multinational companies 216 and this places

governments at a disadvantage, leading to a decrease in control. Additionally many large

multinational companies are international making it difficult for domestic national law to

effectively govern them. In some instances the labour and component market is divided

between nations,217 as in our case where the autonomous weapons component for

movement was in one country and another component in another. Again States have the

power to create agreements within international law to combat such difficulties and ensure

these companies are held to account. States have the requisite power to enable victims to

hold corporations to account through civil litigation218 they just need the will to do so.

In the event States are prepared to risk autonomous weapons on the battlefield it may be

in their best interest to ensure the firms they purchase such weapons from can be held to

account for violations of IHL.

Where to from Here?

To date there are no current international instruments placing a prohibition on

autonomous weapons but the time has come to discuss the implementation of laws

banning them. The urgently required prohibitions may be in the form of amendment by

214 Mongelard, above n 207, 669. 215 Beth Stephens ‘The amorality of profit’ (2002) 20 Berkeley Journal of International Law 45, 57. In 2002 only seven nations had revenues greater than General Motors. 216 Stephens, above n, 215, 58. 217 Ibid 54. 218 Ibid 60.


55

way of an additional protocol to the Convention on Prohibitions or Restrictions on the

Use of Certain Conventional Weapons Which May be Deemed to be Excessively

Injurious or to Have Indiscriminate Effects, Opened for signature 10th of October

1980, 1342 UNTS 132 (entered into force 2nd December 1983). The preamble states

‘that prohibits the employment in armed conflicts of weapons, projectiles and material

and methods of warfare of a nature to cause superfluous injury or unnecessary

suffering’219 This dissertation has made it clear that killer robots will not be able to

emulate human emotion which is necessary not only to restrain acts of war but also to

accurately apply the touchstone principles of IHL, distinction and proportionality which

should satisfy the preamble requirements. An expert panel has met under this

Convention220 to discuss autonomous weapons.

A meeting of experts on Lethal Automatous Weapons Systems was held at the United

Nations in Geneva from the 13th to 16th of May 2014221 as part of this convention 87

countries participated.222 Problems can arise with such instruments as they only bound

states that choose to be bound leaving insurgent groups not bound by these agreements223

and may be able to purchase such weapons from private corporations who will most likely

not be held to account for any violations their product causes. States should ensure their

domestic legislation enables aliens to hold private companies to account for breaches of

IHL.

219 Ibid 220 Ibid. 221 United Nations Office at Geneva, Disarmament, Lethal Autonomous Weapons (2013) United Nations Office at Geneva <http://www.unog.ch/80256EE600585943/%28httpPages%29/6CE049BE22EC75A2C1257C8D00513E26?OpenDocument> 222Human Rights Watch, UN: ‘Killer Robots’ on the Run. (May 16, 2014) Human Rights watch <http://www.hrw.org/news/2014/05/16/un-‐killer-‐robots-‐run> 223 Marchant et al, above n 32, 305.


56

These measures are probably to extreme for the pallets of most states. Perhaps prior to the

introduction of international instruments soft law should be utilised to begin discussions

and awareness of the issues autonomous weapons create. Soft law approaches aim to

create substantive principles and norms which are not legally binding.224 Additionally

they allow flexibility as new issues become better understood.225 The following have been

used for emerging technologies and may be suited to autonomous weaponry, although an

outright ban is preferred. These soft law instruments include, codes of conduct226 which

are often industry based and can link into liability.227 Trans governmental dialogue which

are informal flexible arrangements encouraging states to meet and discuss developments

and policies. International sharing and confidence building measures the aim of which is

to increase trust by sharing information and finally framework conventions that at the

onset may be very basic and may start simply by agreeing an agreement is required.

Conclusion

The removal of humans from decision-making during times of war takes away important

restrictions such as the desire not to kill. Indeed the more robotic humans become the

greater the chance of IHL violation. Human decision-making may appear erroneous but

overall the decisions made by humans include very important emotions providing an

imperative restraining factor. Decisions made without emotion by way of autonomous

weapons will require new law, the inherent restraints which form part of IHL will need to

be explicitly contained within it.

224 Ibid 306. 225 Ibid 305. 226 Ibid 307. 227 Ibid 310.


57

Autonomous weapons cannot replace humans on the OODA loop decision-making

process. The thought capacity of these weapons could not emulate humans as such their

legality becomes questionable. Killer robots will not be able to adequately identify a

combatant or civilian and as such one of the cardinal principle of IHL, the principle of

distinction would be difficult to comply with. The principle of proportionality requires a

subjective judgement and cannot be quantified again it will be impossible for a killer

robot to apply.

Finally autonomous weapons themselves cannot be held to account for violation of IHL,

unpredictability is an inherent aspect of autonomous robots and the result, should private

corporations chose to release them, is an accountability gap unless the weapon was

manufactured in the United States. The use of an autonomous weapon to act as the

principle perpetrator in the commission of a crime will not preclude the modes of liability

from being applied or State Responsibility, both individuals and States can be held to

account for crimes committed by the autonomous weapon.


58

Bibliography A Articles/Books/Reports Adams, Thomas K, ‘Future Warfare and the Decline of Human Decision making’ (2001) 31(4) Parameters: US Army War College 57 Asaro, Peter ‘How Just Could a Robot War Be?’ (2008) Proceeding of the 2008 Conference on Current Issues in Computing and Philosophy, 50 Choi, Stephen and AC Pritchard, ‘Behavioural Economics and the SEC’ (2003) 56(1) Stanford Law Review 1 Clarke, Roger, Asimov's Laws of Robotics Implications for Information Technology-Part II, (1994) 27(1) Computer 57, 65 Creamer, Mark, Ric Marchall and Anne Goyne Military Stress and Performance The Australian Defence Force Experience (Melbourne University Press, 2003) Farnsworth, Allen and Steve Sheppard (ed) An Introduction to the Legal System of The United States (Oxford University Press, 4th ed, 2010) Forrest, CJS, ‘The Doctrine of Military Necessity and the Protection of Cultural Property During Armed Conflicts’ (2007) 37(2) California Western International Law Journal 177 Gardam, Judith, ‘Proportionality as a Restraint on the Use of Force’ (1999) 20 Australian Year Book of International Law, 161 Gigerenzar, Gerd and Wolfgang Gaissmaier ‘Heuristic Decision Making’ (2011) 62 Annual Review of Psychology 451 Grossman, Dave, On Killing The psychological Cost of Learning to Kill in War and Society (Hachette Book Group, revised ed, 2009) Gustafson, Katrina ‘The Requirement of an ‘Express Agreement’ for Joint Criminal Enterprise Liability’ (2007) 5(1) Journal International Criminal Justice 134 Hall, Stephen, Principles of International Law (LexisNexis, 2nd ed, 2011) Henckaerts, Jean-Marie and Lousie Doswald-Beck, Customary International Humanitarian Law Volume I: Rules (Cambridge University Press, 2005) Human Rights Watch, ‘Losing Humanity The Case against Killer Robots’ (2012) International Human Rights Clinic 1 International Committee of the Red Cross, Commentary on the Additional Protocols of 8 June 1977 to the Geneva Conventions of 12 August 1949 (Geneva: Martinus Nijhoff Publishers, 1987) Kanwar, Vik, ‘Post-Human Humanitarian Law: The Law of War in the Age of Robotic Weapons’ (2011) 2 Harvard National Security Journal Vol 616


59

Krishnan, Armin ‘Killer Robots Legality and Ethicality of Autonomous Weapons’ (Ashgate Publishing Limited, 2009) Marchant, Gary E, et al, ‘International Governance of Autonomous Military Robots’ (2011) 12 The Columbia Science and Technology Law Review 272 Marra, William C and Sonia K McNeil ‘Understanding “The Loop” Regulating the Next Generation of War Machines’ (2013) 36(3) Harvard Journal of Law and Public Policy 1139, 1146. McManners, Hugh The Scars of War (Harper Collins Publishers, 1994). Mongelard, Eric ‘Corporate Civil Liability for Violations of International Humanitarian Law’ (2006)88(863) International Review of the Red Cross 665 Munoz-Rojas, Daniel and Jean Jacques Fresard, ‘Roots of Behavior in War’ (2004) 86(853) IRRC 189 Nygh, Peter E and Peter Butt (eds) Butterworths Australian Legal Dictionary (Butterworths 1997) Report of the Commission to the General Assembly on the Work of its fifty-third session [2001] II(2) Yearbook of the International Law Commission 1 Schmitt, Michael ‘Essays on Law and War at the Fault Lines’ (Asser Press, 2012) Schmitt, Michael and Jeffrey S Thurnher ‘Out of the Loop: Autonomous Weapon Systems and the Law of Armed Conflict’ (2012-2013) 4 Harvard National Security Journal 213 Sharkey, Noel, ‘Killing Made Easy: from joysticks to politics’ in Patrick Lin and George Bekey and Keith Abney (eds) The Ethical and Social Implications of Robotics (Cambridge: MIT Press, 2012) Sharkey, Noel ‘Cassandra or False Prophet of Doom: AI Robots and War’ (2008) 23(4) IEEE Intelligent Systems 14 Sharkey, Noel, “Automated Killers and the Computing Profession’ (2007) 40(11) Computer, 122-124. Solis, Gary, ‘The Gotovina Acquittal: A sound Appellate Course Correction’ (2013) 215 Military law Review 78, Sparrow, Peter, ‘Killer Robots’ (2007) 24(1) Journal of Applied Philosophy 62 Stephens, Beth ‘The amorality of profit’ (2002) 20 Berkeley Journal of International Law 45


60

Sumner, Marcus ‘Studies of the Defense Contracting Process’ (1964) Law and Contemporary Problems 19 Tversky, A and D Kahneman ‘The framing of decisions and the psychology of choice’ (1981) 211(4481) Science 453 B Cases Caire Claim (France v Mexico) (Arbitration Tribunal) (1929) 5 RIAA 516. Doe v Unocal, 395 F.3d 932, 42 (9th Circuit) (2002) Nuclear Weapons (Advisory Opinion) [ 1996] ICJ Rep 225 No 93. Prosecutor v Bradanin (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-99-36-A, 3rd April 2007) Prosecutor v Galic, (Trial Judgment) (International Criminal Tribunal for the Former Yugoslavia, Trial Chamber, IT-98-29-A 5th of December 2003) Prosecution v Gotovina (Trail Judgment) (International Criminal Tribunal for the Former Yugoslavia, Trial Chamber IT-06-90-T 15th of April 2011)[ Prosecutor v Kupreskic (Trial Judgment) (International Criminal Tribunal for the Former Yugoslavia, Trial Chamber IT-95-16-T, 14th of January 2000) Prosecutor v Kvocka and others (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-98-30/1-A, 28th February 2005) Prosecutor v Martic (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-95-11-A, 8th October 2008) Prosecutor v Tadic (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-94-1-A, 15th July 1999) Prosecutor v Vasiljevic (Appeals Judgment) (International Criminal Tribunal for the Former Yugoslavia, Appeal Chamber, Case No IT-98-32, 23rd February 2004) Rainbow Warrior (New Zealand v France) (Arbitration Tribunal) (1986) 19 RIAA 199, 251[75]. D Treaties Articles on Responsibility of States for Internationally Wrongful Acts GA Res 56/83, UN GAOR, 53rd Sess , Agenda Item 162, Supp No 10, UN Doc A/RES/56/83 (adopted November 2001) Convention on Cluster Munitions, opened for signature 3rd December 2008, 2688 UNTS (entered into force 1st August 2010) Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May be Deemed to be Excessively Injurious or to Have Indiscriminate Effects,


61

Opened for signature 10th of October 1980, 1342 UNTS 132 (entered into force 2nd December 1983). Convention for the Protection of Cultural Property in the Event of Armed Conflict with Regulation for the Execution of the Convention , opened for signature 14th of May 1954 UNESCO (entered into force 7th of August 1956) Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts (Protocol I) opened for signature 8th June 1977, 1125 UNTS 3 (entered into force 7th December 1979) Rome Statute of the International Criminal Court, opened for signature 17th July 1998, 2187 UNTS 90 (entered into force 1 July 2002). Statute of the International Criminal Tribunal for the Former Yugoslavia SC Res 808, UN SCOR, 3175th mtg, (22nd February 1993) Vienna Convention on the Law of Treaties, opened for signature 23rd May 1969, 1155 UNTS 331 (entered into force 27th January 1980). E Other Asaro, Peter, Modeling the Moral User (16th March 2009) IEEE Explore Digital Library 21. <http://peterasaro.org/writing/Asaro%20Modeling%20Moral%20User.pdf> Alton Parrish, Gee Whiz! Sea Whiz 4500 Rounds Per Minute For Close In Naval Fighting (10th September 2012) Before it’s News <http://beforeitsnews.com/science-and-technology/2012/09/gee-whiz-sea-whiz-4500-rounds-per-minute-for-close-in-naval-fighting-2467644.html>?>. Chow, Denise, Drone Wars: Pilots Reveal Debilitating Stress Beyond Virtual Battlefield (5th November 2013) Live Science <http://www.livescience.com/40959-military-drone-war-psychology.html> Heyns, Christof, Report of the Special Rapporteur on Extrajudicial Summary or Arbitrary Executions Agenda Item 3, A/HRC/23?47/Add.4, (23rd of April 2013), Agenda Item 3 Horowitz, Michael C, The looming Robotics Gap Why America’s Global Dominance in Military Technology is Starting to Crumble (5th May 2014) Foreign Policy The Magazine <http://www.foreignpolicy.com/articles/2014/05/05/the_looming_robotics_gap_us_military_technology_dominance> Human Rights Watch, UN: ‘Killer Robots’ on the Run. (May 16, 2014) Human Rights watch <http://www.hrw.org/news/2014/05/16/un-killer-robots-run> Human Rights Watch ‘Shaking the Foundations, The Human Rights Implications of Killer Robots’ (12th May 2014) Human Rights Watch 12. <http://www.hrw.org/reports/2014/05/12/shaking-foundations> Karl Sims, Evolved Virtual Creatures You Tube <http://www.youtube.com/watch?v=F0OHycypSG8>


62

Lawand, Kathleen and Robin Coupland and Peter Herby, A Guide to the Legal Review of New Weapons, Means and Methods of Warfare. Measures to Implement Articles 36 of API of 1977 November 2006 International Committee of the Red Cross 1,10. < http://www.icrc.org/eng/assets/files/other/icrc_002_0902.pdf > Melzer, Nils, Interpretive Guidance on the Notion of Direct Participation in Hostilities under International Humanitarian Law Page International Committee of the Red Cross 22-24 <http://www.icrc.org/eng/assets/files/other/icrc-002-0990.pdf> Ministry of Defense ‘The UK Approach to Unmanned Aircraft Systems’ (Joint Doctrine Note 2/11 The Development, Concepts and Doctrine Centre, 30th of March 2011) Moynihan, Tim, Watch the Astounding Dexterity of Honda’s Dancing Humanoid Robot (19th of April 2014) Wired <http://www.wired.com/2014/04/honda-asimo/> International Committee of the Red Cross, Commentary – Precautions in attack International Committee of the Red Cross <http://www.icrc.org/ihl.nsf/COM/470-750073?OpenDocument> International Committee of the Red Cross What is International Humanitarian Law? (July 2004) Advisory Service on International Humanitarian Law <http://www.icrc.org/eng/assets/files/other/what_is_ihl.pdf> International Criminal Law Services, International Criminal Law & Practice Training Materials Modes of Liability: Commission & Participation ICL Practice and Training Materials United Nations Interregional Crime and Justice Research Institute ITCY, Final report to the Prosecutor by the Committee Established to Review the NATO Bombing Campaign Against the Federal republic of Yugoslavia. (Initiated, 14th May 1999) International Criminal Tribunal for the for the former Yugoslavia) <http://www.icty.org/sid/10052#pagetop> The Prosecutor v Gotovina and Markac, (Ante Gotovina’s Response to ‘Application and Proposed Amicus Curiae Brief’)(International Criminal Court for former Yugoslavia) IT-06-90-A (13th January 2012) [9(4)]. Russon, Mary-Anne, ‘Should Killer Robots be Banned in Policing?’ International Business Times May 12th 2014. Published in the ‘International Business Times’ (United Kingdom) May 12th 2014. <http://www.ibtimes.co.uk/should-killer-robots-be-banned-policing-1448189> Sharkey, Noel, Computing Experts from 37 Countries Call for Ban on Killer Robots (16th October 2013) International Committee for Robot Arms Control <http://icrac.net/2013/10/computing-experts-from-37-countries-call-for-ban-on-killer-robots/> United Nations Office at Geneva, Disarmament, Lethal Autonomous Weapons (2013) United Nations Office at Geneva


63

<http://www.unog.ch/80256EE600585943/%28httpPages%29/6CE049BE22EC75A2C1257C8D00513E26?OpenDocument> Worsnip, Patrick, ‘UN Official Calls for Study of Ethics, Legality of Unmanned Weapons’ Washington Post, (Washington USA), October 24, 2010

Documents

Meaghan_Brockhoff_Dissertation