Tactics of Adversarial Attack on Deep Reinforcement Learning Agents

Yen-Chen Lin1, Zhang-Wei Hong1, Yuan-Hong Liao1, Meng-Li Shih1, Ming-Yu Liu2, Min Sun1

1National Tsing Hua University, Taiwan2NVIDIA, Santa Clara, California, USA

{yenchenlin@gapp, williamd4112@gapp, s102061137@m102, shihsml@gapp, sunmin@ee}.nthu.edu.twmingyul@nvidia.com

AbstractWe introduce two tactics, namely the strategically-timed attack and the enchanting attack, to attackreinforcement learning agents trained by deep re-inforcement learning algorithms using adversarialexamples. In the strategically-timed attack, the ad-versary aims at minimizing the agent’s reward byonly attacking the agent at a small subset of timesteps in an episode. Limiting the attack activity tothis subset helps prevent detection of the attack bythe agent. We propose a novel method to determinewhen an adversarial example should be crafted andapplied. In the enchanting attack, the adversaryaims at luring the agent to a designated target state.This is achieved by combining a generative modeland a planning algorithm: while the generativemodel predicts the future states, the planning algo-rithm generates a preferred sequence of actions forluring the agent. A sequence of adversarial exam-ples is then crafted to lure the agent to take the pre-ferred sequence of actions. We apply the proposedtactics to the agents trained by the state-of-the-artdeep reinforcement learning algorithm includingDQN and A3C. In 5 Atari games, our strategically-timed attack reduces as much reward as the uniformattack (i.e., attacking at every time step) does by at-tacking the agent 4 times less often. Our enchant-ing attack lures the agent toward designated targetstates with a more than 70% success rate. Examplevideos are available at http://yenchenlin.me/adversarial_attack_RL/.

1 IntroductionDeep neural networks (DNNs), which can extract hierarchi-cal distributed representations from signals, are established asthe de facto tool for pattern recognition, particularly for su-pervised learning. We, as a generation, have witnessed a trendof fast adoption of DNNs in various commercial systems per-forming image recognition [Krizhevsky et al., 2012], speechrecognition [Hannun et al., 2014], and natural language pro-cessing [Sutskever et al., 2014] tasks. Recently, DNNs havealso started to play a central role in reinforcement learning(RL)—a field of machine learning research where the goal is

to train an agent to interact with the environment for maxi-mizing its reward. The community has realized that DNNsare ideal function approximators for classical RL algorithms,because DNNs can extract reliable patterns from signals forconstructing a more informed action determination process.For example, [Mnih et al., 2015] use a DNN to model theaction–value function in the Q-learning algorithm, and [Mnihet al., 2016] use a DNN to directly model the policy. Rein-forcement learning research powered by DNNs is generallyreferred to as deep reinforcement learning (Deep RL).

However, a constant question lingers while we enjoy usingDNNs for function approximation in RL. Specifically, sinceDNNs are known to be vulnerable to the adversarial exampleattack [Szegedy et al., 2014], as a deep RL agent inherits thepattern recognition power from a DNN, does it also inheritits vulnerability to the adversarial examples? We believe theanswer is yes and provide empirical evidence in the paper.

Adversarial attack on deep RL agents is different fromadversarial attack on classification system in several ways.Firstly, an RL agent interacts with the environment througha sequence of actions where each action changes the state ofthe environment. What the agent received is a sequence ofcorrelated observations. For an episode of L steps, an adver-sary can determine whether to craft an adversarial example toattack the agent at each time step (i.e. there are 2L choices).Secondly, an adversary to deep RL agents have different goalssuch as reducing the final rewards of agents or malevolentlylure agents to dangerous states, which is different to an adver-sary to classification system that aims at lowering classifica-tion accuracy. In this paper, we focus on studying adversarialattack specific on deep RL agents. We argue this is important.As considering deep RL agents for controlling machines, weneed to understand the vulnerability of the agents becauseit would limit their use in mission-critical tasks such as au-tonomous driving. Based on [Kurakin et al., 2016], whichshowed that adversarial examples also exist in the real world,an adversary can add maliciously-placed paint to the surfaceof a traffic stop to confuse an autonomous car. How could wefully trust deep RL agents if their vulnerability to adversarialattacks is not fully understood and addressed?

In a contemporary work, [Huang et al., 2017] proposes anadversarial attack tactic where the adversary attacks a deepRL agent at every time step in an episode. We refer to such atactic as the uniform attack and argue it is preliminary. First,

arX

iv:1

703.

0674

8v4

[cs

.LG

] 1

3 N

ov 2

019

the uniform attack ignores the fact that the observations arecorrelated. Moreover, the spirit of adversarial attack is to ap-ply a minimal perturbation to the observation to avoid detec-tion. If the adversary perturbs the observation at every timeinstance, it is more likely to be detected. A more sophisti-cated strategy would be to attack at selective time steps. Forexample, as shown in Fig. 1, attacking the deep RL agenthas no consequence when the ball is far away from the pad-dle. However, when the ball is close to the paddle, attackingthe deep RL agent could cause it to drop the ball. There-fore, the adversarial attacks at different time instances are notequally effective. Based on this observation, we propose thestrategically-timed attack, which takes into account the num-ber of times an adversarial example is crafted and used. Itintends to reduce the reward with as fewer adversarial exam-ples as possible. An adversarial example is only used whenthe attack is expected to be effective. Our experiment resultsshow that an adversary exercising the strategically-timed at-tack tactic can reduce the reward of the state-of-the-art deepRL agents by attacking four times less often as comparing toan adversary exercising the uniform attack tactic.

In addition, we propose the enchanting attack for mali-ciously luring a deep RL agent to a certain state. While thestrategically-timed attack aims at reducing the reward of adeep RL agent, the enchanting attack aims at misguiding theagent to a specified state. The enchanting attack can be usedto mislead a self-driving car controlled by a deep RL agentto hit a certain obstacle. We implement the enchanting attackusing a planning algorithm and a deep generative model. Tothe best of our knowledge, this is the first planning-based ad-versarial attack on a deep RL agent. Our experiment resultsshow that the enchanting attack has a more than 70% successrate in attacking state-of-the-art deep RL agents.

We apply our adversarial attack to the agents trained bystate-of-the-art deep RL algorithms including A3C [Mnih etal., 2016] and DQN [Mnih et al., 2015] on 5 Atari games. Weprovide examples to evaluate the effectiveness of our attacks.We also compare the robustness of the agents trained by theA3C and DQN algorithms to these adversarial attacks. Thecontributions of the paper are summarized below:

• We study adversarial example attacks on deep RL agentstrained by state-of-the-art deep RL algorithms includingA3C and DQN.

• We propose the strategically-timed attack aiming at at-tacking a deep RL agent at critical moments.

• We propose the enchanting attack (the first planning-based adversarial attack) aiming at maliciously luring anagent to a certain state.

• We conduct extensive experiments to evaluate the vul-nerability of deep RL agents to the two attacks.

2 Related WorkFollowing [Szegedy et al., 2014], several adversarial exam-ple generation methods were proposed for attacking DNNs.Most of these methods generated an adversarial example viaseeking a minimal perturbation of an image that can confusethe classifier (e.g., [Goodfellow et al., 2015; Kurakin et al.,

2016]). [Moosavi-Dezfooli et al., 2016] first estimated lineardecision boundaries between classes of a DNN in the imagespace and iteratively shifted an image toward the closest ofthese boundaries for crafting an adversarial example.

While the existence of adversarial examples to DNNs hasbeen demonstrated several times on various supervised learn-ing tasks, the existence of adversarial examples to deep RLagents has remained largely unexplored. In a contempo-rary paper, [Huang et al., 2017] proposed the uniform attack,which attacks a deep RL agent with adversarial examples atevery time step in an episode for reducing the reward of theagent. Our work is different to [Huang et al., 2017] in sev-eral aspects, including 1) we introduce a strategically-timedattack, which can reach the same effect of the uniform attackby attacking the agent four times less often on average; 2)we also introduce an enchanting attack tactic, which is thefirst planning-based adversarial attack to misguide the agenttoward a target state.

In terms of defending DNNs from adversarial attacks, sev-eral approaches were recently proposed. [Goodfellow et al.,2015] augmented the training data with adversarial exam-ples to improve DNNs’ robustness to adversarial examples.[Zheng et al., 2016] proposed incorporating a stability term tothe objective function, encouraging DNNs to generate similaroutputs for various perturbed versions of an image. Defensivedistillation is proposed in [Papernot et al., 2016b] for traininga network to defend both the L-BFGS attack in [Szegedy etal., 2014] and the fast gradient sign attack in [Goodfellow etal., 2015]. Interestingly, as anti-adversarial attack approacheswere proposed, stronger adversarial attack approaches alsoemerged. [Carlini and Wagner, 2016] recently introduced away to construct adversarial examples that is immune to var-ious anti-adversarial attack methods, including defensive dis-tillation. A study in [Rozsa et al., 2016] showed that moreaccurate models tend to be more robust to adversarial exam-ples, while adversarial examples that can fool a more accu-rate model can also fool a less accurate model. As the studyof adversarial attack to deep RL agents is still in its infancy,we are unaware of earlier works on the anti-adversarial attackto deep RL agents.

3 Adversarial AttacksIn this section, we will first review the adversarial exampleattack to DNN-based classification systems. We will thengeneralize the attack to deep RL agents and introduce ourstrategically-timed and enchanting attacks.

3.1 PreliminariesLet x be an image and f be a DNN. An adversarial exam-ple to the DNN can be crafted through solving the followingoptimization problem:

minδDI(x, x+ δ)

subject to f(x) 6= f(x+ δ), (1)

whereDI is an image similarity metric. In words, it looks fora minimal perturbation, δ, of an image that can change theclass assignment of the DNN to the image.

s25 s84 𝜹

+ =

action taken: up action taken: down

𝛽

Figure 1: Illustration of the strategically-timed attack on Pong. We use a function c to compute the preference of the agent in taking the mostpreferred action over the least preferred action at the current state st. A large preference value implies an immediate reward. In the bottompanel, we plot c(st). Our proposed strategically-timed attack launch an attack to a deep RL agent when the preference is greater than or equalto a threshold, c(st) ≥ β (red-dash line). When a small perturbation is added to the observation at s84 (where c(s84) ≥ β), the agent changesits action from up to down and eventually misses the ball. But when the perturbation is added to the observation at s25 (where c(s25) < β),there is no impact to the reward.

An RL agent learns to interact with the environmentthrough the rewards signal. At each time step, it performs anaction based on the observation of the environment for max-imizing the accumulated future rewards. The action determi-nation is through a policy π, which maps an observation toan action. Let the current time step be t, the goal of an RLalgorithm is then to learn a policy that maximizes the accu-mulated future rewards Rt = rt + rt+1 + ... + rL, where Lis the length of an episode.

In a deep RL algorithm, the policy π is modeled througha DNN. An adversary can attack an agent trained by thedeep RL algorithm by perturbing the observations (throughcrafting an adversarial example) to make the agent take non-preferred actions that can result in reduction of the accumu-lated future rewards.

3.2 Adversarial Attacks on RLIn a recent paper, [Huang et al., 2017] propose the uniformattack tactic where the adversary attacks a deep RL agent atevery time step, by perturbing each image the agent observes.The perturbation to an image is computed by using the fastgradient sign method [Goodfellow et al., 2015]. The uniformattack tactic is regarded as a direct extension of the adver-sarial attack on a DNN-based classification system, since theadversarial example at each time step is computed indepen-dently of the adversarial examples at other time steps.

It does not consider several unique aspects of the RL prob-lem. For example, during learning, an RL agent is nevertold which actions to take but instead discovers which actionsyield the most reward. This is in contrast to the classificationproblem where each image has a ground truth class. More-over, an adversarial attack to a DNN is considered a successif it makes the DNN outputs any wrong class. But the successof an adversarial attack on an RL agent is measured based onthe amount of reward that the adversary takes away from the

RL agent. Instead of perturbing the image to make the agenttakes any non-optimal action, we would like to find a pertur-bation that makes the agent takes an action that can reducemost reward. Also, because the reward signal in many RLproblems is sparse, an adversary need not attack the RL agentat every time step. Our strategically-timed attack tactic de-scribed in Section 3.3 leverages these unique characteristicsto attack deep RL agents.

Another unique characteristic of the RL problem is thateach action taken by the agent influenced its future obser-vations. Therefore, an adversary could plan a sequence ofadversarial examples to maliciously lure the agent toward acertain state that can lead to a catastrophic outcome. Our en-chanting attack tactic described in Section 3.4 leverages thischaracteristic to attack RL agents.

3.3 Strategically-Timed AttackIn an episode, an RL agent observes a sequence of obser-vations or states {s1, ..., sL}. Instead of attacking at everytime step in an episode, the strategically-timed attack selectsa subset of time steps to attack the agent. Let {δ1, ..., δL} bea sequence of perturbations. LetR1 be the expected return atthe first time step. We can formulate the above intuition as anoptimization problem as follows:

minb1,b2,...,bL,δ1,δ2,...,δL

R1(s1, ..., sL)

st = st + btδt for all t = 1, ..., L

bt ∈ {0, 1}, for all t = 1, ..., L∑t

bt ≤ Γ (2)

The binary variables b1, ..., bL denote when an adversarial ex-ample is applied. If bt = 1, the perturbation δt is applied.Otherwise, we do not alter the state. The total number of at-tacks is limited by the constant Γ. In words, the adversary

minimizes the expected accumulated reward by strategicallyattacking less than Γ << L time steps.

The optimization problem in (2) is a mixed integer pro-gramming problem, which is difficult to solve. Moreover, inan RL problem, the observation at time step t depends onall the previous observations, which makes the developmentof a solver to (2) even more challenging since the problemsize grows exponentially with L. In order to study adversar-ial attack to deep RL agents, we bypass these limitations andpropose a heuristic algorithm to compute {b1, ..., bL} (solv-ing the when-to-attack problem) and {δ1, ..., δL} (solving thehow-to-attack problem), respectively. In the following, wefirst discuss our solution to the when-to-attack problem. Wethen discuss our solution to the how-to-attack problem.

When to attackWe introduce a relative action preference function c for solv-ing the when-to-attack problem. The function c computes thepreference of the agent in taking the most preferred actionover the least preferred action at the current state st (similarto [massoud Farahmand, 2011]). The degree of preference toan action depends on the DNN policy. A large c value im-plies that the agent strongly prefers one action over the other.In the case of Pong, when the ball is about to drop from thetop of the screen (see s84 in Fig. 1), a well-trained RL agentwould strongly prefer an up action over a down action. Butwhen the ball is far away from the paddle (see s25 in Fig. 1),the agent has no preference on any actions, resulting a smallc value. We describe how to design the relative action pref-erence function c for attacking the agents trained by the A3Cand DQN algorithms below.

For policy gradient-based methods such as the A3C algo-rithm, if the action distribution of a well-trained agent is uni-form at state st, it means that taking any action is equallygood. But, when an agent strongly prefer a specific action(The action has a relative high probability.), it means that itis critical to perform the action; otherwise the accumulatedreward will be reduced. Based on this intuition, we define thec function as

c(st) = maxat

π(st, at)−minat

π(st, at). (3)

where π is the policy network which maps a state–action pair(st, at) to a probability, representing the likelihood that theaction at is chosen. In our strategically-timed attack, the ad-versary attacks the deep RL agent at time step t when therelative action preference function c has a value greater athreshold parameter β. In other words, bt = 1 if and onlyif c(st) ≥ β. We note the β parameter controls how often itattacks the RL agent and is related to Γ.

For value-based methods such as DQN, the same intuitionapplies. We can convert the computed Q-values of actionsinto a probability distribution over actions using the softmaxfunction (similar to [Huang et al., 2017]) with the temperatureconstant T .

c(st) = maxat

eQ(st,at)

T∑ake

Q(st,ak)

T

−minat

eQ(st,at)

T∑ake

Q(st,ak)

T

. (4)

How to attackTo craft an adversarial example at time step δt, we search fora perturbation to be added to the observation that can changethe preferred action of the agent from the originally (beforeapplying the perturbation) most preferred one to the origi-nally least preferred one. We use the attack method intro-duced in [Carlini and Wagner, 2016] where we treat the least-preferred action as the misclassification target (see Sec. 4.1for details). This approach allows us to leverage the output ofa trained deep RL agent as cue to craft effective adversarialexample for reducing accumulated rewards.

3.4 Enchanting AttackThe goal of the enchanting attack is to lure the deep RL agentfrom current state st at time step t to a specified target statesg after H steps. The adversary needs to craft a series ofadversarial examples st+1 + δt+1, ..., st+H + δt+H for thisattack. The enchanting attack is therefore more challengingthan the strategically-timed attack.

We break this challenging task into two subtasks. In thefirst subtask, we assume that we have full control of the agentto take arbitrary actions at each step. Hence, the task is re-duced to planning a sequence of actions for reaching the tar-get state sg from current state st. In the second subtask, wecraft an adversarial example st + δt to lure an agent to takethe first action of planned action sequence using the methodintroduced in [Carlini and Wagner, 2016]. After the agentobserves the adversarial example and takes the first actionplanned by the adversary, the environment will return a newstate st+1. We progressively craft st+1 + δt+1, ..., st+H +δt+H , one at a time, using the same procedure described in(Fig. 2) to lure the agent from state st+1 to the target statesg . Next, we describe an on-line planing algorithm, whichmakes use of a next frame prediction model, for generatingthe planned action sequence.

Future state prediction and evaluationWe train a video prediction model M to predict a future stategiven a sequence of actions based on [Oh et al., 2015], whichused a generative model to predict a video frame in the future:

sMt+H = M(st, At:t+H) , (5)

where At:t+H = {at, ..., at+H} is the given sequence of Hfuture actions beginning at step t, st is the current state, andsMt+H is the predicted future state. For more details about thevideo prediction model, please refer to the original paper.

The series of actions At:t+H = {at, ..., at+H} take theagent to reach the state sMt+H . Since the goal of the enchant-ing attack is to reach the target state sg , we can evaluate thesuccess of the attack based on the distance between sg andsMt+H , which is given by D(sg, s

Mt+H). The distance D is re-

alized using the L2-norm in our experiments. We note thatother metrics can be applied as well. We also note that thestate is given by the observed image by the agent.

Sampling-based action planningWe use a sampling-based cross-entropy method ( [Rubinsteinand Kroese, 2013]) to compute a sequence of actions to steerthe RL agent toward our target state. Specifically, we sam-ple N action sequences of length H: {Ant:t+H}Nn=1, and rank

video prediction model

adversaryunlabeled video

possible sequence of actions adversarial example

agent

Environment

st

st+1

st

at+

target state

training model planning

(1) (2)(3)

(4)

input

crafting

𝜹

Figure 2: Illustration of Enchanting Attack on Ms.Pacman. The blue panel on the right shows the flow of the attack starting at st: (1) actionsequence planning, (2) crafting an adversarial example with a target-action, (3) the agent takes an action, and (4) environment generates thenext state st+1. The green panel at the left depicts that the video prediction model is trained from unlabeled video. The white panel in themiddle depicts the adversary starts at st and utilize the prediction model to plan the attack.

each of them based on the distance between the final stateobtained after performing the action sequence and the targetstate sg . After that, we keep the best K action sequencesand refit our categorical distributions to them. In our experi-ments, the hyper-parameter values are N = 2000, K = 400,and J = 5.

At the end of the last iteration, we set the sampled actionsequence A∗t:t+H that results in a final state that is closest toour target state sg as our plan. Then, we craft an adversar-ial example with the target-action a∗t using the method in-troduced in [Carlini and Wagner, 2016]. Instead of directlycrafting the next adversarial example with target-action a∗t+1,we plan for another enchanting attack starting at state st+1 tobe robust to potential failure in the previous attack.

We note that the state-transition model is different to thepolicy of the deep RL agent. We use the state-transitionmodel to propose a sequence of actions that we want the deepRL agent to follow. We also note that both the state-transitionmodel and the future frame prediction model M are learnedwithout assuming any information from the RL agent.

4 ExperimentsWe evaluated our tactics of adversarial attack to deep RLagents on 5 different Atari 2600 games (i.e., MsPacman,Pong, Seaquest, Qbert, and ChopperCommand) using Ope-nAI Gym [Brockman et al., 2016]. These games represents abalanced collection. Deep RL agents can achieve an above-human level performance when playing Pong and a below-human level performance when playing Ms.Pacman. We dis-cuss our experimental setup and results in details. Our imple-mentation will be released.

4.1 Experimental SetupFor each game, the deep RL agents were trained using thestate-of-the-art deep RL algorithms including the A3C andDQN algorithms. For the agents trained by the A3C algo-rithm, we used the same pre-processing steps and neural net-

work architecture as in [Mnih et al., 2016]. For the agentstrained by the DQN algorithm, we also used the same networkarchitecture for the Q-function as in the original paper [Mnihet al., 2015]. The input to the neural network at time twas theconcatenation of the last 4 images. Each of the images wasresized to 84×84. The pixel value was rescaled to [0, 1]. Theoutput of the policy was a distribution over possible actionsfor A3C, and an estimate of Q values for DQN.

Although several existing methods can be used tocraft an adversarial example (e.g., the fast gradient signmethod [Goodfellow et al., 2015], and Jacobian-basedsaliency map attack [Papernot et al., 2016a]), anti-adversarialattack measures were also discovered to limit their impact[Goodfellow et al., 2015; Papernot et al., 2016b]. We adoptedan adversarial example crafting method proposed by [Carliniand Wagner, 2016], which can break several existing anti-adversarial attack methods. Specifically, it crafts an adversar-ial example by approximately optimizing (1) where the imagesimilarity metric was given by L∞ norm. We early stoppedthe optimizer when D(s, s+ δ) < ε, where ε is a small valueset to 0.007. The value of temperature T in Equation (4) isset to 1 in the experiments.

4.2 Strategically-Timed AttackFor each game and for the agents trained by the DQN andA3C algorithms, we launched the strategically-timed attackusing different β values. Each β value rendered a different at-tack rate, quantifying how often an adversary attacked an RLagent in an episode. We computed the collected rewards bythe agents under different attack rates. The results are shownin Fig. 3 where the y-axis is the accumulated reward and thex-axis is the average portion of time steps in an episode thatan adversary attacks the agent (i.e., the attack rate). We showthe lowest attack rate where the reward reaches the the re-ward of uniform attack. From the figure, we found that onaverage the strategically-timed attack can reach the same ef-fect of the uniform attack by attacking 25% of the time stepsin an episode. We also found an agent trained using the DQN

(a) Pong (b) Seaquest (c) MsPacman (d) ChopperCommand (e) Qbert

Figure 3: Accumulated reward (y-axis) v.s. Portions of time steps the agent is attacked (x-axis) of Strategically-timed Attack in 5 games. Theblue and green curves correspond to results of A3C and DQN, respectively. A larger reward means the deep RL agent is more robust to thestrategically-timed attack.

(a) Pong (b) Seaquest (c) MsPacman (d) ChopperCommand (e) Qbert

Figure 4: Success rate (y-axis) v.s. H steps in the future (x-axis) for Enchanting Attack in 5 games. The blue and green curves correspond toresults of A3C and DQN, respectively. A lower rate means that the deep RL agent is more robust to the enchanting attack.

algorithm was more vulnerable than an agent trained with theA3C algorithm in most games except Pong. Since the A3C al-gorithm is known to perform better than the DQN algorithmin the Atari games, this result suggests that a stronger deepRL agent may be more robust to the adversarial attack. Thisfinding, to some extent, echoes the finding in [Rozsa et al.,2016], which suggested that a stronger DNN-based recogni-tion system is more robust to the adversarial attack.

4.3 Enchanting AttackThe goal of the enchanting attack is to maliciously lure theagent toward a target state. In order to avoid the bias of defin-ing target states manually, we synthesized target states ran-domly. Firstly, we let the agent to apply its policy by t stepsto reach an initial state st and saved this state into a snap-shot. Secondly, we randomly sampled a sequence of actionsof length H and consider the state reached by the agent af-ter performing these actions as a synthesized target state sg .After recording the target state, we restored the snapshot andrun the enchanting attack on the agent and compared the nor-malized Euclidean distance between the target state sg andthe final reached state st+H where the normalization constantwas given by the image resolution.

We considered an attack was successful if the final statehad a normalized Euclidean distance to the target state withina tolerance value of 1. To make sure the evaluation was notaffected by different stages of the game, we set 10 initial timestep t equals to [0.0, 0.1, ..., 0.9] × L, where L was the av-erage length of the game episode played by the RL agents10 times. For each initial time step, we evaluated differentH = [1, 5, 10, 20, 40, 60, 80, 100, 120]. Then, for eachH , wecomputed the success rate (i.e., number of times the adversarymisguided the agent to reach the target state divided by num-ber of trials). We expected that a largerH would correspond amore difficult enchanting attack problem. In Fig. 4, we showthe success rate (y-axis) as a function of H in 5 games. We

found that the agents trained by both the A3C and DQN algo-rithms were enchanted. When H < 40, the success rate wasmore than 70% in several games (except Seaquest and Chop-perCommand). The reason that enchanting attack was less ef-fective on Seaquest and ChopperCommand was because bothof the games include multiple random enemies such that ourtrained video prediction models were less accurate.

5 ConclusionWe introduced two novel tactics of adversarial attack on deepRL agents: the strategically-timed attack and the enchantingattack. In five Atari games, we showed that the accumulatedrewards collected by the agents trained using the DQN andA3C algorithms were significantly reduced when they wereattacked by the strategically-timed attack even with just 25%of the time steps in an episode. Our enchanting attack com-bining video prediction and planning can lure deep RL agenttoward maliciously defined target states in 40 steps with morethan 70% success rate in 3 out of 5 games. In the future,we plan to develop a more sophisticated strategically-timedattack method. We also plan to improve video predictionaccuracy of the generative model for improving the successrate of enchanting attack on more complicated games. An-other important direction of future work is developing de-fenses against adversarial attacks. Possible methods includ-ing augmenting training data with adversarial examples (as in[Goodfellow et al., 2015], or training a subnetwork to detectadversarial input at test time and deal with it properly.

AcknowledgementsWe would love to thank anonymous reviewers, Chun-Yi Lee,Jan Kautz, Bryan Catanzaro, and William Dally for their use-ful comments. We also thank MOST 105-2815-C-007-083-Eand MediaTek for their support.

References[Brockman et al., 2016] Greg Brockman, Vicki Cheung,

Ludwig Pettersson, Jonas Schneider, John Schulman, JieTang, and Wojciech Zaremba. Openai gym, 2016.

[Carlini and Wagner, 2016] Nicholas Carlini and DavidWagner. Towards evaluating the robustness of neuralnetworks. https://arxiv.org/abs/1608.04644, 2016.

[Goodfellow et al., 2015] Ian Goodfellow, Jonathon Shlens,and Christian Szegedy. Explaining and harnessing adver-sarial examples. In ICLR, 2015.

[Hannun et al., 2014] Awni Hannun, Carl Case, JaredCasper, Bryan Catanzaro, Greg Diamos, Erich Elsen,Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, AdamCoates, et al. Deep speech: Scaling up end-to-end speechrecognition. https://arxiv.org/abs/1412.5567, 2014.

[Huang et al., 2017] Sandy Huang, Nicolas Papernot,Ian Goodfellow, Yan Duan, and Pieter Abbeel.Adversarial attacks on neural network policies.https://arxiv.org/abs/1702.02284, 2017.

[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever,and Geoffrey E Hinton. Imagenet classification with deepconvolutional neural networks. In NIPS, 2012.

[Kurakin et al., 2016] Alexey Kurakin, Ian J. Goodfellow,and Samy Bengio. Adversarial examples in the physicalworld. CoRR, abs/1607.02533, 2016.

[massoud Farahmand, 2011] Amir massoud Farahmand.Action-gap phenomenon in reinforcement learning. InJ. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira,and K. Q. Weinberger, editors, Advances in Neural Infor-mation Processing Systems 24, pages 172–180. CurranAssociates, Inc., 2011.

[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu,David Silver, Andrei A Rusu, Joel Veness, Marc G Belle-mare, Alex Graves, Martin Riedmiller, Andreas K Fidje-land, Georg Ostrovski, et al. Human-level control throughdeep reinforcement learning. Nature, 518(7540):529–533,2015.

[Mnih et al., 2016] Volodymyr Mnih, Adria PuigdomenechBadia, and Mehdi Mirza. Asynchronous methods for deepreinforcement learning. In ICML, 2016.

[Moosavi-Dezfooli et al., 2016] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deep-fool: A simple and accurate method to fool deep neuralnetworks. In CVPR, June 2016.

[Oh et al., 2015] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee,Richard L Lewis, and Satinder Singh. Action-conditionalvideo prediction using deep networks in atari games. InNIPS, 2015.

[Papernot et al., 2016a] Nicolas Papernot, Patrick Mc-Daniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik,and Ananthram Swami. The limitations of deep learningin adversarial settings. In Proceedings of the 1st IEEEEuropean Symposium on Security and Privacy, 2016.

[Papernot et al., 2016b] Nicolas Papernot, Patrick Mc-Daniel, Xi Wu, Somesh Jha, and Ananthram Swami.Distillation as a defense to adversarial perturbationsagainst deep neural networks. In Proceedings of the 37thIEEE Symposium on Security and Privacy, 2016.

[Rozsa et al., 2016] Andras Rozsa, Manuel Gunther, andTerrance E. Boult. Towards robust deep neural networkswith BANG. CoRR, abs/1612.00138, 2016.

[Rubinstein and Kroese, 2013] Reuven Y Rubinstein andDirk P Kroese. The cross-entropy method: a unified ap-proach to combinatorial optimization, Monte-Carlo simu-lation and machine learning. Springer Science & BusinessMedia, 2013.

[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, andQuoc V Le. Sequence to sequence learning with neuralnetworks. In NIPS, 2014.

[Szegedy et al., 2014] Christian Szegedy, WojciechZaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, IanGoodfellow, and Rob Fergus. Intriguing properties ofneural networks. In ICLR, 2014.

[Zheng et al., 2016] Stephan Zheng, Yang Song, ThomasLeung, and Ian Goodfellow. Improving the robustness ofdeep neural networks via stability training. In CVPR’2016,2016.