A Self-Adaptive Sleep/Wake-up SchedulingApproach for Wireless Sensor Networks

Dayong Ye and Minjie Zhang

Abstract—Sleep/wake-up scheduling is one of the fundamentalproblems in Wireless Sensor Networks (WSNs), since the energyof sensor nodes is limited and they are usually unrecharge-able. The purpose of sleep/wake-up scheduling is to save theenergy of each node by keeping nodes in sleep mode as longas possible (without sacrificing packet delivery efficiency) andthereby maxmising their lifetime. In this paper, a self-adaptivesleep/wake-up scheduling approach is proposed. Unlike mostexisting studies that use the duty cycling technique, which incursa trade-off between packet delivery delay and energy saving, theproposed approach, which does not us duty cycling, avoids sucha trade-off. The proposed approach, based on the reinforcementlearning technique, enables each node to autonomously decide itsown operation mode (sleep, listen or transmission) in each timeslot in a decentralised manner. Simulation results demonstratethe good performance of the proposed approach in variouscircumstances.

Keywords - Wireless Sensor Networks, Sleep/Wake-upScheduling, Self-Adaptation

I. INTRODUCTION

Due to recent technological advances, the manufacturing ofsmall, low power, low cost and highly integrated sensors hasbecome technically and economically feasible. These sensorsare generally equipped with sensing, data processing andcommunication components. Such sensors can be used tomeasure conditions in the environment surrounding them andthen transform these measurements into signals. The signalscan be processed further to reveal properties about objectslocated in the vicinity of the sensors. The sensors then sendthese data, usually via a radio transmitter, to a command center(also known as a ‘sink’ or a ‘base station’) either directlyor via several relaying sensors [1]. A large number of thesesensors can be networked in many applications that requireunattended operation, hence producing a Wireless SensorNetwork (WSN). Currently, there are various applications ofWSNs, including target tracking [2], health care [3], datacollection [4], security surveillance [5], [6], and distributedcomputing [7], [8].

Typically, WSNs contain hundreds or thousands of sensorswhich have the ability to communicate with each other [9].The energy of each sensor is limited and they are usuallyunrechargeable, so energy consumption of each sensor hasto be minimised to prolong the life time of WSNs. Major

This work was supported by two ARC Discovery Projects (DP140100974and DP150101775) from Australian Research Council, Australia.

D. Ye is with the State Key Laboratory of Software Engineering, WuhanUniversity, Wuhan 430072, China and the School of Software and ElectricalEngineering, Swinburne University of Technology, Melbourne, VIC 3122,Australia, email: dye@swin.edu.au; M. Zhang is with the School of ComputerScience and Software Engineering, University of Wollongong, Australia,email: minjie@uow.edu.au

sources of energy waste are idle listening, collision, overhear-ing and control overhead [10]. Among these, idle listening isa dominant factor in most sensor network applications [11].There are several ways to prolong the life time of WSNs,e.g., efficient deployment of sensors [12], optimisation ofWSN coverage [13] and sleep/wake-up scheduling [14]. Inthis paper, we focus on sleep/wake-up scheduling. Sleep/wake-up scheduling, which aims to minimise idle listening time,is one of the fundamental research problems in WSNs [15].Specifically, research into sleep/wake-up scheduling studieshow to adjust the ratio between sleeping time and awake timeof each sensor in each period. When a sensor is awake, it is inan idle listening state and it can receive and transmit packets.However, if no packets are received or transmitted during theidle listening time, the energy used during the idle listeningtime is wasted. Such waste should certainly be minimisedby adjusting the awake time of sensors, which is the aimof sleep/wake-up scheduling. Recently, many sleep/wake-upscheduling approaches have been developed, e.g., [16], [17],[14]. These approaches roughly fall into three categories: on-demand wake-up approaches, synchronous wake-up approach-es and asynchronous wake-up approaches, as categorised in[18].

In on-demand wake-up approaches [19], [20], out-of-bandsignaling is used to wake up sleeping nodes on-demand. Forexample, with the help of a paging signal, a node listening ona page channel can be woken up. As page radios can operateat lower power consumption, this strategy is very energyefficient. However, it suffers from increased implementationcomplexity.

In synchronous wake-up approaches [16], [21], [22], sleep-ing nodes wake up at the same time periodically to commu-nicate with one another. Such approaches have to synchroniseneighbouring nodes in order to align their awake or sleep-ing time. Neighbouring nodes start exchanging packets onlywithin the common active time, enabling a node to sleep formost of the time within an operational cycle without miss-ing any incoming packets. Synchronous wake-up approachescan reduce idle listening time significantly, but the requiredsynchronisation introduces extra overhead and complexity. Inaddition, a node may need to wake up multiple times duringa full sleep/wake-up period, if its neighbours are on differentschedules.

In asynchronous wake-up approaches [23], [24], [25], eachnode follows its own wake-up schedule in the idle state. Thisrequires that the wake-up intervals among neighbours are over-lapped. To meet this requirement, nodes usually have to wakeup more frequently than in synchronous wake-up approaches.The advantages offered by asynchronous wake-up approachesinclude easiness of implementation, low message overhead for

communication, and assurance of network connectivity evenin highly dynamic networks.

Most current studies use the technique of duty cycling toperiodically alternate between awake and sleeping states [17],[14]. Here, duty cycle is the ratio between the wake up timelength in a pre-defined period and the total length of that period[10]. For example, suppose a period is one second and a nodekeeps awake for 0.3 second and keeps asleep for 0.7 secondin the period. Then, the duty cycle is 30% (or 0.3). The useof duty cycling incurs a trade-off between energy saving andpacket delivery delay [26]: a long wake-up time may causeenergy waste, while a short wake-up time may incur packetdelivery delay. However, in WSNs, both energy saving andpacket delivery delay are important. Because each node inWSNs is usually equipped with an un-rechargeable battery,energy saving is crucial for prolonging the lifetime of WSNs.Because delay is unacceptable in some applications of WSNs,e.g., fire detection and tsunami alarm [27], reducing packetdelivery delay is crucial for the effectiveness of WSNs. Anintuitive solution to this trade-off is to dynamically determinethe length of wake-up time. The solution proposed in [28]can dynamically determine the length of wake-up time bytransmitting all messages in bursts of variable length andsleeping between bursts. That solution can save energy butit may exaggerate packet delivery delay, because each nodehas to spend time to accumulate packets in its queue beforeeach node transmits these packets in bursts. Another solution,proposed in [29], enables senders to predict receivers’ wake-up times by using a pseudo-random wake-up schedulingapproach. In the future, if senders have packets to transmit,senders can wake up shortly before the predicted wake-up timeof receivers, so the energy, which senders use for idle listening,can be saved. In this case, senders do not have to make thetrade-off, because their wake-up times are totally based onreceivers’ wake-up times. Receivers still face the trade-off,however, since a receiver’s wake-up time relies on a pseudo-random wake-up scheduling function and different selectionsof parameters in this function will result in different wake-upintervals. In addition, before a sender can make a predictionabout a receiver’s wake-up time, the sender has to request theparameters in the receiver’s wake-up scheduling function. Thisrequest incurs extra energy consumption.

In this paper, a self-adaptive sleep/wake-up scheduling ap-proach is proposed, which takes both energy saving and packetdelivery delay into account. This approach is an asynchronousone and it does not use the technique of duty cycling. Thus, thetrade-off between energy saving and packet delivery delay canbe avoided. In most existing duty cycling based sleep/wake-upscheduling approaches, the time axis is divided into periods,each of which consists of several time slots. In each period,nodes adjust their sleep and wake up time, i.e., adjustingthe duty cycle, where each node keeps awake in some timeslots while sleeps in other time slots. In the proposed self-adaptive sleep/wake-up scheduling approach, the time axisis directly divided into time slots. In each time slot, eachnode autonomously decides to sleep or wake up. Thus, inthe proposed approach, there is no ‘cycle’ and each timeslot is independent. Fig. 1 roughly displays how the proposedapproach works.

Fig. 1. Overview of the proposed approach

In Fig. 1, A and B are two neighbouring nodes whoseclocks may not be synchronised. They make decisions at thebeginning of each time slot autonomously and independentlywithout exchanging information. There are two points in thefigure which should be noted. First, for the receiver, if thelength of a time slot is not long enough to receive a packet,the length of the time slot will be extended automatically untilthe packet is received successfully (see the first time slot ofnode B). Second, when a node decides to transmit a packet inthe current time slot and the length of the time slot is longerthan the time length required to transmit a packet, the nodewill also decide when in the current time slot to transmit thepacket (see the third time slot of node B).

The proposed approach is not designed incorporating a spe-cific packet routing protocol. This is because if the sleep/wake-up scheduling approach is designed incorporation with aspecific packet routing protocol, the scheduling approach maywork well only with that routing protocol but may workless efficiently with other routing protocols. For example,in [14], Guo et al.’s sleep/wake-up scheduling approach isdesigned incorporation with a packet routing protocol. Theirscheduling approach uses staggered wake-up schedules tocreate unidirectional delivery paths for data propagation tosignificantly reduce the latency of data collection process.Their approach works very well if packets are delivered inthe designated direction, but it is not efficient when packetsare delivered in other directions.

The contributions of this paper are summarised as follows.

• To the best of our knowledge, this approach is the first onewhich does not use the technique of duty cycling. Thusthe trade-off between energy saving and packet deliverydelay, which is incurred by duty cycling, can be avoided.This approach can reduce both energy consumption andpacket delivery delay.

• This approach can also achieve higher packet delivery ra-tios in various circumstances compared to the benchmarkapproaches.

• Unlike recent prediction-based approaches [29], [30],where nodes have to exchange information betweeneach other, this approach enables nodes to approximatetheir neighbours’ situation without requesting informationfrom these neighbours. Thus, the large amount of energyused for information exchange [14] can be saved.

The rest of the paper is organised as follows. In the nextsection, the proposed self-adaptive sleep/wake-up schedulingapproach is described in detail. The simulation and analysisof the proposed approach is given in Section III. Finally, thepaper is concluded in Section IV. Due to the page limitation,The related work, theoretical analysis, a part of simulationand future work are provided in a separate supplementary filewhich can be found in IEEE Digital Library.

II. THE APPROACH DESIGN

In this section, we begin with the formulation of thesleep/wake-up scheduling problem. Then, the details of thealgorithms, which are involved in the proposed approach, areprovided.

A. Formulation of the problemAs described in Section I, the research of sleep/wake-up

scheduling studies how to adjust the ratio between sleepingtime and awake time of each sensor in each period as shownin Fig. 2.

Fig. 2. Formulation of the problem

According to Fig. 2, formally, we have:Definition 1 (Sleep): A sensor cannot receive or transmit

any packets when it is sleeping, i.e., in sleep state. A sensorin sleep state consumes very little energy;

Definition 2 (Wake-up): A sensor can receive and transmitpackets when it is awake, i.e., in wake-up state. A sensor inwake-up state consumes much more energy compared to sleepstate;

Definition 3 (Sleep/wake-up scheduling): Sensors adjustthe sleeping time length and the awake time length in eachperiod in order to save energy and meanwhile guarantee theefficient transmission of packets.

Generally, the radio transceiver in a sensor node has threemodes of operations (termed actions): transmit, listen andsleep [31]. In transmit mode, the radio transceiver can transmitand receive packets. In listen mode, the transmitter circuitryis turned off, so the transceiver can only receive packets. Insleep mode, both the receiver and the transmitter are turned off.Typically, among these actions, the power required to transmitis the highest, the power required to listen is medium and thepower required to sleep is much less compared to the othertwo actions. The example provided in [20] shows these powerlevels: 81mW for transmission, 30mW for listen and 0.003mWfor sleep.

B. Model DescriptionTable I describes the meaning of each symbol or term that

will be used in this paper.Interaction between two neighbouring nodes is modelled

as a two-player, three-action game, where two players1, a rowplayer and a column player, represent two neighbouring nodesand three actions mean transmit, listen and sleep. The threeterms, player, node and sensor, are used interchangeably inthis paper. Game theory is a mathematical technique which canbe used to deal with multi-player decision making problems.During the decision-making process, there may be conflictor cooperation among the multiple players. Such conflictor cooperation can be easily modelled by game theory via

1It is not necessary that one player must be a sender and the otherplayer must be a receiver. Both of them can send packets to the other partysimultaneously, although this will cause collision.

TABLE ITHE MEANING OF EACH SYMBOL OR TERM

symbol or term meaninga player a node/sensor

row player and column player two neighbouring nodes

three actions action 1: transmit, action 2: listen,action 3: sleep

state of a node

Each node has four states:s0, s1, s2, s3,

where s0 means the node does nothave any packet in its buffer,

s1 means the node has 1 packet in its buffer,s2 means the node has 2 packets in its buffer,s3 means the node has 3 packets in its buffer.

R and C payoff matrices for row playerand column player, respectively

rij

payoff obtained by row playerwhen row player takes action i

and column player takes action j

cij

payoff obtained by column playerwhen row player takes action i

and column player takes action j

α1, α2, α3the probability for row player

to select actions 1, 2, 3, respectively

β1, β2, β3the probability for column player

to select actions 1, 2, 3, respectivelyU Successful packet transmission reward

π(s, a)

the probability for a nodeto select action a in state s,

where a is one of the three actionsand s is one of the four states

Q(s, a)

the reinforcement value for a nodeto take action a in state s.

The value is be updated using payoffand then the value is used to update π(s, a).

ξ, δ, ζ, ε Learning ratesγ Discount factorη Gradient step size

TTL Time to live

properly setting the payoff matrices and utility functions.In WSNs, there are conflict and cooperation among sensorsduring many processes, such as packet routing and sleep/wake-up scheduling. Thus, in this paper, game theory is used to dealwith the sleep/wake-up scheduling problem among sensors inWSNs.

The game is defined by a pair of payoff matrices:

R =

r11 r12 r13r21 r22 r23r31 r32 r33

and C =

c11 c12 c13c21 c22 c23c31 c32 c33

,

where R and C specify the payoffs for the row player andthe column player, respectively. Each of the two players selectsan action from the three available actions. The joint action ofthe players determines their payoffs according to their payoffmatrices. If the row player and the column player select actionsi and j, respectively, the row player receives payoff rij andthe column player obtains payoff cij . The players can selectactions stochastically based on a probability distribution overtheir available actions. Let α1, α2, α3 denote the probabilityfor the row player to choose actions 1, 2, 3, respectively, whereα1 + α2 + α3 = 1. Let β1, β2, β3 denote the probability forthe column player to choose actions 1, 2, 3, respectively, whereβ1 + β2 + β3 = 1. The row player’s expected payoff is

Pr =∑

1≤i≤3

(∑

1≤j≤3

rijαiβj) (1)

and the column player’s expected payoff is

Pc =∑

1≤i≤3

(∑

1≤j≤3

cijαiβj). (2)

Let actions 1, 2 and 3 denote transmit, listen and sleep,respectively. The values of those payoffs in the payoff matricescan be defined by the energy used by a node (which isa negative payoff). In addition, if a packet is successfullytransmitted, the payoff of the transmitter/receiver is the energy,used to transmit/receive the packet, plus a positive constant, U ,say U = 98. Constant U is added on the energy consumption,if and only if a packet is successfully transmitted. The payofffor action sleep is −0.003 (the energy consumed during sleep-ing period) irrespective of the opponent’s action, where thenegative sign means that the energy is consumed. The value ofthe constant U is larger than the energy used for transmitting orreceiving a packet. For example, if the row player has a packetto transmit and it selects transmit and the column player selectslisten, the packet can be successfully transmitted. The payoffsfor both players are positive, which can be calculated usingthe energy they use to transmit/receive the packet plus theconstant U . Then, the row player gets payoff −81 + 98 = 17and the column player obtains payoff −30+98 = 68, where 81and 30 are energy consumption for transmitting and receivinga packet, respectively, and the negative sign means that theenergy is consumed. However, if the column player selectssleep, the packet cannot be successfully transmitted. Then, therow player gets payoff −81 (the energy used for transmitting apacket) and the column player gets payoff −0.003 (the energyused for sleeping). It should be noted that if a node does nothave a packet to transmit, it will not select transmit.

In a time slot, each node is in one of several states whichindicate the status of its buffer. For example, if a node’s buffercan store three packets, there are four possible states for thenode: s0, s1, s2 and s3, which imply that the node has 0, 1,2 and 3 packets in its buffer, respectively. The aim of eachnode is to find a policy π, mapping states to actions, that canmaximise the node’s long-run payoff. Specifically, for a node,π(s, a) is a probability, based on which the node selects actiona in current state s, and π(s) is a vector which is a probabilitydistribution over the available actions in current state s. Thus,policy π is a matrix. For example, a node’s buffer can storethree packets, so the node have four states: s0, s1, s2 and s3,as described above. Also, the node has three actions: transmit,listen and sleep, denoted as 1, 2 and 3, respectively. Hence,the policy of the node is:

π =

π(s0, 1) π(s0, 2) π(s0, 3)π(s1, 1) π(s1, 2) π(s1, 3)π(s2, 1) π(s2, 2) π(s2, 3)π(s3, 1) π(s3, 2) π(s3, 3)

.

Initially, as the node does not have any knowledge, eachaction is considered to be equally important in each state.Because for each state, s,

∑1≤i≤3 π(s, i) has to be 1, each

element in π is set to be 13 . Then, through learning, the node

will adjust the value of each element in π. The detail will begive in the following sub-sections.

Here, the terms π(s, a) and α, β denote the probabilityof selecting an action. π(s, a) takes states into considerationwhile α and β do not do so. α and β are used only fordescription convenience of the model and the algorithms.

C. The AlgorithmBased on the proposed model, we present a reinforcement

learning algorithm, which is employed by a player to learnits optimal actions through trial-and-error interactions withina dynamic environment. The algorithm is called Q-learning,Algorithm 1. Q-learning is one of the simplest reinforce-ment learning algorithms. Both reinforcement learning andevolutionary computation are sub-fields of machine learn-ing [32]. Reinforcement learning aims to solve sequentialdecision tasks through trial and error interactions with theenvironment [33]. In a sequential decision task, a participantinteracts with a dynamic system by selecting actions thataffect state transitions to optimise some reward function.Evolutionary algorithms are global search techniques derivedfrom Darwin’s theory of evolution by natural selection [32].An evolutionary algorithm iteratively updates a populationof potential solutions, which are often encoded in structurescalled chromosomes. Thus, the major difference betweenreinforcement learning and evolutionary algorithms is thatreinforcement learning is used by participants to maximisetheir individual rewards while evolutionary algorithms areused to achieve global optimisation. Moreover, reinforcementlearning algorithms are mainly decentralised and participantsneed only local information, whereas evolutionary algorithmsare primarily centralised or require global information [34].In this paper, WSNs are distributed environments and eachsensor has only local information about itself, so reinforcementlearning is more suitable than evolutionary algorithms to thesleep/wake-up scheduling problem.

The benefit of reinforcement learning is that a player doesnot need a teacher to learn how to solve a problem. Theonly signal used by the player to learn from its actions indynamic environments is payoff (also known as reward), anumber which tells the player if its last action was good ornot [33]. Q-learning as the simplest reinforcement learningalgorithm is model-free, which means that players using Q-learning can act optimally in Markovian domains withoutbuilding overall maps of the domains [35]. During the learningprocess, a player takes an action in a particular state based ona probability distribution over available actions. The higher theprobability of an action is, the more possible the action couldbe taken. Then, the player evaluates the consequence of theaction, which the player just takes, based on the immediatereward or penalty, which it receives by taking the action, andalso based on the estimate of the value of the state in which theaction is taken. By trying all actions in all states repeatedly,the player learns which action is the best choice in a specificstate.

Algorithm 1 describes how the proposed approach works inone time slot and is summarised as a flowchart in Fig. 3. Theapproach is described in the perspective of an individual node.According to Fig. 3, it can be seen that the proposed approachis composed of three steps. First, a node selects an action basedon a probability distribution over the three actions: transmit,listen or sleep, in the current state s. Second, the node carriesout the selected action and observes the immediate payoffand the new state s′. Finally, the node adjusts the probabilitydistribution over the three actions in state s based on the payoffand the approximation of the interacted neighbour’s policy.

Algorithm 1: Sleep/wake-up scheduling of a node1 Let ξ and δ be the learning rates and γ be the discount factor;2 For each action, initialise value function Q to 0 and policy π to

1n

, where n is the number of available actions;3 repeat4 select an action a in current state s based on policy π(s, a);5 if the selected action a is transmit then6 the node determines when to transmit the packet in the

time slot; /* ref Algorithm 2 */7 observe payoff p and next state s′, update Q-value

Q(s, a)← (1− ξ)Q(s, a) + ξ(p+ γmaxa′Q(s′, a′));8 if the selected action a is not sleep then9 based on the updated Q-value, approximate the policy

of the neighbour that interacted with the node in thecurrent time slot;

10 based on the approximation, for each action a ∈ A,update the node’s policy π(s, a);

11 else12 calculate the average payoff

P̄ (s)←∑a∈A π(s, a)Q(s, a);

13 for each action a ∈ A do14 π(s, a)← π(s, a) + δ(Q(s, a)− P̄ (s));

15 π(s)← Normalise(π(s)); /* ref Algorithm 3 */16 ξ ← k

k+1· ξ;

17 s← s′;18 until the process is terminated;

Fig. 3. The flowchart of Algorithm 1

The detail of Algorithm 1 is presented as follows.At the beginning of each time slot, Algorithm 1 repeats

from Line 3, except for the first time slot where the algorithmstarts at Line 1. In Line 1, a learning rate determines to whatextent the newly acquired information will override the oldinformation. The value of a learning rate is in the range [0, 1].A factor of 0 means that the node does not learn anything,while a factor of 1 means that the node considers only themost recent information [36]. A discount factor determinesthe importance of future rewards. The value of a discountfactor is in the range [0, 1]. A factor of 0 means that thenode is myopic by only considering current rewards, whilea factor approaching 1 means that the node strives for a long-

term high reward [36]. At the beginning of each time slot,a node has to decide in which mode it will be in this timeslot. The node thus selects an action based on the probabilitydistribution over its available actions in its current state (Line4). The initial probability distribution can be set equally overavailable actions. For example, in this paper, there are threeactions. Initially, the probability of selecting each action canbe set to 1

3 . Later, during the learning process, the probabilitydistribution over the actions will be updated based on theconsequence of each action. If the selected action is transmit,the node needs to decide when to transmit the packet in thetime slot (Lines 5 and 6, where the detail will be described inAlgorithm 2). The node then receives a payoff and reaches anew state. It updates the Q-value of the selected action in itscurrent state based on the received payoff and the maximumQ-value in the new state (Line 7). Here, Q-value, Q(s, a), is areinforcement of taking action a in state s. This information isused to reinforce the learning process. The formula in Line 7is a value iteration update. Initially, Q-value is given arbitrarilyby the designer. Then, the Q-value is updated using thecurrent Q-value, (1 − ξ)Q(s, a), plus the learned knowledge,ξ(p + γmaxa′Q(s′, a′)). The learned knowledge consists ofthe payoff obtained by the node after taking an action plusthe estimate of optimal future value: p+ γmaxa′Q(s′, a′). InLines 8-10, if the selected action is not sleep, the node willapproximate the probability distribution over the neighbour’savailable actions. This neighbour is the one that has interactedwith the node, i.e., transmitted a packet to the node or receiveda packet from the node, in the current time slot. Then, based onthe approximation, the node updates its policy π(s, a) for eachavailable action. In Lines 11-14, if the selected action is sleep,which means that the node does not interact with another nodein the current time slot, the node then updates its policy π(s, a)for each available action based only on its average payoff. InLine 12, the calculation of average payoff is based on theQ-value of an action times the probability of selecting theaction. Certainly, average payoff can also be calculated usingthe sum of the payoff of each action dividing the total numberof actions [37]. The former calculation method, however, ismore efficient and is more widely used than the latter one[38], [39]. In Line 13, the probability of selecting each actionis updated. The update of the probability of selecting an actionis derived using the current probability of selecting the actionplus the difference between the Q-value of the action and theaverage payoff. If the Q-value of an action is larger than theaverage payoff, the probability of selecting the action willbe increased; otherwise, the probability will be decreased. InLine 15, the probability distribution π(s) is normalised to be avalid distribution, where

∑a∈A π(s, a) = 1 and each π(s, a)

is within the range (0, 1). The details of the normalisation willbe described in Algorithm 3. Finally, in Line 16, the learningrate ξ is decayed, where k means that the current time slot isthe kth time slot. The learning rate is decayed to guarantee theconvergence of the algorithm as shown in Theorem 3 in thesupplementary file. The decay method is not unique. Actually,any progressive decay methods can be used here [40]. Then, atthe beginning of the next time slot, the node repeats Algorithm1 from Line 3.

Algorithm 2 and Algorithm 3 are parts of Algorithm 1,

where Algorithm 2 is the expansion of Line 6 in Algorithm 1and Algorithm 3 is the expansion of Line 15 in Algorithm 1.However, the description of Algorithm 2 and Algorithm 3 istoo complex to be integrated into Algorithm 1. Thus, for theclarity purpose, Algorithm 2 and Algorithm 3 are separatedfrom Algorithm 1 to become independent algorithms.

D. The Algorithm in More DetailIn Algorithm 1, there are four issues which have to be

addressed.1) If a node decides to transmit a packet in the current time

slot, how does the node determine when to transmit thepacket in the time slot (Line 6 of Algorithm 1)?

2) How does a node approximate its opponent’s probabilitydistribution without asking any information from theopponent (Line 9 of Algorithm 1)?

3) How does a node update its policy based on the approx-imation (Line 10 of Algorithm 1)?

4) How does an invalid probability distribution becomenormalised to a valid one (Line 15 of Algorithm 1)?

The solutions of the four issues are presented as follows.1) Issue 1: As discussed in Section I, if the time length of

a time slot is longer than the time length required to transmit apacket, a node should determine when to transmit the packetin the current time slot. The time length of a time slot andthe time length for each packet transmission are known aheadof time. Suppose that the time length of a time slot is longenough for transmitting m packets2. A time slot is then dividedinto m sub-slots. The length of a sub-slot is equal to the timelength required to transmit a packet. A node now needs toselect a sub-slot from the m sub-slots to transmit the packet.Two intuitive solutions are that (i) the node randomly picks upa sub-slot to transmit the packet; and (ii) from the beginningof the time slot, the node sends the packet. If the packet isnot sent successfully, the node sends it again. This processcontinues until the packet is sent successfully or the currenttime slot expires. However, the two solutions may incur alarge amount of energy waste, especially the second solution.To handle this issue, another learning algorithm is devised.

Algorithm 2: A node determines when to transmit a packetin a time slot

1 Let ζ and ε be the learning rates;2 For each sub-slot in the current time slot, initialise Q-value to 0

and the probability for selecting each sub-slot is initialised toxi = 1

m, where 1 ≤ i ≤ m and m is the number of sub-slots;

3 select a sub-slot in current time slot based on the probabilitydistribution over the sub-slots x = 〈x1, ..., xm〉;

4 observe payoff p and update Q-value for each sub-slot,Qi ← Qi + xi · ζ · (p−

∑1≤i≤m xiQi);

5 update xi for each sub-slot,

xi =

{(1− ε) + ( ε/m ), if Qi is the highestε/m , otherwise

6 x← Normalise(x);

In Algorithm 2, if a node decides to transmit a packetin a time slot, it selects a sub-slot based on the probability

2In this paper, it is assumed that each packet is transmitted using the sametime length. Relaxing this assumption is one of our future studies.

distribution x over the sub-slots (Line 3)3. The node thenobserves the payoff p obtained for selecting the sub-slot andupdate Q-value for each sub-slot based on payoff p and currentprobability distribution x (Line 4). Probability distribution xis adjusted based on the updated Q-values of sub-slots (Line5) and is normalised to be a valid distribution (Line 6). Theprobability distribution update method (Line 5) is ε-greedyexploration [41]. The ε-greedy exploration defines a semi-uniform probability distribution. The current best sub-slot isselected with probability 1−ε+ ε

m and one of the rest sub-slotsis selected with probability ε

m , where m is the number of sub-slots. The purpose of exploration is to harmonise the trade-offbetween exploitation and exploration such that the node canreinforce the evaluation of the sub-slots it already knows tobe good but also explore new sub-slots. The importance ofobtaining a Q-learning algorithm with ε-greedy exploration isalso justified through a large number of applications [42], [43],[44].

It should be noted that using the above-mentioned intuitivesolutions will incur energy waste, as these solutions do nothave any intelligence. The proposed algorithm, i.e., Algorithm2), however, enables nodes to intelligently select proper sub-slots for efficient packet transmission. Thus, compared tothe intuitive solutions, the energy wasted on failed packettransmission can be saved by using Algorithm 2. Algorithm 2needs time for convergence. However, during the convergenceprocess, the performance of Algorithm 2 increases gradually,whereas the performance of the intuitive solutions is notimproved as time progresses, because they do not have anylearning or adaptive ability. This means that Algorithm 2outperforms the intuitive approaches since the start of running.Then, the performance gap becomes bigger and bigger as timeprogresses.

2) Issue 2: Taking the row player for example (recallSection II-B), the expected payoffs of the row player playsactions 1, 2, 3 are

P (1)r = lim

α1→1

∑1≤i≤3

(∑

1≤j≤3

rijαiβj) =∑

1≤j≤3

r1jβj , (3)

P (2)r = lim

α2→1

∑1≤i≤3

(∑

1≤j≤3

rijαiβj) =∑

1≤j≤3

r2jβj , (4)

P (3)r = lim

α3→1

∑1≤i≤3

(∑

1≤j≤3

rijαiβj) =∑

1≤j≤3

r3jβj , (5)

respectively.It can be proven that if an action is executed in each

state an infinite number of times on an infinite run and thelearning rate ξ is decayed appropriately, the Q-value of thataction (calculated using the equation in Line 5 of Algorithm1) will converge with probability 1 to Q∗, where Q∗ is theexpected payoff of a player playing that action. The detailedproof will be presented in the supplementary file. Accordingto Equations 3, 4 and 5, it can been found that the row player’sexpected payoff for executing an action depend on the column

3Each time (except the first time) Algorithm 2 is called, only Lines 3-6are executed. As Lines 1 and 2 are used for initialisation, they are executedonly at the first time the algorithm is called. Also, Algorithm 2 is called andexecuted only when the node selects action transmit in Algorithm 1. Everytime Algorithm 2 is called, it is executed only once in the current time slot.

player’s probabilities for executing each action. Based on thisconclusion, the row player can use the current Q-value of anaction to approximate the expected payoff of that action (i.e.,Equations 6, 7 and 8).

Q(s, 1) = P (1)r =

∑1≤j≤3

r1jβj , (6)

Q(s, 2) = P (2)r =

∑1≤j≤3

r2jβj , (7)

Q(s, 3) = P (3)r =

∑1≤j≤3

r3jβj . (8)

Using Equations 6, 7 and 8, the row player can calculateβ1, β2 and β3 which are the column player’s probabilities totake actions 1, 2 and 3, respectively. After the calculation,the row player adjusts its own probabilities for executing eachaction, which will be described later. Certainly, it is infeasibleto execute the same action infinite number of times. Thus,the expected payoff can only be approximated but cannot beprecisely predicted. However, as the learning progresses, theQ-learning algorithm will gradually converge to the optimalQ-value, Q∗ and thus the precision of the approximation willincrease. Then, the column player’s probability distribution canbe computed more and more precisely by the row player. Thus,it can be seen that by using the approximation, the row playercan predict the column player’s probability distribution overthe actions without communication with the column player.

3) Issue 3: Using a gradient ascent algorithm [45], [46], aplayer can increase its expected payoff by moving its strategyin the direction of the current gradient with some step size. Thegradient is computed as the partial derivative of the player’sexpected payoff with respect to its probability for selectingeach action. We take the row player for example and set α3 =1− α1 − α2. Then, we have

∂Pr∂α1

=∑

1≤j≤3

r1jβj −∑

1≤j≤3

r3jβj (9)

and∂Pr∂α2

=∑

1≤j≤3

r2jβj −∑

1≤j≤3

r3jβj . (10)

If in the kth time slot, α(k)1 , α(k)

2 and α(k)3 are the row play-

er’s probabilities for selecting actions 1, 2 and 3, respectively,then the new probabilities for the next time slot are

α(k+1)1 = α

(k)1 + η

∂Pr

∂α(k)1

, (11)

α(k+1)2 = α

(k)2 + η

∂Pr

∂α(k)2

, (12)

andα(k+1)3 = 1− α(k+1)

1 − α(k+1)2 , (13)

where η is the gradient step size. Based on mathematicalknowledge [47], it can be known that if η is sufficiently small,the procedure will converge.

4) Issue 4: To normalise an invalid probability distributionto be a valid one, proportion-based mapping is used. Theinvalid probability distribution contains the learned knowledgeand in order to preserve the learned knowledge, the normalisedprobability distribution should be as ‘close’ as possible to theun-normalised one. Proportion-based mapping can adjust eachinvalid probability into the range (0, 1) and meanwhile, cankeep the relative magnitude of the probabilities. The functionNormalise() is presented in pseudocode form in Algorithm3.

Algorithm 3: Normalise()1 Suppose that in state s, there are m available actions, i.e.,a1, a2, ..., am;

2 Let d = min1≤k≤mπ(s, ak), mapping center c0 = 0.5 andmapping lower bound ∆ = 0.001;

3 if d < ∆ then4 ρ← c0−∆

c0−d;

5 for k = 1 to m do6 π(s, ak)← c0 − ρ · (c0 − π(s, ak));

7 for k = 1 to m do8 r ←

∑1≤k≤m π(s, ak);

9 π(s, ak)← π(s,ak)r

;

10 return π(s);

From the above description, it can be seen that, comparedto existing sleep/wake-up scheduling approaches, where n-odes’ sleep/wake-up patterns are almost pre-defined, this ap-proach enables nodes to autonomously and dynamically decidewhether to sleep using learning techniques. Thus, theoretically,nodes in this approach are smarter than nodes in most existingapproaches.

III. SIMULATION AND ANALYSIS

In this section, the simulation results and the correspondinganalysis are presented. The simulation was implemented usingJAVA programming language and was run on Windows 7Professional SP1 system with Intel Core i5 3.4GHz CPU and8GB RAM. ‘Node’ is programmed as a class and each node isan object of this class. To simulate an area, a two-dimensionalarray is defined, which represents the coordinates of an area.The length of the row of the array represents the length ofthe area and the length of the column of the array representsthe width of the area. For the connection of two nodes, iftwo nodes are in the communication range of each other, thetwo nodes are connected. For time implementation, a JAVAfunction is used to read the system time. To measure the timelength of delivering a packet, system time is read at both thebeginning of and the end of the delivery. Then, the time lengthof delivering a packet can be obtained using the system timeread at the beginning of the delivery minus the system timeread at the end of the delivery.

To evaluate the proposed self-adaptive approach (recordedas SA-Mech.), we build a platform using JAVA to test it incomparison with four other approaches: Two-Radio-MAC [20](abbreviated as TR-MAC), DW-MAC [48], EM-MAC [29] andAS-MAC [26]. TR-MAC is an on-demand approach. DW-MAC is a synchronous approach. EM-MAC and AS-MAC

are asynchronous approaches. Through comparing with theseapproaches, the performance of the proposed approach canbe objectively demonstrated. The reason for selecting theseapproaches is that TR-MAC and DW-MAC are the mostefficient on-demand approaches and synchronous approach,respectively4. Both EM-MAC and AS-MAC are the latestasynchronous approaches5, but, to the best of our knowledge,their performance has not been compared yet. Thus, weselect both of them in our simulation. These approaches aredescribed in detail as follows.

1) TR-MAC. In TR-MAC, two radios are used, where one isfor waking up neighbours and the other is for sending packets.Unlike traditional on-demand approaches, in TR-MAC, whena node has a packet to transmit, it does not wake up its entireneighbourhood but selectively wakes up several neighbourswhich have previously engaged in communication through rateestimation.

2) DW-MAC. DW-MAC is a synchronised duty cycle MACprotocol, where each cycle is divided into three periods: Sync,Data and Sleep. DW-MAC has to synchronise the clocks insensor nodes periodically during the Sync period. DW-MACthen sets up a one-to-one proportional mapping between a Dataperiod and the following Sleep period. In a Data period, thesender will send a scheduling frame to the receiver. Based onthe time interval after the beginning of the Data period and theduration of the scheduling frame transmission, both the senderand the receiver will set up their wake-up time interval duringthe following Sleep period to transmit/receive the packet.

3) EM-MAC. In EM-MAC, each node uses a pseudo-random number generator: Xn+1 = (aXn + c) mod m tocompute its wake-up times, where m > 0 is the modulus, a isthe multiplier, c is the increment, Xn is the current seed andthe generated Xn+1 becomes the next seed. In this simulation,m = 65536, each node’s a, c and Xn are independentlychosen following the principles suggested by Knuth [50]. Byrequesting the parameters, m, a, c and Xn, from a receiver,a sender can predict the receivers future wake-up times andprepare to send data at those times. EM-MAC does not needsynchronisation but it requires nodes to exchange informationbefore nodes can make predictions.

4) AS-MAC. In AS-MAC, nodes wake up periodically(but asynchronously) to receive packets. Nodes intending totransmit packets wake up at the scheduled wake-up time ofthe intended target nodes. Neighbouring nodes have to com-municate periodically to exchange information about wake-up schedules to avoid long preambles at the beginning oftransmission.

In addition, we also compare SA-Mech. with its synchro-nised version, SA-Mech.-Syn. In SA-Mech.-Syn, it is assumedthat a sink node periodically broadcasts a special packet to the

4Actually, DW-MAC is not the latest synchronous approach. Instead, Guoet al.’s approach [14] is the latest. However, Guo et al’s approach heavilydepends on predetermined transmission paths, so it is not very efficient oncea path is changed.

5Actually, Hsu et al.’s approach [49] was published later than both EM-MAC and AS-MAC. However, as Hsu et al.’s approach is jointly designedwith a routing protocol, it may not work well with other routing protocols.

entire network to synchronise the nodes’ clocks6. The aim ofintroducing SA-Mech.-Syn for comparison is to test how theintroduction of synchronisation will affect the performance ofSA-Mech.

A. Simulation Setup

The simulation is operated in two types of networks: gridnetworks and random networks. For each type of networks,there are four different scales. This setting is to evaluate theperformance of these approaches in different scales and dif-ferent types of networks. The scale of grid networks fluctuatesfrom 49 nodes to 169 nodes, where 49 nodes are structured asa 7× 7 grid network, 81 nodes are structured as a 9× 9 gridnetwork, 121 nodes are structured as a 11× 11 grid network,and 169 nodes are structured as a 13 × 13 grid network. Inthe four grid networks, each node is 200 meters from itsneighbours and there are 5 sinks which are located at thefour corners and the center of the network. The four scalesof random networks are 50 nodes, 80 nodes, 120 nodes and170 nodes which are randomly located in a 1000m× 1000marea. In the four random networks, there are 5 sinks which arealso randomly located in the area. The communication radiusof each node is set to 200m.

Each node generates a packet at the beginning of each timeslot based on a pre-defined probability: the packet generationprobability. As the state of a node is determined by the numberof packets in its buffer, the packet generation probabilitydirectly affects the state of each node. Then, the actionselection of each node will be indirectly affected. The expirytime of a packet is based on exponential distribution. Theaverage size of a packet is 100 bytes, and the actual size ofa packet is based on normal distribution with variance equalto 10. In this simulation, four packet generation probabilitiesare used: 0.2, 0.4, 0.6 and 0.8. This setting is to evaluate theperformance of these approaches in a network with differentnumber of transmitted packets. For packet routing, we use abasic routing approach, gossiping [51]. Gossiping is a slightlyenhanced version of flooding where the receiving node sendsthe packet to a randomly selected neighbour, which picksanother random neighbour to forward the packet to and so on,until the destination or the maximum hop is reached. It shouldbe noted that when the destination and some other nodes are allin the signal range of the source, based on the routing protocol,the source still relays a packet to one of neighbours andthis process continues until the destination or the maximumhop is reached. The routing process is not optimised in thesimulation, as this paper focuses on sleep/wake-up schedulingonly. This routing protocol is not energy-efficient but it is easyto implement. Because all of the sleep/wake-up schedulingapproaches use the same routing protocol in the simulation,the comparison among them is still fair.

Performance is measured by three quantitative metrics:average packet delivery latency, packet delivery ratio andaverage energy consumption. The minimum time needed by

6Actually, clocks can only be synchronised using broadcasting only if thereare not intermediate nodes for relaying. Otherwise, too much delay will incur.However, as the focus of this paper is not on clock synchronisation, thesynchronisation process is simplified.

nodes to transmit or receive a packet is about 2ms [14], e.g.,using the radio chip Chipcon CC2420. The three metrics aredescribed as follows.

1) Packet delivery latency is measured by the average timetaken by each delivered packet to be transmitted from thesource to the destination. Note that those packets, which donot reach the destination successfully, have also been takeninto account. Their delivery latency is the time interval, duringwhich they exist in the network.

2) Packet delivery ratio is measured by using the percentageof packets that are successfully delivered from the source tothe destination. Each packet comes with a parameter, TTL(Time-To-Live), which is a positive integer. Once a packet istransmitted from a sender to a receiver (no matter whethersuccessfully or unsuccessfully), the TTL of this packet sub-tracts 1. If the TTL of this packet becomes 0 and it has notreached the destination, the delivery of this packet is a failure.

3) Average energy consumption is calculated by using thetotal energy consumption to divide the number of nodes in thenetwork during a simulation run.

In this simulation, we set up the evaluation cases ourselves.The approaches used for comparison in the simulation are fromfour different references which use different evaluation cases.Thus, there are no common evaluation cases among thesereferences. Because all the evaluated approaches are testedin the same cases, the comparison is still fair and the resultsare convincing.

The values and meanings of the parameters used in thesimulation are listed in Table II, where the value of the TTL isset to 8 in grid networks and is set to 15 in random networks.The values of the parameters were experimentally chosen toprovide the best performance of the proposed approach7. Thevalues for energy consumption are from [20].

TABLE IIPARAMETERS SETTING

Parameters Values Explanations81mW Energy used for transmission30mW Energy used for listen

0.003mW Energy used for sleepU 98 Successful packet transmission reward

ξ, δ, ζ, ε 0.8, 0.4, 0.2, 0.2 Learning ratesγ 0.65 Discount factorη 0.0001 Gradient step size

TTL 8, 15 Time to live

For the compared approaches, the duty cycle is set to 5%. Afull sleep/wake-up interval is set to 1s. As our approach doesnot use the duty cycle, the time length of a time slot in ourapproach is set to 8ms. As described in Section II-C, learningrates are used to update the learned knowledge. Because thecompared approaches do not use reinforcement learning, theydo not need update and thus do not need learning rates. In eachsimulation case, each of these approaches has 200 repeats andeach repeat consists of 5000 seconds. The detailed simulationresult data are given in the supplementary file.

7The “best performance” means that the presented results are the best resultsthat we ever attempted. It does not mean that the presented results are the bestin all the combinations of parameter values. As the number of combinationsof parameter values is infinite, it is infeasible to attempt all the combinationsof parameter values.

B. Simulation Results in Grid Networks

The topology of a grid network is like a chessboard asdescribed in the previous sub-section. Grid networks havethree rules: 1) each of the four nodes that are located atthe four corners has two neighbours; 2) each node on theedge has three neighbours; and 3) all the other nodes havefour neighbours. Moreover, in this simulation, five sinks arelocated at the four corners and the center of the network,respectively. Therefore, the grid networks are highly regular.In this situation, the proposed approach, SA-Mech., can workwell, because SA-Mech. is based on reinforcement learning,and something with high regularity can be easily learned [33].In the grid networks, all the packets are transmitted to thefour corners and the center, where sinks are located. Thisregularity can be easily learned by nodes under SA-Mech. astime progresses, and based on the learned knowledge, nodescan precisely approximate their neighbours’ situations. Thedetailed analysis of the simulation results is given in thefollowing sub-sub-sections.

(a) Average delivery latency (ms)

(b) Packet delivery ratio (%)

(c) Average energy consumption (mW)

Fig. 4. Performance of the approaches in different scales of grid networks

1) Performance of the approaches in different scales ofgrid networks: Fig. 4 demonstrates the performance of theseapproaches in different scales of grid networks. The packetgeneration probability is fixed at 0.2. In Fig. 4(a), with the

increase of network scale, the average delivery latency in allof these approaches rises. This is because when the networkscale increases, based on the routing approach used in thissimulation, on average, each packet has to be transmitted withmore steps to its destination. Thus, the average delivery latencywill undoubtedly increase. Specifically, it can be found thatTR-MAC achieves the highest latency (36ms in the 49-nodenetwork, 52ms in the 81-node network, 72ms in the 121-node network and 88ms in the 169-node network). This isbecause in TR-MAC, whenever a sender intends to transmita packet, it has to wake-up the receiver and to wait for theresponse from the receiver. This process will continue untilthe packet reaches the destination or the TTL of the packetreaches 0, so each packet will suffer a large latency duringthe transmission process. DW-MAC and AS-MAC achievenearly the same latency while the latency in EM-MAC isslightly, about 5%, lower than that in DW-MAC and AS-MAC.This can be explained by the fact that both DW-MAC andAS-MAC require nodes to periodically exchange informationto schedule future wake-up intervals but EM-MAC does notneed such a step, so the time used for periodical informationexchange can be saved. Although in EM-MAC, nodes haveto request their neighbours’ information for future wake-upprediction, that request is on-demand and only once. Theproposed approach, SA-Mech., achieves the lowest latency(25ms in the 49-node network, 30ms in the 81-node network,42ms in the 121-node network and 50ms in the 169-nodenetwork), because (i) SA-Mech. does not require a periodicalinformation exchange and unlike EM-MAC, SA-Mech. doesnot require nodes to request their neighbours’ information forprediction, so the corresponding time is saved; (ii) as EM-MAC is an asynchronous approach, two neighbouring nodes’clocks are not synchronised. When a node makes predictionregarding its neighbour’s wake-up interval, it uses its ownclock, so the prediction may not be precise enough. In SA-Mech., nodes can dynamically approximate their neighbours’behaviour based on their own and their neighbours’ currentsituation, so SA-Mech. achieves a lower delivery latency,average 10%, than EM-MAC does.

In Fig. 4(b), with the increase of network scale, the packetdelivery ratios in all of these approaches decrease. This ismainly because of the routing approach. In different scalesof networks, each packet is given the same TTL. When thenetwork scale increases, the TTL value is no longer largeenough to guarantee successful transmission of a packet toits destination. Therefore, more and more packets are unableto be transmitted to their destinations successfully. In thisscenario, TR-MAC obtains the best result (95% in the 49-nodenetwork, 90% in the 81-node network, 87% in the 121-nodenetwork and 85% in the 169-node network). In TR-MAC, thetransmission is on-demand and the transmission happens afterthe sender communicates with the receiver, so packets are verylikely to be transmitted successfully. SA-Mech. achieves thesecond best result (95% in the 49-node network, 89% in the81-node network, 86% in the 121-node network and 83% inthe 169-node network). In SA-Mech., to save energy, nodesare not allowed to communicate to exchange information.Nodes make decisions based only on approximation whichmay contain errors, so SA-Mech. is somewhat, about 2%, less

efficient than TR-MAC in this scenario, however, SA-Mech. isstill better than the three other approaches. Unlike TR-MACand SA-Mech., where senders immediately transmit packetsto intended receivers after communication or approximation,in both DW-MAC and AS-MAC, there is a periodic schedulefor future wake-up times. Because in both DW-MAC and AS-MAC, packet transmission does not happen immediately afterscheduling, collision is possible in the future. For example,several senders schedule with one receiver for future wake-up times and these wake-up times may overlap, which willresult in packet transmission failure, because (i) in DW-MAC, the calculation of future wake-up time is based ona one-to-one proportional mapping between a Data periodand the following Sleep period, so there is a probability thattwo senders have the same or partially overlapping wake-uptimes with one receiver; (ii) in AS-MAC, there is not suchstrict one-to-one mapping but AS-MAC is an asynchronousapproach, so the scheduled wake-up times may still overlapdue to the asynchrony of nodes’ clocks. In EM-MAC, packettransmission does not happen immediately after schedulingand there is no synchronisation in EM-MAC, so it suffers thedrawbacks of both DW-MAC and AS-MAC. Hence, EM-MACachieves the worst result in this scenario 93% in the 49-nodenetwork, 80% in the 81-node network, 75% in the 121-nodenetwork and 68% in the 169-node network.

In Fig. 4(c), with the increase of network scale, the averageenergy consumption reduces gradually. Actually, with theincrease of network scale, the total energy consumption in-creases, as packets may have to be transmitted with more stepsto the destination. However, because the packet generationprobability and TTL are fixed, the total energy consumptioncannot increase much with the increase of network scale.Therefore, the increase speed of total energy consumption isslower than the increase speed of the number of nodes. Thus,the average energy consumption decreases with the increaseof network scale. Recall that average energy consumption iscomputed by using total energy consumption to divide thenumber of nodes. TR-MAC and SA-Mech. are very energyefficient (2230mW in in the 49-node network, 1978mW inthe 81-node network, 1682mW in the 121-node network and1453mW in the 169-node network), as both of them do notrequire nodes to communicate between each other to obtaininformation8. The energy consumption in EM-MAC is lessthan that in both DW-MAC and AS-MAC (about 10% lessthan DW-MAC and 5% less than AS-MAC), because DW-MAC and AS-MAC require an energy-consuming periodicalsynchronisation or scheduling period. This periodical processconsumes much energy. SA-Mech. is more energy efficientthan EM-MAC, since, as described above, EM-MAC suffersthe drawback of asynchrony, where a sender’s prediction,based on its own clock, about an intended receiver’s wake-up interval may not be precise enough.

In Fig. 4, it can be seen that SA-Mech. uses less energythan SA-Mech.-Syn, but SA-Mech.-Syn can achieve higherpacket delivery ratio and less packet delivery latency. Thisis because of the introduction of synchronisation. In order to

8Recall that in TR-MAC, for each node, there is a separate wake-up radiofor communication. The energy consumption of a wake-up radio is muchlower than that of a normal sensor node.

realise synchronisation, a sink has to periodically broadcastto synchronise nodes’ clocks and this synchronisation processinevitably costs some energy. However, synchronisation canmake the behaivour of nodes more predictable comparedwith asynchronised nodes. In SA-Mech., each node adapts itsbehaviour based on the prediction and approximation of itsneighbours’ behaviour and the introduction of synchronisationcan make nodes’ behaviour more predictable. Therefore, SA-Mech.-Syn can achieve higher packet delivery ratio and lesspacket delivery latency than SA-Mech. This phenomenon alsoexists in other circumstances (Figs. 5, 6) due to the samereason.

(a) Average delivery latency (ms)

(b) Packet delivery ratio (%)

(c) Average energy consumption (mW)

Fig. 5. Performance of the approaches under different packet generationprobabilities in grid networks

2) Performance of the approaches under different packetgeneration probabilities in grid networks: Fig. 5 demonstratesthe performance of these approaches under different packetgeneration probabilities. The number of nodes in the gridnetwork is fixed at 121 (11×11). In Fig. 5(a), with the increaseof packet generation probability, the average packet deliverylatency in each of the approaches rises. This is because whenmore packets exist in the network, collision is more likely tohappen, as two or more nodes transmitting might be withineach other’s signal range. Such collision will increase packetdelivery latency. An interesting phenomenon is that the packet

delivery latency in DW-MAC increases more sharply as thepacket generation probability rises compared with the otherapproaches. This is because in DW-MAC, a sender has toschedule packet transmission with a receiver during a Dataperiod. Since the Data period is short, when several senderswant to transmit packets to one receiver, the Data period isnot long enough for all the senders’ scheduling. Thus, someof the senders have to wait until the next cycle, which willprolong the packet delivery latency.

In Fig. 5(b), with the increase of packet generation prob-ability, the packet delivery ratio keeps almost steady in TR-MAC (about 87%), DW-MAC (about 80%), AS-MAC (about79%) and SA-Mech. (about 85%), but it goes down in EM-MAC (from 75% to 60%). This is due to the fact that theasynchrony of EM-MAC will incur senders’ imprecise predic-tion of intended receivers’ wake-up interval. Such impreciseprediction may lead to collision and thus increases the numberof packet delivery failures. As the number of packets in thenetwork increases, collision becomes more likely and thenumber of packet delivery failures rises correspondingly. Also,as shown in Fig. 5(c), because of such a large number ofpacket delivery failures, the corresponding energy is consumedsubstantially. Therefore, in EM-MAC, the energy consumptionrises significantly (from 1898mW to 2952mW ) as the packetgeneration probability increases, whereas in other approaches,the energy consumption just rises steadily due to the increasingnumber of transmitted packets in the network.

3) Performance variance of the approaches as time pro-gresses in grid networks: Fig. 6 demonstrates the performancevariance of these approaches as time progresses. The numberof nodes is fixed at 81 and the packet generation probabilityis fixed at 0.4. It can be seen that as time progresses, onlythe performance of SA-Mech. and SA-Mech.-Syn increasesgradually, whereas the performance of other approaches keepsalmost steady. This is because in both SA-Mech. and SA-Mech.-Syn, nodes have a learning process. Thus, as timeprogresses, nodes obtain more and more knowledge and workbetter and better. In other approaches, nodes do not have sucha learning process, so their performance is relatively stablein comparison with SA-Mech. and SA-Mech.-Syn. It has tobe noted that the convergence speed of SA-Mech. and SA-Mech.-Syn depends on the value of learning rate, ξ. If thevalue of ξ is large, e.g., 0.9, SA-Mech. and SA-Mech.-Syncan quickly converge to a relatively stable state in the earlystages but they may heavily oscillate in the late stages. Onthe contrary, if the value of ξ is too small, e.g., 0.1, SA-Mech. and SA-Mech.-Syn will converge very slowly but oncethey converge to a state, this state will be a very stable state.Therefore, in the simulation, we initially set ξ a large valueand let it automatically decay in each round. There are tworeasons for setting ξ in this way. First, SA-Mech. and SA-Mech.-Syn can quickly move towards a relatively stable statein the early stages and then slowly converge to an optimal statein the late stages. Thus, the oscillation can be avoided andthe convergence speed is relatively quick. The second reasonis that setting ξ in this way can meet the requirements ofTheorem 3 (presented in the supplementary file) and then, theconvergence of SA-Mech. and SA-Mech.-Syn to an optimalresult can be guaranteed.

(a) Average delivery latency (ms)

(b) Packet delivery ratio (%)

(c) Average energy consumption (mW)

Fig. 6. Performance variance of the approaches as time progresses in gridnetworks

The energy cost of learning, e.g., how much energy is usedduring a learning process, is not evaluated in this paper, be-cause the dominant energy cost in WSNs is idle listening [11].The time cost of learning has been empirically evaluated. For asingle process, i.e., the time period during which a node makesa decision about sleeping or awaking, the proposed approachneeds less time than the other compared approaches. This canbe explained by the fact that the proposed approach does notneed communication among neighbouring nodes, while theall of the other approaches need. Communication is usuallymore time consuming than computation. Thus, because ofthe prediction of neighbouring nodes’ behaviour, the proposedapproach needs more computation than the other approachesbut, under the proposed approach, packets need less deliverytime. Because the evaluation of average delivery latency hasalready included the evaluation of time cost, the figure aboutthe time cost for a single process of these approaches is notpresented in the paper.

In summary, the performance of the proposed approach,SA-Mech., is better than the other approaches in most cir-cumstances under the current simulation setup. Although insome circumstances, the performance of TR-MAC is betterthan that of SA-Mech, the difference is quite slight, about 5%,

considering that TR-MAC has to use separate communicationradios. Both SA-Mech. and its synchronised version SA-Mech.-Syn. perform well under the current simulation setup.SA-Mech.-Syn. achieves 8% higher packet delivery ratio and3% less packet delivery latency than SA-Mech. but needs 10%extra energy. Thus, which one is better depends on specificrequirements. Moreover, we have also evaluated a learning-based approach, RL-Mech. [52], in the simulation. Like ourSA-Mech., RL-Mech. is also based on reinforcement learning.Thus, SA-Mech. and RL-Mech. have the same trend in varioussituations. However, RL-Mech. is still based on duty cycletechnique and nodes have to keep awake for D consecutivetime slots, where D is the duty cycle fixed by users. In our SA-Mech., each time slot is independent and nodes do not haveto keep awake for a consecutive series of time slots. Thus, theaverage energy consumption under SA-Mech. is less than thatunder RL-Mech. for 30%. Also, as RL-Mech. is asynchronousand nodes cannot predict their neighbours’ duty cycle, a node’sawake time may not overlap with another node’s awake time.Therefore, the average delivery latency under RL-Mech. islonger than that under SA-Mech. for 20%, and the packetdelivery ratio under RL-Mech. is less than that under SA-Mech. for 15%. In addition, through the simulation, it has beenfound that the performance of these approaches depends, tosome extent, on the routing approach. Such dependency maylimit the applicability of these approaches in real situations.Therefore, such dependency should be avoided. We leave thisas one of our future studies.

IV. CONCLUSION

This paper introduced a self-adaptive sleep/wake-upscheduling approach. This approach does not use the techniqueof duty cycling. Instead, it divides the time axis into a numberof time slots and lets each node autonomously decide to sleep,listen or transmit in a time slot. Each node makes a decisionbased on its current situation and an approximation of itsneighbours’ situations, where such approximation does notneed communication with neighbours. Through these tech-niques, the performance of the proposed approach outperformsother related approaches. Most existing approaches are basedon the duty cycling technique and these researchers have takenmuch effort to improve the performance of their approaches.Thus, duty cycling is a mature and efficient technique forsleep/wake-up scheduling. This paper is the first one whichdoes not use the duty cycling technique. Instead, it proposesan alternative approach which is based on game theory andthe reinforcement learning technique. The performance im-provement of the proposed approach, compared with existingapproaches, may not be big, but the proposed approach pro-vides a new way to study sleep/wake-up scheduling in WSNs.This paper primarily focuses on theoretical study, so there aresome assumptions. These assumptions are set to simplify thediscussion of our approach. Without these assumptions, thediscussion of our approach will become extremely complex,which is harmful for the readability of this paper. The problemitself addressed in this paper, however, is not simplified bythese assumptions. Thus, the problem is still general underthese assumptions.

V. ACKNOWLEDGMENTS

We appreciate A/Prof. Zhen Ye for his help on theoret-ical analysis. We also wish to gratefully acknowledge Dr.Madeleine Strong Cincotta and Dr. Ian C. Piper’s help in thefinal language editing of this paper.

REFERENCES

[1] Y. Xiao, M. Peng, J. Gibson, G. G. Xie, D.-Z. Du, and A. V. Vasilakos,“Tight performance bounds of multihop fair access for mac protocolsin wireless sensor networks and underwater sensor networks,” IEEETransactions on Mobile Computing, vol. 11, pp. 1538–1554, 2012.

[2] S. Zhu, C. Chen, W. Li, B. Yang, and X. Guan, “Distributed optimalconsensus filter for target tracking in heterogeneous sensor networks,”IEEE Transactions on Cybernetics, vol. 43, pp. 1963–1976, 2013.

[3] G. Acampora, D. J. Cook, P. Rashidi, and A. V. Vasilakos, “A survey onambient intelligence in healthcare,” Proceedings of the IEEE, vol. 101,pp. 2470–2494, 2013.

[4] Y. Yao, Q. Cao, and A. V. Vasilakos, “Edal: An energy-efficient, delay-aware, and lifetime-balancing data collection protocol for heterogeneouswireless sensor networks,” IEEE/ACM Transactions on Networking,2014, dOI 10.1109/TNET.2014.2306592.

[5] S. H. Semnani and O. A. Basir, “Semi-flocking algorithm for motioncontrol of mobile sensors in large-scale surveillance systems,” IEEETransactions on Cybernetics, vol. 45, pp. 129–137, 2015.

[6] B. Fu, Y. Xiao, X. Liang, and C. L. P. Chen, “Bio-inspired groupmodeling and analysis for intruder detection in mobile sensor/roboticnetworks,” IEEE Transactions on Cybernetics, vol. 45, pp. 103–115,2015.

[7] Y. Zhao, Y. Liu, Z. Duan, and G. Wen, “Distributed average computationfor multiple time-varying signals with output measurements,” Int. J.Robust Nonlinear Control, vol. 26, pp. 2899–2915, 2016.

[8] Y. Zhao, Z. Duan, G. Wen, and G. Chen, “Distributed finite-time trackingof multiple non-identical second-order nonlinear systems with settlingtime estimation,” Automatica, vol. 64, pp. 86–93, 2016.

[9] M. Li, Z. Li, and A. V. Vasilakos, “A survey on topology controlin wireless sensor networks: Taxonomy, comparative study, and openissues,” Proceedings of the IEEE, vol. 101, pp. 2538–2557, 2013.

[10] W. Ye, J. Heidemann, and D. Estrin, “An energy-efficient mac protocolfor wireless sensor networks,” in Proc. of IEEE INFOCOM, New York,NY, Jun. 2002, pp. 1567–1576.

[11] W. Ye, F. Silva, and J. Heidemann, “Ultra-low duty cycle mac withscheduled channel polling,” in Proc. of ACM SenSys, Boulder, Colorado,USA, Nov. 2006, pp. 321–334.

[12] X. Liu, “A deployment strategy for multiple types of requirements inwireless sensor networks,” IEEE Transactions on Cybernetics, vol. 45,pp. 2364–2376, 2015.

[13] C. Chen and et al., “A hybrid memetic framework for coverage optimiza-tion in wireless sensor networks,” IEEE Transactions on Cybernetics,vol. 45, pp. 2309–2322, 2015.

[14] P. Guo, T. Jiang, Q. Zhang, and K. Zhang, “Sleep scheduling for criticalevent monitoring in wireless sensor networks,” IEEE Transactions onParallel and Distributed Systems, vol. 23, no. 2, pp. 345–352, Feb. 2012.

[15] G. Wei, Y. Ling, B. Guo, B. Xiao, and A. V. Vasilakos, “Prediction-based data aggregation in wireless sensor networks: Combining greymodel and kalman filter,” Computer Communications, vol. 34, pp. 793–802, 2011.

[16] W. Ye, J. Heidemann, and D. Estrin, “Medium access control withcoordinated adaptive sleeping for wireless sensor networks,” IEEE/ACMTransactions on Networking, vol. 12, no. 3, pp. 493–506, Jun. 2004.

[17] Y. Sun, O. Gurewitz, and D. B. Johnson, “Ri-mac: A receiver-initiatedasynchronous duty cycle mac protocol for dynamic traffic loads inwireless sensor networks,” in Proc. of ACM SenSys, Raleigh, NorthCarolina, USA, Nov. 2008, pp. 1–14.

[18] S. Lai, B. Ravindran, and H. Cho, “Heterogeneous quorum-based wake-up scheduling in wireless sensor networks,” IEEE Transactions onComputers, vol. 59, no. 11, pp. 1562–1575, Nov. 2010.

[19] L. Gu and J. Stankovic, “Radio-triggered wake-up capability for sensornetworks,” in Proc. 10th IEEE Real-Time and Embedded Technologyand Applications Symp., 2004, pp. 27–37.

[20] M. J. Miller and N. H. Vaidya, “A mac protocol to reduce sensornetwork energy consumption using a wakeup radio,” IEEE Transactionson Mobile Computing, vol. 4, pp. 228–242, 2005.

[21] Q. Cao, T. Abdelzaher, T. He, and J. Stankovic, “Towards optimal sleepscheduling in sensor networks for rare-event detection,” in Proc. FourthIntl Symp. Information Processing in Sensor Networks (IPSN), 2005.

[22] A. Keshavarzian, H. Lee, and L. Venkatraman, “Wakeup scheduling inwireless sensor networks,” in Proc. Seventh ACM MobiHoc, 2006, pp.322–333.

[23] R. Zheng, J. Hou, and L. Sha, “Asynchronous wakeup for ad hocnetworks,” in Proc. of ACM MobiHoc, 2003, pp. 35–45.

[24] J. Polastre, J. Hill, and D. Culler, “Versatile low power media access forwireless sensor networks,” in Proc. of ACM SenSys, 2004, pp. 95–107.

[25] P. Sommer and R. Wattenhofer, “Gradient clock synchronization in wire-less sensor networks,” in Proc. Ninth ACM/IEEE Intl Conf. InformationProcessing in Sensor Networks (IPSN), 2009, pp. 37–48.

[26] B. Jang, J. B. Lim, and M. L. Sichitiu, “An asynchronous scheduled macprotocol for wireless sensor networks,” Computer Networks, vol. 57, pp.85–98, 2013.

[27] J. Kim, X. Lin, N. B. Shroff, and P. Sinha, “Minimizing delay and max-imizing lifetime for wireless sensor networks with anycast,” IEEE/ACMTransactions on Networking, vol. 18, no. 2, pp. 515–528, 2010.

[28] T. V. Dam and K. Langendoen, “An adaptive energy-efficient macprotocol for wireless sensor networks,” in Proc. of ACM SenSys’03,2003, pp. 171–180.

[29] L. Tang, Y. Sun, O. Gurewitz, and D. B. Johnson, “Pw-mac: An energy-efficient predictive-wakeup mac protocol for wireless sensor networks,”in Proc. of IEEE INFOCOM, 2011, pp. 1305–1313.

[30] L. Tang, Y. Sun, O.Gurewitz, and D. B. Johnson, “Em-mac: A dynamicmultichannel energy-efficient mac protocol for wireless sensor network-s,” in Proc. of ACM MobiHoc, 2011, pp. 23:1–23:11.

[31] D. Niyato, E. Hossain, M. M. Rashid, and V. K. Bhargava, “Wirelesssensor networks with energy harvesting technologies: A game-theoreticapproach to optimal energy management,” IEEE Wireless Communica-tions, vol. 14, pp. 90–96, 2007.

[32] D. E. Moriarty, A. C. Schultz, and J. J. Grefenstette, “Evolutionaryalgorithms for reinforcement learning,” Journal of Artificial IntelligenceResearch, vol. 11, pp. 241–276, 1999.

[33] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcementlearning: A survey,” Journal of AI Research, vol. 4, pp. 237–285, 1996.

[34] Y. Gong and et al., “Distributed evolutionary algorithms and theirmodels: A survey of the state-of-the-art,” Applied Soft Computing,vol. 34, pp. 286–300, 2015.

[35] C. J. C. H. Watkins, “Q-learning,” Machine Learning, vol. 8, no. 3, pp.279–292, 1992.

[36] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction.MIT Press, 1998.

[37] S. Abdallah and V. Lesser, “Learning the task allocation game,” in Proc.of AAMAS06, 2006, pp. 850–857.

[38] ——, “Multiagent reinforcement learning and self-organization in anetwork of agents,” in AAMAS’07, Honolulu, Hawai’i, May 2007, pp.172–179.

[39] C. Zhang, V. Lesser, and P. Shenoy, “A multi-agent learning approachto online distributed resource allocation,” in IJCAI’09, Pasadena, Cali-fornia, Jul. 2009, pp. 361–366.

[40] M. Bowling and M. Veloso, “Multiagent learning using a variablelearning rate,” Artificial Intelligence, vol. 136, pp. 215–250, 2002.

[41] E. R. Gomes and R. Kowalczyk, “Dynamic analysis of multiagent q-learning with e-greedy exploration,” in ICML’09, Canada, 2009.

[42] A. Galstyan, K. Czajkowski, and K. Lerman, “Resource allocation inthe grid using reinforcement learning,” in AAMAS’04, New York, NY,Aug. 2004, pp. 1314–1315.

[43] E. R. Gomes and R. Kowalczyk, “Learning the ipa market with indi-vidual and social rewards,” in Proc. of the International Conference onIntelligent Agent Technology (IAT’07), Fremont, CA, USA, 2007, pp.328–334.

[44] N. P. Ziogos, A. C. Tellidou, V. P. Gountis, and A. G. Bakirtzis, “Areinforcement learning algorithm for market participants in ftr auctions,”in Proc. of the 7th IEEE Power Tech. Conference, 2007, pp. 943–948.

[45] S. P. Singh, T. Jaakkola, M. L. Littman, and C. Szepesvari, “Convergenceresults for single-step on-policy reinforcement learning algorithms,”Machine Learning, vol. 38, pp. 287–308, 2000.

[46] S. Singh, M. Kearns, and Y. Mansour, “Nash convergence of gradientdynamics in general-sum games,” in Proc. of UAI, 2000, pp. 541–548.

[47] D. Koller and N. Friedman, Probabilistic Graphical Models. MIT Press,2009.

[48] Y. Sun, D. Shu, O. Gurewitz, and D. B. Johnson, “Dw-mac: A lowlatency, energy efficient demand-wakeup mac protocol for wirelesssensor networks,” in Proc. of ACM MobiHoc, 2008, pp. 53–62.

[49] C.-C. Hsu, M.-S. Kuo, S.-C. Wang, and C.-F. Chou, “Joint designof asynchronous sleep-wake scheduling and opportunistic routing inwireless sensor networks,” IEEE Transactions on Computers, vol. 63,pp. 1840–1846, 2014.

[50] D. E. Knuth, The Linear Congruential Method, 3rd ed. Addison-Wesley,1997, vol. 2: Seminumerical Algorithms, ch. The Art of ComputerProgramming.

[51] S. Hedetniemi and A. Liestman, “A survey of gossiping and broadcastingin communication networks,” Networks, vol. 18, pp. 319–349, 1988.

[52] M. Mihaylov, Y. L. Borgne, K. Tuyls, and A. Nowe, “Decentralisedreinforcement learning for energy-efficient scheduling in wireless sensornetworks,” Int. J. Communication Networks and Distributed Systems,vol. 9, pp. 207–224, 2012.

Dayong Ye received his MSc and PhD degrees bothfrom University of Wollongong, Australia, in 2009and 2013, respectively. Now, he is a research fellowof computer science at Swinburne University ofTechnology, Australia. His research interests focuson service-oriented computing, self-organisation andmulti-agent systems.

Minjie Zhang received her BSc degree from Fu-dan University, P.R. China, in 1982 and her PhDin Computer Science degree from University ofNew England, Australia, in 1996. Now, she is anAssociate Professor of Computer Science at theUniversity of Wollongong, Australia. Her researchinterests include multi-agent systems, agent-basedsimulation and modeling in complex domains.