In recent years the Ethernet technology has taken several giant evolution steps. From the 10Mbit/s shared cable in 1990, the state of the art is now a switch-based communication technology running at 100 or 1000Mbit/s. The network switches are also getting more sophisticated in that they have started including support for packet priority and virtual networks. Since all this power comes at a steadily decreasing cost, it is steadily invading other cost-sensitive areas, most notably automation networks. This chapter will look at some critical aspects of switched Fast Ethernet as an automation network.

The Switches Are Not the Complete NetworkExchanging the classic Ethernet infrastructure for a switched infrastructure does wonders for the infrastruc-

ture but not necessarily for our automation network. The reason for this fact is illustrated in Figure 1, which shows all the stages in an information transfer between an automation controller and an I/O node.

Controllersoftware

Protocol

Network Interface

I/O nodesoftware

Protocol

Network Interface

Network

Drop Link Drop LinkFigure 1. The complete end-to-end network

The purpose of this chapter is to: Analyze the advantages, disadvantages and peculiarities of a switched Ethernet automation network Point out the remaining bottlenecks between the controller software and the I/O node software Discuss how the characteristics of the switched network affects some specialized automation network tasks

Analysing Switched Fast EthernetLet us start by recapitulating some basic facts about switched Ethernet automation networks.

1. Just as in a hub, an Ethernet switch contains a number of ports. Each port is either active (connected to an active Ethernet node) or passive (disconnected or connected to an inactive Ethernet node).

2. The connection between an active port and its associated Ethernet node is point-to-point. The connection may be full duplex (send and receive simultaneously) if the associated node supports it, otherwise it is half duplex.

3. Each port in an Ethernet switch has a table of network addresses associated with that port. These tables are created by inspecting network packets sent from the node and extracting the source address from the pack-ets.

4. Ethernet switches use a store-and-forward approach, which means that the complete network packet is re -ceived and verified by the switch before it is transferred to the output port.

5. The transfer of a network packet from one port to another inside the Ethernet switch is done by memory-to-memory copy at a very high speed.

6. An Ethernet switch does not use the collision mechanism of classic Ethernet.

7. Several transfers between different ports may take place more or less simultaneously. This means that an Ethernet switch has a potentially much greater data transfer capability than a hub.

Switched Ethernet In Automation Networking

The Learning Process inside the SwitchItem 3 above gave the algorithm the switch uses for the internal transfer of network packets. What happens

if the switch does not have the destination node address in any port table? In this case the switch plays it safe and transfers the packet to every output port in the switch. It reduces the switch performance to that of a hub, but it ensures that if the destination node is anywhere on the network, the packet will get through. In the mean time, the switch has learnt the location of the source node and stored it in the correct table.

This learning process is usually very fast, since each node only has to answer once in order for it to be asso-ciated with the correct port. There are, however some cases when the learning process fails, the most important being: Ethernet broadcast Ethernet multicast

Broadcast: Use with careEthernet inherited the concept of broadcast (an address that is accepted by every node on the network)

from HDLC. This functionality turned out to be very useful in several network protocols: Name announcing in NetBIOS (“Hi, everybody, my name is…”) The Service Advertising Protocol (SAP) in IPX/SPX Network address inquiry in ARP (“Hello everybody, who has got this IP address?”) IP address deployment in BOOTP and DHCP (“I just woke up and I want an IP address!”)

The drawbacks inherent in excessive usage of broadcast messages are: Broadcast messages cannot be filtered by hardware and must perforce be handled by software, thus con -

suming CPU resources If broadcast messages are allowed to pass through bridges and routers unchecked, they may create a broad-

cast storm (broadcast messages propagated in circles and thereby clogging the whole network) If a broadcast address is used as a source address, it may create a lot of network problems (this is a well-

known way of creating network meltdown) And, of course, it reduces the performance of a switched network to that of a hub-based network.

Multicast: A Useful, but Dangerous ConceptA huge number of Ethernet addresses are reserved for multicast usage. The Multicast concept addresses the

need for creating network groups of some sort. Traditionally, the multicast concept has been popular in some automation contexts, usually associated with a philosophy called publish-and-subscribe.

This philosophy allows independently developed distributed applications to be able to exchange informa-tion in an event driven manner without needing to know the source of the data or the network topology. Inform-ation producers publish information anonymously. Subscribers anonymously receive messages without request-ing them.

Like other broadcast-based models, publish-and-subscribe is efficient when used on a classic network. For example if the cost of energy changes in a distribution system, only a single transmission is required to update all of the devices dependent on the energy price (This is of course in the best or most optimistic case).

On a switched Ethernet network, the situation changes drastically. For a standard (unmanaged) switch, mul-ticasting actually uses more bandwidth than sending the same message to one recipient after another. This sur-prising fact is best illustrated with an example.

If we have a 16-port switch and send the same message to 4 nodes, the message will occupy the sending port 4 times and the 4 receiving ports one time each, for a total of 8 message times. Sending it as a multicast means that the message will occupy all 16 ports for one message time – a total of 16 message times.

The moral in this case is: Know your protocols and your infrastructure – and make sure they work together, not against one another.

There are Always BottlenecksReferring again to Figure 1, we observe that the network infrastructure is just one part of the communica-

tion path from the controller application to the I/O node application. The drop links are unchanged from the hub-based network and the end nodes (the controller and the I/O station) may well be unchanged from the days of the 10Mbit network – or even older. We shall now proceed to look at the most important bottlenecks in our switched network.

Even Highways have Queues

Queues in the switchesThe non-deterministic behaviour of traditional switched Ethernet is caused by unpredictable traffic patterns.

At times packets from several input ports will be destined for the same output port and some of them must per -force be queued up waiting for the output port to be free. The reason is that at times there will be a lot of low-priority traffic in the system (node status reports, node firmware update etc.). If we do not introduce some sort of traffic rules, such a situation will give an unpredictable buffering delay depending on the number of involved packets and their length. In worst case packets can be lost when the amount of packets sent to an output port ex-ceeds the bandwidth of this port for a time period that is longer than the output buffer is able to handle.

An automation network will have such co-existing real-time traffic (raw data, time sync data, commands, etc.) and non-critical data (TCP/IP – file transfer, etc.). A maximum size packet (1518 bytes) represents an ex-tra delay of 122s for any following queued packets in case of a 100Mbps network.

Queues on the drop linksEven if we manage to introduce some sort of priority mechanism in our system, we have one queue mech-

anism left. Since some standards (IEC 61850-9 springs to mind) propose to transfer real-time data by using a “publish and subscribe” approach (which is again based on the use of multicast groups), a standard unmanaged Ethernet switch has to route these data packets on to every drop link in the system adhering to the broadcast paradigm. Since all those real-time packets might have the same priority, they will be put in the same switch queue on all output ports. The processing rate for these queues is equal to the bandwidth of the drop link between the switch and the nodes, and therefore is effectively a drop link queue. This way, important data packets destined for one node may be delayed by multicast traffic destined for another node. Obviously, we need to introduce some traffic rules/mechanisms for multicast data packets as well in order to reduce the worst-case transfer time.

Queues in the nodesIntroducing traffic rules in the Ethernet switches will improve the worst-case latency across the network

(from the Ethernet controller in the source node to the Ethernet controller in the destination node). Inside the node, however, there is usually only one single network task, and just a single hardware queue associated with the Ethernet controller. Since a network packet spends a large percentage of its total end-to-end time inside a node, internal packet prioritisation is needed in order to have maximal control over the total packet transfer time (80-90% of the end-to-end message latency is spent within the end nodes, at least when adhering to the IETF protocols UDP/TCP/IP). For the transmit operation one typically could have a situation where multiple max size Ethernet packets,

for example fragments of an FTP transfer, queued up at the Ethernet driver level. In a standard implementa -tion, real-time packets will be added to the end of the queue. Such behavior will cause a non-predictable delay in transmission.

For the receive operation the protocol stack implementation represents a possible bottleneck in the system. This is mainly due to the first-come first-served queue at the protocol multiplexer level (the level where the network packets are routed to the appropriate protocol handler software).

Introducing a Standard for Priority and Delivery

High Priority Packets Jump Ahead in Switch QueuesIEEE 802.1D (see [2]) has been introduced to alleviate the switch queue problem; moreover, the standard

specifies a layer 2 mechanism for giving mission-critical data preferential treatment over non-critical data. The concept has primarily been driven by the multimedia industry and is based on priority tagging of packets and implementation of multiple queues within the network elements in order to discriminate packets. For tagging purposes IEEE 802.1Q [3] defines an extra field for the Ethernet MAC header. This field is called Tag Control Info (TCI) field and is inserted as indicated by Figure 2. This field contains 3 priority bits, thus the standard defines 8 different levels of priority.

Figure 2 - MAC header (layer 2) with tag

Multicast Distribution Rules Reduce Drop Link TrafficFigure 2 defines more than some priority bits. If you look closely, you can see something called a “12 bit

802.1Q VLAN identifier”. This VLAN identifier is a mandatory part of each TCI field and can, when properly used, remove all unnecessary drop link traffic in an automation network based on publish-and-subscribe. We will discuss the handling and usage of this field in the section on Virtual LANs.

High Priority Packets Get High Priority TreatmentA network data packet spends a relatively small percentage of is application-to-application transfer time on

the physical network. The actual percentage varies with the speed of the network and the performance of the node, but for a 100Mbit/s Fast Ethernet, the average percentage is between 20% and 0.1%. For this reason, im -plementing network priority will have little influence on the average application-to-application transfer time (it will, however have a large influence on the worst-case transfer time). In order to improve the average applica -tion-to-application transfer time, the concept of priority must be extended to include the protocol layers in both the sending and receiving end. In order to accomplish this, we must consider: Adjustable process priority for the protocol stack software Several instances of the protocol stack software running at different priority levels Multiple transmit queues at the Ethernet driver level

Matching Network Process Priority to Packet PriorityIn most real-time operating systems, the protocol stack runs as a single thread in the context of some net -

working task. If we want the task priority to depend on the packet priority, we can implement it in a very simple way:1. At compile time, decide on the task priority that should correspond to each packet priority and to an un -

tagged packet. Put those priorities in a table.2. Set a high basic networking task priority (the priority it uses when no packet is being processed).3. When the networking task starts to process a packet, it should extract the packet priority and use it with the

priority table described above to find the appropriate task priority.4. Change the task priority by executing a system call.

Multiple Instances of the Protocol Stack SoftwareThere is, however, a problem with the previous solution. Adjusting to correspond to the packet priority en-

sures that the processing of high priority network packets does not get interrupted by less important adminis -trative tasks. Still networking tasks process incoming messages one at a time in a linear fashion. What we really want is to process high priority messages before low priority ones. In fact, if we could suspend the processing of low priority network packets when a high priority message arrives, we would have the perfect solution.

At first glance, there exists an ideal solution. It goes like this:1. At compile time, decide on the task priority that should correspond to each packet priority and to an un-

tagged packet. Create one task for each priority and put the task IDs in a table.2. When a packet arrives, extract the packet priority, use it with the task table described above, send the packet

to that task and send a signal to the task in order for it to start processing.The problem with this solution is that it supposes that the network software is reentrant, a condition which

is seldom fulfilled. Rewriting the stack software to make it reentrant is not hard, just tedious and time-consum-ing. Of course, it also means that you have to support the rewritten software in the future.

Running multiple instances of the protocol software may be the most elegant and efficient solution, but be prepared to spend some time and resources on it.

Multiple Receive Queues and Adjustable PriorityIf we do not want to spend time and money on making the network software reentrant, there is an alternat -

ive solution available. This solution does not suspend the processing of lower priority packets, but selects the next packet to be processed from a set of criteria based on the packet priority. One possible implementation goes like this:5. At compile time, decide on the task priority that should correspond to each packet priority and to an un-

tagged packet. Create one queue for each priority and put a pointer to the queue in a table.6. Whenever the network software is ready to process the next packet, it pulls all packets from the input queue

and distributes them to the priority queues.7. When the input queue is empty, the priority selection algorithm is run. This algorithm may be implemented

in several different ways, for example: Always pick the packet from the highest priority non-empty queue. Pick the packet from the highest priority non-empty queue a certain number of times. Then move the

first packet from each non-empty queue to the next higher priority queue before picking the packet from the highest priority non-empty queue again.

Introduce a LOW flag. Wait for a packet to appear in the highest priority queue or for the LOW flag to be set. If the packet appeared in the highest priority queue, set the LOW flag and process that packet. If the LOW flag was set, reset it and process the packet from the highest priority non-empty queue.

Implementing Multiple Transmit QueuesOn a transmit request, a standard Ethernet driver takes a buffer, does some housekeeping and transfers it to

the Ethernet controller hardware. Such a driver is simple and “fair” (first come, first served), but may be unsuit-able for an efficient priority implementation. One or more large low-priority packets, once they are scheduled for transmission, will delay high-priority packets for the time it takes to transfer the low-priority ones. If we want high priority packets to go to the head of the transmission queue, and the hardware does not support mul-tiple queues, we must do some priority handling at the driver level.

The simplest solution is to use two queues, the hardware queue and a software queue. Low priority packets go into the software queue, high priority packets go straight into the hardware queue. From this point onward, there exist several algorithms addressing different needs. If the real-time requirements are moderate, move the first packet in the software queue to the hardware

queue whenever the hardware queue is empty. If the real-time requirements are strict, move the first packet in the software queue to the hardware queue

whenever a high priority packet has been placed in the hardware queue.

Bottleneck ConclusionsThe conclusion, based on a large set of real-world measurements (see [1]) is that:

1. At 100Mbit/s, the Ethernet switch and the drop links do not constitute a bottleneck under FTP load.

2. If several switches are interconnected by standard drop link cables, these cables might represent bottle-necks. Most switches have provisions for Gigabit Ethernet (1000Mbit/s) interconnections, however, thereby removing this possibility.

3. The main communication delays are inside the nodes. A suitable queue strategy for high priority packets assures that those will be processed before low priority packets.

4. For a high performance node processor, implementing prioritized transmit queues has a greater overall im-pact on high-priority packet delays than implementing prioritized receive queues.

5. If you want to implement internal receive priority queuing, ensure that it is possible to configure the chosen switch not to remove the priority tagging information.

Time Synchronization Across Switched Ethernet Now and then you come across measurement problems that are tightly associated with the notion of syn-

chronicity, meaning that things need to happen simultaneously. The usual things that need such synchronicity are data sampling and motion control. In the case of data sampling you need to know the value of two different quantities measured at the same time (within a narrow tolerance). If the measurement sources are close to -gether, this is fairly easy to accomplish, but if they are far apart and connected to different measurement nodes, it suddenly gets harder. The usual choices are:1. Use a special hardware signal on a separate cable between the controller and all nodes that need synchro-

nization. If the nodes are far apart and the tolerances are tight, make sure that all cables that carry the syn -chronization signal have the same length.

2. Add a local clock to each node and use the present automation network to keep them in synchronization. Tell each node how often the measurement sources should be sampled and require the node to time stamp each measurement.We shall now take a look at the hardest synchronization requirements for automation purposes and discuss

the possibility of implementing class T5 (1 s) and class T3 (25s) synchronization in a multi-traffic switched Ethernet environment. Common for both solutions is that they adhere to the same standardized time protocol. Such a step would significantly reduce the cabling and transceiver cost, since costly dedicated (separate) links are used for this purpose today.

The Concept of Time StampingLet us start at the very beginning – the concept of time stamping:

Time stamping is the association of a data set with a time value. In this context, “time” may also in-clude “date”.

Why would anybody want to time stamp anything? The closest example may be on your own PC – whenever you create a document and save it, the document is automatically assigned a data-and-time value. This value enables you to look for Documents created on a certain date (for example last Monday) Documents created within a certain time span (for example last half of 1998) The order in which a set of documents were created (for example the e-mails in your inbox).

If we just look at the examples above, we see that the accuracy we need is for the time stamping is about the same as we expect from our trusty old wristwatch. This again means “within a couple of minutes”, but as long as the clock does not stop it does not really matter much how precise it is.

Let Us Synchronize Our Watches!Now we know about time stamping on our own PC. The next step is to connect the PC to a network, maybe

even to the Internet and start exchanging documents and e-mails. What happens if the clock in your PC (the clock that is used for time stamping) is wrong by a significant amount? If you have an e-mail correspondence with someone, a reply (which is time stamped at the other end) might

appear to be written before the question (which is time stamped at your end) If you collaborate on some documents, getting the latest version might be problematical.

Therefore, when several PCs are connected together in any sort of network, the PC clocks are still accurate enough, but a new requirement is that they should be synchronized (show the same time at one point in time). Now, we could go around to each PC, look at our wristwatch and set the PC clock to agree with it. The trouble is that this is a boring and time-consuming job and we should look for a better solution.

One solution is to elect one PC to be the “time reference”, which means that every other PC should get the current time from it at least once a day and set its own clock to agree with that time. This solution works satis-factorily on a local area network (LAN), but all PC clocks will lag the time reference by the time it takes a clock value to travel from the time reference to the synchronizing PC. Except for very unusual cases, this lag is less than one second and thus good enough for office purposes.

Enter the Internet. Suddenly the synchronization problem escalates, since two collaborating PCs may be located in different time zones (remember to compensate for that) and a synchronization message may take a long time from one PC to the other. Fortunately, the Internet Network Time Protocol has a solution to both problems. This protocol involves sending a time stamped time request message to a “timeserver”. This timeserver adds an arrival time stamp and a retransmit time stamp before returning the request message to the requesting PC. The requesting PC time stamps the message when it returns and uses all the time stamps in cal-culating the correct time. This protocol and its little brother, the Simple Network Time Protocol, are able to synchronize computers across the Internet with a precision in the low milliseconds.

Synchronization Requirements in Substation AutomationIn the energy distribution world, a substation is an installation where the energy is combined, split or trans-

formed. A Substation Automation (SA) system refers to tasks that must be performed in order to control, mon -itor, and protect the primary equipment of such a substation and its associated feeders. In addition, the SA sys-tem has administrative duties such as configuration, communication management and software management.

The communication within SA systems is crucial from the point that the functionality demands for very time-critical data exchange. These requirements are substantially harder than the corresponding requirements in general automation. This is also true for the required synchronization accuracy of the IED’s 1 internal clock in order guarantee precise time stamping of current and voltage samples. Various SA protection functions require different levels of synchronization accuracy. IEC has provisionally defined five levels - IEC classes T1 – T5 (IEC 61850-5, sections 12.6.6.1 and 12.6.6.2): IEC class T1: 1 ms IEC class T2: 0.1 ms IEC class T3: 25 s IEC class T4: 4 s IEC class T5: 1 s

Since these definitions and classes are not frozen as yet, we will refer to them as class T1, class T2 etc. (without IEC) here.

At this point in time, the Substation Automation field is also on the edge of migrating towards the usage of Switched Fast Ethernet as the automation network infrastructure. The ultimate vision is to achieve interoperab-ility between products from different vendors on all levels within the substation automation field. A proof of this new trend is the upcoming IEC 61850 standard on Communication networks and systems in substations is-sued by Technical Committee. Inventions of de-facto standard concepts and adoption of off-the-shelf technolo-gies are the key instruments to reach the interoperability goal. Figure 3 illustrates the communication structure of a future substation adhering to switched Ethernet as a common network concept holding multiple coexisting traffic types.1 IED: Intelligent Electrical Device

Figure 3. The communication network in a future substation automation system using Ethernet as a common network infrastructure.

We already know that Switched Fast Ethernet has sufficient real-time characteristics to meet very demand-ing automation requirements. What is left to show is that it is possible to implement the various IEC classes of synchronization accuracy over such a network. This is considered to be the final obstacle of fully migrating to Ethernet in Substation Automation.

Time Synchronization StatusThere is a plethora of proposed theory and methods for synchronizing clocks in distributed systems. The

most prominent public domain synchronization method is the Network Time Protocol (NTP), which is stand-ardized by the Internet Engineering Task Force in RFC1305. A subset of NTP (SNTP - Simple Network Time Protocol), is also defined, and is protocol-compatible with NTP.

The intended use of NTP is to synchronize computer clocks in the global Internet. For this purpose it relies on sophisticated mechanisms to access national time, organize time synchronization subnets possibly imple-mented over various media, and adjustment of local clock in each participating peer. SNTP on the other hand does not implement the full set of NTP algorithms and is targeting simpler synchronization purposes. Common for this body of work is that it does not present solutions for low microsecond accuracy, instead targeting syn -chronization of LANs and WANs in the general sense where a precision of some milliseconds is sufficient.

Looking at the automation field in general and especially at the SA world, we find a diversity of proprietary and patented solutions in order to achieve highly accurate time synchronization over Ethernet. Interoperability concerns are, however,

Stating the Problem – why Network Synchronization is DifficultThe delays from the time stamping of a time synchronization message in the message source node until it is

time stamped in the message destination node are: Message preparation delay Communication stack traversal delay (transmission) Network access delay Network traversal delay Communication stack traversal delay (reception) Message handling delay

Variations in the delays are due to: Real Time Operating System (RTOS) scheduling unpredictability Network access unpredictability

Network transversal time variationsTime stamping at the lowest stack level helps eliminate the stack delay variations and real-time OS

scheduling unpredictability but introduces some complications in the implementation.

How to be Extremely Accurate – IEC Class T3We have mentioned that the precision that may be achieved by the traditional NTP/SNTP implementations

is one millisecond at best. Basically, this stems from the time stamping of incoming and outgoing NTP/SNTP packets at the NTP/SNTP application layer. As stated in the previous paragraph, this makes time stamping a victim of real-time OS scheduling unpredictability.

In this section we describe how a high degree of accuracy can be achieved using a tuned SNTP implement-ation and standard Ethernet switches.

A Tuned SNTP Time Protocol ImplementationThe NTP/SNTP time synchronization algorithm presented has definite limits to the level of attainable ac-

curacy. Let us recapitulate:The NTP/SNTP algorithm is based on a network packet containing three time stamps:

t1: The (client) time the packet was generated in the client asking for the current time t2: The (server) time the packet arrived at the timeserver. t3: The (server) time the packet was updated and put into the transmission queue at the server.

In addition, the calculations require: t4: The (client) time the packet arrived back at the client.

Now, t2 and t4 can easily be determined down to microsecond (and perhaps even better) using hardware or software time stamps based on packet arrival interrupts. The other two have definite problems, however.

For full accuracy, t1 and t3 should have been the time when the packet left the time client or timeserver re-spectively. The problem is that this time stamp is not available until the packet has really left the timeserver or client and then it is, of course, too late to incorporate it into the packet. Therefore the time synchronization in -accuracy for an NTP/SNTP setup is the variation in the delay between t1 and the time the packet leaves the time client plus is the variation in the delay between t3 and the time the packet leaves the timeserver.It turns out that there are several ways of determining the time when the synchronization packet actually leaves the time client. One algorithm runs like this:1. Create the NTP packet and fill in the t1 (Originate Timestamp) as usual. Transmit the package to the time-

server.2. Get hold of the time stamp when the packet leaves the time client, using any appropriate mechanism. Store

this time stamp (t11) in a data structure together with t1.3. When the packet returns from the timeserver, extract t2 from the packet and use t1 to look up the correspond-

ing value of t11. Since t11 and t1 are stored together, there is no chance of confusion here. 4. Substitute t11 for t1 in the packet.

Achieving Precise Time Stamping Within A NodeThe nature of real-time operating systems (they guarantee a maximum response time for an event but allow

for a wide variation below that) introduces a substantial variation in the time spent in the communication stacks. This fact has necessitated an interrupt-level time stamping both in the time client and timeserver. The IEC class T3 solution described here adheres to the principle of interrupt-level time stamping of the SNTP re-quest packet when sent from the time client and when received at the timeserver. Moreover, we propose that the synchronization is based on the (corrected) transmit time stamp set by the client (referred to as t1 in SNTP ter-minology) and the receive time stamp set by the server (referred to as t2). Usage of a possible low-level trans-mit time stamping of the corresponding SNTP reply packet (referred to as t3) necessitates novel techniques for controlling the non-deterministic access of an Ethernet packet to an Ethernet bus.

A side effect of using t1 - t2 only is that no mechanism for automatic calibration of the network latency will be available and therefore a manual calibration of the propagation delays of the drop links and the minimum

switch latency2 must be performed. illustrates the setup of a SNTP time client and timeserver adhering to inter-rupt-level time stamping.

Figure 4. The SNTP time client – server relation using low level time stamping.

Time client implementation issuesThere are several ways of time stamping a network packet. We shall look at three of them and show that

only the two first ones are suitable for accurate time synchronization:1. Hardware time stamping in the Ethernet controller.2. Software time stamping in an Interrupt Service Routine (ISR) outside the RTOS. This ISR should be con-

nected to the Ethernet Interrupt Request signal and have a top hardware priority.3. Software time stamping in an Interrupt Service Routine (ISR) controlled by the RTOS (Ethernet driver).

This ISR is connected to the Ethernet Interrupt Request signal with a normal hardware priority.Using any of this low-level time stamping methods is considered an implementation issue and will not

cause any incompatibly between a low-level time stamping client and a standard high-level time stamping server. In addition to low-level time stamping the time client must consider the following aspects: The interval between time updates. The specifications of the local time-of-day clock with respect to resolution, accuracy/stability, and the

availability of drift and offset correction mechanisms. The usage of adaptive filtering and time stamp validation methods in order to remove network delay varia -

tions.

Timeserver implementation issuesIn order to achieve class T3 accuracy, the timeserver should be able to time stamp an incoming message

with an accuracy of better than 2s independently of network load. The exact time should be taken from a GPS receiver, and the time parameters distributed from the timeserver should be based on GPS time representation instead of absolute time (i.e. UTC timing) in order to cope with the leap second problem. It is also convenient if the timeserver supports full duplex connectivity in order to avoid a situation where upstream data introduces extra switch latency in downstream data (i.e. time requests

Ethernet infrastructure implementation issuesOnly one switch should preferably be allowed between a time client and a timeserver. Having multiple

switch levels will impose increased jitter3 through the infrastructure, which again might call for more complex filtering at the time client side. The Ethernet switch must also have good switch latency characteristics. The switch latency from the client drop link to the server drop link depends on several parameters: General switch load; this means all the network load on the switch except for the packets sent to the time-

server. The variations in the switch latency from the client drop link to the server drop link should be less than 2s.

Timeserver load; this parameter means other packet sent to the timeserver that may introduce extra delay in the transmission of a given SNTP request packet. This delay can be handled at the time client side using various filtering techniques (see time client requirements).

2 The minimum Ethernet switch delay is usually given in the switch data sheet.3 Jitter: variations in the delay

Measurements on an Actual NetworkExtensive tests and measurements regarding time synchronization accuracy on a switched Ethernet network

have been undertaken. The conclusions from this body of work are: Traffic not destined for the timeserver does not interfere with traffic to the timeserver. The switch latency for Ethernet packets to the timeserver depends to a great extent on other traffic to the

timeserver.The full network measurements conclusions are:

Software time stamping using a sufficiently high priority interrupt (preferably non-maskable) is for all practical purposes indistinguishable from time stamping using special-purpose hardware.

Software time stamping using an interrupt under RTOS control needs sophisticated filtering and statistical techniques before it can be used for time synchronization purposes. In that respect this time stamping method is not suitable for IEC class T3 synchronization.

IEC Class T3 time synchronization using tuned SNTP over a switched Ethernet has shown to be eminently feasible.

Beyond the Speed of Light – Class T5It is now possible to procure industrial-class Fast Ethernet switches fulfilling an extensive list of environ-

ment requirements relevant for substation automation applications. Some of these switches can even be de-livered with an integrated SNTP timeserver.

Since the internal switch logic has full control over input and output ports, time stamping an SNTP request packet on arrival is no problem. In addition, the switch logic can insert the transmit time stamp whenever the output port is ready for the reply packet (and even adjust the time stamp for the delay between time stamping and actual transmission). Thus, the traditional problem related to the non-deterministic access to the Ethernet is not a problem here due to the tight interaction between the SNTP timeserver and the switch architecture.

This time synchronization scheme provides the following: Timing synchronization accuracy better than one microsecond if time stamping in the time client is per-

formed in hardware, see “Time client implementation issues”. Both server time stamps - t2 (receive) and t3 (transmit) - may be used at the time client for synchronization

purposes, and the drop link propagation delay can easily be removed based on the calculated round trip de-lay.

The timing accuracy is independent of the network load. No clever filtering/erasure techniques are needed in the time client.

Summary and conclusionsWe have presented general solutions for achieving class T5 (1 s) and class T3 (25 s) time synchroniza-

tion over switched Ethernet. The former is based on a dedicated Ethernet switch/Time Server combination, while the latter relies on standard switches. Common for both solutions is that they adhere to the low-level time stamp implementation of the Simple Network Time Protocol (SNTP).

Hardware time stamping or low-level software time stamping outside the real time operating system elim-inates the client inaccuracy from the error budget of the SNTP time synchronization loop. If the SNTP timeserver relies on the same time stamping techniques, the only remaining factor to be handled in the error budget is possible time delay variations within the infrastructure. In these settings class T3 synchronization is possible over switched Ethernet.

Introducing Virtual SubnetworksWhat is a Virtual Local Area Networks (VLAN)? A concept made possible by Ethernet switches, the multi-

tude of vendor-specific VLAN solutions and implementation strategies has made it very difficult indeed to define precisely what a VLAN is. Nevertheless, most people would agree that a VLAN might be roughly equated to a broadcast domain4. More specifically, VLANs can be seen as analogous to a group of end-stations, 4 Broadcast Domain: A collection of all nodes that can be reached by a broadcast message from one of them.

perhaps on multiple physical LAN segments, which are not constrained by their physical location and can com-municate as if they were on a common LAN.

Currently, three types of VLANs are of interest - those are: Port Group VLANs Layer 2 - Multicast Group-based VLANs Layer 3-Based VLANs

Here we will only give a brief overview of these different types of VLANs. An important aspect to note in this context, the fact that it is now possible to define dynamic VLANs that correspond exactly to a multicast group. This means that multicast frames will only propagate to members of the indicated VLAN and not to any-one else. In particular, the frames will not occupy drop links or CPU resources in nodes that do not belong to the multicast group.

Port Group VLANs Port based VLAN is a concept relying on a pure manual configuration of Ethernet switches (bridges) to set

up VLAN membership. Many initial VLAN implementations defined VLAN membership by groups of switch ports (for example, ports 1, 2, 3, 7, and 8 on a switch make up VLAN A, while ports 4, 5, and 6 make up VLAN B). Furthermore, in most initial implementations, VLANs could only be supported on a single switch.

Second-generation implementations support VLANs that span multiple switches (for example, ports 1 and 2 of switch #1 and ports 4, 5, 6, and 7 of switch #2 make up VLAN A; while ports 3, 4, 5, 6, 7, and 8 of switch #1 combined with ports 1, 2, 3, and 8 of switch #2 make up VLAN B).

Port grouping is still the most common method of defining VLAN membership, and configuration is fairly straightforward. Defining VLANs purely by port group does not allow multiple VLANs to include the same physical segment (or switch port). However, the primary limitation of defining VLANs by port is that the net-work manager must reconfigure the VLAN membership when a node is moved from one port to another.

Group-based VLANs: GARP VLAN Registration Protocol (GVRP)In the 802.1D standard (see [2]), the GARP (Generic Attribute Registration Protocol) is introduced. This is

a general protocol that allows network nodes to control selected switch properties. The two first implementa-tions of this protocol are the GARP Multicast Registration Protocol and the GARP VLAN Registration Pro-tocol (GVRP).

GVRP provides a mechanism that allows switches and end stations to dynamically register (and sub-sequently, de-register) VLAN membership information with the Ethernet switches attached to the same LAN segment, and for that information to be disseminated across all switched in the LAN that support Extended Fil-tering Services. Moreover, no manual configuration of the switches is required opposed to the port based solu-tion – on the other hand the switches must implement extra software as specified by the IEEE 802.1D standard [2].

The operation of GVRP relies upon the services provided by GARP. The information registered, de-re-gistered, and disseminated via GVRP is in the following forms:

a) VLAN membership information. This indicates that one or more GVRP participants that are members of a particular VLAN (or VLANs) exist, and the Ethernet frames carry a 12-bit VID5 (see Figure 2) that state the membership. The act of registering/deregistering a VID affects the contents of Dynamic VLAN Registration Entries to indicate the Port(s) on which members of the VLAN(s) have been re-gistered.

b) Registration of VLAN membership information allows the Ethernet switches in a LAN to be made aware that frames associated with a particular VID should only be forwarded in the direction of the re-gistered members of that VLAN. In this way the VLAN membership is propagated through the Ether-net infrastructure, and forwarding of frames associated with the VID therefore occurs only on Ports on which such membership registration has been received.

GVRP is a very new protocol concept and is not yet widely supported by Industrial Ethernet switches, but is foreseen to be one of the future technologies for handling the publish-and-subscribe automation network philosophy. One such handling algorithm consists of three steps:1. Map every multicast group to one specific VLAN identifier. This is, of course, an off-line activity.5 VID= VLAN Identifier

2. At system start-up time, all nodes send out one small packet for each VLAN (and thereby multicast group) it wants to join. This packet, which is not a part of any complicated protocol stack, gives the network infra-structure the information it needs in order to map VLAN identifiers to physical drop links. For simple data source nodes, these packets can be precompiled.

3. Whenever a multicast packet is to be transmitted, it should be tagged with “real time priority” and the VLAN identifier corresponding to the multicast group.These simple steps ensure that multicast packets do not block any unnecessary drop links. Below we dis-

cuss possible solutions for taking control of the latency within the end-nodes.

Layer-3 Based VLANsVLANs based on layer 3 information take into account protocol type (if multiple protocols are supported)

or network-layer address (for example, subnet address for TCP/IP networks) in determining VLAN member -ship. Although these VLANs are based on layer-3 information, this does not constitute a "routing" function and should not be confused with network-layer routing.

Even though a switch inspects a packet's IP address to determine VLAN membership, no route calculation is undertaken, and frames traversing the switch are usually bridged according to implementation of the Span-ning Tree Algorithm (the purpose of this algorithm is to make sure that no network packet is caught in an end-less loop between switches).

There are several advantages to defining VLANs at layer 3. First, it enables partitioning by protocol type. This may be an attractive option for network managers who are dedicated to a service- or application-based VLAN strategy. Second, nodes can physically move their workstations without having to reconfigure each workstation's network address - a benefit primarily for TCP/IP nodes. Third, defining VLANs at layer 3 can eliminate the need for frame tagging in order to communicate VLAN membership between switches, reducing transport overhead.

One of the disadvantages of defining VLANs at layer 3 (vs. MAC- or port-based VLANs) can be perform-ance. Inspecting layer 3 addresses in packets is more time consuming than looking at MAC addresses in frames. For this reason, switches that use layer 3 information for VLAN definition are generally slower than those that use layer 2 information. It should be noted that this performance difference is true for most, but not all, vendor implementations.

References[1] T. Skeie, S. Johannessen, and Ø. Holmeide, “The Road to an End-to-End Deterministic Ethernet”, In pro-

ceedings of 4th IEEE International Workshop on Factory Communication Systems (WFCS), Septem-ber, 2002.

[2] IEEE 802.1D, Information Technology - Telecommunications and Information exchange between systems – Local and Metropolitan Area Networks - Communication Specification – Part 3: Media Access Con-trol Bridges, 1998.

[3] IEEE Std 802.1Q-1998, IEEE Standards for Local and Metropolitan Area Networks: Virtual Bridged Local Area Networks.