Agile Management and Interoperability Testing of SDN/NFV ... · reference points, leveraging open-source PoC projects and communities to drive open standards of NFV. In 2016, it published

Agile Management and Interoperability Testing of

SDN/NFV-Enriched 5G Core Networks

Taesang Choi, TaeYeon Kim, Wouter Tavernier, Aki Korvala, and Jussi Pajunp€a€a

In the fifth generation (5G) era, the radio internetprotocol capacity is expected to reach 20 Gb/s persector, and ultralarge content traffic will travel across afaster wireless/wireline access network and packet corenetwork. Moreover, the massive and mission-criticalInternet of Things is the main differentiator of 5Gservices. These types of real-time and large-bandwidth-consuming services require a radio latency of less than1 ms and an end-to-end latency of less than a fewmilliseconds. By distributing 5G core nodes closer to cellsites, the backhaul traffic volume and latency can besignificantly reduced by having mobile devices downloadcontent immediately from a closer content server. In thispaper, we propose a novel solution based on software-defined network and network function virtualizationtechnologies in order to achieve agile management of 5Gcore network functionalities with a proof-of-conceptimplementation targeted for the PyeongChang WinterOlympics and describe the results of interoperabilitytesting experiences between two core networks.

Keywords: 5G core network (CN), Agilemanagement, Interoperability between CNs, Networkfunction virtualization (NFV), Software-definednetwork (SDN).

I. Introduction

In the fifth generation (5G) era, the radio internet protocol(IP) capacity is expected to reach 20 Gb/s per sector(mobile speeds up to 20 Gb/s), and ultralarge content traffic(for example, ultrahigh definition video streaming,augmented reality (AR), and virtual reality) will travelacross a faster wireless/wireline access network. All 5Gmobile/fixed traffic has to travel via the packet corenetwork (CN). Currently, in the fourth generation (4G),most mobile operators (even large-scale ones) have only afew sites with packet gateways (PGWs) across their entirenetworks. The software-defined network (SDN) paradigmprovides a new capability for faster service provisioning ofthe 5G CN through standard programmable interfaces.Moreover, with cloud computing, datacenters promote theon-demand provisioning of computing resources andservices [1]. If the 5G core nodes are distributed closer tocell sites, content servers (or caching servers) can be placedon the rack right next to the distributed 5G core withnetwork function virtualization (NFV) technologies. Thiscan help significantly reduce backhaul traffic by havingmobile devices download content immediately from thecontent server. Thus, it is desirable to distribute packet corefunctionality to a number of local sites near end users in thecoming 5G era. The 5G core functionality and applicationscan then run on virtualized servers at the local networksites. Other important 5G services—massive and mission-critical Internet of Things (IoT) services—are the maindifferentiator from 4G services. Mission-critical IoT(ultrareliable and low-latency communications) applicationsinclude remote-controlled machines, autonomous driving,and others. These types of ultra-real-time services require aradio latency of less than 1 ms and an end-to-end latency ofless than a few milliseconds [2].To address such challenges, we present a novel agile

management and orchestration (MANO) architecture

Manuscript received Oct. 13, 2017; revised Dec. 4, 2017; accepted Dec. 18,2017.Taesang Choi (corresponding author, [email protected]) and TaeYeon Kim

([email protected]) are with the Hyper-connected Communication ResearchLaboratory, ETRI, Daejeon, Rep. of Korea.Wouter Tavernier ([email protected]) is with the Department of

Information Technology, Gent University, Belgium.Aki Korvala ([email protected]) and Jussi Pajunpaa (jussi.pajunpaa@

nokia.com) are with Nokia, Oulu, Finland.

This is an Open Access article distributed under the term of Korea OpenGovernment License (KOGL) Type 4: Source Indication + Commercial UseProhibition + Change Prohibition (http://www.kogl.or.kr/info/licenseTypeEn.do).

https://doi.org/10.4218/etrij.2017-0236 © 2018 pISSN: 1225-6463, eISSN: 2233-7326

72ETRI Journal, Volume 40, Number 1, February 2018

http://onlinelibrary.wiley.com/journal/10.4218/(ISSN)2233-7326

http://www.kogl.or.kr/info/licenseTypeEn.do


based on enabling key technologies for 5G corefunctionalities, a proof-of-concept (PoC) implementationtargeted for PyeongChang Winter Olympics, anddeployment and interoperability testing experiences. Theproposed solution is an interim result of a collaborationproject between the Republic of Korea (KR) and theEuropean Union (EU) [3]. The rest of the paper isorganized as follows. We describe the enabling keytechnologies in Section II. We present the agile MANOarchitecture in Section III. Our prototype implementationand deployment experiences are described in Section IV.A performance evaluation of the proposed system,including the interoperability testing results, are providedin Section V. Finally, we conclude our paper with theplans for potential future work in Section VI.

II. Enabling Key Technologies

This section examines the key technologies for theSDN, NFV, MANO, mobile edge computing (MEC),mobility management, and control plane (CP) security andtheir associated design principles for the support of theproposed CN functionalities and their agile management.

1. Software-Defined Networking and Orchestration

Standardization efforts for SDNs were mainly carried outby the Open Networking Forum [4] and the InternationalTelecommunications Union – Telecommunications StudyGroup 13 (ITU-T SG13) [5] by defining the requirements,reference architecture, protocols, and use cases. Open-sourceprojects such as Open Daylight [6] and Open NetworkingOS [7] have played important roles in realizing the SDNconcept in real life. The SDN started with a limitednetworking environment such as cloud data centers andenterprise networks and has widened its coverage to wide-area transport networks and wireless/wireline integratedmultidomain networks. Instead of applying it as a standalonenetwork control tool, it is now used with NFV and as acomponent of an end-to-end orchestration solution. Itprovides an intelligent knowledge plane for making controldecisions via traffic steering, traffic engineering, and flexibleservice chaining for latency-sensitive and reliability-seekingapplications. It can be used in efficient communicationsamong distributed core functional components.

2. Network Function Virtualization

The virtualization of core and radio access networkfunctions will optimize the use of network resources andadd scalability and agility. To this end, the European

Telecommunications Standards Institute (ETSI) NFVIndustry Specification Group has defined the architecture,open application programming interfaces (APIs), andreference points, leveraging open-source PoC projects andcommunities to drive open standards of NFV. In 2016, itpublished Release 2 specifications and reports, includingthe functional requirements, interface, and informationmodel for the reference points for the MANO functionblock called NFV-MANO [8]. These open standardsare intended to enable third-party vendors to developframework components that can collaborate with variousvendor components so that content service providers) arenot restricted in selecting functional and managementcomponents. The main appeal of the use of NFV to deploynetwork elements and virtual network functions (VNFs) isthat services can be launched more quickly by installingsoftware on a standard hardware platform. This is akin tothe way software applications could be developed andlaunched for the personal computer (PC) platform whenit first emerged. Another advantage is lower capitalexpenditures because standardized hardware platformstend to drive costs down. Such advantages can be directlyapplied to the distributed core functional components inthe communications environment.

3. Mobile Edge Computing

In order to support the requirements for the market’sexpected throughput, latency, scalability, and programma-bility, ETSI established the Industry Specification Groupon Mobile Edge Computing in 2014 [9]. It develops astandardized and open environment that offers distributedcloud-computing capabilities and an IT service environ-ment for application developers and content providers. ByFebruary 2016, the group finalized three specifications:the terminology, the technical requirements, and theframework and reference architecture. This group alsoworks on specifications for MEC platform applicationenablement, the API principles and guidelines, the serviceAPIs for radio network information and location, userequipment (UE) identity and bandwidth management,system/host/platform management, lifecycle and policymanagement, the UE application interface, the deploymentof MEC in an NFV environment, and the end-to-endmobility.By offering distributed cloud-computing capabilities

and exposure to real-time radio network and contextinformation, MEC provides the following characteristics:

• Ultralow latency: Mobile edge services can be run closeto end-user devices to provide the lowest possiblelatency,

73Taesang Choi et al.



• Proximity: Being close to the source of information,MEC is particularly useful for capturing key informationfor analytics and big data,

• High Bandwidth: The mobile edge location at the edgeof the network combined with the use of real-time radionetwork information can be used to optimize thebandwidth for applications,

• Location awareness: A mobile edge can leverage thelow-level signaling information to determine thelocation of each connected device,

• Real-time insight into radio network and contextinformation: Real-time network data can be used by theapplications and services to offer context-related services.MEC can provide a significant improvement in a mobile

user’s quality of experience for latency- or quality ofservice (QoS)-sensitive services such as edge videoorchestration, mobile video throughput guidance, AR,intelligent video analytics, and others. Most importantly,MEC enables the implementation of mobile edgeapplications as software-only entities that run on top of avirtualization infrastructure, which is located in or close tothe network edge.

4. Distributed Mobility Management

It is essential to support distributed mobilitymanagement to enable agile management of the CNfunctionality. Currently, the Internet Engineering TaskForce is conducting standardization efforts to define adistributed mobility management architecture andmechanism in a layer 3 IP network environment. The 3rdGeneration Partnership Project (3GPP) also initiated workon defining layer 2 distributed mobility managementrequirements for a mobile communications environment.The functional decomposition and distribution of globalservice management will span multiple points of presence(PoPs) over the network, including network slices in a5G environment. It would be better to determine theanchoring and mobility management tailored to such anetwork environment at the central node, unlike exitinghierarchical and IP mobility. Composition functions andresources will be orchestrated for dynamic mobilitymanagement. Various experiments and simulations areunder study by the research community, and extensivetesting and verification of the concepts of distributedmobility management are needed.

5. Security of the 5G Core Network Control Plane

A software-defined mobile network (SDMN) controllerwill provide the necessary services to the CN functions by

working as an intermediary between the access and corefunctions. The network control functions of the coreelements, for example, the mobility management entity(MME), serving/packet data network gateways (S/P-GWs),and others, will reside in a centralized cloud in the form ofSDN applications that will leverage NFV technology to beinstantiated on different hardware or even at differentnetwork perimeters for a higher scalability and availability.Hence, the main security concern in such architectures willbe the SDN controller since it can become a potentialbottleneck for the overall network.To mitigate the risks of controller failure due to

scalability or the chances of denial of service (DoS)attacks due to its centralized role, controller resiliencestrategies have been proposed. These strategies includecontroller resilience through redundancy, maximizing thestorage and processing capabilities of the controller, anddistributing controller functionalities among multiplecontrol points in the network. The OpenFlow variant of anSDN supports wildcard rules so that the controller sendsan aggregate of client requests to server replicas. Bydefault, microflow requests are handled by the controllerthat can create potential scalability challenges, increasingthe chances of failures due to DoS attacks. Normally,reactive controllers that act on a flow request when itarrives at the controller are used. Proactive controllersinstall flow rules in advance, thus minimizing the flowrequest queue in the controller. Similarly, various load-balancing techniques that would balance the load amongmultiple controllers in a network have been suggested. Wehave worked on a novel communication architecture basedon the host identity protocol (HIP) to secure both controland data channels in SDMNs.

III. System Architecture

This section describes the proposed CN and agilemanagement system architecture based on a combinationof the key technologies described in Section II.

1. Core Network Architecture

We designed our CN architecture (Fig. 1) [10] tosupport CN functionalities and agile management on thebasis of the various key technologies described inSection II. Specifically, the CN functionality is realized byleveraging an SDN and NFV in order to facilitate thedynamic provisioning of CN functions. By using SDNcapabilities, traffic flows can be dynamically controlled,redirecting the traffic to gateways according to theworkloads. Simultaneously, the introduction of NFV

74 ETRI Journal, Vol. 40, No. 1, February 2018

https://doi.org/10.4218/etrij.2017-0236

permits the separation of service functionalities from thecapacity-constrained specific network entities and allowdynamic instantiation in commodity and powerful servers.Starting from late 1990s, the 3GPP has been taking stepstowards a clear separation of the data and control planesand the respective elements in the architecture. Wepropose to take this concept to the next level following theSDN paradigm. Figure 1 also presents the 5G networkcontrol as a group of SDN applications. They are the BaseStation App, Backhaul App, Mobility Management App(MM App), Monitoring App, Access App, and SecureService Delivery App. The network applications areorchestrated via the Controller Northbound API. MultipleSDN applications operate without conflicts.The Base Station App runs the control software that is

now vertically integrated with the evolved Node B (eNB).The physical base stations under its control consist of anantenna, a band-pass filter, and an Ethernet card forbackhaul connectivity [11]. The MM App implementsmobility as a service and incorporates the MME. Inaddition, it needs to manage the QoS for each user,balance the load among alternative paths across theaggregation network, and route the user to a cache whenpossible. The MM App also chooses the path for a device.The load-balancing decision is made on the basis of theinput from the Network Monitoring App. In any case, it isdesirable that the point of attachment of a mobile device tothe Internet is fixed while it remains within the coverageof the current mobile network [11].In one physical mobile network, there may be many

Access Apps. In this case, an Access App is owned andoperated by a particular mobile virtual network operator.Putting mobility aside, the Access App is responsible forproviding data services to mobile users. The keyproperties of the Access App include providing Internetaccess, firewalling unwanted traffic, and providing accessto premium content [11].

The main CN functions are designed and implementedin the form of virtual functions, namely, virtual evolvedpacket cores (vEPCs). Both the EU and KR provide theirown implementations of vEPCs based on this architecture.They are described as follows.

2. European vEPC Architecture (5GTN)

The EU vEPC consists of the following VNFs:

• Mobile gateway: The cloud mobile gateway providesthe service provider-gateway (SP-GW), gateway generalpacket radio service (GPRS) support node, and trafficdetention functions (TDFs), evolved packet datagateway, and trusted wireless access gateway.

• Mobility management: The cloud mobility managerprovides the MME and servicing GPRS support nodefunctions.

• Policy control and charging: The dynamic servicescontroller built on patented agile rules technologyengine provides the policy and charging rules function(PCRF) and wireline radius/change of authorization.Element and network management: The service-aware

manager provides end-to-end network managementvisibility across the entire mobile network.To support the scalability required to meet the expected

5G and IoT service requirements, the packet core VNFsprovide three key design innovations:

• The packet core VNFs are decomposed into separate CPand data-plane virtual machine (VM) instances. Thisenables a distributed architecture where data-planeresources can be deployed in edge data centers closer tothe device, while CP resources can be centralized.

• State-efficient VNF processing unpins the subscriber/device state information from the VMs, freeing up theunderlying computing resources to be reused to processother subscribers/devices.

• The remote cloud database synchronizes the subscriber/device state information into a real-time data store.

• The 5GTN functional architecture [10] is given inFig. 2.

3. Korean vEPC Architecture

The CN of 4G Long Term Evolution (LTE) is in chargeof mobility, authentication, and charging, allowing allmobile traffic to pass through the CN to access servicesincurring traffic congestion in the CNs. Our architecturaldecision for 5G is to distribute mobile core functions tothe edge nodes. A 5G core is generally divided into a 5Gcore user plane (UP) in charge of bearer delivery and a 5GCore CP in charge of signaling and control of the 5G CN.

Applications

InfrastructureCache DPI FW

Internet@ccess

SDN controller NorthboundSouthbound

PCRFHSS

Topologydiscovery

Link provisioning

Load balancingSLA

Service recovery

MME Cache Policy Firewall CES

DNSDHCPBackhaul provisioning

Mobility management

Internet access

Base station Network monitoring

Mobility management

Access control

ChargingSecure service delivery

Fig. 1. CN architecture.




The key CN architectural design principle is a centralizedCP with a distributed UP over the edge nodes.If the CN where bearers are terminated is located closer

to the cell sites, the application servers follow naturally, andthe backhaul traffic will significantly decrease, resulting ina cost reduction for continual backhaul enhancement.A 5G network is supposed to be able to provide

ultra-real-time services such as highly sensitive remotecontrol and automatic driving vehicles. These types ofservices may generate much lesser traffic than videostreaming applications but require an ultralow latency.Figure 3 illustrates the high-level architecture of aKorean vEPC. It is realized as a highly scalable vEPC(HSvEPC) [12]. Its functionality and architecture aredescribed below.

4. HSvEPC Network Architecture

It is possible to deploy different types of virtual mobilepacket cores depending on the demand or network accessenvironment in an HSvEPC network architecture. Twotypes of vEPCs are designed (shown in Figs. 4 and 5):

• Split vEPC (S-vEPC): The first type is an expansion of avEPC by separating conventional consolidated functionsinto UP and CP functions for dynamic scaling operations.

• Mobile hotspot network vEPC (MHN-vEPC): The othertype is an optimized case for a hotspot area to enhancethe agility of the network. For faster and more dynamicmobility management in a mobile hotspot area, the S1(single interface between the LTE radio access networkand the EPC) interface of the virtual EPC has beenmodified in terms of the UP and CP.

5. Management and Orchestration Architecture

Figure 6 shows overall MANO architecture on the EUside based on NFV MANO and an SDN. The architecturehas two management entities:

• The VNF manager is in charge of instantiating andcontrolling EPC functions. It is responsible for

Central data centerCloud packet core

Scale in & out

OAM Q&M

CP DPScale in &

out

OAM-VM

Control plane VM

Load balancer VM

Load balancer VM

Data plane VM

Shared data layer

Edge data center

CP DP

Scale in & outLB LB

LB

LBLB

Fig. 2. 5GTN vEPC functional architecture.

5G service managing Health SecurityEnergy Media Transport

IFA007

IFA005

IFA008 IFA008

VNF managerNFV VIM

Mobility management

Traffic monitoring

Traffic optimization

Service function chaining

High-availability control

SDN controller

Internet(KOREN/KREONET)

Radioaccess

wlan

MHNDistributed vEPC Distributed vEPC

MHN

wlan

RadioaccessApplications

5G core(CP)5G core(UP)

Applications5G core(CP)5G core(UP)

Orchestrator Inventory management

VNF on-boarding Provisioning MonitoringVNF/NS lifecycle management

Fig. 3. High-level architecture of Korea’s distributed vEPC.

UE eNodeB

eNodeB

HSS PCRF SPR

GTP-C(S11)

Diameter(S6a)

*SDP GTP-C(S5)

GTP-C(S5)

GTP-U(S1-U)

Diameter(Gx)

Diameter(Sp)

*SDP SGiPDN

X2-AP/GTP-U(X2)

S1-AP(S1-MME)

vEPCvMME

vSGW-CU

vSGW-DU

vPGW-CU

vPGW-DU

Fig. 4. HSvEPC functional architecture: S-vEPC.

mDUmTE

mRUmGW

(HEvEPC)

AP

AP

SwitchmRU

User

User

mNB

CPRI GbEmmWaveGigaWiFiaccess

User

······

GbE

mRU Access link(inside vehicle)

mRU

mDU mDU

mGW

mRUmRU

mTE

Intra-DUmobility controller

Inter-DUmobility support 5G core

network

···Backhaul link

: mmWavemGW:MHN gatewaymNB: MHN node BmDU: MHN digital unitmRU: MHN radio unit mTE: MHN terminal

equipment

e

IP

Fig. 5. HSvEPC functional architecture: MHN-vEPC.

MME HHS SGW/PGW-S

SDN-C (local)

vCompute vNetworkvStorage

Virtualization

Compute NetworkStorage

vCompute vNetworkvStorage

Virtualization

Compute NetworkStorage

VNF-1 VNF-2 VNF-3 VNF-1 VNF-2 VNF-3

Firewall DPI Video optimization

SDN-C(global)

OF N/W

Internet

MANO

OpenFlow

SDN-C (local)

Orchestrator

Infra manager

VNFM

Fig. 6. Overall EU MANO architecture with an SDN.



interacting with VNFs, chaining VNFs, and handlingtheir lifecycle—instantiation, maintenance, and others. Itis in charge of the operation and configuration of VNFsthrough the operations support system (OSS)/basestation subsystem (BSS). It will handle multifunctionalEPC components such as the MME and homesubscriber server (HSS) as well as specific-functionalityVNFs such as firewalls and deep packet inspectors.

• The infrastructure manager interacts with (or incorporatesthe capability of) the SDN controller in the servicestratum when deploying VNFs for configuring thecomputing and storage resources for the VNF of interest.It also supports the attachment of the VNFs to the borderof the underlying transport network for the networkingpart to make them reachable from outside the data center.This is only for the service-layer part. It also has todetermine a path for the transport-layer VNFs.The KR CN MANO are also based on NFV and SDN

components. Figure 7 shows the MANO architecture. Itcomprises three different entities: the NFV orchestrator(NFVO), VNF manager (VNFM), and virtualinfrastructure manager (VIM).The NFVO is responsible for managing functions such as

network service (NS) lifecycle management and overallresource management. Service management or orchestrationdeals with the creation and end-to-end management ofservices by composing different VNFs. Resourcemanagement helps to ensure that the NFV infrastructure(NFVI) resources are abstracted cleanly (independent of theVIM) to support the services that access these resources.The VNFM oversees the lifecycle (which typically

involves provisioning, scaling, and terminating)management of instances of a VNF. In this case, each VNFis associated with a VNFM that will manage that particularVNF’s lifecycle. A VNFM may manage multiple instancesof the same type of VNF or different types of VNFs.The VIM controls and manages the NFVI computing,

storage, and network resources. The VIM component hasbeen the focus of a large amount of research and variousopen-source solutions such as OpenStack and has beenused to realize the virtualized infrastructure managementfunctionality of MANO.

6. Autoscaling Based on Performance/FaultManagement

In our M&O, autoscaling functionality is provided asshown in Fig. 8. After instantiation of a 5G mobile CNservice, the NFVO sends a supervision request to thesupervisor, which performs performance monitoring andfault notification over virtualized resources and functions.Scaling is conducted autonomously by the orchestrator onthe basis of the information provided by the supervisor.

7. Automation by Event Chaining

An event chaining process is another importantfunctionality that is supported, which is defined as asequence of event units occurring from inside or outsidethe target VNF and virtual data unit (VDU). It enables theautomation of 5G mobile CN management. Acombination of internal events that are significant in asingle VNF or VDU and external events between VNFsand VDUs enables the automated management of alifecycle of a mobile CN service (see Fig. 9).

Designer(NFV graphic

designer)

LCM(life cycle manager)

For VNF and other SW

VM(VNFC)

MONITORING(OMW)

ETSI ISG NFV

Or-V

i

Vi-Vnfm

Or-Vnfm

Ve-Vnfm-vnf

Ve-Vnfm-em

Os-Ma-nfvo

Nf-Vi

Generic VNFM

VIM

TACKER based

Designer

Or-

Vi

RESTfulJSON structure

Public VNFMmade by INSOFT

RESTfulstructure

JSON

RESTfulstructureYAML

*LCA

JSONhttp

*MAxml

socket

NFVO

VNFM

VIM

RESTful

VIM

OPENSTACK

NFVO

VIM DRIVER

Fig. 7. Overall KR M&O architecture with MANO and an SDN.

NFVO

Interface Monitoring

OpenStack infra.

VM &VNF

Monitoringagent

Monitoringagent

Monitoringcollector

VIMOpenStack

2) Instantiation 3) ProvisioningPre.) set aggregate PM/FM and each indicates

1) Send aggregated alarm to NFVO

VDUprocess

OS

2) L

ook

up

NS /

VN

F

3) VNF/VNFC

Performanceinterface

FaultinterfaceIndicateinterfacePM/FM alarm

interface

Performanceinterface

FaultinterfaceIndicateinterface

Alarminterface

Fig. 8. MANO autoscaling process.

Event chain(for automation)

OrderInternal

Internal

VNF1

Event chain(for automation)

VNF2

Order

External(like a

function)

Internal

Internal

Reference

Event(for setting internal)

Event(for setting outside

information)

The event needs information to set

internal config

External(like a

function)

External(like a

function)

Reverseexternal

Fig. 9. MANO automation process.




8. Security Management Architecture

The security of the CN can be grouped into two parts:the security of the CN elements and the security of thecommunication channels in the CN. In SDNs, controllingthe behavior and interworking of different heterogeneousnetworks is carried out with a logically centralized controlarchitecture that has a global view of all forwardingelements. An operating system maps the entire network toservices and applications that are implemented on top ofthe control plane. Hence, security services will beimplemented as security applications using the networkstats provided either proactively or reactively by thenetwork control platform. Centralized control, which canbe either logically or physically centralized, enables theprogrammability of the network and will thus providefine-grained network security control, remote monitoring,and dynamic security service insertion. The securitymanagement architecture is presented in Fig. 10.

IV. Implementation and Deployment Experience

Both the EU and KR edge and CN functions are underdevelopment. The development of some components hasbeen completed, such as EU’s edge and core functions in a5GTN solution. KR’s vEPC development is underwaywith the core functionality completed. The KR vEPCcurrently supports up to 100 UEs and a channelthroughput of 20 Gbps toward an eNB. To meet the 5Gkey performance indicator (KPIs), we are trying to fill thegaps in both vEPC systems. We are targeting thecompletion of our system development by October 2017.We are also developing our agile CN MANO systems

based on the architectures described in Section II. Initialprototypes are available, and their functionality as separatesystems and their interoperability are being tested as well

across the EU and KR over interconnected research anddevelopment networks between the EU and KR via theKorea Research and Education Network (KOREN)–TransEurasia Information Network (TEIN)–Nordic CountriesNetwork (NORDUNET)–Finnish University and ResearchNetwork (FUNET).

1. vEPC Implementation

5GTN vEPC VNF functions have been implemented,deployed, and tested on CloudBand’s NFVI and itsMANO solution. CloudBand is a hardened, production-ready NFV solution based on OpenStack and other open-source technologies. This open approach allows serviceproviders to benefit from a vast community of engineersand supports investments in a mainstream solution withopen interfaces.The HSvEPC implementation consists of a vMME,

vSGW-CU, vPGW-CU, vSGW-DU, and vPGW-DU. TheCU is a CP that controls the device management, and thedata unit (DU) is a UP that controls the data transferbetween devices. The main reason why we separatedfunctions by each plane is to support scalabilitydepending on the demand situation. Since the functionsin the HSvEPC are implemented as VNFs, they can bemodified on demand and controlled per VNF level. Oneimportant use case of such flexibility is network slicingsupport.Figure 11 shows the access point name (APN)-based CN

slicing use case. An IoT device may have a different APNagainst a UE, and discrimination of each device at the MMEis required. The above use case illustrates ourimplementation of an MME that can classify differentdevices by categorization based on their APNs and mapappropriate resources in the SGW and PGW. Moreover, theHS-vEPC can be scaled in or out depending on the demand,which can reduce the cost, and other unused parts of networkfunctions can be relocated to only the necessary parts.

Network administrator

Security policy

Application plane

Control plane

Data plane

Network users Data path elements

OF switches Sec.

MiddleBox

Controller 1 Controller 2 Controller 3App-specific

API

SDN (OpenFlow) controllersNorth-bound API

South-bound API (OpenFlow protocol)

Security analytics Security

monitoring

Authen. & author.

Access control

DPI

SDN (OpenFlow) applications

Fig. 10. SDN architecture showing the security services andtheir deployment.

IoTdevice

PCRF/SPR

eNodeB

MME

SGW PGW Internet

HSS

IMS

eNodeB IoT-SGW IoT-PGW NB IoTUE

IoT-MME

APN: iot.xxx.com

APN: lte.xxx.com, ims.xxx.com

Fig. 11. HSvEPC core slicing use case.



2. Management and Orchestration Implementation

MANO in 5GTN has been implemented and deployed.It consists of CloudBand infrastructure software, aCloudBand application manager, and a CloudBandnetwork director that have been optimized to fit the keyNFV MANO shown in Fig. 12.

• CloudBand Infrastructure SoftwareThe CloudBand infrastructure software is a multipurposeNFVI and VIM. It virtualizes and manages computing,storage, and network resources.

• CloudBand Application ManagerThe CloudBand application manager is a VNFM thatautomates lifecycle management actions by managingresources and applying associated workflows.

• CloudBand Network DirectorThe CloudBand network director is an NFV resource andNS orchestrator. It manages virtual resources acrossgeodistributed NFV infrastructure nodes. It visualizes andautomates the lifecycle of NSs, such as virtual customerpremise equipment (CPE), including their forwardinggraphs and service chains.The KR MANO implementation is shown in Fig. 13.

We have implemented it in a rack of servers consisting ofa VIM built and extended over OpenStack, a VNF

manger, and an orchestrator. The management target is, ofcourse, a set of virtual functions implementing CNfunctionality and networks that interconnect those virtualcore functions.

3. Deployment and EU–KR Interoperability Testing

First, phase field deployment and interoperability testingbetween the EU and KR was conducted in July 2017 [13].Both the EU and KR vEPC and MANO have beendeployed. Figure 14 shows the EU’s 5GTN deploymentnetwork environment. The 5GTN network elements inOulu are physically located at two different sites. TheeNBs and Juniper SRX240 router are located at Site 1.Juniper SRX240 connects external entities to 5GTN. Datacenter Tampere in Oulu is connected via a layer 2 virtualprivate network (L2VPN) connection. KR entities are alsoconnected using an L2VPN connection. The EPC isphysically located at Site 2 and connected throughUniversity of Oulu (UOulu) switches to a radio accessnetwork.Figure 15 shows the 5GTN vEPC AirFrame hardware.

The European testbed vEPC has 11 servers, of which threeservers are used as controller nodes and eight servers areused as compute nodes; one hardware (HW) managementswitch; and two leaf switches.

Serv

ices

Element/network managementNetAct 3rd party5620 SAM

VNFsNokia VNFs 3rd party VNF

Service management/BSSNokia software portfolio 3rd party

Other cloud stacks

3rd party

Hardware infrastructureAirFrame 3rd party

CloudBand network director

CloudBand application manager

3rd partySDN

Nuage networks

OpenStack

CloudBand infrastructure

software

Network service orchestration resource orchestration and optimization aggregated view, distributed infrastructures

Automated life cycle execution, rapid onboarding, 40+ VNFs on boarded

Carrier-grade, pre-configured, highly available software on different hardware configurations, blueprinting, automated operations

Fig. 12. 5GTN MANO implementation.

OSS interface Integrated management GUIAccess control

NS configuration management NS

performance & fault

managementInfrastructure

integrated management

NS control

WAN infrastructuremanagement

Infrastructure resource management

VNFM interface VIM interface WIM interface

NFVO

Generic VNFM

Specific VNFM

VNFM

OpenStackVIM

Or-Vnfm

Vi-Vnfm

VNF

vBYOD VNFvScreen

VNFsvIDS vIPS vADC

Or-Vi

Nf-ViHypervisor (KVM)

Compute Storage NetworkNFVI

VIM1

VIM2

VIM/VNFM

Fig. 13. KR MANO implementation. Fig. 15. 5GTN EPC AirFrame hardware.

HSS PCFRDNS

DC tampere (Nokia)

UOulunetwork S-GW

P-GWMME

Nokia AirFrame CORE

JuniperQFX5100

S1-MMES1-US6aSGi

UOulu UOulu/CWCUOulu site 2

Internet

5GTN VLAN

UOulu site 1

JuniperSRX240

CWC

1 Gbit

1 Gbit 1 Gbit

1 Gbit 10Gbit

40Gbit

Dedicated connection Access, cloud core, services…FUNET → NORDUNET→ Geant

→ TEIN → KOREN Korean testbed

1 Gbit

Fig. 14. EU 5GTN deployment environment.




• Server type: 11 9 Quanta B51BP-1U, manufactured byQuanta Computer Inc.

• HW management switch type: 1 9 Quanta LB9,manufactured by Quanta Computer Inc.

• Leaf switch type: 2 9 Juniper QFX5100-24Q switches,each having 2 9 QFX-EM-4Q expansion modules,manufactured by Juniper Networks Inc.The three uppermost servers are controller nodes. The

remaining eight servers are computing nodes. Themanagement switch is below the servers, and the leafswitches are below the management switch.Figure 16 shows the Korean vEPC and MANO

deployment environment. There are three possible PoPs inKR interconnected over KOREN: Seoul, Daejeon, andGangneung, where the PyeongChang Winter OlympicGames take place in 2018. We plan to deploy mobilecore infrastructure for 5G networks at these three sitesfor service deployment. Currently, we deployed oneset of a vEPC and MANO at the Electronics andTelecommunications Research Institute (ETRI) in Daejeonfor interoperability testing with the EU.Our vEPC supports NS provisioning and monitoring

functionality as follows:

• The vEPC NS consists of an MME, a virtual S-GWcontrol unit (S-GW-CU), a virtual S-GW-DU, a virtualP-GW-CU, and a virtual P-GW-DU.

• In our deployment, the vEPC NS does not cover theremaining functionalities for the vEPC (that is, the HSSand PCRF).

• The virtual S-GW-DU and virtual P-GW-DU must havesingle-root input/output virtualization and sharing (SR-IOV)-enabled ports in order to enhance theirperformance.The NFVO, VNFM, and VIM closely interwork with

each other to create and manage NSs. Figure 17 shows theprocedures to provision an NS in the NFV-enabledinfrastructure.

1. An OSS/BSS (or administrators) requests to create anNS at the NFVO by defining a new NS descriptor orselecting one.2. The NFVO requests the allocation of network resourcesat the VIM, which connects the VNFs composing therequested NS. In this step, management network resourcesare also created for management access.3. Once the network resources are allocated, the NFVOrequests the VNFM to instantiate VNFs. Since our NFVIis in the indirect mode, the VNFM indirectly requests theVNF resource allocation at the NFVO, and the thenrequest is sent to the VIM.4. When VNF resources are allocated, the VNFMconfigures VNFs with any VNF-specific parameters.The states of the NSs are monitored with two metrics: a

service utilization metric and a metric for monitoringresource utilization. The NFVO receives the monitoringresults from the VNFM and VIM and exploits the resultsto perform other management operations such as a scalingoperation.Figure 18 shows the two types of monitoring.

• VNF monitoring: VNF providers can specify someindication of VNF behavior, and they include thisinformation as a parameter (that is, VnfIndicator) of the

PTN node

OSS/BSS

EMS EMS EMSVNF VNF VNF

NFVO

VNFM

VIM

PTN node PTN

node

VIMNFVI

NFVIPTN tunnel

Gangneung

ETRI

Daejeon PoP

(@Daejeon)

KOREN NOC(@Seoul)

Seoul/Gangneung PoP

Computing resources

Networking resources

Storage resources

Computing resources


Storage resources

Fig. 16. KR vEPC and MANO deployment environment.

OSS/BSS

EMS

NFVO

VNFM

VIMNFVI

EMS EMS EMS

NS

Usertraffic

4. Configure VNFs

1. Instantiate NS

3. Allocate VNFresources

2. Allocate network resource

Computing resources


Storage resources

Fig. 17. NS provisioning procedures.

OSS/BSS

EMS

NFVO

VNFM

VIMNFVI

EMS EMS EMS

NS

Usertraffic

VNF monitoring

Virtual resources

monitoringComputing resources


Storage resources

Fig. 18. VNF and virtual resource monitoring.



VNF descriptor. On the basis of this parameter, the VNFMrequests the actual value of a given indicator from theVNFs.

• Virtual resource monitoring: The VIM continuouslymonitors the allocated virtualized resources such as virtualcomputing, virtual storage, and virtual networking.For the preparation of an end-to-end 5G service

demonstration between the EU and KR, we performedinteroperability testing between two CNs as a first step.We are planning to conduct an end-to-end interoperabilitytest including mobile access networks on both sides byNovember 2017, and the results will be described in afuture version of this paper.For CN interoperability testing, we defined the

following two scenarios:

• Scenario 1: There are two users—one connected to theEU vEPC and the other to the KR vEPC. Content isshared between the two users, which is a latency criticalapplication such as shared gaming.

• Scenario 2: In this scenario, a mobile UE on the KR sideis the content provider and is streaming 4K three-dimensional videos to a receiving UE on the EU side. Theaim is to achieve very high data rates across the twovEPCs.For the first phase, we conducted a loose interoperability

test, defined as follows:

• Standard PDN interconnection via IP.

• A dedicated tunnel between the EU and KR test bed,which will provide guaranteed bandwidth and latency.This will ensure that the QoS requirements of the twouse cases are guaranteed.

• A model similar to the DiffServ model to guarantee theQoS. This model must be capable of providing 5Gstandard QoS. The details of such a model need to beworked out further.

• A reachable fixed IP- or DNS-based system, dependingon the actual applications for both use-case scenariosdefined.

• Support for dual stacks. Both IP version 4 and version 6will be supported.

• Dynamic routing protocols (open shortest path first(OSPF), border gateway protocol (BGP)) for advertisingthe PGW IP address to the external network.

• An application server placed strategically between thetwo cores, which will enable the execution of commonapplications such as games with low latencies. Theconnections to and from these servers will also have aguaranteed QoS.The EU–KR interconnectivity is shown in Fig. 19. The

EU–KR dedicated interconnectivity is implemented usingan L2VPN. A dedicated L2VPN connection path is UOulu

, FUNET , NORDUNET , Geant Open , TEIN ,KOREN, ETRI.

4. Dynamic Interoperability Provisioning

The key benefit of an SDN/NFV-enabled mobile corearchitecture is its ability to dynamically adapt requiredresources to the changing context and environment. To takefull advantage of such capabilities, the NFVI on which theEU and KR vEPCs are deployed are (partly) under thecontrol of the same NFVO. Figure 20 illustrates the resultingnetwork architecture, where the common NFVO overseesone or more PoPs, each managed by their own VIM, as wellas the interconnecting wide area network (WAN) managedby its WAN infrastructure manager (WIM).In the static scenario, the NFVO receives an NS request

to deploy the EU and KR vEPCs on their respective PoPsas well as their interconnection via the WAN. As a result,the NFVO will instruct the VIMs to instantiate therequired network function instances as well as the WIM toset up the interconnection.A more advanced and dynamic scenario involves the

dynamic reprovisioning of the interconnected bearers aswell as that of the underlying VNF resources to fulfill thenecessary QoS requirements. This scenario is depicted inFig. 20. In this scenario, the NFVO is used for staticprovisioning of different parts of the mobile coreinteroperability setup and for the dynamic reprovisioning ofthis NS based on monitoring components (see previoussection) as well as other external triggering systems (OSS/BSS or services). These components might, for example,trigger the scale-out of the P-GW-U (1a) or the migration ofthe S-GW (1b) VNFs. Note that the monitoring componentsare not necessarily directly interacting with the NFVO butare usually relying on the interaction of the managementfunctionality of the associated VNF (VNFM) or services. Asa result, the NFVO will (re-)instruct the corresponding VIMsand WIM(s) to instantiate new VNFs (indicated in dark

Interconnection networkCore network Core networkRAN RAN

PCPCBearer Bearer

5GTN PoP Korean PoP

Public internet

jPerf GUI (client)

iPerf server/ 4K video feed

Gangneung city(IoT street)

Daejeon city (ETRI)Oulu (UOULU)

S-GWU

P-GWU

S-GWC

P-GWC

P-GWU

P-GWC

S-GWU

S-GWC

MMEHSSPCRFPCRFHSSMME

FUNET

NORDUNET

Geant

TEIN

KOREN

MPLS L2VPN

Fig. 19. EU-KR interconnectivity.




blue) and rewire the associated network connectivity viathe WIM(s). Future work will refine this process anddetermine the degree of dynamics that will be implementedfor the considered project scenarios and associateddemonstrations.

5. Monitoring

In Sections IV.3 and 4, we described the differentinteroperability scenarios and the necessary steps toinitiate a new connection. We also noted that it is notsufficient to create a connection based on availableresources, but the monitoring of allocated resources isnecessary. By obtaining real information about, forexample, the latency or bandwidth, one can tune the QoSparameters to better align with the application session’srequirements.One of the most trivial metrics to measure is the end-to-

end delay and bandwidth of the newly created path.Depending on the chosen application architecture, one hasto monitor the links between the UEs and the applicationserver or between the two UEs. In case of a client–serverarchitecture, the application server can initiate activemeasurements, or it can passively capture the behavior ofan underlying protocol such as the transmission controlprotocol (TCP) window size and round-trip time (RTT).We have the same possibilities with the point-to-pointarchitecture, except that the UE executes the monitoringapplication. Such an application is the easiest solution, butit would place an unnecessary load on the UE. Moreover,we cannot infer the causes of any quality degradation bymeasuring end-to-end metrics.To overcome these difficulties, we can extend the

vEPCs with monitoring functions or use the existing onesif there are any. One can also install dedicated switchesbetween the serving and PDN gateways in both vEPCs tomonitor the application flows passively. As the traffic isIP-based, we can use OpenFlow switches or even

NetFlow-supporting ones. Moreover, accurate delaymeasurements require clock synchronization between themonitoring nodes. The same synchronization is necessaryfor active measurements, where probes instead of switchesperform the monitoring. These probes have to be aware ofthe properties of the flows in order to inject traffic into thebearers. Therefore, the application server must informthem about the newly created connection.Besides the monitoring of UP traffic, one can also

capture the control traffic (green marker in Fig. 21).Observing the connection setup messages between theMME and the eNB, we can derive the time required forthe initial attachment or a handover. The control messagesbetween the gateways provide information about theduration of the network-initiated (that is, on the request ofan application function) connection setup. Most likely,these setup times have no or little effect on the overall userexperience, but they can inform us about potentialslowdowns. In case of a burst in the number of users, forexample, the application server should refuse some of theconnection requests when it experiences increased setuptimes at one or both of the vEPCs.In addition to the UP and CP traffic as a good indicator of

the performance, we can use also the central processing unit(CPU) usage and memory consumption of the networkelements. Monitoring the CPU usage of the MME, we canforecast system slowdowns, as discussed in the previousparagraph, from the control traffic. A high resource usage atthe gateways in one of the vEPCs indicates failing QoSrequirements, and one can proactively redistribute theresources between the two vEPCs.The monitoring procedures presented so far handle the

functional blocks of the vEPCs as a black box software.They do not require any domain-specific knowledge aboutthe inner workings of the mobile cores, nor do they useany API possible provided by the vendor of the systems(see Fig. 22). Such an API could give us informationabout the number of active connections and the number of

1a. Scale-out trigger 1b. Migrate trigger

2a. Instantiate new P-GW U 2b. Migrate S-GW U

2c. R

e-co

nfig

ure

WA

N c

onne

ctio

n

Dedicated interconnection

Application function

NFVO

MME

S-GW U

S-GW C

P-GW U

P-GW C

HSSPCRF

AP1

S-GW C

P-GW U

P-GW C

PCRF

WAN MGT/CTRL

VIM 2VIM 1

S-GW U

P-GW U

MMEHSS

AP4

AP3

AP2

AP2

eNB

eNB

EU vEPC on PoP 1 KR vEPC on PoP 2

SDN-switch-based user plane

SDN-switch-based user plane

Fig. 20. Dynamic reprovisioning and NW-initiated bearer setup.

MME

S-GW U

S-GW C

P-GW U

P-GW C

HSS PCRF

Bearer

MMEHSSPCRF

Bearer

Korean core networkEuropean core network

AFP-GW

U

P-GW C

S-GW U

S-GW C

Dedicatedinterconnection

Public internet

eNB

Passive monitoring switch or active probe for user planePassive monitoring of the control plane

eNB

Fig. 21. Traffic monitoring locations at the two interworkingsites.



bearers or even indicate if some of the QoS requirementsare failing. Information from the MME could reveal thephysical location of the UE or at least the cell to which itconnects, which can help us to determine the initiallatency ratio.

V. Performance Evaluation and InteroperabilityTesting Results

During the deployment and interoperability testing, weobserved several important performance measures: thevEPC system performance, the end-to-end networkperformance between the two core systems, and theapplication performance.The HSvEPC provides a total channel throughput of

20 Gbps toward an eNB and can accommodate 100simultaneous UEs. The 5GTN vEPC also supports a totalchannel throughput of 20 Gbps toward an eNB and canaccommodate over 500 simultaneous UEs.The network that connects the two core systems

currently supports up to 1 Gbps, and there are plans toupgrade it to 10 Gbps by November 2017. We performedbandwidth throughput and delay tests on both the EU andKR sides, and the end-to-end context and the resultsobtained over this interconnection link are described asfollows [13].

1. EU CORE Integration and System Testing Results

A. EU’s UOulu Site Testing

Testing was performed with LTE access. A PC with anLTE Universal Serial Bus (USB) stick and with jPerf/iPerftools was used. The simplified test scenario is shown inFig. 23. The LTE band used is Band 7, and the bandwidthis 5 MHz.

• UOulu iPerf Testing with TCPWe performed iPerf testing with TCP traffic. The uplinkbandwidth was about 6.2 Mb/s. The performance was as

much as expected with the 5-MHz bandwidth. The iPerftool does not support downlink measurement when thereis network address translation (NAT) between the endpoints (NAT is carried out at the LTE USB stick).

• UOulu iPerf Testing with the user datagram protocol(UDP)

We also performed iPerf testing with UDP traffic. Theuplink bandwidth was about 11 Mb/s. The jitter was about1.6 ms. The performance was as much as expected withthe 5-MHz bandwidth. The iPerf tool does not supportdownlink measurement when there is NAT between theend points (NAT is carried out at the LTE USB stick).The RTT was measured using a ping test, as shown in

Fig. 24. The average was about 44 ms.

2. KR Core Integration and System Testing Results

A. KR’s ETRI Site Testing

The maximum achievable bandwidth during tested onthe 5G mobile core (5GMC) network was measured byiPerf. In this test scenario, the iPerf client was connectedto the iPerf server running on the PDN GW in the 5GMC.The simplified test scenario is shown in Fig. 25.

• iPerf Testing with TCPWe performed iPerf testing with TCP data streams.Figure 26 shows a screenshot from the iPerf client using

MME

S-GWU

S-GW C

P-GW U

P-GW C

HSS PCRF

Bearer

MMEHSSPCRF

Bearer

Korean core networkEuropean core network

AFP-GW

U

P-GW C

S-GW U

S-GW C

Dedicatedinterconnection

Public internet

eNB

CPU and memory usage monitor Vendor API or extension

eNB

Fig. 22. CPU and memory monitoring and vendor APIexposures.

PC/jPerf clienteNB

5GTN core and

EPC

iPerf server at 5GTN

cloud

Fig. 23. UOulu testing scenario with LTE access.

Fig. 24. UOulu ping test.

iPerf client

5GMC

iPerf server

Fig. 25. Testing without LTE connectivity.




jPerf, a graphical user interface front end for iPerf. Thedownlink bandwidth is about 941 Mb/s.

• iPerf Testing with UDPWe performed iPerf testing with UDP streams. Figure 27shows a screenshot of jPerf on the client side. The uplinkbandwidth is about 812 Mb/s. The downlink bandwidth isabout 910 Mb/s.

• Ping TestThe RTT was measured using a ping test, as shown inFig. 28. It was about 0.31 ms.

3. EU–KR Interconnection Integration and SystemTesting Results

A. EU ? KR Testing

Testing was performed with LTE access on the EU sideand on the ETRI side with LTE. A PC with an LTE USBstick with jPerf/iPerf tools was used. The simplified testscenario is shown in Fig. 29. The LTE band used is Band7, and the bandwidth is 5 MHz.

• EU–KR iPerf Testing with TCPWe performed iPerf testing with LTE access with TCPtraffic. The uplink bandwidth was about 9.31 Mb/s. The

performance was as much as expected with the 5-MHzbandwidth. The iPerf tool does not support downlinkmeasurement when there is NAT between the end points(NAT is carried out at the LTE USB stick). The bandwidthbetween the EU and KR at the CN is sufficient to obtainthis uplink bandwidth with LTE access.

• EU–KR iPerf Testing with UDPWe performed iPerf testing with LTE access with UDPtraffic. The uplink bandwidth was about 11.1 Mb/s. Thejitter was about 1.5 ms. The performance was as much asexpected with the 5-MHz bandwidth. The iPerf tool doesnot support downlink measurement when there is NATbetween the end points (NAT is carried out at the LTEUSB stick). The bandwidth between the EU and KR at theCN is sufficient to obtain this uplink bandwidth with LTEaccess.

• EU–KR Ping TestingThe RTT was measured using a ping test, as shown inFig. 30. The average was about 413 ms.

B. KR ? EU Testing

In order to test the performance of the interconnectionlink between the mobile CNs in KR and Europe, a client PC

Fig. 26. iPerf testing with TCP traffic: iPerf client view.

Fig. 27. iPerf testing with UDP traffic: client view.

Fig. 28. Ping testing with the ETRI server.

PC/jPerfclient

eNB5GTN

core and EPC

ETRI coreDedicated connection

iPerf server at ETRI(Korea)

FUNET → NORDUNET → Geant → TEIN → KOREN

Fig. 29. EU–KR testing scenario with LTE access.

Fig. 30. EU–KR ping test.



with jPerf/iPerf tools for initiating the test on the Koreanside was connected to the 5GMC, as shown in Fig. 31.

• KR-EU iPerf testing with TCPFigure 32 shows the results for iPerf testing with TCPtraffic. The green dotted line shows the uplink direction,and the blue line shows the downlink direction. Theuplink bandwidth was about 59.8 Mb/s. The downlinkbandwidth was about 37.8 Mb/s. A long RTT (over300 ms) affected the bandwidth with TCP traffic, whichhas delicate flow control and an error control mechanismas a connection-oriented transport protocol.

• KR–EU iPerf Testing with UDPWith iPerf testing with UDP traffic, both the uplink anddownlink bandwidths were measured to be about 812 Mb/s,similarly to the 1,470-b-sized UDP datagram traffic. Thejitter was 0.023 ms on average. The results encouraginglyshow almost full bandwidth considering that the maximumbandwidth between KR and the EU is 1 Gbps.

• KR–EU Ping TestingThe RTT was measured using a ping test. The average wasabout 304 ms.

C. 4K Video Demo

4K video streaming was demonstrated via our L2VPNdedicated connection and public internet access tocompare the quality. Video servers were located in KR.Video streaming used UDP transfer. The downlinkbandwidth could be verified by the computer’sperformance tools available from the “Task Manager.”With dedicated access, 4K video streaming showed very

good performance, and the downlink bandwidth used wasabout 60 Mb/s to 65 Mb/s. The bandwidth was less than10% of the total available bandwidth. However, 4K videostreaming via the public internet exhibited very badquality. The downlink bandwidth via the public internetwas about 8 Mb/s, which was not sufficient to obtain agood end-user experience.

4. Performance Evaluation of the 5G CN SecurityManagement Mechanism

In present mobile networks, IPsec tunneling and securitygateways are widely used to secure backhaulcommunication. We have worked on a novelcommunication architecture based on the HIP to secureboth the control and data channels in SDMNs. We aimedto analyze the added security features as well as theperformance penalty on both the control and datachannels inherent to the proposed simplified architecture(shown in Fig. 33). These performance penalties areconsidered in terms of the throughput, jitter, and latency.The key performance indicators in our performanceanalysis are [10].

• The performance penalty of security on the TCPthroughput

• The performance penalty of security on the UDPthroughput

• The latency introduced

• The performance penalty of security on the jitter

A. Performance Analysis of the Control Channel

In the first set of experiments, we analyze theperformance penalty of security on the SDMN controlchannel due to the proposed architecture.iPerf client

5GMC(Korea)

5GTN core

iPerf server

Dedicated connection

KOREN → TEIN → Geant → NORDUNET → FUNET

Fig. 31. EU–KR testing scenario.

Fig. 32. EU–KR iPerf testing with TCP traffic in the uplink.

POX controller

SecGW

Data channelControl channel

AttackerAttacker

LSALSA

Open vSwitch

Open vSwitch

Host 1 Host 2 Host 3 Host 4

Fig. 33. Testbed for the IPsec tunneling architecture for SDMNcommunication channels.




• Connection Establishment DelayIn the first experiment, we measure the connectionestablishment delay between Open vSwitch 1 and thePOX SDN controller under different scenarios. Here, weattempt to send a ping request from Host 1 to Host 2 andmeasure the connection establishment delay. Theexperimental results in Fig. 34 reveal that the proposedsecure architecture significantly increases (136%) thetunnel establishment delay. HIP tunnel establishmentbetween the local security authority (LSA) and the securegateway (SecGW) adds an extra delay to tunnelestablishment. However, the impact of this delay can beminimized by maintaining the established HIP tunnels fora long period. It is possible to maintain established HIPtunnels for long periods (that is, 15 min).

• Flow Table Update DelayIn the second experiment, we measure the delay to updatethe flow tables for a new packet flow during steady-stateoperation. In steady-state operation, the HIP tunnelsbetween the LSAs and the SecGW are already establishedand operational. Here, we ping from Host 1 to Host 2 andmeasure the RTT. The experimental results in Fig. 35reveal that the performance penalty of the proposed securearchitecture is less significant in steady-state operation.The extra IPsec encryption increases the flow update delayby only 2%. However, this delay can be further minimizedby using IPsec accelerators. IPsec acceleration is possibleby using external accelerators and/or using new AdvancedEncryption Standard instruction sets for processors.

B. Performance Analysis of the Data Channel

In the second set of experiments, we measure the TCPand UDP throughput performance of the data channel indifferent scenarios.

• Impact on the TCP ThroughputIn third experiment, we establish a TCP connectionbetween Host 1 and Host 3 to measure the TCPthroughput performance of data channel by using the iPerftool. The experimental results in Fig. 36 reveal thatthe proposed secure architecture decreases the TCPthroughput by only 2.3% compared to that of thenonsecure data channel. The extra layer of encryptiondecreases the TCP throughput.

• Impact on the UDP ThroughputIn fourth experiment, we establish a UDP connectionbetween Host 1 and Host 3 to measure the UDPthroughput performance of the data channel. Theexperimental results in Fig. 37 reveal that the proposedsecure architecture decreases the UDP throughput by only2.2% compared to that of the nonsecure data channel. Theextra layer of encryption decreases the UDP throughput.Moreover, the performance penalty of security on thethroughput is around 2% for both the UDP and TCPsessions compared with that of the nonsecure scenario.Thus, we can conclude that the performance penalty of

Number of attempts10 20 30 40 50 60 70 80 90 100

020

40

6080

120

140

160180

100

200

Rou

nd tr

ip ti

me

(ms)

OpenFlow with TLSv1Proposed control channelOpenFlow with TLSv1 (average with CI)Proposed control channel (average with CI)

Fig. 34. Connection establishment delay.

Number of attempts20 40 60 80 100

05

10

1520

30

35

4045

25

50

Rou

nd tr

ip ti

me

(ms)

OpenFlow with TLSv1Proposed control channelOpenFlow with TLSv1 (average with CI)Proposed control channel (average with CI)

0

Fig. 35. Flow table update delay.

Time (s)20 40 70 90 100

60

65

70

75

80

90

95

85

100

TCP

thro

ughp

ut (M

bps)

Without secure channelProposed secure channelWithout secure channel (average with CI)Proposed secure channel (average with CI)

0 10 30 60 8050

Fig. 36. Performance penalty on the TCP throughput.



security on the throughput is independent of the transportlayer protocol.

• Impact on the JitterIn fifth experiment, the jitter performance of a UDPsession between Host 1 and Host 3 is measured by usingthe iPerf tool. The experimental results in Fig. 38 revealthat the performance penalty of the secured architecture is41% relative to the nonsecure data channel. However, thejitter is still well below 500 ls (voice over IP (VoIP)requires a jitter below 4 ms), and the impact of jitter forreal-time applications such as VoIP and video streaming isless significant in a short-range network.

VI. Conclusion and Future Work

In this paper, we proposed an SDN/NFV-enrichedintelligent 5G CN and its agile MANO system to address5G KPIs. As details of the proposed system, we presentedthe architecture of the virtualized CN capabilities andits agile management. Furthermore, we shared ourimplementation, its deployment, and our interoperability

testing experiences with PoC use cases. As described, weare currently in the second phase of conformance andinteroperability testing of the proposed systemfunctionality. Our future work includes a performanceevaluation of the proposed solution in an end-to-end scope(UE-5G access-5G core–data center with applicationservers) in a PoC testing environment, which is thePyeongChang Olympic venue.

Acknowledgements

This work was supported by a grant from the Institutefor Information & Communications TechnologyPromotion (IITP) funded by the Korean government(MSIT) (No. B0115-16-0001, 5G Communication with aHeterogeneous, Agile Mobile Network in thePyeongChang Winter Olympic Competition) and theEuropean Union H2020 5GPPP under grant number723247.

References

[1] H. Shokri, C. Fischione, G. Fodor, P. Popovski, and M.

Zorzi, “Millimeter Wave Cellular Networks: A MAC LayerPerspective,” IEEE Trans. Commun., vol. 63, no. 10, Oct.

2015, pp. 3437–3458.[2] V. Desai, L. Krzymien, P. Sartori, W. Xiao, Z. Soong, and

A. Alkhateeb, “Initial Beamforming for mmWaveCommunications,” Asilomar Conf. Signals, Syst. Comput.,

Pacific Grove, CA, USA, Nov. 2–5, 2014, pp. 1926–1930.[3] E.C. Strinati and H.K. Chung, 5G CHAMPION, 2017. Accessed

Dec. 31, 2017. http://www.5g-champion.eu/

[4] G. Cross, Open Networking Forum SDN Standards, 2017.Accessed Nov. 15, 2017. https://www.opennetworking.

org/software-defined-standards/overview/[5] ITU-T SG13 TSB, ITU-T SG13 WP1 Q.21, 2017, Accessed

Nov 15, 2017. http://www.itu.int/en/ITU-T/studygroups/2017-2020/13/Pages/default.asp

[6] Linux Foundation Projects, ODL Open Source Project,2017, Accessed Nov. 15, 2017. https://www.opendaylight.

org/what-we-do/odl-platform-overview[7] G. Cross, ONOS Open Source Project, 2017, Accessed

Nov. 15, 2017. https://www.opennetworking.org/platforms/

onos/[8] J. Quittek et al., “Network Functions Virtualisation (NFV)

– Management and Orchestration V.1.1.1,” ETSI NFV ISG,Dec. 2014.

[9] R. Schuster et al., “An Introduction to the New ETSIIndustry Specification Group (ISG) for Mobile Edge

Computing (MEC),” ETSI MEC ISG, Oct. 2015.

Time (s) 20 40 70 90 100

60

65

70

75

80

90

95

85

100

UD

P th

roug

hput

(Mbp

s)

Without secure channel Proposed secure channel Without secure channel (average with CI)Proposed secure channel (average with CI)

0 10 30 60 80 50

Fig. 37. Performance penalty on the UDP throughput.

Number of attempts20 40 70 90 100

0

0.1

0.2

0.3

0.5

0.4

Jitte

r (s)

Without secure channelProposed secure channelWithout secure channel (average with CI)Proposed secure channel (average with CI)

0 10 30 60 8050

Fig. 38. Performance penalty on the jitter.



http://www.5g-champion.eu/

https://www.opennetworking.org/software-defined-standards/overview/

https://www.opennetworking.org/software-defined-standards/overview/

http://www.itu.int/en/ITU-T/studygroups/2017-2020/13/Pages/default.asp

http://www.itu.int/en/ITU-T/studygroups/2017-2020/13/Pages/default.asp

https://www.opendaylight.org/what-we-do/odl-platform-overview

https://www.opendaylight.org/what-we-do/odl-platform-overview

https://www.opennetworking.org/platforms/onos/

https://www.opennetworking.org/platforms/onos/


[10] J. Moilanen et al., “Operator Grade NFV-Based and SDN-Enriched EPC Environment at 5GTN (D4.1),” 5GCHAMPION Project, May 2017.

[11] M. Liyanage, A. Gurtov, and M. Ylianttila, “Software

Defined Mobile Networks (SDMN): Beyond LTE Network

Architecture,” Chichester, UK: John Wiley & Sons,2015.

[12] R. Banerjee et al., “5G CHAMPION Architecture, API-andInterface Document (D2.1),” 5GCHAMPION Project, Oct.

2016.[13] J. Moilanen et al., “VNF/SDN/EPC: Integration and System

Testing (D6.2),” 5GCHAMPION Project, June 2017.

Taesang Choi received his MS and PhDdegrees in computer science and

telecommunications from the University ofKansas City, MO, USA in 1988 and 1995,respectively. He joined the ETRI, Daejeon,

Rep. of Korea in 1996 and is currentlyworking as principal research staff. He has

been actively involved in the research and development of trafficengineering, traffic measurement and analysis, SDN/NFV

management, and 5G network slice management. He has alsoactively contributed to various SDOs and open-source activities

such as IETF, ITU-T, ONF, ONOS, and others. He is currentlyacting as an ITU-T SG13 Question 6 Rapporteur and InternationalIT Standardization Expert representing the Rep. of Korea.

TaeYeon Kim received his PhD degree in

computer science from Chungbuk NationalUniversity, Chungju, Rep. of Korea in2007. He also received BS and MS

degrees from Chung-Ang University,Seoul, Rep. of Korea in 1990 and 1992,

respectively. He joined the ETRI, Daejeon,Rep. of Korea in 1992. His current research includes network

and computing convergence platforms and SDN and NFVtechnologies for future networks.

Wouter Tavernier received his BS andMS degrees in computer science in 2002

from Ghent University, Belgium. Hejoined the Internet-Based Communications

Networks group (which became part ofIDLab in October 2016) of Ghent

University in 2006 as researcher on CarrierEthernet. In 2012, he obtained a PhD degree from the sameuniversity on reliable routing and switching. Currently he is

employed as a professor at Ghent University. His currentresearch interests focus on the performance aspects of software-

defined networks and network function virtualization. This workis performed in the context of European projects such as H2020

5G-CHAMPION, SONATA-NFV, and 5G TANGO. Thisresearch has been published in more than 50 scientific

publications.

Aki Korvala received his BS degree from

the Technical Institute of Oulu,Department of Electrical Engineering, in

1997. He has worked at Nokia for18 years in various R&D positions withinmobile phones and network business lines.

Currently, he is working in the 5G area asa program manager. This work is performed in the context of

European projects such as H2020 5G-CHAMPION.

Jussi Pajunp€a€a has worked at Nokia

Networks, Espoo, Finland for 19 years invarious R&D positions in software and

systems engineering in the core networkdomain. Currently, he is working with the

Telco Cloud and virtual network functionarchitecture as a chief architect and R&D

manager and contributing to the 5G test network activities inOulu.


Documents

Agile Management and Interoperability Testing of SDN/NFV ... · reference points, leveraging open-source PoC projects and communities to drive open standards of NFV. In 2016, it published