Building Carrier-Grade On-Device Network Management SystemsAn Enterprise Management Associates White Paper October 2007

Table of Contents

Introduction..........................................................................................................................................................................................1

New.Services.Increase.Network.Complexity.................................................................................................................................1

Enterprise.and.Carrier.Requirements.are.Converging..................................................................................................................1

Network.Management.Design.Challenges......................................................................................................................................1

High.Availability...................................................................................................................................................................................1

Scalability.and.Performance...............................................................................................................................................................2

Security...................................................................................................................................................................................................2

Fine-grained.Control.and.Logging...................................................................................................................................................3

Data.Consistency.and.Integrity.........................................................................................................................................................3

Change.Automation.............................................................................................................................................................................4

Rapid.Development.............................................................................................................................................................................5

Conclusion.............................................................................................................................................................................................5

Building Carrier-Grade On-Device Network Management Systems

Page �

©2007.Enterprise.Management.Associates,.Inc..All.Rights.Reserved.

IntroductionThe.key.issues.involved.in.developing.on-device.network.management. systems. for. carrier-grade. enterprise. and.operator.networks.are.the.same.for.any.other.networked.resource,.only.with.more.complexity.and.built-in.restric-tions.due.to.the.limited.space.and.processing.capabilities.inherent.to.these.types.of .devices..The.requirements.for.on-device.network.management.systems.will.be.reviewed.in.the.context.of .the.business.and.technical.expectations.of .network.operators.that.drive.these.requirements.and.their. ever. changing. network. environmental. demands..Increasing.levels.of .complexity.and.converging.industry.requirements.are.the.key.factors.that.are.continuing.to.shape.this.unique.industry.segment.

New Services Increase Network Complexity Modern.networks.have.become.increasingly.sophisticat-ed.platforms.for.delivering.data,.voice,.video,.and.wire-less.services..Network.management.systems.must.adapt.to.meet.new.challenges..The.old.paradigm.of .humans.administering.networks.with.a.“set.and.forget”.approach.is.long.gone..Customer.expectations.for.network.perfor-mance.have.never.been.higher..Users.want.applications.and services available in their homes, offices, and every-where. between. –. without. interruption. or. limitations..Network.operators.and.equipment.vendors.are.working.hard.to.make.this.a.reality..

Competitive.and.economic.forces.are.also.affecting.the.networking. industry.. While. the. cost. for. the. network.equipment. itself . has. been. squeezed. over. many. years,.the.cost.for.deploying.new.services.and.maintaining.the.networks. has. increased. dramatically.. Service. providers.and. enterprises. are. constantly. looking. to. reduce. the.operating.cost.of .their.networks.as.a.means.to.become.more.competitive.

Enterprise and Carrier Requirements are ConvergingInterestingly,.the.challenges.facing.CIOs.of .enterprises.often. mirror. those. of . commercial. service. providers..Larger.enterprise.networks.have.many.similarities.with.service provider networks. For example, the “five-nines” expectation. for. telco. equipment.has.now.crossed. into.the. enterprise. as. Voice. over. Internet. Protocol. (VoIP).penetration. explodes.. Users. rightly. expect. a. dial. tone.

every.time.they.pick.up.the.phone.whether.it.comes.over.a.Time-Division.Multiplexing.(TDM).or.an.IP.network.

Network Management Design ChallengesThe. developers. of . networking. equipment. and. their.network.management.systems.must.meet.or.exceed.the.expectations. of . network. operators. in. an. environment.that.is.increasingly.complicated.

The.number.of .network.devices.and.their.form.factors.are.proliferating..In.addition.to.appliances.and.Advanced.Telecommunications. Computing. Architecture. (ATCA).compliant. chassis. products,. blade. servers. are. being.increasingly. adopted. for. networking. applications.. The.growth.in.the.number.of .network.devices.requires.mul-tiple devices to be configured and supported for flexible high.availability.frameworks.

The. number. and. complexity. of . required. northbound.management.interfaces.is.also.increasing..In.addition.to.simple.network.management.protocol.(SNMP).and.com-mand.line.interface.(CLI).agents,.web.and.NETCONF/Extensible.Markup.Language. (XML). interfaces. are. in-creasingly.expected.on.most.network.devices..All.these.management.interfaces.must.be.synchronized.to.ensure.consistency..Flexible.provisioning.and.frequent.re-con-figuration is needed to support multi-service networks. As. if . this. was. not. enough,. developers. of . networking.equipment.must.bring.complete.solutions.to.market.in.less. time,. with. constrained. resources,. and. often. using.distributed.teams.responsible.for.modular.components.of .the.overall.system.

High AvailabilityUp.until.recently,.network.management.was.not.high.on.a.network.operator’s.punch.list.for.ensuring.continuous.availability..However,.as.networks.become.dependent.on.frequent. software. updates. and. dynamic. service. provi-sioning,.this.is.changing..

Building.a.highly.available.network.management.appli-cation.requires.a.powerful.software.fabric.with.capabili-ties.including.database.replication,.support.for.master-slave.heart.beat,.full.redundancy,.hot.failover,.multiple.levels. of . security. access. controls,. complete. analytical.drill-down.functionality.and.of .course.a.full.line.of .re-porting.capabilities..

Building Carrier-Grade On-Device Network Management Systems

Page �

©2007.Enterprise.Management.Associates,.Inc..All.Rights.Reserved.

It.is.common.for.chassis-based.network.devices.to.have.a. separate. management. card. managing. individual. line.cards.as.shown.in.Figure.1..This.capability.requires.an.application.programming.interface.(API).to.make.it.pos-sible.for.the.management.cards.to.manage.resources.on.other.hosts.

Figure 1

Dual.management.cards.should.have.the.capability.to.be.configured to ensure high availability, master-slave heart beat, and full redundancy. All configuration changes are always. written. to. both. management. cards. and. when.re-configuring the master all data is also replicated to the.slave..Upon.a.failure.of .the.master,.the.system.will.switch.over. to. the. slave.and. the. slave.will.become.ac-tive. in.the.system..Some.form.of .alerting,.or.warning,.mechanism. is. necessary. to. notify. network. operations.staff .of .the.event.and.subsequent.change.in.operational.status. of . the. device.. Additionally,. drill-down. analytics.is.a.necessity.for.the.network.operations.staff .enabling.them.to.dive. into. the.heart.of .any.network.event.and.ascertain.the.true.cause.and.nature.of .the.event.

With.this.type.of .architecture.it.is.also.possible.to.upgrade.the.system.without.bringing.down.the.entire.service.or.device..To.deliver.redundancy.in.a.scalable.fashion,.data.replication.should.be.possible.from.a.single.master.to.N.number.of .slaves.where.N.is.not.arbitrarily.limited..Slave.units.or. redundant.devices. could.be.physically. located.anywhere.on.the.network..Control.of .the.master,.slaves,.and.redundant.devices.should.be.from.a.single.console.window..The.same.console.window.will.also.be.the.cen-tral alert / notification display mechanism for this type of .network.management.architecture.

Scalability and PerformanceCarrier-grade. network. management. applications. must.have the capability to monitor, provision, and configure very.large.networks.without.impacting.service.delivery.

The first problem is storing large numbers of configura-tion.parameters..In.a.large.complex.network.the.number.of configuration parameters can be in the millions. This implies.the.need.for.a.large,.robust,.real-time.data.store.that.can.scale.linearly.without.performance.degradation..The problem is much more difficult at the network de-vice. level.where.memory.and.processing.resources.are.limited.and.not.easily.expanded.without.additional.hard-ware.additions..The.best.solution.is.to.use.a.data.store.that is optimized for configuration and operational data as well as the types and database transactions specific to network.management..

The.second.issue.is.the.performance.and.scalability.of .the. transaction. engine. that. manages. the. communica-tion stream to end devices. To implement a configura-tion. change. in. a. large. network. implies. the. need. for. a.transaction.engine.that.can.scale.to.tens.of .thousands.of .operations.per.minute..The.growing.use.of .XML.data.for configuration management requires the data stream-ing.process.in.the.transaction.engine.be.highly.tuned.to.these.performance.needs.

In. order. to. address. growing. need. for. scalable. perfor-mance,. some.networking.products. are.based.on.blade.servers.and.stackable.appliances.as.a.means.of .delivering.scalable.performance..Both.these.approaches.share.the.benefit of having the capability to add capacity without disrupting.the.network..

The challenge is for the configuration management sys-tem.to.operate.without.any.dependency.on.the.hardware.configuration. For example, each blade in a cluster should be.unaware.of .the.fact.that.it. is.running.in.a.clustered.environment.. One. of . the. blades. must. be. dynamically.assigned. a. management. role. and. network. administra-tors.must.have.visibility. to.all.network.devices.as. they.are added on the fly. This later capability is sometimes referred.to.as.a.cluster.join.

SecuritySimple. password. protection. is. not. enough. for. a. car-rier-grade. network. management. system.. With. a. high.incidence.of . severe. threats. and. attacks.on. information.assets,.security.has.become.a.priority.at.the.highest.levels.inside.an.organization..In.addition.to.mitigating.threats.to.mission-critical.network.systems,.network.operators.and.enterprises.must.also.comply.with.a.wide.range.of .regula-

Building Carrier-Grade On-Device Network Management Systems

Page �

©2007.Enterprise.Management.Associates,.Inc..All.Rights.Reserved.

tions.that.require.them.to.implement.and.verify.the.effec-tiveness.of .security.information.management.controls.

The first step in providing appropriate security is user authentication..A.user.must.present.credentials,.such.as.a.password.or.a.public.key,.in.order.to.gain.access..The.ability.to.support.security.solutions.based.on.protocols.such. as. Remote. Authentication. Dial-In. User. Service.(RADIUS).and.Lightweight.Directory.Access.Protocol.(LDAP).is.an.important.option.for.any.good.authentica-tion.system..A.remote.authentication.server.will.typically.store.both. the.users’. login. credentials. and. their. group.information.that.can.be.applied.to.authorizing.their.ac-cess.rights.

The.second.step.in.the.security.model.is.for.authoriza-tion.to.be.performed..Even.though.a.set.of .credentials.are presented and confirmed as being acceptable, there’s still.the.question.of .what.actions,.tasks,.changes,.etc..are.to.be.allowed.under.the.scope.of .those.credentials..Once.a.user.is.properly.and.successfully.authenticated,.all.op-erations.performed.by.that.user.must.be.authorized.by.the.appropriate.access.control.source.inside.the.organi-zation. The internal access control source must confirm that. the.credentials.presented.are. in. fact.authorized. to.perform. the. operation. intended.. If . they. are,. then. the.intended.operation.is.allowed.to.be.performed..If .those.credentials.don’t.have.the.necessary.permissions.or.ac-cess.control.rights.to.perform.the.requested.operation,.then.the.request.is.denied.

Accounting.and.auditing.is.the.third.major.leg.of .the.traditional.AAA.(authentication,.authorization.and.ac-counting). services. needed. for. robust. security.. When.logins.fail,.access.controls.are.denied,.or.unauthorized.changes.are.attempted,.those.events.must.be.recorded.and. reported. to. the. appropriate. authorities..There. in.lies.the.need.for.strong,.robust,.reporting.and.auditing.capabilities. that.can.take.on.many.forms,. i.e..compli-ance.reports.(Sarbanes-Oxley,.HIPPA),.forensic.analy-sis,.and.billing.

Since. many. network. managements. systems. include. a.web. interface,. it. is. also. important. that. the. embedded.web.server.does.not.expose.the.system.to.security.vul-nerabilities.. Considerations. here. include. using. HTTPS.for. all. secure. communications. between. the. client. and.the.server.plus.avoiding.cross-site.scripting.and.caching.configuration values.

Fine-grained Control and LoggingThe ability to allow identified users or user groups to perform specific tasks is important in any large network and. is. the.corner. stone.of .a. role-based.system.of .ad-ministration. To provide true fine-grained control over which.users.or.groups.can.execute.particular.actions.re-quires.authorization.control.of .commands,.data.access.and.device.access.

Figure 2

As. illustrated. in.Figure.2,. role-based.access.control.al-lows.users.to.change.while.roles.remain.constant..When.responsibilities.for.network.administration.are.organized.by.seniority,.geography,.or. line.of .business,. role-based.access control is very beneficial.

Providers.of .managed.services.also.need.such.granular.control..Here. the.need. is. for.multi-tenanted. access. to.the.network.management.application..Individual.clients.of . a. managed. security. gateway,. for. example,. must. be.able.to.view.their.virtualized.security.policies.and.only.make. changes. to. rules. and. data. that. only. affect. their.organization.

Data Consistency and IntegrityIt.is.highly.advantageous.to.have.a.single.view,.or.win-dow,.of .all.sessions.in.all.the.northbound.interfaces..As.a. result,. access. rules. and. audit. trails. can.be.uniformly.applied.by.the.CLI.and.web.interfaces..To.ensure.data.consistency,. a. carrier-grade. network. management. sys-tem must guarantee that configuration changes made in.any.of .the.management.interfaces.are.either.entirely.

Building Carrier-Grade On-Device Network Management Systems

Page �

©2007.Enterprise.Management.Associates,.Inc..All.Rights.Reserved.

completed.or.aborted.and.properly.rolled.back–.thereby.ensuring that the integrity of the configuration, and ulti-mately.the.network,.is.maintained.

Figure 3

Traditionally,. network. management. applications. have.used.stovepipe.architecture.as.depicted.in.Figure.3..Here.each. management. interface. talks. to. managed. objects.using.its.own.software.adaptors.and.APIs..This.cumber-some.approach.not.only.strains.development.resources,.but also risks compromising configuration integrity.

Ideally,. backplane. software. should. set. up. a. transac-tion for every committed configuration change from northbound.interfaces..It.then.should.talk.to.all.affected.managed objects in a specific order, waiting for them to.acknowledge.that.the.change.has.been.accepted.and.duly. processed. and. successfully. completed.. If . any. of .the.managed.objects.returns.an.error.(e.g.,.failed.to.set.a specific configuration parameter in the routing stack), the.backplane.ensures.that.any.changes.made.up.to.that.point.are.completely.and.successfully.rolled.back..Figure.4 illustrates an architecture based on a single unified backplane.or.transaction.engine.

Figure 4

The.use.of .a.synchronous.subscription.API.plays.a.use-ful.role.in.ensuring.data.integrity..This.allows.managed.objects to receive a notification over a subscription sock-

et when configuration data changes. Each subscription should have a priority and the subscribers are notified in that.priority.order..

A Dynamic Host Configuration Protocol (DHCP) serv-er configuration change is an example of where ordered notifications are of significant value to the integrity of . the.network.as.a.whole.. If .both. the. IP.address.of .the.network.device,.as.well.as.some.additional.changes.to the DHCP server configuration are performed in a single configuration change, it is imperative that the DHCP configuration code receives the configuration change.after.the.IP.interface.manager.code.has.acted.on.the.change.and.changed.address.on.the.network.device..Otherwise, there is a potential conflict in the consistency and overall integrity of the configuration of network de-vices.versus.the.information.contained.in.any.particular.server or configuration source.

Change AutomationManaging.operating.expenses.is.critical.to.the.success.of .both.service.providers.and.IT.organizations.responsible.to. enterprise.networks..Network. administration. is.one.of . the.many.functions. that.make.up.the.planning,.de-ployment,.management,.and.maintenance.of .networks..Network administration costs are significant both as direct expenses and as a by-product of configuration errors. that.can.cause.service.outages.or.severe.perfor-mance impacts. Service disruptions carry a significant toll.in.terms.of .organizational.productivity,.negative.PR,.and.potential. customer.churn..Until. recently. the. alter-natives for configuration management such as manual configuration, CLI scripting, and SNMP faced multiple limitations.. For. a. complete. discussion. of . these. issues.read.Tail-f .Systems’.whitepaper.“Next Generation Network Management with NETCONF”.

The NETCONF standard for automated configuration was finalized by the Internet Engineering Task Force (IETF).in.December.of .2006.as.Requests.for.Comment.(RFC).4741-4744..This.Remote.Procedure.Call.(RPC)-based protocol uses XML data encoding for configu-ration. data. and. protocol. messages. that. are. exchanged.between.a.manager.and.an.agent..

NETCONF. includes. robust. features. to. ensure. that.configuration changes are made consistently across all network devices. For example, a configuration change will. only. be. committed. if . no. errors. occur.. If . errors.

Building Carrier-Grade On-Device Network Management Systems

Page �

©2007.Enterprise.Management.Associates,.Inc..All.Rights.Reserved.

do.occur.the.changes.will.be.automatically.rolled.back..This.is.illustrated.in.Figure.5..NETCONF.frees.network.management.applications.and.data.stores.from.the.over-head. and. complexity. of . dealing. with. data. consistency.safeguards.at.the.protocol.level.

Equipment.vendors. and.network.operators. are. adopt-ing. NETCONF. to. facilitate. scalable. deployments. of .networks without the risks of disruptive configuration errors..The.need.for.NETCONF.is.also.being.driven.by.the.inherent.complexity.of .networks.and.the.explosion.in. the. number. and. variety. of . network. devices.. Today.many.services.require.network.devices.to.be.changed.in.one.single.transaction.

Rapid DevelopmentNetwork. operators. are. under. pressure. to. quickly.bring. innovative.services. to. their.customers.and.users..Network.equipment.vendors.therefore.require.enabling.technologies.to.shorten.development.time.and.minimize.sustaining.engineering.overhead.

There. are. several. time-to-market. bottlenecks. affecting.network.management.software..Where.northbound.in-terfaces.are.not.tightly.coupled,.additions.and.changes.to.applications.and.managed.objects.are.multiplied.several.fold. as. each.management. interface.must.be. addressed.independently..

The. need. for. increased. code. integration. and. testing.among. management. interfaces. is. another. negative. by-product.of .a.stovepipe.architecture..Interfaces.between.

software.modules.usually.have.higher.than.average.bug.rates..In.addition,.an.uncoupled.architecture.undermines.the benefits of using distributed development teams by creating. interdependencies.at.every.stage.of .the.devel-opment.process..Ideally,.developers.share.the.same.data.store.of .managed.objects.and.that.data.store.allows.indi-vidual developers to add or delete items in the configura-tion.without.impacting.other.developers..But.at.the.same.time,.there’s.a.need.to.monitor.and.control.items.that.are.changed in the configuration.

The. time. to. develop. new. network. management. func-tionality.can.also.be.accelerated.by.integrating.new.func-tions.with.existing.applications.and.data.stores..Reusing.legacy.software.modules.has.trade-offs,.but.existence.of .well-defined APIs provides a practical option to pursue this.approach.vs..reinventing.everything.from.scratch..

ConclusionCarrier-grade.networks.are.larger.and.more.complex.than.ever.before,.and.growing.more.complex.all.the.time..The.software that monitors, configures, and controls these networks.must.be.designed.for.high.performance,.con-tinuous. service,. comprehensive. security,. low. cost. and.reliable operations. This is best achieved with a unified transaction. based. architecture. that. ensures. consistent.execution, fine-grained control, and rapid application development. This unified architecture will also need to be flexible to absorb the constant change brought about in. today’s.networking. environments.due. to. changes. in.mission.scope,.mergers.and.acquisitions,.as.well.as.the.

Figure 5

Building Carrier-Grade On-Device Network Management Systems

Page �

©2007.Enterprise.Management.Associates,.Inc..All.Rights.Reserved.

constant.application.and.network.performance.improve-ments.required.to.keep.pace.with.competition.

That flexibility will also need to extended to those areas where. the.applications,.or.network.devices,. interact.or.interface.with.other.platform.and.tool.architectures.al-ready.present. in. the.network.environment..The.ability.to.interact.and.play.with.other.applications.and.devices.in.today’s.market. is. imperative..Executive.management.wants.everything.to.work.in.a.homogenous.manner.so.that.past,.present,.and.future.investments.can.be.utilized.together without significant effort or loss of capability or.service.delivery.

Software.applications.and.network.devices.will.increas-ing.need. to. route. their.AAA.activities.and.operational.events.through.a.highly.reliable,.robust,.intelligent,.and.centralized.data.repository...A.centralized.data.repository.allows. for. a. consolidated. and. improved. analytic. func-tionality.to.be.provided.for.enhanced.true.cause.trouble-shooting. along. with. incorporating. a. robust. auditing.and. reporting. feature. set. to. aid. in. current. and. future.compliance.adoption..A.centralized.data.repository.also.provides. the. enterprise. or. telco. staff . with. a. complete.and.cohesive.view.into.their.network.environment.that.allows.everyone.to.see.the.same.information.at.the.same.time..Not.only.across.the.organization,.but.also.up.and.down.the.various.organizational.levels.as.well..

About Enterprise Management Associates, Inc.Enterprise Management Associates is an advisory and research firm providing market insight to solution providers and technology guidance to Fortune 1000.companies..The.EMA.team.is.composed.of .industry.respected.analysts.who.deliver.strategic.awareness.about.computing.and.communications.infrastructure..Coupling.this.team.of .experts.with.an.ever-expanding.knowledge.repository.gives.EMA.clients.an.unparalleled.advantage.against.their.competition. The firm has published hundreds of articles and books on technology management topics and is frequently requested to share their observations.at.management.forums.worldwide.

This.report.in.whole.or.in.part.may.not.be.duplicated,.reproduced,.stored.in.a.retrieval.system.or.retransmitted.without.prior.written.permission.of .Enterprise.Management.Associates,.Inc..All.opinions.and.estimates.herein.constitute.our.judgement.as.of .this.date.and.are.subject.to.change.without.notice..Product.names.mentioned.herein.may.be.trademarks.and/or.registered.trademarks.of .their.respective.companies.

©2007.Enterprise.Management.Associates,.Inc..All.Rights.Reserved.

Corporate Headquarters: 5777.Central.Avenue,.Suite.105.Boulder,.CO.80301.Phone:.+1.303.543.9500.Fax:.+1.303.543.7687.www.enterprisemanagement.com

1460.102207