NOC Theory and Practice

NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

NOC Theory and PracticeNOC Theory and Practice

Part II: The Nuts and Bolts(The Practice Of Making a

Good NOC)


DISCLAIMER

I am not affiliated with any corporation. My views are my own.


Intro - What’s In A NOC?

• Purpose - What Type of Company Are You?

• People will always be your key asset.• Systems make the job of people easier.• Managers assist in identifying gaps in both.• Processes allow for flexibility in operations

while still maintaining standards.• And, last but not least, the facility is where

you do your work.


The Facility - Think Data Center

• Location, location, location.• Security• Environmental Concerns

This applies to both your inside and outside environment.

• Ergonomics• Reliability• Sources of Labor


The Goals of a NOC

Our conception is:• At the bare minimum, you will receive

requests from your customers• You will process those requests in

some fashion (engineering, handoff, ...)

• You will communicate resolution to the customer, and/or keep customer updated on status


Goals II

• NOC as first, second, or third tier support. Conception of NOC as “do everything”

group The engineers should be smart at

designing, configuring, etc. - NOC should be divided into categories of engineers able to resolve issues

A smart issue routing process makes this a very effective model


The Process

• Receive events from your “inputs” (phone calls, network monitoring, etc)

• Process inputs into a problem to be resolved

• Route the problem to the correct individual

• Resolve the problem as appropriate• Communicate status continually.• Record the problem for later review and

training.


The Key to Success Is

By constant re-examination of your operation, your

employees, and what inputs you are getting, you can reach

incredible efficiency. Communicate this to other

teams and your operations will prosper.


The Facility (cont.)

• Security applies to keeping other people out, but it also applies to making a safe arrival for your employees, at any hour.

• Environmental concerns: Just like a datacenter, consider climate control and earthquake/act of god resistance. Disaster planning rules should be in effect.

• Sources of labor: Not just engineers, but builders.


A Word on Portability

• As Matt suggests, portability is critical.• Obvious resources: Computers,

communications, phones.• Not obvious resources: Vehicles and quick

transport/assembly, good cases for shipping or relocation, breakdown/reassembly schedules.

• Multiple everything.• Make sure your resources are distributed.

If your network is large, your footprint of mobility should be large.


Hiring the Right People

• Skillset but more importantly, flexibility

• Stress• Scheduling• Stability

NOCs are hard things to staff appropriately. And good people are hard to retain.


Hiring (cont.)

• Make sure they have lives. But know what you can’t ask.

• Can they talk? How are their soft skills?• Do they get flustered easily?

And remember, org charts suck. Classify people according to their skills, not according to some hierarchy which has no meaning outside of some middle-management mind.


Keeping Your People Happy

• Environment• Ergonomics• Creature Comforts

We’re not kidding.

Give some thought to a 24x7 outlook. This means beds, showers, food, and climate.


Environment - People

• Climate controlled centers get cold at night. Make sure you have 24H control over your space.

• When thinking about climate, keep in mind your working conditions are slightly different in a NOC.

• 2:00AM to 4:00AM are the critical hours.


Ergonomics - People

• Hire a good ergonomic consultant.• Positions of lights, phones, keyboards,

monitors, pens - all this matters.• Observation is key. Your engineers will

show you what the problems are. (Cameras)

• Seating. If you’re in a chair for eight hours at a stretch, don’t you want it to be a comfortable one?

• Screen space.


About People. About NOCs.

• It’s a messy business. It’s high stress. It’s easy to let things get out of hand.

• Good management qualities in a NOC: cool heads methodical approach to solving

problems having been on the customer side micromanagement sucks.


Let’s go over retention.

• Retention of NOC staff is 100% harder. Very few people live for this kind of job.

• So, innovate. Think outside the box.• People want to grow. Know what your people

want to do when they grow up.• Remember your core focus. Do operations,

not sales, or account services, or anything else.

• At the same time, make sure your folks are informed on everything. Everything comes out in the wash. All the dirty clothes come to NOC.


Interviewing NOC folks

• You are interested in three things. The candidate’s willingness to learn. The candidate’s 1-2 year outlook on their career. The candidate’s ability to work funny schedules.

• They are interested in three things. Will you offer them growth potential? Can you give them enough perks to enjoy their work? Do you have your stuff together?

Are the procedures well-documented? Will they be called in outside their shift? Will they ever wonder who to go to?


Productivity

• Momentum is key. Keeping your people busy is the most important thing towards maintaining productivity.

• The other component to momentum is keeping them working on different things. This makes them better engineers, and helps feed ego.

• “Everything at your fingertips” - your engineers should never have to make a customer wait more than four minutes on hold for anything. If it has to wait longer than that, tell the customer why, and mark it as something to fix.


The View of the Customer

• A customer wants Professionalism: your engineers must treat

the customer as if they are paying you a lot of money and the engineer wants the customer to continue to do that.

ETR: Your customer wants to know when the problem will be fixed. “I don’t know” is not an acceptable answer. The ETR is the first thing you give to a customer and the one thing you update him on at least every hour (or less if they prefer).


The View of the Customer II

A description: Concise or not, the customer wants to know what the problem appears to be.

Remember your focus. It is not (typically) your responsibility to engineer your customer’s network for maximum reliability. Your responsibility is the operation of that network. The only thing you should do as a NOC is fix problems when they arise in the most expedient and professional manner possible. A customer wants to be comfortable, and assured that that will happen.


Things we never, EVER allow a customer to hear

• I don’t know when it will be fixed.

• I can’t help you.

• I’m not qualified to answer that question.

• Let me transfer you to someone else.


It’s not always ideal. But -

• Where possible, the first person to take the call should be the person who follows it to completion. The customer hates being thrown between different engineers - but not having an answer is worse.


A Little About Systems

• Don’t just focus on ticketing.• Make sure your people have

information on everything they need to operate.

• If all the architects fell off the planet today, could you rebuild?

• It’s about agility.


Policies and Procedures

• If you don’t write it down, it isn’t a procedure or a policy. There are no unwritten rules in a NOC. People cycle.

• Focus on procedures that affect the handling of a customer. The soft skills are usually the hardest to learn.

• Full disclosure?• Troubleshooting Techniques


And a word on incidents.

• Incidents, tickets, call it what you will.• Whatever happens in the NOC:

gets summarized to the account exec if the issue is of appropriate severity

gets recorded - nothing EVER gets deleted from a customer log

gets archived gets resolved - how much time will you

spend on issues that never get fixed?


Some final notes - Conclusion

• It’s easy to throw a bunch of people in a room. It’s hard to build an efficient, productive, and well-oiled NOC (the same goes for networks)

• Record everything. Write everything down.• Try to remain as flexible as possible.• Look to your employees for cues on what their

ideal environment should be.• Remember your focus, and know your customer.• Know what not to say.


Credits

As far as the practice of NOC building goes, we must thank:

• [email protected] - thanks for great private discussions on NOC building

• nanog subscribers - for talking about operational stuff


And finally

• Sean Donelan - for giving network operations folks an impetus for existing (due to his backhoe skills)

• Shouts to EFNet IRC #nanog: you know who you are


Part III: Evaluation Criteria

• Social engineer your own NOC.• We like to rate NOC folks on:

Do they sound professional? Do they have the answers?

Do they follow us all the way through to resolution?

Can they tackle difficult as well as simple problems?


Part III: Evaluation Criteria (cont).

• How many calls a day does your NOC take?

• Do you survey your customers, Cisco-TAC style?

• The three ‘P’s: Perception Passion (Emotion) Presentation


Evaluation Criteria

• They (customer) must perceive you as the people that will help.

• You (NOC) must present your solution with total assurance that you can handle the problem.

• You (NOC) must have passion in dealing with the customer. Make it friendly, even if it’s a network down emergency. It will come back to pay off.


Remember.

• Your goal is to be professional, calm, cool and collected, even in the face of serious adversity.

• If you don’t have an answer, you know where to get it and how long it will take.

• You have to communicate to the customer and keep the customer’s viewpoint in mind, no matter what.

• As a NOC manager/builder, you have to make sure your people have the right tools.

Documents

NOC Theory and Practice