The GlobalNOC: What we do, and how we do it...GRNOC Begins Providing Service Desk and Layer2/3 NOC Support for National Lambda Rail 2004 GRNOC installs and begins supporting CIC OmniPoP

The GlobalNOC: What we do, and

how we do it

1

Friday, July 17, 2009

Topics

2

1.Brief History

2.Supported Services

3.GlobalNOC Organization

4.Life of an Issue

5.Future Plans


GRNOC History

3


4


1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

HOPI Network Built

2005

GRNOC Builds TransPac2 Network

2004

GRNOC Begins Providing Service Desk and Layer2/3 NOC

Support for National Lambda Rail

2004

GRNOC installs and begins supporting CIC OmniPoP Network

2005

IPGrid Created to connect IU and Purdue to the TeraGrid

network

2004

Abilene NOC formed

1998

MAX GigaPoP

2009

Indiana GigaPoP Upgrade

2006

Abilene Upgrade completed

2004

MANLAN Exchange Point Created

2003

GRNOC begins providing NLR Engineering support

2005

Indiana GigaPoP adds IP functionality

2001

Indiana GigaPoP Created (Layer2)

1998

GRNOC Build-out of Ilight Dark Fiber Network

2001

GRNOC Builds out Ilight: Indiana's Optical Statewide Network

2006

GRNOC engineers deploy new Internet2 Network

2007

GENI Meta-Operations Center Project begins

2008

GRNOC begins providing after-hours Service Desk support for

Connecticut Education Network

2008

GRNOC begins providing pilot Software as a Service network

management support for OneNet

2008

GRNOC begins After Hours Service Desk support for Ampath

2004

5

GRNOC History


1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

HOPI Network Built

2005


2004



2004


2005


network

2004

Abilene NOC formed

1998

MAX GigaPoP

2009


2006


2004


2003


2005


2001


1998


2001


2006


2007


2008



2008



2008


2004

6


1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

HOPI Network Built

2005


2004



2004


2005


network

2004

Abilene NOC formed

1998

MAX GigaPoP

2009


2006


2004


2003


2005


2001


1998


2001


2006


2007


2008



2008



2008


2004

7


GRNOC Services

8


Supported Projects

9

OmniPoP

REN-ISAC


Full NOC Services

10

Customized to meet specific needs of each organization

Service Desk Monitoring & Coordination

Tool Suite & Development

Engineering

Can be Tier2 only with Tier3 performed by others, or Full Tier2/3

Architecture/Planning/Direction provided by Network Owners

Examples: Internet2, NLR, CIC, MAX, etc


Service Desk Services

11

24x7 or After-Hours

Problem Verification or Correlation

Work with Network engineers remotely

Examples: CEN, AMPATH


Software Services

12

Users can leverage GRNOC Tool Suite to manage their network

GRNOC manages & operates servers and tools 24x7x365

Updates as toolset is enhanced

Custom Feature Development

Example: OneNet


Network Design/Consultation

13

Design changes

RFP/RFI review

Audits

Other advice as needed

Example: OneNet


Deployment Services

14

Team of engineers experienced in installation of lots of gear

DC Power, CO Installation Standards

can help with advice too


GRNOC Organization

15


The Gist of it

16

GRNOC is part of Indiana University’s IT department, UITS

Overall, about 60 staff

Staff is split between Indianapolis and Bloomington campuses

Plus our British Columbia office!


Jay DuncanManager, Network

Engineering

Luke FowlerManager, Systems

Engineering

Network Planning Team

Jim WilliamsDirector

GRNOC Service Desk

Matt DavyChief Network

Architect

Steve PeckService Desk

Manager

Chris RobbInternet2

Operations Manager

Grover BrowningNLR Director of Operations and

Engineering

Jon-Paul HerronGroup Manager

GRNOC Engineering

Marianne ChitwoodManager

Ilight

Dave JentAssociate Vice

President Networks

Brad WheelerVice President

IT & CIOIndiana

University

17



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

18



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

18

Leadership



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

19



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

19

Service Desk



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

19

Service Desk

• Staff of ~25



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

19

Service Desk

• Staff of ~25• 24x7x365 Customer interaction and service



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

19

Service Desk

• Staff of ~25• 24x7x365 Customer interaction and service • Provides interface to engineering



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

19

Service Desk

• Staff of ~25• 24x7x365 Customer interaction and service • Provides interface to engineering• Staff are network knowledgeable; they

understand network topology, cause & effect, and management processes



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

19

Service Desk



• Single point of contact for network management



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

19

Service Desk



• Single point of contact for network management

• Leveraged resources for cost-effective operations and management



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

20



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

20

Architecture



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

20

Architecture

• Staff of 3



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

20

Architecture

• Staff of 3• Long-term Network Architecture Decisions



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

20

Architecture

• Staff of 3• Long-term Network Architecture Decisions• Architecture Consultation



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

20

Architecture

• Staff of 3• Long-term Network Architecture Decisions• Architecture Consultation• Campus, Regional, or National issues



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

21



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

21

Engineering



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

21

Engineering

• Staff of ~30



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

21

Engineering

• Staff of ~30• 3 Main Teams:



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

21

Engineering

• Staff of ~30• 3 Main Teams:• Systems Engineering

(12): Develop GRNOC Tool Suite, Administer GRNOC Sytems, Operate Measurement Platforms (Observatory)



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

21

Engineering



• Network Engineering (8): Broadly focused 24x7 Network Troubleshooting and Deployment



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

21

Engineering




• Network Planning (8): Dedicated Network Specialists providing Overall planning and final operational escalation point



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

21

Engineering





• 2 members with dual reporting:



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

21

Engineering






• Chris Robb Manages Internet2 Operations



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

21

Engineering






• Chris Robb Manages Internet2 Operations

• Grover Browning leads the NLR Engineering team



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

22



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

22

Ilight



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

22

Ilight

• Staff of ~8



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

22

Ilight

• Staff of ~8• Full remote hands, engineering, design,

planning, and ownership for Indiana’s Ilight network



Engineering


Engineering



GRNOC Service Desk


Architect


Manager

Chris RobbInternet2

Operations Manager


Engineering


GRNOC Engineering


Ilight


President Networks


IT & CIOIndiana

University

22

Ilight

• Staff of ~8• Full remote hands, engineering, design,

planning, and ownership for Indiana’s Ilight network

• Ilight serves 40+ connections exclusively for higher education in Indiana


Life Story of a Network Issue

23


24Problem Report or

Question Proactive Monitoring

Problem Verification & Information Gathering

Issue Correlation

Ticket(s) Created

NotificationOutage?

Issue assigned to Engineer

Initial Priority Set

Further Information Gathering

Troubleshooting/Consultation

Vendor Involvement?

Vendor Coordination

Problem Resolution

Priority Timer Expired? Escalation Process

Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


24Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

Issues may come from reports, and

may be problems or questions


24Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

Issues may come from reports, and

may be problems or questions

Issues may also come from proactive

monitoring from AlertMon


25Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


25Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

SD gets detailed information from

user such as related Service/Circuit ID,

and verifies problem


26Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


26Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

SD Checks existing Tickets,

and Alerts to find related issues


27Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


27Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

Priority of issue set by type of issue

or by request of user


28Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


28Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

One or more Footprints tickets

created


29Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


29Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

Notifications to Community Sent, using Notification

Tools


30Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


30Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

As needed, issues are assigned to

GRNOC OR external engineers for follow-up. If Critical Priority, engineers are

contacted immediately


31Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


31Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

Engineers investigate, and gather further

information from tools, network, and

user


32Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


32Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

Engineer(s) troubleshoot issue


33Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


33Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

If vendor involvement is

needed, Service Desk coordinates external requests


34Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


34Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

Depending on Priority of issue,

issue may be escalated for expediting


35Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no


35Problem Report or



Issue Correlation

Ticket(s) Created

NotificationOutage?





Vendor Involvement?

Vendor Coordination

Problem Resolution


Satisfaction Check

Close Ticket(s)

yes

no

yes

yes

no

no

After problem resolution, Service

Desk verifies satisfactory

resolution with user


GRNOC Plans

36


Maintain service

37

Main thrust is still to maintain the small organization level of service

No phone trees

Immediate access to engineers

No “wack-a-mole” tickets: it’s more important to resolve issues fully than to close tickets quickly

Keep Network Owners Happy...

...and they’re happy when you’re happy.


Notifications

Notification system revamp:

Flexible

More targeted notifications

Amount and types of notifications determined by users

notifications by methods most useful to people - email? phone? SMS? Twitter? Skywriters?

What people want will drive what we do, so let your networks know!

38


What do YOU want?

Let the networks know what you want out of us!

Let us know if you ever have a less than great NOC experience.

39


Documents

The GlobalNOC: What we do, and how we do it...GRNOC Begins Providing Service Desk and Layer2/3 NOC Support for National Lambda Rail 2004 GRNOC installs and begins supporting CIC OmniPoP