1  

TransPAC3-­‐  Asia  US  High  Performance  International  Networking  NSF  Award    #0962968  Year  5  Annual  Report  

1  March  2014  through  28  February  2015  

 Jennifer  Schopf    –  Principal  Investigator  

(Prepared  by  Alice  Jackson,  Jennifer  Schopf)      

Summary  During  the  Year  5,  the  TransPacific  Asia-­‐US  High  Performance  International  Networking  project  (TransPAC3)  continued  its  collaborative  and  engineering  activities  to  support  international  science.    This  report  outlines  collaborations,  software  and  systems  work,  operational  activities,  and  usage  statistics  for  the  project.  Highlights  include  a  99.98%  uptime  record  for  the  circuits  and  their  infrastructure,  hiring  a  replacement  TransPAC  engineer,  participation  in  15  meetings  and  workshops,  including  presentations  on  several  aspects  of  the  TransPAC  project,  and  extending  the  monitoring  infrastructure  for  the  circuits.    

1.  TransPAC    Overview    The  TransPAC  project  supports  three  circuits  and  a  set  of  network  services  between  the  US  West  coast  and  Asia.  In  the  current  set  up,  these  circuits  are:  

• The  TransPAC  Circuit:  a10Gbps  link  between  Los  Angeles,  California,  and  Tokyo,  Japan.  This  is  the  primary,  NSF-­‐funded  circuit  for  the  project  and  used  for  the  bulk  of  the  production  project  network  bandwidth.  

• The  JGN-­‐X  Circuit:  a  10Gbps  layer-­‐2  circuit,  largely  used  for  experiments  and  Software  Defined  Networking  (SDN)  trials.    The  Japan  Gigabit  Network  Extension  (JGN-­‐X)  project  is  a  testbed  funded  by  the  Japanese  National  Institute  of  Information  and  Communications  Technology  (NICT)  (http://www.nict.go.jp/en).  This  link  is  not  supported  by  NSF  funds.  A  backup  routed  peering  connection  between  TransPAC  and  APAN  also  runs  across  this  link.  

• The  CERNET  circuit:  a  10Gbps  link  between  Los  Angeles,  California,  and  Beijing,  China,  which  is  only  partially  funded  by  NSF.  This  is  a  layer3-­‐only  connection  at  this  time.  

 

2  

These  circuits  are  used  in  production  to  support  a  wide  variety  of  science  applications  and  demonstrations  of  advanced  networking  technologies.  In  addition,  the  TransPAC  award  supports  tool  development,  SDN  experimental  work,  measurement  deployments,  and  security  activities.    Please  note  that  some  of  the  activities  (outreach  to  Africa,  PerfSONAR  training,  etc.)  are  also  included  in  the  ACE  annual  report  project,  as  appropriate,  as  joint  funding  supports  these  efforts.  

2.  Financials      The  TransPAC  project  over  the  course  of  Year  5  spent  approximately  $1,134,297,  which  was  on  target  for  the  project  to  be  spent  down  on  schedule  for  the  original  May  31,  2015  end  date.  See  Appendix  A  for  full  year  accounting.  However,  in  September  the  IU  Finance  office  finally  got  a  fully  updated  accounting,  including  a  ground  truth  of  the  state  of  the  finances  for  the  project  as  a  whole.  At  that  time  it  was  discovered  that  the  project  was  estimated  to  be  underspent  by  $1,440,000  at  the  project  end  date  if  spending  did  not  change  significantly.  In  May  it  was  determined  that  the  underspend  amount  was  closer  to  $1,050,000.  Further  investigation  found  a  severe  underspend  in  the  first  2  years  of  the  project,  which  the  current  PI  was  unaware  of.      In  order  to  address  this  issue,  Schopf  met  with  NSF  PO  Thompson  September  30th  to  discuss  future  plans  and  alternatives.  The  project  requested  and  was  granted  a  one-­‐year  No  Cost  Extension,  and  several  possible  plans  were  discussed.      In  November,  Schopf  was  informed  that  the  proposal  for  a  follow-­‐on  project,  TransPAC4,  was  being  recommended  for  funding,  however  at  a  reduced  budget,  with  the  TransPAC3  underspend  to  cover  the  first  year  of  expenses  for  the  new  award.  The  TransPAC4  project  is  being  announced  March  1,  2015.      Over  the  next  3  months,  additional  transition  planning  will  take  place.  A  high  level  spend-­‐down  plan  for  the  last  15  months  (since  this  report  is  being  submitted  Feb  28,  but  the  project  end  date  is  May  31)  includes:  

• Continue  all  staffing  through  May  2016    • Continue  production  LA-­‐Tokyo  circuit  through  December  2016  • Continue  funding  for  GlobalNOC  support  through  December  2016  • Continue  funding  for  production  IU  systems  support  through  December  

2016  • Discontinue  support  for  LA-­‐Beijing  circuit  in  May  2015,  per  prior  agreed  to  

MOU  with  Internet2  • Upgrade  LA  PoP  to  collect  active  and  passive  monitoring  data  • Explore  additional  TransPacific  circuits  to  be  supported  through  TransPAC4,  

to  begin  fall  2015  –  this  may  include  additional  expenditures  if  a  Seattle  PoP  is  involved  

3  

• Expand  application  support  team  and  consider  developing  workshops  to  increase  use  of  circuits  

 With  these  plans  in  place,  the  project  should  spend  out  its  balance  by  the  end  of  the  sixth  year,  and  insure  a  smooth  transition  to  TransPAC4.  

3.  Staffing    There  were  several  staffing  changes  during  Year  5  of  the  project.  At  the  beginning  of  the  year,  project  staff  consisted  of:  

• Jennifer  Schopf,  Director  • Brent  Sweeny,  primary  TransPAC  senior  network  engineer  • Scott  Chevalier,  primary  contact  for  GlobalNOC  support  desk  • Alice  Jackson,  administration  • George  McLaughlin,  Asia-­‐Pacific  network  consultant  • Dale  Smith,  consultant  

 Brent  Sweeny  retired  from  Indiana  University  in  May.  The  job  was  posted,  and  Joe  Lappa,  formerly  of  Pittsburg  SuperComputer  Center  was  hired  in  July  to  be  the  primary  engineer  for  TransPAC.  However,  in  January,  Lappa  shifted  over  to  the  IU  iLight  project.  The  job  was  reposted,  and  Hans  Addleman  will  join  the  team  in  this  role  March  9.  Addleman  has  a  10-­‐year  history  with  the  IU  GlobalNOC,  and  a  depth  of  experience  working  with  other  International  Network  team  members.    George  McLaughlin’s  contract  was  going  to  be  competed  July,  2014,  but  he  has  requested  an  earlier  completion  date  due  to  health  issues.  His  last  charge  for  services  was  in  May.    Over  the  next  6  months,  we  expect  to  bring  on  an  additional  applications  support  staff  member  to  increase  the  use  of  the  circuits.  We  will  also  explore  hiring  an  additional  Asia-­‐area  consultant  to  replace  McLauglin  and  aid  in  science  support  work.  

4.  Collaborations,  Travel,  and  Training    TransPAC  staff  participated  in  various  meetings  to  support  their  role  in  collaborations  in  Asia.    These  included:  

• Internet2  Global  Summit,  Apri  6-­‐10  2014,  Denver,  CO  –  Schopf,  Sweeny,  Smith  

o Sweeny  received  lifetime  achievement  award  o Schopf  ran  session  on  use  of  100G  networks  

• U  Hawaii  Manoa,  April  24-­‐25,  Schopf  met  with  Jason  Leigh,  David  Lassner,  Gwen  Jacobs,  Alan  Whinery  

• CC:NIE  PI  workshop,  April  30-­‐May  1,  2014,  Arlington,  VA  –  Schopf  o Schopf  presented  on  Campus-­‐International  interactions  

4  

• Woods  Hole  Oceanographic  Institution,  May  23-­‐24,  Schopf  met  with  former  collaborators  and  workshop  organizers  

• TERENA  Networking  Conference,  May  17-­‐22,  Dublin,  Ireland  –  Schopf,  Sweeny  

o Sweeny  presented  on  BGP  PathHinting  • 2014  Environmental  Data  Summit  

(http://environmentaldatasummit2014.deltacouncil.ca.gov/  ),  June  4-­‐6,  Sacramento,  CA  

o Schopf  gave  opening  Keynote  o Additional  meetings  with  ESnet    

• Focused  Technical  Workshop  on  Global  climate  science,  July  13-­‐16,  Boulder,  CO  (http://meetings.internet2.edu/2014-­‐cc-­‐climate/)  

o Schopf  organized  • APAN  38,  Aug  9-­‐15,  Nantou,  Taiwan  –  Schopf,  Lappa  

o Schopf  chaired  2  sessions  o Schopf  gave  an  overview  of  TransPAC  plans  

• Lower  Mekong  Water  Resource  Workshop  (http://internationalnetworking.iu.edu/archives/LMI/index.html),  August  18-­‐22,  Hanoi,  Vietnam  –  Schopf,  Lee  

o Schopf  co-­‐organized  • Chinese  American  Networking  Symposium,  September  15-­‐17,  New  York  City  

–  Lappa  o Lappa  presented  on  use  of  CERNET  circuit  

• PRAGMA  27,  Oct  15-­‐17,  Bloomington,  IN  o Schopf  was  a  co-­‐organizer  for  the  event  o Schopf  chaired  the  Network  Engineering  Workshop  o Lappa  and  Sweeny  presented  

• Internet2  Technology  Exchange,  October  26-­‐30,  Indianapolis,  IN  –  Schopf,  Lappa,  Lee  

o Lee  also  attended  the  Sunday  program  “Broadening  the  Reach:    A  Cyberinfrastructure  Program  for  Non-­‐Researh-­‐Intensive  and  EPSCoR  Institutions”      

• SuperComputing,  November  16-­‐21,  New  Orleans  –  Schopf,  Lee,  Lappa  o Schopf  co-­‐organized  “Innovating  the  Network  for  Data-­‐Intensive  

Science”  workshop.    • ESIP  Winter  meeting,  January  6-­‐8,  Washington,  DC  –  Schopf  • PerfSonar  Workshop,  January  21-­‐23,  Columbus,  OH  –  Schopf,  Lee  

   Throughout  the  year,  the  team  increased  its  collaborations  with  international  groups  significantly.  At  the  upcoming  APAN  meeting  in  March  we  will  begin  hosting  TransPAC-­‐specific  sessions.  Over  the  next  year,  we  plan  to  stay  involved  in  many  of  these  conferences  and  workshops,  and  increase  our  work  with  end-­‐users.    

5  

5.  Software  and  Systems  Work  

A.  Tool  Development    Tool  development  continued  this  quarter  with  expected  patches  and  upgrades  to  existing  tools.      

B.  Dynamic  Circuits    The  TransPAC  project  provides  Dynamic  Layer  2  Network  services  through  the  OSCARS  software  suite.    This  service  peers  with  the  Internet2  ION  project  and  the  JGN-­‐X  Dynamic  network  facilities.    Researchers  and  scientists  can  interactively  create  a  layer2  dynamic  circuit  between  Asia  and  the  US  to  transfer  data.      

C.  Software  Defined  Networking  (SDN)  Activities    One  of  the  primary  research  goals  of  the  TransPAC  project  is  to  enable  Software  Defined  Networking  (SDN)  using  Open  Flow  to  provide  a  mechanism  to  dynamically  configure  and  control  circuit  behavior  between  the  US  and  Asia.    Support  for  JGN-­‐X’s  RISE  and  Open  Flow  experiments  continued.        Unfortunately,  due  to  staff  changes,  the  SDN  work  over  TransPAC  did  not  move  forward  as  much  as  hoped  in  Project  Year  5.  This  will  be  a  focus  of  effort  over  the  last  year  of  the  project,  including  several  talks  and  demos  by  the  group.    

D.  Measurement  Activities    The  TransPAC  project  supports  a  perfSONAR  deployment  in  Los  Angeles  that  provides  periodic  testing  between  several  US  and  Asian  sites.  A  testing  matrix  is  available  at  http://tp3-­‐3.transpac3.iu.edu/maddash-­‐webui/  ,  however  it  is  currently  not  collecting  any  data.  We  have  started  up  conversations  with  the  JGN  monitoring  group  3  times  over  the  last  12  months  to  try  to  get  this  addressed  to  no  avail.  We  will  continue  to  press  on  this  issue.    We  negotiated  with  the  CERNet  group  to  begin  to  collect  Flow  data  in  the  form  of  “Top  Talkers”  list.  They  gave  us  a  sample  in  August  and  another  in  December,  see  Appendix  B  for  the  slides.  With  our  new  staffing  we  will  be  following  up  to  collect  this  data  on  a  monthly  basis  for  the  remaining  lifespan  of  the  circuit.    In  Year  6  we  will  also  be  collecting  our  own  flow  data  on  the  NSF-­‐funded  TransPAC  circuit,  as  part  of  the  upgrade  to  the  LA  POP.      

6  

6.  Operational  Activities    The  TransPAC  circuits  were  very  stable  over  the  year,  with  only  6  downtimes,  of  which  only  one  was  unscheduled.  The  circuits  have  not  changed  over  the  course  of  the  project  year.    We  are  in  the  process  of  upgrading  the  Los  Angeles  PoP  to  enable  additional  monitoring  and  measurement  capabilities,  as  well  as  better  supporting  upcoming  SDN  activities.  The  Ixia  tester,  purchased  as  part  of  the  ACE  award,  will  also  be  placed  in  LA  to  allow  better  use  of  it  for  testing.  This  will  require  an  upgrade  to  our  router  and  port  to  support  a  100G  connection.  Figure  1  shows  the  current  set  up  at  the  POP,  and  Figure  2  diagrams  the  planned  upgrade.      

 Figure 1: LA POP at present time.

 Figure 2: LA POP when upgrade is completed.

7  

 

A.  Network  Engineering    The  TransPAC  circuit  between  Los  Angeles,  California,  and  Tokyo,  Japan,  continues  to  function  as  designed  with  no  unscheduled  outages.  It  supports  IPv4,  IPv6,  and  dynamic-­‐circuit  functions,  and  is  prototyping  OpenFlow  1.0  capabilities.      The  JGN-­‐X  circuit  between  Los  Angeles,  California,  and  Tokyo,  Japan,  operates  in  layer  2  between  switches  in  Tokyo  and  Los  Angeles,  and  is  primarily  used  for  experimental  network  research,  particularly  DCN,  OpenFlow,  GLIF,  Optical  testbeds,  and  OpenGOLEs.      The  Beijing  CERNET  link  will  be  no  longer  part  of  the  TransPAC  project  after  May  31,  2015,  per  prior  agreement  with  Internet2.  Planning  for  this  shift  has  taken  place.    

B.  Traffic  Graphs    Figure  3  and  Figure  4  show  the  traffic  on  the  TransPAC  network  during  Project  Year  5,  and  Figures  5  and  6  shows  data  for  the  CERNET  connection  for  the  same  period.  Monitoring  data  is  not  available  for  the  JGN-­‐X  circuit.  

Figure 3: Aggregate traffic using average daily values on the 10Gbps TransPAC (NSF-funded) circuit between Los Angeles and Tokyo

8  

Figure 4: Aggregate traffic using maximum daily values on the 10Gbps TransPAC (NSF-funded) circuit between Los Angeles and Tokyo

Figure 5: Aggregate traffic using average daily values on the 10G CERNET circuit between Beijing and Los Angeles

9  

Figure 6: Aggregate traffic using maximum daily values on the 10G CERNET circuit between Beijing and Los Angeles. Gaps in data are periods where the polling server is unable to contact the switch to gather data.

 

C.  Trouble  Tickets  During  this  year,  there  were  6  reported  downtimes,  five  scheduled  and  one  unscheduled.  Table  1  shows  detail  on  the  unscheduled  downtime.  Table  2  shows  the  detail  on  the  scheduled  downtime.    Additional  trouble  ticket  information  is  available  at  https://tick.globalnoc.iu.edu/fp_tools/public_ticket_viewer/index.cgi  by  entering  the  ticket  number.      The  unscheduled  outage  on  October  31,  2014,  (ticket  1499:62)  on  the  TransPAC  Core  Node  experienced  was  due  to  maintenance  performed  by  Pacwave  during  which  the  buffering  capability  of  the  line  cards  was  overwhelmed.  This  outage  identified  an  issue  with  the  core  equipment  that  eventually  resulted  in  an  upgrade  to  minimize  any  potential  re-­‐occurrence.    

 Ticket

Number Customer

Impact Network Impact Title Outage Type Source Of

Impact Start Time

(UTC) End Time

(UTC)

1499

1-Critical 1-Critical Brief Outage Resolved - TransPAC Core Node LOSA

Unannounced Maintenance

Vendor 10/31/2014 7:26 AM

10/31/2014 7:42 AM

Table 1: Unscheduled maintenance tickets for the TransPAC this quarter

10

 

 

Ticket Number

Customer Impact

Network Impact Title Maintenance

Type Source

Of Impact

Start Time (UTC)

End Time (UTC)

1438

4-Normal 3-Elevated

Maintenance 1 of 2 Completed - TransPAC Backbone TOKY-TransPAC LOSA

Circuit Vendor 03/06/2014 11:39 AM

03/06/2014 12:50 PM

1507

4-Normal 2-High Emergency Maintenance Completed - TransPAC Peer PacWave (All Connections via PacWave)

Hardware

Internal

11/13/2014 3:36 PM

11/13/2014 5:48 PM

1520

4-Normal 2-High Maintenance Completed - TransPAC Core Node LOSA Software and Hardware Upgrade

Hardware Internal 12/12/2014 3:03 PM

12/12/2014 3:13 PM

1532

3-Elevated 3-Elevated

Maintenance Complete - TransPAC Core Node Hardware Upgrade

Hardware Internal

01/21/2015 5:15 PM

01/21/2015 6:25 PM

1537

3-Elevated 2-High Maintenance 1 of 2 Completed - TransPAC Backbone TOKY-TransPAC LOSA

Circuit Vendor 01/27/2015 11:12 AM

01/27/2015 11:33 AM

Table 2: Scheduled maintenance tickets for the TransPAC circuits this year

 

D.  Downtime    Table  3  shows  the  downtime  information  for  TransPAC  core  nodes  and  Table  4  Shows  the  downtime  for  TransPAC  Circuits.    The  single  unscheduled  outage  on  October  31,  2014,  (ticket  1499:62)  on  the  TransPAC  Core  Node  was  due  to  maintenance  performed  by  Pacwave.    During  this  maintenance  the  buffering  capability  of  the  line  cards  was  overwhelmed.  This  outage  identified  an  issue  with  the  core  equipment  that  eventually  resulted  in  an  upgrade  to  minimize  any  potential  re-­‐occurrence.    

TransPAC Core Nodes Down Time Reporting Period Availability 52 Week Availability

TransPAC MX480 - LA 0 hr 16 min 100.00% 100.00%

Brocade MLXe4 3 hr 32 min 99.96% 99.96%

3410 Ethernet Switch 0 hr 0 min 100.00% 100.00%

OOB Router 0 hr 0 min 100.00% 100.00% Aggregate TransPAC Core Nodes 3 hr 48 min 99.99% 99.99%

Table 3: Reported downtimes for core nodes of TransPAC this year

11

 

TransPAC Backbone Circuits Down Time Reporting Period Availability 52 Week Availability

TransPAC LOSA-JGN2 LOSA 10GigE 0 hr 16 min 100.00% 100.00%

TransPAC LOSA-Pacific Wave LOSA 10GigE 1 hr 36 min 99.98% 99.98%

TransPAC TOKY-TransPAC2 LOSA 2 hr 58 min 99.97% 99.97% Aggregate All TransPAC Backbone Circuits 4 hr 50 min 99.98161% 99.98166%

Table 4: Reported downtimes for circuits of TransPAC this year

7.  Security  Events  and  Activities    Basic  security  measures  were  maintained  over  the  course  of  the  year,  and  there  were  no  security  incidences  to  report.  We  began  to  examining  the  possibility  of  contracting  out  for  some  additional  international  security  work,  for  example,  Jacob  Farmer,  IU,  attended  the  CANS  meeting  on  behalf  of  the  TransPAC  project.    

8.  Year  6  and  Transition    As  discussed  in  Section  2,  we  have  requested  and  been  granted  a  one-­‐year  No-­‐Cost-­‐Extension  for  the  TransPAC3  project,  with  a  new  expected  end  date  of  May  2016.  We  have  also  been  informed  that  TransPAC4  has  been  recommended  for  funding,  with  an  expected  start  date  of  March  2015,  however  with  greatly  reduced  Year  1  funding.  Year  6  or  TransPAC3  will  be  spent  in  transition  to  the  new  TransPAC4  project.  We  split  this  plan  into  several  components,  and  address  them  separately  below.    Production  Traffic:  The  current  10G  NSF-­‐funded  TransPAC  circuit  will  be  supported  through  the  end  of  calendar  year  2015.  Between  March  and  August,  we  will  begin  negotiations  for  additional  trans-­‐Pacific  circuits  for  a  2-­‐5  year  contract  duration.  We  will  ensure  a  smooth  transition  of  any  production  traffic  if  the  routes  change.  Per  agreed  upon  MOU,  the  production  CERNET  link  will  transition  to  Internet2  on  June  1,  2015.  However,  it  could  still  carry  production  traffic  for  the  TransPAC  project  is  that  is  deemed  appropriate.    Experimental  SDN  Traffic:  .    JGN  has  informed  us  that  in  their  next  funding  round,  they  will  concentrate  on  intra-­‐Asia  connectivity,  so  it  is  possible  their  10G  experimental  link  will  cease.  Our  new  production  links  will  need  to  also  be  able  to  carry  experimental  traffic  as  well.  In  addition,  SDN  experiments  will  be  a  focus  of  Year  6  of  the  project.    Monitoring  Work:  The  TransPAC  monitoring  work  increased  over  Year  5,  and  is  likely  to  be  extended  going  forward.  We  have  been  arguing  for  a  strong  PerfSonar  presence  in  APAN,  and  are  committed  to  ensuring  this  as  the  project  transitions.  

9.  Reporting  against  Objectives  Year  5    

12

 

1. Overall  a. Planned:  New  director  to  review  activities  and  adjust  as  needed  

(Ongoing)  b. Additional:  Address  budget  surplus  (See  Sec  2)  

2. Staffing  a. Planned:  Recruitment  of  replacement  half-­‐time  engineer  for  the  

project  (See  Sec  3:  Lappa  hired  July  2015,  Addleman  hired  March  2015)  

3. Collaboration,  Travel,  and  Training  a. Planned:  Attend  and  speak  at  major  Asian  meetings  (See  Sec  4)  b. Planned:  Co-­‐chair  PRAGMA  meeting  in  October  with  Networking  focus  

(See  Sec  4)  c. Planned:  Additional  TransPAC  meetings  at  APAN  in  August  (See  Sec  4)  d. Planned:  Identify  scientific  applications  using  TransPAC  more  

specifically  (Delayed  to  Year  6)  4. Systems  and  Software  Work  

a. Planned:  Re-­‐focus  on  monitoring  of  all  circuits  (See  Sec  5.D)  b. Planned:  Identify  and  if  possible  implement  flow-­‐analysis  tools  (See  

Sec  5.D)  c. Planned:  Extend  current  SDN  experiments  with  JGN  (See  Sec  5.C)  

5. Operational  Activities  a. Planned:  Continue  full  support  of  TransPAC  circuit  (See  Sec  6)  b. Planned:  Continue  to  shift  production  system  support  to  production  

GlobalNOC  teams  (See  Sec  6)  

10.  Plans  for  Year  5    1. Overall  

a. New  director  to  review  activities  and  adjust  as  needed  b. Ensure  smooth  transition  to  TransPAC4  project  

2. Staffing  a. Recruitment  of  additional  Outreach  Specialist  

3. Collaboration,  Travel,  and  Training  a. Attend  and  speak  at  major  Asian  meetings  b. Identify  scientific  applications  using  TransPAC  more  specifically  c. Consider  organization  of  application  workshops  to  support  end  users  

4. Systems  and  Software  Work  a. Update  LA  PoP  to  support  additional  monitoring,  use  of  IXIA  tester  b. Update  active  monitoring  of  all  circuits  c. Update  APAN  active  monitoring  systems  d. Identify  and  if  possible  implement  flow-­‐analysis  tools  e. Extend  current  SDN  experiments  with  JGN  

5. Operational  Activities  a. Continue  full  support  of  TransPAC  circuit  

     

13

 

 

Appendix  B:  Sample  Flow  Data  from  CERnet