June 20–23, 2016 | Berlin, Germany

Multi-site OPNFV Testing ChallengesTrevor Cooper (Intel)

Fatih Degirmenci (Ericsson)

Morgan Richomme (Orange)

Agenda

Multi-site Testing Today - Trevor- Pharos Infrastructure

- CI Today

Multi-site Testing Needs - Morgan- Advanced Test-cases

- Interoperability

Multi-site Testing Plans - Fatih- Challenges and Requirements

- Future Plans and Vision for Infra and Testing

Summary and Call to Action - Fatih

Multi-site Testing Today

Multi-site Testing Introduction

• Motivations for Multi-site Testing• Geographically and technically diverse test environments

• Test resources independent of installers and scenarios promotes robust platform

• Inter-deployment testing for interoperability between data-centers

• Future real-world use-cases and deployment scenarios

• Pharos Infrastructure• Infra Working Group: Pharos, Octopus, Releng, Genesis

• Infrastructure labs, process, tools and features

• CI / CD / CT• Environments increasing in number and complexity

• Process to improve platform robustness

• Challenges of a Distributed Infrastructure • Managing a distributed resources

• Automating testing

• Distributing tests

Pharos Infrastructure

• Current priorities: • Allocation and management of Infra resources for Colorado

• Pharos change process to manage special requirements

• Lab resource booking and usage tool/dashboard

• Pharos compliance verification/validation

• Dealing with today’s multi-site challenges• Challenges and how we solved them

Production Today

CI Yesterday CI Today

POD1Installer XScenario Y

POD2

PODi

Deploy, run tests

POD1Installer XScenario Y

POD2

PODi

Deploy, run tests

Lab 1

Lab 2

Lab 1

Lab 2

Production Evolution - Tomorrow

POD1Installer XScenario Y

POD2

PODi

Deploy, run tests

Lab 1

Lab 2

POD1Installer X’Scenario Y’

POD2

Lab 3

CI Tomorrow

Infra Improvements

• Lab as a Service (LaaS)• Accessible virtual sandboxes for development and smoke tests

• More homogeneous environments for development work

• Off-load shortage of bare-metal resources

• Efficient use of Lab Resources• Strive for CD (provide weekly stable versions)

• Provide dynamic use of lab resources (Dev and CI)

• Verify/validate POD setup for automatic configuration

• All labs/pods/installer configurations for automatic POD configuration

• Jenkins / CI improvements - permissions, scheduling, sandboxes, etc.

Multisite Testing Needs

Test cases

• Advanced test cases• OPNFV Multisite Project - evolving

• Brahmaputra -> Use case definition and bug fixes

• Colorado -> Single machine/Multiple OpenStack Instances, Functional Testing, Kingbird

• Future -> Go after true Multisite

• Discussion on multi tenant vIMS for D river

• Interoperability testing

An advanced test case: vIMS

Today for Functest test: Metaswitch vIMS + Cloudify orchestrator installed in a single tenant on one OPNFV POD.

Not acceptable in “real life” (SPOF)

Geographical redundancy on current production IMS system

Architecture Cloud ready (Cassandra ring)

Possible evolutions: 1) test in multi-tenants (local redundancy)2) Test in multi-sites (geographical redundancy)

An advanced test case: vIMS

Need of Multi-sites to evaluate • the connectivity issues (VPN/Flat)• the consequence on performance• The accuracy of placement algorithm:

• Performance (DPDK/ODP area)• Location (legal constraint) …

• Disaster/recovery scenarios/constraints

Similar considerations on vCDN for next releases

An interoperability test

• OPNFV is a great place to automate interoperability testing…

• Several PODs … with several scenarios

• And a centralized CI

An interoperability test: bgpvpn

• SDNVPN project deals with the implementation of the OpenStack bgpvpn API

• Several possible backends• ODL

• Neutron Bagpipe

• Nuage

• …

• Use OPNFV to perform interoperability testing would be valuable

An interoperability test: bgpvpn

POD1os-nofeature-bgpvpn-ha

POD2os-odl_l2-bgpvpn-ha

Deploy, run tests

BGPVN

Multi-site Testing Plans

Lessons learned

• Distributed infrastructure is not trivial

• Automating tests/ state synchro towards multi-sites

• We learn by doing…first requests on multi-sites needed raised in Brahmaputra => Multi-site project

Next steps

• Apply learnings• Bringing up, managing and using distributed infrastructure

• Evolve continuously• No dramatic changes

• E2E View and holistic approach• Features will come together one day, creating more complete

Multisite solutions

Next steps

• Infrastructure• Provision, manage and use resources cleverly

• Increase the utilization rather than burning CPU uselessly

• Get ready to enter uncharted territories - security and access policies

• CI• Evaluate CI to see if it can support Multisite use cases

• Provide feedback that fits to purpose rather than bunch of log files

• Continue applying best practices

• Testing • Test what needs to be tested when it is needed to be tested

Summary

• We already are challenged by what we had in the past and what we currently have

• The complexity will just increase, requiring us as a community to work together like a community

Call to Action