Neutron UpgradeVictor Morales

Agenda• Upgrade - https://governance.openstack.org/reference/tags/assert_supports-upgrade.html

• Rolling upgrade - https://governance.openstack.org/reference/tags/assert_supports-rolling-upgrade.html

• Zero downtime upgrade - https://governance.openstack.org/reference/tags/assert_supports-zero-downtime-upgrade.html• Plan - https://review.openstack.org/#/c/386685/

• Meetings - https://wiki.openstack.org/wiki/Meetings/Neutron-Upgrades-Subteam

• Zero impact upgrade - https://governance.openstack.org/reference/tags/assert_supports-zero-impact-upgrade.html

Upgrade – Part 1That means it supports a controlled and planned upgrade process from release to release.

Requirements:

• Configuration from release N-1 is supported in release N. • oslo-config-generator https://github.com/openstack/oslo.config/blob/master/oslo_config/cfg.py#L401-L442

• Database schema updates are stable and ordered such that moving a database (with actual data in it) from release N-1 to N is possible without data loss. • Independent Sub-Project Tables (networking-odl, networking-onos, networking-calico, etc)• Offline/Online Migration http://docs.openstack.org/developer/neutron/devref/alembic_migrations.html

• Expand - Rules are strictly additive and can be applied while neutron-server is running (creating a new table, adding a new table column, adding a new index, etc.)

• Contract - Rules that are not safe to apply while neutron-server is running (column or table removal, moving data from one part of the database into another, introducing or modifying constraints, etc.)• https://review.openstack.org/#/c/400239/ Forbid contract migration scripts for Ocata

Upgrade – Part 2• A procedure for general upgrades of the project is defined and does not change substantially from cycle to

cycle. http://docs.openstack.org/developer/neutron/devref/upgrade.html• Upgrade Strategy

• All services are shut down, code upgraded, then all services are started again.• Services are upgraded gradually, based on operator service windows(Rolling Upgrade)

• Provides an upgrade impact section on the release notes page that highlights anything that must be done by operators for each cycle outside the normal upgrade procedures. https://docs.openstack.org/releasenotes/neutron/unreleased.html#id41

Upgrade – Part 3• Full stack integration testing is performed on every proposed commit to validate that cold upgrades from the

previous stable release are not broken. https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/projects.yaml#L7709-L7738

• Grenade https://docs.openstack.org/developer/grenade/

1. Get 2 Devstacks (base & target)2. Install base devstack3. Perform some sanity checking (currently tempest smoke) to ensure this is right4. Allow projects to create resources that should survive upgrade - see projects/*/resources.sh5. Shut down all services6. Verify resources are still working during shutdown7. Upgrade and restart all services8. Verify resources are still working after upgrade9. Perform some sanity checking (currently tempest smoke) to ensure everything seems good.

Rolling UpgradeThat means that the code is installed and deployed on many distributed systems and can be upgraded avoiding a significant downtime.

Requirements:

• Supports Upgrade.• The project has a defined plan that allows operators to roll out

new code to subsets of services, eliminating the need to restart all services on new code simultaneously. In other words, “restarting all API services together” is a reasonable restriction. http://docs.openstack.org/developer/neutron/devref/upgrade.html

1. Server Upgrade2. Agents Upgrade3. Networking backends.

https://www.youtube.com/watch?v=UQLBw1_VGcU

Zero Downtime Upgrade – Plan• Isolate layer that has access to database (Oslo-Versioned Objects).

https://docs.google.com/spreadsheets/d/1FeeQlQITsZSj_wpOXiLbS36dirb_arX0XEWBdFVPMB8/edit#gid=1434170112

• Forbid new contract scripts in Ocata. The idea is to reach a point where unsafe database operations (dropping tables and/or columns) become safe to execute it. https://review.openstack.org/#/c/400239/

• Online data migration[Draft] https://review.openstack.org/#/c/432494/

• Neutron PTG Upgrades session https://etherpad.openstack.org/p/neutron-ptg-pike-upgrades

Oslo-Versioned Objects (Approach)

https://www.youtube.com/watch?v=ivcNI7EHyAY

Olso-versioned Objects (NeutronDbObject)@six.add_metaclass(abc.ABCMeta)class NeutronObject(obj_base.VersionedObject, obj_base.VersionedObjectDictCompat, obj_base.ComparableVersionedObject):

@six.add_metaclass(DeclarativeObject)class NeutronDbObject(NeutronObject):

Oslo-Versioned Objects(Example)@obj_base.VersionedObjectRegistry.registerclass QosPolicy(rbac_db.NeutronRbacObject): # Version 1.0: Initial version # Version 1.1: QosDscpMarkingRule introduced # Version 1.2: Added QosMinimumBandwidthRule # Version 1.3: Added standard attributes (created_at, revision, etc) # Version 1.4: Changed tenant_id to project_id VERSION = '1.4’…def obj_make_compatible(self, primitive, target_version):…

_target_version = versionutils.convert_version_to_tuple(target_version) names = [] if _target_version >= (1, 0): names.append(rule_obj_impl.QosBandwidthLimitRule.obj_name())

Zero Downtime UpgradeThat means that the code is installed and deployed on many distributed systems and can be upgraded without downtime (control plane entirely).

Requirements:

• Supports Rolling Upgrades.• Requires services to completely eliminate API downtime

of the control plane during the upgrade. In other words, “restarting all API services together” is not reasonable.

• While all requests to the control plane must be eventually processed, performance degradation during the upgrade is acceptable.

Neutron – Zero Impact UpgradeThat means that the code is installed and deployed on many distributed systems and can be seamlessly upgraded without downtime.

Requirements:

• Supports Zero Downtime Upgrades.• Requires services to completely eliminate any perceivable performance penalty during the upgrade process.

Q & A