Ironic Towards Truly Open and Reliable, Eventually for Mission Critical

Naohiro Tamura

Professional Engineer

Fujitsu Limited

Ironic

towards truly open and reliable,

eventually for mission critical

Copyright 2015 FUJITSU LIMITED

OpenStack Summit October 2015 Tokyo

Thursday, October 29 • 11:00am - 11:40am

TOC

Introduction

Who am I?

What is Fujitsu good at?

Vision

Customer Values: What are the most important for customer?

Mid Term Vision: Truly Open and Reliable

Long Term Vision: Eventually Mission Critical

Contribution

What we have done and are doing

What we are going to do

Conclusion

1 Copyright 2015 FUJITSU LIMITED

TOC

Introduction

Who am I?


Vision




Contribution

What we have done


Conclusion


Who am I?

I joined Ironic Community at the beginning of Kilo development cycle, and focus on Ironic Driver Development.

Before that, I mainly worked on proprietary software developmentfor system management.

I developed bare metal provisioning and IO virtualization for N+1 server redundancyby enhancing PXE server on legacy BIOS and UEFI.

OpenStack is my first Open Source Project, it’s a whole new experience

Joyful part – working with talented and smart people

Interesting phenomenon – bikeshedding

Contacts

Email: [email protected] IRC: naohirot


mailto:[email protected]


Fujitsu sustains many social infrastructures

Banking

Stock exchange

Factory automation

Government agency system

For example, Tokyo Stock Exchange Trading System

Highly Reliable IA x64 server (PRIMEQUEST/PRIMERGY)

Open Source Linux (RHEL)

Fujitsu’s in-memory database system

Fujitsu is good at missions critical systems


TOC

Introduction

Who am I?


Vision

Customer Values



Contribution



Conclusion


Customer Values


What are the most important thingsfor customer?

Three Customer Values We See


Truly open, no vendor lock-into provide customer with freedom to switch any vendor anytime

Reliable, robust, highly available system

to operate customer's business continuously

Responsive, responsible, competent support

to resolve customer's incident quickly and accurately.

Mid Term Vision : Truly Open and Reliable


What should truly open and reliable be?

The Android Robot logo is licensed under the terms of the Creative Commons Attribution license

http://creativecommons.org/licenses/by/2.5/

CurrentUsed to be


Android as a concrete reference model


iOS

Android

Android

iOS

2015Q2 WW Smartphone Shipments

Source: Worldwide Smartphone Growth Expected to Slow to 10.4% in 2015, Down From 27.5% Growth in 2014, According to IDC http://www.idc.com/getdoc.jsp?containerId=prUS25860315

13.9％

82.8％

No vendor lock-in.

Customer can switch

from one vendor

to other vendor

whenever she/he wants

because of truly open

and reliable.

http://www.idc.com/getdoc.jsp?containerId=prUS25860315

Current Future


OpenStack would be in the same situation as Android


Big FourBig Four

Customer can switch

from one cloud

to other cloud

whenever she/he wants

if it’s truly open and reliable.

Interoperability is key

for no vendor lock-in

among public, private

and hybrid cloud.


Android

Android defines Hardware

Customer can switch anytime to whichever vendor he/she likes.

From UI’s point of view, all Android smartphones have same functionality

Hardware reliability and Support Responsiveness are different among vendors.


Ironic

Ironic will define datacenter server hardware specification if the market accepts Ironic.

Customer will be able to switch anytime to whichever vendor he/she likes.

From API, CLI, and UI’s point of view, Ironic will have same functionality to all bare metal servers

Hardware reliability and Support Responsiveness will be different among vendors.

Situation Comparison between Android and Ironic



How can we achieveTruly Open and Reliable?


1. First of all, we need to complete the following table to create the same situationas Android, that is no vendor lock-in situation.

Current status as of Liberty: Ironic BMC Driver Implementation

Legend: ✔done, △ ongoing, ×not yet , － not applicable Green: Contributed, Yellow: Contributing, Pink: plan to contribute

2. And then enhance proactive/reactive features to achieve higher reliability


State Power On/Off Power Off to On Deploy Active Inspect Clean Zap Rescue

I/F Power Boot Deploy Mgmt Inspect Clean Raid Rescue

method

BMC

hard soft pxe vmedia iscsi agent oob ib oob iscsi agent ib oob pxe vmedia

IPMI ✔△

liberty✔ － ✔ ✔ ✔

✔kilo

－×

✔kilo

×－

× －

AMT✔kilo

×✔kilo

×✔kilo

×✔kilo

× × ××

× × × ×

DRAC ✔×

✔ × ✔ × ✔✔kilo

× × × × × × ×

iLO ✔ × ✔ ✔ ✔ ✔ ✔△

liberty

✔kilo

✔kilo

✔kilo

△liberty

×× ×

iRMC✔kilo

△liberty

✔kilo

✔liberty

✔kilo

✔liberty

✔kilo

×△

liberty

×× ×

×× ×

UCS✔

liberty

× ✔liberty

×✔

liberty

✔liberty

✔liberty

×△

liberty× × × × × ×


Can you imagine that Tokyo Stock Exchange Trading System runs inside Ironic?

No, I can’t right now.Stock market involves huge amount of investment money.

Our Vision is really challenging, “Ironic for Mission Critical”.

We believe that it’s difficult to achieve this visionjust by a company.

But we believe that we can achieve this visionby a community.Because there are a lot of things to be done.


https://ja.wikipedia.org/wiki/%E6%9D%B1%E8%A8%BCArrows

TOC

Introduction

Who am I?


Vision




Contribution



Conclusion


To realize the mid term vision1) Complete the table for truly open

2) Proactive/reactive features for reliability

TOC

Introduction

Who am I?


Vision




Contribution



Conclusion



Virtual Media Deployment• Out of Band Boot

Soft Power Off and Inject NMI• Power Control Finite State Machine

• Abort Task


Rescue Mode in Tenant Network• Repair Instance Image in Cinder by Virtual Media Boot

Bare Metal N+1 Redundancy• Cold Migration by Soft Power Off and Virtual Media Boot

Virtual Media Deployment


What is it good for?






Virtual Media enables Out of Band (OOB) Boot

It is good for multi tenant and networked storage environment

Element Technology



How does it work?

Note: Ironic Deploy Basics

Boot methods

PXE (network) - IB (In Band)

Virtual Media - OOB (Out Of Band)

Types of Image

Deploy image (Deploy ramdisk)

User image (Boot ramdisk, Instance boot image, OS instance)

Deploy methods

iSCSI

Ironic Python Agent (http/https)


CIFS/NFS


How does iscsi_irmc driver work?


Bare Metal

ServerIronic Conductor

BMC

Image Service Ironic API

depoy.iso

floppy.img

Instance

Boot Image

Management Network

Tenant Network

1

Local

disk

1) Create virtual floppy and copy it into CIFS/NFS

CIFS/NFS




Bare Metal


BMC


depoy.iso

floppy.img

Instance

Boot Image

Management Network

Tenant Network

1

2

Local

disk

Mount cd/fd


2) Attach virtual cdrom and floppy

CIFS/NFS




Bare Metal


BMC


depoy.iso

floppy.img

Instance

Boot Image

Management Network

Tenant Network

1

2, 3

Local

disk

Mount cd/fd



3) Boot from virtual cdrom

CIFS/NFS




Bare Metal


BMC


depoy.iso

floppy.img

Instance

Boot Image

Management Network

Tenant Network

1

2, 3

4

Local

disk

Mount cd/fd




4) Export local disk as iscsi target,

Call Ironic API to continue




Bare Metal


BMC

Image Service Ironic APIInstance

Boot Image

Management Network

Tenant Network

1

2, 3

45

Local

disk

Mount cd/fd





Call Ironic API to continue5) Dispatch Ironic API call to conductor

CIFS/NFS

depoy.iso

floppy.img




Bare Metal


BMC


Boot Image

Management Network

Tenant Network

1

2, 3

45

6

Local

disk

Mount cd/fd





Call Ironic API to continue5) Dispatch Ironic API call to conductor

6) Call Image service

CIFS/NFS

depoy.iso

floppy.img




Bare Metal


BMC


Boot Image

Management Network

Tenant Network

1

2, 3

45

6

7







7) Download boot image

Local

disk

Mount cd/fd

5) Dispatch Ironic API call to conductor

CIFS/NFS

depoy.iso

floppy.img




Bare Metal


BMC


Boot Image

Management Network

Tenant Network

1

2, 3

45

6

7

8

8) Attach local disk by iscsi,

DD boot image to local disk

Local

disk

Mount cd/fd









CIFS/NFS

depoy.iso

floppy.img




Bare Metal


BMC


Boot Image

Management Network

Tenant Network

1

2, 3, 9

45

6

7

8

8) Attach local disk by iscsi,

DD boot image to local disk9) Boot from local disk

Local

disk

Mount cd/fd

9









CIFS/NFS

depoy.iso

floppy.img


How does agent_irmc driver work?


Bare Metal


BMC


Boot Image

Management Network

Tenant Network

1

Local

disk


CIFS/NFS

depoy.iso

floppy.img




Bare Metal


BMC


Boot Image

Tenant Network

1

2

Local

disk

Mount cd/fd



Management Network

CIFS/NFS

depoy.iso

floppy.img

CIFS/NFS




Bare Metal


BMC


depoy.iso

floppy.img

Instance

Boot Image

Tenant Network

1

2, 3

Local

disk

Mount cd/fd




Management Network




Bare Metal


BMC


Boot Image

Tenant Network

1

2, 3

4

Local

disk

Mount cd/fd




4) Export IPA (Ironic Python Agent) API

Call Ironic API to heartbeat

Management Network

CIFS/NFS

depoy.iso

floppy.img




Bare Metal


BMC


Boot Image

Tenant Network

1

2, 3

45

Local

disk

Mount cd/fd







Management Network

CIFS/NFS

depoy.iso

floppy.img




Bare Metal


BMC


Boot Image

Tenant Network

1

2, 3

456

Local

disk

Mount cd/fd







6) Call IPA API to start boot image download

Management Network

CIFS/NFS

depoy.iso

floppy.img




Bare Metal


BMC


Boot Image

Tenant Network

1

2, 3

456

Local

disk

7

Mount cd/fd








7) Download boot image by HTTP to local disk

Management Network

CIFS/NFS

depoy.iso

floppy.img




Bare Metal


BMC


Boot Image

Tenant Network

1

2, 3

456, 8

Local

disk

7

Mount cd/fd









8) Call IPA API to see if deploy has been done

Management Network

CIFS/NFS

depoy.iso

floppy.img




Bare Metal


BMC


Boot Image

Tenant Network

1

2, 3, 9

456, 8









8) Call IPA API to see if deploy has been done

9) Boot from local disk

Local

disk

7

Mount cd/fd

9

Management Network

CIFS/NFS

depoy.iso

floppy.img

TOC

Introduction

Who am I?


Vision




Contribution



Conclusion





• Abort Task




Usecases of Soft Power Off and Inject NMI*

In what situation or scenario does Soft Power Off help?

Unscheduled Hardware Maintenance, because cloud provider cannot logon customer’s instance.

Scheduled Hardware Maintenance, but customer didn’t shutdown

In what situation or scenario does Inject NMI help?

Cloud provider support can ask customer to provide OS dump to resolve customer's incident quickly and accurately.

Customer can investigate problem by themselves with keeping sensitive business data such as credit card number.


*NMI: Non-maskable interrupt https://en.wikipedia.org/wiki/Non-maskable_interrupt

Benefits of Soft Power Off and Inject NMI

Soft Power Off protects customer’s data

Current Power Control is “hard” only. Imagine in-memory database is running, it’s very dangerous operation!• ironic node-set-power-state off

Soft Power Off shuts down OS gracefully, and it’s abortable• ironic node-set-power-state soft_off

• ironic node-set-power-state abort_soft_off

Inject NMI enables responsive support

Inject NMI behaves like reboot, and take OS dump when reboot has done• ironic node-set-power-state inject_nmi


Soft Power Off and Inject NMI

Power State and Target Power State

Power Control is so basic, but not simple and easy to implement

$ ironic node-show-states $NODE_UUID

+------------------------+---------------------------+

| Property | Value |

+------------------------+---------------------------+

| target_power_state | None |

| target_provision_state | None |

| last_error | None |

| console_enabled | False |

| provision_updated_at | 2015-10-01T05:20:15+00:00 |

| power_state | power off |

| provision_state | available |

+------------------------+---------------------------+42 Copyright 2015 FUJITSU LIMITED

Power On | Power Off | Error

power on | power off

soft power off | inject NMI


Current Implementation

No Power Control Finite State Machine such as Deployment State


Power ON Power OFF

Power ON

Error

Timeout | IOError

Reboot = Power Cycle (Power OFF + Power ON) Stable State Existing Target State

Power OFF


Proposed Implementation

Create Power Control Finite State Machine such as Deployment State

The most difficult part is to support Abort


Power ON Power OFF

Power ON

Power OFF SOFT

Inject NMI Error

Abort | Timeout | IOError

Reboot = Power Cycle (Power OFF + Power ON) New Target StateStable State Existing Target State

Power OFF


How to implement Abort

How to handle Abort/Cancel/Timeout/Exception of background task are common problem in concurrent programming

How should we implement in eventlet green thread?• CSP (Communication Sequential Process) Channel

• Channel Registry such as Erlang process registry


Database

node state

node state

node state

Ironic Conductor

Ironic API

Channel Registry

channelnode uuid

channelnode uuid

channelnode uuid

Soft Power OFF Task (green thread)

AbortTask (green thread)

1) Soft Power Off

Exit

Abort Message







Database

node state

node state

node state

Ironic Conductor

Ironic API

Channel Registry

channelnode uuid

channelnode uuid

channelnode uuid



1) Soft Power Off

Exit

2) Exclusive Node Lock

Abort Message







Database

node state

node state

node state

Ironic Conductor

Ironic API

Channel Registry

channelnode uuid

channelnode uuid

channelnode uuid



1) Soft Power Off

Exit


3) Get Chan

Abort Message







Database

node state

node state

node state

Ironic Conductor

Ironic API

Channel Registry

channelnode uuid

channelnode uuid

channelnode uuid



1) Soft Power Off

4) Read Chan

Exit


3) Get Chan

Abort Message







Database

node state

node state

node state

Ironic Conductor

Ironic API

Channel Registry

channelnode uuid

channelnode uuid

channelnode uuid



1) Soft Power Off

4) Read Chan

Exit


3) Get Chan

5) AbortAbort Message







Database

node state

node state

node state

Ironic Conductor

Ironic API

Channel Registry

channelnode uuid

channelnode uuid

channelnode uuid



1) Soft Power Off

4) Read Chan

Exit


3) Get Chan


6) Get Chan







Database

node state

node state

node state

Ironic Conductor

Ironic API

Channel Registry

channelnode uuid

channelnode uuid

channelnode uuid



1) Soft Power Off

4) Read Chan

Exit


3) Get Chan


6) Get Chan

7) Send Message







Database

node state

node state

node state

Ironic Conductor

Ironic API

Channel Registry

channelnode uuid

channelnode uuid

channelnode uuid



1) Soft Power Off

4, 8) Read Chan

Exit


3) Get Chan


6) Get Chan

7) Send Message







Database

node state

node state

node state

Ironic Conductor

Ironic API

Channel Registry

channelnode uuid

channelnode uuid

channelnode uuid



1) Soft Power Off

4, 8) Read Chan

Exit


3) Get Chan


6) Get Chan

7) Send Message

9) If Abort

Message







Database

node state

node state

node state

Ironic Conductor

Ironic API

Channel Registry

channelnode uuid

channelnode uuid

channelnode uuid



1) Soft Power Off

4, 8) Read Chan

Exit


10) Unock

Node

3) Get Chan


6) Get Chan

7) Send Message

9) If Abort

Message

TOC

Introduction

Who am I?


Vision




Contribution



Conclusion





• Abort Task




Multi Tenant, and

Networked storage

Environment

Rescue Mode in Tenant Network

Rescue Usecase in Multi Tenant Support

The instance image is deployed by pxe (In Band) boot and flip network


Management Network

Tenant Network

Bare Metal

Server 1

BMC

Cinder

Instance

Boot Image

Deploy Network

L2 Switch

NeutronIronic ConductorDeploy

Image

What if the instance is damaged?

Flip NetworkPXE boot

Multi Tenant, and

Networked storage

Environment


Multi Tenant Network Support – Provider Network

Rescue Image needs Rescue Network and Tenant Network

Bare Metal Server 1 now has different network configuration from the production environment which could make rescue difficult


Ironic Conductor

Tenant Network

Bare Metal

Server 1

BMC

Cinder

Instance

Boot Image

Rescue Network

Rescue

Image

L2 Switch

Neutron

Management Network

Fix the damaged instance

in different network configuration

Multi Tenant, and

Networked storage

Environment


Virtual Media Boot provides Out Of Band Rescue Modewith the same tenant network configuration as the real Instance


Management Network

Tenant Network

Bare Metal

Server 1

BMC

Rescue Network

L2 Switch

Neutron

CIFS/NFS

rescue.iso

Ironic ConductorRescue

Image

Fix the damaged instance

in same network configuration

Cinder

Instance

Boot Image

TOC

Introduction

Who am I?


Vision




Contribution



Conclusion





• Abort Task




Multi Tenant, and

Networked storage

Environment

CIFS/NFS

Bare Metal N+1 Redundancy

Cold Migration by Soft Power Off and Virtual Media Boot


Ironic Conductor Bare Metal

Server 2

BMC

Cinder

Instance

Boot Image

Management Network

Tenant Network

Bare Metal

Server N

BMC

Bare Metal

Server 1

BMC

Bare Metal

Server N+1

BMC

…

migration.iso

floppy.img

1) Bare Metal Server 1 is running normally 3) % ironic cold-migration “Bare Metal Server 1”

2) A sign of failure is detected

Spare server

Multi Tenant, and

Networked storage

Environment

CIFS/NFS





Server 2

BMC

Cinder

Instance

Boot Image

Management Network

Tenant Network

Bare Metal

Server N

BMC

Bare Metal

Server 1

BMC

Bare Metal

Server N+1

BMC

…

migration.iso

floppy.img3) % ironic cold-migration “Bare Metal Server 1” 4) Graceful shutdown by

Soft Power Off

Spare server

Multi Tenant, and

Networked storage

Environment

CIFS/NFS





Server 2

BMC

Cinder

Instance

Boot Image

Management Network

Tenant Network

Bare Metal

Server N

BMC

Bare Metal

Server 1

BMC

Bare Metal

Server N+1

BMC

…

migration.iso

floppy.img

5) Boot migration.iso

from Virtual Media,

and set IO to attach Cinder

Spare server

3) % ironic cold-migration “Bare Metal Server 1”

Multi Tenant, and

Networked storage

Environment

CIFS/NFS





Server 2

BMC

Cinder

Instance

Boot Image

Management Network

Tenant Network

Bare Metal

Server N

BMC

Bare Metal

Server 1

BMC

Bare Metal

Server N+1

BMC

…

migration.iso

floppy.img

6) Reboot from the same instance boot

Image in Cinder

Spare server

3) % ironic cold-migration “Bare Metal Server 1”

Recap


Mid Term VisonCostumer Values Contribution

Long Term Vison: Eventually for Mission Critical

Truly Open

Reliable

Truly open,

no vendor lock-in

Reliable, robust,

highly available system

Responsive, responsible,

competent support


Rescue Mode

in Tenant Network

Soft Power Off and

Inject NMI

Bare Metal

N+1 Redundancy

Proactive

Reactive

Conclusion

Thanks!

Q&A


Let’s make Ironic great product together!

66

Software

Ironic Towards Truly Open and Reliable, Eventually for Mission Critical