View
285
Download
7
Category
Preview:
Citation preview
FlexFrame® Orchestrator
Version 1.1A
Administration and Operation
Edition June 2015
Document Version 1.2
Fujitsu Limited
© Copyright Fujitsu Technology Solutions 2015
FlexFrame ®
and PRIMERGY™ are trademarks or registered trademarks of Fujitsu
Limited in Japan and other countries.
SAP® and NetWeaver™ are trademarks or registered trademarks of SAP AG in
Germany and in several other countries
Linux® is a registered trademark of Linus Torvalds
SUSE® Linux is a registered trademark of Novell, Inc., in the United States and
other countries
Oracle™ and Java™ are trademarks of ORACLE Corporation and/or its affiliates
Intel® and PXE
® are registered trademarks of Intel Corporation in the United States
and other countries
MaxDB® is a registered trademark of MySQL AB, Sweden
MySQL® is a registered trademark of MySQL AB, Sweden
NetApp® and the Network Appliance
® logo are registered trademarks and Network
Appliance™ and Data ONTAP™ are trademarks of NetApp, Inc. in the U.S. and
other countries.
EMC®
and Celerra™ are trademarks or registered trademarks of EMC Corporation
in the United States and other countries.
VMware®, ESX
®, ESXi, VMware vCenter, VMware vSphere are registered
trademarks or trademarks of VMware, Inc. in the United States and/or other
jurisdictions.
Ethernet® is a registered trademark of XEROX, Inc., Digital Equipment Corporation
and Intel Corporation
Windows® and Word
® are registered trademarks of Microsoft Corporation
All other hardware and software names used are trademarks of their respective
companies.
All rights, including rights of translation, reproduction by printing, copying or similar
methods, in part or in whole, are reserved.
Offenders will be liable for damages.
All rights, including rights created by patent grant or registration of a utility model or
design, are reserved.
Delivery subject to availability. Right of technical modification reserved.
Administration and Operation
Contents
1 Introduction ..................................................................................................... 1 1.1 Requirements .................................................................................................... 1 1.2 Notational Conventions ..................................................................................... 1 1.3 Document History .............................................................................................. 2 1.4 Related Documents ........................................................................................... 2 1.5 Special Hints for FlexFrame
® 1.1A .................................................................... 2
1.5.1 Incompatibilities (command scripts) .................................................................. 2 1.5.2 Displaced subjects ............................................................................................ 3 1.5.2.1 ServerView Operations Manager ...................................................................... 3 1.5.2.2 Support of EMC Celerra .................................................................................... 3
2 FlexFrame Architecture .................................................................................. 5 2.1 General Notes on FlexFrame ............................................................................ 5 2.2 Hardware and Software ..................................................................................... 6 2.3 Shared Operating System ................................................................................. 9 2.3.1 Shared OS Boot Concept .................................................................................. 9 2.3.2 Control Nodes ................................................................................................... 9 2.3.3 Application Nodes ........................................................................................... 10 2.3.4 Virtual Control Center ...................................................................................... 10 2.4 FlexFrame Structure in LDAP.......................................................................... 11 2.4.1 Working with LDAP ......................................................................................... 13 2.5 Linux-HA Cluster on Control Center ................................................................ 13 2.5.1 Terminology..................................................................................................... 13 2.5.2 Simple Resources ........................................................................................... 15 2.5.2.1 Constraints ...................................................................................................... 15 2.5.2.2 Score ............................................................................................................... 15 2.5.2.3 Stickiness ........................................................................................................ 16 2.5.2.4 Resource Groups ............................................................................................ 18 2.5.3 FlexFrame Specific Configuration ................................................................... 18 2.5.3.1 Configuration of Limits for Cluster Resources ................................................. 19 2.5.3.2 LDAP Configuration ......................................................................................... 20 2.5.3.3 Resource ff_manage ....................................................................................... 22 2.5.3.4 Resource Netboot ........................................................................................... 22 2.5.3.5 STONITH (Physical Control Center) ................................................................ 23 2.5.3.6 STONITH (Virtual Control Center) ................................................................... 25 2.5.3.7 Configuring the Cluster .................................................................................... 26 2.5.3.8 Starting Behavior of the Cluster ....................................................................... 26 2.5.3.9 Status Information of the Clusters ................................................................... 27 2.5.4 Linux-HA CLI Commands ................................................................................ 27 2.5.5 Linux-HA Logfile .............................................................................................. 29 2.5.6 Linux-HA GUI .................................................................................................. 29 2.6 Network ........................................................................................................... 29
Contents
Administration and Operation
2.6.1 LAN Failover .................................................................................................... 30 2.6.2 Segments ........................................................................................................ 30 2.6.3 Network Switches ............................................................................................ 31 2.6.4 Network Speed ................................................................................................ 31 2.6.5 Network Switch Groups ................................................................................... 32 2.6.6 Network Switch Ports ...................................................................................... 32 2.6.7 Automounter Concept ...................................................................................... 33 2.7 Storage Systems ............................................................................................. 35 2.7.1 NAS systems ................................................................................................... 36 2.7.1.1 Built-in Cluster File System .............................................................................. 37 2.7.1.2 Volume Layout ................................................................................................. 37 2.7.1.3 Snapshots ........................................................................................................ 37 2.7.1.4 vFiler ................................................................................................................ 38 2.7.1.5 Storage Virtual Machines (SVMs) .................................................................... 38 2.7.2 SAN Support .................................................................................................... 40 2.7.2.1 SAN Basic Layers ............................................................................................ 40 2.7.2.2 Rules and Restrictions ..................................................................................... 41
3 FlexFrame Basic Administration .................................................................. 42 3.1 Accessing a FlexFrame Landscape (Remote Administration) ......................... 42 3.2 Powering up the FlexFrame Landscape .......................................................... 42 3.3 Powering off the FlexFrame Landscape .......................................................... 43 3.4 Reactivating ANs after Power Shutdown by FlexFrame Agents ...................... 46 3.5 Displaying the Current FlexFrame Configuration State ................................... 47 3.5.1 FlexFrame Web Portal ..................................................................................... 48 3.5.2 FlexFrame Agents ........................................................................................... 49 3.5.2.1 State of Pools .................................................................................................. 49 3.5.2.2 State of Application Nodes .............................................................................. 49 3.5.2.3 State of SAP Systems ..................................................................................... 50 3.5.2.4 State of SID Instances ..................................................................................... 50 3.5.2.5 Virtual IP address for FlexFrame Agents WebGUI .......................................... 50 3.5.3 Networks.......................................................................................................... 51 3.5.4 Cluster Status .................................................................................................. 53 3.5.5 Supported Hardware ....................................................................................... 53 3.5.6 Management Blades ........................................................................................ 54 3.6 FlexFrame Backup with Tape Library .............................................................. 54 3.7 Product Licenses ............................................................................................. 55 3.7.1 How to Work with Licenses .............................................................................. 55 3.7.2 Types of Licenses in FlexFrame Orchestrator ................................................. 55 3.7.3 How to Install a License Key ............................................................................ 56 3.7.4 How to List Installed License Keys .................................................................. 56 3.7.5 How to Control the Use of Licenses ................................................................ 57
4 Pools and Groups .......................................................................................... 58 4.1 Adding a Pool .................................................................................................. 58
Contents
Administration and Operation
4.2 Removing a Pool ............................................................................................. 61 4.3 Listing Pool Details .......................................................................................... 63 4.4 Listing All Pools ............................................................................................... 66 4.5 Changing Pool DNS Domain ........................................................................... 68 4.6 Changing Pool Default Router ......................................................................... 69 4.7 Adding a Group to a Pool ................................................................................ 70 4.8 Removing Pool Group ..................................................................................... 71 4.9 Hosts Database ............................................................................................... 71 4.9.1 Script: ff_hosts.sh ............................................................................................ 72
5 User and Groups Administration ................................................................. 74 5.1 Create, Modify, Delete, or List User(s) for Application Nodes ......................... 74 5.2 Creating, Modifying, Deleting or Listing Group(s) for Application Nodes ......... 76 5.3 Creating, Modifying, Deleting or Listing Service(s) for Application Nodes ....... 78
6 Pool-independent Spare Nodes ................................................................... 81 6.1 Creation of Spare Nodes in the ADMINPOOL ................................................. 81 6.2 Moving of a Spare Node .................................................................................. 81 6.3 Handling Pool-independent Spare Nodes with FlexFrame Agents .................. 82
7 Application Nodes Administration ............................................................... 83 7.1 Listing Application Nodes ................................................................................ 83 7.1.1 Displaying Information on a Specific Application Node ................................... 83 7.1.2 Displaying Information of all Application Nodes............................................... 87 7.1.3 Displaying all available OS images ................................................................. 88 7.2 Adding Application Nodes ............................................................................... 88 7.3 Removing Application Nodes .......................................................................... 92 7.4 Changing the attributes of an Application Node .............................................. 93 7.4.1 Change the group of an Application Node ....................................................... 93 7.4.2 Change the os image of an Application Node ................................................. 94 7.4.3 Change the mac addresses of an Application Node ........................................ 95 7.5 Moving Application Nodes ............................................................................... 95 7.6 Administrating Blade Server Cabinets ............................................................. 97 7.6.1 Listing Blade Server Cabinets ......................................................................... 97 7.6.1.1 Displaying Information on a Specific Blade Server Cabinet ............................ 97 7.6.1.2 Displaying Information on all Configured Blade Server Cabinets .................... 98 7.6.2 Adding Blade Server Cabinets ........................................................................ 99 7.6.3 Removing Blade Server Cabinets ................................................................. 102 7.6.4 Changing Switch Blade Type ........................................................................ 103 7.6.5 Changing Switch Blade Name ....................................................................... 103 7.6.6 Changing Switch Blade Password ................................................................. 104 7.6.7 Getting Switch Blade Configuration ............................................................... 105 7.6.8 Change Switch Blade Uplink ......................................................................... 106 7.6.9 Move a Blade Cabinet to Another Switch Group ........................................... 107
Contents
Administration and Operation
7.7 Administrating Hypervisor Nodes and Virtual Machines ................................ 109 7.7.1 Getting Started with Hypervisor Nodes and VMs ........................................... 110 7.7.2 Virtualization related global FlexFrame parameters ...................................... 111 7.7.2.1 System Code for Hypervisor Nodes and VMs ............................................... 111 7.7.2.2 vCenter Server .............................................................................................. 112 7.7.2.3 Control LAN Default Router for Hypervisor Nodes ........................................ 114 7.7.3 Adding Hypervisor Nodes .............................................................................. 114 7.7.4 Completing Hypervisor Node Configuration................................................... 117 7.7.5 Removing Hypervisor Nodes ......................................................................... 118 7.7.6 Displaying Information about HNs and VMs .................................................. 119 7.7.7 Hypervisor Nodes and Pools ......................................................................... 122 7.7.8 Special Functions for Virtual Machines .......................................................... 123 7.7.9 Virtual Machine Properties and Hypervisor Resources ................................. 125 7.7.10 Using Hypervisor Management Software ...................................................... 127 7.8 Power on / off / reboot of an Application Node .............................................. 128
8 Storage Systems Administration ............................................................... 129 8.1 NAS Systems Configuration .......................................................................... 129 8.1.1 Adding a New NAS ........................................................................................ 129 8.1.2 Removing a NAS ........................................................................................... 131 8.1.3 Displaying all configured NAS ....................................................................... 132 8.1.4 Displaying NAS Configuration ....................................................................... 134 8.1.5 Adding a Pool to a NAS ................................................................................. 135 8.1.6 Removing a Pool from a NAS ........................................................................ 135 8.1.7 Create NAS HA Partnership .......................................................................... 136 8.1.8 Moving a NAS ................................................................................................ 137 8.1.9 Changing NAS Command Shell .................................................................... 138 8.1.10 Changing NAS Ports ..................................................................................... 139 8.1.11 Changing NAS Path Export ........................................................................... 140 8.2 Using SAN Storage ....................................................................................... 141 8.2.1 Getting started with SAN usage in FlexFrame ............................................... 142 8.2.2 Administrating Volume Groups and LUNs ..................................................... 146 8.2.2.1 Adding a Volume Group ................................................................................ 147 8.2.2.2 Adding a LUN ................................................................................................ 149 8.2.2.3 Removing a Volume Group ........................................................................... 151 8.2.2.4 Removing a LUN ........................................................................................... 152 8.2.2.5 Changing Poolgroup Assignment of a Volume Group ................................... 153 8.2.2.6 Displaying Information about Volume Groups and LUNs............................... 153 8.2.3 Completing SAN Setup for a SAP System..................................................... 157 8.2.4 Mounting SAN based File Systems of a SAP Service ................................... 158 8.2.5 SAN Helper Functions ................................................................................... 159
9 Switch Administration ................................................................................. 162 9.1 Adding a Switch to a Switch Group ............................................................... 162 9.2 Removing a Switch from a Switch Group ...................................................... 165
Contents
Administration and Operation
9.3 Listing a Switch Group Configuration ............................................................ 166 9.4 Generating a Switch Group Configuration ..................................................... 168 9.5 Changing the Password of a Switch Group ................................................... 169 9.6 Changing the Host Name of a Switch Group ................................................. 170 9.7 Displaying/Changing Common Network Configuration Parameters .............. 172 9.8 Adding a Switch Group .................................................................................. 174 9.9 Adding an Expansion Module ........................................................................ 177 9.10 Removing an Expansion Module ................................................................... 178 9.11 Removing a Switch Group ............................................................................. 179 9.12 Adding an Uplink to Switch Group ................................................................. 180 9.13 Extend an Uplink of Switch Group ................................................................. 181 9.14 Delete an Uplink of Switch Group .................................................................. 183 9.15 Migrating Switches of a Switch Group ........................................................... 184 9.16 Adding a Switch Port Configuration ............................................................... 185 9.17 Removing a Switch Port Configuration .......................................................... 187 9.18 Displaying a Switch Port Configuration ......................................................... 188 9.19 Displaying the Complete Switch Port Configuration ...................................... 189 9.20 Moving Device Connection to Core Switch.................................................... 191 9.20.1 Move Control Center to Core Switch ............................................................. 191 9.20.2 Move Client LAN to Core Switch ................................................................... 192 9.20.3 Move NAS System to Core Switch ................................................................ 192 9.20.4 Move Application Node to Core Switch ......................................................... 193 9.20.5 Move ESX Server to Core Switch .................................................................. 193 9.20.6 Move BX Cabinet to Core Switch .................................................................. 194
10 SAP System Handling ................................................................................. 195 10.1 Listing SAP SIDs and Instances .................................................................... 196 10.2 Updating System Configuration Files ............................................................ 198 10.3 Adding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)198 10.4 Removing SAP SIDs and Instances .............................................................. 205 10.5 Adding/Removing SAP SIDs (add-on services) ............................................. 206 10.5.1 BOBJ – Business Objects ............................................................................. 206 10.5.2 Content Server (CMS) ................................................................................... 207 10.5.3 MDM – Master Data Management ................................................................ 211 10.5.4 SMD – Solution Manager Diagnostics ........................................................... 214 10.5.5 TREX (Search and Classification Service) .................................................... 218 10.5.6 WebDispatcher .............................................................................................. 220 10.5.7 HANA database (service hdb) ....................................................................... 222 10.5.8 SAP classic NetWeaver services using HANA database .............................. 227 10.6 Cloning a SAP SID into a Different Pool ........................................................ 228 10.6.1 Script: ff_clone_sid.pl .................................................................................... 228 10.6.2 Changing User and Group IDs after Cloning ................................................. 229 10.7 Cloning a SAP system into a different Pool ................................................... 230 10.7.1 Script: ff_clone_sap.sh .................................................................................. 230 10.7.1.1 Cloning SAP systems with MaxDB ................................................................ 235
Contents
Administration and Operation
10.7.1.2 Post cloning steps – DB2............................................................................... 235 10.7.1.3 Post cloning steps – Oracle ........................................................................... 236 10.7.1.4 Post cloning steps – MaxDB .......................................................................... 236 10.7.1.5 Post cloning steps – HANA reference database ............................................ 236 10.7.1.6 Cloning a SAP system example session ....................................................... 237 10.8 Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl .............. 238 10.8.1 Hints for usage .............................................................................................. 242 10.8.1.1 General .......................................................................................................... 242 10.8.1.2 Creating a SID/relocation of sapdata/saplog ................................................. 242 10.8.1.3 Creating a SID/relocation of SID specific data ............................................... 243 10.9 Multiple NAS Systems and Multiple Volumes ................................................ 243 10.9.1 NetApp NAS Systems ................................................................................... 244 10.10 Upgrading a SAP System .............................................................................. 245 10.10.1 Service Port ................................................................................................... 245 10.10.2 FlexFrame Agents ......................................................................................... 246 10.10.3 Support SAP Upgrade (ff_sap_upgrade.pl) ................................................... 246 10.11 SAP Kernel Updates and Patches ................................................................. 249 10.12 Unloading volFF ............................................................................................ 249 10.13 Administrating SAP Services ......................................................................... 250 10.13.1 Displaying Status of SAP Services ................................................................ 250 10.13.1.1 FlexFrame Agents WebGUI ........................................................................... 250 10.13.1.2 List SAP Services .......................................................................................... 251 10.13.2 Starting and Stopping Application Services ................................................... 252 10.13.2.1 Configuration file (ff_service.config) .............................................................. 253 10.13.2.2 Starting and Stopping SAP Services Without Root Privileges ....................... 255 10.13.2.3 Starting and Stopping SAP Services from Control Node using RBAC ........... 256 10.13.2.4 SAP Service Scripts ...................................................................................... 257 10.13.2.5 SAP Service Script Actions ............................................................................ 259 10.13.2.6 SAP Service Script logging ............................................................................ 261 10.13.2.7 SAP Service Script User Exits ....................................................................... 261 10.13.2.8 SAP service script ff_smd_agent.sh .............................................................. 262 10.13.3 Return Code of the SAP Service Script ......................................................... 262 10.13.4 Starting and Stopping Multiple SAP Services ................................................ 262 10.13.5 Removing an Application from Monitoring by FlexFrame Agents .................. 264 10.13.5.1 Stopping and Starting an Application for Upgrades Using r3up ..................... 265 10.13.6 Service Switchover ........................................................................................ 265 10.13.7 Use ServicePings from FlexFrame Agents .................................................... 267
11 Software and Hardware Upgrade and Maintenance ................................. 269 11.1 Upgrading the Entire FlexFrame Landscape ................................................. 269 11.2 Software Upgrade on the Control Node ......................................................... 269 11.2.1 FlexFrame Control Node Software ................................................................ 270 11.2.2 Service Packs ................................................................................................ 271 11.2.3 Updating/Installing a New Linux Kernel ......................................................... 271 11.2.3.1 Software Stage .............................................................................................. 271
Contents
Administration and Operation
11.2.3.2 Install the New Kernel ................................................................................... 272 11.2.3.3 Reboot the Control Node ............................................................................... 272 11.2.3.4 Partner Linux Driver Process......................................................................... 272 11.2.4 Updating RPM Packages on an Control Node .............................................. 273 11.2.4.1 Setup a patch server ..................................................................................... 273 11.2.4.2 Connect the patch server to the FlexFrame environment .............................. 273 11.2.4.3 Prepare the Online Update ............................................................................ 273 11.2.4.4 Perform the Online Update ............................................................................ 274 11.2.5 Backup/Restore of FlexFrame Control Nodes ............................................... 274 11.2.5.1 Backup of a Control Node ............................................................................. 275 11.2.5.2 Restore of a Control Node ............................................................................. 275 11.3 Maintenance of the Control Node - Hardware ............................................... 278 11.3.1 Exchanging a Control Node........................................................................... 278 11.3.1.1 Hardware Failed, Hard Disk and Installed OS are not Affected ..................... 278 11.3.1.2 One Hard Disk is Defect, the Other One is Undamaged ............................... 278 11.3.1.3 The Control Nodes OS is Damaged .............................................................. 278 11.3.2 Replacing a Network Card – Control Node ................................................... 279 11.4 Maintenance of the virtual Control Center ..................................................... 279 11.4.1 Site failover within virtual Control Center ....................................................... 279 11.4.1.1 The naming conventions ............................................................................... 279 11.4.1.2 Scenario of sane KVM Hypervisor Nodes ..................................................... 280 11.4.1.3 KVM Hypervisor Node kvm-vcn1 fails (manual switch over) ......................... 282 11.4.1.4 KVM Hypervisor Node kvm-vcn1 fails (automatic switch over)...................... 286 11.4.2 Restore of damaged virtual Control Node ..................................................... 287 11.4.3 Restore of damaged Hypervisor Node .......................................................... 287 11.5 Maintenance of Application Nodes - Software ............................................... 289 11.5.1 Introduction.................................................................................................... 289 11.5.2 Schematic Overview ...................................................................................... 289 11.5.3 Installing Application Node Images from Installation Media .......................... 291 11.5.3.1 Installing the Application Node Image ........................................................... 291 11.5.4 Step #1: Creating a Maintenance Image ....................................................... 293 11.5.5 Step #2: Assigning the Maintenance Image, booting and maintaining .......... 295 11.5.5.1 Using the Node .............................................................................................. 296 11.5.5.2 Booting .......................................................................................................... 297 11.5.5.3 Maintaining .................................................................................................... 298 11.5.6 Step #3: Reverting the Maintenance Image .................................................. 299 11.5.7 Migrating remaining Application Nodes ......................................................... 302 11.5.8 Re-Using the Maintenance Image ................................................................. 304 11.5.9 Maintaining Use Cases ................................................................................. 306 11.5.9.1 Service Packs ................................................................................................ 306 11.5.9.2 Updating/Installing a New Linux Kernel ......................................................... 306 11.5.9.3 ServerView Update ....................................................................................... 309 11.5.9.4 Upgrading the Application Software .............................................................. 310 11.5.9.5 Updating RPM Packages on an Application Node ........................................ 311 11.5.9.6 Updating vmware-tools .................................................................................. 312
Contents
Administration and Operation
11.5.10 Implement conflicting changes using the ff_new_an.custom hook ................ 313 11.6 Maintenance of the Application Nodes - Hardware ........................................ 317 11.6.1 Changing BIOS Settings for Netboot ............................................................. 317 11.6.2 Replacing a Network Card – Application Node .............................................. 317 11.6.3 Replacing a Switch Blade .............................................................................. 319 11.6.4 Replacing Power Control Hardware .............................................................. 319 11.7 Maintenance of ESXi Servers ........................................................................ 320 11.7.1 BIOS Updates ................................................................................................ 320 11.7.2 Replacing a Network Card - ESXi Server ...................................................... 320 11.7.3 ESXi Updates and Patches ........................................................................... 320 11.8 Maintenance of Other FlexFrame Components ............................................. 321 11.8.1 NetApp Storage ............................................................................................. 321 11.8.2 Switches and Switch Blades .......................................................................... 321 11.8.2.1 Firmware Update ........................................................................................... 321 11.8.2.2 Backup of Switch Configurations ................................................................... 321 11.8.2.3 Restore of Switch Configurations .................................................................. 322 11.8.3 DNS Servers .................................................................................................. 323 11.8.4 Third Party Products ...................................................................................... 323 11.9 FlexFrame Agents ......................................................................................... 324 11.10 Project specific changes ................................................................................ 324 11.10.1 The ff_projectinfo.sh command ..................................................................... 325
12 Troubleshooting .......................................................................................... 328 12.1 Locking of Flex Frame Administration Commands ........................................ 328 12.2 Script Logging in FlexFrame .......................................................................... 329 12.3 RBAC Logging with Syslog ............................................................................ 331 12.4 Log Files ........................................................................................................ 331 12.5 Network Errors ............................................................................................... 333 12.6 NFS Mount Messages ................................................................................... 333 12.7 LDAP Error Codes and Messages ................................................................. 333 12.8 LDAP and Cache Coherence ........................................................................ 335 12.8.1 Linux .............................................................................................................. 335 12.9 Start/Stop Script Errors .................................................................................. 335 12.9.1 Severity ‘INFO’ .............................................................................................. 336 12.9.2 Severity ‘WARN’ ............................................................................................ 336 12.9.3 Severity ‘ERROR’ .......................................................................................... 336 12.9.4 Severity 'DEBUG' .......................................................................................... 336 12.10 Script Debugging ........................................................................................... 336 12.10.1 Shell Scripts ................................................................................................... 336 12.10.2 Perl Scripts .................................................................................................... 337 12.11 Debugging the Linux Kernel .......................................................................... 337 12.11.1 Netconsole ..................................................................................................... 337 12.11.2 Capturing Crash Dumps ................................................................................ 338 12.11.2.1 Common Restrictions for Taking Crash Dumps ............................................. 338 12.11.2.2 Kernel Crash Dump Capturing (Kdump) ........................................................ 338
Contents
Administration and Operation
12.11.2.3 Forcing a Crash Dump .................................................................................. 338 12.12 Activate core dumps on CN or AN ................................................................. 339 12.13 Huge Amount of getattr Calls ........................................................................ 340
13 Abbreviations .............................................................................................. 341
14 Glossary ....................................................................................................... 345
15 Index ............................................................................................................. 351
Administration and Operation 1
1 Introduction
This document provides instructions on administrating and operating an installed
FlexFrame® 1.1A environment. It focuses on general aspects of the architecture as well
as on software updates, hardware extensions and FlexFrame-specific configuration.
It does not cover the installation of an entire FlexFrame environment. Please refer to the
“Installation of a FlexFrame Environment” manual for information on initial installation.
1.1 Requirements
This document addresses administrators on FlexFrame environments. We assume that
the reader of this document has technical background knowledge in the areas of
operating systems (Linux®), IP networking and SAP
® basis.
1.2 Notational Conventions
The following conventions are used in this manual:
Additional information that should be observed.
Warning that must be observed.
fixed font Names of paths, files, commands, and system output.
<fixed font> Names of variables.
fixed font User input in command examples (if applicable using <> with variables).
Command prompt: # The notation
control1:/<somewhere> # <command>
indicates that the command <command> is issued on the first
Control Node in the directory /<somewhere>.
The reader may need to change into the directory first, e.g.
control1:~ # cd /<somewhere>
control1:/<somewhere># <command>.
Introduction Document History
2 Administration and Operation
1.3 Document History
Document
Version
Changes Date
1.0 First Edition 2014-12-03
1.1 Update with small corrections 2015-02-23
1.2 vCC support 2015-06-30
1.4 Related Documents
FlexFrame® – HW Characteristics Quickguides
FlexFrame® – Installation and Configuration of LVM 2.1 Standard Edition
FlexFrame® – Installation Guide for SAP Solutions
FlexFrame® – Installation of a FlexFrame Environment
FlexFrame® – Management Tool
FlexFrame® – Agents Installation and Administration
FlexFrame® – Messenger Concepts and Usage
FlexFrame® – LogAgent Concepts and Usage
FlexFrame® – Network Design and Configuration Guide
FlexFrame® – Security Guide
FlexFrame® – Technical White Paper
FlexFrame® – Upgrading FlexFrame Orchestrator 1.0A to 1.1A
ServerView Documentation
SUSE Linux Enterprise Server Documentation
1.5 Special Hints for FlexFrame® 1.1A
In this document, you often will find console output, configuration data and
installation examples which are based on earlier FlexFrame versions. Please
keep in mind that these are examples and may look slightly different in
FlexFrame® 1.1A.
The two Control Nodes (CN) of FlexFrame are also named as the FlexFrame Control
Center (CC). In this documentation the notation Control Node (CN) is used as a
synonym for Control Center (CC) and the other way round.
1.5.1 Incompatibilities (command scripts)
There have been some changes with supplied scripts:
Special Hints for FlexFrame® 1.1A Introduction
Administration and Operation 3
The script ff_change_id.pl is not further available.
With the script ff_user_adm.pl resp. ff_group_adm.pl you can change
UIDs or GIDs of a specific user/group
The ff_install_an_linux_images.sh script uses a slightly different syntax:
the arguments to the -t and -s options are now names and not any longer
directories, and the -m option needs an argument.
The command line syntax of ff_setup_sid_folder.sh changed to
ff_setup_sid_folder.sh –s <SID> -p <pool>
With script ff_nas_adm.pl, the operation mode conf10gb is no longer
supported. Use operation mode move instead.
The script ff_esx_adm.pl is no longer available. Use the script ff_hn_adm.pl
instead
The script ff_esx_vm.pl is no longer available. Use the script ff_vm.pl instead
1.5.2 Displaced subjects
1.5.2.1 ServerView Operations Manager
The software component ServerView Operations Manager used to be an element of the
Control Node software stack. This component (allowing to monitor the hardware status of
PRIMERGY servers in your FlexFrame® environment) has been removed from the
Control Node as of FlexFrame 5.3A/1.0A. Customers wanting to use this component
have to install and configure ServerView Operations Manager on an own server. See
Description Paper - Server monitoring and trap forwarding on the Fujitsu Extranet or ask
your Fujitsu Customer Support for getting a copy.
Still left on the Control Node (and also on the Application Nodes) are the still active
software components ServerView Agents and ServerView RAID Manager.
1.5.2.2 Support of EMC Celerra
The support for EMC Celerra had already been frozen based on FlexFrame 5.0A for all
successor versions of Flexframe – now with FlexFrame 1.1A the totally new directory and
automount structure caused by the requirement to support NetApp’s ONTAP clustermode
does not allow to continue this frozen support any further, so FlexFrame Orchestrator
1.1A is not prepared to support EMC Celerra without changes – if needed you should
comission Fujitsu to do a Solution project corresponding to your demands.
Administration and Operation 5
2 FlexFrame Architecture
The FlexFrame solution is a revolutionary approach to run complex SAP solutions with
higher efficiency. At the same time some major changes of the configuration paradigms
for infrastructures have been implemented. These changes are:
A shared operating system booted via IP networks for the SAP Application Nodes.
Decoupling of application software and operating system, called virtualization of
SAP software or Adaptive Computing.
Shared Network Attached Storage from Network Appliance® providing Write
Anywhere File Layout (WAFL™) and sophisticated snap functionality.
FlexFrame Agents providing revolutionary mechanisms to implement high-
availability functions without cluster software
The concept of FlexFrame consists of several components, which implement state-of-the-
art functionality. Together with new components, such as the FlexFrame Agents, the
whole solution is far more than just the sum of its components. A major part of the
benefits consists in a dramatic reduction in day to day operating costs for SAP
environments.
It is, of course, possible to use parts of the FlexFrame solution in project-based
implementations. However, they cannot be called FlexFrame.
2.1 General Notes on FlexFrame
FlexFrame was designed and developed as a platform for SAP Applications. Its major
purpose is to simplify and abstract the basic components to enable the administrator of a
SAP system landscape to focus on SAP and not worry about servers, networking and
storage.
The FlexFrame Control Nodes are seen as an appliance. Like a toaster or a micro wave
oven, they have a well defined purpose and the build-components must work together to
achieve that purpose. It happened that Fujitsu picked SUSE® Linux Enterprise Server
(SLES) as the operating system for the Control Nodes; however it is not intended that the
customer is using it as a regular server, meaning installing additional software on it and
applying patches to it is not wanted, unless Fujitsu support line instructs so. Upcoming
versions of the Control Node's operating system may be totally different and may not
allow modifying anything at all. The installation and backup/restore functionality of a
Control Node are based on fixed images which are delivered on DVD. Modifications of
installed images will not be taken care of, if a new version is installed.
Modifications may even lead to errors which may be hard to find. Therefore we strongly
recommend to the customers not to install any software or patches onto the Control
Nodes without confirmation of Fujitsu support.
FlexFrame Architecture Hardware and Software
6 Administration and Operation
Similar to the Control Nodes are the Application Nodes. Fixed images are shipped for
installation of Linux Application Nodes in the FlexFrame landscape. Another aspect of
FlexFrame is the reduction of the TCO (Total Cost of Ownership). A static approach
(once started, never touched) will not be very efficient. To achieve the best savings, it is
recommended to actively manage where a certain SAP application instance or database
instance is running. If, as an example, a SAP instance requires the power of two CPUs of
an Application Node during most days of a month and eight CPUs during month-end
calculations, it is best to move it back and forth to the appropriate Application Node with
the right size. During the time where the application instance is running on the two CPU
Application Node, another SAP instance can use the bigger eight CPU Application Node
and therefore saving the need to have more eight CPU Application Nodes as in a static
approach.
2.2 Hardware and Software
The FlexFrame solution consists of both, hardware and software. To grant proper
function of the whole landscape, the entire software set is strictly defined. Anything other
than the software components listed below is not part of FlexFrame. This applies
unchanged if software from the list below is missing, is installed in other versions than
below, or if software other than the actual SAP components is added.
For detailed information about the hardware supported in a FlexFrame environment, see
the FlexFrame 1.1A Configuration Guide. Any other functions, such as backup, can be
implemented separately as an add-on to FlexFrame and need dedicated hardware,
operating system, high availability, professional service and support etc.
Hardware and Software FlexFrame Architecture
Administration and Operation 7
No. Hardware 1
OS Software Services
1 Control Nodes:
2 x PRIMERGY RX2540 M1 or
2 x PRIMERGY RX300 S8, S7, S6, or S5, or
2 x PRIMERGY rack servers each equipped with a
KVM hypervisor
SLES11 SP3 plus SLES11
HAE or
SLES for SAP 2
FlexFrame Agents (Control Agents)
V9.0, FlexFrame 1.1A File System
Image CN, ServerView Agents,
ServerView RAID, etc.
TFTP, DHCP, LDAP,
(SAPROUTER), etc.
2 Network switches:
pairs of 1 GbE switches (Cisco Catalyst 3750x or
Brocade ICX 6430 / 6450),
and/or
pairs of 10 GbE switches (Cisco Nexus 5K or
Brocade VDX6740 or 3 )
IOS (proprietary)
NX-OS (proprietary)
Brocade Network OS (propr.)
…
(as delivered)
3 Network Attached Storage:
one or more NetApp storage systems
(FASxxxx, vFiler or SVMs),
disk shelves as required 4, hosting shared OS
file systems, and application data
ONTAP (7-mode and cDOT) 5
(proprietary)
NetApp Tools
NFS, optional:
cluster components,
FlexClone, SnapVault,
SnapMirror,
SnapRestore
4 Intel- based PRIMERGY or PRIMEQUEST servers
(standard rack or blade server)
SLES11 SP3 / SLES for SAP 1
or VMware vSphere ESXi running SLES11 SP3 / SLES for SAP
2
or SLES11 SP3 / SLES for SAP
2
running as KVM Hypervisor
FlexFrame V1.1A File System Image,
FlexFrame Agents (Application Agents),
SAP Applications, Database
SAP & DB Services
1 allowed types according to current FlexFrame Support Matrix available on the Service DVD or ask your Fujitsu Customer Support for getting an up-to-date copy
2 based on SLES11 SP3 (x86_64)
3 or other switch pairs when adapter modules exist and are certified by Fujitsu
4 the amount of disks required for customer-specific FlexFrame configurations can be determined together with Fujitsu Customer Support Filer Sizing Team
5 the currently supported ONTAP version is kept up-to-date in the FlexFrame Support Matrix
FlexFrame Architecture Hardware and Software
8 Administration and Operation
Administration and Operation 9
2.3 Shared Operating System
One major aspect of FlexFrame is its shared operating system. Sharing in this case
means that the very same files of essential parts of the underlying operating system are
used to run multiple Application Nodes. This part of the file system is mounted read-only,
so none of the Application Nodes that run the actual applications can modify it. Server
specific information is linked to a file system area that is server-specific and mounted
read-write. The shared operating system is kept to a NAS system (Network Attached
Storage system from NetApp).
2.3.1 Shared OS Boot Concept
The chart below shows the boot process of a FlexFrame Application Node
(PRIMERGY/Linux):
2.3.2 Control Nodes
A productive FlexFrame landscape always includes two Control Nodes. Their purpose is
to be a single point of control for the Application Nodes, as well as to check and manage
the proper function of the Application Nodes.
FlexFrame Architecture Shared Operating System
10 Administration and Operation
Control Nodes are not running SAP software (with the exception of saprouter, as an
option). They exclusively run SUSE Linux Enterprise Server Version 11 (SLES11 SP3),
installed on local disks. Control Nodes provide and run services such as:
Linux-HA high availablity cluster framework
Timeserver for the complete FlexFrame landscape
Control Agents
Web server to provide the Control Agents user interface
DHCP for assignment of IP addresses and TFTP for the boot process of the
Application Nodes
saprouter (optional)
Control Nodes have to be of the type PRIMERGY RX300 (models S8, ..., S5 resp.
RX2540M1).
As of FlexFrame Orchestrator 1.1A00-020 , the Control Nodes can also be virtual
machines on top of PRIMERGY servers using SUSE Linux KVM as a virtualization layer.
2.3.3 Application Nodes
Application Nodes run database and SAP services on the SUSE Linux Enterprise Server
shared operating system. Application Nodes can be physical servers that offer CPU and
memory, or virtual servers built on top of physical servers, using the physical server's
CPU and memory through a virtualization layer.
For FlexFrame the principal types of Application Nodes are PRIMERGY servers running
Linux directly and virtual servers on top of PRIMERGY servers using VMware ESXi or
SUSE Linux KVM as a virtualization layer.
Admissible PRIMERGY servers have to be approved for SAP on Linux by Fujitsu.
During the boot process using Intel's PXE® technology, each Application Node will be
identified using the hardware address of its boot interface (MAC address). The Control
Node will assign an IP address to it and supply the operating system via the network.
File systems, especially the root file system (/), are mounted via the network in read-only
mode. If, for any reason, an Application Node needs to be replaced or added, only a
handful of settings need to be adjusted to integrate it into the FlexFrame landscape.
Intel's PXE technology is implemented in Fujitsu PRIMERGY servers and allows booting
via the network. DHCP will be used with static MAC address relationship for all the
Application Nodes.
2.3.4 Virtual Control Center
As of FFO 1.1A00-020 the Control Center of FlexFrame can either consist of physical
Control Nodes or of virtual Control Nodes (vCN). In case of a virtual Control Center (vCC)
FlexFrame Structure in LDAP FlexFrame Architecture
Administration and Operation 11
both virtual machines are hosted on different physical PRIMERGY servers that run SUSE
KVM as hypervisor. The virtualized Control Center has the following features:
Two physical servers are installed as KVM Hypervisor Nodes. The supported
Hypervisor Nodes can be derived from the FlexFrame Support Matrix.
The virtual Control Node CN1 is hosted on one of the Hypervisor Nodes.
Accordingly Control Node CN2 is hosted on the remaining Hypervisor Node.
Just like their physical counterparts, the virtual Control Nodes are clustered by
LINUX-HA. In case of hardware failure of one of the Hypervisor Nodes LINUX-
HA guarantees that the Control Node functionality is still given.
Both KVM Hypervisor Nodes can host other virtual machines i.e. Application
Nodes.
Running Control Node and additional Application Nodes in parallel allows a
higher grade of utilization of the hardware,
The virtual Control Center allows the configuration of an FFO entry model
consisting of only two physical servers.
There is no migration of virtual Control Nodes within the FlexFrame landscape
A virtual Control Center can only be installed during a new FFO installation. Up to now
you cannot install it during the Upgrade process. The installation process is described in
detail in the Installation Guide. The KVM Hypervisor Nodes hosting a virtual Control
Center must fulfill special requirements as described in the "FlexFrame® HW
Characteristics Quickguides". Especially the KVM Hypervisor Nodes hosting a virtual
Control Node need 4 network interfaces. Within the FFO 1.1A00-020 implementation
there is a fixed assignment of virtual Control Nodes to their Hypervisor Node.
A vCN needs a portion of 60GB diskspace on the physical local disk of the underlying
Hypervisor Node and a corresponding portion of 8GB memory. It is the customer’s
responsibility to care for the sufficient availability of disk and memory space of the
underlying Hypervisor Node for the virtual Control Node.
At the moment it is not possible to migrate a virtual Control Node to another Hypervisor
Node.
In case a virtual Control Node is disturbed or the Hypervisor Node is physically defect
special steps have to be done in order to get the virtual Control Center functional again.
These disaster scenarios are described in detail in chapter 11.4.
2.4 FlexFrame Structure in LDAP
LDAP is used as the central information service for all shared OS nodes within a
FlexFrame environment. The Control Nodes are used as LDAP servers. The LDAP
database is located on shared file systems mounted from the NAS storage. The
FlexFrame Architecture FlexFrame Structure in LDAP
12 Administration and Operation
Application Nodes are configured as LDAP clients. LDAP requests from Application
Nodes are restricted to the data of their own pool.
LDAP provides host-related network information such as:
net boot
automount
user authentication
groups
host names and IP addresses
shared services
networks and netmasks
LDAP client profiles
The FlexFrame LDAP tree roughly looks as illustrated here:
Additional information about configuration data is only applicable for Control Nodes.
These are used FlexFrame-internally to add, remove or modify the configuration of
Application Nodes or SAP services.
Linux-HA Cluster on Control Center FlexFrame Architecture
Administration and Operation 13
FlexFrame utilizes LDAP for two different purposes:
(1) for operating naming services (such as host name resolution, user/password
retrieval, tcp/udp service lookup, etc.) and
(2) for storing FlexFrame specific data on the structure of the installed environment
Application Nodes are only able to search in area (1). It is separated into pool specific
sections in order to protect pools from accessing other pools' data. Each of them contains
pool specific network information service (NIS) like data. The LDAP servers have access
lists to prevent searches outside of the own pool.
The other main DIT part contains FlexFrame configuration data (2). It should only be
accessed through maintenance tools from one of the Control Nodes. This part of the DIT
contains a lot of cross references, which need to be kept in sync. Do not try to change
this data, unless you are explicitly instructed by Fujitsu support to do so.
2.4.1 Working with LDAP
The usage of LDAP specific commands like ldapadd or ldapmodify is limited to very
few actions. One is to create or remove a PUT service for a SAP system copy. This
action is described within the “Installation Guide for SAP Solutions” manual. Other direct
interaction through LDAP commands is limited to service issues.
No other interventions have to and should be done. The FlexFrame maintenance tools
provide the necessary functionality.
2.5 Linux-HA Cluster on Control Center
2.5.1 Terminology
Some of the Linux-HA terms used below are explained here in order to promote broader-
based understanding:
Node
Every computer that is part of a cluster is a node
Pacemaker
Software to perform Linux-HA resource management on a cluster.
Resource
Everything than can be administered by pacemaker is referred to as a resource. For
example, an IP address that is administered by the cluster is a resource.
Resource Agent (RA)
The RA is the connection between pacemaker and the programs that are started
when the RA is called. They are shell scripts, which have to provide a standardized
FlexFrame Architecture Linux-HA Cluster on Control Center
14 Administration and Operation
interface to pacemaker, so that they can be started, monitored and stopped by
pacemaker.
Supported standards:
Linux Standard Base (LSB)
o All scripts under /etc/init.d correspond to this standard
Open Cluster Framework (OCF)
o For examples see /usr/lib/ocf/resource.d
Stonith
Designated Coordinator (DC)
Every cluster has precisely one DC as the central instance in the cluster. It is elected
from all the nodes. It alone is responsible for all the actions in the cluster and has the
only valid cluster information base. All the other nodes only have a duplicate.
STONITH
STONITH is an abbreviation for "Shoot The Other Node In The Head". This is the
name of a method that stops any nodes no longer accessible from "causing damage"
in the cluster by switching it off or alternatively causing a reboot.
openais / corosync
The cluster can be started and stopped with the /etc/init.d/openais script.
This script starts the corosync cluster engine /usr/sbin/corosync.
Cluster Resource Manager (CRM)
The CRM manages the resources, decides which resource is to run where and
ensures that the required status of the cluster is achieved based on the current
status. For this purpose the CRM distributes the work to the LRM and receives
feedback from the latter.
Local Resource Manager (LRM)
The LRM manages the resources on "its" local nodes. The CRM tells the LRM which
resources they are. The LRM communicates with the CRM.
Cluster Information Base (CIB)
The CIB is the central information file for all the resources and nodes of the cluster. It
not only contains information about the static configuration but also about a dynamic
part of the current status of all resources. The data is stored in XML format.
corosync.conf
This configuration file controls the behavior of the cluster software. It is needed to
start pacemaker. For example, it defines the interface via which communication is to
Linux-HA Cluster on Control Center FlexFrame Architecture
Administration and Operation 15
take place in the cluster and the nodes which belong to the cluster. It must be available on every node under /etc/corosync.
authkey
The file authkey is used to authenticate the nodes of a cluster for one another. This file must also be available on every node under /etc/corosync.
2.5.2 Simple Resources
2.5.2.1 Constraints
Simple resources can be linked with constraints, which are triggered when the resource is
started. These are:
Ordering:
specifies the relationship of two resources to each other. This means that you can
only start resource B after resource A.
Collocation:
ensures that two resources are started on the same node.
Location:
defines the node where a resource is to preferably run.
2.5.2.2 Score
Constraints are defined by rules. A system of points is used to decide on which node a
resource is to run. Each applicable rule is linked to a score. In addition to normal inte-
gers, it can also accept special values –INFINITY and INFINITY.
Example:
Resource A is to preferably run on CN1. The rule: NODE eq CN1
Score: 10
If A would run on CN1, the rule would return “true” and CN1 would get the score 10.
If A would run on CN2, the rule would return “false” and CN2 would not get any score,
thus 0.
After all possibilities being evaluated by the cluster, CN1 has a greater score than CN2,
thus CN1 would be chosen to run A.
The extremes:
FlexFrame Architecture Linux-HA Cluster on Control Center
16 Administration and Operation
INFINITY: The resource is started in any case on the node selected by an applicable rule.
If it cannot be started on the specified node, an attempt is made to start it on another
node.
-INFINITY: The resource is not started on the node selected by an applicable rule. If it
cannot be started on any other node than the selected one, it remains stopped.
In other words:
INFINITY ~ is applicable if possible
-INFINITY ~ is never applicable
2.5.2.3 Stickiness
Another value that plays an important role in the decision about the resource where the
node is to run is stickiness. You can regard this value as the level of adhesion, with
which a resource aims to remain on the node on which it is currently running. This value
is defined via the global settings of the CIB by the variable default-resource-
stickiness. The following values are possible.
0 :
Linux-HA attempts to optimally place the resource in the cluster.
In other words, the resource is redistributed in the cluster if Linux-HA finds a more
"suitable" node.
> 0 :
Linux-HA attempts to optimally place the resource in the cluster.
In other words, the resource is redistributed in the cluster if Linux-HA finds a more
"suitable" node.
< 0 :
The resource tends to leave the current node. Lower values reinforce this tendency.
INFINITY :
The resources remain on the current node until they are compelled to leave it, e.g.
when their node is shut down.
-INFINITY :
The resources do not want to remain on the current node.
The two values, score and stickiness, determine the node on which a resource is then
actually started. For this purpose, they are determined and added for each node of the
cluster. The resource is then started on the node that has the highest score.
Linux-HA Cluster on Control Center FlexFrame Architecture
Administration and Operation 17
Example:
Default_resource_stickiness : 100
Resource A is to be preferably run on CN1
Rule: NODE eq CN1
Score: 10
Case 1: A already runs on CN1
Points for CN1 :
Location score =10
Stickiness score = 100
Resulting score CN1 =110
Points for CN2 :
Location score = 0 (rule not met)
Stickiness Score = 0 (because it is currently not running on CN2)
Resulting score CN2 = 0
Result: A remains on CN1, score CN1 > score CN2
Case 2: A already runs on CN2
Points for CN1 :
Location score = 10
Stickiness score = 0 (as not running on CN2)
Resulting score CN1 = 10
Points for CN2 :
Location score = 0
Stickiness score = 100
Resulting score CN2 = 100
Result: As score CN2 > score CN1, resource A remains on CN2, although its location
score expresses the wish to move to CN1.
FlexFrame Architecture Linux-HA Cluster on Control Center
18 Administration and Operation
Case 3: A does not run on any node (e.g. during cluster start)
Points for CN1 :
Location score =10
Stickiness score = 0
Resulting score CN1 = 10
Points for CN2 :
Location score = 0 (rule not met)
Stickiness score = 0
Resulting score CN2 = 0
Result: A starts on CN1, score CN1 > score CN2
In FlexFrame all resources except the ldap services are configured in such a way that
they remain on their current node after a move, even when the other control node is
available again after a downtime. This avoids unnecessary switching processes.
2.5.2.4 Resource Groups
Simple resources can be put together to form groups. It is also possible to create so-
called clones, in which simple resources may run repeatedly in the cluster. Constraints,
such as ordering or colocation, which are defined for the group, apply for all the re-
sources belonging to the group.
2.5.3 FlexFrame Specific Configuration
The basic configuration for pacemaker is created during the initial configuration of
FlexFrame, i.e. the file corosync.conf is set up, a key to authenticate the nodes CN1
and CN2 is created for the file authkey and the CIB is created.
It contains the following resources (type and name of the RA in brackets):
Resource Group: network_CN1_server
ip_CN1_server_<poolname-m> (ocf::heartbeat:IPaddr2)
Resource Group: network_CN2_server
ip_CN2_server_<poolname-m> (ocf::heartbeat:IPaddr2)
ldap_master (ocf::fsc:ff_ha_ldap)
ldap_replica (ocf::fsc:ff_ha_ldap)
Resource Group: netboot
Linux-HA Cluster on Control Center FlexFrame Architecture
Administration and Operation 19
dhcpd (lsb:ff_ha_dhcpd)
tftpd (lsb:ff_ha_tftpd)
Resource Group: ff_manage)
postgresql (ocf::heartbeat:pgsql)
myAMC.FA_Messenger (lsb:ff_ha_myAMC.MessengerSrv)
tomcat (lsb:ff_ha_tomcat)
myAMC.FA_CtrlAgent (lsb:ff_ha_myAMC.CtrlAgent)
Clone Set: clone_ClusterMon
ClusterMon:0 (ocf::heartbeat:ClusterMon)
ClusterMon:1 (ocf::heartbeat:ClusterMon)
stonith_ipmi_CN1 (stonith:external/ipmi
stonith_ipmi_CN2 (stonith:external/ipmi)
2.5.3.1 Configuration of Limits for Cluster Resources
In some large FlexFrame environments the default number of open files
(1024) is not sufficient to solve all requests to ldap or tomcat.
/var/log/messages show entries like:
... slapd[2393]: warning: cannot open /etc/hosts.allow: Too many open
files
... java.rmi.RemoteException: VI SDK invoke
exception:java.net.SocketException: Too many open files
In such a case the number of open files has to be increased before the
cluster resources are (re)started.
The problem only occurs in very large configurations (many pools with many Application
Nodes and a lot of big SAP applications with many active users permanently logging in).
Therefore instead of system-wide increasing of the default number of open files a
FlexFrame specific configuration template is provided. This template can be used to
create a custom specific file limiting the number of open files.
The mentioned template is the
/opt/FlexFrame/etc/ff_ha_settings.conf.template file. You can use it to
create your own configuration file with the settings suitable for your system:
FlexFrame Architecture Linux-HA Cluster on Control Center
20 Administration and Operation
At first copy the template file to /opt/FlexFrame/etc/ff_ha_settings.conf:
control1:~ # cp /opt/FlexFrame/etc/ff_ha_settings.conf.template
/opt/FlexFrame/etc/ff_ha_settings.conf
Now uncomment the line with the 'FF_ULIMIT_SETTINGS' variable. The example
below shows an entry to increase the number of open files up to 8192 (8K):
## Template for config file /opt/FlexFrame/etc/ff_ha_settings.conf
## Used to set special values for our linux ha resources like ldap or tomcat
## The resource scripts /opt/FlexFrame/lib/ha/scripts/ff_ha_* will look
## for active variables in the config file
##
## Copy the template file to /opt/FlexFrame/etc/ff_ha_settings.conf and
## uncomment the needed variables
## increase number of open files for resource ldap and tomcat
FF_ULIMIT_SETTINGS="-n 8192"
Finally the resources tomcat and ldap_master have to be restarted, if they are already
running.
control1:~ # ff_ha_cmd.sh restart tomcat
control1:~ # ff_ha_cmd.sh restart ldap_master
You may verify the results of the changes by checking the limits before (should be 1024)
and after (should now be 8192) the restart of the resources:
cat /proc/<pid_of_tomcat>/limits
cat /proc/<pid_of_ldap>/limits
The configuration file /opt/FlexFrame/etc/ff_ha_settings.conf will be backed
up automatically by ff_cn_backup.sh during a FlexFrame upgrade, but has to be
restored manually after the FlexFrame upgrade has finished.
2.5.3.2 LDAP Configuration
The FlexFrame LDAP concept is implemented by the resource groups net-
work_CN1_server and network_CN2_server as well as the simple resources ldap_master and ldap_replica. Since this concept has dramatically changed with
the introduction of Linux-HA, it is explained in more detail here.
Linux-HA Cluster on Control Center FlexFrame Architecture
Administration and Operation 21
Each network_CN<n>_server (n=1..2) group consists of the simple resources
network_CN<n>_server_<poolname-m> (n=1..2)1. Each of these simple
resources is exactly one server LAN IP of a pool, i.e. the appropriate resource agent
(IPaddr2) accurately monitors a server LAN IP and, in the event of a fault, moves it to the
surviving node of the cluster. Accordingly, there are as many resources in the cluster as
there are server LAN IPs.
This ensures that the entire server LAN IPs of all the pools can always be accessed in
the network.
The resource ldap_master is initially started on the first control node. It starts the LDAP
server process slapd on the same control node with the server LAN IPs of all the pools
of the first control node as defined by the management tool. Analog to this, the resource
ldap_replica on the second control node is started with the appropriate server LAN
IPs of all the pools of the second control node.
This concept ensures that:
1. ldap_master und ldap_replica can always access precisely defined IP
addresses, i.e. the address of the server LAN
2. these IP addresses are managed by the cluster and are thus always available on a
cluster-wide basis
3. ldap_master und ldap_replica can also run in parallel on a node.
In other words, if a node fails, the cluster ensures that its IP addresses and the
appropriate ldap resource are switched over to the other node and are then available
again on a system-wide basis
Example:
2 Pools, pool1, pool2:
Server LAN IPs :
cn1-pool1-se 192.168.12.12
cn1-pool2-se 192.168.102.16
cn2-pool1-se 192.168.12.13
cn2-pool2-se 192.168.102.17
Resource group network_CN1_server contains the two simple resources:
1 "n" is an enumeration consisting of 2 elements. Element 1 for CN1 and element 2 for CN2.
FlexFrame Architecture Linux-HA Cluster on Control Center
22 Administration and Operation
network_CN1_server_pool1 manages 192.168.12.12
network_CN1_server_pool2 manages 192.168.102.16
Resource group network_CN2_server with
network_CN2_server_pool1 manages 192.168.12.13
network_CN2_server_pool2 manages 192.168.102.17
The resource ldap_master starts on CN1 with the IP addresses 192.168.12.12 and
192.168.102.16
The resource ldap_replica starts on CN2 with the IP addresses 192.168.12.13 and
192.168.102.17
In comparison with other resources and resource groups the rule for the "location" con-
straint of the resource groups network_CN<n>_server (n=1..2) is given a score
(=100000) higher than the default-resource-stickiness value (=100). After a
move to the other nodes (e.g. after a control node failure or reboot) this causes the server
LAN and LDAP resources to return to their original nodes as soon as possible.
2.5.3.3 Resource ff_manage
The resource ff_manage is a group of simple resources that is initially started on the
first control node. These are:
postgresql
myAMC.FA_Messenger
tomcat
myAMC.FA_CtrlAgent
Prerequisite for the start of myAMC.FA_messenger is the successful start of the resource
postgresql. This is compelled by the constraint "orders".
Since the start of ff_manage can take longer, an attempt is made as a result of an "or-
ders" constraint to start the resources ldap_master and ldap_replica before the
start of the group ff_manage.
2.5.3.4 Resource Netboot
The resource group netboot contains the simple resources.
dhcp
tftpd
An attempt is also made for this group – as for the group ff_manage – to only start it if
the resources ldap_master and ldap_replica run successfully. The "orders"
constraint is also used for this.
Linux-HA Cluster on Control Center FlexFrame Architecture
Administration and Operation 23
2.5.3.5 STONITH (Physical Control Center)
The stonith agents' purpose is to prevent the cluster entering a non-defined status in the
event of a fault in communications. The corresponding stonith resources are:
stonith_ipmi_CN2
stonith_ipmi_CN1
Due to the "location" constraint stonith_ipmi_CN2 may – under no circumstances –
be started on CN2 and stonith_ipmi_CN1 may not be started on CN1.
These resources monitor the other node respectively. If, after a time interval set during
configuration, a node finds that communication with the other node is no longer possible,
a hard "reboot" or a "reset" is triggered on the inaccessible node via the IPMI interface.
The IP addresses of the IPMI interface as well as user and password were transferred to
the resource during configuration in order to enable this step.
For the reboot to take place via the ipmi interface the latter must be accessible. Therefore stonith_ipmi_CN<n> (n=1..2) monitors the respective interface every 30 seconds. If it
is determined that the interface does not answer (e.g. in the event of a power failure),
pacemaker sets the state of the node in question to UNCLEAN (offline). In this case the
administrator should
1. Call crm node clearstate <node which cannot be reached>
This command sets the state of that node from UNCLEAN (offline) to offline so
that pacemaker is able to take over the resources fom that node.
2. Shutdown / restart that node
Example after power failure of CN2 :
Using the command crm_mon -1 -r -f -n provides an overview of the error counters
and inactive resources:
FlexFrame Architecture Linux-HA Cluster on Control Center
24 Administration and Operation
Calling : crm node clearstate cn2
The state of cn2 was cleared and all resources which are running on cn2 are switched
over to cn1.
Linux-HA Cluster on Control Center FlexFrame Architecture
Administration and Operation 25
1. If the hardware of the node in question was not changed no manually
intervention is required during the reboot.
2. After the reboot pacemaker will start all services but ff_manage on their
default node. Please be patient because this process will take some
time.
2.5.3.6 STONITH (Virtual Control Center)
In case of a virtual Control Center you will find the following stonith resources:
stonith_ipmi_CN2
stonith_vm_CN2
stonith_ipmi_CN1
stonith_vm_CN1
Due to the "location" constraints stonith_ipmi_CN2 and stonith_vm_CN2 are
running on CN1 only. stonith_ipmi_CN1 and stonith_vm_CN1 are running on CN2
only.
FlexFrame Architecture Linux-HA Cluster on Control Center
26 Administration and Operation
stonith_vm_CN2 (running on virtual Control Node CN1) monitors virtual Control Node
CN2 (stonith_vm_CN1 running on CN2 does the same with CN1).
For simplicity we regard from now on only the case that virtual Contol Node CN2 gets into
trouble.
If, after a time interval, the monitored virtual Control Node CN2 is supposed to be dead, stonith_vm_CN2 tries to restart CN2 using libvirt. If this is successful, everything is fine
again.
If the restart of CN2 fails due to an issue on the Hypervisor Node, stonith_ipmi_CN2
becomes active. All IP addresses of FlexFrame virtual Application Nodes that run in
parallel on the affected Hypervisor Node hosting CN2 are known to stonith_ipmi_CN2. A script called by the resource checks via ping if there are still
virtual Application Nodes left that are alive (on the Hypervisor Node hosting CN2). In
case of still living virtual Application Nodes no further action is taken by stonith_ipmi_CN2. If all virtual Application Nodes are judged dead, the Hypervisor
Node hosting CN2 is killed via IPMI interface.
2.5.3.7 Configuring the Cluster
The cluster is automatically configured by the script ff_post_conf.sh during the instal-
lation of FlexFrame. If, in certain situations, it is necessary to subsequently configure the cluster, the command ff_ha_tool.sh [-f] -i must be used. During the setup the
IPMI user and IPMI password are queried if they are not yet known or access is not pos-
sible with the known user and password.
Syntax :
ff_ha_tool.sh [-f] -i
-i: initial configuration of linux-ha for FlexFrame
-f: force execution, purge old configuration
2.5.3.8 Starting Behavior of the Cluster
The cluster is started:
automatically when FlexFrame is installed via the installation scripts
with each reboot
with the command /etc/init.d/openais start
Two cases should be observed both when rebooting and during a manual start:
1. One node is affected – either it has failed or the service openais was stopped
manually. The second node subsequently took on the resources and has the
role of the DC.
Linux-HA Cluster on Control Center FlexFrame Architecture
Administration and Operation 27
2. Both nodes are affected.
In case 1 there is still nothing to be considered. After the node is online again or the clus-
ter could be successfully started on this node, the resources are redistributed according
to their score values and started.
If the cluster has to be restarted on both nodes (case 2) – either through rebooting or
manually – the following must be taken into consideration:
During the start of a node the service openais attempts to communicate with the other
node. If there is no answer after a configurable time interval, the stonith agent is
activated, which via the IPMI interface then causes the second node to reboot. The time
interval is configured in the CIB by the parameter dc-deadtime and is currently
120seconds.
Particularly when switching on an entire FlexFrame environment it is
essential to ensure that both control nodes have to be switched on very
quickly one after each other. If the delay is too great, the control node
switched on last may be reset hard by the stonith agent, but this usually has
no negative effects.
If in this situation the IPMI interface of the other node is not accessible, the stonith IPMI
agent would - after a vain attempt to trigger a reboot on the node - change the state of
that node to UNCLEAN (offline). Then the operator must proceed as described in the
section "STONITH" so that the resources monitored by the cluster can be started.
2.5.3.9 Status Information of the Clusters
Various options can be used to find out the current status of the cluster. One of which is
implemented by the resource agent ClusterMon. It is the task of this RA to regularly
transfer the current status of the cluster to a HTML file. For this purpose, the clones ClusterMon:0 and ClusterMon:1 are configured which are each started on a
control node. They write the status in /srv/www/htdocs/ClusterMon.html, which can be
accessed via the local web server at http://localhost/ClusterMon.html and is also linked
from the default homepage of the web server.
2.5.4 Linux-HA CLI Commands
Various options are available to achieve a required action. For this purpose, FlexFrame provides a simple CLI interface to Linux-HA. The command ff_ha_cmd.sh, which is
internally based on the Linux-HA CLI commands, was provided for the most important ac-
tions, such as status display, starting and stopping a Resource.
Action Command
FlexFrame Architecture Linux-HA Cluster on Control Center
28 Administration and Operation
Action Command
Display the status of all resources ff_ha_cmd.sh status
Display the status of one resource ff_ha_cmd.sh status <resource>
Start a resource ff_ha_cmd.sh start <resource>
Stop a resource ff_ha_cmd.sh stop <resource>
Migrate (move) a resource to another
node – this creates a special location
constraint
ff_ha_cmd.sh migrate <resource>
<node>
Undo the migration – the special
location constraint is deleted ff_ha_cmd.sh unmigrate <resource>
Cleanup a resource, delete all status
flags – this will restart the resource if
possible
ff_ha_cmd.sh cleanup <resource>
Remove a resource from the
administration using pacemaker.
Resources will not be automatically
started, stopped or restarted
ff_ha_cmd.sh unmanage <resource>
Reintegrate a resource in the
administration using pacemaker ff_ha_cmd.sh manage <resource>
Output the CIB (XML format) cibadmin -Q
List all the resources of the cluster crm_resource -L
crm_resource -l
List a resource (XML format) crm_resource -x -r <resource>
Output the node, on which a resource
runs
crm_resource -W -r <resource>
ff_ha_cmd.sh status resource
Regular status display (interval: 15
sec)
crm_mon
Regular status display, interval <n>
sec
crm_mon -i <n>
Once-only status display crm_mon -1
[Once-only] status display with output
of the error counters of all resources
as well as all offline resources
crm_mon [-1] -f -r
Network FlexFrame Architecture
Administration and Operation 29
Action Command
As above, but grouped by nodes crm_mon [-1] -f -r -n
Add resources based on a
configuration template
ff_ha_tool.sh –a <template>
[param=value]
Remove resources based on a
configuration template
ff_ha_tool.sh –d <template>
[param=value]
List available and configured
configuration templates
ff_ha_tool.sh –a –l
If a resource is to be migrated to another node, it is essential that this action creates a
"Location" constraint with the score INFINITY. However, this means that this resource
also remains permanently on this node even after a complete restart of the cluster and is
therefore not moved – unless the node has a problem to the effect that no resources can
run any more. This constraint can be removed with the command ff_ha_cmd.sh un-
migrate <resource>, which nevertheless does not cause the resource to automati-
cally move back; it merely removes the constraint of the preferred node as a result.
2.5.5 Linux-HA Logfile
All messages that concern Linux-HA are written in the log file /var/log/ha-log .
2.5.6 Linux-HA GUI
Linux-HA has its own GUI which can be launched using the "hb_gui" command. However, it is strongly recommended not to use this interface to change configuration parameters or control resources, because each change hides the risk of negatively influencing or even severely damaging the FlexFrame cluster configuration. There also may be situations where it is not possible to use the command line interface after having used the "hb_gui" interface. Please use the command line interface ff_ha_cmd.sh exclusively to control resources.
2.6 Network
The network is the backbone of the FlexFrame solution. Communication between the
various nodes and storage devices is done exclusively via the IP network infrastructure. It
serves both, communication between servers and clients as well as delivering IO data
blocks between the NAS (Network Attached Storage) and the servers.
The IP network infrastructure is essential for every FlexFrame configuration. FlexFrame is
designed with a dedicated network for connections between servers and storage that is
FlexFrame Architecture Network
30 Administration and Operation
reserved for FlexFrame traffic only. One network segment, the Client LAN (see below)
can be routed outside the FlexFrame network to connect to the existing network.
2.6.1 LAN Failover
The term „LAN failover“ describes the ability of a FlexFrame environment to use a logical
network interface that consists of several physical network interface cards (NICs), which
in turn are using redundant network paths (cables and switches). When a network
component (NIC, cable, switch, etc.) fails, the network management logic will switch over
to another network interface card and path.
2.6.2 Segments
FlexFrame uses a network concept providing high availability as well as increased
flexibility in virtualizing the whole FlexFrame landscape.
The FlexFrame network concept relies on VLAN technology that allows running multiple
virtual networks across a single physical network. Additionally, in order to ensure high
network availability, LAN bonding is used on every node. This includes a double switch
and wiring infrastructure, to keep the whole environment working, even when a network
switch or cable fails. Within FlexFrame there are four virtual networks. These networks
run through one logical redundant NIC, using bonding on Linux.
Client LAN
The purpose of the Client LAN segment is to have dedicated user connectivity to the
SAP instances. This segment also allows administrators to access the Control
Nodes.
Control LAN
The Control LAN segment carries all administrative communication, such as IPMI or
similar interfaces.
Server LAN
The Server LAN segment is used for the communication between SAP instances
among each other and the databases.
Storage LAN
The Storage LAN segment is dedicated to NFS communication for accessing the
Application Nodes' shared operating systems, the executables of SAP and the
RDBMS as well as the IO of the database content and SAP instances.
The following figure outlines the basic network segments of a typical FlexFrame
landscape with Application Nodes.
Network FlexFrame Architecture
Administration and Operation 31
2.6.3 Network Switches
Network switching components play a very important role within FlexFrame. Therefore,
only designated switch types are tested and supported (see FlexFrame Support Matrix
https://partners.ts.fujitsu.com/com/products/infrastruc-solutions/FlexFrame or ask your
Fujitsu Customer Support for getting a copy).
All supported switch models support VLAN technology for a flexible configuration for the
various network segments.
2.6.4 Network Speed
FlexFrame supports network connections for data communication with the following
network speeds (see also: FlexFrame® – Network Design and Configuration Guide):
1Gbit/sec (1GbE)
10Gbit/sec (10GbE)
FlexFrame Architecture Network
32 Administration and Operation
To determine the 10GbE use case for end systems which are able for both, 1GbE or 10GbE network connection, use the --10gbit option when adding the device to the
FlexFrame configuration.
For further details about supported end systems see the FlexFrame Support Matrix.
2.6.5 Network Switch Groups
In FlexFrame network switches are grouped. Ports of an end system are building a
redundant connection and are connected to ports of different members of the same
switch group (see also: FlexFrame® – Network Design and Configuration Guide).
The switch groups within the entire FlexFrame environment are numbered starting with 1.
Also the members within a switch group are numbered starting with 1 in each case. In
special cases the use of switch group number 0 (zero) is allowed. This switch group is
neither registered in nor managed from FlexFrame. It helps to identify network devices
outside but with relation to the FlexFrame environment (see also next section Network
Switch Ports).
2.6.6 Network Switch Ports
In FlexFrame network switch ports are uniquely identified within the entire FlexFrame
environment by:
the number of the switch group
the number of the member within the switch group
the port ID within the switch
The portID is
a single number as for Cisco Catalyst 3750 switch fixed ports
a prefixed number
o <slot number>/<port number> as for Cisco Nexus switch ports
o G1 ... G4 as for Cisco Catalyst 3750x 1GbE module
o TE1, TE2 as for Cisco Catalyst 3750x 10GbE module
o F1 ... F4 as for Brocade ICX SFP ports
In case of Cisco Catalyst 3750e the numbers 1 and 2 are used both for GigabitEthernet
Ports and TenGigabitEthernet Ports. To distinguish both cases a --10gbit Option is used
on input and on output (10G) is appended.
Switch ports belonging to the unspecific switch group 0 (zero) are meant to be
somewhere outside the FlexFrame environment.
Network FlexFrame Architecture
Administration and Operation 33
2.6.7 Automounter Concept
The Automounter Concept is based on the ability of Linux to mount file systems
automatically when their mount points are accessed.
During the boot process of an Application Node some file systems are mounted. For
Linux these are the root file system (read-only) as well as the /var file system (read-
write). These are the basic file systems which must be accessible for the Application
Node to function properly. There is no data in the two file systems that is specific for a
SAP service.
Data which is specific for a SAP service, a database or a FlexFrame Agent is found in
directories which are mounted on first access. Some of the mounts will stay as long as
the Application Node is operational. Others will be unmounted again, if directories or files
below that mount point have not been accessed for a certain period of time.
Within the LDAP database there are two types of data which relate to the automounter
configuration: automountMap and automount.
An automountMap is a base for automount objects. Here's how to list the
automountMaps:
control1:~ # ldapsearch -x -LLL '(objectClass=automountMap)'
dn: ou=auto.FlexFrame,ou=Automount,ou=pool2,ou=Pools,ou=FlexFrame,
dc=flexframe,dc=wdf,dc=fujitsu,dc=com
objectClass: top
objectClass: automountMap
ou: auto.FlexFrame
...
The base directory looks like this:
dn: cn=/FlexFrame,ou=auto_master,ou=Automount,ou=pool1,ou=Pools,
ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu,dc=com
objectClass: top
objectClass: automount
cn: /FlexFrame
automountInformation: auto_FlexFrame
Further on there are entries like:
dn: cn=myAMC,ou=auto_FlexFrame,ou=Automount,ou=pool1,ou=Pools,
ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu,dc=com
objectClass: top
objectClass: automount
cn: myAMC
FlexFrame Architecture Network
34 Administration and Operation
automountInformation:
-rw,nointr,hard,rsize=32768,wsize=32768,proto=tcp,nolock,
vers=3 filpool1-st:/vol/volFF/pool-pool1/pooldata/&
There are two things that have to be pointed out.
First, the ou=auto.FlexFrame refers to the base directory as shown before.
The second notable aspect in this entry is the use of the wildcard &. If the folder
/FlexFrame/myAMC is accessed, the autofs process tries to mount it from the path
filpool1-st:/vol/volFF/pool-pool1/pooldata/myAMC. If the folder myAMC is
found and permissions allow the clients to access it, it will be mounted to
/FlexFrame/myAMC/<name>. If myAMC is not found or the client does not have the
permissions, the folder will not be mounted. In such a case, try to mount the folder
manually on a different folder, e.g. like:
an_linux:~ # mount filpool1-st:/vol/volFF/pool-pool1/pooldata/myAMC /mnt
If you get an error message like Permission denied, check the exports on the NAS
system and the existence of the directory myAMC/ itself.
Other entries in LDAP make use of platform specifics. With Linux you can find a number
of variables like ${OSNAME}/${OSDIST}/${ARCH} to make a distinction between
different platforms.
dn: cn=/,ou=auto.oracle,ou=Automount,ou=pool2,ou=Pools,
ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu,dc=com
objectClass: top
objectClass: automount
cn: /
description: catch-all for Linux automount
automountInformation:
-rw,nointr,hard,rsize=32768,wsize=32768,proto=tcp,nolock,
vers=3 filpool2-st:/vol/volFF/pool-pool2/oracle/${OSNAME}/${OSDIST}/${ARCH}/&
On Linux, the automount mount points can be read using the following command:
an_linux:~ # mount
rootfs on / type rootfs (rw)
/dev/root on / type nfs (ro,v3,rsize=32768,wsize=32768,reserved,
hard,intr,tcp,nolock,addr=192.168.11.206)
192.168.11.206:/vol/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64/var_img/var-
c0a80b36
on /var type nfs (rw,v3,rsize=32768,wsize=32768,reserved,
hard,intr,tcp,nolock,addr=192.168.11.206)
192.168.11.206:/vol/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64/var_img/var-
c0a80b36
Storage Systems FlexFrame Architecture
Administration and Operation 35
/dev on /dev type nfs (rw,v3,rsize=32768,wsize=32768,reserved,
hard,intr,tcp,nolock,addr=192.168.11.206)
192.168.11.206:/vol/volFF/os/Linux/pool_img/pool-c0a80bff on /pool_img type nfs
(rw,v3,rsize=32768,wsize=32768,reserved,
hard,intr,tcp,nolock,addr=192.168.11.206)
proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw)
shmfs on /dev/shm type shm (rw)
/dev/ram on /var/agentx type ext2 (rw)
automount(pid1750) on /FlexFrame type autofs (rw)
automount(pid1772) on /saplog/mirrlogA type autofs (rw)
automount(pid1752) on /home_sap type autofs (rw)
automount(pid1788) on /saplog/saplog1 type autofs (rw)
automount(pid1766) on /sapdata/sapdata5 type autofs (rw)
automount(pid1778) on /saplog/origlogA type autofs (rw)
automount(pid1762) on /sapdata/sapdata3 type autofs (rw)
automount(pid1758) on /sapdata/sapdata1 type autofs (rw)
automount(pid1784) on /saplog/saparch type autofs (rw)
automount(pid1764) on /sapdata/sapdata4 type autofs (rw)
automount(pid1786) on /saplog/sapbackup type autofs (rw)
automount(pid1760) on /sapdata/sapdata2 type autofs (rw)
automount(pid1754) on /myAMC type autofs (rw)
automount(pid1796) on /usr/sap type autofs (rw)
automount(pid1780) on /saplog/origlogB type autofs (rw)
automount(pid1768) on /sapdata/sapdata6 type autofs (rw)
automount(pid1792) on /saplog/sapreorg type autofs (rw)
automount(pid1776) on /saplog/oraarch type autofs (rw)
automount(pid1774) on /saplog/mirrlogB type autofs (rw)
automount(pid1770) on /sapdb type autofs (rw)
automount(pid1756) on /oracle type autofs (rw)
automount(pid1790) on /saplog/saplog2 type autofs (rw)
automount(pid1794) on /sapmnt type autofs (rw)
filpool2-st:/vol/volFF/pool-pool2/pooldata on /FlexFrame/pooldata type nfs
(rw,v3,rsize=32768,wsize=32768,reserved,hard,tcp,
nolock,addr=filpool2-st)
The cn: parts show the mount points.
2.7 Storage Systems
FlexFrame uses a central NAS system to store data needed by the FlexFrame system
and its components. Some data areas of a FlexFrame landscape can be distributed to
other NAS systems.
FlexFrame Architecture Storage Systems
36 Administration and Operation
It is also possible to use SAN storage systems for the database datafiles and database
logfiles of SAP systems with demand for higher performance and throughput. Everything
else will still be placed on NAS storage. This explicitly includes the database software,
SAP applications and the OS. Even the database files do not necessarily have to be
placed onto a SAN based storage. One might choose to place large, productive
databases with IO requirements onto the SAN based storage while smaller databases will
be on NAS storage. The choice is based on SID level. One SID might be on SAN and
another SID on NAS.
The following pictures illustrate the components which can be placed onto NAS and SAN
storage.
FlexFrame Components on NAS
FlexFrame Components on SAN
2.7.1 NAS systems
In FlexFrame the storage for all Application Nodes can be one or more NAS systems
(Filers) from Network Appliance, Inc.. The Filer can be connected with 10Gbit/sec or
1Gbit/sec NICs to the FlexFrame system. The parallel use of 1Gbit/sec and 10Gbit/sec
NICs of one Filer is not possible. But you can use on one filer 1Git/sec NICs and on
another 10Gbit/sec NICs.
The operating system of the Filer is called "ONTAP". The disks will be grouped into RAID
groups. A combination of RAID groups will make a volume. Starting with ONTAP 7, also
aggregates can be created. FlexVolumes can be created on top of aggregates. A Filer
volume contains a file system (WAFL - Write Anywhere File Layout) and provides
volumes or mount points for NFS (for UNIX systems) or CIFS (for Windows systems).
The Filer has NVRAM (Non Volatile RAM) that buffers committed IO blocks. The contents
of the NVRAM will remain intact if the power of the Filer should fail. Data will be flushed to
the disks once power is back online.
The minimal FlexFrame landscape has at least the following volumes:
vol0 (ONTAP, configuration of Filer)
sapdata (database files)
saplog (database log files)
Storage Systems FlexFrame Architecture
Administration and Operation 37
volFF (OS images of Application Nodes, SAP and database software,
pool related files)
In FlexFrame, the volume volFF separates FlexFrame data (file system of Application
Nodes and other software) from the Filer's configuration and ONTAP. In larger
installations, multiple sapdata and saplog volumes can be created (e.g. to separate
production and QA etc.).
As of FlexFrame 5.2A the three volumes names volFF, sapdata, saplog are no
more fixed names and may be established as required via Planning Mode of the
Management Tool (chapter “Storage” Object). The given names are the FlexFrame
default names.
In this document the volume names vol0, volFF, sapdata, saplog always refer
to the volumes playing the role mentioned above.
2.7.1.1 Built-in Cluster File System
The Network Appliance implementation of the NFS (Networked File System) allows
sharing of the same data files between multiple hosts. No additional product (e.g. cluster
file system) is required.
2.7.1.2 Volume Layout
The FlexFrame concept reduces the amount of "wasted" disk space since multiple SAP
systems can optionally share the same volume of disks. As the data grow, it is easy to
add additional disks and enlarge the volumes without downtime.
2.7.1.3 Snapshots
When a snapshot is taken, no data blocks are being copied. Just the information where
the data blocks are located is saved. If a data block is modified, it is written to a new
location, while the content of the original data block is preserved (also known as “copy on
write”). Therefore, the creation of a snapshot is done very quickly, since only few data
have to be copied. Besides that, the snapshot functionality provided by NetApp is unique,
because the usage of snapshots does not decrease the throughput and performance of
the storage system.
Snapshot functionality will allow the administrator to create up to 250 backup-views of a
volume. The functionality “SnapRestore” provided by NetApp significantly reduces the
time to restore any of the copies if required. Snapshots will be named and can be re-
named and deleted. Nested snapshots can be used to create e.g. hourly and daily
backups of all databases. In a FlexFrame landscape, a single backup server is sufficient
to create tape backups of all volumes. Even a server-less backup can be implemented.
Off-Line backups require a minimal down-time to the database because the backup to
tape can be done reading from a quickly taken snapshot. Concerning backup using
snapshots please also have a look at 3.6 FlexFrame Backup with Tape Library.
FlexFrame Architecture Storage Systems
38 Administration and Operation
2.7.1.4 vFiler
With activation of a Multistore license a Filer can only support vFiler. Every vFiler may
have its own administrator. A special vFiler named vfiler0 is automatically created and
cannot be deleted. Initially all resources are owned by vfiler0. Only the vfiler0
administrator can create or delete vFiler, volumes and interfaces and assign volumes and
interfaces to vFiler.
As of FlexFrame 5.2 an arbitrary vFiler is allowed to be used as NAS System in the
FlexFrame environment. Other vFiler on the same storage host can be used for other
purposes as far as the sizing requirements are observed. If in doubt consult the FTS
sizing team.
2.7.1.5 Storage Virtual Machines (SVMs)
Together with ONTAP 8.0 NetApp introduced its new paradigma of a Storage Virtual
Machine (SVM) with its “Clustered Data ONTAP (cDOT)”, which is on the way to replace
the conventional ONTAP 7-mode, and which is the basis for NetApp’s scale-out
architecture.
As of FlexFrame Orchestrator 1.1A Clustered Data ONTAP is supported, and the 7-mode
DATA ONTAP support is continued as well.
The storage virtual machine also supplies the foundation for the current NetApp multi-
tenant architecture. From FlexFrame’s perspective an SVM is a NAS system like any
FASxxxx or a vFiler.
The SVM may have a single logical interface (lif) per pool and for all lifs the home
port is created as VLAN port based on a single virtual interface group (ifgroup) consisting
of multiple physical interfaces. The home node is given as parent parameter when adding
the SVM to the FlexFrame landscape. Before adding an SVM the home node and its
partner node must be defined what in turn needs that the cluster is already defined as
NAS system of type “cDOTCluster”:
It is helpful to know the 3 types of vservers because you need to define and/or configure
them all:
admin vserver
This is commonly called the “Cluster”. Even if there is no adminstrative access
given by the owner the Cluster has to be defined. In FlexFrame it has the type
“cDOTCluster”.
node vserver
These are the real nodes a cluster consists of. In FlexFrame they have the type
of the nodes i.e. “FASxxxx”. Each node must have a partner in the Cluster in
order to establish an HA pair. The Cluster has to be defined as the parent.
data vserver
Storage Systems FlexFrame Architecture
Administration and Operation 39
This is actually the “SVM” hosting the data and providing file services. In
FlexFrame it has the type “SVM”. FlexFrame has full administrative access to
the SVM. One of the nodes has to be defined as the parent.
The term cluster has been used historically to refer to an HA pair running
Data ONTAP 7-Mode. This usage has been discontinued, and HA pair is the
only correct term for this configuration. The term cluster now refers only to a
configuration of one or more HA pairs running clustered Data ONTAP.
Example: To configure an SVM mysvm on a HA pair (2 nodes called node1 and node2)
belonging to the cluster newcluster the following steps are needed:
cn1:~ # ff_nas_adm.pl --op add --name newcluster --type cDOTCluster \
--osType cDOT --interface a1a
cn1:~ # ff_nas_adm.pl --op add --name node1 --type FASxxx \
--osType cDOT --parent newcluster --interface a1a
cn1:~ # ff_nas_adm.pl --op add --name node2 --type FASxxx \
--osType cDOT --parent newcluster --interface a1a
cn1:~ # ff_nas_adm.pl --op partner --name node1 --partner node2
cn1:~ # ff_nas_adm.pl --op add --name mysvm --type SVM \
--parent node1 --interface a1a
Please refer to chapter 8.1.1 “Adding a New NAS” for details.
Independent from the ONTAP variant a Filer can be “clustered” via a HA pair to protect
data against the failure of a single Filer. Switching from one Filer to its HA pair
counterpart is transparent to the FlexFrame Application Nodes. HA pair Filer “clustering”
is a functionality of NetApp Filers.
FlexFrame Architecture Storage Systems
40 Administration and Operation
2.7.2 SAN Support
As of FlexFrame Orchestrator 1.1. it is possible to use SAN based storage for the
database datafiles (sapdata) and database logfiles (saplog) of a SAP system.
Due to the more advanced integration into the FlexFrame functionality and the easier
setup and handling of the NAS based storage, it is recommended to use SAN based
storage only for the database servers for which the performance achievable with NAS
storage is not sufficient.
2.7.2.1 SAN Basic Layers
Configuring a SAN environment for one or multiple SAP databases can be a very
complex task. As with many complex tasks they become easier if you break it down into
little pieces. In order to make you understand the various layers of a SAN environment
we divide it along the path as seen from the disk to the database:
Level Description
DISK The real disk(s) contained in the storage system
RAID-GROUP One or multiple disks grouped together based on RAID
mechanisms provided by the storage system
STORAGE-LUN LUNs (or storage volumes) as seen by the storage system
SAN-FABRIC A SAN fabric to connect one or multiple storage systems with
one or multiple hosts. It consists of fibre channel cabling and
switches
HBA The host bus adapter which connects the host to the SAN
fabric
OS-DRIVER Basic OS drivers to make the HBA(s) work
HOST-LUN LUNs as seen by the host (Application Node)
MULTIPATHING OS specific multipathing software to make multiple HBAs to
work as a single virtual interface
VOLUME MANAGER An OS specific volume manager to group multiple LUNs from
the same or different storage systems into volume groups
FILE SYSTEM OS specific file systems created on top of the logical volumes
provided by the volume manager
DATABASE The database files contained on the file system(s)
Each of the layers above can be configured in many different ways using lots of
variations. Therefore not all of those layers are controlled by FlexFrame.
Storage Systems FlexFrame Architecture
Administration and Operation 41
2.7.2.2 Rules and Restrictions
This section lists some rules and restrictions which must be observed when setting up a
SAN configuration for FlexFrame usage.
4. All Application Nodes within one pool group must be configured equally. This
includes the number and type of HBAs. They must also use the same OS image.
5. All Application Nodes within one pool group must have access to the same set of
LUNs of each involved storage subsystem. Each Application Node of a pool group
must be able to access the same LUNs without reconfiguring the access settings of
the storage subsystem.
6. Host based mirroring can be used on volume manager layer to mirror one or multiple
LUNs from one storage subsystem to another (e.g. for a two-site-concept). Copy
mechanisms of the storage subsystems can also be used. If the failover to a different
storage subsystem changes the LUN identification data, the LUN definition in the
FlexFrame LDAP database must be changed accordingly.
7. A LUN is dedicated to a volume group. A volume group may contain more than one
LUN, but a LUN must not be shared between volume groups.
8. A volume group is dedicated to a SID. A SID with SAN support (or a
SID/instance/nodeid tuple, in case of HANA) uses two volume groups, one for
sapdata and one for saplog. A volume group must not contain volumes for more than
one SID.
The appropriate volume groups are switched from one Application Node to
the next if a database is moved.
9. Access to all file systems of a SID must be tested on each Application Node of its
pool group before it can be monitored and used (“watch”).
10. A database which was installed using SAN based database files in one pool group
cannot be started in a different pool group. If this is required the database must be
installed on NAS.
FlexFrame Basic AdministrationAccessing a FlexFrame Landscape (Remote Administration)
42 Administration and Operation
3 FlexFrame Basic Administration
3.1 Accessing a FlexFrame Landscape (Remote Administration)
A FlexFrame landscape can be accessed through Secure Shell (ssh, scp) connections
to the Control Node. Any other remote administration tools like rsh or telnet have been
disabled for reasons of security.
3.2 Powering up the FlexFrame Landscape
If the complete FlexFrame landscape was powered off, the following power-on sequence
is recommended:
Powering off the FlexFrame Landscape FlexFrame Basic Administration
Administration and Operation 43
Before an Application Node can be booted, the NTP server must be set up correctly. This
can be verified by running the following command:
control1:~ # ntpq -p
remote refid st t when poll reach delay offset jitter
=================================================================
*LOCAL(0) LOCAL(0) 5 l 45 64 377 0.000 0.000 0.004
control2-se control1-se 7 u 248 1024 377 2.287 2.587 0.815
This command needs to be repeated until an asterisk (*) is displayed at the beginning of
one of the data lines. This character indicates that the NTP server is now ready. If you do
not wait and continue with booting, the Application Nodes may work with a different time
than the Control Nodes and may (among other possible side effects) create files which
may have wrong time stamp information.
If this sequence is not used and all servers are powered on at the same time, the
Application Nodes will try to boot while the Control Nodes are not ready to
receive the Application Node's boot request. If this is the case, manual
intervention is required to re-initiate the boot process of the Application Nodes.
3.3 Powering off the FlexFrame Landscape
If you need to power-off the complete FlexFrame landscape (e.g. to move it to a different
location) we recommend following the steps as outlined below:
FlexFrame Basic Administration Powering off the FlexFrame Landscape
44 Administration and Operation
Before shutting down all SAP and DB services, check if users, batch jobs, print
jobs and RFC connections to other SAP systems have finished working.
To shutdown the SAP and DB services, use the following command for each pool:
control1:~ # stop_all_sapservices <pool_name>
Before you can stop the NAS-System, you need to stop all processes on the Control
Nodes. Since those processes are under control of the Linux-HA cluster you have to
switch all resources from control node 2 to control node 1 and then stop all resources.
control1:~ # ssh control2 /etc/init.d/openais stop
control1:~ # /etc/init.d/openais stop
Keep in mind, stopping all resources will take some time.
Now stop all running FlexFrame agents, e.g. FlexFrame LogAgent, on both Control
Nodes.
control1:~ # ssh control2 /etc/init.d/myAMC.FA_LogAgent stop
control1:~ # /etc/init.d/myAMC.FA_LogAgent stop
control1:~ # /etc/init.d/myAMC.FA_FrameAgent stop
control1:~ # /etc/init.d/myAMC.FA_DomainManager stop
And umount all NFS file systems on both Control Nodes.
control1:~ # ssh control2 umount -a -t nfs
control1:~ # umount -a -t nfs
Now the NAS systems can be safely switched off without impacting the Control Center
shutdown later on.
To stop the NAS system, use the following command.
For a NetApp Filer:
control1:~ # ssh <filer_name> halt
There is no explicit power-off sequence for the switches. Assuming there are no
other devices connected to the switches, they may simply be plugged-off after all
components were powered down.
If you do not send an explicit halt command to the Filer, the backup-battery of
the Filer may be drained since the Filer assumes a power-loss and tries to
preserve the contents of its NVRAM. If the Filer is powered-off for too long the
result can be a loss of NVRAM data and a long waiting period during next
startup of the Filer.
Powering off the FlexFrame Landscape FlexFrame Basic Administration
Administration and Operation 45
FlexFrame Basic AdministrationReactivating ANs after Power Shutdown by FlexFrame Agents
46 Administration and Operation
3.4 Reactivating ANs after Power Shutdown by FlexFrame Agents
FlexFrame Autonomy places an Application Node out of service if it is confronted with a
problem it cannot solve. The reactions, messages and alarms which take place in this
case are described in the “FlexFrame Agents Installation and Administration” manual in
the context of the switchover scenarios. It is the responsibility of the administrator to
analyze why the node could not be used any longer. We recommend analyzing the FA
log and work files as these may be able to supply valuable information. A node which is
to start operating as a Spare Node after switchover must be validated using suitable test
scenarios.
It may happen in some cases that the "placing out of service" of an Application Node fails
(e.g. when the IPMI-interface has broken down) and so it cannot be decided if the
Application Node is down.
In these cases - as default - the services that were running on the server are not switched
over to another node.
Therefore there is an additional high availability FlexFrame Agents parameter
"IgnoreShutdownFailure" with impact on the failover reaction for services.
Setting this parameter to value "true"
assigns the FlexFrame Agents to deactivate the network interfaces of the server
if successful the FlexFrame Agents automatically switch over services that were
running on the server to another node
the failed server can only be reactivated by manually reactivating its network
interfaces via the command ff_sw_ports.pl
(see the chapters "Switchover Control Parameters" and "Shutdown Configuration" in the
FlexFrame Agents Installation and Administration manual).
Default value "false" means, that a not successful server shutdown has no effect on the
server network interfaces and the services that were running on the server are not
switched over to another node.
Setting this parameter strengthens the service high availability, but the FlexFrame
administrator has to be aware of its impact in certain failure scenarios.
Recommendation is to set the parameter "IgnoreShutdownFailure" to "true" after
installation of the Flex-Frame system, if the FlexFrame administration wants to expand
the high availability of service and is willing to manually reactivate the network interfaces
of a server in such error cases.
Displaying the Current FlexFrame Configuration State FlexFrame Basic Administration
Administration and Operation 47
3.5 Displaying the Current FlexFrame Configuration State
To obtain a general overview of an active FlexFrame system, use the FlexFrame Agents
WebGUI.
In principle, the FlexFrame Agents WebGUI can be used with every browser with SUN
JAVA Plugin V1.4.1 or higher which has access to the page on the Control Node. The
WebGUI can always be accessed when the Apache Tomcat service is running. This
service is normally started by the Linux-HA cluster.
The WebGUI is described in detail in chapter 3.5.1.
The login mask expects a user name and password for authentication purposes. You can
only use the WebGUI with a valid combination of user name and password. For details on
the configuration of the users, see the FlexFrame Agent documentation for the WebGUI.
The WebGUI provides a presentation of all elements of a FlexFrame system. On the left-
hand side the pools, groups, nodes and the active SAP services on the individual nodes
are shown in a TreeView.
FlexFrame Basic Administration Displaying the Current FlexFrame Configuration State
48 Administration and Operation
The TreeView of the WebGUI can show either the physical view or the application-related
view of the active SAP systems and their instances or a mixed view.
The panels derived from the TreeView (Application Server Panel, Application System
Panel, ServiceView and MessageView) always show the objects in relation to the
selected hierarchical level in the TreeView.
The FlexFrame Agents WebGUI is thus the central cockpit for displaying the static
configuration of a FlexFrame infrastructure, but as well it is the display for the active SAP
systems and their instances. Here, all user interactions such as startup and shutdown are
shown directly. All reactions initiate by the FlexFrame Agents are displayed as well.
If the FlexFrame Messenger component has been configured and activated, all traps are
stored in a support database. This permits the temporary process for the traps to be
displayed very simply at pool, group or node level.
3.5.1 FlexFrame Web Portal
The FlexFrame Control Nodes provide a web portal with links to Web interfaces of
several FlexFrame components, i.e.
FlexFrame Agents
Cluster status
Supported hardware
Management Blades of BX cabinets
To access this portal, start firefox and enter the Control Node's IP address in the location
bar. If you are directly on the Control Node you want to configure, just call firefox from the
shell:
control1:~ # firefox localhost
You will see an overview of all Web interfaces installed on the Control Nodes.
Displaying the Current FlexFrame Configuration State FlexFrame Basic Administration
Administration and Operation 49
3.5.2 FlexFrame Agents
Use this tool to manage the virtualized services in your FlexFrame environment.
For information on usage, please refer to the FlexFrame Agents manuals.
You can access the FlexFrame Agents WebGUI directly from the active Control
Node by entering the following URL:
http://localhost:8080/FAwebgui
This only works if the Jakarta Tomcat web server is running. If it is not running,
check the Linux-HA cluster configuration.
3.5.2.1 State of Pools
The active configured pools are displayed on the FlexFrame Agents WebGUI. The nodes
or systems belonging to a pool are displayed in the WebGUI's TreeView. Each node element in the tree which represents a pool is identified by the prefixed keyword Pool.
3.5.2.2 State of Application Nodes
Each Application Node with a running FlexFrame Application Agent is shown in the
FlexFrame Agents WebGUI. It is shown in the Nodes TreeView with its host name
FlexFrame Basic Administration Displaying the Current FlexFrame Configuration State
50 Administration and Operation
(Linux), and also in the Application Server Panel. In the Application Server Panel the data
displayed depend on the hierarchical level selected in the TreeView.
3.5.2.3 State of SAP Systems
The active SAP system IDs can be displayed very easily in the Systems TreeView of the
FlexFrame Agents WebGUI and also in the SAP System Panel. The SAP System Panel
is shown in parallel to the Application Server Panel. In the SAP System Panel, the data
displayed depend on the hierarchical level selected in the TreeView.
3.5.2.4 State of SID Instances
The active instances of a SID can be displayed very simply in the InstancesView of the
FlexFrame Agents WebGUI by clicking on a SID in the SAP System Panel.
For each view, information is provided in tabular form specifying the service's current
pool, group, node, priority and status.
3.5.2.5 Virtual IP address for FlexFrame Agents WebGUI
The FlexFrame Agents WebGUI only works on the Control Node which is currently
running the ff_manage cluster resource group. This may cause confusion if this group is
switched over to the other Control Node.
It is possible to add a virtual IP address to the ff_manage resource group which will
ensure that the administrative access is always accessible by using a single address.
To add such an address, run the following command:
control1:~ # ff_ha_tool.sh -a ffmanageip ip=<address>
It is absolutely necessary to choose an address which meets the following criteria:
1. The address must be part of a subnet which already exists on the Control Node.
Ideally this is an address of the Client LAN.
2. The address must not exist before
Note that these critiera are not actively checked by the ff_ha_tool.sh command, the
address is simply being added to the cluster configuration. If the address cannot be
configured, the ff_manage group may be switched over to the other Control Node.
Verify the configuration and cluster state with ff_ha_cmd.sh status.
To remove a configured address, run:
control1:~ # ff_ha_tool.sh -d ffmanageip ip=<address>
It is possible to add multiple addresses, i.e. one for each Client LAN pool.
Displaying the Current FlexFrame Configuration State FlexFrame Basic Administration
Administration and Operation 51
3.5.3 Networks
The network of FlexFrame is its backbone. Here are some tips to get an overview of the
current situation on the various networks:
To double-check the network addresses, their names and pool assignment you can use
the getent command:
control1:~ # getent networks
loopback 127.0.0.0
control 192.168.20.0
storage_pool1 192.168.10.0
server_pool1 192.168.3.0
client_pool1 10.1.1.0
storage_pool2 192.168.11.0
server_pool2 192.168.4.0
client_pool2 10.1.2.0
The loopback network is local for each host and always has the IP address 127.0.0.0.
The control network is the Control LAN network segment for the complete FlexFrame
landscape.
In the example we have configured two pools called pool1 and pool2. For each pool
there are the three dedicated and distinct segments storage, server and client. The
building rule of the network name is <segment>_<pool_name>.
On the Control Nodes you can see the relation of each pool specific segment to its interface using the netstat -r command like this:
control1:~ # netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
server_pool2 * 255.255.255.0 U 0 0 0 vlan42
control * 255.255.255.0 U 0 0 0 bond0
server_pool1 * 255.255.255.0 U 0 0 0 vlan32
storage_pool2 * 255.255.255.0 U 0 0 0 vlan41
client_pool1 * 255.255.255.0 U 0 0 0 vlan30
storage_pool1 * 255.255.255.0 U 0 0 0 vlan31
client_pool2 * 255.255.255.0 U 0 0 0 vlan40
default gw216p1 0.0.0.0 UG 0 0 0 vlan30
Here you can quickly see that the Server LAN segment of pool2 (server_pool2) is using
the VLAN ID 42 on interface vlan42. Note that the VLAN ID of the Control LAN segment
cannot be seen because that vlan is native on bond0.
Administration and Operation 53
3.5.4 Cluster Status
A short overview of the cluster status is available from the FlexFrame Web Portal.
3.5.5 Supported Hardware
Clicking on the ‘Supported Hardware’ link in the FlexFrame Web Interface displays the
currently supported hardware and the features used by FlexFrame. The list includes
hardware that is supported only by an approved special release request.
If JavaScript is enabled, clicking on the table headers displays information about the
listed content.
FlexFrame Basic Administration FlexFrame Backup with Tape Library
54 Administration and Operation
3.5.6 Management Blades
Clicking on the links in the management blade section of the FlexFrame Web Interface
opens https connections to the respectice management blades of BX cabinets.
Note that the management blade section may be missing if the FlexFrame landscape
contains no BX cabinets.
3.6 FlexFrame Backup with Tape Library
Two differerent ways of backing up a database in a Flexframe environment have been
tested and are recommended. Both use third-party software, either Simpana (Comvault )
or Netbackup (Symantec), and both use different snapshot technology.
Corresponding best practice papers will be provided by Fujitsu Storage Consulting and
are available at https://partners.ts.fujitsu.com/com/products/infrastruc-
solutions/FlexFrame, see Best Practices / Description Papers.
See also
https://partners.ts.fujitsu.com/com/products/storage/software/Pages/default.aspx .
Concerning backup using snapshots there is a speciality with NetApp and
cDOT:
NetApp offers a popular function to access contents of snapshots by way of
filesystem, via the .snapshot directory.
In the cDOT architecture there is a change in the way directories are made
accessible for clients (or "are exported for clients" in the NFS terms) - in 7-
mode it was possible to make certain subdirectories accessible for certain
clients, whereas in clustermode (cDOT) it is only possible to make two levels
of directories accessible for clients, either whole volumes or qtrees (special
subdirectories) of volumes.
In order to make a qtree accessible for a client (= to export a qtree for a
client) it is necessary that the whole superior volume is accessible for the
client.
A problem arises, if snapshot directory access is allowed (when the volume
modify option -snapdir-access is set to true – default in Flexframe is
false).
Then this directory is visible on volume level, and snapshots in this directory
contain all qtrees of that volume, even if they belong to different clients
(application nodes), different pools, different mandants.
Then it would be possible that clients excess their border and are able to
access data of other mandants indirectly by accessing snapshots, which is
Product Licenses FlexFrame Basic Administration
Administration and Operation 55
untolerable due to security reasons.
Thus the snapshot directory access must be prohibited, leading to the fact,
that a snapshot directory cannot be used by backup functions using NFS
mount.
This is a constraint in cDOT environments which only NetApp can set aside
by changing their export-authorization policies.
3.7 Product Licenses
Use of the FlexFrame Software requires specific software licenses depending on the
required functionality.
FlexFrame Orchestrator comes with a license concept where licenses (SID Runtime,
ESX-Server, KVM-nodes, HANA OLAP DB, RBAC, ACCOUNTING, SAN) have been
defined. The FFO SID license is the base runtime license for running FlexFrame
Orchestrator. Every other FFO license allows the additive use of a special functionality of
FlexFrame Orchestrator.
3.7.1 How to Work with Licenses
The customer orders all required FlexFrame licenses by contacting his sales
representative. Beside the license documentation the customer receives an amount
of license key files (one file for every ordered license).
The customer installs each license key in the FlexFrame using the script
ff_license.pl.
If the customer wants to use a feature which is subject to a license in certain cases a
license check is executed.
Every day a special tool checks all features with FlexFrame Orchestrator licenses:
how many applications of this feature are currently available in the FlexFrame and
how many applications of this feature are currently allowed by the corresponding
license keys.
3.7.2 Types of Licenses in FlexFrame Orchestrator
In FlexFrame Orchestrator the following features are requiring an own license:
running SIDs
KVM nodes
ESX server
HANA OLAP DBs
RBAC (Role Based Access Control)
FlexFrame Basic Administration Product Licenses
56 Administration and Operation
Accounting
SAN
Every single license allows the use of a specified amount of applications of the
corresponding feature. First the customer orders a license with type BASE for every
required feature. If the amount of applications allowed by this BASE license is no more
sufficient the customer can order a number of additional licenses of this feature with type
EXTENT (i.e. one license of type BASE and 0 – n licenses of type EXTENT are required).
Special case: the features SID, RBAC and Accounting are additionally provided with the
type ENTRY. In place of a license with type BASE the customer can order a license with
type ENTRY. In this case he can order no additional licenses of type EXTENT for this
feature.
3.7.3 How to Install a License Key
Every required license key has to be installed individually.
To install a license key use the program ff_license.pl as root like:
/opt/FlexFrame/bin/ff_license.pl -–op add -file <license key file path>
An already installed Iicense key is not possible to install again.
The installation of the CNs is executed without FFO license checks. If both CN
installations are finished the customer has to install all the FFO license keys as soon as
possible.
3.7.4 How to List Installed License Keys
To list all installed license keys use the program ff_license.pl as root like:
/opt/FlexFrame/bin/ff_license.pl -–op list-all
Output: a list of all license keys installed in LDAP.
The given example shows the output of FlexFrame Orchestrator.
Example:jer1cn1:/opt/FlexFrame/bin # ff_license.pl -–op list-all
**** all FFO licenses in FlexFrame ****
licensed feature: SID
licensed type: BASE
licensed number of running SIDs: 12
license created at: 2013-07-01 12:00:00
Product Licenses FlexFrame Basic Administration
Administration and Operation 57
license valid till: unlimited
identifier (customer): Lufthansa
license created with FF version: 1.0
licensed feature: SID
licensed type: EXTENT
licensed number of running SIDs: 6
license created at: 2013-07-10 08:17:23
license valid till: unlimited
identifier (customer): Lufthansa
license created with FF version: 1.0
licensed feature: KVM
licensed type: BASE
licensed number of servers: 12
license created at: 2013-07-01 12:00:01
license valid till: unlimited
identifier (customer): Lufthansa
license created with FF version: 1.0
With /opt/FlexFrame/bin/ff_license.pl --op list -–feat <feature> all
license keys of the given feature are listed.
Features allowed with FFO are: SID, KVM, ESX, HANA, RBAC, ACCOUNTING, SAN.
3.7.5 How to Control the Use of Licenses
The new script ff_lic_usage.pl is called daily by a cronjob. With this script the
proper use of the FFO licenses is checked.
The features SID, KVM, ESX, RBAC, ACCOUNTING, SAN are checked by this tool: how
many applications of this feature are currently available in the FlexFrame and how many
applications of this feature are currently allowed by the corresponding license keys.
The summation of running SIDs is done without SMD SIDs.
SIDs without a DB are weighted only with 10 percent in the summation of running SIDs.
I.e with one SID license the customer is allowed to run 10 SIDs without a DB.
The result of this checks is written in the file /var/log/FlexFrame/licstat/FFO_license_status.<date>.
Files FFO_license_status.<date> older than 180 days are removed.
Pools and Groups Adding a Pool
58 Administration and Operation
4 Pools and Groups
4.1 Adding a Pool
A new pool can be added using ff_pool_adm.pl. Some parameters have to be defined
on the command line. They are used to configure switch VLANs and ports, to create the
NAS system volume folder structures, to create LDAP pool subtree and to configure
Control Nodes.
An ADMINPOOL (see chapter 6) is created, if <pool name> is adminpool or
force_admin is given.
If the pool shall be equipped with Application Nodes of type “BX600” (blade
server in a BX600 blade server cabinet with switch blades of type 1GbE-10/6-Q
(SB9a)), then the following restriction concerning the VLAN-ID has to be consi-
dered:
Switch blades of type 1GbE-10/6-Q only allow the VLAN ID range from 1 to
3965!
Creating a new pool adds VLAN associations for new VLANs to some ports, e.g.
to uplink ports. If special ports for Client LAN connection should be used the
behavior is determined according to global parameters originally given by the
Management Tool (MT, see the manual Management Tool, chapter “Network”
Object). These parameters can be displayed and changed by
ff_swgroup_adm.pl --op parameters (SWGADM) (see this manual
chapter 9.7).
The parameters are One ClientLAN per SWP (true/false) at MT or
clanPortPerVlan (yes/no) at SWGADM
true/yes Use an independent switch port pair for each pool Client LAN.
Every port pair will be configured with the pool Client LAN (untagged).
false/no All pool ClientLANs use the same switch port pair. Both ports are
configured as Cisco trunk ports, with tagged VLANs.
Use 1Gbit SWPs for ClientLANs at MT or
useTxToClan at SWGADM
Only one ADMINPOOL is allowed in a FlexFrame system.
Adding a pool changes the exports file on all given NAS systems. Temporary
exports on these NAS systems will be gone after running ff_pool_adm.pl.
Be sure not to have temporary exports.
Adding a Pool Pools and Groups
Administration and Operation 59
true/yes use copper twisted pair ports as CLAN ports.
false/no use fiber optic (SFP) ports as CLAN ports if available else use
copper ports.
So if the original global setting does not match your current wish for wiring the
new pool, you can change it temporarily with the two corresponding ff_swgroup_adm.pl commands.
Synopsis
ff_pool_adm.pl
--op add --name <pool name>
--storage <vlan id>,<network ip>,<netmask>
--server <vlan id>,<network ip>,<netmask>
--client <vlan id>,<network ip>,<netmask>
--dns <domain name>[,<dns server ip>[,<dns server ip>]
[--dns_search_list <list of domains>]
[--sapdata <nas name>[,<volume path>]]
[--saplog <nas name>[,<volume path>]]
[--volff <nas name>[,<volume path>]]
[--defrouter <default router ip>]
[--switchgrp <id>[,<id>]]
[--force_admin]
Options
--op add
Adds a pool and displays information about processing steps, errors and warnings.
--name <pool name>
Name of new pool (has to be unique withinFlexFrame). A pool name may consist of
2 up to 13 characters (letters, numbers and dashes). It must start with a letter and
must not end with a dash.. The pool name adminpool always creates an
ADMINPOOL.
--storage <vlan id>,<network ip>,<netmask>
Pool specific storage network segment. The option is followed by a comma
separated list of the VLAN ID, the network IP and the netmask of the network IP
address.
--server <vlan id>,<network ip>,<netmask>
Pools and Groups Adding a Pool
60 Administration and Operation
Pool specific server network segment. The option is followed by a comma separated
list of the VLAN ID, the network IP and the netmask of the network IP address.
--client <vlan id>,<network ip>,<netmask>
Pool specific client network segment. The option is followed by a comma separated
list of the VLAN ID, the network IP and the netmask of network IP.
--dns <domain name>,[<dns server ip>]
DNS domain name and servers to be used for this pool. More than one DNS server
IP address may be given. Keep in mind to use the default router option if server IP
addresses are given. A DNS option may look like this:
my.domain.com,192.168.251.17,192.168.251.18
--dns_search_list <list of domains>
List of DNS domain names to be used for this pool. At most six domain names are
allowed. For technical reasons the first DNS domain name of this list always must be the DNS domain name given in option dns.
--sapdata <nas name>[,<volume path>]
Optional NAS name and volume path the pool should use for sapdata. A missing
value is auto-filled with the default specified with installation and output with
operation --help.
--saplog <nas name>[,<volume path>]
Optional NAS name and volume path the pool should use for saplog. A missing
value is auto-filled with the default specified with installation and output with
operation --help.
--volff <nas name>[,<volume path>]
Optional NAS name and volume path the pool should use for volFF. A missing value
is auto-filled with the default specified with installation and output with operation --
help.
--defrouter <default router ip>
The default router is a gateway to route IP data to other, non-pool local networks. All
IP data that cannot be addressed to a local network will be sent to the default router
to be forwarded to the destination network. The option parameter is an IP address of
this default router. Use a default router IP address matching the client pool network,
because otherwise it will not be accessible by Application Nodes.
--switchgrp <id>[,<id>]
The switch group ID(s) the Client LAN to corporate LAN ports should be configured.
If not given, the client VLAN is assigned to the existing trunk ports or a new port pair
at first both switch groups. Not more than two switch group IDs are accepted.
Removing a Pool Pools and Groups
Administration and Operation 61
--force_admin
This pool (with any pool name) should be the ADMINPOOL
Example
cn1:~ # ff_pool_adm.pl --op add --name pool4
--storage 30,192.168.30.0,255.255.255.0 --server 31,192.168.31.0,255.255.255.0
--client 32,192.168.32.0,255.255.255.0 --sapdata filer --saplog filer
--volff filer --dns my.domain.com --defrouter 192.168.32.254
update LDAP
...
update switch 1/1 configuration
Notice: Update will take about 1 minute.
vlan: storage-30 has been created
restart cluster service ldap_srv1
Notice: restart will take up to 1 minute.
stop and wait until service is offline
start and wait until service is online
restart cluster service ldap_srv2
Notice: restart will take up to 1 minute.
stop and wait until service is offline
start and wait until service is online
restart cluster service netboot_srv
Notice: restart will take up to 2 minutes.
stop and wait until service is offline
start and wait until service is online
If not reported any warnings or errors all precautions are done
and the pool was successfully created.
Use ff_poolgroup_adm.pl to define the host groups of this pool
to be able to add application nodes.
4.2 Removing a Pool
A pool can be removed using ff_pool_adm.pl. Some parameters have to be defined
on the command line. Switch VLANs will be removed and the affected ports reconfigured.
The LDAP pool subtree will be removed and Control Node configurations rewritten.
The pool created with the attribute "CN Pool = true" may not be removed by an
administration command, since this pool is used to mount file systems from the NAS
device.
If the pool uses volumes different from common volFF these volumes and their data were
not touched. The customer has to remove the data manually.
Pools and Groups Removing a Pool
62 Administration and Operation
A pool may not be removed if any Application Node or SID is defined. The first
pool may not be removed due to system requirements. Removing a pool
changes the exports on all NAS systems used by this pool. Use list or list-
all to get the storage configuration of pool to be removed. Temporary exports
(not written to the exports file /vol0/etc/exports) on these NAS systems will
be gone after running ff_pool_adm.pl. Be sure not to have temporary
exports.
Synopsis
ff_pool_adm.pl --op rem --name <pool name>
Options
--op rem
Removes a pool and displays only errors and warnings.
--name <pool name>
Name of pool to be removed. Use ff_pool_adm.pl --op list-all to get a list
of currently configured pools (see 4.4).
Example
cn1:~ # ff_pool_adm.pl --op rem --name pool4
update LDAP
...
update switch 1/1 configuration
Notice: Update will take about 1 minute.
restart cluster service ldap_srv1
Notice: restart will take up to 1 minute.
stop and wait until service is offline
start and wait until service is online
restart cluster service ldap_srv2
Notice: restart will take up to 1 minute.
stop and wait until service is offline
start and wait until service is online
restart cluster service netboot_srv
Notice: restart will take up to 2 minutes.
stop and wait until service is offline
start and wait until service is online
If not reported any warnings or errors the pool was successfully removed.
Keep in mind, the volumes different from common volFF and their data were not
harmed. It's on you to remove them.
Listing Pool Details Pools and Groups
Administration and Operation 63
4.3 Listing Pool Details
To list the configurations details of a pool like used networks, pool groups, SIDs and Application Nodes, the maintenance tool ff_pool_adm.pl can be used. The pool name
has to be defined on the command line.
Synopsis
ff_pool_adm.pl --op list --name <pool name>
[--list <part>[,<part>]]
Options
--op list
Displays pool configuration details
--name <pool name>
Name of pool to be listed. Use ff_pool_adm.pl --op list-all to get a list of
currently configured pools (see 4.4).
--list <part>[,<part>]
To reduce the output to the interesting parts, use this option. The parameters to this
option are the display sections. Add them as a comma separated list. The default
sections are: network,storage,dns,group,sid,an,cn,NAS system. You may also use a
two character abbreviation instead the full section name like ne for network.
Pools and Groups Listing Pool Details
64 Administration and Operation
Examples
cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list --name p1
Pool configuration details of pool p1
Networks
Client-LAN
Network: 10.10.10.0 Netmask: 255.255.255.0 VLAN ID: 100
Server-LAN
Network: 192.168.10.0 Netmask: 255.255.255.0 VLAN ID: 110
Storage-LAN
Network: 192.168.20.0 Netmask: 255.255.255.0 VLAN ID: 120
Def.Router: 10.10.10.254
Storage Volumes
sapdata fas01-p1-st:/vol/sapdata/p1
saplog fas01-p1-st:/vol/saplog/p1
volFF fas01-p1-st:/vol/volFF/pool-p1
volFF shared fas01-p1-st:/vol/volFF
DNS data
Domain Name: my.domain.com
Pool Groups
Linux
SIDs D01 SAP Version: SAP-6.20 DB Version: Oracle-9 P01 SAP Version: SAP-
6.20 DB Version: Oracle-9
Q01 SAP Version: SAP-6.20 DB Version: Oracle-9
Application Nodes
blade01
Type: BX600 Cabinet ID: 1 Slot/Partition ID: 1
OS: SUSE Linux SLES-11.x86_64
Group: Linux
Client-LAN blade01 10.10.10.23
Server-LAN blade01-se 192.168.10.23
Storage-LAN blade01-st 192.168.20.23
blade02
Type: BX600 Cabinet ID: 1 Slot/Partition ID: 2
OS: SUSE Linux SLES-11.x86_64
Group: Linux
Client-LAN blade02 10.10.10.24
Server-LAN blade02-se 192.168.10.24
Storage-LAN blade02-st 192.168.20.24
blade03
Type: BX600 Cabinet ID: 1 Slot/Partition ID: 3
Listing Pool Details Pools and Groups
Administration and Operation 65
OS: SUSE Linux SLES-11.x86_64
Group: Linux
Client-LAN blade03 10.10.10.25
Server-LAN blade03-se 192.168.10.25
Storage-LAN blade03-st 192.168.20.25
rx801
Type: RX800
OS: SUSE Linux SLES-11.x86_64
Group: Linux
Client-LAN rx801 10.10.10.2
Server-LAN rx801-se 192.168.10.2
Storage-LAN rx801-st 192.168.20.2
Control Nodes
cn1
Client-LAN cn1-p1 10.10.10.10
Server-LAN cn1-p1-se 192.168.10.10
Storage-LAN cn1-p1-st 192.168.20.10
cn2
Client-LAN cn2-p1 10.10.10.11
Server-LAN cn2-p1-se 192.168.10.11
Storage-LAN cn2-p1-st 192.168.20.11
NAS Nodes
fas01-p1
Storage-LAN fas01-p1-st 192.168.20.14
A sample with a reduced output:
cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list --name p1 --list ne,gr
Pool configuration details of pool p1
Networks
Client-LAN
Network: 10.10.10.0 Netmask: 255.255.255.0 VLAN ID: 100
Server-LAN
Network: 192.168.10.0 Netmask: 255.255.255.0 VLAN ID: 110
Storage-LAN
Network: 192.168.20.0 Netmask: 255.255.255.0 VLAN ID: 120
Def.Router: 10.10.10.254
Pool Groups
Linux
Pools and Groups Listing All Pools
66 Administration and Operation
4.4 Listing All Pools
To display an overview of all pools with their used networks, pool groups, SIDs and Control Node and NAS system interfaces, the maintenance tool ff_pool_adm.pl can
be used. No arguments except the operation mode have to be defined on the command
line.
Synopsis
ff_pool_adm.pl --op list-all [--list <part>[,<part>]]
Options
--op list-all
Displays the configuration details of all configured pools.
--list <part>[,<part>]
To reduce output to interesting parts use this option. The parameters to this option
are the display sections. Add them as a comma separated list. The default sections
are: network,storage,group,sid,cn,NAS system. You may also use a two character
abbreviation instead the full section name like ne for network.
Examples
cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list-all
Pool configurations
p1
Pool Networks
Client-LAN
Network: 10.10.10.0 Netmask: 255.255.255.0 VLAN ID: 100
Server-LAN
Network: 192.168.10.0 Netmask: 255.255.255.0 VLAN ID: 110
Storage-LAN
Network: 192.168.20.0 Netmask: 255.255.255.0 VLAN ID: 120
Pool Storage Volumes
sapdata fas01-p1-st:/vol/sapdata/p1
saplog fas01-p1-st:/vol/saplog/p1
volFF fas01-p1-st:/vol/volFF/pool-p1
volFF shared fas01-p1-st:/vol/volFF
Pool Groups
Linux
Pool SIDs
D01 SAP Version: SAP-6.20 DB Version: Oracle-9 P01 SAP Version: SAP-
6.20 DB Version: Oracle-9
Listing All Pools Pools and Groups
Administration and Operation 67
Q01 SAP Version: SAP-6.20 DB Version: Oracle-9
Pool Control Node Interfaces
cn1
Client-LAN cn1-p1 10.10.10.10
Server-LAN cn1-p1-se 192.168.10.10
Storage-LAN cn1-p1-st 192.168.20.10
cn2
Client-LAN cn2-p1 10.10.10.11
Server-LAN cn2-p1-se 192.168.10.11
Storage-LAN cn2-p1-st 192.168.20.11
Pool NAS Node Interfaces
fas01-p1
Storage-LAN fas01-p1-st 192.168.20.14
A sample with a reduced output on a single pool configuration:
cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list-all --list sid,group
Pool configurations
p1
Pool Groups
Linux
Pool SIDs
D01 SAP Version: SAP-6.20 DB Version: Oracle-9 P01 SAP Version: SAP-
6.20 DB Version: Oracle-9
Q01 SAP Version: SAP-6.20 DB Version: Oracle-9
Pools and Groups Changing Pool DNS Domain
68 Administration and Operation
4.5 Changing Pool DNS Domain
Changing a pools DNS configuration can be done using the maintenance tool ff_pool_adm.pl. The DNS configuration is written to
/FlexFrame/volFF/pool-<pool name>/pooldata/config/etc/resolv.conf .
The DNS configuration is newly configured, therefore all domain names and ser-
vers to be used for this pool have to be given in the options. Any modification in resolv.conf, done by the customer, is lost.
If the DNS domain name of a pool is modified, afterwards some modifications
concerning the SAP services have to be done manually. These modifications
need a downtime of the SAP services.
Step 1: find all concerned host names
e.g. switch to an application node of the pool and execute ‘getent hosts | grep
<old dns domain>’.The first columne shows the concerned host names which
have to be modified in LDAP.
Step 2:
Change in LDAP the host entries (stored in Pools - <poolname> - Hosts).
Exchange the old dns domain name to the new one.
Step 3:
provide in SAP profile files the parameter SAPLOCALHOSTFULL with the new
domain name. Maybe other SAP files contain the domain name too and
therefore have to be changed.The changes in SAP files have to be done by a
SAP expert.
If in the command line of ff_pool_adm.pl a search-list with a list of dns domains
is given every dns domain of the list has to be inserted in every concerned host
entry in LDAP.
If until now more than one dns domain was given (by a previous search-list)
remove in LDAP in the concerned host entries all the dns domain names not
included in the new command line of ff_pool_adm.pl.
Synopsis
ff_pool_adm.pl --op dns --name <pool name>
--dns <dns domain>[,<dns server ip>
[,<dns server ip>] ]
[--dns_search_list <list of domains>]
Changing Pool Default Router Pools and Groups
Administration and Operation 69
Options
--op dns
Adds or changes DNS configuration of a pool.
--name <pool name>
Name of pool the DNS configuration has to be changed.
Use ff_pool_adm.pl –op list-all to get a list of currently configured pools
(see 4.4).
--dns <dns domain>[,<dns server ip>]
DNS domain name and servers to be used for this pool.
A dns_server_ip is only allowed if a default router of this pool is already specified.
--dns_search_list <list of domains>
List of DNS domain names to be used for this pool. At most six domain names are
allowed. For technical reasons the first DNS domain name of this list always must be the DNS domain name given in option dns.
Example
cn1:~ # ff_pool_adm.pl --op dns --name pool3 --dns fsc.net
NOTICE: IF YOU CHANGE THE DNS DOMAIN NAME YOU MAYBE HAVE TO DO SOME CHANGES
MANUALLY !
If at this pool some SAP installations are configured you have to change the domain
names of the Host Entries in LDAP. (Attention: this change needs a downtime of the
corresponding services.) If in the SAP profiles the Parameter SAPLOCALHOSTFULL is
using another domain name, the domain name has to be changed manually. The changes
in SAP files have to be done by a SAP expert.
DNS domain successfully changed at LDAP and AN images.
4.6 Changing Pool Default Router
Add or change a pool's default router with opcode defrouter, remove a pool's default
router with opcode rem-defrouter. The default router may be necessary to reach
external DNS servers. Within all cases a default router has to have an IP address
matching the client IP network (. The default router is used to transfer network traffic
between the local IP networks and external IP networks.
Pools and Groups Adding a Group to a Pool
70 Administration and Operation
Synopsis
ff_pool_adm.pl --op defrouter --name <pool name>
--defrouter <default router ip>
ff_pool_adm.pl --op rem-defrouter --name <pool name>
Options
--op defrouter
Adds or changes default router configuration of a pool.
--op rem-defrouter
removes default router configuration of a pool.
--name <pool name>
Name of pool the default router configuration has to be changed and removed
respectively.
Use ff_pool_adm.pl –op list-all to get a list of currently configured pools
(see 4.4).
--defrouter <default router ip>
The default router to be used for this pool to communicate with other, non local
networks. The IP address has to match to client pool network.
Example
cn1:~ # ff_pool_adm.pl --op defrouter --name pool3 --defrouter 192.168.100.200
Defaultrouter successfully changed at LDAP and AN images.
4.7 Adding a Group to a Pool
A pool may have more than one group. To add a group to a pool, the maintenance tool /opt/FlexFrame/bin/ff_poolgroup_adm.pl can be used. To associate a group to
a pool, some parameters have to be defined on the command line. Groups are used with
the FlexFrame Agents.
In the ADMINPOOL only a group named SPARE is allowed. This group acts exclusively
as a container for all types of spare nodes. The group is added to a pool in the LDAP
database.
Synopsis
ff_poolgroup_adm.pl --op add --pool <pool name>
--group <group name>
Removing Pool Group Pools and Groups
Administration and Operation 71
Options
--op add
Adds a group to a pool.
--pool <pool name>
Name of pool the group should be added to.
--group <group name>
The name of the group to be added.
If a new poolgroup is added and you want to run already installed HANA
systems in that new group you have to run ff_setup_sid_folder.sh once more
to create the new file and directory structures for the new poolgroup
according to the HANA file structure requirements. Take care and make sure
that concerned HANA systems are down.
4.8 Removing Pool Group
To remove a group of a pool, use the maintenance tool /opt/FlexFrame/bin/ff_
poolgroup_adm.pl. You have to define the pool and the group name on the command
line.
The group is removed from a pool in the LDAP database.
Synopsis
ff_poolgroup_adm.pl --op rem --pool <pool name>
--group <group name>
Options
--op rem
Removes a group from a pool.
--pool <pool name>
Name of the pool the group should be removed from.
--group <group name>
The name of the group to be removed.
4.9 Hosts Database
It may become necessary to have additional entries in the hosts database. Those entries
may be required by 3rd-party products installed on customized Application Node images.
Pools and Groups Hosts Database
72 Administration and Operation
The hosts database is stored in LDAP. To maintain the additional host entries use the command ff_hosts.sh.
You cannot remove names or addresses which are essential to the FlexFrame
landscape. Each pool has its own hosts database.
Therefore you have to maintain each pool individually.
4.9.1 Script: ff_hosts.sh
This tool allows the administrator to list, add and delete host names or aliases to the
LDAP database. Note: When adding host names with an IP address that matches one of
the pool's network segments or the Control LAN segment, the list of IP addresses for that
segment gets extended by the IP address of this host name to prevent automatic
allocation of the same IP address by other FlexFrame tools.
Only one option of -l, -a or -r can be used at one time.
Synopsis
ff_hosts.sh [-d] -p <pool name> [{-l|-a <ip> -n <host name>|-r
<host name>}]
Options
-d.
This option will log debug information
-l
List all the hosts entries for the pool as provided by option -p.
-p <pool name>
Pool where the host name should be added to.
-a <ip>
Add the IP address <ip>. Has to be used together with option -n. If an entry with
the IP address <ip> already exists, the name provided will be added as an alias.
-n <host name>
Host name <host name> will be added to the list.
-r <host name>
Hosts Database Pools and Groups
Administration and Operation 73
Deletes the host name or alias of name. The host name cannot be deleted if it has
additional aliases. Remove the aliases first.
Examples
The following example will list all additional hosts entries for pool poolname created
using this tool:
cn1:~ # ff_hosts.sh -l -p poolname
The following example will add a host newhost with the IP address 1.2.3.4 to the pool
poolname:
cn1:~ # ff_hosts.sh -p poolname -a 1.2.3.4 -n newhost
The following example will remove the hosts entry for newhost:.
cn1:~
User and Groups AdministrationCreate, Modify, Delete, or List User(s) for Application Nodes
74 Administration and Operation
5 User and Groups Administration
5.1 Create, Modify, Delete, or List User(s) for Application Nodes
Synopsis
/opt/FlexFrame/bin/ff_user_adm.pl
--op add –-user <user name> --pool <pool name>
[–-group {<group name> | <group id>},...]
[–-uid <uid number>]
[--home <home directory>] [–-pass <password>]
[--shell <login shell>] [-–gecos <comment>]
[--shadowmin <number of days>] [--shadowmax <number of days>]
[--shadowwarn <number of days>]
[--shadowinactive <number of days>]
[--shadowexpire <number of days]
/opt/FlexFrame/bin/ff_user_adm.pl
--op mod –-user <user name> --pool <pool name>
[–-pass <password>] --shell <login shell> [-–gecos <comment>]
[--shadowmin <number of days>] [--shadowmax <number of days>]
[--shadowwarn <number of days>]
[--shadowinactive <number of days>]
[--shadowexpire <number of days>]
[--home <home directory>]
[--group {<primary group name> | <primary group id>}]
[--uid <uid number>]
/opt/FlexFrame/bin/ff_user_adm.pl
--op rem –-user <user name> --pool <pool name>
/opt/FlexFrame/bin/ff_user_adm.pl
--op list –-user <user name> --pool <pool name>
/opt/FlexFrame/bin/ff_user_adm.pl
--op list-all --pool <pool name>
Options
--op add –-user <user name> --pool <pool name>
Creates a new user in a given pool.
Create, Modify, Delete, or List User(s) for Application NodesUser and Groups Administration
Administration and Operation 75
--op mod –-user <user name> --pool <pool name>
Modifies a user in a given pool.
--op rem –-user <user name> --pool <pool name>
Deletes a user in a given pool.
--op list –-user <user name> --pool <pool name>
Displays information of a user in a given pool.
--op list-all --pool <pool name>
Displays information of all users in a given pool.
--group <group name> | <group id>,...
A comma separated list of group names/ids the user belongs to. The first group is
taken as primary group. Default: gid number 1 (=other).
--group <primary group name> | <primary group id>
The affiliation to the primary group can be changed with this option.
--uid <uid number>
A specific uid number, if desired.
Default: One higher than the highest used uid number.
SAP recommends usage of uid numbers >= 3600.
In connection with the mod option the existence of the uid is not checked.
--home <home_directory>
Home directory of the user. If this option is not used the home directory is set to
/FlexFrame/pooldata/home/<user> and created if it does not exists.
The following rules are valid if the option is used:
<home_directory>::= <name-1>/<name-2> | <name-3>
Using ‘<name-1>/<name-2>’ or ‘<name3>’ creates a directory in
/FlexFrame/pooldata/<appropriate_dir> if the appropriate directory
does not exist and sets the home directory to
‘/FlexFrame/pooldata/<appropriate_dir> ’.
<home_directory>::= /home_sap/<name-1> | /<name-2>/<name-3>
Using ‘/home_sap/<name-1>’ sets the home directory to
‘/home_sap/<name-1>’ and creates the directory if it does not exist. Using
‘/<name-2>/<name-3>’ sets the home directory to’/<name-2>/<name-3>’
but does not create the appropriate directory
--pass <password>
User and Groups AdministrationCreating, Modifying, Deleting or Listing Group(s) for Application Nodes
76 Administration and Operation
Login password for the user. Default: password.
--shell <login shell>
Login shell of the user. Default: /bin/csh. In connection with the mod option the
existence of the login shell is not checked.
--gecos <comment>
Some comment used for the user's gecos attribute. Default: Normal user. This
replaces the previous used option comment.
--shadowmin <number of days>
Min. number of days until the password can be changed.
--shadowmax <number of days>
Max. number of days until the password have to be changed.
--shadowwarn <number of days>
Number of days you get a warning message before the password expires.
--shadowinactive <number of days>
Number of days of inactivity that is allowed for this account.
--shadowexpire <number of days>
Number of days after which the account expires.
5.2 Creating, Modifying, Deleting or Listing Group(s) for Application Nodes
This command enables you to extend the operating system group entries stored in the
LDAP database. The modification or deletion of group entries is restricted to those entries
which are created with ff_group_adm.pl.
Synopsis
ff_group_adm.pl --op add –-name <osgroup name> --pool <pool name>
[–-gid <group id>] [--member <member, ...>]
[--gmember <member, ...>] [-–text <description>]
[--force] [--fname <filename>]
ff_group_adm.pl --op mod –-name <osgroup name> --pool <pool name>
[–-gid <group id>] [--member <member, ...>]
[--gmember <member, ...>] [--force]
[--fname <filename>]
Creating, Modifying, Deleting or Listing Group(s) for Application NodesUser and Groups Administration
Administration and Operation 77
ff_group_adm.pl --op rem –-name <osgroup name> --pool <pool name>
[--member <member, ...>] [--gmember <member, ...>]
[--fname <filename>]
ff_group_adm.pl --op list –-name <osgroup name> --pool <pool name>
ff_group_adm.pl --op list-all [–-name <osgroup name>]
[--pool <pool name>]
ff_group_adm.pl --help
Options
--op add
Creates a new group in a given pool.
--op mod
Modifies some of the attributes of the group entry in a given pool.
--op rem
.Removes some of the attributes of the group entry or the entry itself.
--op list
Displays information of group entries in a given pool.
--op list-all
Displays information of all group entries in a given pool or in all pools.
--name <osgroup name, ...>
A comma separated list of group names.
--pool <pool name>
Name of the pool the group should belong to.
--gid <gid number>
A certain GID number, if desired. If a list of groups is specified, it is the start GUID
number for the first group. For the next group names the value is always
incremented by one. If the increment means a conflict with an already existing GUID
the script looks for the next free GUID.
You can modify the GID with operation mod, but you are responsible for the effects.
It is also not possible to use a list of groups if you want to change a certain GID.
--member <member, ...>
A comma separated list of user names which should belong to this group. No check
is done if the user really exists.
User and Groups AdministrationCreating, Modifying, Deleting or Listing Service(s) for Application Nodes
78 Administration and Operation
--gmember <member, ...>
A comma separated list of user names which is inserted into the group members list
of a group. This option requires a modified flexframe.schema file. Group
members usually used with a DB2 installation.
--text <description>
A user specific description of the group entry.
--force
The add operation continues with the next group if any add operation before
terminates unexpectedly.
--fname <filename>
Name of the file to store the used LDIF statements for this request.
--help
Shows the usage of the command.
5.3 Creating, Modifying, Deleting or Listing Service(s) for Application Nodes
This command enables you to extend the service entries stored in LDAP database. The
modification or deletion of service entries is restricted to those entries which are created
with ff_services_adm.pl.
Synopsis
ff_services_adm.pl --op add –-name <service name, ...>
--pool <pool name> [–-port <port number>]
[--prot {udp|tcp}] [-–text <description>]
[--force] [--fname <filename>]
ff_services_adm.pl --op mod –-name <service name>
--pool <pool name> [–-port <port number>]
[--prot {udp|tcp}] [--force]
[--fname <filename>]
ff_services_adm.pl --op rem –-name <service name, ...>
--pool <pool name>
ff_services_adm.pl --op list –-name <service name, ...>
--pool <pool name>
Creating, Modifying, Deleting or Listing Service(s) for Application NodesUser and Groups Administration
Administration and Operation 79
ff_services_adm.pl --op list-all [–-name <service name>]
[--pool <pool name>]
ff_services_adm.pl --help
Options
--op add
Creates new services in a given pool.
--op mod
Modifies a service in a given pool.
--op rem
Deletes services in a given pool.
--op list
Displays information of service entries in a given pool.
--op list-all
Displays information of all service entries in a given pool or in all pools.
--name <service name>,...
A comma separated list of service names.
In case of operation mod you cannot use a list of services.
--pool <pool name>
Name of the pool the services should belong to.
--port <port number>
A certain port number, if desired. If a list of services is specified it is the start port
number for the first service. For the next service names the value is always
incremented by one. Portnumbers has to be unique.
--prot {udp|tcp}
Specifies the used protocol (default tcp).
--text <description>
A user specific description of the service entry.
--force
The add operation continues with the next service if any add operation before
terminates unexpectedly.
--fname <filename>
User and Groups AdministrationCreating, Modifying, Deleting or Listing Service(s) for Application Nodes
80 Administration and Operation
Name of the file to store the used LDIF statements for this request.
--help
Shows the usage of the command.
Administration and Operation 81
6 Pool-independent Spare Nodes
Before FlexFrame version 4.2 a group specific failover concept for Application Nodes was
used, i.e. if a node in a certain pool group failed, the FlexFrame Agents searched another
Application Node in this group with enough free capacity to take over the service of the
failed node. As of FlexFrame version 4.2 the new concept of pool-independent spare
nodes was introduced:
If no adequate spare node is found in the group of the failed node, the FlexFrame Agents
can search a spare node in the special pool ADMINPOOL, which mostly is called
adminpool (except in cases, where this name has already been reserved). The
ADMINPOOL is a reservoir of spare nodes for all pools of a FlexFrame landscape, and it
must not serve as a normal production pool.
A spare node in the ADMINPOOL is ready for takeover by the FlexFrame Agents
only if it has been booted at least once!
To move a spare node from the ADMINPOOL to its target pool and target group, the option --op move of ff_an_adm.pl has been implemented.
Pay attention to the restrictions of the move operation of ff_an_adm.pl as
described in section 7.5 “Moving Application Nodes
The following is an overview of the details of this concept.
6.1 Creation of Spare Nodes in the ADMINPOOL
To create spare nodes, you have to perform the following steps using the Management
Tool or the corresponding administration scripts (see parentheses):
1. Create the ADMINPOOL (ff_pool_adm.pl –-op add; see section 4.1)
2. Create the pool group SPARE within this pool (ff_poolgroup_adm.pl –-op add;
see section 4.7)
3. Add the necessary spare nodes to this group (ff_an_adm.pl –-op add; see
section 7.2)
4. Create the os images with ff_new_an.sh and boot them.
6.2 Moving of a Spare Node
To move a spare node from the ADMINPOOL to the desired target pool and target group,
use ff_an_adm.pl –op move (see section 7.5).
Pool-independent Spare NodesHandling Pool-independent Spare Nodes with FlexFrame Agents
82 Administration and Operation
6.3 Handling Pool-independent Spare Nodes with FlexFrame Agents
If no adequate spare node is found in the group of the failed node, the FlexFrame Agents
can search a spare node in the ADMINPOOL. For more information see the “FlexFrame
Agents Installation and Administration“ manual.
Administration and Operation 83
7 Application Nodes Administration
Manual Application Node administration would be very complex and error-prone. The
script /opt/FlexFrame/bin/ff_an_adm.pl does the major changes and supports
adding, changing, removing and listing of Application Nodes. Below, each action is des-
cribed in detail.
7.1 Listing Application Nodes
7.1.1 Displaying Information on a Specific Application Node
Synopsis
ff_an_adm.pl --op list --name <node name> [--cmdline]
Options
--op list
Lists the configuration details of an Application Node.
--name <node name>
The name of the Application Node to be listed.
--cmdline
The command line that used to create the Application Node is displayed at the end
of the node listing. This may be used as a preparation step to rename the AN.
The output is structured in sections: hardware, software, network, pool and group and
LAN interface connections or virtual machine.
Hardware
This section contains information about system type, shutdown facility with IP
address and host name, mac addresses and on blade ser-vers the chassis and slot
ID.
Software
This section contains information on OS type, vendor and version and the root
image path.
Network
This section lists the VLAN ID, IP address and host name of all configured networks,
sorted by LAN segments.
Pool and Group
Application Nodes Administration Listing Application Nodes
84 Administration and Operation
This section lists the names of the assigned pool and group.
LAN Interface Connections
In case link aggregates are configured, this section identifies the aggregate and its
ports. Each used switch port is shown with the switch group ID, switch ID and port ID
(cabinet ID, switch blade ID and port on blade servers) for the common LANs
(Storage, Server and Client LAN) and the Control LAN (if used).
Listing Application Nodes Application Nodes Administration
Administration and Operation 85
Examples
The command displays detailed information on the selected Application Node. The output
differs between blade servers and all others.
Example of a PRIMERGY blade server output:
cn1:~ # ff_an_adm.pl --op list --name bx924
Configuration details of node bx924
Hardware
System: BX924S2
10GBit: yes
Shut.Facil.: Mgmt Blade bx91-co (172.20.13.40)
iRMC Facil.: iRMC bx924-co (172.20.13.41)
MAC Addr.: 00:23:8b:97:ca:02 00:23:8b:97:ca:03
Cabinet/Slot: 1 / 1
Software
OS: SuSE / Linux / SLES-11.x86_64 (Vendor / Type / Version)
OS Path: Linux/FSC_1.1A00-000.SLES-11.x86_64
Network
VlanID Host IP Hostname
Storage LAN: 2021 172.20.21.51 bx924-st
Server LAN: 2022 172.20.22.51 bx924-se
Client LAN: 2020 172.20.20.51 bx924-1
Pool and Group
Pool: pool2
Group: group1
LAN Interface Connections
Cabinet SwBlade Port
data NIC-1: 3 1 1
data NIC-2: 3 2 1
Application Nodes Administration Listing Application Nodes
86 Administration and Operation
Example of a PRIMERGY rack server output:
cn1:~ # ff_an_adm.pl --op list --name rx36-1
Configuration details of node rx36-1
Hardware
System: RX300S6
10GBit: no
Shut.Facil.: IPMI rx36-1-co (172.20.13.23)
MAC Addr.: 00:c0:9f:dc:3a:da 00:c0:9f:dc:3a:db
Software
OS: SuSE / Linux / SLES-11.x86_64 (Vendor / Type / Version)
OS Path: Linux/FSC_1.1A00-000.SLES-11.x86_64
Network
VlanID Host IP Hostname
Storage LAN: 2011 172.20.11.1 rx36-1-st
Server LAN: 2012 172.20.12.1 rx36-1-se
Client LAN: 2010 172.20.10.1 rx36-1
Pool and Group
Pool: pool1
Group: group1
LAN Interface Connections
SwGroup Switch Port
data NIC-1: 1 2 5
data NIC-2: 1 1 5
IPMI NIC-1: 1 2 24
Listing Application Nodes Application Nodes Administration
Administration and Operation 87
7.1.2 Displaying Information of all Application Nodes
Synopsis
ff_an_adm.pl --op list-all [--pool <pool name>]
Options
--op list-all
Lists all configured Application Nodes.
--pool <pool name>
The name of the pool of which the Application Nodes have to be listed.
Example
cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op list-all
Nodes sorted by pool, group and name
Pool pool1
Pool Group group1
bx924
System: BX924S2
MAC Addr.: 00:23:8b:97:ca:02 00:23:8b:97:ca:03
Cabinet/Slot: 1 / 1
OS: SuSE / Linux / SLES-11.x86_64 (Vendor / Type / Version)
OS Path: Linux/FSC_1.1A00-000.SLES-11.x86_64
VlanID Host IP Hostname
Storage LAN: 2021 172.20.21.51 bx924-st
Server LAN: 2022 172.20.22.51 bx924-se
Client LAN: 2020 172.20.20.51 bx924-1
Pool pool2
Pool Group group2
bx36-1
System: RX300S65
MAC Addr.: 00:c0:9f:dc:3a:da 00:c0:9f:dc:3a:db
OS: SuSE / Linux / SLES-11.x86_64 (Vendor / Type / Version)
OS Path: Linux/FSC_1.1A00-000.SLES-11.x86_64
VlanID Host IP Hostname
Storage LAN: 2011 172.20.11.1 rx365-1-st
Server LAN: 2012 172.20.12.1 rx365-1-se
Client LAN: 2010 172.20.10.1 rx365-1
Application Nodes Administration Adding Application Nodes
88 Administration and Operation
The output of list-all is less detailed than the list output. It is used to get an
overview. It shows the Application Nodes sorted by pool and group in alphabetical order.
For each node the system type, the cabinet and slot ID (if node is a blade server), the OS
type, vendor, version, the root image path, the main IP addresses, host names and the
MAC addresses are listed.
7.1.3 Displaying all available OS images
Synopsis
ff_an_adm.pl --op list-images
Lists all available OS images.
Example
cn1:~ # ff_an_adm.pl --op list-images
Available OS Images are:
Linux/FSC_1.1A00-008.SLES-11.x86_64
Linux/MNT_1.1A00-000.SLES-11.x86_64 (maintenance only (pa-rx300-1))
7.2 Adding Application Nodes
This section describes how to provide the required information for adding a new AN to an
existing FlexFrame environment. See also section 11.6.1.
You have to define some parameters at the command line. They are used to configure
switch ports, to create the boot information and the OS image.
Synopsis
ff_an_adm.pl --op add --type <system type> --name <node name>
--pool <pool name> --group <group name>
--swgroup <switch group id>
Adding an Application Node changes the exports file on the common volFF NAS
system. Temporary exports on this NAS system will be gone after running
ff_new_an.sh. or ff_an_adm.pl with option --new-image. Be sure not to
have temporary exports.
Adding Application Nodes Application Nodes Administration
Administration and Operation 89
--mgmtswgroup <switch group id>
--mac <mac addresses>
--ospath <path to os image>
[--new-image]
[--host <ip host number> [,<cntl-LAN ip host number>,
<2nd cntl-LAN ip hostnumber>]]
[--slot <cabinet/slot>]
[--10gbit]
[--port switch:port,switch:port,switch:port,
[switch:port]]
[--hn <hypervisor node name>]
[--vcpus <number of virtual cpus>]
[--vmem <virtual machine memory size]
[--force]
Options
--op add
Adds an Application Node and displays some information about processing steps.
--type <system type>
Specifies the product name and type. Call ff_an_adm.pl without any parameter to
get a list of supported system types. A system type of ESXVM or KVMVM denotes a
virtual machine on an ESXi host or a KVM host, respectively, and a virtual machine
with the same name is implicitly created on the denoted hypervisor node
--name <node name>
The name of the Application Node. This name has to be unique in the FlexFrame
system. All interface names are based on this node name. The name may consist of
2 up to 13 characters (letters, numbers and dashes). It must start with a letter and
must not end with a dash.
--pool <pool name>
The name of the pool this node should belong to. See usage (call ff_an_adm.pl
without any parameter) to get a list of currently configured pools.
--group <group name>
The name of the pool group this node is a member of. A group must consist of
Application Nodes of the same OS image version and should be of the same
capacity (CPU, Memory etc.). Use command ff_pool_adm.pl with op mode list
or list-all to get the pool groups.
Application Nodes Administration Adding Application Nodes
90 Administration and Operation
--swgroup <switch group id>
Defines the switch group the Application Nodes data NIC 1 and 2 are connected to.
This information is necessary to assign and configure switch ports. Call
ff_an_adm.pl without any parameter to get a list of currently configured switch
group IDs.
--mac <mac addresses>
Add here both MAC addresses of the data NICs used for booting. Use the colon
separated hex notation for each MAC address. Concatenate them with a comma.
The MAC address syntax is a six colon separated hex values, eg.
00:e0:00:c5:19:41. For an Application Node with type ESXVM or KVMVM, an
appropriate MAC address is generated by the script.
--ospath <path to os image>
Defines the OS image to be used. Add the relative path to
/FlexFrame/volFF/os/ as seen from the Control Node.
Use ff_an_adm.pl --op list-images to get a list of available OS paths.
--host <ip host number>[,<cntl-LAN ip host number>,
2nd cntl-LAN ip host number>]
Host part to be used to build IP addresses for the three networks. If necessary host
part(s) to be used to build IP addresses for the management network can be added,
separated by commas. If this option is omitted, the script uses free host numbers to
calculate the IP addresses.
--slot <cabinet_id/slot>
With PRIMERGY server blades use this option to define the cabinet and slot ID of
the server blade. New cabinets have to be defined with the
/opt/FlexFrame/bin/ff_bx_cabinet_adm.pl command.
For models that occupy more than one slot in width (e.g.BX630 S2 quad),
the part of the server blade that occupies the slot with the highest slot
number is called the master. The master is usually the rightmost slot and its
slot ID has to be chosen as the master slot ID.
--10gbit
Specifies that the node is used with 10 Gigabit data NICs. The specification can be
omitted if the nodes system type only allows usage with 10 Gigabit data NICs.
--mgmtswgroup <switch group id>
Defines the switch group the Application Nodes management interface (IPMI) is
connected to. If omitted the effective switch group is computed as follows: First the
Adding Application Nodes Application Nodes Administration
Administration and Operation 91
switch group given with --swgroup is considered, second the switch group used for
the swgroup out of band management is considered. Call ff_an_adm.pl --help to get
a list of currently configured switch group IDs.
--new-image
Creates a new os image (need not call ff_new_an.sh !).
--port switch:port,switch:port,switch:port,[switch:port]
Defines the switch ports to be used. First two tuples are for data NIC 1 and 2 and
allocated in the switch group defined with --swgroup. The following tuples are for
mgmt NICs and allocated in the effective switch group for management interfaces. If
--10gbit is specified, the first two ports must be 10 Gigabit capable ports.
--hn <hypervisor node name>
When adding a node with type ESXVM or KVMVM, this parameter is mandatory and
specifies the name of the hypervisor node where the virtual machine must be
created. A hypervisor node with this name must already exist in FlexFrame and must
be configured for FlexFrame usage. For details refer to chapter 7.7 "Administrating
Hypervisor Nodes and Virtual Machines".
--vcpus <number of virtual cpus>
Specifies the number of virtual CPUs when creating a virtual machine. The default is
2 CPUs.
--vmem <virtual machine memory size in MB>
Defines the memory size in MB when creating a virtual machine. The default is 8
GB. The available resources of the hypervisor node must be taken into account
when choosing a value for this parameter. For details refer to section 7.7.9 “Virtual
Machine Properties and Hypervisor Resources“.
--force
Specifies that the memory usage of the hypervisor node may be overcommitted
when creating a virtual machine. Default is to deny creation of a virtual machine on a
host if the total vmem of the virtual machines on this host, including the new one,
exceeds the memory size of the host
Examples
Output for a blade server:
cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op add --type BX924S1
--name bx1-6 --pool pool1 --group bx900
--ospath Linux/FSC_5.1A00-000.SLES-11.x86_64 --host 1 --slot 1/6
--mac 00:C0:9F:99:E6:CC,00:C0:9F:99:E6:CD
update swblade 1/1 configuration
Application Nodes Administration Removing Application Nodes
92 Administration and Operation
Notice: Update will take about 1 minute.
update swblade 1/2 configuration
Notice: Update will take about 1 minute.
If not reported any error all precautions are done to create
application nodes os image. To do this call:
ff_new_an.sh -n bx1-6
Creating and customizing an image may take some minutes.
Don't get anxious.
Output for a non-blade server:
cn1:~ # ff_an_adm.pl --op add --name rx300-1 --type RX300S6 --pool pool2 --group
group1 --swgroup 1 --ospath Linux/FSC_5.1A00-000.SLES-11.x86_64 --mac
00:15:17:2d:ab:a8,00:15:17:2d:ac:02
update switch 1/1 configuration
Notice: Update will take about 1 minute.
Connect your systems LAN interfaces to named switch ports:
SwitchGroup / Switch / Port LAN Interface
1 / 1 / 6 data NIC-1
1 / 2 / 6 data NIC-2
1 / 1 / 24 IPMI NIC-1
If not reported any error all precautions are done to create
application nodes os image. To do this call:
ff_new_an.sh -n rx300-1
Creating and customizing an image may take some minutes.
Don't get anxious.
The script first checks all arguments and aborts with error messages in case of errors.
Then it fetches free IP addresses and switch ports. The switch ports are reconfigured to
match requirements, the LDAP data is created and a netboot file is written. The netboot
file is used by ff_new_an.sh to create Application Node images and extend the NAS
system's exports list. At the end you get a cabling advice and instructions how to call
ff_new_an.sh script to finish the Application Node creation.
7.3 Removing Application Nodes
You only have to specify the node name to be removed at the command line. All switch
ports will be unconfigured and the boot information and OS image are deleted. For an
Application Node of type ESXVM or KVMVM, the virtual machine is also destroyed.
Changing the attributes of an Application Node Application Nodes Administration
Administration and Operation 93
Synopsis
ff_an_adm.pl --op rem --name <node name>
Options
--op rem
Removes an application node.
--name <node name>
The name of the Application Node to be removed. Use operation mode list-all
to get all configured Application Nodes (see 7.1.2).
Example
cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op rem --name rx300-1
7.4 Changing the attributes of an Application Node
There are some operations to change attributes of the application node.
Keep in mind: the name or the pool of an application node can only be changed with
operation move.
7.4.1 Change the group of an Application Node
Synopsis
ff_an_adm.pl --op group --name <node name> --group <group name>
Removing an Application Node results in direct deletion of its image, removal of
its LDAP entries as well as disabling the respective switch ports. All existing
images (OS images and maintenance images) created for this node are
destroyed. Please make sure you really want to remove the Application Node
(AN) when calling the script, the script does not ask for further
confirmation.Removing an Application Node changes the exports file on the
common volFF NAS system. Temporary exports on this NAS system will be
gone after running ff_an_adm.pl. Please make sure not to have temporary
exports.
If the Application Node is used with a host based SMD agent the SMD Agent
instance for this host remains defined in pool and the customer has to change
the SMD configuration manually.
Application Nodes Administration Changing the attributes of an Application Node
94 Administration and Operation
Options
--op group
change the pool group the node belongs to.
--group
the new group of the node.
Example
cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op group --name rx300-1 –group newgroup
7.4.2 Change the os image of an Application Node
Synopsis
ff_an_adm.pl --op os --name <node name> --ospath <path to os
image> [--remove-image]
Options
--op os
change the os image path.
--ospath
The new path of the os image to be used.
--remove-image
On os path changes the former image will be left intact, to be able to change
between them. To remove the old image specify this command line option.
Example
cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op os --name rx300-1 –os_path
Linux/FSC_1.1A00-008.SLES-11.x86_64
Moving Application Nodes Application Nodes Administration
Administration and Operation 95
7.4.3 Change the mac addresses of an Application Node
Synopsis
ff_an_adm.pl --op mac --name <node name> --mac <mac addresses>
[--type <system type>]
Options
--op mac
change the mac addresses of an application node.
--mac
Define all mac addresses of the network interfaces the node should be able to boot
from. The program will change the mac addresses of netboot configuration.
--type
This parameter may be used to change the system type in case of hardware
changes. This may lead to a change of boot configuration files to support different
hardware class due to system type change.
Example
cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op mac --name rx300-1 --mac
c8:0a:a9:57:ad:5a,c8:0a:a9:57:ad:5b
7.5 Moving Application Nodes
Moving of Application Nodes from one pool to another pool may often be necessary, spe-cially for pool-independent spare nodes (see chapter 6). The opcode move with to-
pool, to-group serves to satisfy this requirement. While performing the move action
between pools, the node gets new Vlans in its new target pool, but changes of network
cabling are not necessary.
Changing of node names may be necessary for various reasons. With opcode move the
nodename can be changed either in the same pool or along with moving into another
pool.
The operation move is not supported for virtual machine Application Nodes.
The operation move is not allowed for a move into a new pool and a new pool group,
if the new pool group is assigned to some SAN volume groups.
Application Nodes Administration Moving Application Nodes
96 Administration and Operation
If the Application Node is used with a host based SMD agent the SMD Agent instance for
the host remains unchanged and the customer has to change the SMD configuration
manually.
Only if the move is a renaming of the application node in the same pool with a new name
the SMD Agent instance is changed too. In this case it has to be checked if the renaming
of the node invalidates the reference information. The corrresponding SAP Service
description has to be adpated also.
All existing images (OS images and maintenance images) created for this node are
destroyed, the OS image is newly created.
Synopsis
ff_an_adm.pl --op move --name <node name> [--to-pool <pool name>]
[--to-group <group name>] [--newname <new node name>]
[--to-host <ip host number>[,<cntl-LAN ip host
number>,<2nd cntl-LAN ip host number>]]
[--to-ospath <path to os image>]
[--failed-host <node name>] [--new-image]
[--clean-swap]
Options
--op move
Moves the Application Node given by --name.
--name <node name>
Current name of the node to be moved.
--to-pool <pool name>
Name of target pool.
--to-group <group name>
Name of target group.
--newname <new node name>
New name of the node. The name may consist of 2 up to 13 characters (letters,
numbers and dashes). It must start with a letter and must not end with a dash.
--to-host <ip host number>[,<cntl-LAN ip host number>, <2nd cntl-
LAN ip host number>]
Administrating Blade Server Cabinets Application Nodes Administration
Administration and Operation 97
Host part of Ip address in the target pool.
--to-ospath <path to os image>
Os image path the moved node should use.
--new-image
Creates a new os image (need not call ff_new_an.sh !).
--clean-swap
Clean local disks of node on first boot of moved node. This parameter has only
meaning together with --new-image.
If you specify --failed-host, all other options except –name, --new-image and –
clean-swap are ignored. In this case all necessary information for the moved node in its
target pool is derived from the failed host.
7.6 Administrating Blade Server Cabinets
Some network settings have to be changed to add or remove a blade server cabinet. The
script /opt/FlexFrame/bin/ff_bx_cabinet_adm.pl will simplify the administration
by doing LDAP changes automatically and preparing configurations to be done manually.
The script supports adding, moving, removing and listing of blade server cabinets as well
as modifying switch blade attributes. Each action is described in detail below.
7.6.1 Listing Blade Server Cabinets
7.6.1.1 Displaying Information on a Specific Blade Server Cabinet
Synopsis
ff_bx_cabinet_adm.pl --op list --name <cabinet name>
Options
--op list
Lists the configuration details of a blade server cabinet.
--name <cabinet name>
The name of the blade server cabinet to be listed.
The output is structured in sections: hardware, software, network, assigned pool and
group, switch ports.
Application Nodes Administration Administrating Blade Server Cabinets
98 Administration and Operation
Output example
PRIMERGY Cabinet 1 (cab1)
System Type: BX900S1
Management Blade
Hostname / IP Address: cab1-co 195.40.224.75
Integrated LAN Switch Ports: SwitchGroup SwitchID PortID
1 1 8
1 2 8
Switch Blade
SwitchID Type Switch name Hostname IP Address
1 10GbE-18/8-F cab1-swb1 cab1-swb1-co 195.40.224.76
2 10GbE-18/8-F cab1-swb2 cab1-swb2-co 195.40.224.77
Switch Blade Port Integrated LAN Switch Port
Switch Blade ID PortID <--> SwitchGroup SwitchID PortID
1 19 2 1 1/14
1 20 2 2 1/13
2 19 2 2 1/14
2 20 2 1 1/13
As seen from the sample above, the cabinet ID and name, the cabinet system type, the
management blade and the switch blades are listed.
For the management blade the host name, the IP address and both FlexFrame integrated
LAN switch ports are displayed.
The switch blade information shows the switch and host name, the IP address and the
switch blade port to FlexFrame integrated LAN switch port connections, structured by
switch blade ID.
7.6.1.2 Displaying Information on all Configured Blade Server Cabinets
Synopsis
ff_bx_cabinet_adm.pl --op list-all
Option
--op list-all
Lists all configured blade server cabinets.
Output example
PRIMERGY Cabinets
1 (cab1) BX600S3
Administrating Blade Server Cabinets Application Nodes Administration
Administration and Operation 99
Management Blade: cab1-co / 195.40.224.75
Switch Group ID: 1
Server Blades (by slot id)
1 (blade1) BX924S1
Pool / Group: pool1 / PROD
2 (blade2) BX924S1
Pool / Group: pool1 / PROD
3 (blade3) BX924S1
Pool / Group: pool1 / PROD
4 (blade4) BX924S1
Pool / Group: pool2 / DEV
5 (blade5) BX924S1
Pool / Group: pool2 / DEV
For each cabinet the ID, the cabinet name, the management host name and IP address
and the server blades are displayed.
Each server blade is shown with its slot ID and name, the system type and the pool and
group it belongs to.
7.6.2 Adding Blade Server Cabinets
This section describes how to provide the required information for adding a new blade
server cabinet to an existing FlexFrame environment. You have to define some
parameters at the command line. They are used to configure switch ports and to create
the switch blade configurations.
Synopsis
ff_bx_cabinet_adm.pl --op add --type <system type>
--name <cabinet name>
--swgroup <switch group id>
[--swblades <switch blade type>]
[--swblogin <login name>]
[--swbpwd <password>]
[--host <ip host parts>] [--10gbit]
[--mgmtswgroup <switch group id>]
[--uplinkportcnt <nic count>]
[--port <sw:port>[,...]]
Options
--op add
Adds a blade server cabinet.
Application Nodes Administration Administrating Blade Server Cabinets
100 Administration and Operation
--type <system type>
PRIMERGY blade system type e.g. BX600S3, BX900S1.
Call ff_bx_cabinet_adm.pl without any parameter to get a list of supported
system types.
--name <cabinet name>
Name of the subsystem (cabinet). It is used to generate a name for the management
blade and has to be unique within FlexFrame. It may consist of 2 up to 13 characters
(letters, numbers, dashes). It must start with a letter and must not end with a dash.
--swgroup <switch group id>
Switch group number the cabinet has to be connected to (physically). Call
ff_bx_cabinet_adm.pl without any parameter to get a list of currently configured
switch group IDs.
--mgmtswgroup <switch group id>
Defines the switch group the cabinets management blade interfaces are connected
to. If omitted the effective switch group is computed as follows: First the switch group
given with --swgroup is considered, second the switch group used for the swgroup
out of band management is considered. Call ff_bx_cabinet_adm.pl --help
to get a list of currently configured switch group IDs
--swblades <switch blade type>
The type of switch blades. For valid types see the usage. For the default switch
blades this option may be omitted.
--swblogin <login_name>
Name used to login into the switch blades. If this option is omitted, the login name of
the switch group is used.
--swbpwd <password>
Password used to login into the switch blades. If this option is omitted, the login
password of the switch group is used.
--host <ip host parts>
Host parts to be used to build IP addresses for the control LAN networks. If this
option is omitted the script uses free host numbers to calculate the IP addresses.
Order of comma separated host numbers: first host number for both management
blades and then one for each switch blade.
--10gbit
Use 10 Gigabit switch blade ports as uplink. The specification can be omitted if only
10 Gigabit switch blade ports are available
Administrating Blade Server Cabinets Application Nodes Administration
Administration and Operation 101
--uplinkportcnt <nic count>
Set the NIC count of switch blades uplink channel to given value. Defaults to 2.
--port <sw:port>[,...]
Specifies the ports that are to be used on the connected switch group(s).
‘sw’ is the switch id within the switch group. ‘port’ is the port id on that switch.
Multiple ports are specified by a comma separated list. The first and second
specification are for the management blades, followed by specifications for the first
switch blades. Specifications for the second switch blade constitute the rest of the
list. If the list contains an empty specification, the port is calculated as if the --port
option was omitted (e.g. --port ,,1:13,2:13,1:14,2:14 to specify ports only
for the switch blades).
Output example
At the end of the output, the command displays further instructions.
Configure the ManagementBlade with the control lan settings:
control lan IP address: 195.40.224.75
control lan name: cab1-co
to interoperate correctly with the FlexFrame Agents.
Interconnect the ManagementBlades and SwitchBlades with the switches noted below:
SwitchGroup SwitchID Port Mgmt/SwitchBlade
1 1 8 ManagementBlade 2
1 1 11 SwitchBlade 1 Port 12
1 1 12 SwitchBlade 2 Port 12
1 2 8 ManagementBlade 1
1 2 11 SwitchBlade 1 Port 11
1 2 12 SwitchBlade 2 Port 11
Uploads of initial SwitchBlade configurations have to be done manually.
See the document(s)
'Quick Start Hardware FlexFrame for SAP – PRIMERGY SWB 1GbE-10/6-Q'
in the doc/hwinfo section of the Service CD for details.
The files to be uploaded are named:
SwitchBlade Blade Type File Path
1 1GbE-10/6-Q /tftpboot/swblade-2-1.config
2 1GbE-10/6-Q /tftpboot/swblade-2-2.config
Look at "/opt/FlexFrame/network/wiring-BX600-cab1.txt" to get a copy of this
message.
Set up the management blade initially with name and IP address listed by the output as
seen above. Use console redirection of management blade to connect to the console of
Application Nodes Administration Administrating Blade Server Cabinets
102 Administration and Operation
the switch blades, and upload configuration as described by the FlexFrame manual
“Installation of a FlexFrame Environment”.
Finally, plug in the network cables according to the wiring plan given by the command
output.
7.6.3 Removing Blade Server Cabinets
You only have to specify the ID or the name of the cabinet that is to be removed. All
FlexFrame integrated LAN switch ports will be unconfigured. Removing a blade server
cabinet requires removing of all of its server blades.
Synopsis
ff_bx_cabinet_adm.pl --op rem --id <cabinet id> | --name <cabinet
name>
Options
--op rem
Removes a blade server cabinet.
--id <cabinet id>
Specifies the cabinet ID of the cabinet to be removed. Use the
list-all option to get the ID (see section 7.6.3).
--name <cabinet name>
Specifies the name of the cabinet.
Output examples
If there are any server blades configured for this cabinet, an error message is displayed.
ERROR: there are server blades configured for this cabinet.
To remove the cabinet, remove application nodes (server blades) first.
Use command ff_an_adm.pl to do this.
Use the list operation mode to list the configured server blades. You have to remove
them before you can remove the cabinet they are in.
If no server blades are configured for this cabinet, the command displays a summary at
the end.
If not reported any warnings or errors the cabinet was removed
from LDAP and integrated LAN switches.
Administrating Blade Server Cabinets Application Nodes Administration
Administration and Operation 103
The cabinet has been removed successfully from LDAP and the FlexFrame integrated
LAN switch ports used by the cabinet have been reconfigured to default.
7.6.4 Changing Switch Blade Type
Within service cases it may be necessary to change the type of a switch blade due to a
defective part replacement. Only switching blades can be used for a type change.
To change the switch blade type the cabinet, the switch blade id and the switch blade
type have to be specified.
Synopsis
ff_bx_cabinet_adm.pl --op swb-change --id <cabinet id>
--swbid <switch blade id>
--swbtype <switch blade type>
Options
--op swb-change
Selects the operation mode. Change the type of a switch blade.
--id <cabinet id>
Specifies the cabinet ID of the cabinet. Use the list-all option to get the ID.
--swbid <switch blade id>
Specifies the switch blade ID. The ID is the slot number of the selected switch blade.
--swbtype <switch blade type>
Defines the new type of the switch blade. See usage for the currently supported
types.
Output example
Switch type of switch blade 2 was successfully changed from "1GbE-10/6-Q" to "1GbE-
10/2+4-C" at LDAP database.
The switch blade type was changed at LDAP database. To get the initial configuration use operation mode swb-config of this program. It will display instructions how to
upload the configurations too.
7.6.5 Changing Switch Blade Name
On adding a new cabinet, the name of the switch blade is like the cabinet name with a
slot extension. In some cases the names of the switch blades have to be changed to
match naming conventions.
Application Nodes Administration Administrating Blade Server Cabinets
104 Administration and Operation
Synopsis
ff_bx_cabinet_adm.pl --op swb-name --id <cabinet id>
--swbid <switch blade id>
--swbname <switch blade name>
Options
--op swb-name
Selects the operation mode. Change the name of a switch blade.
--id <cabinet id>
Specifies the cabinet ID of the cabinet. Use the list-all option to get the ID.
--swbid <switch blade id>
Specifies the ID of the switch blade. The ID is the slot number of the selected switch
blade.
--swbname <switch blade name>
Defines the new name of the switch blade.
Output example
If not reported any warnings or errors the hostname was successfully changed at
switch blade, hosts files and LDAP.
As noted by the program the name of the switch will be changed at /etc/hosts of both
control nodes, the LDAP database and at least the hostname and, if possible, the SNMP
sysname at the selected switch blade itself.
7.6.6 Changing Switch Blade Password
If on adding a cabinet the switch blade password could not be derived from the switch
group and was not specified in the command, a default password is assigned. This is not
secure.
Synopsis
ff_bx_cabinet_adm.pl --op swb-passwd --id <cabinet id>
--swbid <switch blade id>
--swbpwd <password>
Options
--op swb-passwd
Administrating Blade Server Cabinets Application Nodes Administration
Administration and Operation 105
Selects the operation mode. Change the login password of a switch blade.
--id <cabinet id>
Specifies the subsystem (cabinet) ID of the cabinet. Use the list-all option to get the
ID.
--swbid <switch blade id>
Specifies the switch blade ID. The ID is the slot number of the selected switch blade.
--swbpwd <password>
Defines the new login and enable password of the switch blade.
Output example
If not reported any warnings or errors the password was successfully changed at
switch blade and LDAP.
As noted by the program the password will be changed at the LDAP database and the
selected switch blade. At the switch blade the login password and the enable password
are changed and have to be the same.
7.6.7 Getting Switch Blade Configuration
In case of a service issue it may be necessary to get a switch blade configuration, which
has to be uploaded manually. Use this command to get the actual necessary
configuration according to LDAP for a specific switch blade.
Customizations, if necessary, have to be reapplied. Inspection of the project
information see 11.10 and the last saved configurations in /tftpboot/saved_switchconfigs may help.
Synopsis
ff_bx_cabinet_adm.pl --op swb-config --id <cabinet id>
--swbid <switch blade id>
Options
--op swb-config
Selects the operation mode. Create the switch blade configuration.
--id <cabinet id>
Specifies the cabinet ID of the cabinet. Use the list-all option to get the ID.
--swbid <switch blade id>
Application Nodes Administration Administrating Blade Server Cabinets
106 Administration and Operation
Specifies the ID of the switch blade. The ID is the slot number of the selected switch
blade.
Output example
jercn1:~ # ff_bx_cabinet_adm.pl --op swb-config --id 2 --swbid 1
Upload the generated SwitchBlade configurations noted below.
See the Quick Start Hardware FlexFrame document from the doc/hwinfo
section of the Service CD for the particular switch type and use the
following data:
swblade 1 cabinet 2 (hoctest-swb1) type 10GbE-18/8+2-F
IP and Mask: 172.20.13.42 255.255.255.0
Config: /tftpboot/hoctest-swb1.config
The configuration file is stored directly to /tftpboot. Upload of configuration is
described using TFTP which uses /tftpboot as top level directory. A detailed
instruction can be found in the FlexFrame manual „Installation of a FlexFrame
Environment“.
7.6.8 Change Switch Blade Uplink
More than the two default ports can be used for the uplink to switch group. To change the
count of ports used by the uplink link aggregates, including switch port configuration, use
the operation mode swb-uplink of ff_bx_cabinet_adm.pl. On switch group and
switch blades the appropriate link aggregates will be expanded by new ports until count
of given ports is reached.
Synopsis
ff_bx_cabinet_adm.pl --op swb-uplink --id <cabinet id>
--uplinkportcnt <nic count>
Options
--op swb-uplink
Change the count of NICs of uplink link aggregate for each switch blade of cabinet.
--id <cabinet id>
Defines the cabinet to change.
--uplinkportcnt <nic count>
Set the NIC count of switch blades uplink channel to given value. Defaults to 2
Administrating Blade Server Cabinets Application Nodes Administration
Administration and Operation 107
Example
cn1:~ # ff_bx_cabinet_adm.pl --op swb-uplink --id 1 –-uplinkportcnt 6
update LDAP
...
update swblade 1/1 configuration
Notice: Update will take about 1 minute.
..........
update swblade 1/2 configuration
Notice: Update will take about 1 minute.
..........
Interconnect additional NICs between SwitchBlades and the switches of
SwitchGroup 1 as noted below:
SwitchBladeID/Port SwitchID/Port
1 / 13 <==> 1 / 9
1 / 14 <==> 2 / 9
1 / 15 <==> 1 / 11
1 / 16 <==> 2 / 11
2 / 13 <==> 1 / 14
2 / 14 <==> 2 / 14
2 / 15 <==> 1 / 16
2 / 16 <==> 2 / 15
Unless any errors are reported follow instructions above to extend
switch blade channels to switch group.
Look at "/opt/FlexFrame/network/wiring-channel-extend-BX600-1.txt"
to get a copy of this message.
7.6.9 Move a Blade Cabinet to Another Switch Group
On growing installations sometimes the load of a switch group is split onto two groups.
While doing this sometimes entire blade cabinets have to be moved to another switch
group. To simplify this action use the operation mode move of
ff_bx_cabinet_adm.pl.
Synopsis
ff_bx_cabinet_adm.pl --op move --id <cabinet id>
--swgroup <switch group id>
[--mgmtswgroup <switch group id>]
[--port <sw:port>[,...]]
Application Nodes Administration Administrating Blade Server Cabinets
108 Administration and Operation
Options
--op move
Change the switch group of a cabinet, including switch blade but not server blade
connections.
--id <cabinet id>
Defines the cabinet to be moved.
--swgroup <switch group id>
The switch group id the cabinet should be moved to.
--mgmtswgroup <switch group id>
Defines the switch group the cabinet’s management blade interfaces should be
moved to. If omitted the effective switch group is computed as follows: First the
switch group given with --swgroup is considered, second the switch group used for
the swgroup out of band management is considered.
--port <sw:port>[,...]
Specifies the ports that are to be used on the connected switch group(s).
'sw' is the switch id within the switch group. 'port' is the port id on that switch.
Multiple ports are specified by a comma separated list. The first and second
specification are for the management blades, followed by specifications for the first
switch blades. Specifications for the second switch blade constitute the rest of the
list. If the list contains an empty specification, the port is calculated as if the -port
option was omitted (e.g. -port ,,1:13,2:13,1:14,2:14 to specify ports only for the
switch blades).
Example
cn1:~ # ff_bx_cabinet_adm.pl --op move --id 1 –-swgroup 2
update LDAP
...
update swblade 1/1 configuration
Notice: Update will take about 1 minute.
..........
update swblade 1/2 configuration
Notice: Update will take about 1 minute.
..........
Interconnect NICs between MgmtBlades,SwitchBlades and the switches of
the new SwitchGroup 2 as noted below:
SwGrpID/SwitchID/Port Mgmt/SwitchBlade
2 / 1 / 1 <==> master ManagementBlade
2 / 1 / 2 <==> SwitchBlade 1 Port 11
Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration
Administration and Operation 109
2 / 1 / 3 <==> SwitchBlade 2 Port 11
2 / 2 / 1 <==> slave ManagementBlade
2 / 2 / 2 <==> SwitchBlade 1 Port 12
2 / 2 / 3 <==> SwitchBlade 2 Port 12
Unless any errors are reported follow instructions above to move
cabinet to new switch group.
Look at "/opt/FlexFrame/network/wiring-cabinet-move-BX600-1.txt"
to get a copy of this message.
7.7 Administrating Hypervisor Nodes and Virtual Machines
Instead of being used directly as Application Nodes, PRIMERGY servers may also be
used in FlexFrame as Hypervisor Nodes. A Hypervisor Node can host a number of virtual
machines that are used as Application Nodes.
VMware ESXi and VMware ESX are "bare-metal" hypervisors that form the foundation of
VMware vSphere. The VMware vSphere product line supports three different types of
hypervisors - ESX classic, ESXi installable and ESXi embedded. In FlexFrame only ESXi
installable and ESXi embedded are supported. However, the terms "ESX" and "ESXi" are
both used in the FlexFrame documentation and code to denote the VMware hypervisor
and always mean "ESXi" unless explicitly stated otherwise.
With FlexFrame Orchestrator, SUSE Linux KVM is also supported as an additional
hypervisor type.
Hypervisor Nodes of type KVM can also host the virtual Control Center, i.e. the Control
Nodes run in virtual machines. A Hypervisor Node hosting a virtual Control Node may
additionally host virtual machines used as Application Nodes. Hypervisor Nodes and
virtual machines used as Control Nodes are predefined with the FlexFrame Management
Tool and put into operation during the FlexFrame installation process.
Hypervisor Nodes and virtual machines used as Application Nodes can be predefined
with the FlexFrame Management Tool and put into operation during the FlexFrame
installation process, or added to a FlexFrame configuration later with FlexFrame
administrative tools as described in section 7.7.1 below. The FlexFrame administrative
tools also offer functions to display or adjust the configuration as shown in detail in the
following sections.
Besides the FlexFrame tools which focus mainly on the FlexFrame specific aspects of
ESXi servers and virtual machines, there are several ways to access virtualization
components, such as the vSphere Client or the vSphere Command-Line Interface (vCLI)
for vSphere based components or the Virtual Machine Manager (virt-manager) and
the virsh command line for KVM. These tools can be used in addition to the FlexFrame
Application Nodes AdministrationAdministrating Hypervisor Nodes and Virtual Machines
110 Administration and Operation
tools, but certain actions must be avoided to insure proper operation of the FlexFrame
system. A short overview of this topic is given in section 7.7.10.
7.7.1 Getting Started with Hypervisor Nodes and VMs
This section gives an overview of the steps needed to add a Hypervisor Node (HN) and
virtual machines used as Application Nodes on the HN to a FlexFrame system when
using FlexFrame administrative tools.
1. Add the HN to the FlexFrame configuration.
cn1:~ # ff_hn_adm.pl --op add --name <node name> --type <system type> ...
Depending on system type, more options are needed when entering this
command.
See section 7.7.3 for details.
2. Install/Boot the HN.
The necessary actions depend on the used hypervisor type. For embedded ESXi,
the server is already installed on an internal USB device and the only action is to
boot it from this device. For ESXi installable, install the ESXi server from its CD or
DVD (an ISO-Image of this software is also contained on the FlexFrame Service
DVD) and then boot it from local hard disk. For SUSE Linux KVM, install SUSE Linux
Enterprise Server (SLES) with KVM from the FlexFrame Control Node DVD. Refer to
the "Install and Boot Hypervisor Node" section of the manual "Installation of a
FlexFrame Environment" for details and to the section "Setup the <server type> for a
Hypervisor Node installation" in the appropriate "FlexFrame® HW Characteristics
Quickguide ("Hardwaresteckbrief")".
3. Do a minimal configuration of the Hypervisor Node using its server console. This
includes setting a password and selecting the network adapters for the network
connection of the Hypervisor Node. Refer to the "Hypervisor Node Preconfiguration
on the Server Console" section of the manual "Installation of a FlexFrame
Environment" for details.
4. Complete the Hypervisor Node configuration.
cn1:~ # ff_hn_adm.pl --op complete-config --name <node name>
--user <user name> --pass <password>
See section 7.7.4 below for details.
5. No explicit creation of virtual machines is needed.
Instead of that, simply use the well-known ff_an_adm.pl --op add command
with a system type of ESXVM or KVMVM to specify that the "hardware" of the new
Application Node is a virtual machine on an ESXi host or a KVM host. In addition to
its usual functions such as reserving IP addresses or creating LDAP data, the script
Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration
Administration and Operation 111
then creates a virtual machine with the same name as the Application Node on the
Hypervisor Node specified on the command line. The memory size and number of
CPUs for the virtual machine are set to default values, but can also be specified on
the command line (options --vmem and --vcpus). To select appropriate values for
these parameters, the available resources of the Hypervisor Node must be taken
into account, as well as the intended use of the Application Node. See section 7.7.9
”Virtual Machine Properties and Hypervisor Resources” on page 125 for details on
this topic.
cn1:~ # ff_an_adm.pl --op add --name <node name> --type ESXVM
--pool <pool name> --group <group name>
--ospath <path to os image> --hn <node name> ...
or
cn1:~ # ff_an_adm.pl --op add --name <node name> --type KVMVM
--pool <pool name> --group <group name>
--ospath <path to os image> --hn <node name> ...
Proceed as you would do it for any other Application Node.
Create a new personalized boot image for the Application Node (if not implicitly done using the --new-image operand of ff_an_adm.pl):
cn1:~ # ff_new_an.sh -n <node name>
Then boot the node by calling:
cn1:~ # ff_ponpoff.pl --op power-on --name <node name>
7.7.2 Virtualization related global FlexFrame parameters
Besides the Hypervisor Nodes and virtual machines, some FlexFrame global parameters
must be taken into account when planning a FlexFrame system using server
Virtualization functionality. They are usually set with the FlexFrame Management Tool,
but can be also set with administrative tools in some special cases.
7.7.2.1 System Code for Hypervisor Nodes and VMs
The FlexFrame system code for Hypervisor Nodes and virtual machines is a numeric
value between 0 and 63 that is used to generate MAC addresses for FlexFrame virtual
machines and to build names of some Hypervisor Node related resources such as Port
Groups and Datastores (VMware terminology) or Networks and Storage Pools
(KVM/libvirt terminology). Its purpose is to differentiate these objects belonging to
different FlexFrame systems used in a common environment, for example if using a
common vCenter Server or a not completely separated ethernet network.
Application Nodes AdministrationAdministrating Hypervisor Nodes and Virtual Machines
112 Administration and Operation
The FlexFrame system code is usually set with the FlexFrame Management Tool and
must not be changed for a system with already configured Hypervisor Nodes and virtual
machines.
The ability to set the System Code using the tool ff_hn_adm.pl is mainly provided for
cases when the FlexFrame system has been upgraded from an older version without
server virtualization support or if the initial FlexFrame setup has been done without server
virtualization usage.
Use a different system code for each FlexFrame system in your environment.
Do not change the system code for an existing FlexFrame system with
configured Hypervisor Nodes and virtual machines.
Synopsis
ff_hn_adm.pl --op set-syscode --syscode <system code>
Options
--op set-syscode
Sets the FlexFrame system code for Hypervisor Nodes and virtual machines.
--syscode <system code>
Numeric value between 0 and 63 to be used as FlexFrame system code
7.7.2.2 vCenter Server
More complex administrative actions and functionalities, as well as the centralized view
and monitoring of the VMware Infrastructure are outside the scope of FlexFrame. For
these functions, a vCenter Server can be used.
If FlexFrame ESX servers are administrated by a vCenter Server, the name and IP
address of the vCenter Server have to be made known to FlexFrame. This is usually
done with the FlexFrame Management Tool, but can also be done using the tool ff_hn_adm.pl, for example in cases when the FlexFrame system has been upgraded
from an older version without VMware support or if the initial FlexFrame setup has been
done without VMware usage. You can also remove the previously set vCenter Server
information, e.g. if the information set with the FlexFrame Management Tool is not
correct, and you can set the authentication data for the vCenter Server in FlexFrame, so
that the FlexFrame tools can access this server.
Synopsis
ff_hn_adm.pl --op vcenter-add --vc-name <vcenter name>
[--vc-ip <vcenter ip>] [--vc-in-cntllan]
Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration
Administration and Operation 113
ff_hn_adm.pl --op vcenter-rem
ff_hn_adm.pl --op vcenter-auth --user <user name>
--pass <password>
Options
--op vcenter-add
Add vCenter Server hostname and IP address to FlexFrame.
--op vcenter-rem
Remove vCenter Server information from FlexFrame.
--op vcenter-auth
Set authentication data for vCenter Server in FlexFrame.
--vc-name <vcenter name>
Defines the name of the vCenter Server with --op vcenter-add.
--vc-ip <vcenter-ip>
Defines the IP address of the vCenter Server with --op vcenter-add. It can be
an address in the FlexFrame Control LAN or an address outside FlexFrame, that is
reachable from the Control Nodes.
If the --vc-ip option is omitted, and --vc-in-cntllan is specified, a free
address from Control LAN is selected.
--vc-in-cntllan
Specifies that the vCenter IP address is in the Control LAN. If no address is given,
advises the script to select a free address from Control LAN.
--user <user name>
Specifies the user name for accessing the vCenter Server with --op vcenter-
auth.
--pass <password>
Specifies the password for accessing the vCenter Server.
If using vCenter Server in FlexFrame, the names and IP addresses of the ESX servers
must be resolvable on a DNS server that is accessible for the vCenter Server.
The FlexFrame ESX servers must be added manually to vCenter Server using their
hostnames as they are known in FlexFrame. Add the ESX servers to a Datacenter called
FlexFrame or, if more than one FlexFrame system is administrated with the same
Application Nodes AdministrationAdministrating Hypervisor Nodes and Virtual Machines
114 Administration and Operation
vCenter Server, use a distinct Datacenter called FlexFrame-<i> for each FlexFrame
system, where i is the FlexFrame system code explained in section 7.7.2.1.
7.7.2.3 Control LAN Default Router for Hypervisor Nodes
If FlexFrame ESX servers are administrated by a vCenter Server, it may also be
necessary to use a Control LAN router that is set on the Hypervisor Nodes as default router. This can be done using the tool ff_hn_adm.pl with operation mode
defrouter.
With this operation mode, the address of a default router for the Control LAN can be set
or removed in the FlexFrame configuration, including the DHCP configuration. The
Hypervisor Nodes get this address via DHCP together with the IP address for HN
management.
Synopsis
ff_hn_adm.pl --op defrouter --defrouter <default router ip | 0>
Options
--op defrouter
Add or remove the address of the Control LAN default router to the FlexFrame
configuration
--defrouter <default router ip | 0>
Defines the IP address of the Control LAN default router. Specify a Control LAN
address to add a default router for the Hypervisor Nodes or to change the current
setting, or use the value 0 to remove it.
7.7.3 Adding Hypervisor Nodes
This section describes how to add a Hypervisor Node to an existing FlexFrame
environment.
Adding a Hypervisor Node to a FlexFrame environment may require a specific
FlexFrame license. For details on FlexFrame licenses refer to chapter 3.7
“Product Licenses”.
Synopsis
ff_hn_adm.pl --op add --type <system type> --name <node name>
--hntype <hypervisor node type>
--swgroup <switch group_ id> --mac <mac addresses>
Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration
Administration and Operation 115
[--host <ip host number>[,<IPMI ip host number>
[,<2nd IPMI ip host number>]]]
[--slot <cabinet_id/slot>] [--10gbit]
[--mgmtswgroup <switch group id>]
[--port switch:port,switch:port,switch:port
[,switch:port]]
Options
--op add
Adds a Hypervisor Node and displays some information about processing steps.
--hntype <hypervisor node type>
Specifies the hypervisor node type like ESX or KVM. Call ff_hn_adm.pl --help to
get a list of supported types
--type <system type>
Specifies the product name and type. Call ff_hn_adm.pl --help to get a list of
supported system types
--name <node name>
The name of the Hypervisor Node. This name has to be unique for the entire
FlexFrame system. All interface names are based on this node name. The name has
to be lowercase and consists of 2 up to 13 characters (letters, digits and dashes). It
must start with a letter and must not end with a dash.
--swgroup <switch group id>
Defines the switch group the Hypervisor Node is connected to. This information is
necessary to assign and configure switch ports. Call ff_hn_adm.pl --help to get
a list of currently configured switch group IDs.
For blade servers, the --swgroup option may be omitted, as the switch group is
already defined with the blade cabinet.
--mac <mac addresses>
Add here the MAC addresses of the server's data NICs.They are used to configure
DHCP. Use the colon separated hex notation for each MAC address. Concatenate
the two MAC addresses with a comma. A MAC address syntax example is
00:e0:00:c5:19:41.
For blade servers, the --mac option may be omitted. The MAC addresses are then
fetched via SNMP from the management blade.
--host <ip host number>[,<IPMI ip host number>
[,2nd IPMI ip host number>]]
Application Nodes AdministrationAdministrating Hypervisor Nodes and Virtual Machines
116 Administration and Operation
Host part to be used to build the IP addresses for hypervisor management and,
depending on system type, for IPMI, all of them in the Control LAN.
If the --host option is omitted, the script uses free host numbers to calculate the IP
addresses.
--slot <cabinet_id/slot>
With PRIMERGY server blades use this option to define the cabinet and slot ID of
the server blade. New cabinets have to be defined with the
ff_bx_cabinet_adm.pl command.
--mgmtswgroup <switch group id>
Defines the switch group the server management interface (IPMI) is connected to. If
omitted the effective switch group is computed as follows: First the switch group given with --swgroup is considered, second the switch group used for the swgroup
out of band management is considered. Call ff_hn_adm.pl --help to get a list of
currently configured switch group IDs.
--10gbit
Specifies that the node is used with 10 Gigabit data NICs. The specification can be
omitted if the nodes system_type only allows usage with 10 Gigabit data NICs.
--port switch:port,switch:port,switch:port,[switch:port]
Defines the switch ports to be used. First two tuples are for data NIC 1 and 2 and allocated in the switch group defined with --swgroup. The following tuples are for
mgmt NICs and allocated in the effective switch group for management interfaces. If
--10gbit is specified, the first two ports must be 10 Gigabit capable ports
If --port is not specified, the script assigns free ports according to an internal
algorithm.
Example
cn1:~ # ff_hn_adm.pl --op add --name rx300s5-esx1 –hntype ESX
--type RX300S5 --swgroup 2 --mac 00:19:99:48:e2:aa,00:19:99:48:e2:ab
update LDAP
.........
update switch 2/1 configuration
Notice: Update will take about 1 minute.
restart cluster service dhcpd
stopping dhcpd (timeout=20) . dhcpd:done . done.
starting dhcpd (timeout=20) .. dhcpd:done . done.
Connect your systems LAN interfaces to named switch ports:
LAN Interface SwGroup / Switch / Port
data NIC-1 2 / 2 / 2
data NIC-2 2 / 1 / 2
IPMI NIC-1 2 / 2 / 24
Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration
Administration and Operation 117
The script first checks all arguments and aborts with an error message in case of errors. It
then fetches free IP addresses and switch ports or checks whether the given ones can be
used for the node. LDAP data are created and the switch ports are reconfigured to match
the requirements. The DHCP service on the control nodes is reconfigured to assign the
hypervisor management IP address in the Control LAN based on the given MAC
addresses.
At the end, you get a cabling advice if the node must be connected to a switch group.
How to continue
The next step is to do the cabling according to the advice if applicable.
Then proceed with installing/booting and preconfiguring the Hypervisor Node using the
server console (see section "Getting Started with Hypervisor Nodes and VMs" on page
110 and the sections "Install and boot Hypervisor Node" and "Hypervisor Node
Preconfiguration on the Server Console" of the manual "Installation of a FlexFrame
Environment").
Finally, complete the Hypervisor Node configuration as shown in the next section.
7.7.4 Completing Hypervisor Node Configuration
This section describes how to complete the Hypervisor Node configuration.
Synopsis
ff_hn_adm.pl --op complete-config [--name <node name>]
--user <user name> --pass <password>
Options
--op complete-config
Completes the Hypervisor Node configuration and makes provisions so that
FlexFrame scripts can access the server without user interaction. In case of an ESX
server, the authentication data are stored in FlexFrame for subsequent access of the
server.
--name <node name>
Name of the Hypervisor Node. If the name is omitted, the operation is done for all
defined Hypervisor Nodes. This possibility is intended for usage during the
installation process.
--user <user name>
Specifies the user name for accessing the Hypervisor Node.
--pass <password>
Specifies the password for accessing the Hypervisor Node.
Application Nodes AdministrationAdministrating Hypervisor Nodes and Virtual Machines
118 Administration and Operation
Example
cn1:~ # ff_hn_adm.pl --op complete-config --name rx300s5-esx1 --user
root
--pass password
The script accesses the ESXi server using the given authentication data and completes
the ESXi server configuration to match the FlexFrame needs, e.g. the FlexFrame Control
Nodes are set as NTP servers and as trap targets for SNMP. If it is already known for
which pools the ESXi server must be prepared, the corresponding Port Groups and
Datastores are created on the ESXi server (see section "Hypervisor Nodes and Pools" on
page 122 for details).
Finally, the script stores the authentication data needed to access the ESXi server to
enable the subsequent access of the server by FlexFrame scripts.
7.7.5 Removing Hypervisor Nodes
To remove a Hypervisor Node from the FlexFrame configuration, the only needed
parameter is the node name. The node is removed from the LDAP database, the hosts
file and the DHCP configuration. The switch ports used by this node are also
unconfigured. A Hypervisor Node cannot be removed if there are still virtual machine
Application Nodes that use this Hypervisor Node in the FlexFrame configuration or if the
Hypervisor Node hosts a Control Node.
Please make sure that you really want to remove the Hypervisor Node, as the
script does not ask for confirmation.
Synopsis
ff_hn_adm.pl --op rem --name <node name>
Options
--op rem
Removes a Hypervisor Node from the FlexFrame configuration.
--name <node_name>
Name of the Hypervisor Node.
Example
cn1:~ # ff_hn_adm.pl --op rem --name rx300s8-kvm1
Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration
Administration and Operation 119
7.7.6 Displaying Information about HNs and VMs
Using the script ff_hn_adm.pl, one can get an overview about all Hypervisor Nodes,
or detailed information about a specific Hypervisor Node including its virtual machines.
With ff_vm.pl, it is possible to get an overview of all FlexFrame virtual machines,
irrespectively of the involved Hypervisor Node.
Synopsis
ff_hn_adm.pl --op list-all --hntype <hypervisor node type>
ff_hn_adm.pl --op list --name <node name> [--cmdline]
Options
--op list-all
Displays an overview of all Hypervisor Nodes and also shows virtualization related
global FlexFrame parameters, such as the vCenter Server name and address or the
FlexFrame system code.
--hntype <hypervisor node type>
Specifies the hypervisor node type together with --op list-all in order to get
all Hypervisor Nodes of this type listed.
--op list
Displays detailed information about a specific Hypervisor Node, including its virtual
machines.
--name <node name>
Name of the Hypervisor Node.
--cmdline
Used with --op list, the command line that can be used to recreate the
Hypervisor Node is displayed at the end of the node listing.
Synopsis
ff_vm.pl --op list-all
Options
--op list-all
Displays an overview of all FlexFrame virtual machines, irrespective of the hosting
Hypervisor Node.
Application Nodes AdministrationAdministrating Hypervisor Nodes and Virtual Machines
120 Administration and Operation
Examples
Overview of all Hypervisor Nodes and global information:
cn1:~ # ff_hn_adm.pl --op list-all
Global information
vCenter Server Hostname: vcenter-vm
vCenter Server IP: 172.50.6.68
FlexFrame Systemcode: 4
Control LAN Default Router: 172.50.6.251
Hypervisor Nodes sorted by name
bx92-01-esx
Hypervisor Type: ESX
System Type: BX924S4
Cabinet/Slot ID: 2/1
Mgmt IP/Hostname: 172.50.6.62 / bx92-01-esx
Mac Addr.: 00:1b:24:3d:a2:57 00:1b:24:3d:a2:58
bx92-02-kvm
Hypervisor Type: KVM
System Type: BX924S4
Cabinet/Slot ID: 2/2
Mgmt IP/Hostname: 172.50.6.63 / bx92-02-kvm
Mac Addr.: 00:1b:24:3d:a2:63 00:1b:24:3d:a2:64
Detailed information about a specific Hypervisor Node:
cn1:~ # ff_hn_adm.pl --op list --name rx3s7-kvm1
Configuration details of KVM node rx3s7-kvm1
Hardware
System: RX300S7
10GBit: No
Shut.Facil.: IPMI rx3s7-kvm1-co (10.10.13.27)
Mac Addr.: 00:19:99:48:e2:aa 00:19:99:48:e2:ab
Hypervisor management interface (Control LAN)
Host IP: 10.10.13.16
Hostname: rx3s7-kvm1
LAN Interface Connections
LAN Interface SwGroup / Switch / Port
data NIC-1 2 / 2 / 2
data NIC-2 2 / 1 / 2
Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration
Administration and Operation 121
IPMI NIC-1 2 / 2 / 24
Hypervisor resources
Memory Size: 32753 MB
CPU Cores: 8
Virtual Machines
List of FlexFrame Virtual Machines:
Name Type State CPUs Memory Pool Group
kvm-vm13 AN Off 2 2048 pool1 p1-vms
cn1 CN On 2 8192
-------------------------------------
Total 4 10240
Overview of all FlexFrame virtual machines:
cn1:~ # ff_vm.pl --op list-all
List of Virtual Machine - Application Nodes registered in LDAP
VM-AN Name Type Pool Group Hypervisor Node
esx-vm31 ESXVM pool3 p3-vms rx3s7-esx1
esx-vm32 ESXVM pool3 p3-vms rx3s7-esx2
kvm-vm13 KVMVM pool1 p1-vms rx3s7-kvm1
List of Virtual Machine - Control Nodes registered in LDAP
VM-CN Name Type Hypervisor Node
cn1 KVMVM rx3s7-kvm1
cn2 KVMVM rx3s7-kvm2
Collecting information from Hypervisor Nodes:
rx3s7-esx1 ... done
rx3s7-esx2 ... done
rx3s7-kvm1 ... done
rx3s7-kvm2 ... done
List of FlexFrame Virtual Machines found on available hypervisor nodes
VM Name Hypervisor Node State
cn1 rx3s7-kvm1 On
cn2 rx3s7-kvm2 On
esx-vm31 rx3s7-esx1 On
esx-vm32 rx3s7-esx2 On
kvm-vm13 rx3s7-kvm1 Off
Application Nodes AdministrationAdministrating Hypervisor Nodes and Virtual Machines
122 Administration and Operation
7.7.7 Hypervisor Nodes and Pools
A Hypervisor Node can host virtual machine Application Nodes from different pools. For
each pool the server needs some specific resources, such as a virtual machine Port
Group or a virtual network for each of the pool networks and two datastores or storage
pools (config and software) in the pool specific volFF volume of the pool.
Usually, these resources are implicitly prepared when the first virtual machine Application
Node of a pool is created on a Hypervisor Node, so that there is no need to do an explicit
pool preparation of the Hypervisor Node.
However, if a FlexFrame virtual machine is moved to another Hypervisor Node by
mechanisms outside FlexFrame (see also section 7.7.10), these resources must be
prepared in advance. This can be done by using the add-pool operation of the
ff_hn_adm.pl script.
If no longer needed, the prepared resources can be frred by using the rem-pool
operation of the ff_hn_adm.pl script.
Synopsis
ff_hn_adm.pl --op add-pool --name <node name> --pool <pool name>
ff_hn_adm.pl --op rem-pool --name <node name> --pool <pool name>
Options
--op add-pool
Prepares a Hypervisor Node for usage by virtual machine Application Nodes from
given pool.
--op rem-pool
Removes pool specific resources from a Hypervisor Node.
--name <node name>
Name of the Hypervisor Node.
--pool <pool name>
Name of the pool.
Example
cn1:~ # ff_hn_adm.pl --op add-pool --name rx3s7-kvm1 --pool pool3
Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration
Administration and Operation 123
7.7.8 Special Functions for Virtual Machines
A virtual machine designated for use as a FlexFrame Application Node is usually created when the Application Node is added to the FlexFrame configuration with ff_an_adm.pl
--op add. Likewise, the virtual machine is destroyed when the associated Application
Node is removed using ff_an_adm.pl --op rem.
Under special circumstances, it is necessary to create FlexFrame virtual machines for
Application Nodes already defined in LDAP, for example during the FlexFrame
installation process. For special purposes, it may also be helpful to destroy a FlexFrame
virtual machine while preserving its LDAP entry, or if the LDAP entry is missing, e.g. as
an effect of an Application Node removal that failed to destroy the virtual machine.
The script ff_vm.pl provides these functionalities, as well as some other special
functions acting on virtual machines. This script is not intended for usual administrative
interactions with virtual machine Application Nodes. Use ff_an_adm.pl to administrate
these Application Nodes too in almost the same manner as other Application Nodes.
Synopsis
ff_vm.pl --op create [--name <node name>]
[--hn <node name>]
[--vcpus <number of virtual cpus>]
[--vmem <virtual machine memory size]
[--force]
ff_vm.pl --op destroy --name <nodename> [--force]
ff_vm.pl --op move --name <node name>
--to-hn <node name>
ff_vm.pl --op refresh-ldap [--name <node name>]
ff_vm.pl --op list --name <node name>
ff_vm.pl --op list-all
Application Nodes AdministrationAdministrating Hypervisor Nodes and Virtual Machines
124 Administration and Operation
Options
--op create
Creates virtual machines for Application Nodes that are already defined in the
FlexFrame configuration database (LDAP). If called without option --name, a virtual
machine is created for each Application Node with system type ESXVM or KVMVM
found in LDAP, optionally restricted to the Application Nodes for which the Hypervisor Node specified with option --hn is preset in LDAP. Otherwise, only the
Application Node with the specified name is considered. The virtual machines get
the same names as the Application Nodes. The target Hypervisor Node, the number
of virtual CPUs and the memory size of the virtual machine are taken from LDAP by default, but can be overridden using the options --hn, --vcpus and --vmem.
--op destroy
Destroys a virtual machine that has been created for an Application Node while
preserving its LDAP entry or if the LDAP entry is missing. If the LDAP entry is missing, the operation is started only if the option --force is specified.
--op move
Moves a powered off virtual machine associated with an Application Node to another
Hypervisor Node.
--op refresh-ldap
Adjusts the LDAP information for a specific or for all FlexFrame virtual machine(s)
after changes done with mechanisms outside FlexFrame, e.g. when a virtual
machine associated with an Application Node has been moved to another
Hypervisor Node. For more information about using other mechanisms on
FlexFrame virtual machines see section 7.7.10 below.
--op list
Displays information about a specific FlexFrame virtual machine.
--op list-all
Displays information about all FlexFrame virtual machines. See also section 7.7.6.
--name <node name>
The name is used to identify a specific node. If specified, an Application Node with
this name and system type ESXVM or KVMVM must exist in LDAP, except for the
case when the --force option is specified with --op destroy.
With --op list and --op refresh-ldap, it is also possible to specify the name
of a Control Node with type KVMVM.
--hn <node name>
Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration
Administration and Operation 125
Specifies the node name of the Hypervisor Node where the virtual machine must be created. By default, the value from LDAP is used. When no <node name> is given,
the --hn option is used to select all VMs for creation for which the specified
Hypervisor Node is preset in LDAP.
--vcpus <number of virtual cpus>
Defines the number of virtual CPUs when creating a virtual machine. By default, the
value from LDAP is used. The argument must be a number between 1 and 500.The
really accepted numbers may depend on the underlying technology and licensing.
Moreover, the available resources of the Hypervisor Node must be taken into
account. See also section 7.7.9 below.
--vmem <virtual machine memory size>
Defines the memory size in MB when creating a virtual machine. By default, the
value from LDAP is used. The memory size must be a number between 256 and
261120 (=255GB). Moreover, the available resources of the Hypervisor Node must
be taken into account. See also section 7.7.9 below.
--force
Specifies that the memory usage of the Hypervisor Node may be overcommitted
when creating virtual machines. Default is to deny creation of virtual machines on a
host if the total vmem of the virtual machines on this host, including the new ones,
exceeds the memory size of the host. With operation mode destroy, --force
allows to destroy a FlexFrame virtual machine even when no corresponding LDAP
entry exists.
--to-hn <node name>
This option is used with --op move to specify the name of the target Hypervisor
Node.
7.7.9 Virtual Machine Properties and Hypervisor Resources
In a vSphere environment, ESX/ESXi hosts provide resources for virtual machines.
Similarly, KVM hosts provide resources for virtual machines in a SUSE Linux KVM based
environment. Resources include CPU, memory, power, storage and network resources.
In this section, we focus on CPU and memory resources, as well as on storage
requirements.
As a part of the virtual hardware properties, the memory size and the number of virtual
CPUs are defined when creating a virtual machine. In a FlexFrame environment, values
for these parameters are either predefined with the FlexFrame Management Tool for
each virtual machine Application Node and can be later overridden when creating the
virtual machine using the ff_vm.pl script, or are defined when creating the Application
Node including its virtual machine with ff_an_adm.pl. If using a virtual Control Center,
the values for the memory size and number of CPUs for the virtual machine Control
Application Nodes AdministrationAdministrating Hypervisor Nodes and Virtual Machines
126 Administration and Operation
Nodes are predefined with the FlexFrame Management Tool and cannot be overridden
later.
When choosing values for these parameters, a careful planning must be done that takes
into account which machines will be powered on on a given Hypervisor Node at the same
time and for what kind of SAP workload they are planned to be used.
VMware techniques allow to configure more memory for the virtual machines on an ESX
server than the available physical memory of the host. This feature is called memory
overcommit and basically assumes that a VM does not use all assigned memory and
therefore it can be shared with other VMs. Similar techniques also exist for KVM.
For SAP systems it is strongly recommended not to overcommit memory usage, because
SAP allocates memory permanently and does not release it again. Furthermore, a
minimum of 4GB should be used, and for Unicode systems, a minimum of 8GB.
You should also be aware that if the memory size defined for a virtual machine is not
sufficient for the workload, excessive Linux swapping in the virtual machine may occur
which has a negative impact on performance and leads to an increased I/O load on the
filer and network which may even disturb the correct operation of the system.
A virtual machine can be configured with a number of virtual CPUs that depends on the
virtualization platform and the hypervisor vendor’s licensing, but it cannot be powered on
if it has more CPUs than the number of logical processors of the host - that is the number
of physical processor cores if hyperthreading is disabled or two times the number of
physical processor cores if hyperthreading is enabled. Similarly as with memory,
virtualization techniques allow to overcommit processor usage. When a host runs multiple
virtual machines that require more than the available CPU resources, the host time-slices
the physical processors across all virtual machines so that each virtual machine runs as if
it has its specified number of virtual processors. The CPU virtualization adds varying
overhead with performance implications.
SAP has successfully run performance tests in vSphere virtual machines (utilizing all
available virtual CPUs to 100%) which overcommitted the host system up to 200%. The
performance degradation inside the virtual machines was linear reciprocal to the
overcommitment. You may exceed the 200% overcommitment, but keep in mind, that the
performance of virtual machines in such a scenario is not guaranteed. In case of
performance problems, SAP can demand you to shutdown or pause other running virtual
machines to check if the overcommitment caused problems.
When creating virtual machines using FlexFrame scripts, the parameters given for the memory size and the number of virtual CPUs (--vmem and --vcpus) are roughly
checked for plausible values and it is checked that the memory usage of the designated
Hypervisor Node is not overcommitted, but no further checks and resource evaluations
are carried out. This task is in the responsibility of the specialists that make the
configuration and sizing planning for the FlexFrame system and assumes a good
knowledge of the virtualization techniques and the requirements of the used SAP
systems.
Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration
Administration and Operation 127
A FlexFrame virtual machine associated with an Application Node also needs some
space on the pool specific volFF of the pool to which the Application Node belongs. The
most significant resources located here are the file representing the virtual hard disk of
the virtual machine, and for virtual machines on ESXi hosts, the file used by the VMkernel to swap out the virtual machine’s physical memory in certain shortage cases (vswp). The
size of the file representing the virtual hard disk is defined by the size of the virtual hard
disk, which is 36 GB for FlexFrame virtual machines. The size of the vswp file is
correlated to the memory size of the virtual machine. If there is not sufficient space
available, the creation of the virtual machine may fail or the virtual machine fails to start.
These space requirements must also be taken into account during the configuration and
sizing planning mentioned above.
7.7.10 Using Hypervisor Management Software
While the FlexFrame tools focus mainly on the FlexFrame specific aspects of Hypervisor
Nodes and virtual machines, there are other ways to access virtualization components,
such as the vSphere Client and the vSphere Command-Line Interface (vCLI) for vSphere
based components or the Virtual Machine Manager (virt-manager) and the virsh
command line for KVM. Moreover, for more complex administrative actions and
functionalities, as well as the centralized view and monitoring of the VMware
Infrastructure, a vCenter Server can be used.
These tools and functions are outside the scope of FlexFrame, and for information on
their usage please refer to the appropriate vendor documentation.
However, when using them for Hypervisor Nodes and virtual machines known in
FlexFrame, it must be taken care not to disturb the FlexFrame functionality. Some special
requirements are:
1. Virtual machines for FlexFrame usage must be created using FlexFrame tools
2. Do not rename FlexFrame virtual machines
3. Do not destroy FlexFrame virtual machines while the associated Application Node
still exists. Removing the Application Node usually also destroys the virtual machine.
4. Do not change the virtual machine properties and devices. As an exception, you may
change the number of CPUs and the memory size of a powered off virtual machine,
but take into account the resources of the involved Hypervisor Node. After such
changes, please call ff_vm.pl --op refresh-ldap to adjust the corresponding
LDAP settings, or use the script ff_ponpoff.pl to power on the virtual machine,
as this does an implicit refresh of the LDAP settings.
5. Do not move a FlexFrame virtual machine to a Hypervisor Node that is not part of the
same FlexFrame system. Virtual machines associated with Application Nodes may
be moved to other Hypervisor Nodes of the same FlexFrame system, but take care
of the hints listed below.
To move a powered off virtual machine associated with an Application Node to
Application Nodes Administration Power on / off / reboot of an Application Node
128 Administration and Operation
another Hypervisor Node of the FlexFrame system, you can use ff_vm.pl --op
move. Otherwise, you must also take care that the target Hypervisor Node is
prepared for the pool the virtual machine Application Node belongs to. After a move
done with non-FlexFrame tools, please call ff_vm.pl --op refresh-ldap to
adjust the corresponding LDAP setting as soon as possible.
If moving a powered on virtual machine (Live Migration, VMware vMotion), which
may be done as part of a project specific solution, also take into account the possible
higher network utilization and service unresponsiveness during the move. In case of
KVM Live Migration, also take care to make the move persistent and to “undefine” the virtual machine on the source host (e.g. with virsh commandline use the
additional options --persistent --undefinesource).
6. Do not suspend/resume FlexFrame virtual machines. This may result in a service
being executed twice when a service formerly running on the suspended virtual
machine has been moved to a spare node and then the other VM is resumed. For
virtual machines created by FlexFrame on ESXi 5.x hosts, a special attribute is set
which denies the suspend action on these virtual machines.
7.8 Power on / off / reboot of an Application Node
Manually powering on or off an Application Node would be very inconvienent. The script
/opt/FlexFrame/bin/ff_ponpoff.pl supports power on, power off and reboot of
Application Nodes using adequate controllers like iRMC, Management Blade or
Hypervisor. The script depends on credentials you configure when adding the Application
Node as described in the corresponding “FlexFrame® HW Characteristics Quickguide”.
Notice that if you power off an Application Node using ff_ponpoff.pl, the
FlexFrame Agents will see an outage and are reacting as configured if the
Application Node is under control of the FlexFrame Agents.
Synopsis
ff_ponpoff.pl --op power-on | power-off | reboot
--name <node name | @all>
Options
--op power-on power-on the node(s)
power-off power-off the node(s)
reboot reboot the node(s)
--name <node name> execute the above actions for a single node <node name>
@all execute the above actions for all nodes of a FlexFrame
NAS Systems Configuration Storage Systems Administration
Administration and Operation 129
8 Storage Systems Administration
8.1 NAS Systems Configuration
8.1.1 Adding a New NAS
Use ff_nas_adm.pl --op add to add a NAS to the FlexFrame environment.
Afterwards configure the NAS manually according the appropriate "FlexFrame® HW
Characteristics Quickguide ("Hardwaresteckbrief")" using the data displayed.
Synopsis
ff_nas_adm.pl --op add --name <node name> --type <system type>
[--swgroup <switch group id>]
[--host <ip host part>]
[--ports <port count>]
[--interface <interface name>]
[--rootvolume <path to root volume>]
[--parent <node name>]
[--ostype <ONTAP|cDOT>]
[--10gbit]
Options
--op add
Adds a NAS and displays some information about processing steps.
--name <node name>
Defines the node name of the NAS. The name may consist of 2 upt to 13 characters
(letters, digits and dashes). It must start with a letter and must not end with a dash.
--type <systemtype>
Defines the type of the NAS. See usage for a list of known NAS types.
--swgroup <switch group id>
Defines the switch group the NAS should be added to. See usage for a list of
configured switch group IDs.
--host <ip host part>
Defines the host part to be used to build IP addresses for the Control or Storage
LAN networks. If this option is omitted the script uses a free host number to calculate
the IP address.--ports <port count>
Storage Systems Administration NAS Systems Configuration
130 Administration and Operation
Defines the count of ports to be used with pool Storage LAN networks. If this option
is omitted the script uses the default of two ports. The maximum number of ports per
channel is 8 (dependant on hardware capabilities)..
--interface <interface name>
Defines the interface to be used for filer access. Default: storage
--rootvolume <path to root volume>
Defines the root volume of the filer. Default: /vol/vol0
--parent <node name>
Defines the parent NAS node. The parent of a NAS of type SVM resp. VFILER is the
NAS hosting the SVM resp. vfiler. The parent of a NAS running NetApp DATA
ONTAP in clustermode is the NAS of type cDOTCluster representing the cluster
management.
--ostype <ONTAP|cDOT>
Defines the os type the NAS is running.
--10gbit
NAS is a 10Gbit system.
Example for a NetApp FAS System
cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op add --name filer2 --type FAS3220
--swgroup 1
update LDAP
.....
update switch 1/1 configuration
Notice: Update will take about 1 minute.
................................+
Some manual interventions are necessary to integrate the NAS
into FlexFrame environment.
The following list of actions has to be performed in order to integrate the NAS
into your FlexFrame landscape. Since your exact configuration may vary, these steps
have to be performed manually. However, the VIF must be named 'storage'.
The /etc/rc, /etc/exports and /etc/hosts.equiv have to be edited
at volume vol0.
These lines have to be added to or changed at /etc/rc:
hostname filer2
vif create multi storage <-- add your NICs here, eg. e0a e0b for 1GB/sec ports;
e1 e2 for 10GB/sec ports.
NAS Systems Configuration Storage Systems Administration
Administration and Operation 131
vlan create storage 13
ifconfig storage-13 192.168.20.4 netmask 255.255.255.0 broadcast 192.168.20.255
mtusize 1500 -wins up
options dns.enable off
options nis.enable off
savecore
These lines have to be added to or changed at /etc/exports:
/vol/vol0 -sec=sys,rw=192.168.20.2:192.168.20.1,anon=0
These lines have to be added to /etc/hosts.equiv:
192.168.20.2 root
192.168.20.1 root
As the switch ports are already configured the correct wiring between NAS and
switch ports has to be done. See below a list of cable connections.
Connect your NAS LAN interfaces to named switch ports:
SwitchGroup / Switch / Port LAN Interface
1 / 2 / 3 filer2 (FAS3220): port "data NIC-1"
1 / 1 / 3 filer2 (FAS3220): port "data NIC-2"
Finally execute command "mount /FlexFrame/filer2/vol0"
on both Control Nodes to mount filers vol0 at Control Nodes. This is necessary for
further automated configuration of filer.
The complete instruction above is listed at file /tmp/filer2-add/todo
8.1.2 Removing a NAS
To remove a NAS from FlexFrame landscape it must not be used by any pool.
ff_nas_adm.pl --op list will display the current configuration of the NAS.
The switch port configuration will be removed from switches and LDAP database.
Synopsis
ff_nas_adm.pl --op rem --name <node name>
Options
--op rem
Storage Systems Administration NAS Systems Configuration
132 Administration and Operation
Removes a NAS from FlexFrame landscape about the NAS configuration and
displays some information about processing steps.
--name <node name>
Defines the node name of the NAS.
Example
cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op rem --name filer2
update switch 1/1 configuration
Notice: Update will take about 1 minute.
.............................+
update LDAP
.....
NAS successfully removed from network and LDAP.
8.1.3 Displaying all configured NAS
To get an overview of all configured NAS within the FlexFrame environment use the
operation mode list-all of the program ff_nas_adm.pl. It displays IP addresses
and names, type, switch ports and the link aggregation id, separated by NAS.
Synopsis
ff_nas_adm.pl --op list-all
Option
--op list-all
Displays all configured NAS.
Example
cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op list-all
NAS configurations
filer
Control Lan
192.168.20.3 filer-co
Type: FAS3220
Shell: /usr/bin/ssh -l flexframe
Switch Link Aggregation
Port Count: 2
Link Aggr.ID: 5
Storage LAN switch ports
1 / 1 / 13 SwGroup / Switch / Port
NAS Systems Configuration Storage Systems Administration
Administration and Operation 133
1 / 2 / 13 SwGroup / Switch / Port
Control LAN switch ports
1 / 2 / 15 SwGroup / Switch / Port
Pools
pool1 192.168.2.131 filer-pool1-st master
pool2 10.3.1.3 filer-pool2-st master
pool5 192.168.40.3 filer-pool5 master
usr 192.168.5.3 filer-usr-st master
Storage Systems Administration NAS Systems Configuration
134 Administration and Operation
8.1.4 Displaying NAS Configuration
To display the detailed configuration of a NAS as known by LDAP database, use the
command ff_nas_adm.pl with operation mode list.
Synopsis
ff_nas_adm.pl --op list --name <node name>
Options
--op list
Displays the configuration of a switch port.
--name <node name>
Defines the node name of a NAS.
Example for a NetApp Filer:
cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op list --name filer
NAS configurations
filer
Control Lan
192.168.20.3 filer-co
Type: FAS3220
Shell: /usr/bin/ssh -l flexframe
Switch Link Aggregation
Port Count: 2
Link Aggr.ID: 5
Storage LAN switch ports
1 / 1 / 13 SwGroup / Switch / Port
1 / 2 / 13 SwGroup / Switch / Port
Control LAN switch ports
1 / 2 / 15 SwGroup / Switch / Port
Pools
pool1 192.168.2.131 filer-pool1-st master
pool2 10.3.1.3 filer-pool2-st master
pool5 192.168.40.3 filer-pool5 master
usr 192.168.5.3 filer-usr-st master
NAS Systems Configuration Storage Systems Administration
Administration and Operation 135
8.1.5 Adding a Pool to a NAS
To be able to use an existing NAS for a pool, the network connection to the NAS has to
be enhanced. On the NAS, a new virtual interface has to be created and on the switch
ports the new VLAN has to be configured. All these steps are done with ff_nas_adm.pl
using the operation mode add-pool, but the program will not change any exports.
Synopsis
ff_nas_adm.pl --op add-pool --name <node name> --pool <pool name>
[--host <ip host part>]
Options
--op add-pool
Adds the given pool to named NAS.
--name <node name>
Defines the node name of NAS.
--pool <pool name>
Defines the pool name the NAS has to be support. See usage for a list of configured
pools.
--host <ip host part>
Defines the host part to be used to build IP addresses for the Control or Storage
LAN networks. If this option is omitted, the script uses a free host number to
calculate the IP address.
Example
cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op add-pool --name filer --pool pool4
update LDAP
....
update switch 1/1 configuration
Notice: Update will take about 1 minute.
...........+
vlan: storage-25 has been created
Pool pool4 successfully added to NAS, LDAP and network.
8.1.6 Removing a Pool from a NAS
The rem-pool mode is the opposite to the add-pool operation mode of
ff_nas_adm.pl. It removes the virtual interface to the pool and removes VLAN access
on NAS switch ports for the given pool. This action will not be permitted if any SID of the
Storage Systems Administration NAS Systems Configuration
136 Administration and Operation
pool uses this NAS or the NAS is the default NAS of the pool. Within the last case, the
pool interface of the NAS will be removed when removing the pool.
Synopsis
ff_nas_adm.pl --op rem-pool --name <node name> --pool <pool name>
Options
--op rem-pool
Removes the given pool from the named NAS.
--name <node name>
Defines the node name of the NAS.
--pool <pool name>
Defines the pool name the NAS has to be support. See usage for a list of configured
pools.
Example
cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op rem-pool --name filer --pool pool4
update switch 1/1 configuration
Notice: Update will take about 1 minute.
...........+
update LDAP
....
Pool pool4 successfully removed from NAS, LDAP and network.
8.1.7 Create NAS HA Partnership
Synopsis
ff_nas_adm.pl --op partner –-name <node name> --partner
<node name>
Options
--op partner
Define a HA partnership for a NAS.
--name <node name>
Name of the NAS.
--partner <node name>
NAS Systems Configuration Storage Systems Administration
Administration and Operation 137
Name of a second NAS which should be partner to this NAS.
8.1.8 Moving a NAS
There are various reasons a NAS should be moved within the FlexFrame landscape.
A NAS should change its network connection, e.g. move from one switch group to another. This includes moving from and to FlexFrame external switches, distinguished by
the switch group id ‘0’ or changing the network speed. Use ff_nas_adm.pl --op
move to get the new network connection then change the connection to the displayed
ports manually. The NAS will not be available until the new connection is established.
A NAS should be replaced by a NAS of another type. Use ff_nas_adm.pl --op
move to set the new type then replace the NAS manually. The NAS will not be available
until the replacement is completed.
A vFiler should be migrated to another storage host. Use ff_nas_adm.pl --op move
to set the new storage host, then perform the migration manually using proper NetApp
commands. After migration adjust the vlan interface partnership manually both on the
former and on the new storage host. When performing offline migration the NAS will not
be available until the migration is completed.
Synopsis
ff_nas_adm.pl --op move --name <node name>
[--type <system type>]
[--swgroup <switch group id>]
[--1gbit|--10gbit]
[--parent <node name>]
[--interface <interface name>]
[--rootvolume <path to root volume>]
Options
--op move
move a NAS
--name <node name>
Defines the node name of the NAS to be considered.
--type <system type>
Defines the new type if the NAS should be replaced by another type.
--swgroup <switch group id>
Defines the switch group the data ports of the NAS should be connected to. Use
switch group id 0 to connect to switches not managed from FlexFrame.
Storage Systems Administration NAS Systems Configuration
138 Administration and Operation
--10gbit
move to 10 Gbit Connection .
--1gbit
move to 1Gbit Connection.
--parent <node name>
Defines the new parent the NAS should be moved to.
--interface <interface name>
Defines the new interface the NAS should use.
--rootvolume <path to root volume>
Defines the new root volume the NAS should use.
8.1.9 Changing NAS Command Shell
To change the command shell used to configure the NAS, the ff_nas_adm.pl
command supports the change_sh operation mode.
The given shell command replaces the currently used command or default configuring the
named NAS. To change the shell for more than one NAS the procedure has to be done
for each NAS.
Be sure there is no password request using this shell command to connect the
NAS.
Synopsis
ff_nas_adm.pl --op change-sh --name <node name>
--shcmd <shell command>
Options
--op change-sh
Changes the sh command for NAS configuration.
--name <node name>
Defines the node name of the NAS.
--shcmd <shell command>
NAS Systems Configuration Storage Systems Administration
Administration and Operation 139
Shell command (absolute path) used to configure NAS. If this option is omitted the
script uses the defaults /usr/bin/ssh for NAS systems.
Example
cn1:~ # ff_nas_adm.pl --op change-sh --name filer
–-shcmd '/usr/bin/ssh -l flexframe'
update LDAP
.
NAS shell command changed at LDAP.
Shell command changed to "/usr/bin/ssh -l flexframe".
8.1.10 Changing NAS Ports
To change the count of NICs of a virtual interface the ff_nas_adm.pl commands
supports the change-ports operation mode. The given number is the count of NICs the
interface should use. The difference between actual and given count of ports will be
allocated at switch group, proper configured and displayed at end of program run. The
additional wiring and configuration of NAS have to be done manually.
Synopsis
ff_nas_adm.pl --op change-ports --name <node name>
--ports <port count>
Options
--op change-ports
Change the count of NICs of a virtual interface.
--name <node name>
Defines the node name of the NAS.
--ports <port count>
Count of NICs the interface should use.
Example
cn1:~ # ff_nas_adm.pl --op change-ports --name filer –-ports 4
update LDAP.
Expand StorageLAN vif with 2 NICs up to 4 NICs.
Do this before(!) connecting new NICs to switch ports.
The additional switch ports at Switch Group are already configured.
Interconnect additional NICs between NAS and the switches of
Storage Systems Administration NAS Systems Configuration
140 Administration and Operation
SwitchGroup 1 as noted below:
SwGrpID/SwitchID/Port Filer
1 / 1 / 20 <==> Filer data NIC-3
1 / 2 / 20 <==> Filer data NIC-4
Unless any errors are reported follow instructions above to extend
NAS link aggregate(s) to switch group.
Look at "/opt/FlexFrame/network/wiring-channel-extend-filer.txt"
to get a copy of this message.
8.1.11 Changing NAS Path Export
To change the export of a NAS path the ff_nas_adm.pl commands supports the
export operation mode. Generally FlexFrame scripts are exporting a path according the
need. In special cases manual use of this operation mode may be considered, e.g. when
described to do so in the corresponding "FlexFrame® HW Characteristics Quickguide
("Hardwaresteckbrief")".
FlexFrame distinguishes the following datatypes to determine a export:
datatype export for access of
cn control nodes.
pool control nodes, hypervisors and pool given wih --pool.
sid control nodes and sids in pool given with --pool.
an control nodes and node given with --an.
allro control nodes and readonly for everyone.
Synopsis
ff_nas_adm.pl --op export --name <node name>
--path <path>
--datatype <cn|pool|sid|an|allro>
[--pool <pool name>]
[--an <node name>]
[--createpath]
[--setvoloptions]
Options
--op export
export a path on given NAS.
--name <node name>
Using SAN Storage Storage Systems Administration
Administration and Operation 141
Defines the node name of the NAS.
--path <path>
path the NAS has to export.
--datatype <cn|pool|sid|an|allro>
type of data to determine the export
--pool <pool name>
pool the data is associated with.
--an <node name>
aplication node the data is associated with.
[--createpath]
create the path to be exported, the volume must already exist.
[--setvoloptions]
set the FlexFrame default vol options.
Example
cn1:~ # ff_nas_adm.pl --op export --name filer \
--path /vol/pool3volff --datatype cn --setvoloptions
NAS path exported.
8.2 Using SAN Storage
Instead of using volumes from a NAS system over NFS, the database datafiles (sapdata)
and database logfiles (saplog) of a SAP system can be placed on SAN based storage.
Configuring the SAN storage environment for a FlexFrame landscape is a complex task
that needs careful planning and design. A team of specialists for storage systems,
servers and SAN infrastructure must work out a concept that will be a guideline for the
complete setup.
Furthermore, it is required to consult the actual recommendations from the manufacturer
of the used storage and SAN components and from the involved partners as well as the
relevant documents from Fujitsu. These documents can contain important information
that must be taken into account when planning the configuration and when configuring
the individual components.
Storage Systems Administration Using SAN Storage
142 Administration and Operation
8.2.1 Getting started with SAN usage in FlexFrame
This section gives an overview of the steps needed when setting up a SAN configuration
for FlexFrame usage. Please note that some of these steps are outside the scope of
FlexFrame and are not described in detail here. The SAN configuration for FlexFrame will
usually be done by a storage systems expert in cooperation with a member of the
FlexFrame implementation team.
Before starting with the SAN setup, a detailed storage concept should be worked out.
Amongst others, this concept has to define which SAP systems will use SAN based
storage for the database, the number and size of LUNs that will be used for these SAP
systems and the Application Nodes on which the database instances with SAN based
storage will run.
In FlexFrame context, the term LUN refers to a storage area provided by a
storage system that can be seen as a disk device by a host. Depending on
system type, it may be referred to by a different term (e.g. volume).
It must be taken into account that all Application Nodes on which a database instance will
run must be able to access the LUNs used by the database. This are the Application
Nodes of the poolgroup to which the Node where the instance is first started manually
belongs.
Assign only a small number of Application Nodes to such a poolgroup and plan
to start only very few SAN based database instances in one poolgroup.
Avoid hundreds of parallel FC logins, as this may lead to problems in case of
some events like powering on all Application Nodes or rebooting a FC Switch.
The steps listed below give only a rough outline of what must be done. Some actions may
be bundled or executed in another sequence than shown, others may be omitted as the
environment already exists.
1. Install and configure the storage system(s)
The SAN storage system is not considered as a part of the FlexFrame system; you
can use LUNs from a storage system which is also used for other purposes in your
computing environment. The FlexFrame software itself does not need an
administrative access to the SAN storage system. However, if an administrative
access to the storage system from inside the FlexFrame environment is desired, you
can add it as an “External Connectivity”.
Perform this step based on the storage system documentation or use an already
configured storage system according to the advice of its administrator.
2. Install and configure the Fibre Channel switches
The Fibre Channel switches are also not a part of the FlexFrame system. The
administration and configuration of the SAN fabric and the Fibre Channel switches is
Using SAN Storage Storage Systems Administration
Administration and Operation 143
completely in the responsibility of the customer. The FlexFrame software itself does
not need an administrative access to the FC switches, but you can add the FC
switches as “External Connectivity” if wanted.
Perform this step based on the vendor documentation or use the existing
environment according to the advice of its administrator.
3. Prepare the Application Nodes
All Application Nodes that will run database instances of SAP systems configured for
SAN usage must be equipped with Host Bus Adapters (HBAs). To ensure that the
correct server type and type of Fibre Channel HBA is used, please consult the
FlexFrame Support Matrix. For performance reasons as well as to protect against
hardware failure, it is strongly recommended to use more than one HBA or at least a
dual ported HBA on each Application Node. Using more than four HBAs (or HBA
ports) is not recommended.
Depending on the used combination of OS/HBA/storage system, specific driver
settings may be needed in the Application Node image. These must be set using a
Maintenance Cycle as described in section 11.5 “Maintenance of Application Nodes -
Software”.
To manage the multiple paths between host and storage system, a multipath
software is needed.The FlexFrame Linux Application Node Images contain a ready-
to-use installation of the Linux native multipath software DM-MPIO.
4. Connect storage and Application Nodes
Before connecting the Application Nodes to the SAN storage system, please ensure
that your configuration including server, HBA, Fibre Channel switch and storage
system is supported according to the FlexFrame Support Matrix. Also check that all
used components are correctly configured according to the actual recommendations
from the respective manufacturer or partner and from the MatrixEP.
The storage and the Application Nodes are connected via the SAN fabric. Therefore,
connect the Fibre Channel ports of the storage system and of the Application Nodes
with the FC switches. For redundancy purposes, take care to connect the ports of an
Application Node to two different switches that are part of independent fabrics.
Arrange for a reasonable number of pathes between Application Nodes and storage
systems. There should be two or four pathes for each LUN. More than four pathes
are not recommended.
5. Implement zoning
Create zones on the FC switches to protect servers and storage systems. There is
no specific requirement on how zoning is to be done for a FlexFrame environment,
but it must be ensured that all Application Nodes on which a SAN based database
instance will be started can access the storage systems on which the LUNs of this
database are located.
Storage Systems Administration Using SAN Storage
144 Administration and Operation
6. Create LUNs on the storage system
For each SAN based database instance, create at least two LUNs on the storage
system, one for sapdata and one for saplog. The concrete number and attributes of
the LUNs are defined as a part of the storage concept.
For information on how to create LUNs (may be also referred to as volumes, storage
volumes, devices, depending on storage system type) refer to the documentation of
the used storage system.
7. Make LUNs accessible for Application Nodes (LUN masking/mapping)
On the storage system, perform actions to ensure that all LUNs needed for database
instances that will be started in a specific poolgroup are accessible for all Application
Nodes of this poolgroup.
Refer to the documentation of the used storage system for a description of the
specific actions.
8. Verify LUN visibility on Application Nodes
Check that all needed LUNs are visible on the designated Application Nodes. You
may use the ff_san_helper.pl script as an assistance
an1:~ # ff_san_helper.pl --op lun-info
If not all needed LUNs are visible on a specific Application Node, reboot the Node.
You may also use the script rescan-scsi-bus.sh according to the SLES
documentation.
9. Add volume group and LUN definitions to the FlexFrame configuration
Based on the LUNs provided by the storage system, volume groups using Linux
Logical Volume Manager (LVM or, more exactly, LVM2) will be created in a later
step. These volume groups will then be assigned to SAP systems for usage as
sapdata or saplog storage. As a prerequisite for these steps, the definitions of the
volume groups and optionally the LUNs on which the volume groups are based must
be added to the FlexFrame configuration.
You can add volume group and LUN definitions with the FlexFrame Management
Tool or using the command line:
cn1:~ # ff_volgroup_adm.pl --op add ...
cn1:~ # ff_volgroup_adm.pl --op add-lun ...
For details and examples see section 8.2.2 “Administrating Volume Groups and
LUNs” below.
10. Assign volume groups as sapdata/saplog storage to SAP systems (SIDs)
Using SAN Storage Storage Systems Administration
Administration and Operation 145
This can be done directly when adding a SID with ff_sid_adm.pl, or after the SID
has been added with ff_sid_mnt_adm.pl. The syntax is equivalent to the syntax
used for specifying an SID specific NAS volume, but use the keyword ‘_SAN’ instead
of the NAS system name and the volume group name instead of the NAS volume
name. For details on these commands see chapter 10 “SAP System Handling”.
Below only a part of the syntax is shown.
cn1:~ # ff_sid_adm.pl --op add --pool <pool name> --sid <SID> ...
cn1:~ # ff_sid_mnt_adm.pl --op add --pool <pool name> --sid <SID> \
cn1:~ # ff_sid_mnt_adm.pl --op add --pool <pool name> --sid <SID> \
--name sapdata:_SAN:<vg name>:<instnr>:<nodeid> \
--name saplog:_SAN:<vg name>:<instnr>:<nodeid>
It is also possible to assign SAN volume groups as sapdata/saplog storage to SIDs
using the FlexFrame Management Tool, in planning mode as well as in admin mode.
Please note that a SAN based database instance needs two different volume groups,
one for sapdata and one saplog and that these volume groups cannot be used for
another SID (or SID/instance/nodeid tuple, in case of HANA).
11. Create directory and file structure for SAP systems (SIDs)
With ff_setup_sid_folder.sh, some directories, files and symbolic links are
created on the NAS based volumes that are used by a SAP system. This must be
done also for SIDs using SAN based storage, as only the sapdata/saplog area is
located on SAN. Furthermore, this also creates links to the directories where the
respective SAN areas will be accessible on the Application Nodes instead of the
corresponding links into the NAS based sapdata/saplog volumes, so that one can
use the same specification for data and log volumes during the SAP installation
irrespective of the used storage type.
cn1:~ # ff_setup_sid_folder.sh --pool <pool> --sid <SID> [--instance <instnr>]
For SIDs created with the FlexFrame Management Tool, this step is done implicitly
either during the initial configuration or when the SID is added.
In any case, ff_setup_sid_folder.sh does not create the structure needed on
the SAN based storage. This task is completed during the next step based on an advice file that is created by ff_setup_sid_folder.sh if the SID is defined to
use SAN based storage.
12. Create volume groups, volumes and file systems for SIDs
For each SAP system that uses SAN based storage, volume groups containing the
designated LUNs, logical volumes as part of the volume groups and file systems on
top of the volumes must be created before the SAP installation of this system can be
started. Depending on SAP system type, also some directories must be created on
the file systems.
Storage Systems Administration Using SAN Storage
146 Administration and Operation
This step can be done manually or assisted by ff_san_helper.pl on an
Application Node that can access the LUNs needed by the respective SID.
an1:~ # ff_san_helper.pl --op setup-sid-stor --sid <SID> ...
For details and examples see section 8.2.3 “Completing SAN Setup for a SAP
System” below.
13. Check usability of file systems on all Application Nodes of the designated poolgroup
On each Application Nodes of the poolgroup where a database service will be
started, mount the file systems needed by the service using ff_san_mount.pl (--
action start), then umount them again using ff_san_mount.pl (--action
stop). This also handles the underlying volume groups and volumes as needed.
Begin with umounting on the Application Node where you created the file systems.
an1:~ # ff_san_mount.pl --action stop --sid <SID> ...
an2:~ # ff_san_mount.pl --action start --sid <SID> ...
an2:~ # ff_san_mount.pl --action stop --sid <SID> ...
For details about ff_san_mount.pl see section 8.2.4 “Mounting SAN based File
Systems of a SAP Service” below.
14. Start SAP installation on an Application Node of the poolgroup
Before starting the installation, mount the file systems again on the selected
Applicaton Node as shown above. You may also want to check that the file systems
are mounted on this Application Node:
an1:~ # ff_san_mount.pl --action status --sid <SID> ...
Refer to the “FlexFrame Orchestrator – Installation Guide for SAP Solutions” for brief
instructions for installating SAP in a FlexFrame environment.
8.2.2 Administrating Volume Groups and LUNs
Use ff_volgroup_adm.pl to administrate SAN based volume groups and LUNs in
FlexFrame.
In FlexFrame context, a LUN (Logical UNit) is a storage area provided by a storage
system that can be used on a host. Depending on storage system type, it may be called
also volume, storage volume or device. A LUN is identified in FlexFrame by a symbolic
name that is assigned by the FlexFrame administrator.
The LUN provided by the storage system is described by several detail data, like the LUN
global unique ID, the host specific LUN ID (number) and the storage system specific LUN
ID in conjunction with the storage system identification.
Using SAN Storage Storage Systems Administration
Administration and Operation 147
LUN administration in FlexFrame does not imply any activities on the storage systems,
which must be done manually by the storage administrator; it is restricted to recording of
information and some plausibility checks.
Volume groups are built on top of LUNs seen on a host and can be assigned to SAP
systems for usage as sapdata or saplog storage. This also needs some manual
preparations, like the creation of volume groups, volumes and file systems, which must
be done on an Application Node that has access to the underlying LUNs.
The script ff_volgroup_adm.pl only administrates the information about volume
groups and LUNs in the FlexFrame configuration.
8.2.2.1 Adding a Volume Group
Use ff_volgroup_adm.pl --op add to add a SAN volume group definition to the
FlexFrame configuration, optionally together with the definition of a LUN on which the
volume group is based. You can also create the LUN definition separately, or add
additional LUNs if the volume group is based on more than one LUN, as shown in next
section “Adding a LUN”. For more information about LUNs in FlexFrame refer to section
“Adding a LUN”.
After adding a volume group to the FlexFrame configuration, it can be assigned as
sapdata or saplog storage for the database of a SAP system with the command ff_sid_adm.pl or ff_sid_mnt_adm.pl. For details see chapter 10 “SAP System
Handling”.
Before a volume group can be actually created and used, it is also necessary that the
LUNs on which it is based are created on a storage system and that these LUNs are
accessible on at least one Application Node. Before the SAP system database instance
using the volume group is put into operation, it is to be ensured that all Application Nodes
of the poolgroup where the instance is started can access these LUNs. To document the
poolgroup where the volume group will be used, specify this poolgroup as an option of
the volume group definition.
Synopsis
ff_volgroup_adm.pl --op add
--name <vg name> --pool <pool name>
[--group <poolgroup name>] [--lun <lun symbolic name>
[--lunguid <global unique lun id> --hostlun <host lun id>
--storlun <storsys spec lun id> --storsysid <ssys id>]]
Options
--op add
Add a SAN volume group definition to the FlexFrame configuration
--name <vg name>
Storage Systems Administration Using SAN Storage
148 Administration and Operation
Name of the new volume group definition that is added to the FlexFrame
configuration. A volume group definition with the same name must not already exist
in the pool given with option --pool. The name may consist of 1 up to 32
characters (letters, digits, underscores, dots and dashes) and must not start with a
dash. The name must reflect the name of the real volume group that will be created
based on the designated LUNs on an Application Node with access to these LUNs.
--pool <pool name>
Name of the FlexFrame pool to which the volume group definition is added.
--group <poolgroup name>
Name of the poolgroup within the FlexFrame pool to which the Application Nodes
where the volume group can be used must belong.
--lun <lun symbolic name>
The LUN symbolic name is used to identify a specific LUN definition from the pool
given with option --pool. It is a FlexFrame internal name, while the real LUN is
identified by the LUN detail options --lunguid, --hostlun, --storlun and
--storsysid. If a LUN symbolic name is given, it is used to add a LUN reference
to the volume group definition that is created. If a LUN with this symbolic name is not
already known, it is added implicitly to the FlexFrame configuration. In this case, all
LUN detail options must be specified.
--lunguid <global unique lun id>
Globally unique LUN ID as assigned by the storage system (32 hex chars).
--hostlun <host lun id>
LUN ID (number) as seen on the host where the LUN is accessed.
--storlun <storsys spec lun id>
Storage system specific LUN ID.
--storsysid <storage system id>
Storage system identification or name.
The options --lunguid, --hostlun, --storlun and –storsysid must be
specified when a LUN definition is created and should reflect the corresponding
attributes oft he real LUN. For more information, see next section, “Adding a LUN”.
Example
cn1:~ # ff_volgroup_adm.pl --op add --name vg_data_ha1 --pool pool1 --group san1
update LDAP
cn1:~ # ff_volgroup_adm.pl --op add --name vg_logs_ha1 --pool pool1 --group san1
update LDAP
Using SAN Storage Storage Systems Administration
Administration and Operation 149
Example with LUN definitions
cn1:~ # ff_volgroup_adm.pl --op add --name vg_data_ha5 --pool pool1 --group san1 \
--lun HA5data --lunguid 600000E00D280000002802D6000B0000 --hostlun 6 \
--storlun 11 --storsysid dx200s3
update LDAP
cn1:~ # ff_volgroup_adm.pl --op add --name vg_logs_ha5 --pool pool1 --group san1 \
--lun HA5logs --lunguid 600000E00D280000002802D6000C0000 --hostlun 7 \
--storlun 12 --storsysid dx200s3
update LDAP
8.2.2.2 Adding a LUN
Use ff_volgroup_adm.pl --op add-lun to add a LUN reference to a volume group
or to add a LUN definition to the FlexFrame configuration.
Please note that for basic FlexFrame functionality it is not absolutely necessary to define
LUNs in FlexFrame, but when the LUNs on which the volume groups are based are
known, some enhanced functions are available.
Synopsis
ff_volgroup_adm.pl --op add-lun
[--name <vg name>] --pool <pool name>
--lun <lun symbolic name>
[--lunguid <global unique lun id> --hostlun <host lun id>
--storlun <storsys spec lun id> --storsysid <ssys id>]
Options
--op add-lun
Add a LUN reference to a volume group or a LUN definition to the FlexFrame
configuration
--name <vg name>
If this option is given, it specifies the name of an already known volume group to
which a LUN reference will be added. Without this option, only a LUN definition is
created.
--pool <pool name>
Name of the FlexFrame pool to which the volume group and LUN definition belongs
to or is added.
--lun <lun symbolic name>
Storage Systems Administration Using SAN Storage
150 Administration and Operation
The LUN symbolic name is used to identify a specific LUN definition from the pool given with option --pool. It is a FlexFrame internal name, while the real LUN is
identified by the LUN detail options --lunguid, --hostlun, --storlun and
--storsysid. If a LUN with this symbolic name is already known, it is used to add
a LUN reference to the volume group given with option --name. Otherwise, a LUN
definition is created. In this case, all LUN detail options must be specified.
--lunguid <global unique lun id>
Globally unique LUN ID as assigned by the storage system (32 hex chars).
--hostlun <host lun id>
LUN ID (number) as seen on the host where the LUN is accessed.
--storlun <storsys spec lun id>
Storage system specific LUN ID.
Choose a value that, together with the storage system identification or name, makes
it easy to identify the LUN (storage volume, device) when communicating with the
storage administrator.
--storsysid <storage system id>
Storage system identification or name.
The options --lunguid, --hostlun, --storlun and –storsysid must be
specified when a LUN definition is created and should reflect the corresponding
attributes of the real LUN. The example below shows how you can get the needed
information from an ETERNUS storage system.
Example
cn1:~ # ff_volgroup_adm.pl --op add-lun --name vg_data_ah2 --pool pool1 \
--lun AH2data --lunguid 600000E00D280000002802D600020000 --hostlun 0 \
--storlun 2 --storsysid dx200s3
update LDAP
cn1:~ # ff_volgroup_adm.pl --op add-lun --name vg_logs_ah2 --pool pool1 \
--lun AH2logs --lunguid 600000E00D280000002802D600030000 --hostlun 1 \
--storlun 3 --storsysid dx200s3
update LDAP
To get the data for the LUN definitions shown in this example, we used the CLI of an
ETERNUS storage system. In ETERNUS terminology, our LUNs are volumes.
In this example, we use the storage system name as storsysid and the volume
number as storlun. The volume name is used as lun symbolic name. This mapping
Using SAN Storage Storage Systems Administration
Administration and Operation 151
scheme is only a suggestion. It is also possible to use a part of the boxid as storsysid,
or to use the volume name as storlun.
The following mapping is mandatory: the volume UID must be specified as lunguid and
the LUN must be specified as hostlun.
CLI> show storage-system-name
Name [dx200s3]
Installation Site [Augsburg DC1a_004]
Contact []
Description []
CLI> show volume-mapping -volume-number 2,3
Volume Type UID
No. Name
----- -------------------------------- --------- --------------------------------
2 AH2data Standard 600000E00D280000002802D600020000
<Mapping>
LUN LUN Group Port
No. Name
---- ---- ---------------- ----------------
0 1 sid_group2 -
Volume Type UID
No. Name
----- -------------------------------- --------- --------------------------------
3 AH2logs Standard 600000E00D280000002802D600030000
<Mapping>
LUN LUN Group Port
No. Name
---- ---- ---------------- ----------------
1 1 sid_group2 -
8.2.2.3 Removing a Volume Group
Use ff_volgroup_adm.pl --op rem to remove a volume group definition from the
FlexFrame configuration.
Synopsis
ff_volgroup_adm.pl --op rem
--name <vg name> --pool <pool name> [--keep] [--force]
Options
--op rem
Storage Systems Administration Using SAN Storage
152 Administration and Operation
Remove a volume group definition from the FlexFrame configuration.
--name <vg name>
Name of an existing volume group from pool given with option --pool.
--pool <pool name>
Name of the FlexFrame pool to which the volume group definition belongs.
--keep
Keep definitions of referenced LUNs. Without this option, the definitions of
referenced LUNs are also removed.
8.2.2.4 Removing a LUN
Use ff_volgroup_adm.pl --op rem-lun to remove a LUN reference from a volume
group or a LUN definition from the FlexFrame configuration.
Synopsis
ff_volgroup_adm.pl
--op rem-lun [--name <vg name>] --pool <pool name>
--lun <lun symbolic name> [--keep] [--force]
Options
--op rem-lun
Remove a volume LUN reference from a volume group or a LUN definition from the
FlexFrame configuration.
--name <vg name>
If this option is given, it specifies the name of the volume group from which the LUN
reference will be removed. Without this option, only a LUN that is not referenced by
a volume group can be removed.
--pool <pool name>
Name of the FlexFrame pool to which the volume group or LUN definition belongs.
--keep
Keep definition of LUN. Remove only the LUN reference from thevolume group.
Using SAN Storage Storage Systems Administration
Administration and Operation 153
8.2.2.5 Changing Poolgroup Assignment of a Volume Group
Use ff_volgroup_adm.pl --op group to change the poolgroup assignment of a
volume group. The poolgroup assignment of a volume group documents on which
Application Nodes this volume group will be used. It must be ensured that all Application
Nodes of the poolgroup can access the LUNs on which the volume group is based.
When changing the poolgroup assignment, make sure that all Application Nodes of the
new poolgroup can access these LUNs. Likewise, the Application Nodes of the old
poolgroup no longer need access to these LUNs.
Synopsis
ff_volgroup_adm.pl
--op group --name <vg name> --pool <pool name>
--group {<poolgroup name>|0}
Options
--op group
Change poolgroup assignment of a volume group.
--name <vg name>
Name of an existing volume group from pool given with option --pool.
--pool <pool name>
Name of the FlexFrame pool to which the volume group definition belongs.
--group {<poolgroup name>|0}
Name of the poolgroup within the FlexFrame pool to which the Application Nodes where the volume group can be used must belong. Use value 0 to remove the
current poolgroup assignment.
8.2.2.6 Displaying Information about Volume Groups and LUNs
Use ff_volgroup_adm.pl --op list to display details of a specific volume group,
--op list-all to display an overview of all volume groups, --op list-luns to
display an overview of all LUNs or --op list-select to display volume groups
according to selection criteria.
Synopsis
ff_volgroup_adm.pl
--op list --name <vg name> --pool <pool name> [--cmdline]
Storage Systems Administration Using SAN Storage
154 Administration and Operation
ff_volgroup_adm.pl
--op list-all [--pool <pool name>]
ff_volgroup_adm.pl
--op list-luns [--pool <pool name>] [--unused] [--cmdline]
ff_volgroup_adm.pl --op list-select --pool <pool name>
[--group {<poolgroup name>|0}]
[--lun {<lun symbolic name>|0}] [--sid {<SAP system ID>|0}]
Options
--op list
Display details of a specific volume group
--op list-all
Display an overview of all volume groups
--op list-luns
Display an overview of all LUNs
--op list-select
Display volume groups according to selection criteria
--name <vg name>
Identifies the volume group from the pool given with option --pool.
--pool <pool name>
Identifies the FlexFrame pool.
--group {<poolgroup name>|0}
Select volume groups assigned to given poolgroup. Use the value 0 to select volume
groups with no poolgroup assignment.
--lun {<lun symbolic name>|0}
Select volume groups using the given LUN. Use the value 0 to select volume groups
with no LUN reference.
--sid {<SAP system ID>|0}
Select volume groups used as sapdata or saplog storage for given SID. Use the
value 0 to select volume groups not used by any SID.
--cmdline
If used with --op list or --op list-luns, the command line(s) that can be
used to recreate the dispIayed definitions is also shown.
Using SAN Storage Storage Systems Administration
Administration and Operation 155
Examples
Display details of a specific volume group and its used LUN(s), with command line(s):
cn1:~ # ff_volgroup_adm.pl --op list --name vg_data_ah2 --pool pool1 --cmdline
Configuration details of volume group vg_data_ah2 in pool pool1
Volume Mgr Type: LVM
Assigned poolgroup: san1
Used as: sapdata for SID AH2
Mountpath Info:
/dev/vg_data_ah2/data_AH2 on /var/FlexFrame/SAN/sapdata/AH2
Used LUNs: AH2data
Configuration details of used LUNs, grouped by storage system
Storage system dx200s3
LUN name LUN GUID h-LUN s-LUN
AH2data 600000E00D280000002802D600020000 0 2
Command line used to create volume goup definition:
ff_volgroup_adm.pl --op add --name vg_data_ah2 --pool pool1 --group san1
Command line(s) used to create LUN definition(s) and reference(s):
ff_volgroup_adm.pl --op add-lun --lun AH2data --name vg_data_ah2 --pool pool1
--lunguid 600000E00D280000002802D600020000 --hostlun 0 --storlun 2
--storsysid dx200s3
Display an overview of all LUNs:
cn1:~ # ff_volgroup_adm.pl --op list-luns --pool pool1
LUNs sorted by pool, grouped by storage system and sorted by lun name
(output reduced to pool pool1)
POOL pool1
Storage system dx200s3
LUN name LUN GUID h-LUN s-LUN
AH2data 600000E00D280000002802D600020000 0 2
AH2logs 600000E00D280000002802D600030000 1 3
HA3data 600000E00D280000002802D600050000 2 5
HA3logs 600000E00D280000002802D600060000 3 6
HA4data 600000E00D280000002802D600090000 4 9
HA4logs 600000E00D280000002802D6000A0000 5 10
HA5data 600000E00D280000002802D6000B0000 6 11
Storage Systems Administration Using SAN Storage
156 Administration and Operation
HA5logs 600000E00D280000002802D6000C0000 7 12
Selecting volume groups of a pool assigned to a given poolgroup:
cn1:~ # ff_volgroup_adm.pl --op list-select --group san1 --pool pool1
Volume Groups from pool pool1
Selecting only volume groups assigned to poolgroup san1
vg_data_ah2
Used LUNs: AH2data
Assigned poolgroup: san1
Used as: sapdata for SID AH2
vg_data_ha4
Used LUNs: HA4data
Assigned poolgroup: san1
Used as: -
vg_data_ha5
Used LUNs: HA5data
Assigned poolgroup: san1
Used as: sapdata for SID HA5, instnr 05, nodeid 001
vg_logs_ah2
Used LUNs: AH2logs
Assigned poolgroup: san1
Used as: saplog for SID AH2
vg_logs_ha4
Used LUNs: HA4logs
Assigned poolgroup: san1
Used as: -
vg_logs_ha5
Used LUNs: HA5logs
Assigned poolgroup: san1
Used as: saplog for SID HA5, instnr 05, nodeid 001
Selecting volume groups of a pool not used by any SID:
cn1:~ # ff_volgroup_adm.pl --op list-select --sid 0 --pool pool1
Volume Groups from pool pool1
Selecting only volume groups not used by any SID
vg_data_ha4
Used LUNs: HA4data
Assigned poolgroup: san1
vg_logs_ha4
Used LUNs: HA4logs
Using SAN Storage Storage Systems Administration
Administration and Operation 157
Assigned poolgroup: san1
8.2.3 Completing SAN Setup for a SAP System
Before the volume groups that have been defined as shown in chapter 8.2.2 and
assigned to a SAP system (SID) for usage as sapdata or saplog storage can be used for
the database of this SAP system, it is also required to actually create these volume
groups and further objects based on the volume groups. This must be done on an
Application Node that can access the LUns on which the volume group is based.
These objects must complement the file and directory structure as created by the script ff_setup_sid_folder.sh and match the LDAP data that reflect the volume group
definitions and usage. You may use ff_san_helper.pl --op setup-sid-stor as
described in section 8.2.5 “SAN Helper Functions” or do the needed actions manually.
The script can only be used if LUN definitions exist for the volume groups.
1. For each defined volume group, create a Linux LVM (or, more exactly LVM2) volume
group with the defined name based on the multipath maps corresponding to the
designated LUNs as physical volumes, Tag the volume group with a “_FF4S” tag to
mark it for FlexFrame usage and a tag with the name or the current Application
Node.
2. In each volume group, create a logical volume named data_<SID> or log_<SID>,
respectively.
3. Create an ext3 file system on top of each logical volume,
4. For all database types except HANA, several directories must be created on each file
system.The needed informations can be found in the files dirlist_san_sapdata_<SID>.cfg and dirlist_san_saplog_<SID>.cfg
in the directory /FlexFrame/pooldata/config/scripts_config/ (as seen
on Application Node).
Before creating the directories, create mount points and mount the file systems on
the appropriate mount points as recorded with the volume group definitions (see
output of ff_volgroup_adm.pl --op list ...).
After completing these actions, umount the file systems (if mounted), deactivate and
export the volume groups and remove the host specific tag from the volume groups. Do this by calling ff_san_mount.pl --action stop --sid ... as shown in section
8.2.4.
Example
The example below shows a shortened list of the commands executed by
ff_san_helper.pl –op setup-sid-stor –sid AH4.
Storage Systems Administration Using SAN Storage
158 Administration and Operation
vgcreate --addtag _FF4S --addtag lucrx1 vg_data_ah4 \
/dev/disk/by-id/dm-name-3600000e00d280000002802d600090000
lvcreate --name data_AH4 --extents 100%VG vg_data_ah4
mkfs.ext3 /dev/vg_data_ah4/data_AH4 -L data_AH4
mkdir -v -p /var/FlexFrame/SAN/sapdata/AH4
mount /dev/vg_data_ah4/data_AH4 /var/FlexFrame/SAN/sapdata/AH4
mkdir -v -m 775 -p /var/FlexFrame/SAN/sapdata/AH4/sapdata1
chown -c 3608:3602 /var/FlexFrame/SAN/sapdata/AH4/sapdata1
mkdir -v -m 775 -p /var/FlexFrame/SAN/sapdata/AH4/sapdata2
chown -c 3608:3602 /var/FlexFrame/SAN/sapdata/AH4/sapdata2
<...>
vgcreate --addtag _FF4S --addtag lucrx1 vg_logs_ah4 \
/dev/disk/by-id/dm-name-3600000e00d280000002802d6000a0000
lvcreate --name log_AH4 --extents 100%VG vg_logs_ah4
mkfs.ext3 /dev/vg_logs_ah4/log_AH4 -L log_AH4
mkdir -v -p /var/FlexFrame/SAN/saplog/AH4
mount /dev/vg_logs_ah4/log_AH4 /var/FlexFrame/SAN/saplog/AH4
mkdir -v -m 775 -p /var/FlexFrame/SAN/saplog/AH4/saplog1
chown -c 3608:3602 /var/FlexFrame/SAN/saplog/AH4/saplog1
<...>
8.2.4 Mounting SAN based File Systems of a SAP Service
The script ff_san_mount.pl is used on Application Nodes during start or stop of the
database service of a SAP system to control the availability of SAN based volume groups
and file systems needed by the service.
Besides this, it must be used manually during the installation of a SAP service that is
configured for SAN usage.
The related information is maintained in the FlexFrame configuration database (LDAP).
This can be done with the FlexFrame Management Tool or with the commands ff_volgroup_adm.pl to administrate volume groups and LUNs and
ff_sid_mnt_adm.pl to assign volume groups to SAP services.
Caution must be applied when using ff_san_mount.pl directly. The configured volume
groups and file systems must not be in use by another host.
Once the database is properly installed, the script ff_service.sh calls
ff_san_mount.pl during the start or stop phase implicitly.
Synopsis
ff_san_mount.pl --action {start | stop | status}
--sid <SAP system ID>
[--instance <HANA instance number> --nodeid <HANA node ID>
[--replid <HANA replication ID> ]]
Using SAN Storage Storage Systems Administration
Administration and Operation 159
Options
--action {start | stop | status}
Action mode.
Choose one of start to start, stop to stop or status to show status of SAN based
volume groups and file systems needed by the database service of a SAP system.
With action mode start, the volume groups are imported and activated and the
associated file systems are mounted. Activation also implies setting of a node
specific tag on the volume group.
With action mode stop, the file systems are umounted and the volume groups are
deactivated and exported. With the deactivation, the node specific tag is also
removed from the volume group.
Action mode status shows if the volume groups are active on this or another node
and if the file systems are mounted on this node. It does not imply a check of the
functional availability.
--sid <SAP system ID>
Identifies the SAP system to which the database service belongs.
--instance <HANA instance number>
HANA instance number (2-digit number), required for SAP HANA service.
--nodeid <HANA node ID>
Identifier of HANA database node (3-digit number), required for SAP HANA service.
--replid <HANA replication ID>
Identifier of HANA database replication node (1-digit number), only relevant for SAP
HANA service.
8.2.5 SAN Helper Functions
Use ff_san_helper.pl on an Application Node for helper functions related to the
preparation of storage for the database service of SAP systems that use SAN based
storage.
Synopsis
ff_san_helper.pl --op setup-sid-stor --sid <SAP system ID>
[--instance <HANA instance number> --nodeid <HANA node ID>
[--replid <HANA replication ID> ]]
ff_san_helper.pl --op hba-info
Storage Systems Administration Using SAN Storage
160 Administration and Operation
ff_san_helper.pl --op target-info
ff_san_helper.pl --op lun-info
Options
--op setup-sid-stor
Helps to complete the setup of storage for a SAP system. Refer to section 8.2.3
above for details.
--op hba-info
Display some brief information about FC HBAs of the Application Node
--op target-info
Display some brief information about FC targets seen on the Application Node
(usually corresponding to the ports of the SAN storage systems).
--op lun-info
Display some brief information about FC LUNs seen on the Application Node.
The following options are relevant only for --op setup-sid-stor.
--sid <SAP system ID>
Identifies the SAP system to which the database service belongs.
--instance <HANA instance number>
HANA instance number (2-digit number), required for SAP HANA service.
--nodeid <HANA node ID>
Identifier of HANA database node (3-digit number), required for SAP HANA service.
--replid <HANA replication ID>
Identifier of HANA database replication node (1-digit number), only relevant for SAP
HANA service.
Examples
an1:~ # ff_san_helper.pl -op hba-info
HBA WWNN WWPN
host7 20:00:00:90:fa:02:a7:22 10:00:00:90:fa:02:a7:22
host8 20:00:00:90:fa:02:a7:23 10:00:00:90:fa:02:a7:23
an1:~ # ff_san_helper.pl -op target-info
Using SAN Storage Storage Systems Administration
Administration and Operation 161
Target WWNN WWPN
7:0:0 50:00:00:e0:da:02:d6:00 50:00:00:e0:da:02:d6:20
8:0:0 50:00:00:e0:da:02:d6:00 50:00:00:e0:da:02:d6:30
an1:~ # ff_san_helper.pl -op lun-info
H-B-T-L Dev Dev-Id(GUID)
7:0:0:0 sdb 600000e00d280000002802d600020000
7:0:0:1 sdc 600000e00d280000002802d600030000
7:0:0:2 sdd 600000e00d280000002802d600050000
7:0:0:3 sde 600000e00d280000002802d600060000
8:0:0:0 sdj 600000e00d280000002802d600020000
8:0:0:1 sdk 600000e00d280000002802d600030000
8:0:0:2 sdl 600000e00d280000002802d600050000
8:0:0:3 sdm 600000e00d280000002802d600060000
Switch Administration Adding a Switch to a Switch Group
162 Administration and Operation
9 Switch Administration
Switch administration means adding or removing a switch group to resp. from the
FlexFrame landscape, adding or removing a switch to resp. from a switch group or
adding or removing a module to resp. from a switch.
A switch group consists of at least two switches of a switch family. Only switch groups
building a switch stack can consist of more than two switches. A switch stack is seen from
the network like one switch.
For a description how to interconnect switches, please refer to the vendors
Installation Manual.
Adding or removing a switch to or from a switch group means to add or remove
a switch to/from a switch stack.
To ensure safe operation we recommend doing this at a downtime to minimize
influence on running systems. This requires shutting down all systems con-
nected to this switch group. In case a NAS system is connected, all systems that
have mounted file systems from the NAS system have to be shut down as well.
Ensure all switches in the stack have a proper Software Version. If necessary
perform an upgrade resp. downgrade.
In special scenarios an upgrade can only be done step by step one switch after
the other. Please refer to original vendors documents for that task.
9.1 Adding a Switch to a Switch Group
A switch may be added to a switch group building a switch stack. See notes above for
recommendations.
The following processing has to be performed.
1. Mount the new switch next to the existing switches
2. Run ff_save_switch_config.pl to save configurations
3. Write down switch ids of each stack member of the switch stack as they may change
inserting the new switch to the stack.
4. Check the OS versions of the switches in the existing switch stack and the new
switch. Switches of the same model (G or E model) must have the same version. If
the versions are different, upgrade resp. downgrade the OS of the new switch.
5. Power off all switches of the switch stack, connect the new switch to the stack using
the provided stacking cable and stacking ports (see the vendors installation manual
for details), power on all switches of the stack except the new one.
Adding a Switch to a Switch Group Switch Administration
Administration and Operation 163
6. Compare the actual stack member ids with your noticed ids. In case of differences
use the following switch OS command for renumbering: switch <Switch Number> renumber <New switch number>
7. Power on the new switch, once again compare the stack member ids, set all
interfaces of the new switch to shutdown
8. Use ff_swgroup_adm.pl --op add-sw to add the switch to the FlexFrame
configuration as described below.
The switch group is now ready for use for further configuration.
Synopsis
ff_swgroup_adm.pl --op add-sw --group <switch group id>
--type <switch type> [--dryrun]
Options
--op add-sw
Adds a new member to the switch group and displays some information about
processing steps.
--group <switch group id>
Defines the switch group to be used.
--type <switch type>
Defines the type of the new switch to be added to the switch group. Call
ff_swgroup_adm.pl without any parameter to get a list of supported switch types
The maximum number of switches per switch group is 9. For more than 4 switches
with 10GbE ports the StackWise cabling may be a bottleneck.
--dryrun
For test purpose you can perform the function without changing the LDAP Database.
Switch Administration Adding a Switch to a Switch Group
164 Administration and Operation
Example
cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op add-sw --group 1
--type cat3750g-24ts
If program is aborted by Ctrl-C or a failure remove left overs
by calling:
ff_swgroup_adm.pl --op rem-sw --group 1 --switch 3
Switch was added to LDAP data.
Keep in mind: INSERTING SWITCH TO STACK NEEDS A DOWN TIME !
To add the switch to switch stack (switch group 1) write down the current switch
ids as they may change inserting the new switch to stack.
To connect the switch to the stack use the provided stacking cable and stacking
ports at rear side. See Cisco installation manual for details.
If switch ids get scrambled use the IOS command
"switch <current_no> renumber <new_no>"
to put them in same order as before.
In short the to do list:
-> write down switch ids of each switch of group
-> power down entire switch group
-> insert switch into stack
-> power on entire switch group
-> look at switch ids and compare with your noticed
-> in case of differences use IOS command to renumber switches
Switch group is ready for use
See file /tmp/swgrp-add-1-3/next_steps for same instructions as above.
Removing a Switch from a Switch Group Switch Administration
Administration and Operation 165
9.2 Removing a Switch from a Switch Group
A switch may be removed from a switch group building a switch stack. Remove it first from the LDAP database with the command ff_swgroup_adm.pl --op rem-sw as
described below and then from the stack. See notes above for recommendations.
All ports of the switch must be unused.
Only the switch with the highest member ID within the stack can be removed.
Synopsis
ff_swgroup_adm.pl --op rem-sw --group <switch group id>
--switch <switch id> [--dryrun]
Options
--op rem-sw
Removes the last member from a switch group and displays some information about
processing steps.
--group <switch group id>
Defines the switch group to be used.
--switch <switch id>
Defines the stack ID of the switch to be removed from a switch group.
--dryrun
For test purpose you can perform the function without changing the LDAP Database.
Example
cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op rem-sw --group 1 --switch 3
Switch was successfully removed from LDAP data.
Keep in mind: REMOVING SWITCH FROM STACK NEEDS A DOWN TIME !
In short the to do list:
-> power down entire switch group
-> remove switch from stack
-> power on entire switch group
Switch group is ready for use
See file /tmp/swgrp-rem-1-3/next_steps for same instructions as above.
Switch Administration Listing a Switch Group Configuration
166 Administration and Operation
9.3 Listing a Switch Group Configuration
Invoking the command ff_swgroup_adm.pl with the list operation mode displays
the configuration of a switch group like used switch types, port channels, port usage
statistics and used switch ports.
Synopsis
ff_swgroup_adm.pl --op list --group <switch group id>
Options
--op list
Displays switch group configuration.
--group <switch group id>
Defines the switch group to be used.
Example
cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op list --group 1
Switch Group 1
Name/IP: switch-i / 192.168.13.15
Login: root
Password: passwort
SNMP Community: public;ro
Switch Types: (switch id, switch type)
1 cat3750g-24t
2 cat3750g-24t
3 cat3750g-24t
Port Channels: (channel id, switch ports, connected device)
2 1/5,2/5 swb-1-1/11,swb-1-1/12
3 1/6,3/1 swb-1-2/11,swb-1-2/12
4 1/15,3/7 intfiler
5 1/19,2/11 swb-2-1/39,swb-2-1/40
6 2/22,3/22 swb-2-2/39,swb-2-2/40
Switch port usage: (switch id, used, free tx, free fx, free 10GB ports)
1 11 used 13 free tx 0 free fx 0 free 10Gb
2 10 used 14 free tx 0 free fx 0 free 10Gb
3 8 used 16 free tx 0 free fx 0 free 10Gb
Listing a Switch Group Configuration Switch Administration
Administration and Operation 167
Switch port list: (switch id, port id, connected device, vlans)
1 1 BX cabinet 2 u13
1 2 --- unused ---
1 3 rx300-14 t10,t12,u11
1 4 rx300-15 t10,t12,u11
1 5 swb-1-1/11 t13,t10,t12,t11,t1
1 6 swb-1-2/12 t13,t10,t12,t11,t1
1 7 --- unused ---
1 8 --- unused ---
1 9 --- unused ---
1 10 --- unused ---
1 11 cn1 u13,t11,t12,t10
1 12 cn2 u13,t11,t12,t10
1 13 --- unused ---
1 14 --- unused ---
1 15 intfiler t11,t13
1 16 extern. Connect t13,t10,t11,t12
1 17 --- unused ---
1 18 --- unused ---
1 19 swb-2-1/39 t13,t10,t12,t11,t1
1 20 --- unused ---
1 21 --- unused ---
1 22 --- unused ---
1 23 Corporate LAN u10
2 1 BX cabinet 2 u13
2 2 --- unused ---
2 3 rx300-14 t10,t12,u11
2 4 rx300-15 t10,t12,u11
2 5 swb-1-1/12 t13,t10,t12,t11,t1
2 6 --- unused ---
2 7 --- unused ---
2 8 --- unused ---
2 9 --- unused ---
2 10 --- unused ---
2 11 swb-2-1/40 t13,t10,t12,t11,t1
2 12 cn2 mgmt u13
2 13 --- unused ---
2 14 --- unused ---
2 15 cn1 mgmt u13
2 16 --- unused ---
2 17 --- unused ---
2 18 --- unused ---
2 19 --- unused ---
2 20 --- unused ---
2 21 BX cabinet 1 u13
Switch Administration Generating a Switch Group Configuration
168 Administration and Operation
2 22 swb-2-2/39 t13,t10,t12,t11,t1
2 23 Corporate LAN u10
3 1 swb-1-2/11 t13,t10,t12,t11,t1
3 2 --- unused ---
3 3 --- unused ---
3 4 --- unused ---
3 5 cn1 u13,t11,t12,t10
3 6 cn2 u13,t11,t12,t10
3 7 intfiler t11,t13
3 8 --- unused ---
3 9 --- unused ---
3 10 --- unused ---
3 11 --- unused ---
3 12 --- unused ---
3 13 --- unused ---
3 14 --- unused ---
3 15 --- unused ---
3 16 --- unused ---
3 17 --- unused ---
3 18 --- unused ---
3 19 BX cabinet 1 u13
3 20 rx300-15 mgmt u13
3 21 rx300-14 mgmt u13
3 22 swb-2-2/40 t13,t10,t12,t11,t1
9.4 Generating a Switch Group Configuration
Use the command ff_swgroup_adm.pl with the gen-config operation mode to
generate the configuration of switch group switches according to LDAP. The generated
configuration can be uploaded manually according to the corresponding "FlexFrame®
HW Characteristics Quickguide ("Hardwaresteckbrief")".
Customizations, if necessary, have to be reapplied. Inspection of the project
information see 11.10 and the last saved configurations in
/tftpboot/saved_switchconfigs may help.
If the connection to the NAS system may be lost because of the manually upload
change /tftpboot to be a local directory before generating the configuration. If the
generated configuration should be accessed by use of the tftp protocol also
restart tftpd.
mv /tftpboot /tftpboot.sav
mkdir /tftpboot
ff_ha_cmd.sh restart tftpd
Changing the Password of a Switch Group Switch Administration
Administration and Operation 169
After the upload go back using
mv /tftpboot/* /tftpboot.sav
rmdir /tftpboot
mv /tftpboot.sav /tftpboot
ff_ha_cmd.sh restart tftpd
Synopsis
ff_swgroup_adm.pl --op gen-config --group <switch group id>
Options
--op gen-config
Generates configuration of switch group switches.
--group <switch group id>
Defines the switch group to be used.
Example
Cn1:~ # ff_swgroup_adm.pl --op gen-config --group 1
Switch configuration generated. Upload has to be done manually.
Follow the instructions of the applicable Quick Start Hardware
FlexFrame for SAP document in the doc/hwinfo section of the Service CD
and use the following data:
Switch Group swg1
IP and Mask: 172.20.13.6 255.255.255.0
Config: scp://root@172.20.13.3//tftpboot/swg1.config
You can find a copy of the message above in
/var/log/FlexFrame/tmp/swgroup_adm-gen-config-1/msg_of_swgroup1_swg1.txt
9.5 Changing the Password of a Switch Group
To change the access password of a switch group it has to be changed at switch group
as well as LDAP database. The command ff_swgroup_adm.pl changes both. If it is
not possible to change the password on the switch because of password complexity
restrictions use this command to reset to the previous password to get resynchronized.
Switch Administration Changing the Host Name of a Switch Group
170 Administration and Operation
Synopsis
ff_swgroup_adm.pl --op pass --group <switch group id>
--passwd <password> [--dryrun]
Options
--op pass
Changes the switch group access.
--group <switch group id>
Defines the switch group to be used.
--passwd <password>
Defines the new password as clear text.
--dryrun
For test purpose you can perform the function without changing the LDAP Database.
Example
cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op pass --group 1 --passwd berta
update switch 1/1 configuration
Notice: Update will take about 1 minute.
............+
Password changed from "anton" to "berta".
See file /tmp/swgrp-pass-1/info for same information as above.
9.6 Changing the Host Name of a Switch Group
To change the host name of a switch group it has to be changed at switch group as well
as in the LDAP database and host files on both control nodes. The command
ff_swgroup_adm.pl changes all of them.
Synopsis
ff_swgroup_adm.pl --op name --group <switch group id>
--name <name> [--dryrun]
Changing the Host Name of a Switch Group Switch Administration
Administration and Operation 171
Options
--op name
Changes the switch group host name. The new name may consist of 2 up to 13
characters (letters, digits and dashes). It must start with a letter and must not end
with a dash.
--group <switch group id>
Defines the switch group to be used --name <name>
Defines the new host name to be used.
Switch Administration Displaying/Changing Common Network Configuration Parameters
172 Administration and Operation
--dryrun
For test purpose you can perform the function without changing the LDAP Database.
Example
cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op name --group 1 --name swg1
update switch 1/1 configuration
Notice: Update will take about 1 minute.
...+
Switch name changed from "swg-1" to "swg1".
See file /tmp/swgrp-name-1/info for same information as above.
9.7 Displaying/Changing Common Network Configuration Parameters
Some parameters of network configuration influence the switch group configuration and
were defined with the Management Tool at initial installation.
To display or change these parameters use the ff_swgroup_adm.pl command.
Synopsis
ff_swgroup_adm.pl --op parameter [–-parameter <parameter name>
--value <parameter value>] [--dryrun]
Options
--op parameter
Displays (without the following options) or changes (in conjunction with the following
options) network configuration parameters.
--parameter <parameter name>
Defines the name of the parameter to be changed. Known parameters are:
usePortsToClan
The parameter specifies if dedicated switch ports for Client LAN connection should
be used:
● no
No dedicated switch ports for Client LAN connection are used. Uplinks are
connected to customer corporate LAN and the Client LANs are part of the
uplink or special uplinks for Client LAN connection are configured (see chapter
9.12).
Displaying/Changing Common Network Configuration Parameters Switch Administration
Administration and Operation 173
● yes
Dedicated switch ports for Client LAN connection should be used.
clanPortPerVlan
The parameter depends the way the Client LANs are connected to the corporate
LAN (with the sapgui users). There are two modes:
● no
Use two dedicated switch ports (for redundancy) and use them for all pool
Client LANs (as tagged VLANs onto these two ports).
● yes
Use two dedicated switch ports (for redundancy) for each pool Client LAN
(as untagged VLAN on both ports).
useTxToClan
If the switch group has fiber optic ports (SFP ports) these may be preferred as ports
to the corporate LAN transferring the pool Client-LAN data. To be able to customize
port type the parameter has two values:
● no
The port type will be fiber optic. Be sure to have matching modules for the
SFP ports modules for the SFP ports.
● yes
The port type will be Cat5 twisted pair, commonly abbreviated as TX
useTxUplink
If the switch group has fiber optic ports (SFP ports) these may be preferred as ports
to connect another switch group directly or to use them as uplink ports to the
FlexFrame integrated LAN switch. The parameter has two values:
● no
The port type for uplink will be fiber optic. Be sure to have matching
modules for the SFP ports.
● yes
The port type for uplink will be Cat5 twisted pair, commonly abbreviated as TX
uplinkPortCnt
Defines the count of ports aggregated to the uplink. The uplink channel consists of at
least two ports and may have a maximum of eight ports. The parameter value is the
count of ports.
Switch Administration Adding a Switch Group
174 Administration and Operation
--value <parameter value>
Defines the value of the parameter to be changed. For uplinkPortCnt use a
range between 2 and 8. For all other parameters use yes or 1 and no or 0 as value.
--dryrun
For test purpose you can perform the function without changing the LDAP Database.
Examples
cn1:~ # ff_swgroup_adm.pl –op parameter
Common Network Configuration Parameters
Parameter Name Parameter Value
use ports to Client LAN no
Client LAN port per VLAN yes
use TX ports to Client LAN yes
use TX ports as uplink no
uplink port count 2
timezone Europe/Berlin
POSIX timezone CET
cn1:~ # ff_swgroup_adm.pl --op parameter --parameter useTxToClan --value
yes
Parameter successfully changed at LDAP.
9.8 Adding a Switch Group
To add an entire new switch group, use the ff_swgroup_adm.pl command. The
command adds data to the LDAP database and creates initial configuration which has to
be uploaded manually according to the corresponding "FlexFrame® HW Characteristics
Quickguide ("Hardwaresteckbrief")". Values appropriate to the current configuration
needed for the manual upload are displayed.
One uplink channel is configured which should be connected to an uplink
channel of another switch group or to switches outside the FlexFrame
environment. If more than one uplink channel is needed you can use
ff_swgroup_adm.pl --op add-uplink to add more channels. Afterwards
use ff_swgroup_adm.pl --op gen-config to create a new initial
configuration for upload purposes.
For details about uplink channels and supported topologies see the FlexFrame®
– Network Design and Configuration Guide.
Adding a Switch Group Switch Administration
Administration and Operation 175
Synopsis
ff_swgroup_adm.pl --op add --group <switch group id>
--type <list of switch types>
--name <switch group name>
--passwd <password> [--login <login name>]
[--host <ip host part>[,ip host part]]
[--uplinkportcnt <port count per channel>]
[--uplinkportmedia {tx|fx|10gb}]
[--mgmtswgroup <switch group id>]
[--vpcdomain <vpc domain id>]
[--dryrun]
Options
--op add
Adds a switch group and displays some information about processing steps.
--group <switch group id>
Defines switch group to operate. Within this operation mode used to define the id of
the new switch group.
--type <list of switch types>
Defines the switch types of the switches the switch group will consist of. The types
are comma separated without any white spaces. The types must belong to the same switch family as described above. Call ff_swgroup_adm.pl --help to get a list
of supported switch types.
In case of types building a switch stack the first switch is intended to be the first
member of the switch stack and so on. The maximum number of switches per switch
stack is limited. Be aware that for more switches the stack cabling may be a
bottleneck.
In case of types not building a switch stack exactly two types have to be specified. It
is recommended to use switches with the same number of ports because ports are
needed always in same quantity from both switches.
For switch types providing expansion module slots, slots can immediately be
occupied by appending modules to the switch type separated with colon.
Switch Administration Adding a Switch Group
176 Administration and Operation
--name <switch group name>
Name string to be set as switch group node name.
--passwd <password>
Clear text password to be set at switch group.
--login <login name>
Login name to be set at switch group. Defaults to flexframe.
--host <ip host part>[,ip host part]
Host part to be used to build IP addresses for the control lan network. If this option is
omitted the script uses a free host number to calculate the IP address. Give
additional host parts if necessary.
--uplinkportcnt <port count for uplink channel>
Count of ports to be used for an uplink channel (default: 2).
--uplinkportmedia {tx|fx|10gb}
Media type of ports to be used for the uplink channel (default: depends on switch
type).
--mgmtswgroup <switch group id>
Defines the switch group the switch management interface should be connected to.
The information is necessary if the switches are managed out of band. Call
ff_swgroup_adm.pl --help to get a list of currently configured switch group IDs.
--vpcdomain <vpc domain id>
Defines a unique vPC domain within the network. The information is necessary if the
switches form a type of cluster, e.g. Nexus vPC or Brocade VCS.
Adding an Expansion Module Switch Administration
Administration and Operation 177
--dryrun
For test purpose you can perform the function without changing the LDAP Database
and updating the switchports.
cn1:~ # ff_swgroup_adm.pl --op add \
--type cat3750x-24t:2x10GbE,cat3750x-24t:2x10GbE \
--name swg6 --passwd passwort
If program is aborted by a failure remove leftovers by calling:
ff_swgroup_adm.pl --op rem --group 6
NOTICE: Ctrl-C will be ignored.
update LDAP
..................................................
........
New SwitchGroup 6 was added to LDAP.
Interconnect the Switch Group as noted below:
SwitchID Port
1 TE2 Uplink port of EtherChannel 1
2 TE2 Uplink port of EtherChannel 1
Upload of initial switch configuration has to be done manually.
Follow the instructions of the applicable Quick Start Hardware
FlexFrame for SAP document in the doc/hwinfo section of the Service CD
and use the following data:
Switch Group swg6
IP and Mask: 172.20.13.31 255.255.255.0
Config: scp://root@172.20.13.3//tftpboot/swg6.config
You can find a copy of the message above in
/var/log/FlexFrame/tmp/swgroup_adm-add-6/wiring_of_swgroup6_swg6.txt
9.9 Adding an Expansion Module
A switch may have slots where you can insert expansion modules to get more available
Ports. Use ff_swgroup_adm.pl --op add-module to add the expansion modules
ethernet ports to the FlexFrame configuration as described below.
Switch Administration Removing an Expansion Module
178 Administration and Operation
Synopsis
ff_swgroup_adm.pl --op add-module --group <switch group id>
--switch <switch id>
--slot <slot id>
--module <module type>
Options
--op add-module
Adds an expansion module to a switch.
--group <switch group id>
Defines the switch group to operate.
--switch <switch id>
Defines the switch to operate.
--slot <slot id>
Defines the slot to be used.
--module <module type>
Defines the type of the expansion module to be added.
9.10 Removing an Expansion Module
Before you remove an expansion module physically from a switch use
ff_swgroup_adm.pl --op rem-module to remove the expansion modules ethernet
ports from the FlexFrame configuration as described below.
All ports of the expansion module must be unused.
Synopsis
ff_swgroup_adm.pl --op rem-module --group <switch group id>
--switch <switch id>
--slot <slot id>
Options
--op rem-module
Removes an expansion module from a switch.
Removing a Switch Group Switch Administration
Administration and Operation 179
--group <switch group id>
Defines the switch group to operate.
--switch <switch id>
Defines the switch to operate.
--slot <slot id>
Defines the slot to operate.
9.11 Removing a Switch Group
To remove an entire switch group use the ff_swgroup_adm.pl command. The
command removes data from LDAP database. As a precaution the switch group may no
longer be in use by any FlexFrame component.
All ports of the switchgroup except the ports of one uplink must be unused.
Synopsis
ff_swgroup_adm.pl --op rem --group <switch group id> [--dryrun]
Options
--op rem
Removes named switch group.
--group <switch group id>
Defines switch group to operate. Within this operation mode used to define the id of
the switch group to be removed.
--dryrun
For test purpose you can perform without changing the LDAP Database.
Example
cn1:~ # ff_swgroup_adm.pl --op rem --group 2
update LDAP.
........
Unless any errors are reported disconnect uplink ports
1/49, 2/49
and switch group is removed from FlexFrame environment.
Switch Administration Adding an Uplink to Switch Group
180 Administration and Operation
9.12 Adding an Uplink to Switch Group
Use ff_swgroup_adm.pl --op add-uplink to create an additional uplink at a
switch group. Use the command line arguments to specify options like port count and port
media.
Synopsis
ff_swgroup_adm.pl --op add-uplink --group <switch group id>
[--uplinkportcnt <port count per channel>]
[--uplinkportmedia {tx|fx|10gb}]
[--uplinkchanneltype {UPLINK|CLANUPLINK}]
[--dryrun]
Options
--op add-uplink
Creates a new uplink link aggregate.
--group <switch group id>
Defines the switch group to change.
--uplinkportcnt <port count per channel>
Count of ports to be used for an uplink channel (default: 2). The maximum number of
ports per channel is 8.
--uplinkportmedia {tx|fx|10gb}
Media type of ports to be used for the uplink channel (default: depends on switch
type).
--uplinkchanneltype {UPLINK|CLANUPLINK}
Defines the type of an uplink channel. UPLINK defines a channel with control LAN
and all pool VLANs. CLANUPLINK defines a channel with only the pool client LANs.
If omitted UPLINK is assumed.
--dryrun
For test purpose you can perform the function without changing the LDAP Database.
Example
cn1:~ # ff_swgroup_adm.pl --op add-uplink --group 2 --uplinkportcnt 4
--uplinkportmedia fx
If program is aborted by Ctrl-C or a failure remove left overs
Extend an Uplink of Switch Group Switch Administration
Administration and Operation 181
by calling:
ff_swgroup_adm.pl --op rem-uplink --group 2 --channel 2
update LDAP
....
update switch 2/1 configuration
Notice: Update will take about 1 minute.
..........
New uplink with channel id 2 was created for switch group 2.
It was added to switch group and LDAP.
See below the configured uplink ports and connect them to the peer
switch.
SwitchID / Port
1 / 51 Uplink port of EtherChannel 2 to peer switch
1 / 52 Uplink port of EtherChannel 2 to peer switch
2 / 51 Uplink port of EtherChannel 2 to peer switch
2 / 52 Uplink port of EtherChannel 2 to peer switch
Unless any errors are reported cable switch ports to use the uplink
channel.
Look at "/opt/FlexFrame/network/wiring_of_swgroup2__swg2.txt"
to get a copy of this message.
9.13 Extend an Uplink of Switch Group
Use ff_swgroup_adm.pl --op ext-uplink to extend an existing uplink at a switch
group. New ports are added until given port count is reached.
Synopsis
ff_swgroup_adm.pl --op ext-uplink --group <switch group id>
--channel <uplink channel id>
[--uplinkportcnt <port count per channel>]
[--dryrun]
Options
--op ext-uplink
Expands an existing uplink link aggregate.
--group <switch group id>
Switch Administration Delete an Uplink of Switch Group
182 Administration and Operation
Defines the switch group to change.
--channel <uplink channel id>
Defines the uplink channel of switch group to changed.
--uplinkportcnt <port count per channel>
Count of ports to be used for an uplink channel (default: 2). The maximum number of
ports per channel is 8.
--dryrun
For test purpose you can perform the function without changing the LDAP Database
and updating the switchports.
Example
cn1:~ # ff_swgroup_adm.pl --op ext-uplink --group 2 --channel 2 --
uplinkportcnt 8
update LDAP
.......
update switch 2/1 configuration
Notice: Update will take about 1 minute.
...................
Uplink with channel id 2 of switch group 2 was extended
up to 8 ports. Switch group configuration and LDAP are updated.
See below the new configured uplink ports and connect them to the peer
switch.
SwitchID / Port
2 / 51 Uplink port of EtherChannel 2 to peer switch
3 / 50 Uplink port of EtherChannel 2 to peer switch
3 / 51 Uplink port of EtherChannel 2 to peer switch
4 / 51 Uplink port of EtherChannel 2 to peer switch
Unless any errors are reported cable switch ports to use the new uplink
channel ports. Look at
"/opt/FlexFrame/network/wiring_of_swgroup2__swg2.txt"
to get a copy of this message.
Delete an Uplink of Switch Group Switch Administration
Administration and Operation 183
9.14 Delete an Uplink of Switch Group
To delete an existing uplink of a switch group use operation mode rem-uplink of
ff_swgroup_adm.pl. It will remove the channel with its link aggregate and all
associated ports at switch group.
Synopsis
ff_swgroup_adm.pl --op rem-uplink --group <switch group id>
--channel <uplink channel id>
[--dryrun]
Options
--op ext-uplink
Expands an existing uplink link aggregate.
--group <switch group id>
Defines the switch group to change.
--channel <uplink channel id>
Defines the uplink channel of switch group to removed.
--dryrun
For test purpose you can perform the function without changing the LDAP Database
and updating the switchports.
Example
cn1:~ # ff_swgroup_adm.pl --op rem-uplink --group 2 --channel 2
update LDAP
........
update switch 2/1 configuration
Notice: Update will take about 1 minute.
........................
Uplink with channel id 2 removed at switch group 2.
It was removed from switch group and LDAP.
See below the freed uplink ports.
SwitchID / Port
1 / 51
2 / 51
3 / 49
3 / 50
Switch Administration Migrating Switches of a Switch Group
184 Administration and Operation
3 / 51
4 / 49
4 / 50
4 / 51
Unless any errors are reported switch ports of uplink channel are now
unused.
Look at "/opt/FlexFrame/network/wiring_of_swgroup2__swg2.txt"
to get a copy of this message.
9.15 Migrating Switches of a Switch Group
Switch migration means the replacement of a single or all switches of a switch group by
another switch exemplars. Switch migration may require a downtime. For details see the
vendors manuals.
In principle the processing is as follows.
1. Run ff_swgroup_adm.pl –-op migrate-sw with --dryrun option to check if
the migration is allowed and possible. For details see below.
2. Run ff_save_switch_config.pl to save the configuration for fall back.
3. Check the software version of the new switches. If the versions are not as requested
from the vendor perform an upgrade resp. downgrade.
4. Power off switches of the switch group as necessary.
5. Remove all network cables from the switches you want to replace.
6. Replace the switches.
7. Plug in all network cables into the new switches.
8. Power on the powered off switches. Ensure every switch has its intended ids
(member and/or cluster).
9. Use ff_swgroup_adm.pl –-op migrate-sw without --dryrun option to
accomplish the migration in LDAP.
10. Run ff_swgroup_adm.pl –-op gen-config to get the configuration of the
migrated switch group. Upload the configuration according to the corresponding
"FlexFrame® HW Characteristics Quickguide ("Hardwaresteckbrief")" or check for
necessary changes and perform the necessary configuration change.
The switch group is now ready for use or further configuration.
Adding a Switch Port Configuration Switch Administration
Administration and Operation 185
Synopsis
ff_swgroup_adm.pl --op migrate-sw --group <switch group id>
[--switch <switch id>] --type <switch type>
[--dryrun]
Options
--op migrate-sw
Migrates the type of switches of a switch group.
--group <switch group id>
Defines the switch group to be used.
--switch <switch id>
Defines the switch within the switch group to be migrated. If omitted all switches of
the switch group are migrated to the given switch type.
--type <switch type>
Defines the type the switch should migrate to. Call ff_swgroup_adm.pl without
any parameter to get a list of supported switch types. Migration is allowed according
to the migration table above.
For switch types providing expansion module slots, slots can immediately be
occupied by appending modules to the switch type separated with colon.
--dryrun
For test purpose you can perform the function without changing the LDAP Database.
Example
cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op migrate-sw --group 1 –
switch 2 --type cat3750e-24td
9.16 Adding a Switch Port Configuration
Switch ports are typically directly configured by maintenance tools. But some issues like
configuring ports for gateways, backup or migration systems need a way to do this on a
per port basis. For this type of configuration a special peer type is used. The program
ff_swport_adm.pl is used to configure or remove this type of port configuration.
Synopsis
ff_swport_adm.pl --op add --port <swgroup:switch:port>
[ --10gbit]
Switch Administration Adding a Switch Port Configuration
186 Administration and Operation
--lan <pool:lan[:lan][,pool:lan[:lan]]>
[--native <pool:lan>]
[--desc <description>]
[--dryrun]
Options
--op add
Adds a switch port configuration and displays some information about processing
steps.
--port <swgroup:switch:port>
Defines the switch group, switch and port ID of the port to be used.
--10gbit
Defines the port number used in --port to be the number of a TenGigabitEthernet
port.
--lan <pool:lan[:lan][,pool:lan[:lan]]>
Defines the accessible VLANs. For better readability, a VLAN is specified with its
pool and LAN name. Use only client, server or storage as LAN names. For more
than one LAN per pool the LAN names may be added to the same pool statement.
The VLANs are not restricted to belong to the same pool. To directly add VLAN IDs
not used within any pool, use '#' as pool name and the VLAN ID(s) as LAN(s).
The accessible VLANs. For better readability, a VLAN is specified with its pool and
LAN name. Use only client, server or storage as LAN names. For more than one
LAN per pool the LAN names may be added to the same pool statement. The
VLANs are not restricted to belong to the same pool. To directly add VLAN IDs not used within any pool, use '#' as pool name and the VLAN ID(s) as LAN(s).
If only a single VLAN is configured this is accessible as native VLAN. This means
the data packet contains no VLAN tag. This is the behavior used by a standard
server network interface. For more than one LAN they are configured as tagged. To
define which of them should be used as native VLAN use the option --native.
Examples:
--lan poolA:client:server,poolB:client:server
--lan poolA:client,poolA:server,poolB:client:server
--lan poolA:storage,poolB:storage
--lan poolA:server
--lan '#:417:891'
--lan poolA:server,'#:417:891'
--lan 'poolA:server,#:417:891'
Removing a Switch Port Configuration Switch Administration
Administration and Operation 187
--native <pool:lan>
Use this option to define the native VLAN of the accessible VLANs defined with
option --lan. To directly add VLAN ID not used within any pool, use '#' as pool
name and the VLAN ID as LAN.
Examples:
--native poolA:server
--native '#:417'
--desc <description>
The description is added to configuration of switch port and the LDAP data of the
switch port configuration.
--dryrun
For test purpose you can perform the function without changing the LDAP Database
and updating the switchports.
Example
cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op add --port 1:1:15
--lan ip1:storage:server,'#:4000' --native ip1:storage
Execution may take some minutes. If program is aborted
by Ctrl-C or a failure remove left overs by calling:
ff_swport_adm.pl --op rem --port 1:1:15
update switch 1/1 configuration
Notice: Update will take about 1 minute.
...........+
If not reported any error the port is configured and LDAP is updated
successfully.
9.17 Removing a Switch Port Configuration
To remove a switch port configuration that was previously configured by ff_swport_adm.pl or as external connectivity with the Management Tool, use the
ff_swport_adm.pl command. Other ports are configured with maintenance tools like
ff_an_adm.pl, ff_pool_adm.pl or ff_bx_cabinet_adm.pl.
The switch port configuration will be removed from the switch and the LDAP database.
Synopsis
ff_swport_adm.pl --op rem --port <swgroup:switch:port> [ -10gbit]
Switch Administration Displaying a Switch Port Configuration
188 Administration and Operation
Options
--op rem
Removes the configuration of a switch port and displays some information about
processing steps.
--port <swgroup:switch:port>
Defines the switch group, switch and port ID of the port to be used.
--10gbit
Defines the port number used in --port to be the number of a TenGigabitEthernet
port.
--dryrun
For test purpose you can perform the function without changing the LDAP Database
and updating the switchports.
Example
cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op rem --port 1:1:15
Execution may take some minutes. If program is aborted
by Ctrl-C or a failure remove left overs by calling:
ff_swport_adm.pl --op rem --port 1:1:15
update switch 1/1 configuration
Notice: Update will take about 1 minute.
.............+
If not reported any error the port is unconfigured and LDAP is updated
successfully.
9.18 Displaying a Switch Port Configuration
To display the configuration of a switch port as known by LDAP database in detail, use
the command ff_swport_adm.pl with operation mode list.
Synopsis
ff_swport_adm.pl --op list --port <swgroup:switch:port> [ -10gbit]
Options
--op list
Displaying the Complete Switch Port Configuration Switch Administration
Administration and Operation 189
Displays configuration of the switch port.
--port <swgroup:switch:port>
Defines the switch group, switch and port ID of the port to be used.
--10gbit
Defines the port number used in --port to be the number of a TenGigabitEthernet
port.
Examples
cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op list --port 1:1:4
Switch Port Configuration of
1:1:4 (Switch Group : Switch : Port)
assigned VLAN IDs: 24,25,26
assigned VLAN Names: pool1:client,pool1:server,pool1:storage
native VLAN: 26
Port Peer Type: AN
Peer Node: rx300-1
Display of an unconfigured port:
ERROR: wrong switch port "1:1:8".
Port configuration unknown.
9.19 Displaying the Complete Switch Port Configuration
To display the complete configuration of all switch ports as known by LDAP database in detail, use the command ff_swport_adm.pl with operation mode list-all.
Synopsis
ff_swport_adm.pl --op list-all
Options
--op list-all
Displays configuration of all switch groups.
Switch Administration Displaying the Complete Switch Port Configuration
190 Administration and Operation
Examples
cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op list-all
Switch Port Configuration of
1:1:1 (Switch Group : Switch : Port)
assigned VLAN IDs: 1010,1011,1012
assigned VLAN Names: pool1:client,pool1:storage,pool1:server
native VLAN: 1011
Port Peer Type: GW
Peer Node: sno1apl5p1
Switch Port Configuration of
1:1:2 (Switch Group : Switch : Port)
assigned VLAN IDs: 1010,1011,1012
assigned VLAN Names: pool1:client,pool1:storage,pool1:server
native VLAN: 1011
Port Peer Type: AN
Peer Node: sno1apl2
Switch Port Configuration of
1:1:3 (Switch Group : Switch : Port)
assigned VLAN IDs:
1,90,91,92,93,94,95,1010,1011,1012,1013,1020,1021,1022,1030,1031,1032
assigned VLAN Names: -:default-
vlan,toni:storage,toni:client,toni:server,toni1:storage,toni1:client,ton
i1:server,
pool1:client,pool1:storage,pool1:server,-
:control,pool2:client,pool2:storage,pool2:server,pool3:client,pool3:stor
age,pool3:
server
Port Channel ID: 3
Port Peer Type: SWB
Switch Port Configuration of
1:1:4 (Switch Group : Switch : Port)
assigned VLAN IDs:
1,90,91,92,93,94,95,1010,1011,1012,1013,1020,1021,1022,1030,1031,1032
assigned VLAN Names: -:default-
vlan,toni:storage,toni:client,toni:server,toni1:storage,toni1:client,ton
i1:server,
Moving Device Connection to Core Switch Switch Administration
Administration and Operation 191
pool1:client,pool1:storage,pool1:server,-
:control,pool2:client,pool2:storage,pool2:server,pool3:client,pool3:stor
age,pool3:
server
Port Channel ID: 4
Port Peer Type: SWB
...
9.20 Moving Device Connection to Core Switch
Devices (e.g. application nodes or NAS storage etc) in FlexFrame are usually connected
to the FlexFrame internal (LAN) switch groups. When the configuration grows e.g. if a lot
of servers connected to several switch groups occupy the same NAS system it would be
more convenient if devices e.g. like this NAS system are directly connected to core
switches. Core switches are customer switches the switch groups have uplinks to. Core
switches are not under control of FlexFrame and not represented in LDAP. Devices di-
rectly connected to core switches are still represented in LDAP with a connection to an
imaginary switch group zero. For more detailed information to the core network concept
see also: FlexFrame - Network Design and Configuration Guide.
ff_move_to_core.pl supports the move of device connections to core switches. The
switch group ports occupied from the device to be moved are displayed and if the --doit
option is used the configuration of the affected ports is changed and an update of LDAP
is performed.
Synopsis
ff_move_to_core.pl --help
ff_move_to_core.pl --version
Options
--help
Display usage.
--version
Display script version.
9.20.1 Move Control Center to Core Switch
Synopsis
ff_move_to_core.pl --op cn [--doit]
Switch Administration Moving Device Connection to Core Switch
192 Administration and Operation
Options
--op cn
The switch group ports of both control nodes should be released. The affected ports
are displayed.
--doit
The configuration of the affected ports is changed and an update of LDAP is per-
formed.
Example
cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op cn
9.20.2 Move Client LAN to Core Switch
Synopsis
ff_move_to_core.pl --op clan [--doit]
Options
--op clan
The switch group ports of the client LAN should be released. The affected ports are
displayed
--doit
The configuration of the affected ports is changed and an update of LDAP is per-
formed.
Example
cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op clan
9.20.3 Move NAS System to Core Switch
Synopsis
ff_move_to_core.pl --op nas --name <node name> [--doit]
Moving Device Connection to Core Switch Switch Administration
Administration and Operation 193
Options
--op nas
The switch group ports of a NAS system should be released. The affected ports are
displayed.
--name <node name>
The node name of the NAS system which should be moved.
--doit
The configuration of the affected ports is changed and an update of LDAP is per-
formed.
Example
cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op nas --name filer1
9.20.4 Move Application Node to Core Switch
Synopsis
ff_move_to_core.pl --op an --name <node name> [--doit]
Options
--op an
The switch group ports of an application node should be released. The affected
ports are displayed.
--name <node name>
The node name of the application node which should be moved.
--doit
The configuration of the affected ports is changed and an update of LDAP is per-
formed.
Example
cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op an --name rx300
9.20.5 Move ESX Server to Core Switch
Synopsis
ff_move_to_core.pl --op esx --name <node name> [--doit]
Switch Administration Moving Device Connection to Core Switch
194 Administration and Operation
Options
--op esx
The switch group ports of an application node should be released. The affected
ports are displayed.
--name <node name>
The node name of the application node which should be moved.
--doit
The configuration of the affected ports is changed and an update of LDAP is per-
formed.
Example
cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op esx --name rx300
9.20.6 Move BX Cabinet to Core Switch
Synopsis
ff_move_to_core.pl --op bx --name <cabinet name> [--doit]
Options
--op bx
The switch group ports of a BX cabinet should be released. The affected ports are
displayed.
--name <cabinet name>
The cabinet name of a BX chassis which should be moved.
--doit
The configuration of the affected ports is changed and an update of LDAP is per-
formed.
Example
cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op bx --name bx61
Moving Device Connection to Core Switch SAP System Handling
Administration and Operation 195
10 SAP System Handling
This chapter describes the management of SAP System IDs (so-called SIDs) and their
respective instances within the FlexFrame environment. It further describes how to clone
SIDs as well as their instances into another pool than the one they were installed to.
The tools described below only maintain the LDAP database entries, rather than adding
or removing the data and binaries of the respective SAP systems. These steps need to
be performed manually.
Listing, adding, removing and cloning the above entities in the LDAP server is supported
by two tools, ff_sid_adm.pl and ff_clone_sid.pl. Both scripts will take care of
keeping the data accessed by the operating system's naming service mechanism in sync
with the FlexFrame internal configuration data, both of which reside in the LDAP server.
There is also a Graphical User Interface available which supports the
handling of SIDs in an easy way.
This data should not be manipulated manually.
Preventing port conflicts
Please take into account that depending on the instance number ports are
reserved for the SAP application. That could mean that you conflict with other
non-SAP applications on your nodes (e.g. SAP's ICM uses ports 80nn which
maybe conflicts with an application which uses port 8081). Please refer to
SDN document “TCP/IP Ports Used by SAP Applications” to prevent conflicts.
If a new SAP system is added by ff_sid_adm.pl and the modification of the LDAP
database fail, ff_sid_adm.pl always tries to remove all changes in LDAP which are
made until the failure. This rollback may cause error messages because the rollback
operation does not exactly know where the error occurred (it cannot recognize the failed
command). Those messages can be ignored.
The script ff_sid_adm.pl tries to generate files to support diagnostics. Those files are:
/var/log/FlexFrame/tmp/ff_sid_adm.pl/ldap.<pool>.<SID>.ldif
/ var/log/FlexFrame/tmp/ff_sid_adm.pl/ldap.error.<pool>.<SID>
/ var/log/FlexFrame/tmp/ff_sid_adm.pl//ldap.log.pool>.<SID>
/ var/log/FlexFrame/tmp/ff_sid_adm.pl/ldap.rollback.<pool>.<SID>
In addition to the described SAP System handling in this chapter here there is another
possibility to handle SAP systems using LVM (Landscape Virtualization Management).
SAP System Handling Listing SAP SIDs and Instances
196 Administration and Operation
For more information see manual "Installation and Configuration LVM 1.0" and additional
documentation from SAP.
10.1 Listing SAP SIDs and Instances
Synopsis
ff_sid_adm.pl –-op list-sids [ –-pool <pool name> ]
ff_sid_adm.pl --op list-ids [ --pool <pool name> --sid <SID> ]
ff_sid_adm.pl –-op list-all [ –-pool <pool name> --sid <SID>]
Options
--op list-sids
The SIDs of the specified pool/all pools are displayed with information on SAP
version, database type and database version
--op list-ids
The instance numbers of the specified SID are displayed related to the given
pool/SID or all pools/all SIDs.
--op list-all
Shows extended information about a specific SID (instance types, instances
numbers, used ip@, used hostnames) in a specific pool.
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
The instance numbers of the specified SID are displayed related to the given pool.
Examples
%> ff_sid_adm.pl –-op list-ids –-pool Pan –-sid sht
Configuration Data of SID – instance number and types: SHT (Pool: Pan)
05 scs
DB0 db
07 ci
Listing SAP SIDs and Instances SAP System Handling
Administration and Operation 197
%> ff_sid_adm.pl –-op list-sids –-pool Pan
List of SIDs in pool: Pan
O01 SAP-7.1 Oracle-10
S02 SAP 7.0 MaxDB-76
%> ff_sid_adm.pl –-op list-all –-pool Pan –-sid SHT
Configuration data of SID: SHT (Pool: Pan)
SID global Data:
SAP version specified: SAP-7.0
Database specified: Oracle-10
Instance Data:
Instance type: ci
Instance number: 00
Client-Lan Host-Ip: nnn.nn.nn.nnn
Client-Lan Hostname: cisht
Server-Lan Host-IP: nnn.nn.mm.nnn
Serber-Lan Hostname: cisht-se
SAP System Handling Updating System Configuration Files
198 Administration and Operation
10.2 Updating System Configuration Files
Synopsis
ff_sid_adm.pl --op db2adm --pool <pool name> --sid <SAP system id>
Options
--op db2adm
Updates the system configuration files with DB2 specific data (host name, service
names) for a specific SID.
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
10.3 Adding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)
Synopsis
ff_sid_adm.pl –-op add –-pool <pool name> --sid <SAP system id>
--sapversion {4.6 |6.20 |6.40 |7.0 |7.1 |7.2 |7.3 |7.4 |
7.3-HANA |7.4-HANA | 7.3-NLS | 7.4-NLS}
--db {ORACLE{9|10|11}|
MAXDB{76|77|78|79}|
DB2V{91|95|97|10|105}|
SYBV{157|160} | HANADB-REF}|
SIQ-V16.0
[:<db_loghost_client_ip>|*]:{<db_loghost_server_ip>|*}
--lc {MAXDB76|MAXDB77|MAXDB78|MAXDB79} {<lc_loghost_ip>|*}
--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,...
--user <username1>:<uidnumber1>,<username2>:<uidnumber2>,...
:{<db_loghost_ip>|*}
--sap {ci|pai|app|jc|j|scs|ascs|ers}:<SYSNR>:
:{<loghost_client_ip>|*}
:{<loghost_server_ip>|*}
[--os <instance_type>:<os>,<instance_type>:<os>,...]
Adding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)SAP System Handling
Administration and Operation 199
[--ips <old_ip>:<new_ip>,<old_ip>:<new_ip>,...]
[--db2srv sapdb2<SID>:<port>,DB2_db2<sid>:<port>,
DB2_db2<sid>_1:<port>, DB2_db2<sid>_2:<port>,
DB2_db2<sid>_END:<port>]
[--sapdata <nas system>:/vol/<volume path>]
[--saplog <nas system>:/vol/<volume path>]
ff_sid_adm.pl –-op mod –-pool <pool_name> --sid <SAP system id>
[--os <instance_type>:<os>,<instance_type>:<os>,...]
ff_sid_adm.pl –-op mod –-pool <pool name> --sid <SAP system id>
[--ips <old_ip>:<new_ip>,<old_ip>:<new_ip>,...]
ff_sid_adm.pl –-op del –-pool <pool name> --sid <SAP system id>
[--sysnr <SYSNR>]
Options
--op add
Determines the add operation.
--op mod
Determines the mod operation. This option is only used to modify OS specifications
and exchanges IP adresses of specific SID instances.
--op { del | rem }
Determines the delete operation.
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
--sapversion {4.6|6.20|6.40|7.0|7.1|7.2|7.3|7.4|
7.6 | 7.7 | 7.9 |
7.3-HANA|7.4-HANA }
Specifies the SAP basis version being used. '7.3-HANA' and '7.4-HANA' each
specify a SAP NetWeaver installation which does not have an own classic database
server. The installation uses an existing HANA database (service 'hdb') and the
SAP System HandlingAdding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)
200 Administration and Operation
option 'db' specifies the reference to that HANA database.
‘7.6, ‘7.7’ and ‘7.9’ are only used with LiveCache. The versions refers to the MaxDB
versions used.
--db {ORACLE{10|11} |
MAXDB{76|77|78|79} |
DB2V{95|97|10|105} |
SYBV{157|160} |
SIQ-V16.0 |
HANADB-REF}
Specifies the database type as well as the respective version being used. The
specification of the database type is not case-sensitive. Please take care of the
restrictions given by SAP (e.g. SAP 7.3 only allows at least Oracle11, MaxDB78 or
DB2V97). SIQ-V16.0 is only used with sapversion 7.3-NLS and 7.4-NLS.
{<db_loghost_server_ip>|*}}
The logical host name is used for the database (generated automatically: db<sid>-
se) as well as the IP address for that host name. Use an asterisk if you want it to be
chosen automatically. All the entries need to be specified in a colon separated for-
mat.
You can omit the network part of the IP, e.g. 10.10.12, and specify only the last tupel
of the IP, e.g. 123.
[ {<db_loghost_client_ip>|*}} ]
The logical host name is used for the database tools (generated automatically:
db<sid>) as well as the IP address for that host name. Use an asterisk if you want it
to be chosen automatically. All the entries need to be specified in a colon separated
format. This parameter is optional.
You can omit the network part of the IP, e.g. 10.10.12, and specify only the last tupel
of the IP, e.g. 123.
[ { hana database reference host – client} ]
The virtual client lan hostname of the HANA database (service ‘hdb’).
{ hana database reference host – server}
The virtual server lan hostname of the HANA database (service ‘hdb’)
--lc {MAXDB76|MAXDB77|MAXDB78|MAXDB79}
Specifies the database type as well as the respective version being used. The
specification of the database type is not case-sensitive.
{<lc_loghost_ip>|*}}
Adding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)SAP System Handling
Administration and Operation 201
The logical host name is used for the database (generated automatically: lc<sid>-
se) as well as the IP address for that host name. Use an asterisk if you want it to be
chosen automatically. All the entries need to be specified in a colon separated for-
mat.
You can omit the network part of the IP, e.g. 10.10.12, and specify only the last tupel
of the IP, e.g. 123.
--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,...
--user <username1>:<uidnumber1>,<username2>:<uidnumber2>,...
User and group enable specially selected user numbers and group numbers to be
assigned to SAP users and SAP groups respectively. In this case a check is made to
see whether the user or group has already been defined for the DB system involved.
A user or group is created only if they do not already exist. For example, a group dba which already exists cannot be assigned a group number which deviates from
the default value.
--sap {ci|pai|app|jc|j|scs|ascs|ers}:<SYSNR>:
{<loghost_client_ip>|*}
{<loghost_server_ip>|*}
Specifies an SAP instance (optionally multiple of those) through its type (ci, pai,
app, jc, j, scs, ascs, ers), its SAP system number, the logical host name
(generated automatically: <type><sid>) in the client network, the respective IP
address, the logical host name (generated automatically: <type><sid>-se) in the
server network and the respective IP address. Again, the IP addresses can be
replaced with asterisks in order to have them chosen automatically.
With support of SAP 7.4 the CI specification is replaced by PAI specification. With
SAP 7.4 we do not allow specification of CI furthermore.
ERS is only supported with --sapversion 7.0 up or higher. For Enqueue Repli-
cated Servers (ERS) the asterisk should be used because we do not need an IP
address from the client and server LAN (given IP addresses will be ignored).
All the entries need to be specified in a colon separated format.
With SAP 7.1 up SAP systems do not know instance type jc for JAVA systems.
Therefore the hostname used changed from jc<sid> to j<sysnr><sid> and from
jc<sid>-se to j<sysnr><sid>-se.
ff_sid_adm.pl still requires the type jc used in option —-sap to make a distinc-
tion between central instance and application instance. Please take into account the
different hostname specifications depending on SAP version used.
--sysnr <SYSNR>
Removes a specific SAP instance instead of the entire system (SID).
SAP System HandlingAdding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)
202 Administration and Operation
--os <instance_type>:<os>,<instance_type>:<os>,...]
Specifies the OS type for the given SID and/or their instances.
instance_type::= {default|ci|app|jc|j|scs|ascs|ers}
os::= {SLES-11.x86_64}
In combination with the –-add option, default:<os> sets the operating system of
the SID itself and all their instances which no own operating system is assigned to.
In combination with the –-mod option, default:<os> sets the operating system of
the SID only. The specifications of their instances are not changed.
Examples:
default:SLES-11.x86_64 - all instances are set to SLES-11.x86_64
The consistence of the given instance type/OS combinations is not checked
by FlexFrame. For a list of allowed combinations see SAP note 1067221.
--ips <old_ip>:<new_ip>,<old_ip>:<new_ip>,...]
Allows to exchange the IP addresses of specific SID instances. This option is only
used with —-op mod. You have to specify the full IP you want to exchange.
ff_sid_adm.pl searches for the specific instances within a SID and exchanges all
corresponding entries in LDAP concerned by that request.
Please pay attention that you make critical changes within your configuration.
We strongly recommend to save a backup of the LDAP database before changing
any IP addresses.
--db2srv sapdb2<SID>:<port>,DB2_db2<sid>:<port>,
DB2_db2<sid>_1:<port>,DB2_db2<sid>_2:<port>,
DB2_db2<sid>_END:<port>
This option is only used for DB2, otherwise it is ignored. You can specify a list of
services you need. The services are written to LDAP and the pool specific /etc/services.
--sapdata <nas system>:/vol/<volume path>
Allows to request an implicit call of ff_sid_mnt_adm.pl to relocate location to store
sapdata. It is expected that the requested volume already exists. The option is
valid only with services using a database. <nas system> is the NAS filer node name
ending with ‘-st’.
This option is not used with installations using a reference to a HANA database.
--saplog <nas system>:/vol/<volume path>
Allows to request an implicit call of ff_sid_mnt_adm.pl to relocate location to store
saplog. It is expected that the requested volume already exists. The option is
Adding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)SAP System Handling
Administration and Operation 203
valid only with services using a database. <nas system> is the NAS filer node name
ending with ‘-st’.
This option is not used with installations using a reference to a HANA database.
Examples
Adding a SID with one Central Instance:
control1:~ # ff_sid_adm.pl –-op add –-sid SHT –-pool otto --sapversion 6.40
--db ORACLE9:192.168.1.1 --sap ci:00:\*:\*
Adding an instance to an existing SAP System:
control1:~ # ff_sid_adm.pl –-op add –-sid SHT –-pool otto --sapversion 6.40
--sap app:01:\*:\*
Adding a SID for LiveCache:
control1:~ # ff_sid_adm.pl --op add --pool test --sid LCA --sapversion 7.0 --lc MAXDB76:\*Adding a
SID with ERS support:
control1:~ # ff_sid_adm.pl --op add --pool pool1 –-sid S04 --sapversion 7.0
--db ORACLE10:dbs04-se:10.10.12.100 --sap ers:12:\*:\*
--sap ci:13:10.10.11.102:10.10.12.102
Adding a SID with operation system:
control1:~ # ff_sid_adm.pl --op add --pool pool1 –-sid OS1 --sapversion 7.0
--db ORACLE10:10.10.12.159
--sap ci:57:10.10.10.157:10.10.12.157
--sap ascs:55:10.10.10.155:10.10.12.155
--os default:SLES-11.x86_64,ascs:SLES-11.x86_64
Adding a SID with DB2 services:
control1:~ # ff_sid_adm.pl --op add --pool pool1 –-sid LB5 --sapversion 7.4
--db DB2V91:10.10.12.159
--sap pai:57:10.10.10.157:10.10.12.157
--sap ascs:55:10.10.10.155:10.10.12.155
--db2srv sapdb2LB5:60000,DB2_db2lb5:60001,DB2_db2lb5_1:60002,DB2_db2lb5_2:60003,
DB2_db2lb5_END:60004
Adding a SID (SID specific volumes for sapdata/saplog):
SAP System HandlingAdding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)
204 Administration and Operation
control1:~ # ff_sid_adm.pl --op add --pool pool1 –-sid JA0 --sapversion 7.0
--db MAXDB77:158
--sap jc:57:157
--sap j:58:156
--sap ascs:55:155
--sapdata filer-1-st:/vol/JA0_data
--saplog filer-1-st:/vol/JA0_logs
Adding a SID (old syntax for hosts):
control1:~ # ff_sid_adm.pl --op add --pool pool1 –-sid JA0 --sapversion 7.0
--db MAXDB77:10.10.12.158
--sap jc:57:10.10.10.157:10.10.12.157
--sap j:58:10.10.10.156:10.10.12.156
--sap ascs:55:10.10.10.155:10.10.12.155
Adding a SID with old syntax (SAP 7.1, JAVA):
control1:~ # ff_sid_adm.pl --op add --pool pool1 –-sid JA1 --sapversion 7.1
--db ORACLE10:10.10.12.158
--sap jc:57:10.10.10.157:10.10.12.157
--sap j:58:10.10.10.156:10.10.12.156
--sap ascs:55:10.10.10.155:10.10.12.155
Adding a SID with reference to HANA database:
control1:~ # ff_sid_adm.pl -–op add –-pool pool1 -–sid HRF –-sapversion 7.4-HANA
--db HANAREF-DB:hdb01ha1-001:hdb01ha1-001-se
--sap ascs:20:120:120
--sap pai:21:121:121
Removing an entire SID (including its instances):
control1:~ # ff_sid_adm.pl –-op del –-sid SHT –-pool otto
Removing a single instance of a SID:
control1:~ # ff_sid_adm.pl –-op del –-sid SHT –-pool otto –-sysnr 01
Removing SAP SIDs and Instances SAP System Handling
Administration and Operation 205
Modifying IP adresses:
control1:~ # ff_sid_adm.pl –-op mod –-sid SHT –-pool otto
-–ips 10.10.10.123:10.10.10.132,10.10.12.123:10.10.12.132
Modifying OS version of the SID entry:
control1:~ # ff_sid_adm.pl –-op mod –-sid SHT –-pool otto
-–os "default:SLES-11.x86_64"
10.4 Removing SAP SIDs and Instances
Synopsis
ff_sid_adm.pl –-op del –-pool <pool name> --sid <SAP system id>
ff_sid_adm.pl –-op [ del | rem ] –-pool <pool name> \
--sid <SAP system id> [--sysnr <SYSNR>]
Options
--op { del | rem }
Determines the delete operation.
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
--sysnr <SYSNR>
Removes a specific SAP instance instead of the entire system (SID).
Examples
Removing an entire SID (including all instances):
%> ff_sid_adm.pl –-op del –-sid SHT –--pool otto
Removing an specifc instance:
%> ff_sid_adm.pl –-op del –-sid SHT –--pool otto –-sysnr 01
SAP System Handling Adding/Removing SAP SIDs (add-on services)
206 Administration and Operation
10.5 Adding/Removing SAP SIDs (add-on services)
10.5.1 BOBJ – Business Objects
Synopsis
ff_sid_adm.pl –-op add –-pool <pool name> --sid <SAP system id> \
--sapversion {3.1 | 3.2| 4.0 | 4.1} \
--bobj <client lan hostip> --os <spec>
ff_sid_adm.pl –-op del –-pool <pool name> --sid <SAP system id>
Options
--op add | del
Determines the operation, add or delete.
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
--sapversion {3.1|3.2|4.0|4.1}
Specifies the SAP basis version being used.
--bobj <client lan hostip>
Specifies the hostip in the client lan for BOBJ service.
--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,...
--user <sid>adm:<uidnumber>
user and group enable specially selected user numbers and group numbers to be
assigned to SAP users and SAP groups respectively. In this case a check is made to
see whether the user or group has already been defined for the SAP system
involved. A user or group is created only if they do not already exist. For example, a
group sapsys which already exists cannot be assigned a group number which
deviates from the default value.
--bobj <client lan hostip>
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 207
The IP address for the clien lan host name (generated internally: bobj<sid>). Use
an asterisk if you want it to be chosen automatically.
You can omit the network part of the IP, eg. 10.10.12, and specify only the last
tupel of the IP, e.g. 123.
--os <instance_type>:<os>,<instance_type>:<os>,...]
Specifies the OS type for the given SID and/or their instances.
instance_type::= {default|bobj}
os::= {SLES-11.x86_64}
In combination with the –-add option, default:<os> sets the operating system of the
SID itself and all their instances which no own operating system is assigned to.
In combination with the –-mod option, default:<os> sets the operating system of the
SID only. The specifications of their instances are not changed.
The consistence of the given instance type/OS combinations is not checked
by FlexFrame. For a list of allowed combinations see SAP note 1067221.
Examples
Adding a SID with BOBJ service:
control1:~ # ff_sid_adm.pl –-op add –-sid bob –--pool otto --sapversion 3.2 \
––bobj 160
Removing a BOBJ service:
control1:~ # ff_sid_adm.pl –-op del –-sid bob –--pool otto
10.5.2 Content Server (CMS)
Adding a CMS service at first time means that you need to specify all the instances
required by CMS (database and cms client spec).
CMS service consists of
a database instance
a client instance
Synopsis
ff_sid_adm.pl –-op add –-pool <pool name> --sid <SAP system id> \
--db MaxDB<version>:{<db_loghost_ip>|\*} \
--cms <client lan_ip>|\*} \
SAP System Handling Adding/Removing SAP SIDs (add-on services)
208 Administration and Operation
--sapversion 6.50 | 6.40 [--os <spec> ] \
[ --sapdata <nas_system>:/>volume path> ]
[ --saplog <nas_system>:/>volume path> ]
ff_sid_adm.pl –-op del –-pool <pool name> --sid <SAP system id>
Options
--op add | del
Determines the operation, add or delete.
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
--sapversion {6.50 | 6.40}
Specifies the SAP basis version being used. Version 6.40 is not supported further by
SAP.
--db MaxDb76|77|78:79:{<db loghost ip> | \*}
Specifies the database version being used and the server lan host IP
--cms <client lan hostip>
Specifies the hostip in the client lan for BOBJ service.
{<db_loghost_ip>|*}}
The logical host name (generated internally: db<sid>-se) is used for the database as
well as the IP address for that host name. Use an asterisk if you want it to be chosen
automatically. All the entries need to be specified in a colon separated format.
You can omit the network part of the IP, e.g. 10.10.12, and specify only the last
tupel of the IP, e.g. 123.
--cms <client lan hostip>
The IP address for that host name (generated internally: cms<sid>). Use an asterisk
if you want it to be chosen automatically.
You can omit the network part of the IP, e.g. 10.10.12, and specify only the last
tupel of the IP, e.g. 123.
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 209
--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,...
--user <username1>:<uidnumber1>,<username2:uidnumber2>,…
user and group enable specially selected user numbers and group numbers to be
assigned to SAP users and SAP groups respectively. In this case a check is made to
see whether the user or group has already been defined for the SAP system
involved. A user or group is created only if they do not already exist. For example, a
group sapsys which already exists cannot be assigned a group number which
deviates from the default value.
--os <instance_type>:<os>,<instance_type>:<os>,...]
Specifies the OS type for the given SID and/or their instances.
instance_type::= {default|bobj}
os::= {SLES-11.x86_64}
In combination with the –-add option, default:<os> sets the operating system
of the SID itself and all their instances which no own operating system is assigned
to.
--sapdata <nas system>:/vol/<volume path>
Allows to request an implicit call of ff_sid_mnt_adm.pl to relocate location to store
sapdata. It is expected that the requested volume already exists. The option is valid
only with services using a database. <nas system> is the NAS filer node name
ending with ‘-st’.
--saplog <nas system>:/vol/<volume path>
Allows to request an implicit call of ff_sid_mnt_adm.pl to relocate location to store
saplog. It is expected that the requested volume already exists. The option is valid
only with services using a database. <nas system> is the NAS filer node name
ending with ‘-st’.
In combination with the –-mod option, default:<os> sets the operating system
of the SID only. The specifications of their instances are not changed.
The consistence of the given instance type/OS combinations is not checked
by FlexFrame. For a list of allowed combinations see SAP note 1067221.
Examples
Adding a SID with CMS service:
control1:~ # ff_sid_adm.pl –-op add –-sid cmx –--pool otto --sapversion 6.40 \
--db MaxDB76:170 \
SAP System Handling Adding/Removing SAP SIDs (add-on services)
210 Administration and Operation
––cms 170 ]
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 211
Adding a CMS client instance:
control1:~ # ff_sid_adm.pl –-op add –-sid cmx --pool otto --sapversion \
6.40 ––cms 170 ]
Removing a CMS Service:
control1:~ # ff_sid_adm.pl –-op rem –-sid cmx –-pool otto
Removing the CMS client instance:
control1:~ # ff_sid_adm.pl –-op rem –-sid cmx –-pool otto –sysnr CMS
10.5.3 MDM – Master Data Management
Adding a MDM service for the first time means that you need at least to specify the
database instance.
MDM service contains:
a database instance
a number of services of type 'mds'
a number of services of type 'mdss'
a number of services of type 'mdis'
Synopsis
ff_sid_adm.pl –-op add –-pool <pool name> --sid <SAP system id> \
--db {ORACLE{10|11}|
MAXDB{77|78|79}|
SYBV{157|160}
DB2V{95|97|10|105}}:{<db_loghost_ip>|*}
--mdm mds:<nr>:{<client lan_ip>|\*}>:{<server
lan_ip>|\*} \
--mdm mdss:<nr>:{<client lan_ip>|\*}>:{<server
lan_ip>|\*} \
--mdm mdis:<nr>:{<client lan_ip>|\*}>:{<server
lan_ip>|\*} \
--sapversion 7.1 [--os <spec> ] \
[ --sapdata <nas_system>:/<volume path> ] \
[ --saplog <nas_system>:/<volume path> ] \
ff_sid_adm.pl –-op del –-pool <pool name> \
SAP System Handling Adding/Removing SAP SIDs (add-on services)
212 Administration and Operation
--sid <SAP system id>
--op add
Determines the add operation.
--op mod
Determines the mod operation. This option is only used to modify OS specifications
and exchanges IP adresses of specific SID instances.
--op del
Determines the delete operation.
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
--sapversion {7.1}
Specifies the SAP basis version being used.
--db {ORACLE{10|11}|MAXDB{77|78|79}|DB2V{95|97|10|105}}
Specifies the database type as well as the respective version being used. The
specification of the database type is not case-sensitive.
--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,...
--user <username1>:<uidnumber1>,<username2>:<uidnumber2>,...
user and group enable specially selected user numbers and group numbers to be
assigned to SAP users and SAP groups respectively. In this case a check is made to
see whether the user or group has already been defined for the DB system involved.
A user or group is created only if they do not already exist. For example, a group
dba which already exists cannot be assigned a group number which deviates from
the default value.
{<db_loghost_ip>|*}}
The logical host name is used for the database (generated automatically: db<sid_-
se) as well as the IP address for that host name. Use an asterisk if you want it to be
chosen automatically. All the entries need to be specified in a colon separated for-
mat.
You can omit the network part of the IP, e.g. 10.10.12., and specify only the last
tupel of the IP, e.g. 123.
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 213
--mdm
{mds|mdss|mdis}:<SYSNR>:{<loghost_client_ip>|*}:{<loghost_server_i
p>|*}
Specifies an SAP instance (optionally multiple of those) through its type, its SAP
system number, the client network IP address, the the server network IP address.
Again, the IP addresses can be replaced with asterisks in order to have them cho-
sen automatically.
All the entries need to be specified in a colon separated format.
FlexFrame expects that the syntax of loghost-client follows the rule <type>sid and
loghost-server follows <type>sid-se.
--sysnr <SYSNR>
Removes a specific SAP instance instead of the entire system (SID).
--os <instance_type>:<os>,<instance_type>:<os>,...]
Specifies the OS type for the given SID and/or their instances.
instance_type::= {default|mds|mdss|mdis}
os::= {SLES-11.x86_64}
In combination with the –-add option, default: <os> sets the operating system of the
SID itself and all their instances which no own operating system is assigned to.
In combination with the –-mod option, default:<os> sets the operating system of the
SID only. The specifications of their instances are not changed.
--sapdata <nas system>:/vol/<volume path>
Allows to request an implicit call of ff_sid_mnt_adm.pl to relocate location to store
sapdata. It is expected that the requested volume already exists. The option is valid
only with services using a database. <nas system> is the NAS filer node name
ending with ‘-st’.
--saplog <nas system>:/vol/<volume path>
Allows to request an implicit call of ff_sid_mnt_adm.pl to relocate location to store
saplog. It is expected that the requested volume already exists. The option is valid
only with services using a database. <nas system> is the NAS filer node name
ending with ‘-st’.
The consistence of the given instance type/OS combinations is not checked
by FlexFrame. For a list of allowed combinations see SAP note 1067221.
--ips <old_ip>:<new_ip>,<old_ip>:<new_ip>,...]
SAP System Handling Adding/Removing SAP SIDs (add-on services)
214 Administration and Operation
Allows to exchange the IP addresses of specific SID instances. This option is only used with —-op mod. You have to specify the full IP you want to exchange.
ff_sid_adm.pl searches for the specific instances within a SID and exchanges all
corresponding entries in LDAP concerned by that request.
Please pay attention that you make critical changes within your configuration. So we
strongly recommend you to take a backup of LDAP database before changing IP
addresses.
Examples
Adding a SID with MDM services:
control1:~ # ff_sid_adm.pl –-op add –-sid mdm –--pool otto --sapversion 7.1 \
--db MaxDB76:170
––mdm mds:01:171:171
––mdm mdss:02:172:172
––mdm mdis:03:173:173
Adding an instance of type mdss to an existing MDM SID:
control1:~ # ff_sid_adm.pl –-op add –-sid mdm –--pool otto --sapversion 7.1 \
––mdm mdss:04:174:174
Removing an specific instance of a MDM SID:
control1:~ # ff_sid_adm.pl –-op rem –-sid mdm –--pool otto –sysnr 02
Removing a SID with MDM services:
control1:~ # ff_sid_adm.pl –-op rem –-sid mdm –--pool otto
10.5.4 SMD – Solution Manager Diagnostics
Current SAP releases using SAP Solution Manager and SMD agents. We
distinct the type of agentsA service based agent as known from SAP 7.1 up.
Each services (e.g. CI) needs its own agent
A host based agend as introduced with SMD 7.3 SP02. For each physical host
one agent is defined
The first type should be found in older installations. It is recommended to specify a SID
with multiple SMD instances, e.g. one SMD SID with one instance per monitored service
within a NetWeaver SID.
The second type should be the choice in newer configurations. The property 'host based'
has to be enabled at customer side. If ‘host based’ is enabled you can only define a SMD
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 215
SID with one instance assigned to the monitored physical host. Otherwise the SMD agent
is configured like an service based type.
A host based SMD agent is assigned to a physical host. If you are moving an Application
Node from one pool to another, or if you are using pool independent spares, you have to
take care that your SMD configuration is consistent. There is no possibility to a complete
automatism.
The SMD SID should be specified before starting NetWeaver installation because during
installation procedure you are asked for SMD agent SID.
Synopsis
ff_sid_adm.pl –-op add –-pool <pool name> --sid <SAP system id> \
--smd { <nr>:<client lan_monitored host> |
<nr>:<application node name> \
--sapversion [ 7.1 | 7.3-SRV | 7.3-HOST ]
[--os <spec> ]
ff_sid_adm.pl --op add --pool <pool name> --sid <SAP system id> \
--smd <nr>:<db server_lan monitored host> \
--sapversion [ 7.1 | 7.3-SRV | 7.3-HOST ] [--os
<spec> ]
ff_sid_adm.pl --op add --pool <pool name> --sid <SAP system id> \
--smd <nr>:<client lan application node> \
--sapversion [ 7.3S-SRV | 7.3-HOST ] [--os <spec>
]
ff_sid_adm.pl --op mod --pool <pool name> --sid <SAP system id> \
--smd <nr>:<client lan application node>
ff_sid_adm.pl –-op del –-pool <pool name> --sid <SAP system id>
--op add
SAP System Handling Adding/Removing SAP SIDs (add-on services)
216 Administration and Operation
Determines the add operation.
--op del
Determines the delete operation.
--op mod
Allows to modify physical hostname referenced with host-based
SMD spec. For modification use option ‘—smd’ as used with
opcode ‘add’
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
--sapversion {7.1}
Specifies the SAP basis version being used.
--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,...
--user <username1>:<uidnumber1>,<username2>:<uidnumber2>,...
user and group enable specially selected user numbers and group numbers to be
assigned to SAP users and SAP groups respectively. In this case a check is made to
see whether the user or group has already been defined for the DB system involved.
A user or group is created only if they do not already exist. For example, a group
dba which already exists cannot be assigned a group number which deviates from
the default value.
--smd <SYSNR>:[ <client lan name – monitored host> |
<db server lan name - monitored host> |
<client lan application node>
Specifies the name of the host which should be monitored. The name of the host
depends on the SAP service type. With SMD 7.1 and 7.3 we have a service-based
monitoring. The following services types can be monitored
● app – app<nr><sid>
● j – j<nr><sid>
● ci – ci<sid>
● pai - pai<sid>
● jc – jc<sid> (less than SAP 7.1), j<nr><sid> (SAP > 7.1)
● ascs – ascs<SID>
● db - db<sid>-se
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 217
● wd – wd<nr><sid>
Parameteroption ‘SYSNR’ specifies the instance number of the SMD agent
installation..
--sysnr <SYSNR>
Removes a specific SAP instance instead of the entire SID .
--os <instance_type>:<os>,<instance_type>:<os>,...]
Specifies the OS type for the given SID and/or their instances.
instance_type::= {default|smd}
os::= {SLES-11.x86_64}
In combination with the –-add option, default:<os> sets the operating system of the
SID itself and all their instances which no own operating system is assigned to.
In combination with the –-mod option, default:<os> sets the operating system of the
SID only. The specifications of their instances are not changed.
The consistence of the given instance type/OS combinations is not checked
by FlexFrame. For a list of allowed combinations see SAP note 1067221.
Examples
Adding a SID with SMD services (service-based monitoring):
control1:~ # ff_sid_adm.pl –-op add –-sid SMD –-pool otto \
--sapversion 7.1 \
––smd 01:j12abc smd 02:cixyz
Adding an instance of type mdss to an existing SMD SID:
control1:~ # ff_sid_adm.pl –-op add –-sid SMD –-pool otto \
--sapversion 7.1 \
––smd 04:app12xyz
Adding a SID for host-based monitoring
control1:~ # ff_sid_adm.pl –-op add –-sid SMD –--pool otto \
--sapversion 7.3S-HOST \
––smd 97:appnode1
Removing an specific instance of a SMD SID:
SAP System Handling Adding/Removing SAP SIDs (add-on services)
218 Administration and Operation
control1:~ # ff_sid_adm.pl –-op rem –-sid mdm –-pool otto –sysnr 02
Removing a SID with SMD services:
control1:~ # ff_sid_adm.pl –-op rem –-sid mdm –-pool otto
10.5.5 TREX (Search and Classification Service)
Synopsis
ff_sid_adm.pl –-op add –-pool <pool name> --sid <SAP system id> \
--trx <nr>:<client lan_ip>>:<server
lan_ip> \
--sapversion 7.1 [--os <spec> ]
ff_sid_adm.pl –-op del –-pool <pool_name> --sid <SAP_system_id>
--op add
Determines the add operation.
--op del
Determines the delete operation.
--op mod
Determines the mod operation. This option is only used to modify OS specifications
and exchanges IP adresses of specific SID instances.
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
--sapversion {7.1}
Specifies the SAP basis version being used.
--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,...
--user <username1>:<uidnumber1>,<username2>:<uidnumber2>,...
user and group enable specially selected user numbers and group numbers to be
assigned to SAP users and SAP groups respectively. In this case a check is made to
see whether the user or group has already been defined for the DB system involved.
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 219
A user or group is created only if they do not already exist. For example, a group
dba which already exists cannot be assigned a group number which deviates from
the default value.
--smd <SYSNR>:<client lan hostname – monitored host>>
The logical host name is used for the database (generated automatically:
trx<nr><sid>, trx<nr><sid-se>) as well as the IP address for that host
name. Use an asterisk if you want it to be chosen automatically. All the entries need
to be specified in a colon separated format.
You can omit the network part of the IP, e.g. 10.10.12. and specify only the last
tupel of the IP, e.g. 123.
--sysnr <SYSNR>
Removes a specific SAP instance instead of the entire system (SID).
--os <instance_type>:<os>,<instance_type>:<os>,...]
Specifies the OS type for the given SID and/or their instances.
instance_type::= {default|trx}
os::= {SLES-11.x86_64}
In combination with the –-add option, default:<os> sets the operating system of the
SID itself and all their instances which no own operating system is assigned to.
In combination with the –-mod option, default:<os> sets the operating system of the
SID only. The specifications of their instances are not changed.
The consistence of the given instance type/OS combinations is not checked
by FlexFrame. For a list of allowed combinations see SAP note 1067221.
--ips <old_ip>:<new_ip>,<old_ip>:<new_ip>,...]
Allows to exchange the IP addresses of specific SID instances. This option is only
used with —-op mod. You have to specify the full IP you want to exchange.
ff_sid_adm.pl searches for the specific instances within a SID and exchanges all
corresponding entries in LDAP concerned by that request.
Please pay attention that you make critical changes within your configuration. So we
strongly recommend you to take a backup of LDAP database before changing IP
addresses.
Examples
Adding a SID with TRX services:
control1:~ # ff_sid_adm.pl –-op add –-sid TR1 –--pool otto --sapversion 7.1 \
––trx 01:201:201
SAP System Handling Adding/Removing SAP SIDs (add-on services)
220 Administration and Operation
––trx 02:200:200
Adding an instance of type mdss to an existing TRX SID:
control1:~ # ff_sid_adm.pl –-op add –-sid TR1 –--pool otto --sapversion 7.1 \
––smd 04:202:202
Removing an specific instance of a TRX SID:
control1:~ # ff_sid_adm.pl –-op rem –-sid tr1 –--pool otto –sysnr 02
Removing a SID with TRX services:
control1:~ # ff_sid_adm.pl –-op rem –-sid tr1 –--pool otto
10.5.6 WebDispatcher
Synopsis
ff_sid_adm.pl --op add --pool <pool name> --sid <SAP system id> \
--wd <nr>:<client lan_ip>:<server lan_ip>> \
--sapversion { 7.1 | 7.2 | 7.3 | 7.4 } [--os <spec>]
ff_sid_adm.pl --op del --pool <pool_name> --sid <SAP_system_id>
--op add
Determines the add operation.
--op del
Determines the delete operation.
--op mod
Determines the modify operation. This option is only used to modify OS
specifications and exchanges IP adresses of specific SID instances.
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
--sapversion {7.1| 7.2 | 7.3 | 7.4 }
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 221
Specifies the SAP basis version being used.
--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,...
--user <username1>:<uidnumber1>,<username2>:<uidnumber2>,...
User and group enable specially selected user numbers and group numbers to be
assigned to SAP users and SAP groups respectively. In this case a check is made to
see whether the user or group has already been defined for the DB system involved.
A user or group is created only if they do not already exist. For example, a group
dba which already exists cannot be assigned to a group number which deviates from
the default value.
--wd <SYSNR>:<client_lan_ip>:<server lan_ip>
The logical host name is used for the instance (generated automatically:
wd<nr><sid>) as well as the IP address for that host name. Use an asterisk if you
want it to be chosen automatically. All the entries need to be specified in a colon
separated format.
You can omit the network part of the IP, e.g. 10.10.12. and specify only the last tupel
of the IP, e.g. 123.
--sysnr <SYSNR>
Removes a specific SAP instance instead of the entire system (SID).
--os <instance_type>:<os>,<instance_type>:<os>,...]
Specifies the OS type for the given SID and/or their instances.
instance_type::= {default|wd}
os::= {SLES-11.x86_64}
In combination with the --add option, default:<os> sets the operating system of
the SID itself and all their instances which no own operating system is assigned to.
In combination with the --mod option, default:<os> sets the operating system of
the SID only. The specifications of their instances are not changed.
--ips <old_ip>:<new_ip>,<old_ip>:<new_ip>,...]
Allows to exchange the IP addresses of specific SID instances. This option is only
used with --op mod. You have to specify the full IP you want to exchange.
ff_sid_adm.pl searches for the specific instances within a SID and exchanges all
corresponding entries in LDAP concerned by that request.
Please pay attention that you make critical changes within your configuration. So we
strongly recommend you to take a backup of LDAP database before changing IP
addresses.
SAP System Handling Adding/Removing SAP SIDs (add-on services)
222 Administration and Operation
Examples
Adding a SID with WD services:
control1:~ # ff_sid_adm.pl --op add --sid WD1 --pool otto \
--sapversion 7.2 \
--wd 01:200:200 --wd 02:201:201
Removing a specific instance of a WebDispatcher SID:
control1:~ # ff_sid_adm.pl --op rem --sid wd1 --pool otto -sysnr 02
Removing a SID with WebDispatcher services:
control1:~ # ff_sid_adm.pl --op rem --sid wd1 --pool otto
10.5.7 HANA database (service hdb)
We support service ‘hdb’ HANA database in Single Node Configurations. Please refer to
the corresponding hardware restrictions specified by SAP.
At now we also support HANA System Replication feature in a Single Node Environment.
A HANA database requires specific volumes for shared administration data and the
database data- and logfiles. Please refer to the preconditions specified by SAP
concerning the file system structure.
FlexFrame Orchestrator only supports HANA single node configurations.
The specification of a hdb service requires
The specification of the SID using ff_sid_adm.pl
Availability of specific filer volumes used for HANA
o SID specific volume for shared data (/hana/shared)
o Instance- and Node-specifc volumes for data (/hana/data ) and logs,
(/hana/log). Each instance and node can use own volumes for data and
logs.
All volumes are assigned to a specific HANA database service and volumes
should not be used over multiple pools
Specifying HANA volumes for SID using ff_sid_mnt_adm.pl
Creation of file system structure using ff_setup_sid_folder.sh. The script expects
that all HANA volumes (shared/data/logs) are assigned to SID. Otherwise the
scripts aborts.
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 223
Please refer also to SAP Installation how to specify and install SAP HANA Database
service.
Please make sure to have an installation of the (maybe) required FlexFrame Orchestrator
software licenses.
With HANA System Replication you have
A SID up to 9 times within a pool
Each replication instance has its own volumes for shared/data/log
The instance number has to be equal in all replicas
The user-/group environment has to be equal
A replication instance is assigned to a specific poolgroup in which it can run
The specification in FlexFrame is quite the same as HANA without HSR.
Synopsis
ff_sid_adm.pl --op add --pool <pool name> --sid <SAP system id> \
--db HANA | R-HANA\
--hdb <SYSNR>:<client lan_ip>:<server lan \
ip>:<nodeid> \
--hdbtype { OLTP | OLAP } \
--sapversion { 1.0 | R-1.0 } [--os <spec> ]
[ --replid < replication id> ]
[ --runingroup <assigned poolgroup for operating> ]
ff_sid_adm.pl --op del --pool <pool_name> --sid <SAP_system_id> \
[ --replid <replication id> ]
--op add
Determines the add operation.
--op { del | rem }
Determines the delete operation.
--op mod
Determines the modify operation. This option is only used to modify OS
specifications and exchanges IP adresses of specific SID instances or modifies/add
properties of HANA System Replication.
SAP System Handling Adding/Removing SAP SIDs (add-on services)
224 Administration and Operation
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
--sapversion {1.0 | R-1.0 }
Specifies the Flexrame internally used version
--db { HANA | R-HANA }
Specifies the type of HANA configuration. ‘HANA’ means the non-HRS configuration
as known from FFO 1.0.
--hdb <SYSNR>:<client_lan_ip>:<server lan ip>:<nodeid>
The logical host name is used for the database (generated automatically:
hdb<nr><sid>-<nodeid>) as well as the IP address for that host name. Use an
asterisk if you want it to be chosen automatically. All the entries need to be specified
in a colon separated format.
You can omit the network part of the IP, e.g. 10.10.12. and specify only the last tupel
of the IP, e.g. 123.
This is also valid for the server la nip.
<SYSNR> specifies the instance number of the service. Valid range is 00 .. 97. At
now SAP supports only one instance.
<node> has to be set always to ‘001’ because of single node configuration. At now
we support only one node per instance.
--replid <replication id>
With HANA System Replication you need the replication id to distinct a SID within a
pool. The range for repkid is 0 to 9. The replication id ‘0’ has to be defined always
first and is the last one deleted.
--runingroup <name of poolgroup>
A HANA replica can have roles like ‘primary’ or ‘secondary’. To separate volumes
and userdata we need to assign a replica instance to a specific poolgroup for
running. Different replication instances (identified by replid) must not run in the
same poolgroup.
--runingroup <old poolgroup>:<new poolgroup>
This works only with operationcode ‘mod’ and moves a HRS instance to another
poolgroup.
--runingroup _migrate_to_hsr:<name of poolgroup>
This works only with operationcode ‘mod’ and add HRS attributes to a former
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 225
non-HRS specification. This only changes the LDAP attributes of the SID and
prepares file structure to HRS requirements. It guess that the automount
entries are HRS-ready (pools created with FFO 1.1 are HRS-ready).
--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,...
--user <username1>:<uidnumber1>,<username2>:<uidnumber2>,...
User and group enable specially selected user numbers and group numbers to be
assigned to SAP users and SAP groups respectively. In this case a check is made to
see whether the user or group has already been defined for the DB system involved.
A user or group is created only if they do not already exist. For example, a group
dba which already exists cannot be assigned to a group number which deviates from
the default value.
--sysnr <SYSNR>
Removes a specific SAP instance instead of the entire system (SID) in case of
delete operation.
[ --sapshared <nas system>:/vol/<volume path> ]
Specifies the SID global HANA volume for storing shared data. <nas system> is the
NAS filer node name ending with ‘-st’.
[ --sapdata <nas system>:/vol/<volume path>:<SYSNR>:<nodeid> …]
Specifies instance and node specific HANA volumes for storing HANA data files. At
now there is only one volume because of the restriction to HANA single node
configuration. <nas system> is the NAS filer node name ending with ‘-st’.
[ --saplog <nas system>:/vol/<volume path>:<SYSNR>:<nodeid> … ]
Specifies instance and node specific HANA volumes for storing HANA log files. At
now there is only one volume because of the restriction to HANA single node
configuration. <nas system> is the NAS filer node name ending with ‘-st’.
--os <instance_type>:<os>,<instance_type>:<os>,...]
Specifies the OS type for the given SID and/or their instances.
instance_type::= {default|hdb}
os::= {SLES-11.x86_64}
In combination with the --add option, default:<os> sets the operating system of
the SID itself and all their instances which no own operating system is assigned to.
SAP System Handling Adding/Removing SAP SIDs (add-on services)
226 Administration and Operation
In combination with the --mod option, default:<os> sets the operating system of
the SID only. The specifications of their instances are not changed.
--ips <old_ip>:<new_ip>,<old_ip>:<new_ip>,...]
Allows to exchange the IP addresses of specific SID instances. This option is only
used with --op mod. You have to specify the full IP you want to exchange.
ff_sid_adm.pl searches for the specific instances within a SID and exchanges all
corresponding entries in LDAP concerned by that request.
Please pay attention that you make critical changes within your configuration. So we
strongly recommend you to take a backup of LDAP database before changing IP
addresses.
Examples:
Adding a SID with HDB services (non HRS):
control1:~ # ff_sid_adm.pl --op add --sid HA1 --pool otto \
--sapversion 1.0 --db HANA \
--hdb 01:150:150:001 \
--hdbtype OLAP
Removing a SID with HDB service (non HRS):
control1:~ # ff_sid_adm.pl --op rem --sid HA1 --pool otto
Add HRS attributes to HDB services (switch to HRS):
control1:~ # ff_sid_adm.pl --op mod --sid HA1 --pool otto –replid 0 \
–runingroup _migrate_to_hsr_:DC_3
Adding a SID with HANA System Replication:
control1:~ # ff_sid_adm.pl --op add --sid HR1 --pool hrs \
--sapversion R- 1.0 --db R-HANA \
--replid 0 --runingroup DC_1
--hdb 01:150:150:001 \
--hdbtype OLTP
Removing a SID with HANA System Replication:
control1:~ # ff_sid_adm.pl --op rem --sid HR1 --pool hrs –replid 0
Adding/Removing SAP SIDs (add-on services) SAP System Handling
Administration and Operation 227
Switch poolgroup for HANA System Replication:
control1:~ # ff_sid_adm.pl --op mod --sid HR1 --pool hrs \
--replid 0 –runingroup DC_1:DC_2
10.5.8 SAP classic NetWeaver services using HANA database
Instead of specifiy a classic database type like Oracle or Sybase ASE HANA database
can be used with SAP NetWeaver 7.4 or 7.3. That means you have to
Specify a HANA database service (hdb) as described in chapter 10.5.7
Specify classic NetWeaver services referring to the selected HANA database
service
Synopsis
ff_sid_adm.pl --op add --pool <pool name> --sid <SAP system id> \
--db hanadb-ref[:<clientlan-ref>][:<serverlan-ref \
--sapversion { 7.4-HANA | 7.3-HANA } \
<sap classic specifications> …
ff_sid_adm.pl --op del --pool <pool_name> --sid <SAP_system_id>
--op add
Determines the add operation.
--op { del | rem }
Determines the delete operation.
--op mod
Determines the modify operation. This option is only used to modify OS
specifications and exchanges IP adresses of specific SID instances.
--pool <pool name>
Specifies the FlexFrame pool to which the operation should be applied.
--sid <SAP system id>
Specifies the SID being used.
SAP System Handling Cloning a SAP SID into a Different Pool
228 Administration and Operation
--sapversion {7.4-HANA | 7.3-HANA }
Specifies the SAP NEtWeaver version
--db hanadb-ref[:<client-lan-ref>]:<server-lan-ref>
The client-lan-ref or server-lan-ref is the virtual hostname of the HANA database
service (hdb). There is no validation if the hostname refers to a HANA database
service.
With HRS you have to specify the hostname of replication id 0.
‘<sap classic specifications>’ describe the other instances like ascs or pai or other
options like group or user as described in chapter 10.3
Examples
Adding a SID with HDB services:
control1:~ # ff_sid_adm.pl --op add --sid HR1 --pool otto \
--sapversion 7.4-HANA \
--db hanadb-ref:hdb01ha1-se \
--sap ascs:01:100:100 \
--sap pai:02:101:101
Removing a SID with referenced HANA database service (only ‘HA1’ is removed, the
HANA database service remains untouched):
control1:~ # ff_sid_adm.pl --op rem --sid HR1 --pool otto
10.6 Cloning a SAP SID into a Different Pool
10.6.1 Script: ff_clone_sid.pl
The script ff_clone_sid.pl allows users to clone (basically copy) an entire SID from
one pool to another. It needs to be clearly understood that only the FlexFrame-specific
administrational data in the LDAP server as well as the required information for the
operating system's naming services are copied and/or added to the LDAP database. Any
additional work (like copying SAP/database binaries and database content) will not be
performed by this tool and needs to be performed, as well.
Cloning a SAP SID into a Different Pool SAP System Handling
Administration and Operation 229
To get a real clone the script has to expect that there are no conflicts in userids, groupids,
services and other properties between the origin and the target pool. You can check it by
using option ‘—dryrun’.
If there are conflicts and you are sure it does not matter you can clone the SID and the
script will select other values than defined in origin. Otherwise the cloning could mean
inconsistencies in your LDAP database. Be careful !
Synopsis
ff_clone_sid.pl –-sid <sid>
--srcpool <pool name>
--trgtpool <pool name>
[--dryrun ]
[--renew ]
Options
--sid <sid>
Specifies the SID which should be cloned.
-srcpool <pool name>
Specifies the pool with the origin SID specification.
--trgtpool <pool name>
Specifies the pool wherein the clone should be created.
[ --dryrun ]
Shows you what happens if you request a clone. You can use the option to check if there
are inconsistencies between the source and the target pool.
[ --help ]
Shows the usage.
[ --renew ]
You can enforce cloning if you have only conflicts with IP addresses between source and
target. If you are sure that it does not matter you can use this option and the script selects
new IP addresses independent from the source.
10.6.2 Changing User and Group IDs after Cloning
The ff_change_id.pl script is not further available.
To change the UID of an OS user you can use ff_user_adm.pl.
To change the GID of an OS group you can use ff_group_adm.pl.
SAP System Handling Cloning a SAP system into a different Pool
230 Administration and Operation
10.7 Cloning a SAP system into a different Pool
10.7.1 Script: ff_clone_sap.sh
The script ff_clone_sap.sh allows users to clone (basically copy) an entire SAP
installation from one pool to another. It means also the binaries are copied to the target
pool and so you get a complete independent 'copy' of your source system. The command
tries to do all configuration tasks needed to run the SAP system in the target pool.
You can clone a SAP system also if the system is active and the database is online. SAP
systems based on HANA database can only be cloned if database is offline.
The script is based on NetApp FlexClone feature. You cannot clone SAP
installations having their sapdata/saplog data on SAN.
The volume to be cloned and the clone volume have to be on the same
filer. You cannot use different filers for original and clone volume !
The script cannot check if your clone request overwrites newer versions of
files already copied earlier with older file versions. You are responsible for
that issue. Use cloning with care !
With ff_clone_sap.sh you copying a complete SAP system. You are
responsible for license issues using SAP software.
At the current release the command supports SAP services with configured
databases and also other services with the only exception of the service
BOBJ.
The script copies the SAP system files (which each clone request)
located in /usr/sap/<SID>
located in /sapmnt/<SID>
home directories of SAP specific users like <sid>adm
/usr/sap/sapservices
Database files (software, configuaration) depending on database type
Cloning a SAP system into a different Pool SAP System Handling
Administration and Operation 231
Some files are only copied if not available in the target configuration or on force request
/usr/sap/hostctrl
/usr/sap/tmp
/oracle/client
/sapdb/data
/sapdb/programs
/sapdb/sql
There are some preconditions you should take into account if you want to clone SAP
systems
the environment used (users, groups, etc.) has to be equal in source and target
pool. Otherwise you may get problems accessing files and directories. It is an
essential precondition.
do not clone SIDs from different source pools to one target pool. We
recommend to create always an own target pool to each pool from which you
want to clone SIDs.
only SAP systems residing on NetApp NAS storage are supported.
Be careful when cloning with different SAP versions. The script cannot prevent
overwriting files with older revisions.
It is necessary to store the files from sapdata/saplog in an own volume to
optimize disk storage.
Newer SAP release require an SMD agent per default. Do not forget to clone
also those installation (if SMD is supported in later). There is no automatic clone
of depending installations.
If you want to unload volFF we recommend to relocate SID specific directories in
the target pool using relocation feature.
At start of the clone process it is not known how much storage is necessary for
the running clone. You are responsible to supply enough storage. In the worst
case the clone needs storage resources as much as the origin.
SAP System Handling Cloning a SAP system into a different Pool
232 Administration and Operation
HANA configurations are only cloned if the configuration to be cloned is a single
node configurationHANA configurations can be cloned only if the original
configuration is offline
Synopsis
ff_clone_sap.sh [ --op | -o ] { create | destroy | crclone |
rmclone | initpool | check }
--srcpool <pool name>
--trgtpool <pool name>
--sid | -s <sid>
--flexdata <sapdata clone volume >
--flexlog <saplog clone volume>
[ --flexshare <sapshared clone volume> ]
[ --nodeid <nodeid of HANA master node> ]
[ --replid <replication ID> ]
[ --runingroup <poolgroup to be used> ]
[ --instance <instance number of HANA service> ]
[ --reloc <nas system>:<volume> ]
[ --dbhost <virtual DB hostname> ]
[ --config <user defined config file> ]
[ --prefix <prefix used for snapshot name> ]
[ --fcopy ]
[ --help | -h ]
[ --version | -v ]
[ --debug | -d ]
[ --dryrun ]
Options
--op initpool
After introducing the target pool by ff_pool_adm.pl use this opcode to create global
users and groups to get a consistent environment between source and target. This
step has to be done before creating first clone.
--op create
Creates a complete clone of your SAP system (cloning of database files, binaries
and global files if necessary)
--op destroy
Cloning a SAP system into a different Pool SAP System Handling
Administration and Operation 233
Deletes the clone completely. Only global files are not deleted.
--op check
Runs a check to examine your environment. It is useful to recognize violations of
most of the preconditions
--op crclone
Creates only a clone of your database
--op rmclone
Removes the database clone only
--srcpool <pool name>
Name of the pool which contains the SID to be cloned..
--trgtpool <pool name>
Name of the pool the SID should be cloned into.
--sid <sid>
Specifies the SID to be cloned.
--flexdata <nas system>:<volume name>
Name of the clone volume for sapdata. This volume is created through FlexClone
feature, not by the caller of the script.
The name of the filer is the filer’s name as used in first pool. It does not contain the
name of the target pool. <nas system> is the NAS filer node name ending with ‘-st’.
Depending on syntax of ‘<filer-name>’ you may get an error message like ‘Error:
Invalid filername used - pool specification invalid: zrfiler-1-st’. In this case set <filer-
name> to ‘zrfiler-1’ without the storage lan suffix and run command again.
--flexlog <nas system>:<volume name>
Name of the clone volume for saplog. This volume is created through FlexClone
feature, not by the caller of the script.
The name of the filer is the filer’s name as used in first pool. It does not contain the
name of the target pool. <nas system> is the NAS filer node name ending with ‘-st’.
Depending on syntax of '<filer-name>' you may get an error message like 'Error:
Invalid filername used - pool specification invalid: zrfiler-1-st'. In this case set <filer-
name> to 'zrfiler-1' without the storage lan suffix and run command again.
--flexshare <filer name>:<volume name>
Name of the clone volume for shared volume. This volume is created through
FlexClone feature, not by the caller of the script.
The name of the filer is the filer's name as used in first pool. It does not contain the
name of the target pool.
This option is only used when cloning HANA configuration
SAP System Handling Cloning a SAP system into a different Pool
234 Administration and Operation
--reloc <nas system>:<volume name>
Name of specific volume in target pool to store SID specific data (unload volFF). You
are responsible for the creation and availability of the volume. You can use the
command ff_sid_mnt_adm.pl to modify filer exports and create a script to create the
necessary directories on your volume. <nas system> is the NAS filer node name
ending with ‘-st’.
--dbhost <dbhost name>
Contains the virtual name of the database host (db<sid>-se). It is used only if the
clone should be taken from an inactive system (database offline).
--nodeid <nodeid>
Contains the node ID of the HANA configuration. It is always set to ‘001’ because we
do not support multimode configurations
--replid <replication id>
Replication ID of HANA System Replica. The valid range is 0 to 9. The replication
with ID ‘0’ has to be cloned always first.
--runingroup <poolgroup>
Poolgroup specification which contains Application Nodes which are assigned to run
specific replication instance. If omitted the script guess that the clone pool consist of
the same poolgroups as the original.
--fcopy
Enforce copy of global files. If the directory already exists in target configuration it is
overwritten (use it with care !).
--prefix <name>
Name of the prefix used for naming snapshot. Default is FF_DATA resp. FF_LOG
--help
Shows the usage
--debug
Switch on debug mode
--dryrun
Starts a cloning without changing anything. You will see what will happen if you are
requesting a real clone.
--version
Shows the script version.
Return Codes
0 means all o.k.
Cloning a SAP system into a different Pool SAP System Handling
Administration and Operation 235
9 at least one warning, but the clone request succeeded
1 abortion of the request because of errors
10.7.1.1 Cloning SAP systems with MaxDB
Some files of MaxDB database configuration are stored in /sapdb/data/config or
/sapdb/FLM/data/config. To ensure consistency you have to move those files to
/sapdata/sapdata<n>/config and create a link at origin site before you start the cloning
operation
<SID>.<nn>
<SID>.cfg
<SID>.pah
<SID>.upc>
<SID>
.M*
10.7.1.2 Post cloning steps – DB2
After cloning a SAP system based on DB2 database you have to execute
exchange hostname in /home_sap/db2<sid>/sqllib/db2nodes.cfg with the name
of the application node you want to start the cloned system first
/home_sap/db2<sid>/db2_software/instance/db2iupdt db2<sid> under control
of user ‘root’
Before calling ‘db2iupdt’ please check if there is a file named
‘/home_sap/db2<sid>sqllib/global.reg’ and check for that file in directory
‘/var/db2’. With some software release it could be necessary to make a copy
from the file into ‘/var/db2’. Please make the copy always. Otherwise the
command could fail.
We guess you changed the password of the corresponding DB2 users
(/sap<sid>, db2<sid>, etc.). After cloning the passwords for the users in the
clone are set to default values. Change your passwords using ff_user_adm.pl.
start the database using ff_service.sh
login as db2<sid> and execute ‘db2inidb <SID> as snapshot’ if there are
connection problems (error message SQL20153 by executing ‘db2 connect’)
For further informations please refer to ‘DB2: Cloning a Database Using NetApp
FlexClone Technology’ by NetApp
SAP System Handling Cloning a SAP system into a different Pool
236 Administration and Operation
10.7.1.3 Post cloning steps – Oracle
If your clone is based on an online system you have to execute a ‘recover database’
using ‘sqlplus’ under control of user ‘ora<sid>’.
We guess you changed the password of the corresponding Oracle users (/ora<sid>,
<sid>adm) in your origin installation. After cloning the passwords for the users in the
clone are set to default values. Change your passwords using ff_user_adm.pl.
10.7.1.4 Post cloning steps – MaxDB
If you are cloning a system with MaxDB less than 7.8 the configuration files from
/sapdb/data/config (7.8: /sapdb/<SID>/data/config) are copied at the time the first clone
operations took place. All further cloning operations for MaxDB do not change
/sapdb/data/config and the configuration files are possibly not up to date. This may cause
problems while starting database.
We guess you changed the password of the corresponding Oracle users (/sqd<sid>,
<sid>adm) in your origin installation. After cloning the passwords for the users in the
clone are set to default values. Change your passwords using ff_user_adm.pl.
After cloning the SAP system ensure the consistency of the configuration files by
executing
:~ # : cd /FlexFrame/volFF/pool-<target>/sapdb/<platform>/data/config
cn1:~ # rm <SID>.???
cn1:~ # cd /FlexFrame/volFF/pool-<source>/sapdb/<platform>/data/config
cn1:~ # find . | cpio –dump \
cn1:~ # /FlexFrame/volFF/pool-<target>/sapdb/<platform>/data/config
10.7.1.5 Post cloning steps – HANA reference database
SAP NetWeaver lately often uses SSFS (Secure Storage in File System) to store
authorization data, e.g. to store login data for database connects.
The data files for connection HANA are always stored at '<home-of-sidadm>/.hdb' (don't
confuse it with the settings made for NetWeaver itself written in profile). But each
application host has its own subdirectory.
Either you rename the host specific directory using the application hostnames of your
targetpool or you copy 'SSFS_HDB.DAT' from one host directory to missing host
directories (may be you have to create the directory first) .
Cloning a SAP system into a different Pool SAP System Handling
Administration and Operation 237
10.7.1.6 Cloning a SAP system example session
We guess you have a pool ‘source’ with installed SAP services using SID ‘ORA’. You
want to clone that installation to a pool ‘target’. We further guess that your sapdata/saplog
are stored on a SID specific volume (command ff_sid_mnt_adm.pl used before executing
ff_setup_sid_folders.sh). Otherwise your clone will include other databases which
belongs to SIDs you do not want to clone. This is just a matter of needed disk space.
At first you have to create your target pool by executing
Control1: ff_pool_adm.pl –op add –name target –storage 16,
172.20.16.0,255.255.255.0 –server 17,172.20.17.0,255.255.255.0
–client 15,172.20.15.0,255.255.255.0 –dns ff4s.mch.fts.net
–defrouter 172.20.18.251
The used IPs are just examples, you insert your own configuration values. If you want to
specify a pool specific volFF volume you have to extent the command by ‘—volFF
<filer>,/vol/<volume name>’. Before using the volume within this command you have to
create it manually by NetApp filer commands or ask your storage administrator.
In the next step you initialize your target pool to make sure that UIDs and GIDs are
in consistent state with the source pool. Please execute
Control1: ff_clone_sap.sh –-op initpool -–srcpool source -–trgtpool target
Now you can clone your SAP system. Please take into account that SAP systems with
MaxDB needs to move configuration files from /sapdb/data/config or
/sapdb/FLM/data/config to a sapdata directory to ensure consistencies (for requested files
see above).
We guess our filer is named ‘jer1na2’ and the volume names for sapdata/saplog are
‘/vol/sid_ora_data’/’/vol/sid_ora_log’.
You don not need to create the volumes for the clone, you just give it names. Those
volumes are created by the FlexClone feature of NetApp. You are only responsible to
supply enough disk space for clone operation and the later operation of the clone in
target pool.
To clone your system (SAP services are down; database offline) you execute
Control1: ff_clone_sap.sh -–op create -–srcpool source -–trgtpool target \
--sid ORA \
--flexdata jer1na2:/vol/sid_ora_data \
--flexlog jer1na2:/vol/sid_ora_log \
--dbhost dbora-se
Some directories/files are copied one time (e.g. if you have MaxDB 7.7 database
software and it is valid for all cloned systems, /sapdb/programs needs to be copied only
with the first clone operation).
If you want to force the copy of those files you additionally use the option ‘—fcopy’. With
SAP System Handling Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl
238 Administration and Operation
older versions of MaxDB software you should use ‘—fcopy’ for each clone because of
global directory ‘/sapdb/data’
If you need additional diagnostic support you can use the option ‘—debug’.
Please check the filer configuration by executing
Control1: ff_nas_adm.pl –op list –name jer1na2
If you cannot find ‘jer1na2-target-st’ in output of the command you need to execute
Control1: ff_nas_adm.pl –op list –name jer1na2 --pool target
Now you should be ready to operate your clone in target pool. Because of changes within
the automount configuration you should execute
applnode: /etc/init.d/autofs restart
Please check also for necessary manual operations depending on database type
described above.
10.8 Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl
To unload volumes containing database files or SID specific directories/files you can use
ff_sid_mnt_adm.pl. As of FlexFrame 5.2 this command takes over the functionality of
former ff_relocate_sid_data.pl. This means some minor incompatibilities depending on
the type of request.
Due to the combination of both commands you have to pay attention for some restrictions
using this command
The specification to relocate sapdata/saplog/sapshared has to be done before
ff_setup_sid_folder.sh is called
All other relocations have to be specified after the call of ff_setup_sid_folder.sh
SAN storage is only supported for sapdata/saplog volumes
If you have relocated sapdata/saplog/sapshared and you want to change it once more
again you have to delete the storage location description (automount entries in case of
NAS storage) and then add it again. But in that case the script does not offer support to
move the data to the new location. It has to be done manually. This can also cause that
you cannot use automounts on Application Nodes without restarting ‘autofs’ or even the
nodes.
If you have relocated other SID specific data and you want to change it once more again
you use the same procedure as used with sapdata/saplog.
If you want to withdraw your relocation to the default sites (volFF) you have to delete the
corresponding automounts and move back the data manually. But this move back needs
Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl SAP System Handling
Administration and Operation 239
some knowledge of FlexFrame internals and should be executed by FlexFrame
professional service people.
Synopsis (using common syntax)
ff_sid_mnt_adm.pl --op add
--pool <pool name>
--sid <sid>
--name <type>:<volume spec> …
[ --dryrun ]
[ --debug ]
ff_sid_mnt_adm.pl --op { del | rem }
--pool <pool name>
--sid <sid>
--name <type> …
[ --dryrun ]
[ --debug ]
ff_sid_mnt_adm.pl --op list
--pool <pool name>
--sid <sid>
[ --debug ]
ff_sid_mnt_adm.pl --help
Options
--op add
Adds new storage location descriptions (automount entries in case of NAS) into
LDAP to relocate sapdata and/or saplog.
--op { del | rem }
Deletes the relocation entries for sapdata and/or saplog
SAP System Handling Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl
240 Administration and Operation
--op list
Displays information about relocation of sapdata and/or saplog for specific SID.
--sid <sid>
Specifies the SID for which the relocation entries should be handled..
--pool <pool name>
Specifies the corresponding pool which contains the SID specification.
--name <type>:<nas system>:<volume path>[/<pool name>/<SID>]…
With opcode ‘add’ the option ‘name’ is used to select the data you want to relocate to
another volume and requires a type and a volume specification. <nas system> is the
NAS filer node name ending with '-st'. With type ‘sapdata’ and ‘saplog’, it is also
possible to use SAN. In this case, the volume specification must be given as
_SAN:<volume group name>.
With opcode _’rem’/’del’ you only need the type specification within the option
’name’.
These options could be repeated several times. But each item can be used one time
per command.
The type specification is one of
oracle
sapdb
db2
sybase
sapmnt
usr_sap
<sid>adm
sqd<sid>
db2<sid>
sap<sid>
sap<sid>db
sapdata
saplog
sapbackup
This type is normally part of type ‘saplog’ and normally there is no request to
split that data to an own volume. But with backup tools like Simpana it could be
Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl SAP System Handling
Administration and Operation 241
helpful to assign an own volume.
The explicit relocation of sapbackup works only with NAS storage, not
with SAN.
With HANA Database Service ‘hdb’ there is an extended syntax used. The synopsis
for operation code ‘add’ is:
--name sapshared:<nas system>:<volume path> …
Specifies the SID global HANS volume for storing shared data. <nas system> is
the NAS filer node name ending with '-st'.
--name sapdata:<nas system>:<volume path>:<instnr>:<nodeid> …
Specifies the instance and node specific volume for storing HANA database
datafiles. <nas system> is the NAS filer node name ending with '-st'.
--name saplog:<nas system>:<volume path>:<instnr>:<nodeid> …
Specifies the instance and node specific volume for storing HANA database
logfiles. <nas system> is the NAS filer node name ending with '-st'.
With type ‘sapdata’ and ‘saplog’, it is also possible to use SAN. In this case, the volume specification must be given as _SAN:<volume group name>
The synopsis for opcode delete is:
--name sapshared
--name sapdata:<instnr>:<nodeid>
--name saplog:<instnr>:<nodeid>
Hints
Moving sapmnt is a special case. If new SID instances are introduced by
ff_sid_adm.pl the mount points SID and SID_exe are already created.
ff_sid_mnt_adm.pl renames these two nodes and generates new entries. If you
are deleting sapmnt, the previous entries are restored.
If you are using ‘--name sapdata …’ and/or ‘--name saplog …’ you cannot use
the old options ‘--sapdata’ and/or ‘--saplog’ and reverse.
With the ‘ff_sid_mnt_adm.pl’ from a version FF 5.0 or 5.1 all relocated entries
(sapdata/sapog) were deleted if you request a delete operation. Now you should use the name option ‘--name sapdata’ and/or ‘--name saplog’ to specify what you
SAP System Handling Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl
242 Administration and Operation
want to delete. So you are able to delete either sapdata entries or saplog entries or
both.
--help
Shows the usage.
--dryrun
Shows what would happen if you execute the command
--debug
Enables script to write additional debug messages in debug file located in
‘/var/log/FlexFrame/<debug_log>’.
--force
Option ‘force’ can be used to enforce the operation in case of ‘minor’ errors. But we
strongly recommend to use it very carefully. For normal operation the option is not
really necessary and should only be used by professional service guys.
10.8.1 Hints for usage
10.8.1.1 General
Filer exports are modified internally by ff_sid_mnt_adm.pl with opcode ‘add’. This feature
is available for NetApp filers.
If you are removing relocated data the exports have to be modified manually by
ff_nas_adm.pl. If the volume should be deleted no action is necessary.
The use of type 'sapdata/saplog' in name option in case of opcode ‘add’ only change the
automount entries in LDAP and ‘ff_setup_sid_folder.sh’ is used to make directory
structures available. To relocate sapdata/saplog/sapshared you have to call
ff_sid_mnt_adm.pl before calling ff_setup_sid_folder.sh. No further actions are necessary
to enable application nodes to use the new volume. In rare cases a ‘/etc/init.d/autofs
restart’ on application nodes is necessary.
For all other types a script is generated (name is displayed by script) to move the data
from origin storage to relocated storage location. You only have to call the script to initiate
copying of data. The origin data are not deleted automatically. It has to be done manually.
10.8.1.2 Creating a SID/relocation of sapdata/saplog
If you want to use own volumes to store sapdata/saplog you have to
Specifying SID using ‘ff_sid_adm.pl’
Relocation of sapdata/saplog using ‘ff_sid_mnt_adm.pl’
Multiple NAS Systems and Multiple Volumes SAP System Handling
Administration and Operation 243
Call of ‘ff_setup_sid_folder.sh’ to create all directories needed for SID
The request for relocation has to be done before ‘ff_setup_sid_folder.sh’ is called !
10.8.1.3 Creating a SID/relocation of SID specific data
Additionally to the relocation of sapdata/saplog you can also relocate other SID specific
data. You have to
Specifying SID using ‘ff_sid_adm.pl’
Relocation of sapdata/saplog using ‘ff_sid_mnt_adm.pl’ (optional)
Call of ‘ff_setup_sid_folder.sh’ to create all directories needed for SID
On Application Node login to relocated user directories, e.g.
'/home_sap/<sid>adm' by 'su - <sid>adm', and execute a 'touch .history' (only
requested with older releases)
Relocation of SID specific data (other than sapdata/saplog) using
‘ff_sid_mnt_adm.pl’
Run generated scriptfile to move data to relocated storage area
The request for relocation has to be done after 'ff_setup_sid_folder.sh' is called !
‘ff_setup_sid_folder’ does not create directory and file structures on relocated storage
area.
10.9 Multiple NAS Systems and Multiple Volumes
During the installation process, FlexFrame assumes that there is one NAS system with sapdata and saplog volumes. Larger installations may require more than one NAS
system or multiple volumes on the same NAS system.
It is possible to distribute SAP databases across multiple NAS systems and multiple
volumes under the following conditions:
9. All NAS systems were entered in the FlexFrame Management Tool, prior to
installation of the FlexFrame landscape.
10. The software for SAP and the database are located in the centralized volume volFF
of the first NAS system or alternatively (if wanted) in a pool-specific volFF volume in
an arbitrary NAS system (defined within the FlexFrame environment).
11. For each SID you can assign a sapdata volume for the database's data files. This
sapdata volume can be shared with other SIDs or solely for this SID. But if you
want to share multiple SIDs on a volume we recommend you to use pool-specific
volumes for sapdata. It will minimize your efforts for administration tasks.
SAP System Handling Multiple NAS Systems and Multiple Volumes
244 Administration and Operation
12. For each SID you can assign a saplog volume for the database's online redolog
files. This saplog volume can be shared with other SIDs or solely for this SID. But if
you want to share multiple SIDs on a volume we recommend you to use pool-specific
volumes for saplog. It will minimize your efforts for administration tasks.
10.9.1 NetApp NAS Systems
The volumes must be created manually on the Filer, e.g. like:
filer2> vol create dataC11 10
filer2> vol create logC11 4
Here, dataC11 and logC11 are the new names of the volumes and 4 and 10 are the
numbers of disks to be used.
We recommend using the volume names sapdata and saplog (if on a different
Filer) or data<SID> and log<SID> for SID specific volumes on the same Filer.
You may use FlexVols (ONTAP 7G) or regular volumes.
The following options must be set for the volume:
filer2> vol options dataC11 nosnap on
filer2> vol options dataC11 nosnapdir on
filer2> vol options dataC11 minra off
filer2> vol options dataC11 no_atime_update on
filer2> vol options logC11 nosnap on
filer2> vol options logC11 nosnapdir on
filer2> vol options logC11 minra off
filer2> vol options logC11 no_atime_update on
Next, create qtrees for each FlexFrame pool which will store data and logs in those
volumes:
filer2> qtree create /vol/dataC11/pool1
filer2> qtree create /vol/logC11/pool1
If you use more than the first Filer, make sure it is reachable using, e.g.:
control1:~ # ping filer2-st
PING filer2-st (192.168.10.203) from 192.168.10.201 : 56(84) bytes ofdata.
64 bytes from filer2-st (192.168.10.203): icmp_seq=1 ttl=255 time=0.117 ms
64 bytes from filer2-st (192.168.10.203): icmp_seq=2 ttl=255 time=0.107 ms
64 bytes from filer2-st (192.168.10.203): icmp_seq=3 ttl=255 time=0.103 ms
Now the LDAP database has to be told that this SID is not using the default (first) Filer
and sapdata/saplog volumes:
Upgrading a SAP System SAP System Handling
Administration and Operation 245
control1:~ # ff_sid_mnt_adm.pl –-op add –-pool pool1 –-sid C11
--sapdata filer2-st:/vol/dataC11/pool1/C11 –-saplog filer2-st:/vol/logC11/pool1/C11
The step to modify filer’s export file is not necessary as of FlexFrame 5.1 because this
step is integrated in command ff_sid_mnt_adm.pl. This feature is only available with
NetApp.
The network 192.168.10.0/24 must match the Storage LAN segment of pool
pool1.
Before you can install SAP and the database on those volumes, some folders for the SID
in question have to be created in advance. To do so, run the following command for each
SID (replace "pool1" with your pool name and "C11" with your SID:
control1:~ # ff_setup_sid_folder.sh –p pool1 –s C11
Now you can continue with the SAP installation.
10.10 Upgrading a SAP System
10.10.1 Service Port
If you plan to upgrade your SAP release, you have to add a special service port (called
shadow instance) to LDAP.
SAP shadow service ports are required during an SAP release upgrade. To list, add or
remove the service ports in the LDAP database for service entries, you can use this
tool.
Synopsis
ff_sap_shadowport.sh [-d] -l -p <pool name> [-s <sid>]
[-o <portnr>]
ff_sap_shadowport.sh [-d] {-a|-r} -p <pool name> -s <sid>
[-o <portnr>]
Options
-d
Writes debugging information.
-l
Lists all SAP shadow service ports of the pool provided with the -p option.
If the option -s is used, only the port of the specified SID is displayed.
SAP System Handling Upgrading a SAP System
246 Administration and Operation
-a
Adds an entry.
-r
Removes an entry.
-p <pool name>
Specifies the name of the pool (e.g. pool1).
-s <sid>
Specifies the SAP System ID (SID) by a 3 character string (e.g. C11).
-o <portnr>
Specifies the service port number. The default is 3694 (optional).
10.10.2 FlexFrame Agents
Please make sure that the FlexFrame Application Agents are stopped on the hosts while
you are installing, updating or removing any SAP or database software:
Stop the FlexFrame Agent and check the status:
control1:~ # /etc/init.d/myAMC.FA_AppAgent stop
control1:~ # /etc/init.d/myAMC.FA_AppAgent status
There should be no running processes listed.
10.10.3 Support SAP Upgrade (ff_sap_upgrade.pl)
This command is used to support the migration from a previous SAP/database versions
to a higher one. It could be also useful if you are migrating from an older release of FF to
a newer one. As of FlexFrame 5.2 you can also downgrade the information in LDAP
given for SAP and database release. This downgrade only means changing information
in LDAP, nothing more.
‘ff_sap_upgrade.pl’ cannot be used to change the database type (e.g. from Maxdb to
Oracle). A type change is also not allowed (e.g. SAP to MDM). We recommend a deletion
of the SID and the recreation with the new type/database type.
Upgrading a SAP System SAP System Handling
Administration and Operation 247
You are able to change the SID specific settings in LDAP database like version of SAP or
database installation. It also tries to introduce the LDAP data which are requested with
newer SAP versions. The script tries to cover all requirements concerning version of SAP
or database service.
Sometimes users want to migrate from one database type to an other. The script cannot
execute all actions requested by such a migration. Manual actions could be required.
SAP System Handling Upgrading a SAP System
248 Administration and Operation
Sometimes a new SAP release requires additional operations after the upgrade of the
SAP version in LDAP. So the virtual hostname may change (e.g. for JAVA from SAP 7.0
or less to SAP 7.1 or higher). For that cases you will get a hint after processing the
command. This meant that SAP owned files like profiles are have to be changed, too.
Synopsis
ff_sap_upgrade.pl --pool <pool name> --sid {<SID>|\*}
[ --sapvo <old version> ] --sapvn <new version>
[ --dbvo <old version> ] --dbvn <new version>
--type <service type>
[ --version ] [ --debug ] [ --dryrun ]
ff_sap_upgrade.pl [ --help ]
Options
--pool <pool name>
Name of the pool the SID should belong to.
--sid <SID>
SAP system identifier. Either you specify a specific SID or you use the asterisk. Asterisk means all SIDs of the specific pool which match to sapvo and dbvo are
modified.
--sapvo <old version>
Current value of the SAP version stored in LDAP (without leading SAP-).
Available values <old version> can be displayed with option ‘—help’
--sapvn <new version>
SAP version the SID is migrated to. <new_version>. Available versions are
displayed with option ‘—help’.
--dbvo <old version>
Version of previous used database version. Available versions are
displayed with option '-help'..
SAP Kernel Updates and Patches SAP System Handling
Administration and Operation 249
--dbvn <new version>
Version of database the SID is migrated to. Available versions are displayed with option '-help'.
--type <service type>
Specifies the SAP service. Default value is ‘SAP’, other values are BOBJ, CMS,
MDM, SMD, TREX and WD. Except ‘SAP’, all services are new. This is the reason
why option ‘type’ has been introduced.
--version
Show version of command.
--debug
Sets the debug option to enlarge logging.
--dryrun
Just to see which changes would be made in LDAP if script is executed.
--help
Display usage.
10.11 SAP Kernel Updates and Patches
For an SAP kernel update (binary patches), logon to the Application Node with the CI of
the SAP system that is to be updated. Please make sure that the FlexFrame Application
Agents are stopped on the host while you are updating the SAP kernel (see also section
“FlexFrame Agents” on page 246).
SAP's OSS note 19466 describes where to find kernel patches and how to handle the
installation.
For the installation of SAP ABAP patches or similar, please refer to the SAP
documentation. There are no FlexFrame specific changes.
10.12 Unloading volFF
All SID specific data are located on the same volume (default: volFF). In case of request
of diagnostics it could happen that the volume space is not sufficient or it is not easy to
find the specific data you need because of all SIDs write into the same location.
The command ff_sid_mnt_adm.pl offers the possibility to relocate most of the SID specific
data. Please refer to corresponding command description.
SAP System Handling Administrating SAP Services
250 Administration and Operation
10.13 Administrating SAP Services
In a FlexFrame environment, any type of SAP instance, i.e. database (DB), SAP Central
Services (SCS), Java central instance (JC), Central Instance (CI), application instance
(APP) or Java-only stack (J), is called an SAP service. The management can be done
either web based with the FlexFrame Agents WebGUI or script based with the adapted
start/stop scripts.
If the SAP LVM is running, SAP services can be managed by LVM.
10.13.1 Displaying Status of SAP Services
10.13.1.1 FlexFrame Agents WebGUI
The FlexFrame Agents WebGUI can be used to display the states of SAP services in a
FlexFrame environment. To see all active SAP services and nodes in the WebGUI, the
FlexFrame Application Agent should be started on all Application Nodes. The displayed
active SAP services can be shown system-related or node-related.
The following table shows the node and service states of FlexFrame Autonomy and the
corresponding colors of these states in the FlexFrame Agents WebGUI:
Administrating SAP Services SAP System Handling
Administration and Operation 251
white
inactive or no
further
information
green
normal,
everything ok
yellow
warning
red
critical
black
critical
Node states
RUNNING
SWITCH_INT
SWITCH_EXT
PowerOff
Service states
SHUTDOWN
DOWN
WATCH
NOWATCH
UNKNOWN
NULL
RUNNING
STOPPING
STARTING
REBOOT
RESTART
RESTARTING
REBOOT
REBOOTING
RBGET
SWITCH
SWITCHOVER
SWGET
ERROR
If you don't want to configure the parameters in the plain xml files, you may use the
FlexFrame Agents WebGUI to do this more conveniently. Details are provided in the
“FlexFrame Agents Installation and Administration“ manual.
10.13.1.2 List SAP Services
The script ff_list_services.sh lists the status of all installed SAP services. If no
pools are specified, services of all pools will be shown.
SAP System Handling Administrating SAP Services
252 Administration and Operation
Synopsis
ff_list_services.sh [-ivcCH] [<pool> ...]
Options
-i
Shows inactive services
-v
Verbose mode
-c
Force use of colors for status information
-C
Suppress colors for status information
-H
Suppress headers
-h or -?
Shows usage
10.13.2 Starting and Stopping Application Services
Virtualization of the applications and services demands special measures for starting,
stopping, restarting them etc. These measures are cared for by an SAP service script for
each service type.
As of FlexFrame 5.0 there is an incompatibility concerning starting and
stopping SAP services. The standard sapxx scripts are deprecated and all
requests are now handled by ‘ff_service.sh’ script.
We also now use Linux command ‘ip’ to start and stop virtual interfaces.
Therefore you will not see the interfaces using ‘ifconfig’ as expected. To show
the interfaces you should use now ‘ip -4 –o addr show’ or ‘ff_service.sh’ with
action code ‘icheck’.
Only project specific solutions like sapxprint will be supported furthermore.
Administrating SAP Services SAP System Handling
Administration and Operation 253
The application or service must not be started directly, e.g. for an SAP instance as <sid>adm with startsap, since in this case the interfaces are neither
supplied with IP addresses, nor is the service control file maintained. The
started application will not work due to the lack of a network connection.
We strongly recommend you not to logon at application node with the client
LAN interface which you want to stop. In case of stopping a service you would
loose your connection and your terminal would hang up.
10.13.2.1 Configuration file (ff_service.config)
ff_service.sh reads settings from file ff_service.config. This file contains defaults used
while starting a service
location of profiles (depending on installed SAP release)
service specific parameters (e.g. depending on database service)
service specific kernel parameters (depending on SAP service)
cpupower parameter settings (depending on SAP service)
Most of the default settings are suitable for running SAP services. But sometimes the
behaviour of start and stop has to be switched to non-default or in your environment you
recognize that the given defaults are insufficient.
You can modify the defaults in a configuration file of your own
make a copy of ff_service_config.custom.TEMPLATE to
/FlexFrame/scripts/ff_service.config.custom
modify the corresponding configuration parameter you want (do not forget to
remove ‘# ’ in first column to activate the change)
you can restrict an option to a SID (replace SID by your SID)
you can restrict an option to a SID within a pool (replace SID by your SID and
POOL with your poolname
you can globally set an option if SID and POOL are omitted
You can only modify the values available in the template file. The value in
ff_service.config.custom overwrites the values set in ff_service.config.
If there are multiple values for an option there is a priority order, SID, POOL-
SID and global.
IF POOL and/or SID are not mentioned it is a values which can be set only
globally.
SAP System Handling Administrating SAP Services
254 Administration and Operation
10.13.2.1.1 MaxDB
If you are concerned mit SAP case 1242763 you can set the parameter to ‘on’.
# CONFIG:maxdb:SID_xserver_start_options=-a off
# CONFIG:maxdb:xserver_start_options=-a off
If you have a performance issue with a large number of dbmsrv processes (see also SB-
FF-13P07) you kann use a host specific Databases.ini file. Set the option to ‘yes’
# CONFIG:maxdb:private_configuration_file=no
If the previous option is set to ‘yes’ you can set this option also to ‘yes’ and the scripts will
use attach/detach for MaxDB registration.
# CONFIG:maxdb:private_configuration_file_use_saphostctrl=no
10.13.2.1.2 Sybase
To connect to Sybase ASE database using ‘isql’ in newer versions it is necessary to set
the command line option ‘-X’. If you run older version it is not necessary to set the option
# CONFIG:sybase:SID_POOL_ecryption_switch=-X
# CONFIG:sybase:POOL_encryption_switch=-X
# CONFIG:sybase:encription_switch=-X
10.13.2.1.3 DB2
Some DB2 database configurations have the problem of long lasting DB Crash Recovery
runs. In that case DB dependent services may run into a restart loop. Setting the option to
‘yes’ means that dependent services will wait a while for end and set the ‘nowatch’ flag
for FlexFrame Agent for that time.
# CONFIG:db2:SID_POOL_db2crash_set_nowatch_flag=no
# CONFIG:db2:POOL_db2crash_set_nowatch_flag=no
# CONFIG:db2:db2crash_set_nowatch_flag=no
10.13.2.1.4 HANA
If there is a need to increase the wait time (e.g. you always get timeout failures uring
start/stop) you can increase the value (in seconds) to a value of your choice.
# CONFIG:hana:SID_POOL_hdb_server_stopWaitTime=400
# CONFIG:hana:POOL_hdb_server_stopWaitTime=400
Administrating SAP Services SAP System Handling
Administration and Operation 255
The following values are defined by SAP Notes concerning HANA configurations. If there
are changes requested you can easy change the values. But change only if there is a
concrete request !
# CONFIG:hana:sysctl:vm.memory_failure_early_kill=1
# CONFIG:hana:sysctl:kernel.shmmni=65536
# CONFIG:hana:sysctl:vm.dirty_ratio=5
# CONFIG:hana:sysctl:net.ipv4.conf.all.arp_ignore=1
# CONFIG:hana:sysctl:# NetApp note TR 3700'
# CONFIG:hana:sysctl:fs.file-max=20000000
# CONFIG:hana:sysctl:net.ipv4.ip_local_port_range=40000 65500
# CONFIG:hana:sysctl:fs.aio-max-nr=1000000
HANA configuration expects some CPU power settings. If there are new rquests by SAP
Notes you can change the values. But change only if there is a concrete request.
# CONFIG:hana:cpupower:POOL_POOLGROUP_set_cpu_governor=performance
# CONFIG:hana:cpupower:POOL_set_cpu_governor=performance
# CONFIG:hana:cpupower:set_cpu_governor=performance
# CONFIG:hana:cpupower:POOL_POOLGROUP_c-state-disable=yes
# CONFIG:hana:cpupower:POOL_c-state-disable=yes
# CONFIG:hana:cpupower:c-state-disable=yes
10.13.2.2 Starting and Stopping SAP Services Without Root Privileges
In order to enable user to start and stop SAP services without having root privileges a
sudo (see man sudo) configuration is shipped with the Application Nodes. This
configuration allows the SAP and the DB administration users to start their respective
services (including their virtual hostname's IP addresses). The user command line
requires a prefix of sudo in addition to the standard command line (see example below):
%> sudo /FlexFrame/scripts/ff_service.sh –s <SID> -t db -a start
The sudo mechanisms will check against its configuration file if the user (or the
responsible group) is allowed to run the command and run it with root privileges.
The configuration file is stored on the Application Node and can be found in
Linux: /etc/sudoers
If FlexFrame itself needs extensions these are made in our standard images.
SAP System Handling Administrating SAP Services
256 Administration and Operation
If there are any requirements to extent the configuration at customer site you can make
these extensions in the corresponding configuration files. We strongly recommend to
adjust the intended modifications with professional service support because in the worst
case your system will not work correctly furthermore.
To make these modifications persistent in Linux you can modify the root image directly
without a maintenance cycle. But with each update of FlexFrame you need to modify your
image again.
Content of FlexFrame configuration file on Linux:
# sudoers for FlexFrame Application Nodes
Command definitions
Cmd_Alias SERVICE = /FlexFrame/scripts/ff_service.sh,
/FlexFrame/scripts/view_hosts
Cmd_Alias SAN = /FlexFrame/scripts/ff_san_*
This configuration means, all members of the operating system group sapsys are
allowed to execute the script named ff_service.sh and view_*. Members of oper,
sdba or dba just have the permission to execute the scripts ff_san_* additionally
(although currently these are not supported any more). Entering a password is not
needed.
10.13.2.3 Starting and Stopping SAP Services from Control Node using RBAC
For customers that are primarily working as RBAC enabled users on the Control Node,
there is a possibility to remotely call the ff_service.sh script without additional
configuration of ssh public key authorization for <sid>adm users or the need to login to an
Application Node as <sid>adm using a password.
On the Control Node, call the ff_service_an.sh script as follows:
cn1:~ # ff_service_an.sh <node> <cmdline>
The <cmdline> string will be directly passed to a ff_service.sh command on the specified
Application Node.
Example:
cn1:~ # ff_service_an.sh an100 –a start –t db –s TST
This command will will start the “TST” database on Application Node “an100”.
Internally, it performs these steps:
- Check permissions utilizing RBAC
Administrating SAP Services SAP System Handling
Administration and Operation 257
- Switch to root if user has appropriate rights
- Calls “ssh an100 ff_service.sh -a start -t db -s TST”
- Returns the return code of ff_service.sh or 255 if an ssh error occured
10.13.2.4 SAP Service Scripts
All Services (except project specific ones) are started by:
ff_service.sh { --sid | –s } SID> \
{ --type | -t } <type> \
{ --action | –a } <action> \
[ { --instance | -I } <nr> ] \
[ { --user | –u } <user> ] \
[ { --credential | -c } <password> ] \
[ { --nodeid | -N } <node_id ] \
[ { --replid | -r } <replication_id ] \
[ --debug | -d ]
[ --help | -h ]
[ --version | -v ]
The call parameters are:
{ --instance | -i} <nr>
Distinction of several similar instances of a service type of an SID; 2 digits,
numerical.
{ --sid | -s } <SID>
System ID (SID),.
{ --action | -a } <action>
The action to be performed with the application or service. The actions are start,
stop, restart, status, cleanup, watch, nowatch, istart,istop,
setcfg and resetcfg.
{ --type | -t } <type>
Distinction of the different service types.
ascs (classic, message server)
app (classic, application server, ABAP)
bobj (BOBJ - Business Objects)
ci (classic, central instance, ABAP)
SAP System Handling Administrating SAP Services
258 Administration and Operation
pai (classic, primary application instance, ABAP, replaces ci
as of SAP 7.4)
cms (CMS - Content Server, CMS – Cache Server)
db (Database)
ers (classic, enqueue replicated server)
j (classic, application server, Java,; also used for jc if version ge
SAP 7.1)
jc (classic, central instance, Java)
lc (Live Cache)
hdb (HANA database)
mds (MDM server)
mdss (MDM syndication server)
mdis (MDM import server)
scs (classic, message server)
smd (SMD -Solution Manager Diagnostics)
trx (TREX – Ssearch and Classification)
wd (WebDispatcher)
--debug | -d
Switch on the debug mode.
If you need the Debuglog from requests done by FlexFrame Agents you create a
file by ‘touch /tmp/switch_on_debug.<SID>’ for the SID you want to debug. All
requests for ‘SID’ store a Debugfile in /FlexFrame/scripts/log. Please do not forget
to delete the file and the Debuglogs if you will not need Debuglogs furthermore.
{ --user | -u } <user>
To get the state of BOBJ 4.0 an authentification is needed. <user> is the login user
for CMC specified in installation procedure. Default user is ‘Administrator’.
With Sybase ASE this option is used to specify the Sybase Administration User.
Default is ‘sapsa’.
{--credential | -c } <password>
Is the password of <user> used for authentification to access BOBJ.
Administrating SAP Services SAP System Handling
Administration and Operation 259
With Sybase ASE this option is used to specifiy the password of the Sybase
Administration User.
{ --nodeid | -N } <nodeid>
specifies the nodeid of the HANA database service which should be started.
Default is set to ‘001’.
{ --replid | -r } <replication id>
specifies the replication id of a HANA System Replication configuration
--help | -h
Shows the usage.
--version | -v
shows the script’s version
10.13.2.5 SAP Service Script Actions
In the following, the SAP service script actions are described:
start
This checks whether the application or service in question is already running. If it is,
it is not restarted. It also checks whether required applications or services are
running.
The required virtual IP addresses are assigned to the relevant interfaces for the
Client LAN and Server LAN (ifconfig <ifc> <ip-adr> netmask <netmask>
up). The application is started and the service control file is written.
Some services are dependent on already started other services. As far as possible
FlexFrame tries to give a hint about such dependencies. Some examples for
dependent services are CI (depends on ASCS) or WebDispatcher (depends on
message server of other SAP service).
stop
This checks whether the application or service in question is running. If it is running,
it is stopped.
The application is terminated, the service control file is deleted and the virtual IP
addresses are separated again from the interfaces (ifconfig <ifc> down).
SAP System Handling Administrating SAP Services
260 Administration and Operation
status
This checks the logical status of the application or service. The functional availability
is not tested.
With BOBJ 4.0 (SBOP) the request to get the state by using
‘/usr/sap/<SID>/…/ccm.sh’ requires an authentification
restart
This merges the actions stop, cleanup and start in one call; restart is
intended for restarting a malfunctioning application.
cleanup
This kills application processes that are still running and deletes occupied resources
such as shared memory, semaphores and the message queue.
Note that this action may only be performed after an attempt to stop the application
has failed.
nowatch
This removes the application from monitoring by the high-availability software
FlexFrame Agent without the application having to be restarted. The application itself
retains its current status. The default status after application startup is 'watch'.
watch
This includes the application again in monitoring by the high-availability software
(FlexFrame Agents) without having to restart the application. The application itself
retains its current status.
istart
Start the service interfaces. You do not need to make a distinction between server
and client lan. The script itself selects the necessary interfaces.
This action code cannot be used with type ‘ERS’ because it does not have an own
IP in client or server lan during operation.
istop
Stop the service interfaces.
This action code cannot be used with type ‘ERS’ because it does not have an own
IP in client or server lan during operation.
setcfg
Change only the hardware and OS settings as defined for SAP HANA for the
current Application node. This is useful to setup a system for running benchmarks
without a real HANA database installation.
Administrating SAP Services SAP System Handling
Administration and Operation 261
sesetcfg
Resets the changes of hardware and OS settings
10.13.2.6 SAP Service Script logging
For diagnostic issues SAP service script writes different levels of logging information
standard logging is always written to stdout
/FlexFrame/scripts/log/sapservice.log logs each call of ff_service.sh and its
exitcode. You should reorganize it periodically to get rid of old entries.
/FlexFrame/scripts/log/sap<type>_<SID>_<action>.log logs each call of
ff_service.sh and its exitcode. Additionally the messages written to stdout are
shown, too. A new request overwrites an already existing file for the same SID,
service and action.
/FlexFrame/scripts/log/sapservice.<pid>.debug.<SID>.<type>.<inr>.<action>
contains an extended debuglog if option ‘-d’ is set by ff_service.sh.
10.13.2.7 SAP Service Script User Exits
The FlexFrame script for SAP Services (ff_service.sh) provide a user exit as a shell
script /FlexFrame/scripts/user_script. If this user_script exist, the
FlexFrame script will call it twice, 1st in the beginning, 2
nd at the end of the start-/stop-
scripts. This means two phases, a pre-phase and a post-phase. The pre-phase will be
run before any action, the post-phase will run after all action. Because of plausibility
errors while executing the FlexFrame start-/stop-script, it is possible that the post-phase
will be omitted.
A sample user_script is delivered as user_script.TEMPLATE.
With FlexFrame 5.0 new service types were introduced. Those new service
types have to be added to customer specific user script. The name is built by
the rule ‘sap<service>’, e.g. ‘sapsmd’, ‘saptrx’. Please refer to template file
available in directory /FlexFrame/scripts.
To achieve a pool dependent function of user_script, put the script into the pool image
and create a symbolic link to it in FlexFrame/scripts. Seen from the Control Node:
Control1: # ln -s
/FlexFrame/pooldata/config/scripts_config/user_script/FlexFrame/scripts/user_script
SAP System Handling Administrating SAP Services
262 Administration and Operation
If no pool dependent function is needed, the script user_script can be located directly
in /FlexFrame/scripts.
10.13.2.8 SAP service script ff_smd_agent.sh
With support of SMD 7.3SP02 we introduced a new script which is called internally and
by FlexFrame Agents. The script just selects SMD Agent instance information from
configuration file ‘agents-on-the-fly.config’ and calls ff_service.sh internally. All other
users have to use ‘ff_service.sh’ as before to start SMD instances.
10.13.3 Return Code of the SAP Service Script
The SAP service scripts issue a return code (exit code). The meaning of this code can be
looked up in the sapservice_functions file:
#================================================================
# common exit codes for service scripts
#================================================================
no_error=0 # no error
wrong_parameter_count=1 # Bit 0, wrong number of parameters
plausibility_error=2 # Bit 1, plausibility error
interface_server_lan_error=4 # Bit 2, error at server lan interface up/down
interface_client_lan_error=8 # Bit 3, error at client lan interface up/down
service_start_stop_error=16 # Bit 4, error at service start/stop/status/...
any_error=32 # Bit 5, any other error
san_mount_error=64 # Bit 6, san mount error
user_script_error=128 # Bit 7, user_script error
For actions like start, stop, nowatch and watch, the exit code should be no_error
normally. For the action status, the exit code is dependent from various factors:
The installation check against the SAP service may have failed
(plausibility_error)
the SAP service is possibly not running or not running on this host (any_error)
the SAP service is running on this host (no_error).
In addition to these exit codes, the SAP service scripts print out messages as described
in the section “Start/Stop Script Errors” on page 335.
10.13.4 Starting and Stopping Multiple SAP Services
The following scripts are provided for starting and stopping multiple applications and
services:
Script name Application Place of execution
Administrating SAP Services SAP System Handling
Administration and Operation 263
Script name Application Place of execution
start_all_sapservices Initial start of all configured
applications
Only on a Control
Node
stop_all_sapservices Stopping all running
applications
Only on a Control
Node with option
<poolname>.
stop_all_sapservices_SID Stopping all running
applications of one SID
Only on a Control
Node
stop_all_sapservices_local Stopping all running
applications on the local
node
Only on an
Application Node
Ff_startstop_services.sh Starting/stopping
applications at predefined
locations
Only on a Control
Node
Call syntax for pool-dependent scripts:
stop_all_sapservices [<pool name>]
stop_all_sapservices_SID <sid> [<pool name>]
Functionality of the SAP service scripts for multiple services:
start_all_sapservices
Initial start script for customer defined sap service landscapes. This can be the
whole FlexFrame landscape, a pool, a SAP system landscape with a productive
system, a quality assurance system and a development system, or a single SAP
system with database, central instance and application instances. This script can be
duplicated to create various initial start scripts with different names, e.g.
start_all_sapservices_pool2. A sample start_all_sapservices script is
delivered as start_all_sapservices.TEMPLATE.
stop_all_sapservices
Shutdown script for all running sap services in a dedicated pool. The advantage of
this script is that sap services have not to be stopped one-by-one.
On Control Node you have to specify the pool with the services which should be
stopped. On Application Node no additional option (<pool name>) is allowed.
stop_all_sapservices_SID
Shutdown script for a singe SAP system (one SID) with all belonging sap services
like application instances, central instances und database instance in the right order.
stop_all_sapservices_local
SAP System Handling Administrating SAP Services
264 Administration and Operation
Shutdown script for all running sap services on one Application Node. This script will
be integrated via run levels. The information base for these
stop_all_sapservices* scripts are the /FlexFrame/scripts/log/*_host
files.
ff_startstop_services.sh
Starting and Stopping of applications concerning a configuration file. A configuration
file describes where a service should be startet. This could be helpful after a
downtime if you want to start the services on servers the instances run before. This
script is only available in the toolbox supplied with FlexFrame Service CD. Please
take care of the Disclaimer on Service CD because the script is just a helper, not an
official FlexFrame script.
10.13.5 Removing an Application from Monitoring by FlexFrame Agents
If the applications or services are started with the scripts for virtualization, they are
monitored by the FlexFrame Agents. If you do not want this, be it for tests, installation or
upgrades, you have to inform the high-availability software of this, using the additional parameter nowatch when the application is started.
Example:
The central instance of BW1 is to be started without monitoring by the high-availability
software:
blade1 # ff_service.sh -s bw1 –t ci –a nowatch
blade1 # ff_service.sh –s bw1 –t ci –a start
or from a Control Node:
control1 # ssh blade1 ff_service.sh -s bw1 –t ci –a nowatch
control1 # ssh blade1 ff_service.sh –s bw1 –t ci –a start
If a running application is to be excluded from monitoring by the high-availability software
without being restarted, this is possible, using the nowatch option. The application then
retains its current status.
Example:
The central instance of BW1 is to be excluded from monitoring by the high-availability
software while running:
blade1 # ff_service.sh -s bw1 –t ci –a nowatch
Alternatively, call the following from a Control Node:
Administrating SAP Services SAP System Handling
Administration and Operation 265
control1 # ssh blade1 ff_service.sh -s bw1 –t ci –a nowatch
If a running application is to be included (again) into monitoring by the high-availability software without being restarted, this is possible using the watch option. The application
then retains its current status.
Example:
The central instance of BW1 is to be included into monitoring by the high-availability
software while running:
blade1 # ff_service.sh -s bw1 –t ci –a watch
or from a Control Node:
control1 # ssh blade1 ff_service.sh -s bw1 –t ci –a watch
10.13.5.1 Stopping and Starting an Application for Upgrades Using r3up
The r3up upgrade control program starts and stops the central instance or application
server in various upgrade phases, using the conventional startsap and stopsap. The
start and stop scripts for visualization are therefore left out.
Suitable measures must be taken to ensure that the virtual host name for <sid>adm is
always available and that the interfaces are supplied with the correct IP addresses. In
addition, the application has to be removed from monitoring by the high-availability
software.
Note that for the following solution steps, the application must have been started
correctly beforehand with start and stop scripts for virtualization, so that the
matching virtual host name is set by default in the $HOME/hostname_default
script.
Remove the application from monitoring by the high-availability software and run it.
r3up can now shut down and start the application directly with stopsap and
startsap; the IP addresses remain assigned to the interfaces and the host name
is retained for <sid>adm without any changes on the correct virtual host name.
Include the application again into monitoring by the high-availability software after
the upgrade.
10.13.6 Service Switchover
There is no command line tool to enforce a direct switchover. To switch over an SAP
service like a Central Instance, Database Instance, Application Instance etc., it is required
SAP System Handling Administrating SAP Services
266 Administration and Operation
to first stop this SAP service on the actual Application Node and to start this SAP service
on the destination Application Node thereafter.
Examples:
The central instance of OL4 from pool pool1 is running on klinge4 and should
switchover to klinge5.
First, check whether the Central Instance OL4 is running on klinge4:
For a first overview to see which SAP service should be running on which Application Node in the pool pool1, run
control1 # view_hosts pool1
app_44_ol4 should be running on klinge4
app_56_ml4 should be running on Baby_4
ci_ml4 should be running on RX300-01
ci_ol4 should be running on klinge1
ci_osm should be running on Baby_3
db_osm should be running on Baby_3
db_ml4 should be running on RX300-01
db_ol4 should be running on klinge4
The output should be means that the script view_hosts does not really check the
state of the SAP services. It only checks the FlexFrame/script/log/*_host files
which are created immediately after a successfully start of the corresponding SAP
service.
For a detailed status information you can use:
control1 # ssh klinge4 ff_service.sh -s ol4 –t ci –a status
The exit code from our SAP service script will be “0” in this case. For any other errors
while executing our SAP service script, it will return an exit code not equal “0”.
cn1:~ # / # ssh klinge1 ff_service.sh -s ol4 –t ci –a status
INFO: Version of ff_service.sh: 1.20.2.21
INFO: …….
INFO: Service ci for SID CL4 should be running on this host
INFO: ….
INFO: server_lan interface cis02-se 172.20.20.129 is (already) up
INFO: Exit code: 0 (no error)
Stop the running Central Instance OL4 on klinge4:
control1 # ssh klinge4 ff_service.sh -s ol4 –t ci –a stop
And now start the stopped Central Instance OL4 on the new destination klinge5:
Administrating SAP Services SAP System Handling
Administration and Operation 267
control1 # ssh klinge5 ff_service.sh -s ol4 –t ci –a start
The manual switch over should now be complete.
10.13.7 Use ServicePings from FlexFrame Agents
With newer SAP releases it could be insufficient of FlexFrame Agents just check the
process list of a host to decide if o service is running or not.
Now the FlexFrame Agents offers the possibility to check it using SAP executables like
’sapcontrol … GetProcessList’. To activate that feature you
Switch to /opt/myAMC/scripts
Remove ‘exit 99’ from ServicePings*.sh
Activate ‘ServicePing’ for each pool you want to use it by the FlexFrame Agents
WebGUI
Restart ‘ff_manage’ either bei ‘hb_gui’ or by ‘ff_ha_cmd.sh’
Administration and Operation 269
11 Software and Hardware Upgrade and Maintenance
11.1 Upgrading the Entire FlexFrame Landscape
Upgrading the entire FlexFrame to the 1.1A release depends on the release you are
starting from:
Please note that only a „FlexFrame® Orchestrator“ V1.0A can be used for upgrading to
„FlexFrame® Orchestrator“ V1.1A. To upgrade a „FlexFrame® Orchestrator“ V1.0A”
please refer to the so called “Upgrade Guide”: “Upgrading FlexFrame Orchestrator 1.0A
to 1.1A”.
Older FlexFrame versions, for example „FlexFrame® for SAP®“ must be upgraded to
“FlexFrame® Orchestrator” V1.0A first. See the document “Upgrading FlexFrame® 5.1A
or 5.2A to 5.3A/1.0A”.
11.2 Software Upgrade on the Control Node
This section describes FlexFrame software installation and upgrades on the Control
Node. For upgrade of the Control Node see previous chapter. Control Nodes are not
designed to run any third party software (except saprouter).
Since FlexFrame 5.1A most of the installation of FlexFrame specific software (products)
on top of the underlying SLES takes place during first boot of the Control Node. You
know this from the “Quick Installation” section in the “Installation of a FlexFrame
Environment” manual.
That way the installation of these products moved from Fujitsu Mastering Site to
customer’s site. The advantage is that all those products are now available at your site
which might become important in case of needs to reinstall them.
In addition this means that these software products are no longer inside the pre-built
Control Node image.
To find out more details about the performed installation of these software products you
can inspect the well-known log file directory /var/log/FlexFrame. The very first
log_<control node name> file contains the complete installation logs.
To find out details about the software products before the actual installation you may
inspect the Service CD. Insert the Service CD and run the following commands:
cn1:~ # mount /media/dvd
cn1:~ # cd /media/dvd/CONOS
cn1:/media/dvd/CONOS # ./installer.sh –l
This will list (-l) the products and product versions.
Software and Hardware Upgrade and Maintenance
270 Administration and Operation
Do not run this installer in any other than the mentioned list (-l) mode unless
instructed to do so!
Running in install (-i) or upgrade (-u) mode may damage your FlexFrame.
New versions of these mentioned software products will be deployed with new Service
CDs. These will contain instructions how to install new versions if needed (see also
chapter “11.2.1 FlexFrame Control Node Software”
You may want to install other software products (i.e. Third Party Products) beside the
mentioned from the Service CD. Remember (as stated in chapter 2.1):
The FlexFrame Control Nodes are seen as an appliance
Installing third party software on Control Nodes or Application Nodes may cause
functional restrictions and other malfunctions of the FlexFrame system software.
Fujitsu shall have no liability whatsoever whether of a direct, indirect or
consequential nature with regard to damages caused by the third party software
or its erroneous installation.
SLES patches may have effects on the functionality of FlexFrame Control Node
and Application Node images. So be careful with installing any SLES patches to
the FlexFrame Nodes.
If you install or upgrade software products on a project specific base you must use the ff_projectinfo.sh command to document this change!
See chapter 11.10 Project specific changes for details on this command.
Nevertheless we will describe here the general approach to update/install software on the
Control Nodes, on a few examples. In your special case this approach can differ to the
ones described here.
11.2.1 FlexFrame Control Node Software
As stated in the previous chapter, new versions of the FlexFrame Control Node products
will be deployed with a new Service CD.
If needed or instructed by Fujitsu support an update of these components is possible by
launching the respective installer command: Basically you just have to change the
directory to the product location and start the installer. If you need for example to (re-)
install the “example” product insert the Service CD and run the following commands:
Software and Hardware Upgrade and Maintenance
Administration and Operation 271
cn1:~ # mount /media/dvd
cn1:~ # cd /media/dvd/CONOS/example
cn1:/media/dvd/CONOS/example # ./installer.sh –ivf
Run these installers only if needed or instructed by Fujitsu support.
Normally there is a README or other installation instructions available which
guides through the process.
All of these product installer cover the Control Node only where the Service CD has
been mounted. You will have to run them on the other Control Node as well. The
mentioned README or installation instructions will point out to that fact
11.2.2 Service Packs
It is not allowed to install any Service Packs on the Control Node manually. The only way
to obtain a new service pack for the Control Node is to install new Conttrol Node image
delivered from Fujitsu. Service Packs contain a lot of changes that can have influences to
the FlexFrame functionality. Service Packs can also contain initial configuration files that
will substitute the FlexFrame configurations.
11.2.3 Updating/Installing a New Linux Kernel
The following sections describe how to install a new Linux kernel for Control Nodes in
FlexFrame. This only has to be done if instructed to do so by Fujitsu support.
The process will be described on examples accomplished on a development system. The
version numbers here in this document are not generally binding; they must be replaced
with the correct information for your system.
This approach describes the kernel installation without the recommended Server Start
CD.
11.2.3.1 Software Stage
To install and update any of the software packages in FlexFrame, it is useful to mount
your NAS system or jukebox or any other software stage on a local mount point.
For example, your software stage is /FlexFrame/volFF/FlexFrame/stage/ and
the new kernel to install is 2.6.16.60-0.63.1.
Copy the delivered software packages to the software stage:
cn1:~ # cd /FlexFrame/volFF/FlexFrame/stage
cn1:/FlexFrame/volFF/FlexFrame/stage # mkdir -p CN/kernel
Kernel:
Software and Hardware Upgrade and Maintenance
272 Administration and Operation
cn1:~ # cp <somewhere>/kernel-smp-2.6.16.60-0.63.1.x86_64.rpm \
/FlexFrame/volFF/FlexFrame/stage/CN/kernel
Kernel kdump:
cn1:~ # cp <somewhere>/kernel-kdump-2.6.16.60-0.63.1.x86_64.rpm \
/FlexFrame/volFF/FlexFrame/stage/CN/kernel
Kernel source:
cn1:~ # cp <somewhere>/kernel-source-2.6.16.60-0.63.1.x86_64.rpm \
/FlexFrame/volFF/FlexFrame/stage/CN/kernel
With SLES 10 (Service Pack 1 and later) Novell has introduced the Partner
Linux Driver Process (PLDP). There is no need to download drivers for the
update kernel. The existing drives are reused for the new kernel.
11.2.3.2 Install the New Kernel
To install the new kernel, execute the following command:
cn1:~ # cd /FlexFrame/volFF/FlexFrame/stage/CN/kernel
cn1:~ # pwd
/FlexFrame/volFF/FlexFrame/stage/CN/kernel
cn1:~ # rpm -ivh kernel-smp-2.6.16.60-0.63.1.x86_64.rpm
cn1:~ # rpm –ivh kernel-kdump-2.6.16.60-0.63.1.x86_64.rpm
cn1:~ # rpm –ivh kernel-source-2.6.16.60-0.63.1.x86_64.rpm
11.2.3.3 Reboot the Control Node
Reboot the Control Node and test the functionality of the updated Control Node, Linux-
HA, network connectivity, and so on.
The uname -r command should display the new kernel version:
cn1:~ # uname -r
2.6.16.60-0.63.1-smp
11.2.3.4 Partner Linux Driver Process
The Partner Linux Driver Process (PLDP) replaces the traditional method of driver
provisioning for SLES 10 (Service Pack 1 and later). The most benefit of this process is
the support for reusing existing drivers for errata kernels. There is no need to download
drivers for those kernels.
Software and Hardware Upgrade and Maintenance
Administration and Operation 273
Nevertheless if updated driver packages are released (e.g. a new version) these
packages can be installed on Control Node or Application Nodes like an “installable
patch”.
Depending on a special configuration the document “Novell Partner Linux Driver Process
for PRIMERGY servers” describes the update procedure in detail. This document is
available at URL:
http://ts.fujitsu.com/support/downloads.html
11.2.4 Updating RPM Packages on an Control Node
To update the SLES RPM packages you can use the original Novell tools to perform an
online update.
11.2.4.1 Setup a patch server
It is not recommended to connect your FlexFrame environment directly to the Internet to
download patches or updates. Therefore it is advisable to set up an own patch server,
which provides the necessary update packages. See: http://www.novell.com/de-
de/linux/smt/ and http://www.novell.com/communities/files/appnote-2009-12-10-v1_7.pdf
for hints.
Provided that this patch server is accessible for the Control Node a normal online update
with YaST can be performed.
11.2.4.2 Connect the patch server to the FlexFrame environment
There are several possibilities to connect your local patch server to your FlexFrame
environment. It’s up to your internal network and security policy, which way is the right
way for you.
Some of them are described in http://www.novell.com/communities/files/appnote-2009-
12-10-v1_7.pdf
It is possible to setup an ‘External Connectivity’ on the Control Node to connect your
patch server to the FlexFrame environment. Or you can copy the data from your patch
server to an ‘USB disk’ and connect this disk directly to the Contorl Node. Or copy the
data to the local disk of the Control Node or to a directory on the filer and share (NFS,
SMB/CIFS, HTTP(S) or FTP) this directory for the Control Node.
11.2.4.3 Prepare the Online Update
To prepare the online update you have to define the installation source. That’s the way,
how you provide the update packages, i.e. your patch server
On a SLES11 Control Node:
YaST Software Software Repositories Add Media Type
Software and Hardware Upgrade and Maintenance
274 Administration and Operation
See Section 9.3 “Managing Software Repositories and Services”
http://www.novell.com/documentation/sles11/book_sle_deployment/data/sec_y2_sw_inst
source.html#sec_y2_sw_instsource for more information.
The Media Type strongly depends on the way you provide the update packages (see
section 11.5.9.5.2 ”Connect the patch server to the FlexFrame environment” on page
320).
11.2.4.4 Perform the Online Update
Now you can perform the online update:
YaST Software Online Update
For SLES11 see section 1.0 “YaST Online Update” from
http://www.novell.com/documentation/sles11/book_sle_admin/data/cha_onlineupdate_yo
u.html#cha_onlineupdate_you for more information.
If the online update has installed a new kernel, it is necessary to reboot the
Control Node.
11.2.5 Backup/Restore of FlexFrame Control Nodes
To back up or restore Control Nodes use the ff_cn_backup.sh script.
Synopsis
ff_cn_backup.sh [-dr] [-f <file name>] [-l <list>]
Options
-d
Debug
-r
Restore
-f <file name>
Use file name for restore
-l <list>
Blank separated list of files (quoted)
Software and Hardware Upgrade and Maintenance
Administration and Operation 275
11.2.5.1 Backup of a Control Node
Most of the files on a Control Node are shipped with the installation DVD. Necessary data
can be backed-up using ff_cn_backup.sh.
The default behavior (without any options) is to back up the configuration files in a zipped tar file at /FlexFrame/volFF/FlexFrame/backup/<name_of_cn><date>.tgz.
Additional files can be backed-up using the -l option.
The Control Nodes are not intended to have additional software installed on.
Therefore backup in the regular sense is not required. Installation from the DVD
is much faster than any restore.
11.2.5.2 Restore of a Control Node
If needed a restore of a Control Node can be done quite fast. The restore is basically a
new installation of the node, using FlexFrame configuration data provided by the
remaining, still running Control Node, and the existing Backups of the Control Node.
To do this the following media is needed:
Control Node DVD (labeled “FlexFrame V1.1A00 Orchestrator Control Node –
Images”)
“Service CD” (labeled “FlexFrame V1.1A00 Orchestrator Service and
Information”)
USB stick (will host the configuration file)
Make sure to use the same Control Node DVD and Service CD as before! Refer to the
remaining, still running Control Node to check the FlexFrame release.
In detail, follow these steps to restore a Control Node. In this example the remaining, still running Control Node is called control1. The Control Node to be restored is called
control2.
1. Login to the remaining, still running Control Node (control1):
2. Create a configuration file of the current (running) configuration:
control1:~# java –jar /opt/FlexFrame/MgmtTool/MgmtTool.jar
Choose “File” – “Open” – “Local LDAP Connection” to read the FlexFrame
Configuration from LDAP. Then validate configuration by help of “Tools” – “Validate Configuration”. Finally write the configuration file ff_config.xml by help of “File” –
“Save For Installation” to the USB stick.
3. If applicable, copy the SFG (‘Request for Special Release’) file (/opt/FlexFrame/etc/ff_hardware_sfg.xml) to the the USB stick, too.
Software and Hardware Upgrade and Maintenance
276 Administration and Operation
4. Boot the Control Node to be restored (in this example: control2) with the Control
Node DVD and choose “Control Node Installation”
5. Provide the configuration file ff_config.xml (from step 2) with the USB stick.
6. Insert the Service CD and continue the installation as described in the “Installation of
a FlexFrame Environment” manual. Again: You need to use use same Control Node
DVD and Service CD as before.
7. If the passwords are requested, enter the current passwords.
8. Select the correct Node (“first” or “second”) for this Control Node installation; in this
example: “second”.
9. Once the system is booted do not follow the installation instructions from the
“Installation of a FlexFrame Environment” manual. Invoke the following commands
instead:
First run the ff_init_conf.pl command:
control2:~# mkdir –p /opt/FlexFrame/init_conf
control2:~# ff_init_conf.pl --upgrade
This will create several files in the /opt/FlexFrame/init_conf directory.
10. Then execute the following commands to create the directory structure and mount
needed file systems:
control2:~# cp /opt/FlexFrame/init_conf/fstab /etc
control2:~# cp /opt/FlexFrame/init_conf/hosts /etc
control2:~# mkdir -p $(cat /opt/FlexFrame/init_conf/mountpoints)
control2:~# mount –a
Before continuing check if everything has been mounted:
control2:~# for mp in \
$(cat /opt/FlexFrame/init_conf/mountpoints); do df $mp; done
11. Now restore the original contents of the Control Node's configuration files. Use
ff_cn_backup.sh with the option -r and optionally -f <file name> to do this.
The backups are located in /FlexFrame/volFF/FlexFrame/backup and, by
default, the restore will pick the latest backup it can find. If this should not be the
appropriate backup file, use the option -f <file name>:
control2:~# ff_cn_backup.sh –r [–f <file name>]
Software and Hardware Upgrade and Maintenance
Administration and Operation 277
12. Add additional security related changes to some config files using the command:
control2:~# ff_enh_sec.sh
13. Update the rpm database regarding the FlexFrame Agents packages
This is needed since the FlexFrame Agents packages (rpms) reside on the shared
storage already. It is sufficient to update the rpm database. All of the missing
packages are available on the respective Service DVD. Insert the Service DVD,
mount it and update the database:
control2:~# rpm –U --justdb /media/dvd/FA_Suite/myAMC_FA/*.rpm
14. Start FlexFrame Agents next and activate them permanently:
control2:~# insserv myAMC.FA_LogAgent
control2:~# /etc/init.d/myAMC.FA_LogAgent start
control2:~# insserv myAMC.FA_FrameAgent
control2:~# /etc/init.d/myAMC.FA_FrameAgent start
15. Compare the two Control Nodes using the command:
control2:~# ff_cn_cmp.sh
If there are other differences in files listed, copy them over from the other Control
Node using:
control2:~# scp -p control1:/<path> /<path>
16. Reboot the just installed Control Node:
control2:~# init 6
17. Start the Linux HA software stack:
control2:~# /etc/init.d/openais start
After the command returned, it will take a while until the complete Linux HA software
stack is up and running. This can be verified with the crm_mon command; both
Nodes should be shown as online and the ldap master or replica should be switched over to the restored Control Node. An ff_ha_cmd.sh cleanup of some resources
might be necessary in some cases, especially if the old Control Node was hard
powered off.
18. If the Linux HA cluster works as expected, activate it permanently:
control2:~# insserv openais
Software and Hardware Upgrade and Maintenance
278 Administration and Operation
11.3 Maintenance of the Control Node - Hardware
11.3.1 Exchanging a Control Node
If a Control Node must be replaced for any reason, attention has to be paid to the fact
that the replacing hardware has to be the same as the replaced component. Otherwise,
problems with missing Linux drivers might occur. Problems due to hard disk failures
should usually not occur, since the Control Node's hard disks are mirrored through
hardware-based RAID-1.
11.3.1.1 Hardware Failed, Hard Disk and Installed OS are not Affected
In case that other hardware components than one of the HDs are broken (e.g. memory,
CPU, etc.) and the installed OS image is still operational, these hardware components
have to be replaced with equivalent ones. Even the entire hardware can be replaced -
excluding the HDs. The HDs can be removed from the existing hardware, plugged into
the new hardware and booted, after checking the RAID controller's BIOS for hardware
RAID settings being enabled. Even if one of the hard disks is broken, it can be replaced
with the same model and synchronized with the working hard disk through the RAID
controller's BIOS.
In this case the Control Node hardware must be replaced by the same model with the
same disk controller and the same network interface controllers. The approach is very
simple, plug in your hard disks in the new Control Node in the same order as the old one,
power on the Control Node, enter the RAID controller BIOS and check the parameters;
hardware raid should be enabled as before.
See also the manual “Installation of a FlexFrame Environment“, chapter “Control Node
Host RAID Configuration”.
11.3.1.2 One Hard Disk is Defect, the Other One is Undamaged
The Control Nodes OS installation must be on a RAID-1 disk array.
Replace the defect hard disk with the same model and synchronize it with the
undamaged one via the RAID controller BIOS. This approach is dependent from the
RAID controller model and its BIOS.
11.3.1.3 The Control Nodes OS is Damaged
If, for any reason, the operating system of a Control Node should be damaged, it has to
be installed anew with the original Control Node installation DVD. The configuration has
to be recovered from a previous backup. See section “Backup/Restore of FlexFrame
Control Nodes” on page 273.
Software and Hardware Upgrade and Maintenance
Administration and Operation 279
11.3.2 Replacing a Network Card – Control Node
If the motherboard or the PCI network card in a Control Node is replaced, no further
actions are required since there is no reference to the replaced MAC addresses.
11.4 Maintenance of the virtual Control Center
This chapter describes different scenarios of failure within the virtual Control Center. The
first scenario described in chapter 11.4.1 deals with the failure of a Hypervisor Node. All
necessary steps to get information in order to switch over SAP instances are described.
The second scenario (chapter 11.4.2) deals with the restore of a damaged virtual Control
Node. The third scenario (chapter 11.4.3) deals with the restore of a damaged Hypervisor
Node.
11.4.1 Site failover within virtual Control Center
This scenario describes the consequences if a Hypervisor Node hosting a virtual Control
Node does fail. If the Hypervisor Nodes of the virtual Control Center are hosting virtual
Application Nodes in addition, the failure of one of the Hypervisor Nodes can be
compared with a site failover within a FFO landscape consisting of physical Control and
Application Nodes. Within this chapter the different steps are described in order to handle
the situation of failure. The whole scenario is described with the naming conventions of
one of our development machines.
11.4.1.1 The naming conventions
KVM Hypervisor Nodes hosting virtual Control Center:
kvm-vcn1
kvm-vcn2
Virtual Control Nodes:
cn1 (hosted on kvm-vcn1)
cn2 (hosted on kvm-vcn2)
Virtual Application Nodes:
van1-kvm1 (hosted on kvm-vcn1)
o Pool: sles11
van3-kvm1 (hosted on kvm-vcn1)
o Pool: dp2
o Group: sap
van2-kvm2 (hosted on kvm-vcn2)
Software and Hardware Upgrade and Maintenance
280 Administration and Operation
o Pool: sles11
van4-kvm2 (hosted on kvm-vcn2)
o Pool: dp2
o Group: sap
SIDs:
CL1 (running on van3-kvm1)
11.4.1.2 Scenario of sane KVM Hypervisor Nodes
The following figure shows the situation with both Hypervisor Nodes up and running:
0 Copyright 2012 FUJITSUFTS INTERNAL
KVMH-Ausfall im vCC cont.
Copyright 2012 FUJITSU
kvm-vcn2
van2-vcn2cn2
kvm-vcn1
van1-vcn1cn1FA_CtrlAgent
van4-vcn2van3-vcn1
SAP Sid:
CL1
We can get informed about:
Cluster state of virtual Control Center
SAP services
Configuration of Virtual Machines
by the following commands.
The cluster state of the virtual Control Center can be displayed by:
cn1:~# crm_mon -1 -r -f -n
…
Software and Hardware Upgrade and Maintenance
Administration and Operation 281
Current DC: cn1 - partition with quorum
…
Node cn1: online
myAMC.FA_Messenger (lsb:ff_ha_myAMC.MessengerSrv): Started
ldap_master (ocf::fsc:ff_ha_ldap): Started
…
myAMC.FA_CtrlAgent (lsb:ff_ha_myAMC.CtrlAgent): Started
stonith_vm_CN2 (stonith:external/libvirt): Started
Node cn2: online
ldap_replica (ocf::fsc:ff_ha_ldap): Started
…
From the output we can see that both Control Nodes are online.
The following command gives information of the system status concerning the SAP
services:
cn1:~# ff_list_services.sh
Pool Group Hostname SID Type Id Node State
dp2 sap van3-kvm1 CL1 db running
dp2 sap van3-kvm1 CL1 ascs 01 running
dp2 sap van3-kvm1 CL1 scs 02 running
dp2 sap van3-kvm1 CL1 pai 03 running
dp2 sap van3-kvm1 CL1 ers 06 running
dp2 sap van3-kvm1 CL1 ers 07 running
By the following command, system information concerning the Virtual Machines hosted
by FFO Hypervisor Nodes can be derived. With operation mode list, all virtual
machines that use the specified hypervisor are shown, according to the FFO
configuration. An excerpt from our example configuration is shown hereafter:
cn1:~# ff_hn_adm.pl --op list --name kvm-vcn1
Configuration details of KVM node kvm-vcn1
Hardware
System: RX300S8
10GBit: No
Shut.Facil.: IPMI kvm-vcn1-co (192.168.73.177)
...
Virtual Machines
List of FlexFrame Virtual Machines:
Name Type State CPUs Memory Pool Group
van1-kvm1 AN Off 4 8192 dp1 sles11
Software and Hardware Upgrade and Maintenance
282 Administration and Operation
van3-kvm1 AN On 4 8192 dp2 sap
cn1 CN On 4 8192
-------------------------------------
Total 12 24576
11.4.1.3 KVM Hypervisor Node kvm-vcn1 fails (manual switch over)
The following figure shows the situation when Hypervisor Node kvm-vcn1 does fail:
0 Copyright 2012 FUJITSUFTS INTERNAL
KVMH-Ausfall im vCC cont.
Copyright 2012 FUJITSU
kvm-vcn2
van2-vcn2cn2
kvm-vcn1
van1-vcn1cn1FA_CtrlAgent
van4-vcn2van3-vcn1
SAP Sid:
CL1
If Hypervisor Node kvm-vcn1 fails, the consequences are:
Virtual Control Node cn1 is no longer running
o FA_CtrlAgent does no longer run
Virtual machine Application Nodes van1-kvm1 and van3-kvm1 are no longer
running
o SAP SID CL1 is no longer running
After Hypervisor Node failure the surviving virtual Control Node (in our example cn2) has
no information of the status of failed virtual Control Node cn1. Within this scenario the
iRMC interface of Hypervisor Node kvm-vcn1 is assumed dead (no power supply). As a
consequence cn2 does not know if kvm-vcn1 is really switched off.
Software and Hardware Upgrade and Maintenance
Administration and Operation 283
If we call the crm_mon command in this situation, we get the following output:
cn2:~# crm_mon -1 -r -f -n
…
Current DC: cn2 - partition WITHOUT quorum
…
Node cn1: UNCLEAN (offline)
myAMC.FA_Messenger (lsb:ff_ha_myAMC.MessengerSrv): Started
ldap_master (ocf::fsc:ff_ha_ldap): Started
…
Node cn2: online
ldap_replica (ocf::fsc:ff_ha_ldap): Started
…
This means that manual operator intervention is needed in order to clear the split brain
situation within the Control Center. This is done by the following command:
cn2:~# crm node clearstate cn1
This command sets the state of node cn1 from UNCLEAN (offline) to offline so that
pacemaker is able to take over the resources from that node
The state of virtual Control Node cn1 has been cleared now and all resources which are
running on cn1 are switched over to cn2. Now the Linux-HA Cluster consists only of one
active virtual Control Node cn2 but the status of the cluster is defined and clean. All
services running before on cn1 are switched over to cn2. This is especially true for the
FA_CtrlAgent.
Let us look at the output of crm_mon after Linux-HA Cluster has switched over all services:
cn2~# crm_mon -1 -r -f -n
…
Current DC: cn2 - partition WITHOUT quorum
…
Node cn1: OFFLINE
Node cn2: online
myAMC.FA_Messenger (lsb:ff_ha_myAMC.MessengerSrv): Started
ldap_replica (ocf::fsc:ff_ha_ldap): Started
ldap_master (ocf::fsc:ff_ha_ldap): Started
…
myAMC.FA_CtrlAgent (lsb:ff_ha_myAMC.CtrlAgent): Started
Software and Hardware Upgrade and Maintenance
284 Administration and Operation
The system administrator has to be aware of the resulting situation and its
consequences:
FA_Ctlr Agent did not run until failover of resources by Linux-HA Cluster
Operator intervention was needed to trigger switchover of resources
Virtual machine Application Nodes (van1-kvm1 and van3-kvm) hosted by failed
hypervisor node kvm-vcn1 did no longer write into livelist
SAP service SID CL1 running on failed virtual machine Application Nodes van3-
kvm1 does no longer run
Restarted FA_CtrlAgent (by default) does not read in the past in live list (this
behavior can be changed by parameter: dT_LivelistRead).
By default (dT_LivelistRead=0) failed/missing virtual machine Application
Nodes and SAP services are neither reported by FA_CtrlAgent nor restarted
automatically.
Manual intervention is needed to restart SAP services on virtual machine Application
Nodes hosted on other Hypervisor Nodes. A problem might be to find out in this situation
which virtual machine Application Nodes are missing?
which SAP services have been running on the failed virtual machine Application
Nodes?
Information about virtual machine Application Nodes that were hosted on failed
Hypervisor Node can be retrieved by the following commands:
cn2~# ff_hn_adm.pl --op list --name kvm-vcn1
Configuration details of KVM node kvm-vcn1
Hardware
System: RX300S8
10GBit: No
…
Hypervisor resources
no information available, because of: Cannot ping IP address 192.168.73.77 of
kvm-vcn1
Virtual Machines
List of FlexFrame Virtual Machines:
Name Type State CPUs Memory Pool Group
van1-kvm1 AN - 4 8192 dp1 sles11
van3-kvm1 AN - 4 8192 dp2 sap
Software and Hardware Upgrade and Maintenance
Administration and Operation 285
cn1 CN - 4 8192
-------------------------------------
Total 12 24576
The information is retrieved out of LDAP data base. Of cause there is no longer
information about the state of the virtual machines, because Hypervisor Node kvm-vcn1
is no longer reachable.
The following command is helpful in order to find out which SAP services (SIDs) have
been running on virtual Application Node van3-kvm1 before hypervisor kvm-vcn1 did fail.
cn2~# /FlexFrame/scripts/view_hosts dp2
ascs_cl1 should be running on van3-kvm1
db_cl1 should be running on van3-kvm1
ers_06_cl1 should be running on van3-kvm1
ers_07_cl1 should be running on van3-kvm1
pai_cl1 should be running on van3-kvm1
scs_cl1 should be running on van3-kvm1
The command evaluates logging information written by ff_service.sh when services
are started or stopped.
The command can be called either on Application Node or on Control Node. The output
of view_hosts has the following format:
<service>[instanznr]_<SID> should be running on <name of
application node)
Let’s analyze one line of the output above:
…
db_cl1 should be running on van3-kvm1
In this case the SAP service is: db and the SID is: cl1. Further van3-kvm1 is the name
of the virtual Application Node the SAP service has been started on.
Command view_hosts relies on the fact that start/stop of SAP services is
exclusively done by the corresponding FFO scripts. If you don’t use these scripts, view_hosts will not produce correct results.
Software and Hardware Upgrade and Maintenance
286 Administration and Operation
Anyway it is a good idea not only to rely on information retrieved in a disaster situation. A
responsible system administrator knows which SAP instances are running on which
application node. He is able to prioritize the importance of SAP instances and is able to
decide where these instances are restarted in case of disaster.
In order to manually restart SAP services it might be necessary to cleanup state
Files in Directory: /FlexFrame/scripts/log. Before starting SAP services it is
necessary to remove the *_host Files of the missing SID instances.
11.4.1.4 KVM Hypervisor Node kvm-vcn1 fails (automatic switch over)
It is possible to automate the switch over of SAP services in case of KVM Hypervisor
failure if the iRMC interface of the Hypervisor Node is still working. In this case the
surviving Control Node cn2 is able to shutdown kvm-vcn1. In this scenario no split brain
situation within the virtual Control Center occurs. Control Node cn2 automatically takes
over all resources, especially it restarts the FA_CtrlAgent. There is no manual
intervention needed on the Control Center, but SAP Applications require further
configuration by setting the following parameter:
dT_LivelistRead =600
Parameter dT_LivelistRead is evaluated when the FA_CtrlAgent is started. The
parameter instructs the Control Agent to look backward into the live list and to
automatically restart failed SAP instances on available spare Application Nodes. The
value of dT_LivelistRead is specified in seconds.
Setting dT_LivelistRead assumes that a sufficient number of spare nodes do exist.
The parameter dT_LivelistRead is pool-specific and and can be configured by the
FA_Config GUI. Within the pool-specific configuration window:
Local->FlexFrame->Pools->Pool Configuration
you have to select:
Autonomy->Timing
Now a window opens that allows configuring the basic timing settings for myAMC
FA_Agents. dT_LivelistRead is one of the parameters that can be configured.
A value of 600 seconds (10 minutes) is a meaningful value because the HA-Cluster
needs some minutes in order to take over resources (i.e. FA_CtrlAgent). The restarted
FA_CtrlAgent now looks back for 600 seconds into live list and will detect the failed virtual
Application Nodes and SAP services. In case of sufficient spare nodes, all failed SAP
services can be automatically restarted.
Software and Hardware Upgrade and Maintenance
Administration and Operation 287
11.4.2 Restore of damaged virtual Control Node
This scenario is analoguous to “Restore of a physical Control Node” described in chapter
11.2.5.2. Before restore operation the (existing) damaged virtual Control Node has to be
erased. This can be done by calling the virt-manager on Hypervisor Node that hosts
the damaged virtual Control Node. After shutdown and erase of the damaged virtual
Control Node the virtual Control Node has to be created again using the ff_setup_vcn.sh command. You have to provide the correct ff_config.xml file (i.e. on
mounted USB stick) that has been used when originally installing the (now damaged)
virtual Contol Node. The command for creation of virtual Control Node could be in case of
a damaged Control Node cn1:
ff_setup_vcn.sh –r 1 –c /mnt/ff_config.xml
From here on you can follow the steps described in chapter “Restore of a physical
Control Node”.
11.4.3 Restore of damaged Hypervisor Node
This scenario describes the steps that have to be executed in case a Hypervisor Node
hosting a virtual Control Node does crash and needs new installation. The scenario is
described with the following naming conventions:
KVM Hypervisor Node hosting virtual Control Node that needs new installation:
kvm-vcn1
Virtual Control Node:
cn1 (hosted on kvm-vcn1)
Virtual Application Node:
van3-kvm1 (hosted on kvm-vcn1)
1. The hypervisor node needs to be installed from Control Node DVD. You need the
original Control Node DVD and the original ff_config.xml File stored on USB stick as
well.
2. After Hypervisor Node has been installed create the virtual Control Node using the
ff_setup_vcn.sh command and restore the virtual Control Node as described in
the chapter before.
3. Erase kvm-vcn1 from: /root/.ssh/known_hosts on both: cn1 and cn2
Software and Hardware Upgrade and Maintenance
288 Administration and Operation
4. Now you have to erase all virtual Application Nodes that have been running
(originally) on the Hypervisor Node. In our example this is: van3-kvm1. If you type:
cn1:~ # ff_vm.pl --op list --name van3-kvm1
Virtualization specific configuration details of node van3-kvm1 LDAP view:
Node Type : AN
Type : KVMVM
Hypervisor Node : kvm-vcn1
Number of CPUs : 4
Memory size in MB: 8192
VM created : yes …
you can see that van3-kvm1 is still created.
Virtual Application Node van3-kvm1 can be erased using the following command:
cn1:~ # ff_vm.pl --op destroy --name van3-kvm1 --force
Password:
ERROR: Virtual machine van3-kvm1 not found in virtualization
environment update LDAP
The ERROR message can be ignored.
If you type: ff_vm.pl --op list --name van3-kvm1 again, you will see that the
“VM created” flag is now set to: “no”.
5. Configuration of the virtual Hypervisor Node can now be compled using the following
command:
cn1:~ # ff_hn_adm.pl --op complete-config --name kvm-vcn1 --user root
--pass passwort
6. In order to recreate the virtual Application node van3-kvm1 you can type:
cn1:~ # ff_vm.pl --op create --name van3-kvm1 update LDAP
7. Now you can power on the virtual Application Node using:
cn1:~ # ff_ponpoff.pl --op power-on --name van3-kvm1
Software and Hardware Upgrade and Maintenance
Administration and Operation 289
In case you exchange the physical Hypervisor Node because it is damaged, the
scenario is the same as described here. For the new Hypervisor node you have
to choose the same IMPI address that was configured with the damaged node
11.5 Maintenance of Application Nodes - Software
11.5.1 Introduction
Maintenance of Linux Application Node Images – often referred to as “the Maintenance
Cycle” – allows modification of the Application Node’s software which is normally hardly
possible owing to the shared image concept.
After a short schematic overview this section describes the following:
installing Application Node images from installation media
creating a Maintanance Image
assigning the Maintenance Image, booting and maintaining
reverting the Maintenance Image
migrating remaining Application Nodes
maintaining use cases
FlexFrame 5.0 introduced a new Application Node Image concept. It is still a shared
image concept as known from earlier FlexFrame releases but was revolutionized
particularly in order to improve the Maintenance of Application Node Images.
For details refer to the section "Setting up Application Nodes" of the "Installation of a
FlexFrame Environment" manual.
By help of the new concept the Maintenance of an Application Node Image is now as
easy as 1-2-3 as you will see in this section.
11.5.2 Schematic Overview
This section gives an idea how the Maintenance Cycle works. Basically this cycle
consists of just three steps:
1. Creating a Maintenance Image
(ff_install_an_linux_images.sh -m ...)
2. Assigning the Maintenance Image, creating boot image, booting the Application
Node and maintaining
(ff_an_adm.pl –op os ...; ff_new_an.sh –n ...)
3. Reverting the Maintenance Image (ff_install_an_linux_images.sh –r ...)
Software and Hardware Upgrade and Maintenance
290 Administration and Operation
Then all of the Application Nodes can use the maintained image instead of the old one to
boot.
The following figure visualizes the entire process of maintaining of an Application Node
Image i.e. the “Maintenance Cycle”. You will recognize the figure since it has been
introduced in the “Setting up Application Nodes” section of “Installation of a FlexFrame
Environment” manual.
1
`
os/Linux/FSC_1.1A*/
productive mode
generation 1
/FlexFrame/volFF/
Base I
mag
es
root_img and
var_img of Node #n
root_img and
var_img of Node #2root_img and
var_img of Node #1
base_img/
root_img/
os/Linux/CST_1.1A*/
root_img and
var_img of Node #n
root_img and
var_img of Node #2root_img and
var_img of Node #1
base_img/
root_img/
os/Linux/MNT_1.1A*/
root_img of
Maintenance-Node
base_img/
root_img/
maintenance mode
generation 1’
productive mode
generation 2
Ro
ot
Imag
es
1.
2.
3.
MO
VE
Software and Hardware Upgrade and Maintenance
Administration and Operation 291
Please take notice of:
The Maintenance Cycle is based on a Root Image which is copied, cleaned and
modified to fulfill the needs of an exclusively used OS image
This Maintenance Image does not have to be a newly installed image from an
“Application Node – Images Linux SLES1x” DVD - you can use an image from an
earlier Maintenance Cycle. This implies the option to “maintain a maintained
image”.
There is no need to disable/enable FlexFrame Agents manually.
There is no need to change exports on NAS manually.
11.5.3 Installing Application Node Images from Installation
Media
This section describes how to install an Application Node Image delivered by Fujitsu. You
have to follow the instructions in this section only if you did not install a FlexFrame 5.1A
(or later) Application Node Image yet.
Due to the changed Application Node Image concept it is not possible to use old
Application Node Images. You have to install at least FlexFrame Orchestrator
1.1A (or later) Application Node Images in order to run a Maintenance Cycle.
You need to install an Application Node image if you did not have installed it yet.
This Application Node image comes as a so called “Base Image” and contains all needed
components in a single file delivered by the “Application Node – Images Linux SLES1x”
DVD(s). To install the Application Node image you just have to launch one command: ff_install_an_linux_images.sh which was already introduced in the “FlexFrame
– Installation of a FlexFrame Environment” guide’s section “Setting up Application
Nodes”. The following sections describe the ff_install_an_linux_images.sh
command in more detail since it plays an important role in the Maintenance of Linux
Application Node Images.
11.5.3.1 Installing the Application Node Image
To install Application Node Images delivered by Fujitsu insert one of the “Application
Node – Images Linux SLES1x” DVDs into the DVD device of one of the Control Nodes.
You can only install one “Application Node – Images Linux SLES1x” DVD at the
same time onto your FlexFrame system.
Since the ff_install_an_linux_images.sh command locks system wide
you must install the needed DVDs one after another.
Software and Hardware Upgrade and Maintenance
292 Administration and Operation
The following examples use the first Control Node and the “Linux SLES11” DVD. At first
mount it:
cn1:/ # mount /media/dvd
After changing to the /media/dvd directory you will see the
ff_install_an_linux_images.sh command. This is actually only a wrapper to the
/opt/FlexFrame/bin/ff_install_an_linux_images.sh command in order to
increase the ease of use. It has been placed there since older FlexFrame DVDs had that
command as well and you can use it in the same way:
cn1:/media/dvd # ./ff_install_an_linux_images.sh
You might consider using the –v option when running the command for the first
time.
This will install the contents of the DVD to the Control Node as defined by the DVD itself: the cfg/install.cfg configuration file contains the information. In the example you
will find after installation the following directory:
cn1:/FlexFrame/volFF/os/Linux # l
total 20
drwxr-xr-x 6 root root 4096 Sep 23 18:18 ./
drwxr-xr-x 3 root root 4096 Sep 4 13:50 ../
drwxr-xr-x 3 root root 4096 Sep 8 16:39 FSC_1.1A00-000.SLES-11.x86_64/
The contents of the DVD will aways be located in /FlexFrame/volFF/os/Linux after
installation: Only images located there can be used by FlexFrame administration
commands. If you want to install an image under a different name use the –t option of
the ff_install_an_linux_images.sh command:
Please note that the current version of the
ff_install_an_linux_images.sh command uses a slightly different
syntax. The arguments to the –t and –s options of the command are now
names and not any longer directories.
This means for this example: The argument to the -t option is the name of the
target image for operation: The target image will be created in
/FlexFrame/volFF/os/Linux, or more precisely in the
/FlexFrame/volFF/os/Linux/<name> directory.
If a directory instead of a name is specified, the command will attempt to make a
reasonable assumption.
This helps to save your time.
Software and Hardware Upgrade and Maintenance
Administration and Operation 293
cn1:/media/dvd # ./ff_install_an_linux_images.sh -v -t \
FSC_1.1A00-000.SLES-11.x86_64-another-example
It is worthwhile not changing the prefix of the directory name to keep track of the
images. All images delivered by Fujitsu are named FSC_<FlexFrame_version>.<OS_version>.<OS_architecture> where
FSC_ stands for FlexFrame Software Container.
The ff_install_an_linux_images.sh(8) manual page contains further compre-
hensive information about ff_install_an_linux_images.sh.
11.5.4 Step #1: Creating a Maintenance Image
As stated in the “Schematic overview” the first step in the Maintenance Cycle is to create
a so called “Maintenance Image”.
Software and Hardware Upgrade and Maintenance
294 Administration and Operation
1
`
os/Linux/FSC_1.1A*/
productive mode
generation 1
/FlexFrame/volFF/
Bas
e Im
ag
es
root_img and
var_img of Node #n
root_img and
var_img of Node #2root_img and
var_img of Node #1
base_img/
root_img/
os/Linux/CST_1.1A*/
root_img and
var_img of Node #n
root_img and
var_img of Node #2root_img and
var_img of Node #1
base_img/
root_img/
os/Linux/MNT_1.1A*/
root_img of
Maintenance-Node
base_img/
root_img/
maintenance mode
generation 1’
productive mode
generation 2
Ro
ot
Ima
ge
s
1.
2.
3.
This “Maintenance Image” is ultimately a “Root Image” and therefore it can be created by help of the ff_install_an_linux_images.sh command.
Since a “Maintenance Image” has to be created the –m option has to be used.
Please note that the current version of the
ff_install_an_linux_images.sh command uses a slightly different
syntax. The –m option needs an argument: the name of the node which should
run on the Maintenance Image. The node <node> must already exist.
Software and Hardware Upgrade and Maintenance
Administration and Operation 295
Based on an existing Linux Application Node Image (specified by -s option) the
ff_install_an_linux_images.sh command will spawn another Linux Application
Node Image (specified by -t option) with special properties: it allows software installation
and modifications. You need to assign the named system (specified by -m option) to this
image by using the ff_an_adm.pl command and create the Node by using the
ff_new_an.sh command. This non-productive Node is in an exclusive state and not
recognizable by FA Agents.
Bear in mind:
this spawned image can be used for this particular Application Node (specified
by -m option) only!
the Application Node (specified by -m option) must already exist i.e. known to
ff_an_adm.pl
Synopsis
ff_install_an_linux_images.sh -s <name> -t <name> <-m node> \
[-v] [-d] [-i]
Example
cn1:~ # ff_install_an_linux_images.sh -m pa-rx300-1 \
-s FSC_1.1A00-000.SLES-11.x86_64 -t MNT_1.1A00-000.SLES-11.x86_64
It is worthwhile using a meaningful prefix for the directory name to keep track of
the images. A good start for choosing the name is to replace the original prefix
(here: FSC_) by MNT_ indicating a “Maintenance” image.
The example uses the just installed Linux Application Node Image of section “Installing
the Application Node Image” i.e. the source image (specified by -s option) is an image
delivered by Fujitsu and therefore has the FSC_ prefix. This is easier to explain in this
example. But this is not a constraint: you can use any image from an earlier Maintenance
Cycle i.e. you can “maintain a maintained image”.
11.5.5 Step #2: Assigning the Maintenance Image, booting and maintaining
As stated in the “Schematic overview” we now need to configure a Root Image since we
want to boot a real Linux Application Node in order to have a system for the software
maintenance:
Software and Hardware Upgrade and Maintenance
296 Administration and Operation
11
`
os/Linux/FSC_1.1A*/
productive mode
generation 1
/FlexFrame/volFF/
Ba
se
Im
ag
es
root_img and
var_img of Node #n
root_img and
var_img of Node #2
root_img and
var_img of Node #1
base_img/
root_img/
os/Linux/CST_1.1A*/
root_img and
var_img of Node #n
root_img and
var_img of Node #2
root_img and
var_img of Node #1
base_img/
root_img/
os/Linux/MNT_1.1A*/
root_img of
Maintenance-Node
base_img/
root_img/
maintenance mode
generation 1’
productive mode
generation 2
Ro
ot
Ima
ge
s
1.
2.
3.
11.5.5.1 Using the Node
Since we have already chosen a particular Application Node (-m option) to run on the
Maintenance Image we now have to prepare it.
Notice that during the Maintenance Cycle this Application Node is not available
for Applications like SAP services, databases etc. It is not visible to FlexFrame
and can therefore not serve as a spare node during that time.
Move running applications to the remaining Application Nodes and shut it down.
Software and Hardware Upgrade and Maintenance
Administration and Operation 297
Notice that if you power off the Application Node, e.g. using ff_ponpoff.pl, the
FlexFrame Agents will see an outage and are reacting as configured.
cn1:~ # ssh pa-rx300-1 init 0
Now follows the actual step #2: the assigning of the created “Maintenance Image” of step
#1 to this node by help of the well known ff_an_adm.pl command:
cn1:~ # ff_an_adm.pl --name pa-rx300-1 --op os \
--ospath Linux/MNT_1.1A00-000.SLES-11.x86_64
Create the image for the chosen Application Node by help of the well known ff_new_an.sh command:
cn1:~ # ff_new_an.sh -f –n pa-rx300-1
11.5.5.2 Booting
Finally boot the Node by help of the well known ff_ponpoff.pl:
cn1:~ # ff_ponpoff.pl --op power-on --name pa-rx300-1
After a short time the Application Node pa-rx300-1 will be up and running. During the
boot process you certainly have seen the following message:
###########################
RUNNING IN MAINTENANCE MODE
###########################
After login you will get the information again:
cn1:~ # ssh pa-rx300-1
FlexFrame(R) 1.1A00
Authorized uses only. All activity may be monitored and reported.
Last login: Wed Jun 4 11:05:26 2014 from 172.25.101.181
Authorized uses only. All activity of your current session may be
monitored and reported.
Welcome to FlexFrame(R)
(c) 2010-2013 Fujitsu Technology Solutions
This is Application Node pa-rx300-1
Running in MAINTENANCE mode
Have fun.....
pa-rx300-1:~ #
Software and Hardware Upgrade and Maintenance
298 Administration and Operation
You see: without manipulating anything the system booted into the “Maintenance Mode”.
Checking the file systems will reveal:
pa-rx300-1:~ # mount -t nfs | grep " / "
10.10.11.2:/vol/volFF/node-pa-rx300-1/root_img/MNT_1.1A00-000.SLES-
11.x86_64 on / type nfs (rw,relatime,vers=3,…)
pa-rx300-1:~ # mount -t nfs | grep " /var "
pa-rx300-1:~ #
pa-rx300-1:~ # pgrep -lf AppAgent
pa-rx300-1:~ #
Looking closer to this command output shows that
the root file system is mounted read-write
In contrast to the normal operation the root file system is now writable and allows
installation of software
the /var file system is not mounted
In contrast to the normal operation the /var directory resides on the root file
system and is no separate file system. This means that installation of software
affecting the /var will go into the root file system.
the FlexFrame Agents are not running
In contrast to the normal operation the FlexFrame will not recognize the Application
Node pa-rx300-1 due to the non-started FlexFrame Agents.
11.5.5.3 Maintaining
This Application Node runnig in Maintenance Mode is prepared to get maintained now.
Before starting please keep in mind the following:
Changing the Application Nodes‘ software is delicate since any change can affect
the functionality of the entire FlexFrame. Therefore it is essential to test carefully all
changes.
Installing/Upgrading software should be done based on RPMs since it is easier to
keep track of the changes you made.
Installing SLES patches may have huge effects on the functionality of FlexFrame
Application Node images. Therefore evaluate security and recommended patches
whether they are applicable within a FlexFrame Application Node image.
Installing of complete SLES Service Packs is not allowed during the Maintenance
Mode. Service Packs contain a lot of changes that can have influences to the
FlexFrame functionality. Service Packs can also contain initial configuration files
that will substitute the FlexFrame configurations.
Software and Hardware Upgrade and Maintenance
Administration and Operation 299
The only way to obtain a new SLES Service Pack for Application Nodes is to install
new Application Node images delivered from Fujitsu on new “Application Node –
Images Linux SLES1x” DVDs.
The section 11.5.8 Maintaining Use Cases contains detailed use cases of maintaining
work.
Finish maintaining of the Application Node’s software always with intensive tests. The
maintained Application Node Image will become the image for other Application Nodes as
well.
Notice that all Application Nodes you will boot later on from that maintained
image will inherit all the changes you made. Therefore keep an eye on unwished
remnants i.e. unwished temporary files, notes, directories etc. Clean them up.
Finally shut down the Application node running in Maintenance mode i.e. in the example
do:
pa-rx300-1:~ # init 0
11.5.6 Step #3: Reverting the Maintenance Image
As stated in the “Schematic overview” the last step in the Maintenance Cycle is to revert
the Maintenance Image in order to be able to run productive Nodes on it:
Software and Hardware Upgrade and Maintenance
300 Administration and Operation
1
`
os/Linux/FSC_1.1A*/
productive mode
generation 1
/FlexFrame/volFF/
Ba
se
Im
ag
es
root_img and
var_img of Node #n
root_img and
var_img of Node #2
root_img and
var_img of Node #1
base_img/
root_img/
os/Linux/CST_1.1A*/
root_img and
var_img of Node #n
root_img and
var_img of Node #2
root_img and
var_img of Node #1
base_img/
root_img/
os/Linux/MNT_1.1A*/
root_img of
Maintenance-Node
base_img/
root_img/
maintenance mode
generation 1’
productive mode
generation 2
Ro
ot
Ima
ge
s
1.
2.
3.
The “Maintenance Image” resulting of step #2 needs to be transformed into a new Root
Image”. This - i.e. creating an Image - can be done by help of the already known ff_install_an_linux_images.sh command. Since a revert is needed the –r option
has to be used.
Please note that the current version of the
ff_install_an_linux_images.sh command uses a slightly different
syntax. The –r option needs an argument: the name of the node which should
be reverted from Maintenance Image.
Software and Hardware Upgrade and Maintenance
Administration and Operation 301
The reverting is required since only a temporary “Base Image” can be used to create a
“Root Image” which is needed to run productive Application Nodes on it.
Based on the maintained Linux Application Node Image (specified by -s option) the
ff_install_an_linux_images.sh command will spawn another Linux Application
Node Image (specified by -t option). You can assign systems to this Linux Application
Node Image by using ff_an_adm.pl and create Nodes by using the ff_new_an.sh
command. All of the Nodes running on it will benefit from the software maintenance you made. This launch of ff_install_an_linux_images.sh command will copy the
entire Image to a new Image.
Synopsis
ff_install_an_linux_images.sh -s <name> -t <name> <-r node> \
[-v] [-d] [-i]
Example
(Remember the step #2 asked to shut down the used Application Node)
cn1:~ # ff_install_an_linux_images.sh -r pa-rx300-1 \
-s MNT_1.1A00-000.SLES-11.x86_64 -t CST_1.1A00-000.SLES-11.x86_64
It is worthwhile using a meaningful prefix for the directory name to keep track of
the images. A good start for choosing the name is to replace the original prefix
(here: MNT_) by CST_ indicating a “Customer” image.
As mentioned earlier the SLES operating systems, which are used within a FlexFrame
environment, have been changed in a number of places to implement the concepts,
which make FlexFrame unique. In some situations the changes made during
Maintenance (Step #2) may conflict with FlexFrame’s concepts. In order to keep
FlexFrame going smoothly the FlexFrame implementations have higher priority than the
changes made during Maintenance mode. This means: these changes made during
Maintenance mode will be replaced during reverting2. If possible the
ff_install_an_linux_images.sh command will point out to these conflicts:
cn1:~ # ff_install_an_linux_images.sh -r pa-rx300-1 \
-s MNT_1.1A00-000.SLES-11.x86_64 -t CST_1.1A00-000.SLES-11.x86_64
...
The following files were modified during maintenance. To ensure
FlexFrame's functionality they will be replaced now by their
2 FlexFrame Infrastructure Consultants might seek advice in /opt/FlexFrame/lib/an.
Software and Hardware Upgrade and Maintenance
302 Administration and Operation
FlexFrame specific versions. See "Administration and Operation
Guide" for details:
/etc/issue
/root/.ssh/authorized_keys
/etc/fstab
A copy of the maintenance version is available as
"<file>.maintenance".
...
Here you must compare for example /etc/fstab.maintenance (this is the version of
this file after leaving maintenance and before reverting) and the /etc/fstab (this is the
needed FlexFrame version of this file, installed after revert). You may have to merge
entries from the /etc/fstab.maintenance file into the /etc/fstab file.
Note: it is important to double-check the reverted image whether it contains the expected
changes: This is possible for example by simply comparing the just created image with
the old maintenance image. If there are changes which were replaced by FlexFrame
settings it is needed to decide whether they can be inserted again. Please take notice of
section 11.5.8 Maintaining Use Cases.
After reverting the source of the operation the “Maintenance Image” seems to be
needless: it was the source for the revert and can be removed now. But in section 11.5.8
Re-Using the Maintenance Image on page 304 you will see how experienced
administrators might use it again.
11.5.7 Migrating remaining Application Nodes
Now, after testing and reverting the “Maintenance Image” it is time to go productive with
all the Application Nodes.
The created image acts as a normal image i.e. it can now serve as a source for other
Application Node Images. All Application Nodes inherit the changes from Maintenance
Mode.
To do this it is only necessary to use the well known administration commands. Since this
is not “Maintenance Mode” specific – the procedure and all the commands are already
known - now only the example follows:
At first assign the created image to the Application Nodes by help of the ff_an_adm.pl
command and then create an image by help of the ff_new_an.sh command and boot.
Assuming we have another 3 Application Nodes in the group sles11_x64 of the pp1
pool where our pa-rx300-1 belongs to.
The first one – named pa-rx300-1 – was used for the Maintenance Cycle and is
currently down. Let us start with that node:
Software and Hardware Upgrade and Maintenance
Administration and Operation 303
cn1:~ # ff_an_adm.pl --name pa-rx300-1 --op os \
--ospath Linux/CST_1.1A00-000.SLES-11.x86_64
The ff_an_adm.pl command asked for it so let us launch the ff_new_an.sh
command:
cn1:~ # ff_new_an -n pa-rx300-1
Finally boot the Application Node:
cn1:~ # ff_ponpoff.pl --op power-on --name pa-rx300-1
All the other nodes of the pool group – named pa-rx300-2, pa-rx300-3 and pa-
rx300-4 – which are still running on the old image can be handled in a similar way.
Instead of powering them on they just need to be rebooted after assigning the image:
Notice that if you power off the Application Node, e.g. using ff_ponpoff.pl, the
FlexFrame Agents will see an outage and are reacting as configured.
cn1:~ # ff_an_adm.pl --name pa-rx300-2 --op os \
--ospath Linux/CST_1.1A00-000.SLES-11.x86_64
cn1:~ # ff_new_an -n pa-rx300-2
cn1:~ # ssh pa-rx300-2 init 6
cn1:~ # ff_an_adm.pl --name pa-rx300-3 --op os \
--ospath Linux/CST_1.1A00-000.SLES-11.x86_64
cn1:~ # ff_new_an -n pa-rx300-3
cn1:~ # ssh pa-rx300-3 init 6
cn1:~ # ff_an_adm.pl --name pa-rx300-4 --op os \
--ospath Linux/CST_1.1A00-000.SLES-11.x86_64
cn1:~ # ff_new_an -n pa-rx300-4
cn1:~ # ssh pa-rx300-4 init 6
The switch to the maintained image costs only a few minutes! In case of problems you
can switch back in the same way.
Please bear in mind: It is very important that all the Linux Application Nodes of
one pool group have the same configuration (software, patches) since each one
must be able to take over if another one fails. That is why all of the Linux
Application Nodes of a pool group must run on the same image.
In the course of time you might get a lot of Application Node Images in
/FlexFrame/volFF/os/Linux/. In order to keep track of their history you might use
the “.itoc” file of the “Root Image”.
Software and Hardware Upgrade and Maintenance
304 Administration and Operation
11.5.8 Re-Using the Maintenance Image
This section describes a time-saving extension to the “Maintenance Cycle”. You should
use it not before you have got some experience with the “Maintenance Cycle” itself.
Reverting the “Maintenance Image” as described in section 11.5.6
Step #3: Reverting the Maintenance Image on page 299 ended with an apparently
needless “Maintenance Image”: it was the source of the operation. If you are satisfied
with the results of the Maintenance Cycle i.e. the migrated Application Nodes are running
fine you can remove it indeed.
But why it is not removed automatically?
Because it is possible to configure the Application Node (used to perform the
maintenance tasks) and to boot it from the “Maintenance Image” again: this allows fixing
errors you found while testing the just generated images without starting a new
“Maintenance Cycle”.
Be aware that only the same Application Node can be used to boot the “Maintenance Image” again. You can use the ff_an_adm.pl command to get the needed information.
Launch it with no option (output trimmed):
cn1:/ # ff_an_adm.pl
…
--ospath <path to os image>
The path of the os image to be used.
Currently available os image pathes are:
Linux/CST_5.3A00-001.SLES-11.x86_64
Linux/FSC_5.3A00-001.SLES-11.x86_64
Linux/MNT_5.3A00-001.SLES-11.x86_64 (maintenance only (pa-rx300-
1))
Linux/TST_5.3A00-002.SLES-11.x86_64
…
It reveals the name of the only Application Node that can use this “Maintenance Image”:
the pa-rx300-1 node.
To assign the “Maintenance Image” again to this Node use the well known
ff_an_adm.pl command:
cn1:~ # ff_an_adm.pl --name pa-rx300-1 --op os \
--ospath Linux/MNT_1.1A00-000.SLES-11.x86_64
After that create the image for the Application Node:
cn1:~ # ff_new_an.sh –B -n pa-rx300-1
Software and Hardware Upgrade and Maintenance
Administration and Operation 305
Notice that ff_new_an.sh needs to be called with the “-B” option to only configure boot
information for this node because the image is already configured for this node and
cannot be reconfigured.
Finally, reboot the maintenance node:
cn1:~ # ssh pa-rx300-1 init 6
or switch it on if it is powered off:
cn1:~ # ff_ponpoff.pl --op power-on --name pa-rx300-1
After some time the node will come up running the “Maintenance Image" and you can
perform further maintenance tasks and fix errors you found while testing.
As soon as you have finished maintaining this image, shut down the node and revert the
“Maintenance Image” to a new image again as described in section 11.5.6
Step #3: Reverting the Maintenance Image on page 299. Note that you cannot use the
same target directory name for the revert operation (it already exists.) Use a different
directory name or remove the old one if you want to use the same target directory name
as before.
Now it is possible to have one node for maintaining the Maintanance Image and
another node to test the reverted image in production mode because the
Maintenance Images and the images used for production reside inside two
different directory trees.
You will not have to switch one Application Node forth and back between
Maintenance Image and Base Image again and again. You still need to shut
down the maintenance Application Node before reverting to ensure a consistent
image state.
In theory it is also possible to run one Application Node always in “Maintenance
Mode” in order to keep it up to date with security updates and create an image
from it once in a while as needed.
However, it is strongly recommended to remove the Maintance Image once the
maintained Image has been tested intensively in production mode and perform
further maintenance tasks by starting over with a complete new Maintance Cycle
based on the Maintained Images, or in other words: perform the whole 1-2-3
procedure again on the already customized Images.
Following this procedure closely allows keeping all maintained image versions
and rolling back to working images if something failed during a Maintenance
Cycle.
You can keep track of all changes by just looking into the mentioned “.itoc” file
of the “Root Images”.
Software and Hardware Upgrade and Maintenance
306 Administration and Operation
11.5.9 Maintaining Use Cases
We would like to point out that changes due to 3rd party software or patches or
customer modifications may have heavy (negative) side effects to the FlexFrame
functionality.
Installing third party software on Application Nodes may cause functional
restrictions and other malfunctions of the FlexFrame system software. Fujitsu
shall have no liability whatsoever whether of a direct, indirect or consequential
nature with regard to damages caused by the third party software or its
erroneous installation.
Installation of software or other maintaining work as described in this section has
to be done by help of a Maintenance Cycle as described in section 11.5
Maintenance of Application Nodes - Software on page 289. Not following this
procedure i.e. maintain software when running in normal operation may harm the
entire Application Node group. Resulting malfunctions are not an issue of
FlexFrame.
11.5.9.1 Service Packs
It is not allowed to install any Service Packs on Application Node images manually. The
only way to obtain a new service pack for Application Nodes is to install new Application
Node images delivered from Fujitsu. Service Packs contain a lot of changes that can
have influences to the FlexFrame functionality. Service Packs can also contain initial
configuration files that will substitute the FlexFrame configurations.
11.5.9.2 Updating/Installing a New Linux Kernel
This section describes how to install an additional (alternative) Linux kernel into an
existing Application Node image.
The netboot configuration /tftpboot/pxelinux.cfg/<hw class>_<pool name>_<image tree name>
will determine which Linux kernel and initial ramdisk need to be used for a subset of
Application Nodes. The image tree name of original images delivered by Fujitsu begins with FSC_ and follows the scheme
FSC_<FF version>.<OS version>.<OS architecture>.
11.5.9.2.1 Software Stage
To install and update any software packages in FlexFrame, it is useful to mount your NAS
system or jukebox or any other software stage on a local mount point at your Control
Node. The same software stage must be mounted on an Application Node which is
appointed to install the new kernel-RPMs during the maintenance cycle.
Software and Hardware Upgrade and Maintenance
Administration and Operation 307
For example, your software stage on the Control Node is /FlexFrame/volFF/FlexFrame/stage and the new kernel to install is
2.6.16.60-0.63.1.
Create a subdirectory for the delivered software:
control1: # cd /FlexFrame/volFF/FlexFrame/stage
control1:/FlexFrame/volFF/FlexFrame/stage # mkdir -p SLES10/kernel
Copy the delivered software packages to the software stage.
Kernel RPM:
control1: # cp <somewhere>/kernel-smp-2.6.16.60-0.63.1.x86_64.rpm
/FlexFrame/volFF/FlexFrame/stage/SLES10/kernel
Kernel kdump:
control1: # cp <somewhere>/kernel-kdump-2.6.16.60-0.63.1.x86_64.rpm
/FlexFrame/volFF/FlexFrame/stage/SLES10/kernel
Software and Hardware Upgrade and Maintenance
308 Administration and Operation
Kernel source:
control1: # cp <somewhere>/kernel-source-2.6.16.60-0.63.1.x86_64.rpm
/FlexFrame/volFF/FlexFrame/stage/SLES10/kernel
With SLES 10 (Service Pack 1 and later) Novell has introduced the Partner
Linux Driver Process (PLDP). There is no need to download drivers for the
update kernel. The existing drivers are reused for the new kernel.
11.5.9.2.2 Installing a New Linux Kernel
Logon to the appointed Application Node running the maintenance cycle.
Check the currently installed Linux kernels:
pa-rx300-1:/ # rpm -qa | grep kernel
kernel-kdump-2.6.16.60-0.59.1
kernel-smp-2.6.16.60-0.59.1
kernel-source-2.6.16.60-0.59.1
Mount the software stage:
pa-rx300-1: # mkdir /mnt/stage
pa-rx300-1: # mount -t nfs <filer>:/vol/volFF/FlexFrame/stage /mnt/stage
To install the new kernel, execute the following command:
pa-rx300-1: # cd /mnt/stage/SLES10/kernel
pa-rx300-1:/mnt/stage/SLES10/kernel # export YAST_IS_RUNNING=instsys
pa-rx300-1:/mnt/stage/SLES10/kernel # rpm -ihv kernel-smp-2.6.16.60-
0.63.1.x86_64.rpm
pa-rx300-1:/mnt/stage/SLES10/kernel # rpm –ivh kernel-kdump-2.6.16.60-
63.1.x86_64.rpm
pa-rx300-1:/mnt/stage/SLES10/kernel # rpm –ivh kernel-source-2.6.16.60-
63.1.x86_64.rpm
‘YAST_IS_RUNNING=instsys’ is used to skip updating the bootloader
menu.
11.5.9.2.3 New initrd, Kernel and Netboot Configuration
Before booting the maintained Application Node image, it is necessary to create a new
initrd, install the initrd and kernel in /tftpboot and update the PXE configuration file.
All these steps will be automatically performed on the Control Node by the command:
control1:~ # ff_new_an.sh -B -n pa-rx300-1
Software and Hardware Upgrade and Maintenance
Administration and Operation 309
Internally, this will call the ff_mkinitrd.sh utility, update the storage export
configuration and create a new PXE configuration file.
The latest kernel will be automatically used in the future. If you like to revert to a previous
kernel, follow the instructions described below in Chapter 11.5.9.2.5.
Now reboot the Application Node and verify that it is working properly:
pa-rx300-1:~ # reboot
11.5.9.2.4 Partner Linux Driver Process
The Partner Linux Driver Process (PLDP) replaces the traditional method of driver
provisioning for SLES 10 (Service Pack 1 and later). The most benefit of this process is
the support for reusing existing drivers for errata kernels. There is no need to download
drivers for those kernels.
Nevertheless if updated driver packages are released (e.g. a new version) these
packages can be installed on Control Node or Application Nodes like an “installable
patch”.
Depending on a special configuration the document “Novell Partner Linux Driver Process
for PRIMERGY servers” describes the update procedure in detail. This document is
available at URL:
http://ts.fujitsu.com/support/downloads.html
11.5.9.2.5 Switching Between Different Kernels
If you have multiple kernel releases installed in the Application Node image and like to
switch to a different kernel, change the the symlink …/root_img/boot/vmlinuz and
run ff_new_an.sh -B -n <node name> again.
This will change the PXE configuration files for this image and hardware class and will
affect all Application Nodes of the same hardware class using this image.
The initrd does not need to be re-created nor does the initrd-symlink need to be
changed because it always contains all kernel releases found in an image.
11.5.9.3 ServerView Update
The ServerView update comprises the update of the components “Agents” and “RAID
Manager” on the Application Node.
The components for the Application Nodes are the same as for the Control
Nodes.
Software and Hardware Upgrade and Maintenance
310 Administration and Operation
README files or other installation instructions are available in the download
area.
Always use the instructions that come with the downloaded software!
Mount your software stage:
pa-rx300-1:/mnt # mkdir /mnt/stage
pa-rx300-1:/mnt # mount -t nfs <filer>:/vol/volFF/FlexFrame/stage
/mnt/stage
11.5.9.3.1 ServerView Agents
Go to your software stage directory. There you will find the following RPMs: srvmagt-agents-*.i386.rpm
srvmagt-mods_src-*.i386.rpm
srvmagt-eecd-*.i386.rpm
ServerViewConnectorService-*.386.rpm
And the associated installation script: srvmagt.sh.
Upgrade the ServerView Agents by using srvmagt.sh:
pa-rx300-1:~ # cd /mnt/stage
pa-rx300-1:/mnt/stage # sh srvmagt.sh install
11.5.9.3.2 ServerView RAID Manager
Go to your software stage directory. There you will find the
ServerView_RAID-*.i386.rpm RPM.
Upgrade the ServerView RAID Manager by:
pa-rx300-1:~ # cd /mnt/stage
pa-rx300-1:/mnt/stage # rpm -U ServerView_RAID-*.i386.rpm
The ServerView update is now finished.
11.5.9.4 Upgrading the Application Software
The rules for installation of application software (SAP/Oracle) apply here as well.
An Application Node should be available exclusively for upgrading the application
software, so that the physical host name can be set to be identical with the virtual host
name of the application.
Software and Hardware Upgrade and Maintenance
Administration and Operation 311
Provided this has been done, upgrading can be carried out in accordance with the
software vendor's standard guides.
All parts of the application software must reside on the network files system mounted
from the NAS system.
For more details see the SAP Installation Guides.
11.5.9.5 Updating RPM Packages on an Application Node
To update the SLES RPM packages you can use the original Novell tools to perform an
online update.
11.5.9.5.1 Setup a patch server
It is not recommended to connect your FlexFrame environment directly to the Internet to
download patches or updates. Therefore it is advisable to set up an own patch server,
which provides the necessary update packages. See: http://www.novell.com/de-
de/linux/smt/ and http://www.novell.com/communities/files/appnote-2009-12-10-v1_7.pdf
for hints.
Provided that this patch server is accessible for the Application Node in maintenance
mode, a normal online update with YaST can be performed.
11.5.9.5.2 Connect the patch server to the FlexFrame environment
There are several possibilities to connect your local patch server to your FlexFrame
environment. It’s up to your internal network and security policy, which way is the right
way for you.
Some of them are described in http://www.novell.com/communities/files/appnote-2009-
12-10-v1_7.pdf
It is possible to setup an ‘External Connectivity’ on the Control Node to connect your
patch server to the FlexFrame environment. On the Application Node in maintenance
mode you have to setup a ‘SSH tunnel’. Or you can copy the data from your patch server
to an ‘USB disk’ and connect this disk directly to the Application Node in maintenance
mode. Or copy the data to the local disk of the Control Node or to a directory on the filer
and share (NFS, SMB/CIFS, HTTP(S) or FTP) this directory for the Application Node in
maintenance mode.
11.5.9.5.3 Prepare the Online Update
To prepare the online update you have to define the installation source. That’s the way,
how you provide the update packages, i.e. your patch server
Software and Hardware Upgrade and Maintenance
312 Administration and Operation
On an SLES11 Application Node:
YaST Software Software Repositories Add Media Type
See Section 9.3 “Managing Software Repositories and Services”
http://www.novell.com/documentation/sles11/book_sle_deployment/data/sec_y2_sw_inst
source.html#sec_y2_sw_instsource for more information.
The Media Type strongly depends on the way you provide the update packages (see
section 11.5.9.5.2 ”Connect the patch server to the FlexFrame environment” on page
311).
11.5.9.5.4 Perform the Online Update
Now you can perform the online update:
YaST Software Online Update
For SLES11 see section 1.0 “YaST Online Update” from
http://www.novell.com/documentation/sles11/book_sle_admin/data/cha_onlineupdate_yo
u.html#cha_onlineupdate_you for more information.
If the online update has installed a new kernel, it is necessary to create a new
initrd and update the netboot configuration, before you can reboot the
maintained Application Node.
All these steps will be automatically performed on the Control Node by the command:
cn1:~ # ff_new_an.sh -B -n pa-rx300-1
Internally, this will call the ff_mkinitrd.sh utility, update the storage export
configuration and create a new PXE configuration file.
11.5.9.6 Updating vmware-tools
To update the vmware-tools to a newer version, we recommend to use our provided
Application Node Images containing this new version of vmware-tools.
To install a newer version of vmware-tools during a FlexFrame maintenance cycle please
refer to the OSP documentation of VMware, which can be found at
http://packages.vmware.com
Installation of vmware-tools from an ESXi host is not permitted.
Software and Hardware Upgrade and Maintenance
Administration and Operation 313
Log on to the Application Node in Maintenance, copy the downloaded vmware-tools
packages to this Application Node and follow the instructions to install the vmware-tools
on SLES11 or SLES10 as described in the VMware OSP documentation mentioned
above.
11.5.10 Implement conflicting changes using the ff_new_an.custom hook
If possible, all modifications of an Application Node image should be configured during a
Maintenance Cycle. However there are some cases where this is not possible, especially
when changing files or configuration directives that are automatically configured during
ff_install_an_linux_images.sh or ff_new_an.sh.
Examples of such cases are:
● SSH Host Keys, authorized_keys, known_hosts and other configuration
files
● Changes on the /etc/fstab file
● Disabling Services that are automatically re-enabled by FlexFrame tools while
reverting an image from Maintenance Mode
To be able to perform such modifications, they have to be done at the end of the ff_new_an.sh command, which is possible by creating a script called
ff_new_an.custom within the Image directory.
As this is a highly flexible way to modify images, it is strongly recommended to
be used by certified FlexFrame professionals with decent bash scripting
knowledge only. There is a high risk of damaging the whole FlexFrame
environment.
After the first call of ff_new_an.sh for an image, a template for this custom hook is
linked into the image directory:
cn1:/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64 # ls -l
total 32
-rw-r--r-- 1 root root 2906 Sep 25 13:43 .cleanup_root
Software and Hardware Upgrade and Maintenance
314 Administration and Operation
lrwxrwxrwx 1 root root 47 Oct 11 12:39 ff_new_an.custom.TEMPLATE ->
/opt/FlexFrame/lib/an/ff_new_an.custom.TEMPLATE
-rw-r--r-- 1 root root 632 Sep 25 13:02 ff_pxe.conf
-rw-r--r-- 1 root root 170 Sep 25 13:02 modules.conf
drwxr-xr-x 35 root root 4096 Oct 11 13:01 root_img
drwx------ 4 root root 4096 Oct 11 13:36 var_img
To create a new ff_new_an.custom script, just copy the template:
cn1:/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64 # cp
ff_new_an.custom.TEMPLATE ff_new_an.custom
cn1:/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64 # ls -l
total 28
-rw-r--r-- 1 root root 2906 Sep 25 13:43 .cleanup_root
-rwxr-x--- 1 root root 3958 Nov 2 12:06 ff_new_an.custom
lrwxrwxrwx 1 root root 47 Oct 11 12:39 ff_new_an.custom.TEMPLATE ->
/opt/FlexFrame/lib/an/ff_new_an.custom.TEMPLATE
-rw-r--r-- 1 root root 632 Sep 25 13:02 ff_pxe.conf
-rw-r--r-- 1 root root 170 Sep 25 13:02 modules.conf
drwxr-xr-x 35 root root 4096 Oct 11 13:01 root_img
drwx------ 4 root root 4096 Oct 11 13:36 var_img
You now have created a new custom user script from the template which contains a
couple of examples and information about how to implement custom modifications.
To get started, edit the file and activate the line SHOW_ALL_VARIABLES=1:
...
# uncomment the following line to print the above variables and
get_cn_path examples:
SHOW_ALL_VARIABLES=1
#
...
Run ff_new_an.sh in test mode to run the ff_new_an.custom script:
cn1:/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64 #
ff_new_an.sh -T -n bx202
Info: Starting configuration of Application Node bx202
Info: Using Image /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64
Info: Executing custom script /FlexFrame/volFF/os/Linux/FSC_1.0A00-
017.SLES-11.x86_64/ff_new_an.custom
Software and Hardware Upgrade and Maintenance
Administration and Operation 315
Variables:
----------
MAINTENANCE=""
ROOT_IMG="/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/root_img"
VAR_IMG="/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/var_img/var-c0a81e20"
POOLDATA_PATH="/FlexFrame/volFF/pool-pooldev/pooldata"
POOL_IMG="/FlexFrame/volFF/os/Linux/pool_img/pool-c0a81eff"
VAR_TEMPLATE="/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/root_img/var"
OSDIST="SLES-11"
OSARCH="x86_64"
CLIENT_NODE_NAME="bx202"
POOL_NAME="pooldev"
CLIENT_PROD_NAME="BX6924S3"
NIC_10G="no"
CONFIG="/tftpboot/config/netboot_pooldev_bx202.cfg"
get_cn_path examples:
----------
get_cn_path "/etc/passwd" =>
/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/root_img/etc/passwd
get_cn_path "/etc/resolv.conf" => /FlexFrame/volFF/pool-
pooldev/pooldata/config/etc/resolv.conf
get_cn_path "/etc/hosts" =>
/FlexFrame/volFF/os/Linux/pool_img/pool-c0a81eff/etc/hosts
get_cn_path "/etc/fstab" =>
/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64/var_img/var-
c0a81e20/FlexFrame/etc/fstab
Info: Configuration of bx202 was successful
These variables can and should be used where applicable.
In addition the these variables, there is also a function get_cn_path which maps a path
as seen on the application node to a path as seen from the control node.
For example, if you like to add a line to /etc/fstab, you would call this:
echo "# some content" >>$(get_cn_path /etc/fstab)
Software and Hardware Upgrade and Maintenance
316 Administration and Operation
This makes sure that you custom script always finds the correct file location on the
Control Node, no matter if the file is linked to the var image or whether you are running in
maintenance mode or not.
Because of these pre-defined variables and functions, the ff_new_an.custom script
can only be called through ff_new_an.sh.
Also, be very careful about what is done in this script. Before dealing with files, especially
with the get_cn_path function, you should always just echo the determined filename in
a first run before actually modifying that file without having tested the code.
Also, be aware that writing to symlinks in the Application Node image may redirect these
writes to the Control Node, especially with absolute symlinks, so please double check
what files are really being accessed.
There are a couple of commented examples for common use cases in the
template which should be used if possible. These examples have been
thoroughly tested and can be modified to your needs.
About the Mechanism
The mechanism how the ff_new_an.custom script is called is as follows:
User calls ff_new_an.sh
internally calls ff_create_an_linux.sh
if ff_new_an.custom exists in the image directory, selected internal
variable and functions are being exported the ff_new_an.custom is
called
Each time, ff_new_an.custom is called, this is being logged in the FlexFrame logs.
The current script is also archived in the subdirectory ff_new_an.custom.archive
within the image directory, the archived name contains the md5 sum of the script.
With the help of the FlexFrame logs it is therefore possible to reconstruct which scripts
had been called in the past:
cn1:~ # grep ff_new_an.custom /var/log/FlexFrame/log_cn1_2012-11-02 |
tail -2
2012-11-02 12:11:45 INFO [root,30499,cn1] ff_create_an_linux.sh
Archiving /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
Software and Hardware Upgrade and Maintenance
Administration and Operation 317
11.x86_64/ff_new_an.custom to /FlexFrame/volFF/os/Linux/FSC_1.0A00-
017.SLES-
11.x86_64/ff_new_an.custom.archive/ff_new_an.custom.c61da8adb37d70bdcf7f
cb44ec476102
2012-11-02 12:11:45 INFO [root,30499,cn1] ff_create_an_linux.sh
Executing custom script /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/ff_new_an.custom
11.6 Maintenance of the Application Nodes - Hardware
11.6.1 Changing BIOS Settings for Netboot
For detailed information on setting BIOS parameters refer to the hardware
documentation. Look at some of the configuration examples below. The exact procedure
may differ depending on the BIOS version of your hardware. For newer server models
refer to the corresponding documentation called "FlexFrame® HW Characteristics
Quickguide ("Hardwaresteckbrief")" – in these documents (delivered with the Service CD
of a FlexFrame version and also on the FTS Extranet on
https://partners.ts.fujitsu.com/com/service/ps/solutions/flexframe) the server specific
properties relevant for installing the server within a FlexFrame environment are collected.
11.6.2 Replacing a Network Card – Application Node
There are two cases for replacing a network card on an Application Node:
The card has failed and you want to replace it by a card of same type.
The network card is replaced by a card of another technology, e.g. 1GBit to 10GBit
migration or vice versa.
If a network card on an Application Node fails and is replaced, further actions are
necessary to ensure proper operation.
During their boot processes, PRIMERGY nodes need to retrieve some information
through lower level protocols which are based on hardware addresses (MAC addresses).
The PRIMERGY's BIOS utilizes DHCP for that purpose.
The MAC address is the built-in, physical address which is unique to that network card.
Since there are two network cards configured for booting you may not even notice the
MAC address change of a single one, because the other card will be used for booting.
Once the server is up, operation is normal. For full redundancy – even for booting – we
recommend to adjust the configuration accordingly.
Software and Hardware Upgrade and Maintenance
318 Administration and Operation
On PRIMERGY blade systems, use the management blade to determine the server
blade MAC addresses.
On PRIMERGY non-blade systems, use BIOS or network card labels to determine
the MAC addresses.
To change the MAC addresses for booting, use the commands below. Replace the node
name and MAC addresses of the sample with your Application Node's name and noted
addresses:
control:~ # ff_an_adm.pl --op mac --name appnode5
--mac 00:e0:00:c5:19:41,00:e0:00:c5:19:41
MAC addresses have been changed.
The AN boot information has to be recreated. To do this exec:
ff_new_an.sh -B -n appnode5
control:~ # ff_new_an.sh -B -n appnode5
The first command changed the MAC addresses of the node within DHCP configuration
and LDAP and the second one recreated the AN boot information. No further intervention
is necessary.
Some software licenses are derived from information based on server specific
numbers, such as MAC addresses of network cards. On Linux, double-check
your SAP licenses after you have replaced a network card.
If a network card on an Application Node is replaced for technology change reasons the
following steps are necessary:
1. Remove the Application Node from the FlexFrame landscape with ff_an_adm.pl --op rem
2. Remove the 1GB network cables of the Application Node
3. Remove the 1GB NIC cards from the Application Node
4. Plug in the 10GB NIC cards into the Application Node (see corresponding
"FlexFrame® HW Characteristics Quickguide ("Hardwaresteckbrief")")
5. Install the Application Node with ff_an_adm.pl
(for details see the corresponding "FlexFrame® HW Characteristics Quickguide
("Hardwaresteckbrief")")
Example:
ff_an_adm.pl --op add --name sno2rx1 --type RX600S4 --pool
pool2 --group default --swgroup 1 --ospath Linux/FSC_5.1A00-
005.SLES-11.x86_64 --host 25 --mac
00:1b:21:24:4f:5d,00:1b:21:24:4f:5c -–10gbit
6. Plug in the network cables according to the wiring output given by ff_an_adm.pl
Software and Hardware Upgrade and Maintenance
Administration and Operation 319
7. Call ff_new_an.sh to generate an image for the new Application Node
8. Boot the Application Node
11.6.3 Replacing a Switch Blade
There are two cases for replacing a switch blade:
In case of an HW error you have to replace one switch blade.
You can change the type of the used switch blade. In this case you have to replace
both switch blades.
In the first case (HW error) perform the following actions:
1. You have to replace the switch blade as described in the HW description of the
BX Server.
2. Create the configuration file for the switch blade with the function ff_bx_cabinet.pl –-op swb-config. See chapter 7.6.7 Getting Switch Blade
Configuration for a detailed description of the function.
3. Upload configuration to switch as displayed by ff_bx_cabinet.pl –-op swb-
config.
In the second case (change the type) perform the following actions:
1. Deactivate the usage of each server blade.
2. Remove the network cables from the switches.
3. Replace both switches.
4. Plug in the network cables.
5. Call ff_bx_cabinet_adm.pl --op swb-change to update LDAP Database.
6. Create the configuration file for the switch blade with the function ff_bx_cabinet.pl –-op swb-config. See chapter 7.6.7 Getting Switch Blade
Configuration for a detailed description of the function.
7. Upload configuration to switch as displayed by ff_bx_cabinet.pl –-op swb-
config.
8. Now you can use the server blades again.
11.6.4 Replacing Power Control Hardware
This section describes what to do after the replacement of power control hardware,
depending on your Application Nodes and Control Nodes hardware.
Software and Hardware Upgrade and Maintenance
320 Administration and Operation
For instructions on replacing the power control hardware, please refer to the
documentation delivered with your hardware.
11.7 Maintenance of ESXi Servers
11.7.1 BIOS Updates
To update the BIOS of a PRIMERGY server that is used as an ESXi server, use the
standard BIOS update procedure for PRIMERGY servers based on a bootable USB
memory stick. Please note that the method using ASPs (Autonomous Primergy Support
Packages) is not applicable in this case.
11.7.2 Replacing a Network Card - ESXi Server
If a network card of an ESXi server fails and is replaced, further actions are necessary to
ensure proper operation.
PRIMERGY servers used as ESXi servers in FlexFrame get the address used for ESXi
management from a DHCP server running on the FlexFrame Control Center. The DHCP
server determines the correct IP address based on the requester's MAC address.
The MAC address is the built-in, physical address which is unique to that network card.
As the MAC address is changed after replacing a network card, we recommend adjusting
the configuration accordingly,
● On PRIMERGY blade systems, use the management blade to determine the server
blade MAC addresses.
● On PRIMERGY non-blade systems, use BIOS or network card labels to determine
the MAC addresses.
To change the MAC addresses for DHCP, use the command below. Replace the node
name and MAC addresses of the sample with your ESXi server's name and noted
addresses:
control:~ # ff_hn_adm.pl --op mac --name esxnode5
--mac 00:e0:00:c5:19:41,00:e0:00:c5:19:42
The program changes the MAC addresses of the node within DHCP configuration and
LDAP. No further intervention is necessary.
11.7.3 ESXi Updates and Patches
VMware provides patches and updates for ESXi servers. While updates are for
maintenance releases, patches address critical security fixes or urgent bug fixes. An
update or patch can include a new build of firmware, an update of VMware Tools, or an
update of the vSphere Client.
Software and Hardware Upgrade and Maintenance
Administration and Operation 321
Please note that patches and updates of VMware Tools cannot be applied using the
standard VMware procedures. The FlexFrame Application Node images contain the
VMware Tools operating system specific packages (OSPs). For a description on how to
upgrade these tools refer to section 11.5.9.6 .
VMware provides several tools for installing updates and patches to ESXi hosts:
● esxcli - command-line utility
● vCenter Update Manager - automates patching and updates
The esxcli utility is part of the vSphere CLI, which is available on the FlexFrame Control
Nodes.
For more information on these tools as well as further usage hints and recommendations
please refer to the appropriate VMware documentation.
11.8 Maintenance of Other FlexFrame Components
11.8.1 NetApp Storage
ONTAP Patches
Before applying a new patch to the Filer's ONTAP make sure that it is supported for the
FlexFrame concept. In doubt contact the Fujitsu support.
Download the tar file(s) for the new ONTAP onto one of your Control Nodes.
Create a temporary directory e.g. in /FlexFrame/volFF/FlexFrame/stage/ontap
(create the directory if it does not exists) and extract the tar file there.
Proceed as described in the manual or README information of that patch.
Before you activate the new patch by rebooting the Filer, make sure that no SAP system
or Database is busy. You do not need to reboot the Application Nodes, however we
recommend double-checking the mount points after the Filer is up again. If an Application
Node should get stuck, reboot it.
11.8.2 Switches and Switch Blades
11.8.2.1 Firmware Update
It may be possible that an update of the switch firmware is necessary. You will find a
detailed description in the manual of the specific vendor.. For Switch Blades have a look
at the corresponding user manual.
11.8.2.2 Backup of Switch Configurations
The switch configuration may be different to that at the time of installation due to the
operation of a FlexFrame landscape. New Application Nodes and switches may be
Software and Hardware Upgrade and Maintenance
322 Administration and Operation
added, ports configured and so on. For system backup, there are a lot of well known
products and programs.
To simplify backing up the configuration of all switches used within the FlexFrame
landscape the program /opt/FlexFrame/bin/ff_save_switch_config.pl should
be used. It is designed to be run by cron(8) and recommended to call it once a day.
Default is to save the switch configurations of all FlexFrame switches into the default
backup directory /tftpboot/saved_switchconfigs. Only human readable format is
selected for backup. Startup configuration (the configuration the switch reads when
booting) is preferred before the running configuration for backup.
Synopsis
ff_save_switch_config.pl [--name <switch name>]
[--dir <backup directory>]
[--running]
[--ip <control node ip>]
Options
--name <switch name>
Defines the name of the switch to be backed up. Keep in mind to use the name
listed from ff_swgroup_adm.pl respectively ff_bx_cabinet_adm.pl. If omitted all
Switches are backed up.
--dir <backup directory>
Defines the directory where to store the backup. If omitted the backups are stored
into /tftpboot/saved_switchconfigs.
--running
Defines the running configuration to be preferred before the startup configuration for
backup. If omitted the startup configuration is preferred before the running
configuration for backup.
--ip <control node ip>
Defines the control lan ip address of the control node running the netboot service. If
omitted the ip is determined by the program itself if needed.
11.8.2.3 Restore of Switch Configurations
To restore a backed-up switch configuration clear the actual configuration on the switch
then use copy and paste for the switch config commands. Don’t forget to copy the
Software and Hardware Upgrade and Maintenance
Administration and Operation 323
restored new running switch configuration on the switch as startup-configuration for the
next reboot.
Be aware, the backed-up switch configuration may not be the last effective
switch configuration because of changes applied after backup.
You can get an actual configuration according to LDAP
for switch blades using
ff_bx_cabinet_adm.pl --op swb-config ... (see 7.6.7)
for switch groups using
ff_swgroup_adm.pl --op gen-config ... (see 9.4)
11.8.3 DNS Servers
If the customer wants to change IP addresses of his DNS servers he also has to change
the corresponding FlexFrame configurations:
The DNS servers of each pool can be modified with the command ff_pool_adm.pl (see
4.5).
The DNS servers of the Control Center can be modified by editing the “forwarders” in
the file /etc/named.conf . Originally the DNS Servers of the Control Center came into
FlexFrame by specifying them in the “Miscellaneous Control Center Settings” in the
Management Tool.
Any other DNS server (e.g. explicitly indicated DNS servers in hardware configuration
files), which manually has been entered by the customer, is not under FlexFrame control
and it is the customers responsibility to modify these addresses at the right place.
11.8.4 Third Party Products
If the customer needs to install third party software on the Application Nodes, the
maintenance procedures described in this manual must be followed. Installation of
software or patches may have to be performed again after installing a maintenance
image shipped by Fujitsu.
Software and Hardware Upgrade and Maintenance
324 Administration and Operation
If you have installed additional 3rd
party software to your FlexFrame landscape,
please make sure to back up this software components and their configuration
data for later restore. Please consider: If you install a new FlexFrame image, the
whole file system and directories are overwritten by a complete new root file
system image. Afterwards, any 3rd
party software and configurations have to be
re-installed or restored.
Installing third party software on Control Nodes or Application Nodes may cause
functional restrictions and other malfunctions of the FlexFrame system software.
Fujitsu shall have no liability whatsoever whether of a direct, indirect or
consequential nature with regard to damages caused by the third party software
or its erroneous installation.
11.9 FlexFrame Agents
For information on the FlexFrame Agents please refer to the "FlexFrame Agents
Installation and Administration" manual, chapter 2 "First Steps". Here you will find
detailed information on installation requirements, installation, configuration, the
FlexFrame Agents web interface and the FlexFrame Domain Manager.
11.10 Project specific changes
As mentioned in chapter 2 “FlexFrame Architecture” the FlexFrame solution consists of
both hardware and software. Most of the components are seen as an appliance: they
have well defined purposes and must work together in order to achieve a smoothly
running FlexFrame landscape. Normally only a well defined set of software and hardware
(see section 2.2 “Hardware and Software”) is allowed.
However there might be the need to change or modify components, or configuration
details in order to satisfy project specific requirements.
These special changes – project specific changes - mean that your configuration is
outside the FlexFrame standards and lead to a project specific FlexFrame.
These project specific changes may affect the operation of the entire FlexFrame
landscape. An example: Consider you run a SFG (‘Request for Special Release’) for a
server which is unsupported in standard FlexFrame landscapes. Additionally you
changed some configuration in order to use some nice features of this server. Everything
runs fine. But: You have always to remember these changes. You have to tell the support
team (or other professionals working on this FlexFrame) about this. Without knowing
these project specific modifications a patch installation or upgrade installation may fail or
lead to unsatisfying results.
In order to keep track of these project specific changes a “Project Activities
Communication Interface” has been established. It creates a communication channel
from human beings to human beings (and to FlexFrame commands).
Software and Hardware Upgrade and Maintenance
Administration and Operation 325
The most important command of this interface is ff_projectinfo.sh which will be
introduced in the following chapter.
11.10.1 The ff_projectinfo.sh command
Name
ff_projectinfo.sh - add entry to projectinfo database file
Synopsis
ff_projectinfo.sh -n <name> \
-t <text> | -r <text file> \
[-c <customer specific SFG#> ] \
[-s <severity> ] \
[-C ] \
[-v] [-d]
-n: (name)identify yourself (email address preferred).
Mandatory!
-t: (text) Text to be added (if it is a short one)
-r: (read text file) Text to be added will be read out of
this file
-c: (custom) If a custom specific Special Release
(Sonderfreigabe) has been applied this option has to
name the SFG number
-s: (severity) classification of project specific change
where severity = ‘info’ or ‘warning’ or ‘error’
-C: (CLEAR) change state of projectinfo database file to
READ
-v: (verbose)
-d: (debug)
Description
This script and the projectinfo database file are for use by FlexFrame certified
personnel only.
The objective of the project activities interface is to store and share information about
project specific activities at the FlexFrame landscape.
Software and Hardware Upgrade and Maintenance
326 Administration and Operation
The ff_projectinfo.sh command is part of the project activities interface for the
FlexFrame solution. It helps to establish a reliable communication between the
professionals. This interface comprises
· this ff_projectinfo.sh command
· the FlexFrame projectinfo database file
· the ff_projectinfo_detector.sh
The ff_projectinfo.sh command serves as the only interface that creates or fills the
FlexFrame projectinfo database file.
Options
-n <name>
This option is mandatory: you need to identify yourself. The preferred argument is your
email address. If you want to use a name containing spaces you have to embed it into a
pair of double quotes.
-t <text>
Choose this option if the information you want to add is a short one and can be typed in
the command line. You will have to embed the text into a pair of double quotes (if it is not only a single word which seems unlikely). For longer texts use the -r option. But: Usage
of either -t or -r option is mandatory.
-r <text file>
Choose this option if you want to add information you prepared previously, or which is simply too long to be used at command line with the -t option. But: Usage of either -t or
-r option is mandatory. Note: You can add any text you like as long as file(1)
recognizes it as "text". Do not try to add core, dump, tar, files etc.
-c <custom specific Special Release number>
If a custom specific Special Release (Sonderfreigabe) has been applied to the system
this option is mandatory. It has to name the SFG number which you find in the Special
Release documents. It has typically a SFG-20??-??? appearance.
-s <severity>
This option is dedicated to classify the information you add: It helps the professionals
working later on the system to understand the significance resp. severity. The severity
option accepts the following values: info or warning or error. Without giving an explicit
severity a severity of info will be assumed.
-C
Software and Hardware Upgrade and Maintenance
Administration and Operation 327
This option clears the default unread state of the projectinfo database file i.e. it sets
the state of the projectinfo database file to read.
Explanation: The projectinfo database file can have two states: unread or read.
Each appending of information (i.e. each successful call of the ff_projectinfo.sh
command without the -C option) will set the status to unread. This signals the
professionals the necessity to read it. Note: A FlexFrame with an unread projectinfo
database file is not entitled for an upgrade installation. Use the -C option not until you run
the FlexFrame upgrade. For details see ff_project_detector.sh(8) and the
“FlexFrame - Upgrade Guide”. This state of read will remain for a limited period only.
-v
The command will give some more detailed information during its work.
-d
The command will write detailed debug information into FlexFrame's default debug files.
More details and comprehensive use cases are available in the online manual pages of
ff_projectinfo.sh and ff_project_detector.sh.
Troubleshooting
328 Administration and Operation
12 Troubleshooting
12.1 Locking of Flex Frame Administration Commands
FlexFrame administrative commands stored at /opt/FlexFrame/bin use a resource
locking mechanism to prevent data corruption.
The implementation on the control nodes uses lock files on a shared file system mounted
from the common NAS. Each lock file represents a resource access. Each administrative
command uses typically only a single lock at a time to prevent deadlocks. The lock files
contain information about:
the locking user,
the executing host and the process id,
the timeout set,
the date and time of day when the lock expires,
the name and command line arguments of the locking process.
If a command tries to set a lock which is already set the command will retry to set the lock
for five times with a delay of one second. If the lock could not be set the command will
abort with a notice to user.
To get a robust locking mechanism locks will be removed if a timeout of 300 seconds was
reached AND the locking process no longer exists. If the locking process is still running,
an error message will be displayed by newly called program. The error message and the
exit code define the type or error as shown below.
These locking conditions get detected:
Exit Code Condition Description
0 normal
operation
All fine, locking works as expected. No resource collision with
other processes.
2 normal
operation
There is already a lock and the timeout is not reached.
3 Error IO error setting or unsetting a lock. This should not happen and
points to hanging mounts or filled up file systems.
Troubleshooting
Administration and Operation 329
Exit Code Condition Description
4 Error Lock set, timeout reached but locking process is still alive. May
be the locking process hangs. Only an administrator is able to
decide if locking process works or hangs. An administrator may
have to clear the lock manually
5 Error Lock set but lock file is empty, therefore no further checks are
possible. This should not happen as FlexFrame locking writes
data to lock files. An administrator has to clear the lock
manually.
Manual administration is necessary for all three error conditions. The administrator has to
check for source of trouble and resolve it. Some examples:
Resource File path
LDAP /FlexFrame/volFF/FlexFrame/locks/\
ldap_lock
ff_new_an.sh /FlexFrame/volFF/FlexFrame/locks/\
ff_new_an.sh_lock
ff_install_an_linux_images.sh /FlexFrame/volFF/FlexFrame/locks/\
ff_install_an_linux_images.sh_lock
The content of the lock files:
User: <login name>
Pid: <id of locking process>
Host: <node name where locking process lives>
Timeout: <time set in seconds> (lock valid till <date and time>)
<command line of locking process>
12.2 Script Logging in FlexFrame
All CN scripts in /opt/FlexFrame/bin log into one file, which is
/var/log/FlexFrame/log_(TIMESTAMP) by default, e.g.
/var/log/FlexFrame/log_2010_05_31. This file is changed daily.
Messages with priority greater than DEBUG (i.e. INFO, WARN and ERROR) are written
into this logfile.
Troubleshooting
330 Administration and Operation
All CN scripts in /opt/FlexFrame/bin debug (if called with option DEBUG) into one
file, which is /var/log/FlexFrame/debug_(TIMESTAMP) by default, e.g.
/var/log/FlexFrame/debug_2010_05_31 . This file is changed daily.
All messages are written into this debug file.
To get further debug information, all Perl scripts can be called with option –debug, as
well as some Shell scripts can be called with option –d .
Exception : The installation scripts ff_init_conf.pl, ff_post_conf.sh and
ff_slapd_init.sh log/debug to /opt/FlexFrame/init_conf/.
A log entry contains timestamp, priority, username, process ID, hostname, script name
and message text. It looks like this:
2009-04-22 12:29:52 INFO [root,9061,jer1cn1] ff_pool_adm.pl.sh
log example
Two configuration files, log4sh.conf and log4perl.conf reside under
/opt/FlexFrame/etc/.
The destination for Logging/Debugging can be changed in these configuration files by
modifying the corresponding filename lines.
You have to change the destination filenames in both configuration files.
Example:
If you want all scripts (excluding the exceptions) to log into
/FlexFrame/volFF/FlexFrame/log_(TIMESTAMP) you have to change
log4perl.appender.LogFileApp.filename=/var/log/FlexFrame/log
to
log4perl.appender.LogFileApp.filename=/FlexFrame/volFF/FlexFrame/l
og
in log4perl.conf
and
log4sh.appender.A1.File = '/var/log/FlexFrame/log'
to
log4sh.appender.A1.File = '/FlexFrame/volFF/FlexFrame/log'
in log4sh.conf
Troubleshooting
Administration and Operation 331
Attention:
The directory of the log-/debugfiles (in the example above
/FlexFrame/volFF/FlexFrame) has to exist.
The user is responsible for the back up/delete of older logfiles.
12.3 RBAC Logging with Syslog
It is possible to send the RBAC Logs to Syslog.
To do so, you have to make some changes in the file /opt/FlexFrame/etc/log4perl.conf :
- You have to add “SyslogApp” at the end of the first line
- You have to activate the last 3 lines of the file by removing the “#” .
- You have to adapt the facility of SyslogApp, e.g. to local1 , so that it fits to your
adjustments in /etc/syslog-ng/syslog-ng.conf
12.4 Log Files
The default location for logging information of the FlexFrame scripts is /var/log/FlexFrame on the Control Nodes.
Additional information can be found under /var/log/FlexFrame/tmp .
For easy gathering of log file information on all Application Nodes in FlexFrame, it is not
necessary to login on each Application Node because all log files are directly readable at
the Control Nodes on the mounted NAS system.
To elevate the admin to read these log files it is useful to create symbolic links on the
Control Nodes in the directory /var like this:
control1:/ # cd /var
control1:/var # ls -ld log*
drwxr-xr-x 20 root root 1440 May 2 14:01 log
lrwxrwxrwx 1 root root 58 Apr 29 17:28
log_pool1_klinge1 -> /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/var_img/var-ac10020e/log
lrwxrwxrwx 1 root root 58 Apr 29 17:29
log_pool1_klinge2 -> /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/var_img/var-ac10020f/log
lrwxrwxrwx 1 root root 58 Apr 29 17:30
log_pool1_klinge3 -> /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/var_img/var-ac100210/log
Troubleshooting
332 Administration and Operation
lrwxrwxrwx 1 root root 58 Apr 29 17:30
log_pool1_klinge4 -> /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/var_img/var-ac100211/log
lrwxrwxrwx 1 root root 58 Apr 29 17:30
log_pool1_klinge5 -> /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/var_img/var-ac100212/log
lrwxrwxrwx 1 root root 58 Apr 29 17:31 log_otto_RX300-
01 -> /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/var_img/var-ac100113/log
lrwxrwxrwx 1 root root 58 Apr 29 17:33 log_otto_RX300-
02 -> /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-
11.x86_64/var_img/var-ac100114/log
control1:/var #
Troubleshooting
Administration and Operation 333
To read the /var/log/messsages from Application Node klinge5 in pool pool1
invoke this:
control1:/ # cd /var
control1:/var # less log_pool1_klinge5/messages
Or enter the log directory from this Application node and look for log files:
control1:/ # cd /var/log_pool1_klinge5
control1:/var/log_pool1_klinge5 # ls -lrt|tail
-rw-r--r-- 1 root root 278 May 4 16:23 log.scagt
-rw-r--r-- 1 root root 174 May 4 16:24 log.vvagt
-rw-r--r-- 1 root root 257 May 4 16:24 log.statusagt
-rw-r--r-- 1 root root 612 May 4 16:28 ntp
-rw-r--r-- 1 root root 5542 May 4 16:35 warn
-rw-r--r-- 1 root root 1696 May 4 16:35 auth.log
-rw-r--r-- 1 root root 484 May 4 18:16 log.busagt
drwxr-xr-x 2 root root 4096 May 4 18:19 sa
-rw-r--r-- 1 root root 27622 May 4 18:19 messages
prw------- 1 root root 0 May 4 18:24 psadfifo
12.5 Network Errors
In most cases, network problems are most times configuration mistakes. All switches are
configured to send SNMP traps and log via syslog to Control Nodes. To see what happened, look at the Control Nodes' /var/log/messages. Any conditions reported by
the switches can be found here. For SNMP traps look at the FlexFrame Agents support
database. The FlexFrame Agents collect all kinds of SNMP traps for the entire FlexFrame
environment.
12.6 NFS Mount Messages
During start/stop procedures of SAP instances, NFS mount failures for /usr/sap/SYS
and /oracle/db_sw can be seen in /var/log/messages. These messages are not
FlexFrame-specific failures, but may occur on any system that has /usr/sap and
/oracle in its automount maps. These messages are caused by the way SAP is linking
binaries during software development process.
12.7 LDAP Error Codes and Messages
LDAP failures may have various reasons. To determine the reason, first have a look at
the Control Node's /var/log/messages. If slapd does not report anything you may
have to increment the log level configured with /etc/openldap/slapd.conf. Log
levels are additive, and available levels are:
Troubleshooting
334 Administration and Operation
Log level Meaning
1 Trace function calls
2 Debug packet handling
4 Heavy trace debugging
8 Connection management
16 Print out packets sent and received
32 Search filter processing
64 Configuration file processing
128 Access control list processing
256 Stats log connections/operations/results
512 Stats log entries sent
1024 Print communication with shell backends
2048 Entry parsing
Most LDAP problems are related to access control and connection management.
If you have changed the log level send a -HUP to the slapd to advice it to reread
configuration files (use pkill -HUP slapd).
Another reason may be the LDAP client configuration. Check the following issues:
Does the client use the correct server address?
Look at the client messages file to get this information.
If ldapsearch -x on Linux and works properly the problem may be the nsswitch
configuration. These are the top most problems.
Troubleshooting
Administration and Operation 335
When reporting problems with the administration scripts, please provide the
following information:
The error message of the admin tool
A dump of the LDAP database (ldapsearch -x > <file name>)
The configuration files /etc/openldap/slapd.conf and
/FlexFrame/volFF/FlexFrame/ldap/common/slapd.acl.conf
12.8 LDAP and Cache Coherence
While using command scripts to administer your FlexFrame environment, it is possible
that you do not see the changes on Application Nodes immediately. The reason is the
usage of a cache daemon (nscd) which refreshes the cache content in defined time
periods.
There are different caches for different types (e.g. passwd, group, etc.) The configuration of the cache properties is done in /etc/nscd.conf (default). Normally you must not
change the FlexFrame default configuration. Please ask professional service collegues
before doing any change.
You can display your current configuration with nscd –g.
For further information see the corresponding manpages (nscd, nscd.conf).
12.8.1 Linux
To force the refresh of the cache content use nscd –i or nscd restart.
12.9 Start/Stop Script Errors
While executing the SAP service start/stop scripts, several errors may occur for arbitrary
reasons. The messages are grouped by the severity of the message.
The scripts write three types of logs (/FlexFrame/script/logs)
sapservice.log
The log is always available and logs the requests. It contains only the requested
command and the exit state of the request
sap<type>_<SID>_<action>.log
That log contains the messages also written to STDOUT/STDERR
sapservice.<nnn>.debug.<SID>.<type>.<instnr>.action>
That log is only written if option -d is set. It contains a detailed logging of the
course of actions.
The message text below uses ‘%%’ to indicate there will be a variable output in the
message. There are up to four varying strings in a message.
Troubleshooting
336 Administration and Operation
If you are creating a debug file you are requested to delete debug files by yourself. Otherwise they resist in /FlexFrame/script/log forever.
12.9.1 Severity ‘INFO’
Messages from severity ‘INFO’ should show the progress of a request and they are always written to STDOUT and to the debug file if option -d is set.
12.9.2 Severity ‘WARN’
Messages from severity ‘WARN should give you a hint to possible configuration errors.
They are always written to STDOUT and to the debug file if option -d is set.
Most of the WARN messages are not critical but you should check if such a message is
displayed.
12.9.3 Severity ‘ERROR’
Messages from severity ‘ERROR’ are always written to STDOUT and to the debug file if
option -d is set. You have to check it and resolve the issue which leads to the error
message. With opcode ‘cleanup’ sometimes messages with severity ‘ERROR’ are shown
but can be ignored.
12.9.4 Severity 'DEBUG'
Debug messages are only printed to the debug file if option -d is used. Those message
are only written to support diagnostics.
12.10 Script Debugging
12.10.1 Shell Scripts
For many FlexFrame scripts, debugging is activated by default. If debugging is not active,
it can be activated with the option –d:
ff_netscan.sh -d
The default location for the debug log file is /var/log/FlexFrame
Further on, shell scripts can be traced using the option -x.
For details, see the man page of the shell in use (man sh; man bash; etc.)
sh -x ff_netscan.sh
Troubleshooting
Administration and Operation 337
12.10.2 Perl Scripts
Before debugging, activate the logging functions of the shell by calling script (For
details, see man script).
The debugger is called using
perl -d <script_name>
Functions:
h
Help
x <expression>
Evaluate an expression (hash/array) and display the result
p <string>
Returns strings
s
Execute next command; follow subroutines
n
Execute next command; skip subroutines
b <line_number>
Set break point at line <line number>
q
Quit.
For further information, see the man page of perldebug.
12.11 Debugging the Linux Kernel
For the Linux Kernel, debugging is disabled by default. There are several approaches to
collect debugging information. Everything is only described briefly; please contact Fujitsu
support for further instructions.
12.11.1 Netconsole
If the Linux kernel detects an internal error, it tries to write some information about the so
called “Oops” into the messages file.
Troubleshooting
338 Administration and Operation
If an error completely crashes the kernel (which is known as a kernel panic), this
information may be generated but cannot be written to the disk or the NFS mount.
To get this information anyway, activate the netconsole feature to redirect kernel logging
over the network.
If the netconsole output does not provide the required information, you should configure
the crash dump functionality.
Information about using the netconsole can be found in the Linux Kernel documentation
in /usr/src/linux/Documentation/networking/netconsole.txt.
Please note that due to the use of bonding interfaces in FlexFrame there may be some
limitations, netconsole may not work properly with the bonding slaves. It is recommended
to use a dedicated interface.
12.11.2 Capturing Crash Dumps
12.11.2.1 Common Restrictions for Taking Crash Dumps
If taking crash dumps is configured and activated, the process of writing the dump
must not be interrupted, so the Linux HA cluster and FlexFrame Agent power shutdown
features must be disabled for this specific node, for example by pulling the IPMI cable or
setting appropriate timeouts. On the Control Nodes, STONITH should be disabled if the
crash dump cannot be written within 45 seconds which is very likely.
This can be done by disabling the “Enable STONITH” checkbox of the “linux-ha” proper-
ties in the Heartbeat GUI interface.
Amongst others that is one reason for not activating crash dump capturing by default.
12.11.2.2 Kernel Crash Dump Capturing (Kdump)
Since SLES10, the implementation how to take crash dumps was changed to the kdump
method. If a kernel panic occurs, a special kdump-kernel will be directly executed by the
running kernel and during the boot process of this kernel, the dump of the whole system
memory will be written to a sparse file in /var/log/dump.
This feature is well integrated in FlexFrame Application Nodes and may be activated by
specifying a kernel command line option in the PXE configuration file:
Open the corresponding file in /tftpboot/pxelinux.cfg
Append crashkernel=128M@16M to the APPEND line
Reboot the corresponding Application Node(s)
12.11.2.3 Forcing a Crash Dump
There may be situations where the kernel does not initiate a crash dump automatically.
Troubleshooting
Administration and Operation 339
You can manually start a crash dump by using the sysrq shortcuts.
Activate sysrq as follows:
cn1:~ # echo 1 >/proc/sys/kernel/sysrq
To make this setting persistent, set ENABLE_SYSRQ="yes" in
/etc/sysconfig/sysctl.
Now a manual's crash dump capture can be manually initiated by pressing
Alt+SysRq+c altogether on the keyboard. On most keyboards, the SysRq key is equal
to the PrintScrn key.
If the graphical desktop is running, it is recommended to switch to a text console by pressing Ctrl+Alt+F10 before starting the dump, otherwise it might not be possible to
see anything while the kdump-kernel is being executed and the dump is being written.
12.12 Activate core dumps on CN or AN
To investigate problems (e.g. bus errors) with Linux commands it is sometimes helpfull to
get a ‘core dump’. Normally the core file creation is disabled by default.
To activate core dumps in the current running shell use the following command:
In a ‘bash’ shell use
an_Linux# ulimit –c unlimited
In a ‘csh’ shell use
rx300-15 /home_sap/sapadm> unlimit coredumpsize
Now run the faulty command within this shell. The core dump can be found in the
directory /var/log/coredumps.
The filename follows the format:
core-<executable filename>-<time of dump in seconds>
Troubleshooting
340 Administration and Operation
If it is necessary to enable core file creation for a complete login session, you have to globally enable core dump writing .Change the file /etc/security/limits.conf on
the CN or /FlexFrame/volFF/os/Linux/FSC*/root_img/etc/security/limits.conf
for an Application Node and add the line
* soft core unlimited
In the next login session the creation of core dump files is enabled. To check use the command ulimit –a for a bash shell environment or the command limit for a csh
shell environment.
To disable this functionality remove the above entry from the limits.conf file.
12.13 Huge Amount of getattr Calls
If you expect a very huge amount of getattr NFS calls on your network or on your NAS
system, please check the following issues:
Application Node Image
Please use the current application node image available for your FlexFrame release. We
have done a lot of smooth improvements over the last years to limit the getattr calls within
the image to an absolutely needed minimum.
SAP-HANA
If using SAP HANA within FlexFrame, please check the SAP HANA profile.
The file sapprofile.ini under $SAP_RETRIEVAL_PATH (e.g.
/usr/sap/HD1/HDB02/hdb02hd1-001-se) should contain an entry like
HDB/NameServer/RWLockPath=/dev/shm
Please note, that this entry should contain a destination, which is not residing on an NFS
file system.
For more information please look at the FlexFrame documentation “FlexFrame® –
Installation Guide for SAP Solutions” (in the ‘post installation’ section of the HANA
installation)
Miscellaneous
Please check, if you are running additional tools or services (e.g. Citrix Server, Backup
solutions, Tools analyzing file system content by accessing all directories and files, etc..)
on the FlexFrame internal network or against your NAS system used in FlexFrame.
If possible, please turn off everything except the FlexFrame base functionality to find out,
if it is really FlexFrame, who caused the high number of getattr calls.
Abbreviations
Administration and Operation 341
13 Abbreviations
ABAP Advanced Business Application Programming
ACC Adaptive Computing Controller
ACI Adaptive Computing Infrastructure
ACPI Advanced Configuration and Power Interface
APM Advanced Power Management
APOLC Advanced Planner & Optimizer Life Cache
CCU Console Connection Unit
CIFS Common Internet File System
cDOT Clustered Data ONTAP
DHCP Dynamic Host Configuration Protocol
DIT Domain Information Tree
ERP Enterprise Resource Planning
EULA End User License Agreement
FA FlexFrame Agents
FC Fiber Channel
FF FlexFrame
FF4S FlexFrame for SAP
FFO FlexFrame Orchestrator
FSC FlexFrame Software Container, name part used for
Images and CDs/DVDs
FTP File Transfer Protocol
IP Internet Protocol
KVM Kernel based Virtual Machine
LAN Local Area Network
LDAP Lightweight Directory Access Protocol
LUN Logical Unit Number
LVM Landscape Virtualization Management
Abbreviations
342 Administration and Operation
LVM Linux Volume Manager
MAC Media Access Control
MINRA Minimal Read Ahead
NAS Network Attached Storage
NDMP Network Data Management Protocol
NFS Network File System
NIC Network Interface Card
NVRAM Non-Volatile Random Access Memory
OLAP Online Analytical Processing
OLTP On-Line Transaction Processing
ONTAP Open Network Technology for Appliance Products
OSS Open Source Software
PXE Preboot Execution Environment
PY PRIMERGY
QA Quality Assurance
QS Quality of Service
RAID Redundant Array of Independent (or Inexpensive) Disks
RDBMS Relational Database Management System
SAN Storage Area Network
SAP BW SAP Business Warehouse
SAPGUI SAP Graphical User Interface
SAPOSS SAP Online System Service
SID System Identifier
SLD System Landscape Directory
SLES SUSE Linux Enterprise Server
SMB Server Message Block
SNMP Simple Network Management Protocol
SPOC Single Point Of Control
SVM Storage Virtual Machine
Abbreviations
Administration and Operation 343
TELNET Telecommunications Network
TFTP Trivial File Transfer Protocol
UDP User Datagram Protocol
UPS Uninterruptible Power Supply
vCC / vCN virtual Control Center / virtual Control Node
VLAN Virtual Local Area Network
VTOC Virtual Table Of Contents
WAN Wide Area Network
WAS Web Application Server
WAFL Write Anywhere File Layout
Administration and Operation 345
14 Glossary
Adaptive Computing Controller (ACC)
SAP system for monitoring and controlling SAP environments, mostly superceeded
by LVM nowadays.
Advanced Business Application Programming (ABAP)
Proprietary programming language of SAP.
Advanced Power Management
Advanced Power Management defines a layer between the hardware and the
operating system that effectively shields the programmer from hardware details.
Application Agent
A software program for monitoring and managing applications.
Application Node (AN)
A host for applications (e.g. SAP instances db, ci, agate, wgate, app etc.). This
definition includes Application Servers as well as Database Servers.
Automounter
The automounter is an NFS utility that automatically mounts directories on an NFS
client as they are needed, and unmounts them when they are no longer needed..
Blade
A special form factor for computer nodes.
BladeRunner
The working title for the solution part of SAP for FlexFrame.
BRBACKUP
SAP backup and restore tools.
Client LAN
Virtual network segment within FlexFrame, used for client-server traffic.
Clustered Data ONTAP (cDOT)
One of two modes of NetApp’s NAS operating system ONTAP. In clustermode either
a single controller pair or multiple controller pairs are connected (clustered) via a
back-end 10GbE network; The other mode is called 7-mode; this mode will gradually
be replaced by the clustermode.
Common Internet File System (CIFS)
A protocol for the sharing of file systems (same as SMB).
Computing Node
From the SAP ACI perspective: A host that is used for applications.
Control Agent
A software program for monitoring and managing nodes within FlexFrame.
Glossary
346 Administration and Operation
Control Center (CC)
The two Control Nodes (CN) of FlexFrame are also named as the FlexFrame Control
Center (CC). In this documentation the notation Control Node (CN) is used as a
synonym for Control Center (CC) and vice versa.
Control LAN
Virtual network segment within FlexFrame, used for system management traffic.
Control Node (CN)
A physical computer system, controlling and monitoring the entire FlexFrame
landscape and running shared services in the rack (dhcp, tftp, ldap etc.).
Control Station
A Control Node in an SAP ACI environment.
Dynamic Host Configuration Protocol (DHCP)
DHCP is a protocol for assigning dynamic IP addresses to devices on a network.
Dynamic Host Configuration Protocol server
A DHCP server provides configuration parameters specific to the DHCP client host,
required by the host to participate on the Internet.
Enterprise Resource Planning (ERP)
Enterprise Resource Planning systems are management information systems that
integrate and automate many of the business practices associated with the
operations or production aspects of a company.
Ethernet
A Local Area Network which supports data transfer rates of 10 megabits per second.
Fiber Channel (FC)
Fiber Channel is a serial computer bus intended for connecting high-speed storage
devices to computers.
Filer
Network attached storage for file systems of NetApp.
FlexFrame® (FF)
The name FlexFrame® is a generic term for both „FlexFrame
® for SAP
®“ and
„FlexFrame® Orchestrator“.
FlexFrame Agents (FA)
Central system management and high availability software component of FlexFrame
FlexFrame® Orchestrator (FFO)
This is the advancement of the Fujitsu solution FlexFrame for SAP and means a new
approach to offer enhanced functionality and features step by step and become more
and more independent from certain hardware and software components.
FlexFrame® for SAP
® (FF4S)
FlexFrame® for SAP
® is a Fujitsu solution and means a radically new architecture for
Glossary
Administration and Operation 347
SAP environments. It exploits the latest business-critical computing technology to
deliver major cost savings for SAP customers. FlexFrame for SAP is a joint project in
which the main partners are SAP, Network Appliance, Intel and Fujitsu.
FlexFrame internal LAN Switch
Network switches which are integral part of the FlexFrame hardware configuration
and which are automatically configured by the FlexFrame software.
Gigabit Ethernet
A Local Area Network which supports data transfer rates of 1 gigabit (1,000
megabits) per second.
Host name
The name of a node (assigned to an interface) that is resolved to a unique IP
address. One node can have multiple host names (cf. node name). In SAP
environments host names are currently limited to 13 alphanumeric characters
including the hyphen (“ - “). The first character must be a letter. In the SAP
environment host names are case-sensitive.
Image
In the FlexFrame documentation, “Image” is used as a synonym for “Hard Disk
Image”.
Internet Protocol Address
A unique number used by computers to refer to each other when sending information
through networks using the Internet Protocol.
Kernel based Virtual Machine (KVM)
Virtualization solution which turns the Linux kernel into a hypervisor.
Lightweight Directory Access Protocol (LDAP)
Protocol for accessing on-line directory services.
Local Area Network (LAN)
A computer network that spans a relatively small area. Most LANs are confined to a
single building or group of buildings. However, one LAN can be connected to other
LANs over any distance via telephone lines and radio waves. A system of LANs
connected in this way is called a Wide Area Network (WAN).
Local host name
The name of the node (physical computer); it can be displayed and set using the
command /bin/hostname.
Logical Unit Number (LUN)
An address for a single (SCSI) disk drive.
MAC address
Device identifier number of a Network Interface Card. In full: "media access control
address".
Glossary
348 Administration and Operation
MaxDB
A relational database system from mySQL (formerly ADABAS and SAPDB).
Media Access Control address
An identifier for network devices, usually unique. The MAC address is stored
physically on the device.
NAS system
Network Attached Storage of any vendor (in our context: NetApp Filer).
NDMPcopy
NDMPcopy transfers data between Filers using the Network Data Management
Protocol (NDMP).
Netboot
A boot procedure for computers where the operating system is provided via a
network instead of local disks.
Netweaver
SAP NetWeaver is the technical foundation of SAP solutions.
Network Appliance Filer
See “Filer”.
Network Attached Storage (NAS)
A data storage device that is connected via a network to one or multiple computers.
Network File System (NFS)
A network protocol for network-based storage access.
Network Interface Card (NIC)
A hardware device that allows computer communication via networks.
Node
A physical computer system controlled by an OS.
Node name
The name of a physical node as returned by the command uname -n. Each node
name within a FlexFrame environment must be unique.
Non-Volatile Random Access Memory
A type of memory that retains its contents when the power is turned off.
On-Line Analytical Processing (OLAP)
The use of analytical informationssystems via computer networks.
On-Line Transaction Processing (OLTP)
Transaction processing via computer networks.
OpenLDAP
An Open Source LDAP Service Implementation.
Glossary
Administration and Operation 349
Open Network Technology for Appliance Products (ONTAP)
The operating system of Network Appliance Filers.
Open Source Software
Software that is distributed free of charge under an open source license, such as the
GNU Public License.
Oracle RAC
A cluster database by Oracle Corporation.
Physical host
Name of a physical computer system (node).
Power-On Self Test
Part of a computer's boot process; automatic testing of diverse hardware
components.
Preboot Execution Environment (PXE)
An environment that allows a computer to boot from a network resource without
having a local operating system installed.
PRIMERGY
Fujitsu's i386-based server product line.
SAP Service
In FlexFrame: SAP Service and DB Services.
SAP service script
An administration script for starting and stopping an SAP application on a virtual host.
SAP Solution Manager
Service portal for the implementation, operation and optimization of an SAP solution.
SAPLogon
Front-end software for SAPGUI.
SAPRouter
Router for SAP services like SAPGUI or SAPTELNET.
Server
A physical host (hardware), same as node.
Service
A software program providing functions to clients.
Service type
The type of an application or service (db, ci, app, agate, wgate etc.).
Single Point of Control
In FlexFrame: One user interface to control a whole FlexFrame environment.
Glossary
350 Administration and Operation
Storage LAN
A virtual LAN segment within a FlexFrame environment, carrying the traffic to NAS
systems.
SUSE Linux Enterprise Server (SLES)
A Linux distribution by Novell, specializing in server installations.
Telecommunications Network
A terminal emulation program for TCP/IP networks such as the Internet.
Trivial File Transfer Protocol (TFTP)
A simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram
Protocol (UDP) and provides no security features. It is often used by servers to boot
diskless workstations, X-terminals, and routers.
TFTP server
A simple FTP implementation.
Virtual host
The name of the virtual host on which an application runs; it is assigned to a physical
node when an application is started.
Virtual Local Area Network (VLAN)
A VLAN is a logically segmented network mapped over physical hardware according
to the IEEE 802.1q standard.
Virtualization
Virtualization means the separation of hardware and processes. In a virtualized
environment (FlexFrame), a process can be moved between hardware nodes while
staying transparent to the user and application.
.
Administration and Operation 351
15 Index
abbreviations 341
Application
removing from monitoring by
FlexFrame Agents 264
stopping ans starting for upgades
using r3up 265
Application Node
hardware maintenance 317
software maintenance 289
Application Node Images
installation 291
Application Nodes 10
adding 88
administrating 83
displaying information on a specific
83
displaying information on all 87
listing 83
reactivating after power shutdown by
FlexFrame Agents 46
removing 92, 93, 94, 95, 191
state of 49
updating RPM packages 311
application services
starting and stopping 252
application software
upgrading 310
automounter concept 33
Blade Server Cabinets
adding 99
administrating 97
listing 97
removing 102
cloning of SIDs 229, 238
cluster
configuring 26
starting 26
status information 27
cluster file system, built 37
Cluster Status 53
common network configuration
parameters
display and change 172
constraints 15
Control Node
defect hardware 278
exchanging 278
failed hardware 278
hardware maintenance 278
OS damaged 278
replacing network card 279
software updates 269
Control Nodes 9
backup 275
backup/restore 274
core dump 339
crash dump 338
352 Administration and Operation
create new initrd 308
Debugging
Linux kernel 337
script 336
document history 2
ESXi server maintenance 320
BIOS update 320
replace network card 320
update and patches 320
FA Autonomous Agents 49
ff_change_id.pl 229
ff_clone_sid.pl 206, 207, 228, 230
ff_hosts.sh 72
ff_list_services.sh 251
ff_nas_adm.pl 129
ff_sap_shadowport.sh 245
ff_sap_upgrade.pl 246
ff_sid_adm.pl 196
ff_swgroup_adm.pl 162, 184
ff_swport_adm.pl 185
ff_user_adm.pl 74
Filer cluster 38
FlexFrame
architecture 5
backup with tape library 54
basic administration 42
general notes 5
network 51
FlexFrame configuration state,
displaying 47
FlexFrame landscape
accessing 42
power-off 43
power-on 42
FlexFrame Web portal 48
getattr 340
glossary 345
host
database 71
install kernel 308
install new initrd 308
installing new linux kernel 308
Jakarta Tomcat 49
kernel switching 309
LDAP
FlexFrame structure in 11
working with 13
LDAP and Cache Coherence 335
LDAP configuration 20
LDAP error codes and messages 333
LINUX Cluster Components /
Functionality 13
Linux kernel
install new kernel 272
mount software stage 271
reboot control node 272
update/install new 271
update/install new 306
linux-ha cli commands 27
Linux-HA cluster 49
Index
Administration and Operation 353
Linux-HA cluster on control center 13
linux-ha gui 29
linux-ha logfile 29
locking 328
log files 331
maintenance cycle 289, 290, 291, 293,
295, 296, 299
assigning, booting, maintaining (step
#2) 295
maintaining use cases 306
maintenance base image (step #1)
293
migrating remaining Application
Nodes (step #3) 302
overview 289
re-using maintenance images 304
reverting (step #3) 299
NAS
add new 129
changing LinkAggregate ports 139
display all configured 132
display configuration 134
remove 131
NAS systems
multiple 243
NAS systems configuration 129
netboot change BIOS settings 317
network 29
automounter concept 33
LAN failover 30
segments 30
switches 31
Network Appilance Filer 35
network card
replacing 317
network errors 333
NFS mount messages 333
notational conventions 1
ONTAP patches
installing 321
Partner Linux Driver Process (PLDP)
272, 309
PLDP 272, 309
pool
add to a NAS 135
remove from a NAS 135
pool default router
changing 69
pool DNS domain
changing 68
pool groups
removing 71
pools
adding 55, 58
adding a group 70
listing details 63, 66
removing 61
state of 49
pools and groups 58
power control hardware
354 Administration and Operation
replacing 319
related documents 2
remote administration 42
replace
ESXi network card 320
network card 317
power control hardware 319
switch blade 319
requirements 1
resource ff_manage 22
resource groups 18
resource netboot 22
SAN
basic layers 40
rules and restrictions with FlexFrame
41
SAP kernel
updates and patches 249
SAP service
scripts 257, 261
starting and stopping without root
privileges 255, 257
SAP service scripts
actions 259
return code 262
user exits 261
SAP services
administrating 250
display status 250
list status 251
starting and stopping multiple 262
SAP SIDs
adding instances 198, 205, 206, 211,
214, 218
modifying instances 198, 205, 206,
211, 214, 218
removing instances 198, 205, 206,
211, 214, 218
SAP SIDs
adding 198
listing 196
listing instances 196
modifying 198
removing 198
SAP SIDs
adding 205
SAP SIDs
removing 205
SAP SIDs
modifying 205
SAP SIDs
adding 206
SAP SIDs
removing 206
SAP SIDs
modifying 206
SAP SIDs
adding 211
SAP SIDs
removing 211
Index
Administration and Operation 355
SAP SIDs
modifying 211
SAP SIDs
adding 214
SAP SIDs
removing 214
SAP SIDs
modifying 214
SAP SIDs
adding 214
SAP SIDs
removing 214
SAP SIDs
modifying 214
SAP SIDs
adding 218
SAP SIDs
removing 218
SAP SIDs
modifying 218
SAP SIDs
adding 218
SAP SIDs
removing 218
SAP SIDs
modifying 218
SAP SIDs
adding 218
SAP SIDs
removing 218
SAP SIDs
modifying 218
SAP SIDs
cloning 228
SAP SIDs
listing instances 228
Sap system
handling 195
SAP system
upgrading 245
SAP systems
state of 50
score 15
script logging 329
scripts
ff_list_services.sh 251
ff_sap_shadowport.sh 245
ServerView
update 309
update agents 310
update RAID manager 310
ServerView update via RPM 270
service packs 271, 306
service switchover 265
severity
DEBUG 336
ERROR 336
INFO 336
356 Administration and Operation
WARN 336
shared operating system 9
SID instances
state of 50
simple resources 15
snapshot 37
software stage 306
spare nodes
pool-independent 81
specific configuration 18
start/stop script errors 335
state
of Applicatrion Nodes 49
of pools 49
of SAP systems 50
of SID instances 50
stickiness 16
stonith 23
storage systems administration 129
switch
add to a switch group 162
remove from a switch group 165
switch administration 162
switch blade
change blade uplink 106
changing name 103
changing password 104
changing type 103
getting initial configuration 105
replacing 319
switch configuration
backup 321
restoring 322
switch group
adding 174
adding an uplink 180
change host name 170
change password 169
deleting an uplink 183
extending an uplink 181
list configuration 166, 168
moving a blade cabinet to another
switch group 107
removing 179
switch port
add configuration 185
display configuration 188, 189
remove configuration 187
Third Party products 323
troubleshooting 328
update and maintenance 269
update PXE config file 308
updating RPM packages 273, 311
patch server connection 273, 311
patch server setup 273, 311
perfrom online update 274, 312
prepare online update 273, 311
updating vmware-open-vm-tools 312
upgrade FF landsacpe 269
Index
Administration and Operation 357
user and groups administration 74
volFF
unloading 249
volume layout 37
volumes
multiple 243
Recommended