FlexFrame for SAP 4 - manuals.ts.fujitsu.commanuals.ts.fujitsu.com/file/6789/ff4s-adm.pdf · MaxDB ® is a registered ... 2.6.3 Disaster Recovery – Turning Replica into Master

User Guide

FlexFrame™ for SAP® 4.0Administration and Operation

English

FlexFrame™ for SAP® Version 4.0

Administration and Operation

Edition March 2007 Document Version 1.0

Fujitsu Siemens Computers GmbH

© Copyright Fujitsu Siemens Computers GmbH 2007 FlexFrame™, PRIMECLUSTER™. PRIMEPOWER™ and PRIMERGY™ are trademarks of Fujitsu Siemens Computers SPARC64® is a registered trademark of Fujitsu Ltd. SAP® and NetWeaver™ are trademarks or registered trademarks of SAP AG in Germany and in several other countries Linux® is a registered trademark of Linus Torvalds SUSE® Linux is a registered trademark of Novell, Inc., in the United States and other coun-tries Java™ and Solaris™ are trademarks of Sun Microsystems, Inc. in the United States and other countries Intel® and PXE® are registered trademarks of Intel Corporation in the United States and other countries MaxDB® is a registered trademark of MySQL AB, Sweden MySQL® is a registered trademark of MySQL AB, Sweden NetApp®, Network Appliance®, Open Network Technology for Appliance Products™, Write Anywhere File Layout™ and WAFL™ are trademarks or registered trademarks of Network Appilance, Inc. in the United States and other countries Oracle® is a registered trademark of ORACLE Corporation EMC®, CLARiiON®, Symmetrix®, PowerPath®, Celerra™ and SnapSure™ are trademarks or registered trademarks of EMC Corporation in the United States and other countries SPARC™ is a trademark of SPARC International, Inc. in the United States and other coun-tries Ethernet® is a registered trademark of XEROX, Inc., Digital Equipment Corporation and Intel Corporation Windows®, Excel® and Word® are registered trademarks of Microsoft Corporation All other hardware and software names used are trademarks of their respective companies. All rights, including rights of translation, reproduction by printing, copying or similar methods, in part or in whole, are reserved. Offenders will be liable for damages. All rights, including rights created by patent grant or registration of a utility model or design, are reserved. Delivery subject to availability. Right of technical modification reserved.

Requirements Introduction

Administration and Operation

Contents 1 Introduction ..................................................................................................... 1 1.1 Requirements .................................................................................................... 1 1.2 Notational Conventions ..................................................................................... 1 1.3 Document History.............................................................................................. 2 1.4 Related Documents........................................................................................... 2 1.5 Special Hints for FlexFrame 4.0 ........................................................................ 2

2 FlexFrame Architecture .................................................................................. 3 2.1 General Notes on FlexFrame 4.0 ...................................................................... 3 2.2 Hardware........................................................................................................... 4 2.3 Software ............................................................................................................ 5 2.4 Shared Operating System ................................................................................. 8 2.4.1 Shared OS Boot Concept.................................................................................. 8 2.4.2 Control Nodes ................................................................................................. 10 2.4.3 Application Nodes ........................................................................................... 10 2.4.3.1 Linux Application Nodes.................................................................................. 10 2.4.3.2 Solaris Application Nodes ............................................................................... 11 2.5 Switch and NAS Configuration ........................................................................ 11 2.5.1 Adding a Switch to a Switch Group ................................................................. 11 2.5.2 Removing a Switch from a Switch Group ........................................................ 13 2.5.3 Listing a Switch Group Configuration .............................................................. 14 2.5.4 Changing the Password of a Switch Group..................................................... 16 2.5.5 Changing the Host Name of a Switch Group................................................... 16 2.5.6 Displaying/Changing Common Network Configuration Parameters ................ 17 2.5.7 Adding a Switch Group.................................................................................... 19 2.5.8 Removing a Switch Group............................................................................... 22 2.5.9 Adding a Switch Port Configuration................................................................. 22 2.5.10 Removing a Switch Port Configuration............................................................ 24 2.5.11 Displaying a Switch Port Configuration ........................................................... 25 2.5.12 Adding a New NAS.......................................................................................... 26 2.5.13 Configuring SNMP traps for NetApp Filers...................................................... 29 2.5.14 Displaying All Configured NAS........................................................................ 29 2.5.15 Displaying NAS Configuration ......................................................................... 30 2.5.16 Adding a Pool to a NAS................................................................................... 31 2.5.17 Removing a Pool from a NAS.......................................................................... 33 2.5.18 Adding a Blade (Datamover) to an EMC Celerra NAS .................................... 34 2.5.19 Removing a NAS............................................................................................. 35 2.5.20 NAS Disk Free................................................................................................. 35 2.6 LDAP............................................................................................................... 36 2.6.1 FlexFrame Structure in LDAP.......................................................................... 37 2.6.2 Working with LDAP ......................................................................................... 37 2.6.3 Disaster Recovery – Turning Replica into Master ........................................... 37

Contents

2.7 PRIMECLUSTER ............................................................................................ 38 2.7.1 PRIMECLUSTER Components ....................................................................... 38 2.7.2 FlexFrame Specific RMS Configuration........................................................... 38 2.7.3 RMS Configuration – Schematic Overview...................................................... 39 2.7.4 Node Failures .................................................................................................. 40 2.7.5 PRIMECLUSTER CLI Commands................................................................... 41 2.7.6 Usage Example: Switch Application ................................................................ 42 2.7.7 PRIMECLUSTER Log Files ............................................................................. 44 2.7.8 Disaster Repair................................................................................................ 44 2.8 Network ........................................................................................................... 45 2.8.1 LAN Failover.................................................................................................... 45 2.8.2 Segments ........................................................................................................ 45 2.8.3 Network Switches ............................................................................................ 47 2.8.4 Automounter Concept...................................................................................... 47 2.9 NAS System .................................................................................................... 51 2.9.1 Network Appliance Filer................................................................................... 51 2.9.1.1 Built-in Cluster File System.............................................................................. 52 2.9.1.2 Volume Layout................................................................................................. 52 2.9.1.3 Snapshots........................................................................................................ 52 2.9.1.4 Filer Cluster ..................................................................................................... 52 2.9.2 EMC NAS (Celerra) ......................................................................................... 52 2.9.2.1 Control Station................................................................................................. 53 2.9.2.2 Data Mover ...................................................................................................... 53 2.9.2.3 Cluster Architecture ......................................................................................... 54 2.9.2.4 Snapshots........................................................................................................ 54 2.10 SAN Support.................................................................................................... 55 2.10.1 SAN Basic Layers............................................................................................ 55 2.10.2 Scope of the FlexFrame SAN Integration ........................................................ 56 2.10.3 Rules and Restrictions..................................................................................... 57

3 FlexFrame Basic Administration.................................................................. 59 3.1 Accessing a FlexFrame Landscape (Remote Administration) ......................... 59 3.2 Powering up the FlexFrame Landscape .......................................................... 59 3.3 Powering off the FlexFrame Landscape .......................................................... 61 3.4 Reactivating ANs after Power Shutdown by FA Agents .................................. 62

4 Displaying the Current FlexFrame Configuration State ............................. 63 4.1 Networks.......................................................................................................... 64 4.1.1 Script: ff_netscan.sh ........................................................................................ 65 4.1.2 Script: ff_pool_dnssrv.sh ................................................................................. 72 4.2 State of Pools .................................................................................................. 73 4.3 State of Application Nodes .............................................................................. 73 4.4 State of SAP Systems ..................................................................................... 73 4.5 State of SID Instances..................................................................................... 74

Contents

Administration and Operation 5

5 Web Interfaces............................................................................................... 75 5.1 FlexFrame Web Portal .................................................................................... 75 5.2 FA Autonomous Agents .................................................................................. 76 5.3 PRIMECLUSTER Administration..................................................................... 77 5.3.1 Cluster Foundation (CF).................................................................................. 81 5.3.2 Reliant Monitor Services (RMS) ...................................................................... 82 5.3.3 Inconsistent and Faulted Applications ............................................................. 83 5.3.4 Switching Applications..................................................................................... 85 5.3.5 Application Maintenance Mode ....................................................................... 85 5.4 ServerView S2................................................................................................. 86

6 Hardware Changes........................................................................................ 95 6.1 Changing BIOS Settings for Netboot............................................................... 95 6.2 Replacing Network Cards.............................................................................. 102 6.2.1 Replacing a Network Card – Control Node ................................................... 102 6.2.2 Replacing a Network Card – Application Node.............................................. 102 6.3 Replacing Power Control Hardware .............................................................. 103 6.4 Exchanging a Control Node .......................................................................... 103 6.4.1 Hardware Failed, Hard Disk and Installed OS are not Affected..................... 104 6.4.2 One Hard Disk is Defect, the Other One is Undamaged ............................... 104 6.4.3 The Control Nodes OS is Damaged .............................................................. 104 6.5 Replacing Switch Blades............................................................................... 104

7 Software Updates ........................................................................................ 105 7.1 Upgrading the Entire FlexFrame Landscape................................................. 105 7.1.1 Upgrading from FlexFrame 3.0 or Lower Version to FlexFrame 4.0.............. 105 7.1.2 Upgrading from FlexFrame 3.1 or 3.2 to FlexFrame 4.0 ............................... 105 7.2 Software Upgrade on the Control Node ........................................................ 105 7.2.1 ServerView Update via RPM......................................................................... 106 7.2.2 Updating/Installing a New Linux Kernel......................................................... 108 7.2.2.1 Software Stage.............................................................................................. 108 7.2.2.2 Install the New Kernel ................................................................................... 109 7.2.2.3 New Kernel Source for PRIMECLUSTER and ServerView ........................... 110 7.2.2.4 Reboot the Control Node............................................................................... 110 7.3 Installation of a New FA Agent Version ......................................................... 110 7.3.1 Migration at Pool Level.................................................................................. 111 7.3.2 FlexFrame Autonomy Command Line Interface............................................ 112 7.3.3 Migration of FA Agent Versions on Pool Level .............................................. 112 7.4 FA Migration Tool .......................................................................................... 114 7.4.1 Pool Mode ..................................................................................................... 114 7.4.2 File Mode ...................................................................................................... 114 7.4.3 Usage of Help................................................................................................ 115 7.4.4 Parameters of the FA Migration Tool............................................................. 115 7.5 Installing NAS Patches.................................................................................. 116 7.5.1 Installing ONTAP Patches............................................................................. 116

Contents

7.5.2 Installing DART Patches for Celerra Data Movers......................................... 116 7.6 Third Party Software...................................................................................... 117

8 Administrating Application Nodes............................................................. 119 8.1 Listing Application Nodes .............................................................................. 119 8.1.1 Displaying Information on a Specific Application Node.................................. 119 8.1.2 Displaying Information on all Application Nodes............................................ 122 8.2 Adding Application Nodes ............................................................................. 124 8.3 Removing Application Nodes......................................................................... 127 8.4 Renaming Application Nodes ........................................................................ 128 8.5 Application Nodes and SAN .......................................................................... 129 8.6 Administrating Blade Server Cabinets ........................................................... 130 8.6.1 Listing Blade Server Cabinets ....................................................................... 130 8.6.1.1 Displaying Information on a Specific Blade Server Cabinet........................... 130 8.6.1.2 Displaying Information on all Configured Blade Server Cabinets .................. 131 8.6.2 Adding Blade Server Cabinets....................................................................... 132 8.6.3 Removing Blade Server Cabinets.................................................................. 136 8.6.4 Changing Switch Blade Type......................................................................... 137 8.6.5 Changing Switch Blade Name....................................................................... 138 8.6.6 Changing Switch Blade Password................................................................. 139 8.6.7 Getting Switch Blade Initial Configuration...................................................... 140 8.7 Maintenance of Linux Application Node Images............................................ 141 8.7.1 Installing an Application Node Image from Installation Media ....................... 142 8.7.1.1 Installing the Application Node Image ........................................................... 142 8.7.1.2 Creating the New Netboot Configuration ....................................................... 143 8.7.1.3 Creating the New var Image.......................................................................... 143 8.7.2 Creating a New Linux OS Image for Application Nodes ................................ 144 8.7.2.1 Schematic Overview of the Maintenance Cycle............................................. 145 8.7.2.2 Creating a Custom Image Tree for Linux Application Nodes......................... 146 8.7.2.3 Disabling the FA Agents ................................................................................ 146 8.7.2.4 Modifying the Netboot Configuration ............................................................. 146 8.7.2.5 Creating the New var Image.......................................................................... 147 8.7.2.6 Enabling Read-Write Access to the New Root Image ................................... 147 8.7.2.7 Maintaining the Application Node Image ....................................................... 148 8.7.2.8 Creating the New var_template ..................................................................... 148 8.7.2.9 Enabling the FA Agents................................................................................. 149 8.7.2.10 Disabling Read-Write Access to the New Root Image................................... 149 8.7.2.11 Migrating the Remaining Application Nodes.................................................. 149 8.7.3 Service Packs................................................................................................ 150 8.7.4 Updating/Installing a New Linux Kernel ......................................................... 150 8.7.4.1 Software Stage .............................................................................................. 150 8.7.4.2 Installing a New Linux Kernel ........................................................................ 151 8.7.4.3 New initrd for Application Nodes.................................................................... 153 8.7.4.4 New Kernel and Netboot Configuration ......................................................... 153 8.7.4.5 Netboot Configuration for the First Test......................................................... 153

Contents


8.7.4.6 Removing Write Permissions for Maintenance Application Node on the Root Image.................................................................................................... 154

8.7.4.7 Changing the default Netboot Configuration to the New Linux Kernel........... 154 8.7.5 ServerView Update ....................................................................................... 155 8.7.6 Upgrading RPM Packages on an Application Node ...................................... 158 8.7.7 Upgrading the Application Software .............................................................. 158 8.7.8 Migrating Remaining Application Nodes to the New Application

Node Image................................................................................................... 159 8.7.8.1 Modifying the Netboot Configuration ............................................................. 159 8.7.8.2 Creating the New var Images ........................................................................ 159 8.8 Installation / Activation of New Solaris Images.............................................. 160 8.8.1 Introduction ................................................................................................... 160 8.8.1.1 General Notes ............................................................................................... 160 8.8.1.2 How to Access the Console of a PW 250 / PW 450 ...................................... 161 8.8.1.3 OBP Flag: Local-mac-address? .................................................................... 162 8.8.1.4 OBP Flag: Auto-boot? ................................................................................... 162 8.8.1.5 What Happens When the Solaris Client Boots .............................................. 162 8.8.2 Solaris Image – rc scripts .............................................................................. 164 8.9 Solaris Image Maintenance Cycle ................................................................. 165 8.9.1 Introduction ................................................................................................... 165 8.9.2 Overview ....................................................................................................... 166 8.9.3 Running the Cycle ......................................................................................... 168 8.9.4 Create Solaris Application Nodes on the Maintained Image ......................... 171 8.10 Image Customization for Experts .................................................................. 172 8.11 Troubleshooting............................................................................................. 173 8.11.1 Solaris Image – Traces of Solaris rc-scripts .................................................. 173 8.11.2 Problems With /usr During Maintenance Cycle ............................................. 173 8.11.3 Boot Hangs With “Timeout waiting…” ........................................................... 174 8.11.4 Boot Hangs After “router IP is…”................................................................... 174 8.11.5 Boot Stops Complaining About /usr............................................................... 174 8.11.6 Boot Hangs When Trying to Mount /usr ........................................................ 174 8.11.7 Boot Asks for Date, Time and/or Locale........................................................ 174 8.12 Pools and Groups.......................................................................................... 175 8.12.1 Adding a Pool ................................................................................................ 175 8.12.2 Removing a Pool ........................................................................................... 178 8.12.3 Listing Pool Details........................................................................................ 179 8.12.4 Listing all Pools ............................................................................................. 183 8.12.5 Changing Pool DNS Domain......................................................................... 185 8.12.6 Changing Pool Default Router....................................................................... 186 8.12.7 Adding a Group to a Pool .............................................................................. 187 8.12.8 Removing Pool Group ................................................................................... 188 8.12.9 Changing Group Assignment of Application Nodes....................................... 188 8.12.10 Changing Group and Pool Assignment of Application Nodes ....................... 189 8.13 Hosts Database............................................................................................. 189 8.13.1 Script: ff_hosts.sh.......................................................................................... 189

Contents

8.14 Rebooting All Application Nodes ................................................................... 191

9 User Administration .................................................................................... 193 9.1 Create, Modify, Delete, or List User(s) for App Nodes................................... 193 9.2 Create, Modify, Delete, or List Project(s) for Solaris 10 App Nodes .............. 195

10 SAN Configuration in a FlexFrame Environment..................................... 199 10.1 Setting Up the SAN Configuration ................................................................. 199 10.1.1 Configuring Storage....................................................................................... 199 10.1.1.1 General Remark for the Use of Navisphere................................................... 199 10.1.1.2 Storage System Access ................................................................................ 200 10.1.1.3 LUN Creation................................................................................................. 200 10.1.1.4 Recording LUN Information ........................................................................... 202 10.1.2 Configuring Application Nodes ...................................................................... 203 10.1.2.1 Installing the SFS driver for Solaris 9 Application nodes ............................... 205 10.1.2.2 Enabling Solaris Native Multipathing ............................................................. 206 10.1.2.3 Installing EMC PowerPath............................................................................. 207 10.1.2.4 Installing Veritas Software ............................................................................. 207 10.1.3 Connecting the Storage to the Application Nodes ......................................... 208 10.1.3.1 Creating Zones on the Fibre Channel Switches ............................................ 209 10.1.3.2 Checking Visibility of the Storage System on the Application Node .............. 210 10.1.3.3 Registering Host Initiators with a CLARiiON/FibreCAT CX ........................... 211 10.1.3.4 Mapping LUNs to the Application Nodes ....................................................... 212 10.1.3.5 Checking Visibility of the LUNs on the Application Node............................... 213 10.1.4 Creating Volumes and Filesystems for a SAP System .................................. 215 10.1.4.1 Creating a Solaris VM Volume Group for FlexFrame Usage ......................... 216 10.1.4.2 Creating a Linux LVM2 Volume Group for FlexFrame Usage........................ 218 10.1.4.3 Completing the configuration and testing usability of SAN for an SID ........... 218 10.2 FlexFrame SAN Configuration....................................................................... 220 10.2.1 Script: ff_san_ldap_conf.pl ............................................................................ 220 10.2.2 FlexFrame SAN Configuration File ................................................................ 222 10.2.2.1 Syntax ........................................................................................................... 222 10.2.2.2 Description..................................................................................................... 223 10.3 SAN Support Scripts...................................................................................... 229 10.3.1 Script: ff_san_mount.sh................................................................................. 229 10.3.2 Script: ff_san_info.sh ..................................................................................... 230 10.3.3 Script: ff_qlascan.sh ...................................................................................... 231

11 Security ........................................................................................................ 233 11.1 Requested Passwords and Password Settings During Installation................ 233 11.1.1 Requested Passwords of the Control Nodes................................................. 233 11.1.2 Setting Passwords of a NAS System............................................................. 234 11.1.3 Initial SSH Configuration ............................................................................... 234 11.2 Password Settings During Operation, Update and Upgrade.......................... 235 11.2.1 User Administration ....................................................................................... 235

Contents


11.2.2 Password Management on Control Nodes.................................................... 236 11.2.2.1 Passwords for Root and Standard Unix Users .............................................. 236 11.2.2.2 Password for Root of LDAP Server and Replica ........................................... 236 11.2.2.3 Password for LDAPadmins ........................................................................... 239 11.2.2.4 Password for SNMP Community ................................................................... 239 11.2.2.5 Key for Access to the Name Server (named) ................................................ 239 11.2.2.6 Password for myAMC WebGUI ..................................................................... 239 11.2.2.7 Password for mySQL Database .................................................................... 240 11.2.2.8 Password for Power Shutdown ..................................................................... 240 11.2.2.9 Password for SAPDB/MaxDB in FlexFrame Start/Stop Scripts ..................... 240 11.2.3 Password Management on Linux Application Nodes .................................... 241 11.2.3.1 Password for Admin on BX(3,6)00 ................................................................ 241 11.2.3.2 Password for PowerShutdown on BX(3,6)00, RXxxx .................................... 241 11.2.3.3 Passwords for Root / Standard Unix Users and SAP Users.......................... 241 11.2.3.4 Changing Root Password for Linux Application Nodes ................................. 241 11.2.4 Password Management on Solaris Application Nodes .................................. 242 11.2.4.1 Password for PowerShutdown ...................................................................... 242 11.2.4.2 Passwords for Root / Standard Unix Users and SAP Users.......................... 242 11.2.5 Passwords for Networking Components ....................................................... 243 11.2.5.1 Password for Cisco Switch Enable/Login and SNMP_Community................ 243 11.2.5.2 Admin Password for SwitchBlade BX(3,6)00 ................................................ 243 11.2.5.3 Admin and SNMP_Community Storage Passwords...................................... 243 11.2.6 Preparation for Linux Application Nodes ....................................................... 243 11.2.6.1 Script: ff_install_an_linux_images.sh ............................................................ 243 11.2.7 Preparation for Solaris Application Nodes..................................................... 244 11.2.7.1 Script: nb_unpack_bi..................................................................................... 244 11.3 Passwords Stored in Initialization Files ......................................................... 244 11.3.1 Switch Group Switch Definition Files............................................................. 244 11.4 Configuration of the SCON Shutdown Agent on the Control Nodes.............. 245 11.5 FA Agents ..................................................................................................... 245 11.5.1 Power-Shutdown Configuration..................................................................... 246 11.5.1.1 BX300/600 .................................................................................................... 246 11.5.1.2 RX300/RX300 S2.......................................................................................... 247 11.5.1.3 RX600 ........................................................................................................... 247 11.5.1.4 RX800 ........................................................................................................... 247 11.5.1.5 PRIMEPOWER 250/450 ............................................................................... 248 11.5.1.6 PRIMEPOWER 650/850 ............................................................................... 248 11.5.2 SNMP Traps.................................................................................................. 249 11.5.2.1 General ......................................................................................................... 249 11.5.2.2 Configuring User, Password and Community................................................ 249 11.5.2.3 Configuring Management Blades .................................................................. 249 11.6 PRIMECLUSTER .......................................................................................... 250

12 Administrating SAP Systems..................................................................... 251 12.1 Listing SAP SIDs and Instances.................................................................... 251

Contents

12.2 Adding / Removing SAP SIDs and Instances ................................................ 252 12.3 Cloning a SAP SID into a Different Pool ........................................................ 254 12.3.1 Script: ff_clone_sid.pl .................................................................................... 254 12.3.2 Script: ff_change_id.pl ................................................................................... 255 12.3.3 Changing User and Group IDs after Cloning ................................................. 258 12.4 Multiple NAS Systems and Multiple Volumes................................................ 258 12.4.1 NetApp Filer................................................................................................... 259 12.4.2 EMC Celerra.................................................................................................. 261 12.5 Upgrading a SAP System.............................................................................. 262 12.5.1 Service Port ................................................................................................... 262 12.5.2 FA Agents...................................................................................................... 263 12.6 SAP Kernel Updates and Patches................................................................. 263

13 Administrating SAP Services ..................................................................... 265 13.1 Displaying Status of SAP Services ................................................................ 265 13.1.1 myAMC.FA WebGUI ..................................................................................... 265 13.1.2 Script: ff_list_services.sh ............................................................................... 266 13.2 Starting and Stopping Application Services................................................... 267 13.2.1 Starting and Stopping SAP Services without Root Privileges ........................ 267 13.2.2 SAP Service Scripts ...................................................................................... 267 13.2.3 SAP Service Script Actions............................................................................ 268 13.2.4 SAP Service Scripts User Exits ..................................................................... 269 13.3 Return Code of the SAP Service Scripts ....................................................... 270 13.4 Starting and Stopping Multipe SAP Services................................................. 271 13.5 Removing an Application from Monitoring by FA Agents............................... 272 13.5.1 Stopping and Starting an Application for Upgrades Using r3up.................... 273 13.6 Service Switchover ........................................................................................ 273

14 SAP ACC ...................................................................................................... 275 14.1 Integration of New Servers, Pools and SAP Services ................................... 275 14.1.1 Integration of New ACC Pools (=FF groups) ................................................. 275 14.1.2 Integration of New Servers ............................................................................ 275 14.1.3 Integration of new SAP Services ................................................................... 276 14.1.3.1 Moving SAP Services into SLD ..................................................................... 276 14.2 User Administration ....................................................................................... 276 14.3 Usage of ACC................................................................................................ 276 14.3.1 Displaying Status of SAP Services ................................................................ 276 14.3.2 Starting SAP Services ................................................................................... 277 14.3.3 Stoping SAP Services ................................................................................... 277 14.3.4 Relocating SAP Services............................................................................... 277 14.3.5 Archiving the ACC Log .................................................................................. 277

15 Configuring FA Agents ............................................................................... 279 15.1 Groups........................................................................................................... 280 15.1.1 General.......................................................................................................... 280

Contents


15.1.2 Service Classes............................................................................................. 281 15.1.3 Service Priority .............................................................................................. 281 15.1.4 Service Power Value ..................................................................................... 282 15.1.5 Class Creation Rules..................................................................................... 282 15.2 Traps ............................................................................................................. 282 15.2.1 General ......................................................................................................... 282 15.2.2 Changing the Trap Destinations .................................................................... 282 15.3 FlexFrame Autonomy .................................................................................... 283 15.3.1 General Parameters ...................................................................................... 283 15.3.2 Node-Related Parameters............................................................................. 284 15.3.3 Service-Related Parameters ......................................................................... 286 15.3.4 Path Configuration......................................................................................... 287 15.4 Power Management (On/Off/Power-Cycle) ................................................... 289 15.4.1 General ......................................................................................................... 289 15.4.2 Architecture ................................................................................................... 290 15.4.3 Configuring User, Password and Community................................................ 291 15.4.4 Configuring Management Blades .................................................................. 291 15.5 Linux Kernel Crash Dump (LKCD) Utilities.................................................... 291

16 Data Protection............................................................................................ 293 16.1 Backup/Restore for NetApp Filer................................................................... 293 16.1.1 Backup of Filer Volumes with NetApp SnapShot........................................... 294 16.1.1.1 Filer Volumes ................................................................................................ 294 16.1.1.2 Snapshot Schedules ..................................................................................... 295 16.1.2 Backup of SAP Databases ............................................................................ 296 16.1.3 Restore SnapShot ......................................................................................... 300 16.2 Backup/Restore for EMC Celerra .................................................................. 301 16.2.1 Backup of Celerra with EMC SnapSure ........................................................ 301 16.2.1.1 Volumes on Celerra....................................................................................... 302 16.2.1.2 SnapSure Schedules..................................................................................... 302 16.2.2 Backup of SAP Databases ............................................................................ 302 16.2.3 Restore Snapshots........................................................................................ 302 16.3 FlexFrame Backup with Tape Library............................................................ 303 16.3.1 NetWorker ..................................................................................................... 303 16.3.2 Arcserve ........................................................................................................ 304 16.4 Backup/Restore of FlexFrame Control Nodes............................................... 304 16.4.1 Backup of a Control Node ............................................................................. 304 16.4.2 Restore of a Control Node............................................................................. 305 16.4.3 Restore PRIMECLUSTER............................................................................. 306 16.5 Backup of Switch Configurations................................................................... 309 16.6 Restore of Switch Configurations .................................................................. 310

17 Error Handling & Trouble Shooting ........................................................... 311 17.1 Log Files........................................................................................................ 311 17.2 Network Errors .............................................................................................. 312

Contents

17.3 NFS Mount Messages ................................................................................... 312 17.4 LDAP Error Codes and Messages................................................................. 312 17.5 FA Agents Error Diagnosis ............................................................................ 314 17.6 FA Agents Operation and Log Files............................................................... 318 17.6.1 General.......................................................................................................... 318 17.6.2 Overview, important Files and Directories ..................................................... 318 17.6.3 Special Files .................................................................................................. 322 17.6.3.1 Livelist ........................................................................................................... 322 17.6.3.2 Services List .................................................................................................. 322 17.6.3.3 Services Log.................................................................................................. 322 17.6.3.4 Reboot ........................................................................................................... 322 17.6.3.5 Switchover ..................................................................................................... 323 17.6.3.6 XML Repository ............................................................................................. 323 17.6.3.7 BlackBoard .................................................................................................... 323 17.6.4 FA Autonomy Diagnostic Tool ....................................................................... 323 17.6.5 Data for Diagnosis in the Support Department .............................................. 323 17.7 Start/Stop Script Errors.................................................................................. 324 17.7.1 Common Error Messages for all Start/Stop Scripts ....................................... 324 17.7.2 SAPDB Specific Error Messages................................................................... 327 17.7.3 Sapci-specific Error Messages ...................................................................... 328 17.7.4 Sapscs-specific Error Messages ................................................................... 330 17.7.5 Sapascs-specific Error Messages ................................................................. 330 17.7.6 Sapjc-specific Error Messages ...................................................................... 330 17.7.7 Sapapp-specific Error Messages................................................................... 330 17.7.8 Sapj-specific Error Messages ........................................................................ 330 17.8 SAP ACC Troubleshooting ............................................................................ 330 17.8.1 ACC Logging ................................................................................................. 330 17.8.2 Missing Server in the ACC Physical Landscape............................................ 331 17.8.3 Reset of Service Status in Case of Failures .................................................. 331 17.8.4 Hanging Locks............................................................................................... 331 17.9 PRIMECLUSTER .......................................................................................... 332 17.9.1 Problem Reporting......................................................................................... 332 17.9.2 Removing “Ghost Devices” from RMS GUI ................................................... 332 17.10 Script Debugging ........................................................................................... 332 17.10.1 Shell Scripts................................................................................................... 332 17.10.2 Perl Scripts .................................................................................................... 332 17.11 Debugging the Linux Kernel .......................................................................... 333 17.11.1 Netconsole..................................................................................................... 333 17.11.2 Capturing Crash Dumps ................................................................................ 333 17.11.2.1 Common Restrictions for Taking Crash Dumps............................................. 333 17.11.2.2 SLES9: Linux Kernel Crashdump Utilities (LKCD)......................................... 334 17.11.2.3 SLES10: Kdump Kernel Crash Dump Capturing ........................................... 334 17.11.2.4 Forcing a Crash Dump .................................................................................. 334

18 Abbreviations............................................................................................... 335

Contents


19 Glossary....................................................................................................... 339

20 Index............................................................................................................. 345

Requirements Introduction

1 Administration and Operation

1 Introduction This document provides instructions on administrating and operating an installed Flex-Frame™ 4.0 environment. It focusses on general aspects of the architecture as well as on software updates, hardware extensions and FlexFrame-specific configuration.

It does not cover the installation of an entire FlexFrame environment. Please refer to the “Installation of a FlexFrame Environment” manual for information on initial installation.

1.1 Requirements This document addresses administrators on FlexFrame environments. We assume that the reader of this document has technical background knowledge in the areas of operat-ing systems (Linux®, Solaris™), IP networking and SAP® basis.

1.2 Notational Conventions The following conventions are used in this manual:

Additional information that should be observed.

Warning that must be observed.

fixed font Names of paths, files, commands, and system output.

<fixed font> Names of variables.

fixed font User input in command examples (if applicable using <> with variables).

Command prompt: # The notation

control1:/<somewhere> # <command>

indicates that the command <command> is issued on the first Control Node in the directory /<somewhere>.

The reader may need to change into the directory first, e.g.

control1:~ # cd /<somewhere> control1:/<somewhere># <command>.

Introduction Document History


1.3 Document History Document Version Changes Date

1.0 First Edition 2007-03-01

1.4 Related Documents FlexFrame™ for SAP® – Planning Tool

FlexFrame™ for SAP® – Installation of a FlexFrame Environment

FlexFrame™ for SAP® – Installation Guide for SAP Solutions

FlexFrame™ for SAP® – Network Design and Configuration Guide

FlexFrame™ for SAP® – FA Agents - Installation and Administration

FlexFrame™ for SAP® – Installation ACC 1.0 SP13

FlexFrame™ for SAP® – myAMC.FA_LogAgent - Concept and Usage

FlexFrame™ for SAP® – Upgrading FlexFrame 3.1 or 3.2 to 4.0

FlexFrame™ for SAP® – White Paper

PRIMECLUSTER Documentation

ServerView Documentation

SUSE Linux Enterprise Server Documentation

Solaris Documentation

1.5 Special Hints for FlexFrame 4.0 In this document, you often will find console output, configuration data and installation examples which are based on earlier FlexFrame versions. Please keep in mind that these are examples and may look slightly different on the new operating systems introduced in FlexFrame 4.0.

General Notes on FlexFrame 4.0 FlexFrame Architecture


2 FlexFrame Architecture The FlexFrame solution V4.0 is a revolutionary approach to run complex SAP solutions with higher efficiency. At the same time some major changes of the configuration para-digms for infrastructures have been implemented. These changes are:

● A shared operating system booted via IP networks for the SAP Application Nodes.

● Decoupling of application software and operating system, called virtualization of SAP software or Adaptive Computing.

● Shared Network Attached Storage from Network Appliance® providing Write Any-where File Layout (WAFL™) and sophisticated snap functionality.

● Shared Network Attached Storage by EMC® Celerra.

● FlexFrame Autonomous Agents (FA Agents) providing revolutionary mechanisms to implement high-availability functions without cluster software

● SAN storage

The concept of FlexFrame for SAP consists of several components, which implement state-of-the-art functionality. Together with new components, such as the FlexFrame Autonomous Agents, the whole solution is far more than just the sum of its components. A major part of the benefits consist in a dramatic reduction in day to day operating costs for SAP environments.

It is, of course, possible to use parts of the FlexFrame solution in project-based imple-mentations. However, they can not be called FlexFrame.

2.1 General Notes on FlexFrame 4.0 FlexFrame was designed and developed as a platform for SAP Applications. Its major purpose is to simplify and abstract the basic components to enable the administrator of a SAP system landscape to focus on SAP and not worry about servers, networking and storage.

The FlexFrame Control Nodes are seen as an appliance. Like a toaster or a micro wave oven, they have a well defined purpose and the build-components must work together to achieve that purpose. It happened that Fujitsu Siemens Computers picked SUSE® Linux Enterprise Server (SLES) as the operating system for the Control Nodes, however it is not intended that the customer is using it as a regular server, meaning installing addi-tional software on it and applying patches to it is not wanted, unless Fujitsu Siemens Computers support line instructs so. Upcoming versions of the Control Node’s operating system may be totally different and may not allow modifying anything at all. The installa-tion and backup/restore functionality of a Control Node are based on fixed images which are delivered on DVD. Modifications of installed images will not be taken care of, if a new version is installed.

FlexFrame Architecture Hardware


Modifications may even lead to errors which may be hard to find. Therefore we strongly recommend to the customers not to install any software or patches onto the Control Nodes without confirmation of Fujitsu Siemens Computers support.

Similar to the Control Nodes are the Application Nodes. Fixed images are shipped for installation of Linux Application Nodes in the FlexFrame landscape. For Solaris Applica-tion Nodes it is possible to create the images directly out of the Solaris OS DVD/CDs.

Another aspect of FlexFrame is the reduction of the TCO (total cost of ownership). A static approach (once started, never touched) will not be very efficient. To achieve the best savings, it is recommended to actively manage where a certain SAP application instance or database instance is running. If, as an example, a SAP instance requires the power of two CPUs of an Application Node during most days of a month and eight CPUs during month-end calculations, it is best to move it back and forth to the appropriate Ap-plication Node with the right size. During the time where the application instance is run-ning on the two CPU Application Node, another SAP instance can use the bigger eight CPU Application Node and therefore saving the need to have more eight CPU Application Nodes as in a static approach.

2.2 Hardware A typical FlexFrame environment consists of the following hardware types:

Software FlexFrame Architecture


1. Two Control Nodes: PRIMERGY RX300 S3 with SUSE Linux Enterprise Server 9 (SLES 9 32Bit) installed on local disks.

2. Two (equal) network switches Cisco Catalyst 3750G.

3. Network Attached Storage with one Network Appliance Filer or EMC Celerra hosting shared OS file systems and application data. Mirroring of data between separate lo-cations is possible (with NetApp Filer Cluster or EMC Celerra Replicator).

4. Intel-based PRIMERGY Server (standard rack server or blade server) serving as Application Nodes with SUSE Linux Enterprise Server 10 or SUSE Linux Enterprise Server 9 (Shared OS). SUSE Linux Enterprise Server 8 is still supported for some elderly PRIMERGY models.

5. SPARC64 V-based PRIMEPOWER Server serving as Application Nodes with Solaris 10 or Solaris 9 (Shared OS). Solaris 8 is still supported for some elderly PRIME-POWER models.

6. Optionally and new as of FF V4.0: SAN Storage Subsystems (from EMC or Network Appliance) containing LUNs carrying SAP Databases and Log Volumes.

One of the new FlexFrame functions coming with version 4.0 is the integration of EMC Celerra Systems into a FlexFrame environment. For hosting the central shared OS file systems customers are now free to choose Network Attached Storage from the FSC storage partners Network Appliance and EMC; NAS systems from both storage manufac-turers fit smoothly into the FlexFrame architecture.

For detailed information about the hardware supported in a FlexFrame environment, see the FlexFrame 4.0 Configuration Guide. Any other functions, such as backup, can be implemented separately as an add-on to FlexFrame and need dedicated hardware, oper-ating system, high availability, professional service and support etc.

2.3 Software The FlexFrame solution consists of both, hardware and software. To grant proper func-tion of the whole landscape, the entire software set is strictly defined. Anything other than the software components listed below is not part of FlexFrame. This applies unchanged if software from the list below is missing, is installed in other versions than below, or if soft-ware other that the actual SAP components is added.

FlexFrame Architecture Software


No. Hardware OS Software Services

1 Control Nodes:

2 x PRIMERGY RX300 S3 or 2 x PRIMERGY RX300 S2

SLES 9 32Bit SP3 PRIMECLUSTER 4.2A00, FA Agents (Control Agents) V3.0, FlexFrame 4.0 File System Image CN, ServerView, etc.

TFTP, DHCP, RARP, BOOTPARAM, LDAP, (SAPROUTER), etc.

2 Network switches:

2 x Cisco Catalyst 3750G-24TS or

2 x Cisco Catalyst 3750G-24T or 2 x Cisco Catalyst 3750G-48TS

SMI (proprietary)

(as delivered)

3 Network Attached Storage

one or more Network Appliance Filer heads (FASxxx), disk shelves as required 1, hosting shared OS file systems and application data

or

one or more EMC Celerra NSxxx, disk shelves as required 1, hosting shared OS file systems and application data.

ONTAP 7.2 recommended 2 min 7.0.5 (GD) or 7.1.2 (GA) (proprietary)

DART min 5.5.28 3 (proprietary)

ONTAP NetApp Tools

DART EMC Tools

NFS, SnapRestore

optional:

Cluster components, FlexClone, SnapVault, SnapMirror NFS, …

1 The amount of disks required for customer-specific FlexFrame configurations can be determined together with Fujitsu Siemens Computers' Customer Support

Filer Sizing Team 2 Some SAN software stacks afford 7.2, see Support Matrix Flexframe for SAP 4.0A

Software FlexFrame Architecture


No. Hardware OS Software Services

4 SAN Storage Multipath SW SLES 9 / SLES 10: – DM-MPIO integrated Multipath SW

Solaris 9 / 10: – MPxIO 4.4.9 or optional for Solaris OS – EMC Powerpath 4.5.0

HA Services

5 SAN Storage Volume Manager SLES 9 / SLES 10: – LINUX Volume Manager LVM2

Solaris 9 / 10: – Solaris Volume Manager SVM or for all supported OS – Veritas (VxVM, VxFS)

Volume Management Services

6 Intel-based PRIMERGY server (standard rack or blade server)

SLES 10 (x86_64), SLES 9 SP3 (x86_64) and / or SLES 9 SP2 (i686)

FlexFrame 4.0 File System Image, FA Agents (Application Agents), SAP Applications, Database

SAP & DB Services

7 SPARC64 V-based PRIMEPOWER server

Solaris 10 and / or Solaris 9

FlexFrame 4.0 File System Image, FA Agents (Application Agents), SAP Applications, Database

SAP & DB Services

3 Before GA of 5.5.28 the patch version 5.5.25.205 must be used

FlexFrame Architecture Shared Operating System


2.4 Shared Operating System One major aspect of FlexFrame is its shared operating system. Sharing in this case means, that the very same files of essential parts of the underlying operating system are used to run multiple Application Nodes. This part of the file system is mounted read-only, so none of the Application Nodes that run the actual applications can modify it. Server specific information is linked to a file system area that is server-specific and mounted read-write. The shared operating system is kept to a NAS system (Network Attached Storage system from Network Appliance or EMC Celerra).

2.4.1 Shared OS Boot Concept The chart below shows the boot process of a FlexFrame Application Node (PRIMERGY/Linux):

The FlexFrame Concept – Shared Linux

PRIMERGYBX300

PRIMERGYRX600

PRIMERGYRX800

FSC3

FSC4

FSC2

1 get boot info

FSC1

2 Server Name IP addresswhere to boot from

3 mount root directory

4 boot L inux

NAS Storage System

OS-shared

Control Node (clustered)assigns server namesprovides boot infodistributes SAP servicesadmin/monitoring

The network boot concept used for Solaris-based PRIMEPOWER servers is derived from the Solaris Diskless Client Concept. A shared /usr area is mounted read-only. Each server has its own root file system which is mounted read-write.

Shared Operating System FlexFrame Architecture


The following graphic explains the boot process.

In Step (1) the PRIMEPOWER server sends an RARP (Reverse Address Resolution Protocol) request into the Storage LAN network segment. The Control Node with the active in.rarpd process is receiving this request. The request asks for an IP address based on the MAC address of the initiating NIC (Network Interface Card).

The in.rarpd process on the Control Node will search the file /etc/ethers for the MAC address provided. If it finds a line with the MAC address, it will return the associated IP address as an RARP response (2).

Now the PRIMEPOWER server can setup the NIC with the IP address. Next step (3) is to get a file via the TFTP (Trivial File Transfer Protocol) from the Control Node which re-sponded to the RARP request. The file name is build by the hex notation of the IP ad-dress.

The in.tftpd of the Control Node will send this file to the PRIMERPOWER server. This file contains the first step of the Solaris kernel for booting via the network.

Now additional parameters are required. The Solaris kernel sends a bootparam request to the Control Node (5). The rpc.bootparamd process on the Control Node will read the file /etc/bootparams for the given Application Node’s host name and replies with the detail information about its NFS server and path to the root file system along with mount options and other parameters.

FlexFrame Architecture Shared Operating System


2.4.2 Control Nodes A productive FlexFrame landscape always includes two Control Nodes. Their purpose is to be a single point of control for the Application Nodes, as well as to check and manage the proper function of the Application Nodes.

Control Nodes are not running SAP software (with the exception of saprouter, as an op-tion). They exclusively run SUSE Linux Enterprise Server Version 9 (SLES9), installed on local disks. Control Nodes provide and run services such as:

● Timeserver for the complete FlexFrame landscape

● Control Agents

● Web server to provide the Control Agents user interface

● Assignment of IP addresses to Application Nodes using DHCP (Linux) or RARPD and BOOTPARAMD (Solaris)

● TFTP server for the boot process

● saprouter (optional)

Control Nodes have to be of the type PRIMERGY RX300, RX300 S2 or RX300 S3.

2.4.3 Application Nodes Application Nodes are the workhorses of the FlexFrame solution. They offer CPU and memory and run database and SAP services. Application Nodes must have local disks, though, which are exclusively used for swap space and do not contain any other data.

For FlexFrame Version 4.0 there are the following principal types of Application Nodes:

● PRIMERGY server running SLES 10 64bit ● PRIMERGY server running SLES 9 64bit ● PRIMERGY server running SLES 9 32bit ● PRIMEPOWER server running Solaris 10 ● PRIMEPOWER server running Solaris 9

and for compatibility reasons

● “old” PRIMERGY server running SLES 8 ● “old” PRIMEPOWER server running Solaris 8

Admissible PRIMERGY servers have to be approved for SAP on Linux by FSC.

2.4.3.1 Linux Application Nodes

During the boot process using Intel’s PXE® technology, each Application Node will be identified using the hardware address of its boot interface (MAC address). The Control Node will assign an IP address to it and supply the operating system via the network.

Switch and NAS Configuration FlexFrame Architecture


File systems, especially the root file system (/), are mounted via the network in read-only mode. If, for any reason, an Application Node needs to be replaced or added, only a handful of settings need to be adjusted to integrate it into the FlexFrame landscape.

Intel's PXE technology is implemented in Fujitsu Siemens Computers PRIMERGY serv-ers and allows booting via the network. DHCP will be used with static MAC-Address rela-tionship for all the Application Nodes.

2.4.3.2 Solaris Application Nodes

For Solaris Application Nodes, the diskless client concept of Sun Microsystems, Inc. was enhanced for the FlexFrame solution. This concept has a dedicated root file system for each Application Node while sharing the same /usr file system amongst a group of servers.

The boot process is similar to the above Linux network boot. As mechanisms for booting via the network, RARP and BOOTPARAM are used. IP address information is delivered by RARP, while boot information and the location of the root file system of the Application Nodes are delivered by BOOTPARAM.

2.5 Switch and NAS Configuration A switch group consists of at least two switches of the Cisco Catalyst 3750G switch fam-ily. They are interconnected on the back with Cisco StackWise technology to form a stack and behave like a virtual modular switch.

For a description how to interconnect the switches, please refer to Cisco Cata-lyst 3750 Installation Manual or “Cisco StackWise Technology” White Paper.

Adding or removing a switch to or from a switch group means to add or remove a switch to/from a stack. Cisco notes this may be done during normal operation (see Catalyst 3750 White Paper “Cisco StackWise Technology”).

To ensure safe operation we recommend doing this at a downtime to minimize influence on running systems. This requires shutting down all systems con-nected to this switch group. In case a NAS system is connected, all systems that have mounted file systems from the Filer have to be shut down as well.

2.5.1 Adding a Switch to a Switch Group To add a new switch to an existing switch group, the new switch has to be inserted into the stack. Mount switch into system cabinet next to mounted switch group and insert it into StackWise cabeling. See notes above for recommendations.

To be able to use the new switch with the maintenance tools, it has to be added to LDAP database. Use the command ff_swgroup_adm.pl to do this.

FlexFrame Architecture Switch and NAS Configuration


Synopsis

ff_swgroup_adm.pl --op add-sw --group <switch_group_id>

--type <switch_type>

Options

--op add-sw Adds a new member to the switch group and displays some information about processing steps.

--group <switch_group_id> Defines the switch group to be used.

--type <switch_type> Defines the type of the new member to be added to the switch group. Currently the type has to be cat3750g-24t, cat3750g-24ts, or cat3750-48ts.

Output example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op add-sw --group 1 --type cat3750g-24ts If program is aborted by Ctrl-C or a failure remove left overs by calling: ff_swgroup_adm.pl --op rem-sw --group 1 --switch 3 Switch was added to LDAP data. Keep in mind: INSERTING SWITCH TO STACK NEEDS A DOWN TIME ! To add the switch to switch stack (switch group 1) write down the current switch ids as they may change inserting the new switch to stack. To connect the switch to the stack use the provided stacking cable and stacking ports at rear side. See Cisco installation manual for details. If switch ids get scrambled use the IOS command "switch <current nmbr> renumber <new nmbr>" to put them in same order as before. In short the to do list: -> write down switch ids of each switch of group -> power down entire switch group -> insert switch into stack -> power on entire switch group -> look at switch ids and compare with your noticed



-> in case of differences use IOS command to renumber switches Switch group is ready for use See file /tmp/swgrp-add-1-3/next_steps for same instructions as above.

2.5.2 Removing a Switch from a Switch Group A switch may be removed if it is unused and has the highest ID within the stack. Remove it first from the LDAP database with the command ff_swgroup_adm.pl and then from the stack. See notes above for recommendations.

Synopsis

ff_swgroup_adm.pl --op rem-sw --group <switch_group_id>

--switch <switch_id>

Options

--op rem-sw Removes the last member from a switch group and displays some information about processing steps.


--switch <switch_id> Defines the stack ID of the switch to be removed from a switch group.

Output example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op rem-sw --group 1 --switch 3 Switch was successfully removed from LDAP data. Keep in mind: REMOVING SWITCH FROM STACK NEEDS A DOWN TIME ! In short the to do list: -> power down entire switch group -> remove switch from stack -> power on entire switch group Switch group is ready for use See file /tmp/swgrp-rem-1-3/next_steps for same instructions as above.



2.5.3 Listing a Switch Group Configuration Invoking the command ff_swgroup_adm.pl with the list operation mode displays the configuration of a switch group like used switch types, port channels, port usage sta-tistics and used switch ports.

Synopsis

ff_swgroup_adm.pl --op list --group <switch_group_id>

Options

--op list Displays switch group configuration.


Output example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op list --group 1 Switch Group 1 Name/IP: switch-i-2 / 192.168.13.15 Password: passwort SNMP Community: public;ro Switch Types: (switch id, switch type) 1 cat3750g-24ts 2 cat3750g-24ts Port Channels: (channel id, switch ports, connected device) 3 1/3,2/3 swb-1-1/11,swb-1-1/12 4 1/4,2/4 swb-1-2/11,swb-1-2/12 5 1/13,2/13 filer-ip1 Switch port usage: (switch id, used, free tx, free fx ports) 1 21 used 3 free tx 4 free fx 2 21 used 3 free tx 4 free fx Switch port list: (switch id, port id, connected device, vlans) 1 1 pw250-10 t10,t12,u11 1 2 pw250-11 t10,t12,u11 1 3 swb-1-1/11 t1,t10,t11,t12,t13 1 4 swb-1-2/11 t1,t10,t11,t12,t13 1 5 pw650-20 t10,t12,u11



1 6 rx300-13 t10,t12,u11 1 7 rx300-14 t10,t12,u11 1 8 pw250-12 t10,t12,u11 1 9 pw250-13 t10,t12,u11 1 10 cn1 t10,t11,t12,u13 1 11 cn2 t10,t11,t12,u13 1 12 filer-ip1 t11 1 13 extern. Connect t10,t11,t12 1 14 --- unused --- 1 15 --- unused --- 1 16 --- unused --- 1 17 cn1 mgmt u13 1 18 pw250-12 mgmt u13 1 19 rx300-13 mgmt u13 1 20 - u13 1 21 pw650-20 mgmt u13 1 22 pw250-10 mgmt u13 1 23 Corporate LAN u10 2 1 pw250-10 t10,t12,u11 2 2 pw250-11 t10,t12,u11 2 3 swb-1-1/12 t1,t10,t11,t12,t13 2 4 swb-1-2/12 t1,t10,t11,t12,t13 2 5 pw650-20 t10,t12,u11 2 6 rx300-13 t10,t12,u11 2 7 rx300-14 t10,t12,u11 2 8 pw250-12 t10,t12,u11 2 9 pw250-13 t10,t12,u11 2 10 cn1 t10,t11,t12,u13 2 11 cn2 t10,t11,t12,u13 2 12 filer-ip1 t11 2 13 extern. Connect t10,t11,t12 2 14 --- unused --- 2 15 --- unused --- 2 16 --- unused --- 2 17 cn2 mgmt u13 2 18 pw250-13 mgmt u13 2 19 rx300-14 mgmt u13 2 20 - u13 2 21 - u13 2 22 pw250-11 mgmt u13 2 23 Corporate LAN u10



2.5.4 Changing the Password of a Switch Group To change the access password of a switch group it has to be changed at switch group as well as LDAP database. The command ff_swgroup_adm.pl changes both.

Synopsis

ff_swgroup_adm.pl --op pass --group <switch_group_id>

--passwd <password>

Options

--op pass Changes the switch group access.


--passwd <password> Defines the new password as clear text.

Output example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op pass --group 1 --passwd passwort update switch 1/1 configuration Notice: Update will take about 1 minute. ............+ Password changed from "password" to "passwort". See file /tmp/swgrp-pass-1/info for same information as above.

2.5.5 Changing the Host Name of a Switch Group To change the host name of a switch group it has to be changed at switch group as well as LDAP database and host files on both control nodes. The command ff_swgroup_adm.pl changes all of them.

Synopsis

ff_swgroup_adm.pl --op name --group <switch_group_id>

--name <name>



Options

--op name Changes the switch group host.


--name <name> Defines the new host name to be used.

Output example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op name --group 1 --name swg1 update switch 1/1 configuration Notice: Update will take about 1 minute. ...+ Switch name changed from "swg-1" to "swg1". See file /tmp/swgrp-name-1/info for same information as above.

2.5.6 Displaying/Changing Common Network Configuration Parameters

Some parameters of network configuration influence the switch group configuration and were defined with PlanningTool at initial installation.

To display or change these parameters use the ff_swgroup_adm.pl command.

Synopsis

ff_swgroup_adm.pl --op parameter [–-parameter <parameter_name> --value <parameter_value>]

Options

--op parameter Displays (without the following options) or changes (in conjunction with the following options) network configuration parameters.



--parameter <parameter_name> Defines the name of the parameter to be changed. Known parameters are:

clanPortPerVlan The parameter depends the way the Client LANs are connected to the corporate LAN (with the sapgui users). There are two modes:

no Use two dedicated switch ports (for redundancy) and use them for all pool Client LANs (as tagged VLANs onto these two ports).

yes Use two dedicated switch ports (for redundancy) for each pool Client LAN (as untagged VLAN on both ports).

useTxToClan If the switch group has fiber optic ports (SFP ports) these may be prefered as ports to the corporate LAN transfering the pool Client-LAN data. To be able to customize port type the parameter has two values:

no The port type will be fiber optic. Be sure to have matching modules for the SFP ports modules for the SFP ports.

yes The port type will be Cat5 twisted pair, commonly abbreviated as TX.

useTxUplink If the switch group has fiber optic ports (SFP ports) these may be prefered as ports to connect another switch group directly or to use them as uplink ports to the FlexFrame integrated LAN switch. The parameter has two values:

no The port type for uplink will be fiber optic. Be sure to have matching modules for the SFP ports.

yes The port type for uplink will be Cat5 twisted pair, commonly abbreviated as TX.

uplinkPortCnt Defines the count of ports aggregated to the uplink. The uplink channel consists of at least two ports and may have a maximum of eight ports. The parameter value is the number of ports.

--value <parameter_value> Defines the value of the parameter to be changed. For uplinkPortCnt use a range between 2 and 8. For all other parameters use yes or 1 and no or 0 as value.

Output examples

cn1:~ # ff_swgroup_adm.pl –op parameter Common Network Configuration Parameters Parameter Name Parameter Value Client LAN port per VLAN yes



spread Client LAN ports over switch groups no use TX ports to Client LAN yes use TX ports as uplink no uplink port count 2 timezone Europe/Berlin POSIX timezone CET

cn1:~ # ff_swgroup_adm.pl ––op parameter ––parameter useTxToClan -–value yes Parameter successfully changed at LDAP.

2.5.7 Adding a Switch Group To add an entire new switch group use ff_swgroup_adm.pl command. The command adds data to LDAP database and creates an initial configuration file. The configuration file has to be manually uploaded to switch. The instruction to do this is displayed by com-mand with values appropriate to current configuration.

Synopsis

ff_swgroup_adm.pl --op add --group <switch_group_id> --type <list_of_switch_types>

--name <name_of_switch_group>

--passwd <password> [--login <login_name>] [--host <ip_host_part>]

[--snmp <community_name>;<permission>]

[--syslog <syslog_server_ips>;<syslog_facility>] [--ntp <server_ips>]

Options

--op add Adds a switch group and displays some information about processing steps.

--group <switch_group_id> Defines switch group to operate. Within this operation mode used to define the id of the new switch group.

--type <switch_type> Defines the switch types the switch group consists of. Keep note on order if different switch types are used. The first type belongs to the lowest stack member id and so forth. The types are comma separated without any whitespaces.

--name <name_of_switch_group> Name string to be set as switch group node name.



--passwd <password> Clear text password to be set at switch group.

--login <login_name> Login name to be set at switch group. Defaults to flexframe.

--host <ip_host_part> Host part to be used to build IP addresses for the control lan network. If this option is omitted the script uses a free host number to calculate the IP address.

--snmp <community_name>;<permission> Defines the community name and its permission. Defaults to public;RO.

--syslog <syslog_server_ips>;<syslog_facility> Names the IP addresses of syslog server(s) and the facility to log messages. Join IP addresses by comma. Defaults to <cn_control_lan_ips>;local0.

--ntp <server_ips> Names the IP addresses of ntp server(s). Join IP addresses by comma. Defaults to <cn_control_lan_ips>.

Output example

cn1:~ # ff_swgroup_adm.pl --op add --group 2 --type cat3750g-48ts,cat3750g-48ts --name swg2 --passwd pass If program is aborted by Ctrl-C or a failure remove left overs by calling: ff_swgroup_adm.pl --op rem --group 2 update LDAP. ........ New SwitchGroup 2 was added to LDAP. The switch is configured with 1 channel for uplink to an integrated LAN switch or another switch group (in case of only two switch groups). See below the configured ports for uplink and connect them to the integrated LAN switch or the other switch group. SwitchID / Port 1 / 49 Uplink port of EtherChannel 1 to integrated LAN switch 2 / 49 Uplink port of EtherChannel 1 to integrated LAN switch Upload of initial switch configuration has to be done manualy. See installation guide for details. For a quick instruction see below. The file to be uploaded is named: /tftpboot/swg2.config



Quick install instruction for switch group 2 Plug only one port to integrated LAN switch to prevent disabling of portchannel at integrated LAN switches. Connect a RS232 cable between a serial port of ControlNode 1 and the console of the switch. Use console of first switch in stack (stack master). At ControlNode 1 use "minicom" to connect to the serial port. See below a session snippet as sample how to upload the configuration: Press RETURN to get started! ... --- System Configuration Dialog --- Would you like to enter the initial configuration dialog? [yes/no]: no Switch> enable Switch # configure terminal Switch (config)# vtp mode transparent Switch (config)# interface vlan 100 Switch (config-if)# ip address 192.168.100.1 255.255.255.0 Switch (config-if)# no shutdown Switch (config-if)# end Switch # copy tftp://192.168.100.202/swg2.config startup-config Accessing tftp://192.168.100.202/swg2.config... Loading swg2.config from 192.168.100.202 (via Vlan100): ! [OK - 3630 bytes] [OK] 3630 bytes copied in 0.377 secs (9629 bytes/sec) Switch # reload Proceed with reload? [confirm] y Plug all other ports to integrated LAN switches. The switch group should be fully operational. Unless any errors are reported follow instructions above to solve all precautions needed to integrate switch group into FlexFrame. Look at "/opt/FlexFrame/network/wiring_of_swgroup2__swg2.txt" to get a copy of this message.



2.5.8 Removing a Switch Group To remove an entire switch group use the ff_swgroup_adm.pl command. The com-mand removes data from LDAP database. As a precaution the switch group may no longer be in use by any FlexFrame component. Only the uplink port may be configured.

Synopsis

ff_swgroup_adm.pl --op rem --group <switch_group_id>

Options

--op rem Removes named switch group.

--group <switch_group_id> Defines switch group to operate. Within this operation mode used to define the id of the switch group to be revoved.

Output example

cn1:~ # ff_swgroup_adm.pl --op rem --group 2 update LDAP. ........ Unless any errors are reported disconnect uplink ports 1/49, 2/49 and switch group is removed from FlexFrame environment.

2.5.9 Adding a Switch Port Configuration Switch ports are typically directly configured by maintenance tools. But some issues like configuring ports for gateways, backup or migration systems need a way to do this on a per port basis. For this type of configuration a special peer type is used. The program ff_swport_adm.pl is used to configure or remove this type of port configuration.

Synopsis

ff_swport_adm.pl --op add --port <swgroup:switch:port>

--lan <pool:lan[:lan][,pool:lan[:lan]]> [--native <pool:lan>] [--desc <description>]



Options

--op add Adds a switch port configuration and displays some information about processing steps.

--port <swgroup:switch:port> Defines the switch group, switch and port ID of the port to be used.

--lan <pool:lan[:lan][,pool:lan[:lan]]> Defines the accessible VLANs. For better readability, a VLAN is specified with its pool and LAN name. Use only client, server or storage as LAN names. For more than one LAN per pool the LAN names may be added to the same pool statement. The VLANs are not restricted to belong to the same pool. To directly add VLAN IDs not used within any pool, use '#' as pool name and the VLAN ID(s) as LAN(s).

The accessible VLANs. For better readability, a VLAN is specified with its pool and LAN name. Use only client, server or storage as LAN names. For more than one LAN per pool the LAN names may be added to the same pool statement. The VLANs are not restricted to belong to the same pool. To directly add VLAN IDs not used within any pool, use '#' as pool name and the VLAN ID(s) as LAN(s).

If only a single VLAN is configured this is accessible as native VLAN. This means the data packet contains no VLAN tag. This is the behavior used by a standard server network interface. For more than one LAN they are configured as tagged. To define which of them should be used as native VLAN use the option --native.

Examples:

--lan poolA:client:server,poolB:client:server

--lan poolA:client,poolA:server,poolB:client:server

--lan poolA:storage,poolB:storage --lan poolA:server

--lan '#:417:891'

--lan poolA:server,'#:417:891' --lan 'poolA:server,#:417:891'

--native <pool:lan> Use this option to define the native VLAN of the accessible VLANs defined with op-tion --lan. To directly add VLAN ID not used within any pool, use '#' as pool name and the VLAN ID as LAN.

Examples:

--native poolA:server

--native '#:417'



--desc <description> The description is added to configuration of switch port and the LDAP data of the switch port configuration.

Output example

cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op add --port 1:1:15 --lan ip1:storage:server,’#:4000’ --native ip1:storage Execution may take some minutes. If program is aborted by Ctrl-C or a failure remove left overs by calling: ff_swport_adm.pl --op rem --port 1:1:15 update switch 1/1 configuration Notice: Update will take about 1 minute. ...........+ If not reported any error the port is configured and LDAP is updated suc-cessfully.

2.5.10 Removing a Switch Port Configuration To remove a switch port configuration that was previously configured by ff_swport_adm.pl or as external connectivity with PlanningTool, use the ff_swport_adm.pl command. Other ports are configured with maintenance tools like ff_an_adm.pl, ff_pool_adm.pl or ff_bx_cabinet_adm.pl.

The switch port configuration will be removed from the switch and the LDAP database.

Synopsis

ff_swport_adm.pl --op rem --port <swgroup:switch:port>

Options

--op rem Removes the configuration of a switch port and displays some information about processing steps.




Output example

cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op rem --port 1:1:15 Execution may take some minutes. If program is aborted by Ctrl-C or a failure remove left overs by calling: ff_swport_adm.pl --op rem --port 1:1:15 update switch 1/1 configuration Notice: Update will take about 1 minute. .............+ If not reported any error the port is unconfigured and LDAP is updated successfully.

2.5.11 Displaying a Switch Port Configuration To display the configuration of a switch port as known by LDAP database in detail, use the command ff_swport_adm.pl with operation mode list.

Synopsis

ff_swport_adm.pl --op list --port <swgroup:switch:port>

Options

--op list Displays configuration of the switch port.


Output examples

cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op list --port 1:1:4 Switch Port Configuration of 1:1:4 (Switch Group : Switch : Port) assigned VLAN IDs: 24,25,26 assigned VLAN Names: pool1:client,pool1:server,pool1:storage native VLAN: 26 Port Peer Type: AN Peer Node: rx300-1



Display of an unconfigured port:

ERROR: wrong switch port "1:1:8". Port configuration unknown.

2.5.12 Adding a New NAS To add a NAS to FlexFrame environment the NAS has to be configured, the network prepared and some data has to be stored at LDAP database. The NAS configuration has to be done manually. But all nescessary data and locations are displayed by this pro-gram. The network and LDAP preparation is done directly by ff_nas_adm.pl.

The network configuration differs by NAS type. FAS2xx family Filers are limited to two ethernet interfaces (NICs). For this type of Filers the Control LAN is configured to use interface e0 and the multi vif is reserved for pool LANs.

Synopsis

ff_nas_adm.pl --op add --name <node_name> --type <nas_type>

--swgroup <switch_group_id> [--host <ip_host_part>]

[--ports <port_count>] [--blades <blade_count>]

Options

--op add Adds a NAS and displays some information about processing steps.

--name <node_name> Defines the node name of the NAS.

--type <nas_type> Defines the type of the NAS. See usage for a list of known NAS types.

--swgroup <switch_group_id> Defines the switch group the NAS should be added to. See usage for a list of configured switch group IDs.

--host <ip_host_part> Defines the host part to be used to build IP addresses for the Control or Storage LAN networks. If this option is omitted the script uses a free host number to calculate the IP address.

--ports <port_count> Defines the count of ports to be used with pool Storage LAN networks. If this option is omitted the script uses the default of two ports.



--blade <blade_count> Blades/data mover count of an EMC Cellera NAS. If this option is omitted the script uses the default of one blade.

Output examples

For a NetApp Filer:

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op add --name filer2 --type FAS270 --swgroup 1 update LDAP ..... update switch 1/1 configuration Notice: Update will take about 1 minute. ................................+ Some manual interventions are nescessary to integrate the NAS into FlexFrame environment. The following list of actions have to be performed in order to integrate the NAS into your FlexFrame landscape. Since your exact configuration may vary, these steps have to be performed manually. However, the VIF must be named 'storage'. The /etc/rc, /etc/exports and /etc/hosts.equiv have to be edited at volume vol0. These lines have to be added to or changed at /etc/rc: hostname filer2 vif create multi storage <-- add your NICs here, eg. e0a e0b vlan create storage 13 ifconfig storage-13 192.168.20.4 netmask 255.255.255.0 broadcast 192.168.20.255 mtusize 1500 -wins up options dns.enable off options nis.enable off savecore These lines have to be added to or changed at /etc/exports: /vol/vol0 -sec=sys,rw=192.168.20.2:192.168.20.1,anon=0 These lines have to be added to /etc/hosts.equiv: 192.168.20.2 root 192.168.20.1 root



As the switch ports are already configured the correct wiring between NAS and switch ports has to be done. See below a list of cable connections. Connect your NAS LAN interfaces to named switch ports: SwitchGroup / Switch / Port LAN Interface 1 / 2 / 3 filer2 (FAS270): port "data NIC-1" 1 / 1 / 3 filer2 (FAS270): port "data NIC-2" Finally execute command "mount /FlexFrame/filer2/vol0" on both Control Nodes to mount filers vol0 at Control Nodes. This is nescessary for further automated configuration of filer. The complete instruction above is listed at file /tmp/filer2-add/todo

For an EMC Celerra:

cn1:~ # ff_nas_adm.pl --op add --name emc01 --type NS700 --swgroup 1 --blades 2 update LDAP ........... update switch 1/1 configuration Notice: Update will take about 1 minute. .........................................................................+ Some manual interventions are nescessary to integrate the NAS into FlexFrame environment. The following list of actions have to be performed in order to integrate the NAS into your FlexFrame landscape. Since your exact configuration may vary, these steps have to be performed manually. As the switch ports are already configured the correct wiring between NAS and switch ports has to be done. See below a list of cable connections. Connect your NAS LAN interfaces to named switch ports: SwitchGroup / Switch / Port LAN Interface 1 / 2 / 8 emc01 (NS700): server_2: port "data NIC-1" 1 / 1 / 8 emc01 (NS700): server_2: port "data NIC-2" 1 / 2 / 9 emc01 (NS700): server_3: port "data NIC-1" 1 / 1 / 9 emc01 (NS700): server_3: port "data NIC-2" 1 / 2 / 20 emc01 (NS700): port "mgmt NIC-1" Mounting of any Celerra filesystems to Control Nodes is not nescessary for further configuration. The complete instruction above is listed at file /tmp/emc01-add/todo



2.5.13 Configuring SNMP traps for NetApp Filers The NetApp Filers within FlexFrame should send their messages (SNMP traps) to the Control Nodes (resp. myAMC.Messenger). This has to be configured on the Ffilers side. Therefore logon to the Filer(s) with telnet <filer> and continue with the following steps:

filer> snmp traphost add 192.168.20.1 filer> snmp traphost add 192.168.20.2

Use IP addresses of the Control LAN segment.

filer> snmp community add ro public

public may be replaced by the community you have specified in the Planning Tool.

For further informations how to configure thresholds on Filer specific traps, please see the NetApp Filer documentation.

2.5.14 Displaying All Configured NAS To get an overview of all configured NAS within the FlexFrame environment use the op-eration mode list-all of the program ff_nas_adm.pl. It displays IP addresses and names, type, switch ports and the link aggregation id, separated by NAS.

Synopsis

ff_nas_adm.pl --op list-all

Option

--op list-all Displays all configured NAS.

Output example

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op list-all NAS configurations filer Control Lan 192.168.20.3 filer-co Type: FAS940 Switch Link Aggregation



Port Count: 2 Link Aggr.ID: 5 Storage LAN switch ports 1 / 1 / 13 SwGroup / Switch / Port 1 / 2 / 13 SwGroup / Switch / Port Control LAN switch ports 1 / 2 / 15 SwGroup / Switch / Port Pools pool1 192.168.2.131 filer-pool1-st master pool2 10.3.1.3 filer-pool2-st master pool5 192.168.40.3 filer-pool5 master usr 192.168.5.3 filer-usr-st master

2.5.15 Displaying NAS Configuration To display the detailed configuration of a NAS as known by LDAP database, use the command ff_nas_adm.pl with operation mode list.

Synopsis

ff_nas_adm.pl --op list --name <node_name>

Options

--op list Displays the configuration of a switch port.

--name <node_name> Defines the node name of a NAS.

Output examples

For a NetApp Filer:

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op list --name filer NAS configurations filer Control Lan 192.168.20.3 filer-co Type: FAS940 Switch Link Aggregation Port Count: 2 Link Aggr.ID: 5



Storage LAN switch ports 1 / 1 / 13 SwGroup / Switch / Port 1 / 2 / 13 SwGroup / Switch / Port Control LAN switch ports 1 / 2 / 15 SwGroup / Switch / Port Pools pool1 192.168.2.131 filer-pool1-st master pool2 10.3.1.3 filer-pool2-st master pool5 192.168.40.3 filer-pool5 master usr 192.168.5.3 filer-usr-st master

For an EMC Celerra:

cn1:~ # ff_nas_adm.pl -op list –name cel NAS configurations cel Control Lan 192.168.100.204 cel-co Type: NS-Type Switch Link Aggregation DataMover Port Count Link Aggr.ID 2 2 5 Storage LAN switch ports DataMover SwGroup / Switch / Port Control LAN switch ports 1 / 1 / 20 SwGroup / Switch / Port Pools DataMover Pool IP Name 2 p1 192.168.11.204 cel-p1-st

2.5.16 Adding a Pool to a NAS To be able to use an existing NAS for a pool, the network connection to the NAS has to be enhanced. On the NAS, a new virtual interface has to be created and on the switch ports the new VLAN has to be configured. All these steps are done with ff_nas_adm.pl using the operation mode add-pool, but the program will not change any exports.

For EMC NAS the switch ports to any data mover of an EMC Celerra are allowed for all VLANs the NAS is using. So any data mover is able to take over the role of any other data mover.



Synopsis

ff_nas_adm.pl --op add-pool --name <node_name> --pool <pool_name>

[--role {master|slave}] [--host <ip_host_part>] [--blade <blade_id>]

Options

--op add-pool Adds the given pool to named NAS.

--name <node_name> Defines the node name of NAS.

--pool <pool_name> Defines the pool name the NAS has to be support. See usage for a list of configured pools.

--role {master|slave} Defines the role of NAS within given pool. This information is used by the ff_setup_sid_folder program. If this option is omitted the script uses the default role master.

--host <ip_host_part> Defines the host part to be used to build IP addresses for the Control or Storage LAN networks. If this option is omitted, the script uses a free host number to calculate the IP address.

--blade <blade_id> Blade/data mover Id of an EMC Celerra NAS. The Id is the numerical value of the blade/ data mover, eg. server_2 means Id 2. If this option is omitted the script uses the default of Id 2.

Output example

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op add-pool --name filer --pool pool4 update LDAP .... update switch 1/1 configuration Notice: Update will take about 1 minute. ...........+ vlan: storage-25 has been created Pool pool4 successfully added to NAS, LDAP and network.



2.5.17 Removing a Pool from a NAS The rem-pool mode is the opposite to the add-pool operation mode of ff_nas_adm.pl. It removes the virtual interface to the pool and removes VLAN access on NAS switch ports for the given pool. This action will be permitted if any SID of the pool uses this NAS or the NAS is the default NAS of the pool. Within the last case, the pool interface of the NAS will be removed when removing the pool.

For EMC NAS the pool’s VLAN will only be removed from EMC Celerra switch ports if no longer any data mover is configured for this pool.

Synopsis

ff_nas_adm.pl --op rem-pool --name <node_name> --pool <pool_name>

--blade <blade_id>

Options

--op rem-pool Removes the given pool from the named NAS.


--pool <pool_name> Defines the pool name the NAS has to be support. See usage for a list of configured pools.

--blade <blade_id> Blade/ data mover Id of an EMC Celerra NAS. The Id is the numerical value of the blade/ data mover, eg. server_2 means Id 2. If this option is omitted the script uses the default of Id 2.

Output example

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op rem-pool --name filer --pool pool4 update switch 1/1 configuration Notice: Update will take about 1 minute. ...........+ update LDAP .... Pool pool4 successfully removed from NAS, LDAP and network.



2.5.18 Adding a Blade (Datamover) to an EMC Celerra NAS An EMC Celerra may consist of more than one data mover. To add one or more data movers to an existing NAS use this command. It will add configuration data to LDAP and assign and configure switch ports. It will display instructions how to interconnect the new data movers with the FlexFrame switch group.

Synopsis

ff_nas_adm.pl --op add-blade --name <node_name>

--ports <port_count> --blades <blade_count>

Options

--op add-blade Adds blades/ data movers to NAS.


--ports <port_count> Count of ports to be used with pool storage lan networks. If this option is omitted the program uses the default of 2.

--blade <blade_id> Blades/data mover count to be added. If this option is omitted the program uses the default of 1.

Output example

cn1:~ # ff_nas_adm.pl --op add-blade --name cel update LDAP ... update switch 1/1 configuration Notice: Update will take about 1 minute. ..................................+ As the switch ports are already configured the correct wiring between DataMover and switch ports has to be done. See below a list of cable connections. Connect your DataMover LAN interfaces to named switch ports: SwitchGroup / Switch / Port LAN Interface 1 / 2 / 11 cel (NS-Type): server_4: port "data NIC-1" 1 / 1 / 11 cel (NS-Type): server_4: port "data NIC-2" The complete instruction above is listed at file /tmp/cel-add/todo



2.5.19 Removing a NAS To remove a NAS from FlexFrame landscape it may not be used by any pool. ff_nas_adm.pl --op list will display the configuration of NAS. Ports on NAS were configured with maintenance tools ff_pool_adm.pl and ff_nas_adm.pl.

The switch port configuration will be removed from switch and LDAP database.

Synopsis

ff_nas_adm.pl --op rem --name <node_name>

Options

--op rem Removes a NAS from FlexFrame landscape about the NAS configuration and dis-plays some information about processing steps.


Output example

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op rem --name filer2 update switch 1/1 configuration Notice: Update will take about 1 minute. .............................+ update LDAP ..... NAS successfully removed from network and LDAP.

2.5.20 NAS Disk Free In change to FlexFrame 3.x some NAS file systems are no longer mounted. Formerly a df command listed the mounted file systems and the available space. To get the same information the new program ff_nas_df.pl may be used. It connects to all FlexFrame NAS and views the disk free (df) information. To be comparable all size values are given as kilo bytes. The view may be restricted to a single NAS.

Synopsis

ff_nas_df.pl [--name <node_name>] [--mounted]

FlexFrame Architecture LDAP


Options

--name <node_name> Name of NAS to be viewed. Omit this option to view all.

--mounted Adds Mounted on column at view. Typically it is the same like the file system and therefore omitted.

Output example

cn1:~ # ff_nas_df.pl cel server_2 Filesystem total KB used KB avail KB capacity mxi_data 20166984 18441704 1725280 91% mxi_logs 5040984 4098672 942312 81% root_fs_2 114592 712 113880 1% root_fs_common 13624 5256 8368 39% bar_default_data 302511984 265544152 36967832 88% bar_default_logs 70585984 22607672 47978312 32% volFF_bar 161339984 120676000 40663984 75% f940 Filesystem total KB used KB avail KB capacity /vol/data_o10/ 188743680 174010928 14732752 92% /vol/foo_default_data/ 308281344 273950812 34330532 89% /vol/foo_default_logs/ 51380224 24319148 27061076 47% /vol/linux_luns/ 16777216 872 16776344 0% /vol/logs_o10/ 18874368 1459008 17415360 8% /vol/oxi_data/ 20552092 16974148 3577944 83% /vol/oxi_logs/ 2055212 424528 1630684 21% /vol/sapdata/ 10276048 26080 10249968 0% /vol/saplog/ 2055212 384920 1670292 19% /vol/vol0/ 8388608 380648 8007960 5% /vol/volFF/ 99614720 76749520 22865200 77% /vol/volFF_foo/ 209631316 181518524 28112792 87%

2.6 LDAP A directory service is typically a specialized database which is optimized for reading and searching hierarchical structures. Unlike RDBMS, these databases do not support trans-action based processing. LDAP (Lightweight Directory Access Protocol, see RFC 2251) defines means of setting and retrieving information in such directories. The structure of the LDAP data is called Domain Information Tree (DIT).

LDAP FlexFrame Architecture


2.6.1 FlexFrame Structure in LDAP FlexFrame utilizes LDAP for two different purposes:

(1) for operating naming services (such as host name resolution, user/password re-trieval, tcp/udp service lookup, etc.) and

(2) for storing FlexFrame specific data on the structure of the installed environment

Application Nodes are only able to search in area (1). It is separated into pool specific sections in order to protect pools from accessing other pools’ data. Each of them contains pool specific network information service (NIS) like data. The LDAP servers have access lists to prevent searches outside of the own pool.

The other main DIT part contains FlexFrame configuration data (2). It should only be accessed through maintenance tools from one of the Control Nodes. This part of the DIT contains a lot of cross references, which need to be kept in sync. Do not try to change this data, unless you are explicitly instructed by Fujitsu Siemens Computers support to do so.

2.6.2 Working with LDAP The usage of LDAP specific commands like ldapadd or ldapmodify is limited to very few actions. One is to create or remove a PUT service for a SAP system copy. This ac-tion is described within the “Installation Guide for SAP Solutions” manual. Other direct interaction through LDAP commands is limited to service issues.

No other interventions have to and should be done. The FlexFrame maintenance tools provide the necessary functionality.

2.6.3 Disaster Recovery – Turning Replica into Master As there are two LDAP databases running in a FlexFrame environment (one on each Control Node), one of them is running as writable master DB while the other one is a non-writable replica.

In case of a disaster, one Control Node may fail, and therefore its LDAP server becomes unusable.

In case (only) the replica fails, the remaining master LDAP DB continues to work.

In case the master DB fails, the remaining replica LDAP DB will provide the functionality. However, if the LDAP DB needs to be changed, the replica needs to act as master (i.e. become read-writable).

This requires manual interaction. The procedure is described in the document "Disaster Procedures for FlexFrame Landscapes", slide “FlexFrame Control Node (LDAP master)".

FlexFrame Architecture PRIMECLUSTER


2.7 PRIMECLUSTER This section briefly describes how PRIMECLUSTER is implemented in FlexFrame.

PRIMECLUSTER is used for high availability of Control Nodes only, not for Application Nodes.

2.7.1 PRIMECLUSTER Components PRIMECLUSTER in FlexFrame consists of several components:

● Cluster Founation (CF) )

CF controls the cluster node interoperability and builds a base for all other PRIME-CLUSTER components.

● Shutdown Facility (SF)

SF controls power shutdown of the cluster nodes in case of cluster node failures. In FlexFrame, Shutdown Facility agents will also be used by the FA Control Agent.

● Reliant Monitor Services (RMS)

RMS manages all defined service groups (“userApplication”) by starting, stopping and monitoring services. It will react if services fail and switch it over to another clus-ter node.

2.7.2 FlexFrame Specific RMS Configuration FlexFrame delivers a pre-configured RMS configuration divided in five service groups, called “userApplication”. The table below shows the application groups and other respec-tive properties:

userApplication may switch over?

service used by / needed for

FA Control Agent n/a

mySQLdatabase myAMC FA Messenger Server

myAMC FA Mes-senger

n/a

ff_manage

This application contains services for automatic and manual manage-ment facilities.

yes

jakarta tomcat myAMC FA Messenger Server web interface

PRIMECLUSTER FlexFrame Architecture


userApplication may switch over?

service used by / needed for

tftp daemon Solaris and Linux netboot

DHCP daemon Linux netboot

RARP daemon Solaris netboot

netboot_srv

This application contains services required for net-boot

yes

bootparam daemon Solaris netboot

ldap_srv1 no, CN1 only

LDAP master server (slapd)

ldap replication daemon (slurpd)

ldap_srv2 no, CN2 only

LDAP replica server (slapd)

important component of the complete FlexFrame environ-ment

saprouter (optional)

yes optional saprouter initially unconfigured

n/a

All these services are monitored by PRIMECLUSTER and automatically restarted, if they fail. If a recovery is not possible, the application will be set into faulted state and is tried to start on the other Control Node. This process is called “failover”.

Individual userApplications may be switched to the other Control Node, i.e. it is possible to run ff_manage on the first and netboot_srv on the second Control Node.

PRIMECLUSTER uses the scripts in /etc/PRIMECLUSTER/bin for starting, stopping and monitoring services. These scripts are delivered with the ffPCL RPM.

If you try to stop running services manually, RMS will restart them. To prevent this, you should set the corresponding userApplication into maintenance mode. See the command overview table below and section “PRIMECLUSTER Admini-stration” on page 77.

You may damage your configuration if you ignore this!

2.7.3 RMS Configuration – Schematic Overview The following picture shows the initial state of PRIMECLUSTER RMS: ● ldap_srv1 runs on Control Node 1 and may not be switched to Control Node 2 ● ldap_srv2 runs on Control Node 2 and may not be switched to Control Node 1 ● netboot_srv, ff_manage and saprouter (if configured) run on Control Node 1

and can be individually switched to Control Node 2



s t a

r t

s t o pldap_srv1

ControlNode 1

jakarta tomcat

myAMC Msg

mysql

myAMC Ctrl

bootparamd

rarpd

dhcpd

tftpd

ldap masterslapd + slurpd

ldap replicaslapd

netboot_srv ff_manage ldap_srv2

ControlNode 2

saprouter

saprouter

virtual ip

switchover

optional

switchover

switchover

active service(s)on Control Node 1

active service(s)on Control Node 2

2.7.4 Node Failures When a Control Node crashes or freezes, CF will detect an interconnect timeout. SF will then try to issue a power cycle command to the IPMI interface of the failed Control Node.

If this command is successful, RMS will switch over netboot_srv, ff_manage and, if configured, saprouter if they were online on the faulted Control Node.

If the IPMI interface does not respond, PRIMECLUSTER cannot exactly determine which Control Node is still alive. To prevent inconsistencies and data loss, RMS will not switch over any applications. In this case you have to interact manually.

PRIMECLUSTER cannot handle situations where a node is completely powered off in-cluding the on-board IPMI interface. Therefore two power supplies have to be installed in the Control Nodes to prevent inconsistent cluster states in case of power failures.

If you boot with only one Control Node, RMS will not start switchable services to prevent inconsistencies. In this situation you have to do a forced online switch of netboot_srv, ff_manage, ldap_srv1 (on the first Control Node), ldap_srv2 (on the second Control Node) and, if configured, saprouter.

See command table below or use the PRIMECLUSTER Administration interface.



2.7.5 PRIMECLUSTER CLI Commands Here is a quick overview of PRIMECLUSTER CLI commands. For further information, read the appropriate man pages by typing man <command> and the official PRIMECLUSTER manuals.

Action Command

Display state of all resources hvdisp -a

Display and continuously update state of all resources

hvdisp -u

Display state of all resources of the specified type (-u = update)

hvdisp [-u] -T <resource_type>

Start or switch over the specified user application (-f = forced switch)

hvswitch [-f] <userApplication> [<node>]

Stop the specified user application hvutil -f <userApplication>

Clear faulted user application hvutil -c <userApplication>

Set the specified user application into maintenance mode

hvutil -m on <userApplication>

Set all user applications into mainte-nance mode

hvutil -M on

Exit the specified user application’s maintenance mode (forced exit)

hvutil -m [force]off <userApplication>

Exit cluster maintenance mode (forced exit)

hvutil -M [force]off

Stop RMS on all nodes (do offline processing)

hvshut -a

Stop RMS on all nodes (don’t do offline processing)

hvshut -A

Forced/emergency stop of RMS on local node (don’t do offline processing)

hvshut -f

Stop RMS on the local node (do offline processing)

hvshut -l

Forces stop of RMS on local node (don’t do offline processing)

hvshut -L



Action Command

Start RMS on local node hvcm

Verify shutdown facility status sdtool -s

Reinitialize shutdown facility sdtool -r

Stop PRIMECLUSTER CF including all dependent components

/opt/SMAW/SMAWcf/dep/master unload

/etc/init.d/cf stop

Start PRIMECLUSTER CF including all dependent components

/etc/init.d/cf start

/opt/SMAW/SMAWcf/dep/master load

Test the IPMI Login ipmipower -s <IP_address> -u OEM -p <password>

Display the Cluster Foundation nodes and their current status

cftool -n

Display all LAN interfaces recognized by the OS (Solaris/Linux)

cftool -d

Some commands (hvswitch, hvutil, etc.) return without showing any result. Verify the result using hvdisp -a or hvdisp -u and/or check the logfiles (es-pecially switchlog, see below).

2.7.6 Usage Example: Switch Application If you are not familiar with PRIMECLUSTER, here is an example how to switch the appli-cation netboot_srv from the second Control Node to the first one.

First, take a look at the initial situation:

control1:~ # hvdisp -T userApplication Local System: control1RMS Configuration: /opt/SMAW/SMAWpcs/Config/FlexFrame/runtime/FlexFrame.us Resource Type HostName State StateDetails ------------------------------------------------------------------------ netboot_srv userApp Offline netboot_srv userApp control2RMS Online ldap_srv2 userApp ldap_srv2 userApp control2RMS Online ldap_srv1 userApp Online ff_manage userApp Online



As you can see, netboot_srv is online on the second Control Node.

To switch the netboot services to the first Control Node, call:

control1:~ # hvswitch netboot_srv control1

The suffix RMS of the target node is not necessary for most commands.

The target node is optional. If not specified, the application will be switched to the pre-ferred node, therefore in this case you could have also used:

control1:~ # hvswitch netboot_srv

This command returns immediately.

In the background, RMS will perform the switch operation in two steps:

1. The netboot services rarpd, bootparamd, inetd and dhcpd will be taken offline on the second Control Node

2. The netboot services will be taken online on the first Control Node

To see the result, call:

control1:~ # hvdisp -T userApplication Local System: control1RMS Configuration: /opt/SMAW/SMAWpcs/Config(FlexFrame/runtime/FlexFrame.us Resource Type HostName State StateDetails ------------------------------------------------------------------------ netboot_srv userApp Online ldap_srv2 userApp ldap_srv2 userApp control2RMS Online ldap_srv1 userApp Online ff_manage userApp Online As you can see after a few seconds, netboot_srv has been switched over to the first Control Node.

The command hvswitch will also be used to start an application that is offline on both nodes. If nothing happens after issuing the hvswitch command, see switchlog and netboot_srv.log for the cause (see below).

Remember that some applications (especially ff_manage) can take minutes to stop and start.

For information how to access the administration interface of PRIMECLUSTER, please refer to section “PRIMECLUSTER Administration” on page 77.



2.7.7 PRIMECLUSTER Log Files All PRIMECLUSTER components log their actions to log files. Every action will be logged. These files are important to find out the reasons for errors.

Whenever PRIMECLUSTER reports an error, it is very unlikely that PRIME-CLUSTER is the cause for this fault. In most situations, a service controlled by PRIMECLUSTER has failed (i.e. due to misconfiguration) and PRIMECLUSTER has detected and reported this failure.

Please take a look into the logfiles before contacting Fujitsu Siemens Computers support.

Here is an overview of important log files:

Component Log File

RMS switchlog /var/opt/reliant/log/switchlog

RMS application log /var/opt/reliant/log/<userApplication>.log

CF /var/log/messages (dmesg)

SF /var/opt/SMAWsf/log/rcsd.log

2.7.8 Disaster Repair If the FlexFrame landscape is distributed over two locations and if one side fails com-pletely, it may be necessary to clean up the state of the services monitored by PRIME-CLUSTER. This can be done using the tool ff_rms.sh.

The purpose of this tool is to support the administrator in recovering the RMS services in a disaster situation only.

The following services are monitored by PRIMECLUSTER's RMS component:

netboot_srv Monitoring inetd, dhcpd, rarpd and bootparamd

ldap_srv1 Monitoring LDAP server on Control Node 1 (master)

ldap_srv2 Monitoring LDAP server on Control Node 2 (replica)

ff_manage Monitoring FA Control Agents

If these services are not in state Online, this tool tries to restart them. If the partner Control Node is not available, ALL services (which may run locally will be stopped and restarted again).

Network FlexFrame Architecture


Those services will be unavailable during the restart.

The program will stop if a service is in maintenance mode. Since this indicates an opera-tor intervention, it must be fixed manually, e.g. by calling:

hvutil -m off <service> Debugging information can be found in /tmp/ff_rms.DEBUGLOG. In case of problems, please provide this file.

2.8 Network The network is the backbone of the FlexFrame solution. Communication between the various nodes and storage devices is done exclusively via the IP network infrastructure. It serves both, communication between servers and clients as well as delivering IO data blocks between the NAS (Network Attached Storage) and the servers.

The IP network infrastructure is essential for every FlexFrame configuration. FlexFrame is designed with a dedicated network for connections between servers and storage that is reserved for FlexFrame traffic only. One network segment, the Client LAN (see below) can be routed outside the FlexFrame network to connect to the existing network.

2.8.1 LAN Failover The term „LAN failover“ describes the ability of a FlexFrame environment to use a logical network interface that consists of several physical network interface cards (NICs), which in turn are using redundant network paths (cables and switches). When a network com-ponent (NIC, cable, switch, etc.) fails, the network management logic will switch over to another network interface card and path.

2.8.2 Segments FlexFrame 4.0 introduces a new network concept, providing higher availability as well as increased flexibility in virtualizing the whole FlexFrame landscape.

This concept relies on VLAN technology that allows running multiple virtual networks across a single physical network. Additionally, in order to ensure high network availability, LAN bonding is used on every node. This includes a double switch and wiring infrastruc-ture, to keep the whole environment working, even when a network switch or cable fails. Additionally, the network segements are prioritized, to ensure that important connections are prefered.

FlexFrame Architecture Network


Similar to former versions, there are four virtual networks within FlexFrame. In FlexFrame 4.0 these networks run through one logical redundant NIC, using bonding on Linux and IPMP on Solaris.

The following figure outlines the basic network segments of a typical FlexFrame land-scape with Linux-based Application Nodes.

Virtual Network Segments

Application Nodes

Blade servers

NAS System

Mgmt.BladeRSB * IPMI *Clients

bond0

eth0 eth1

bond0

eth1 eth2

bond0

eth0 eth1

*) Power shut down facility my be: IPMI, RSB, RPS, XSCF or SCF

Server LAN

Client LAN

Storage LAN

Control LANVLANs:

Control Node 1

Bond 0cip0

Onboard

IPMI eth 0 eth 1

LAN A LAN B

eth 2PCI - NIC

V2 V3 V4

Control Node 2

Bond 0cip0

Onboard

IPMI eth0 eth 1

LAN A LAN B

eth 2PCI - NIC

V2 V3 V4

XSCF

IPMP

fjgi0 fjgi1

The following virtual network segments are mandatory:

● Client LAN

The purpose of the Client LAN segment is to have dedicated user connectivity to the SAP instances. This segment also allows administrators to access the Control Nodes.

● Control LAN

The Control LAN segment carries all RSB, IPMI, XSCF, RPS, e0 and administrative communication.

Control LAN access to the Application Nodes is required for interventions using the Remote Service Board (RSB).



● Server LAN

The Server LAN segment is used for the communication between SAP instances among each other and the databases.

● Storage LAN

The Storage LAN segment is dedicated to NFS communication for accessing the executables of SAP and the RDBMS as well as the IO of the database content and SAP instances.

The network bandwidth has to be one Gigabit for all components, since intense storage traffic has to be handled.

2.8.3 Network Switches Network switching components play a very important role within FlexFrame. Therefore, only the following switch types are tested and supported:

● Cisco Catalyst WS-C3750G-24TS

● Cisco Catalyst WS-C3750G-24T

● Cisco Catalyst WS-C3750G-48TS

with IOS 12.2.25 or higher.

These models support VLAN technology for a flexible configuration for the various net-work segments and QoS (quality of service) to prioritize traffic (e.g. higher priority for production systems over test systems etc).

2.8.4 Automounter Concept The Automounter Concept is based on the ability of Linux and Solaris to mount file sys-tems automatically when their mount points are accessed.

During the boot process of an Application Node some file systems are mounted. For Linux these are the root file system (read-only) as well as the /var mount point (read-write). For Solaris Application Nodes the root file system is mounted read-write and the /usr mount point is mounted read-only. These are the basic file systems which must be accessible for the Application Node to function properly. There is no data in the two file systems that is specific for a SAP service.

Data which is specific for a SAP service, a database or a FlexFrame Autonomous Agent is found in directories which are mounted on first access. Some of the mounts will stay as long as the Application Node is operational. Others will be unmounted again, if directo-ries or files below that mount point have not been accessed for a certain period of time.

Within the LDAP database there are two types of data which relate to the automounter configuration: automountMap and automount.



An automountMap is a base for automount objects. Here’s how to list the automount-Maps:

control1:~ # ldapsearch -x -LLL '(objectClass=automountMap)' dn: ou=auto.FlexFrame,ou=Automount,ou=pool2,ou=Pools,ou=FlexFrame, dc=flexframe,dc=wdf,dc=fujitsu-siemens,dc=com objectClass: top objectClass: automountMap ou: auto.FlexFrame dn: ou=auto_FlexFrame,ou=Automount,ou=pool2,ou=Pools,ou=FlexFrame, dc=flexframe,dc=wdf,dc=fujitsu-siemens,dc=com objectClass: top objectClass: automountMap ou: auto_FlexFrame ...

The base directory looks like this:

dn: cn=/FlexFrame,ou=auto_master,ou=Automount,ou=pool1,ou=Pools, ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu-siemens,dc=com objectClass: top objectClass: automount cn: /FlexFrame automountInformation: auto_FlexFrame

Further on there are entries like:

dn: cn=myAMC,ou=auto_FlexFrame,ou=Automount,ou=pool1,ou=Pools, ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu-siemens,dc=com objectClass: top objectClass: automount cn: myAMC automountInformation: -rw,nointr,hard,rsize=32768,wsize=32768,proto=tcp,nolock, vers=3 filpool1-st:/vol/volFF/pool-pool1/pooldata/&

There are two things that have to be pointed out.

First, the ou=auto_FlexFrame denotes an entry which is dedicated for Solaris Applica-tion Nodes and referes to the base directory as shown before. The same entries for Linux Application Nodes are named ou=auto.FlexFrame (point instead of an underscore).

The second notable aspect in this entry is the use of the wildcard &. If the folder /FlexFrame/myAMC is accessed, the autofs process tries to mount it from the path filpool1-st:/vol/volFF/pool-pool1/pooldata/myAMC. If the folder myAMC is found and permissions allow the clients to access it, it will be mounted to /FlexFrame/myAMC/<name>. If myAMC is not found or the client does not have the



permissions, the folder will not be mounted. In such a case, try to mount the folder manu-ally on a different folder, e.g. like:

an_linux:~ # mount filpool1-st:/vol/volFF/pool-pool1/pooldata/myAMC /mnt If you get an error message like Permission denied, check the exports on the NAS system and the existence of the directory myAMC/ itself.

Other entries in LDAP make use of variables like ${OSNAME} which is either Linux or SunOS:

dn: cn=/,ou=auto.oracle,ou=Automount,ou=pool2,ou=Pools, ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu-siemens,dc=com objectClass: top objectClass: automount cn: / description: catch-all for Linux automount automountInformation: -rw,nointr,hard,rsize=32768,wsize=32768,proto=tcp,nolock, vers=3 filpool2-st:/vol/volFF/pool-pool2/oracle/${OSNAME}/&

On Linux, the automount mount points can be read using the following command:

an_linux:~ # mount rootfs on / type rootfs (rw) /dev/root on / type nfs (ro,v3,rsize=32768,wsize=32768,reserved, hard,intr,tcp,nolock,addr=192.168.11.206) 192.168.11.206:/vol/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/var_img/var-c0a80b36 on /var type nfs (rw,v3,rsize=32768,wsize=32768,reserved, hard,intr,tcp,nolock,addr=192.168.11.206) 192.168.11.206:/vol/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/var_img/var-c0a80b36 /dev on /dev type nfs (rw,v3,rsize=32768,wsize=32768,reserved, hard,intr,tcp,nolock,addr=192.168.11.206) 192.168.11.206:/vol/volFF/os/Linux/pool_img/pool-c0a80bff on /pool_img type nfs (rw,v3,rsize=32768,wsize=32768,reserved, hard,intr,tcp,nolock,addr=192.168.11.206) proc on /proc type proc (rw) devpts on /dev/pts type devpts (rw) shmfs on /dev/shm type shm (rw) /dev/ram on /var/agentx type ext2 (rw) automount(pid1750) on /FlexFrame type autofs (rw) automount(pid1772) on /saplog/mirrlogA type autofs (rw) automount(pid1752) on /home_sap type autofs (rw) automount(pid1788) on /saplog/saplog1 type autofs (rw) automount(pid1766) on /sapdata/sapdata5 type autofs (rw)



automount(pid1778) on /saplog/origlogA type autofs (rw) automount(pid1762) on /sapdata/sapdata3 type autofs (rw) automount(pid1758) on /sapdata/sapdata1 type autofs (rw) automount(pid1784) on /saplog/saparch type autofs (rw) automount(pid1764) on /sapdata/sapdata4 type autofs (rw) automount(pid1786) on /saplog/sapbackup type autofs (rw) automount(pid1760) on /sapdata/sapdata2 type autofs (rw) automount(pid1754) on /myAMC type autofs (rw) automount(pid1796) on /usr/sap type autofs (rw) automount(pid1780) on /saplog/origlogB type autofs (rw) automount(pid1768) on /sapdata/sapdata6 type autofs (rw) automount(pid1792) on /saplog/sapreorg type autofs (rw) automount(pid1776) on /saplog/oraarch type autofs (rw) automount(pid1774) on /saplog/mirrlogB type autofs (rw) automount(pid1770) on /sapdb type autofs (rw) automount(pid1756) on /oracle type autofs (rw) automount(pid1790) on /saplog/saplog2 type autofs (rw) automount(pid1794) on /sapmnt type autofs (rw) filpool2-st:/vol/volFF/pool-pool2/pooldata on /FlexFrame/pooldata type nfs (rw,v3,rsize=32768,wsize=32768,reserved,hard,tcp, nolock,addr=filpool2-st)

If your Application Node is a Solaris server, use the following command:

an_solaris:~ # ldaplist -l ou=auto_master,ou=Automount dn: cn=/FlexFrame,ou=auto_master,ou=Automount,ou=pool1,ou=Pools, ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu-siemens,dc=com objectClass: top objectClass: automount cn: /FlexFrame automountInformation: auto_FlexFrame dn: cn=/home_sap,ou=auto_master,ou=Automount,ou=pool1,ou=Pools, ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu-siemens,dc=com objectClass: top objectClass: automount cn: /home_sap automountInformation: auto_home_sap dn: cn=/myAMC,ou=auto_master,ou=Automount,ou=pool1,ou=Pools, ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu-siemens,dc=com objectClass: top objectClass: automount cn: /myAMC automountInformation: auto_myAMC

NAS System FlexFrame Architecture


dn: cn=/sapdata/sapdata1,ou=auto_master,ou=Automount,ou=pool1, ou=Pools,ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu-siemens, dc=com objectClass: top objectClass: automount cn: /sapdata/sapdata1 automountInformation: auto_sapdata1

... The cn: parts show the mount points.

2.9 NAS System

2.9.1 Network Appliance Filer In FlexFrame 4.0, the storage for all Application Nodes can be one or more NAS systems (Filers) from Network Appliance.

Fujitsu Siemens Computers are working jointly with Network Appliance (see http://www.netapp.com) on the development of the FlexFrame concept. The Network Appliance product class “Filer” is an essential part of the FlexFrame solution.

The operating system of the Filer is called "ONTAP". The disks will be grouped into RAID groups. A combination of RAID groups will make a volume. Starting with ONTAP 7, also aggregates can be created. FlexVolumes can be created on top of aggregates. A Filer volume contains a file system (WAFL - Write Anywhere File Layout) and provides vol-umes or mount points for NFS (for UNIX systems) or CIFS (for Windows systems). The Filer has NVRAM (Non Volatile RAM) that buffers committed IO blocks. The contents of the NVRAM will remain intact if the power of the Filer should fail. Data will be flushed to the disks once power is back online.

The minimal FlexFrame landscape has at least the following volumes:

● vol0 (ONTAP, configuration of Filer)

● sapdata (database files)

● saplog (database log files)

● volFF (OS images of Application Nodes, SAP and database software, pool related files)

In FlexFrame 4.0, the volume volFF separates FlexFrame data (file system of Applica-tion Nodes and other software) from the Filer's configuration and ONTAP. In larger instal-lations, multiple sapdata and saplog volumes can be created (e.g. to separate produc-tion and QA etc.).

FlexFrame Architecture NAS System


2.9.1.1 Built-in Cluster File System

The Network Appliance implementation of the NFS (Networked File System) allows shar-ing of the same data files between multiple hosts. No additional product (e.g. cluster file system) is required.

2.9.1.2 Volume Layout

The FlexFrame concept reduces the amount of "wasted" disk space since multiple SAP systems can optionally share the same volume of disks. As the data grow, it is easy to add additional disks and enlarge the volumes without downtime.

2.9.1.3 Snapshots

When a snapshot is taken, no data blocks are being copied. Just the information where the data blocks are located is saved. If a data block is modified, it is written to a new location, while the content of the original data block is preserved (also known as “copy on write”). Therefore, the creation of a snapshot is done very quickly, since only few data have to be copied. Besides that, the snapshot functionality provided by NetApp is unique, because the usage of snapshots does not decrease the throughput and performance of the storage system.

Snapshot functionality will allow the administrator to create up to 250 backup-views of a volume. The functionality “SnapRestore” provided by NetApp significantly reduces the time to restore any of the copies if required. Snapshots will be named and can be re-named and deleted. Nested snapshots can be used to create e.g. hourly and daily back-ups of all databases. In a FlexFrame landscape, a single backup server is sufficient to create tape backups of all volumes. Even a server-less backup can be implemented. Off-Line backups require a minimal down-time to the database because the backup to tape can be done reading form a quickly taken snapshot.

2.9.1.4 Filer Cluster

A Filer can be clustered to protect data against the failure of a single Filer. Switching from one Filer to its cluster counterpart is transparent to the Application Nodes.

Filer clustering is a functionality of Network Appliance Filers.

2.9.2 EMC NAS (Celerra) In FF4SAP V4.0 you can use an EMC NAS (Network Attached Storage of EMC = Cel-erra) instead of only a NetApp Filer. In the following we use “NAS system” or “NAS” to indicate that we mean NetApp Filer and/or EMC NAS. Several times we use “Celerra” for EMC NAS.

NAS System FlexFrame Architecture


An EMC Celerra consists of control station(s), data mover(s) and a storage system, which is a CLARiiON® or a Symmetrix®, depending on whether it is an integrated or a gateway variant.

The EMC NAS family is composed of two - for FlexFrame for SAP relevant - product lines:

● The NS line - available in both integrated and gateway models offering choice of 1, 2 or 4 data movers that attach to CLARiiON, or in the case of a gateway, to CLARiiON or Symmetrix storage subsystems.

● The high end CNS with a high performance fault tolerant cluster of data movers attached to enterprise class Symmetrix storage subsystems and/or modular CLARiiON subsystems.

The minimal FlexFrame landscape has at least the following volumes:

● sapdata (database files)

● saplog (database log files)

● volFF (OS images of Application Nodes, SAP and database software, pool related files)

2.9.2.1 Control Station

The control station is a management computer which controls components of the Celerra such as data movers (see below). The control station works with RedHat Linux as ope-rating system, which is customized to it’s needs.

The control station is able to connect to each data mover in the Celerra and send com-mands to the them. After the data movers are booted, they do not depend on the control station for normal operation. In the unlikely event the control station fails, the data movers continue to serve files to clients. Depending on the model, your Celerra Network Server may include an optional standby control station that takes over if the primary control sta-tion fails.

2.9.2.2 Data Mover

The data movers are the Celerra components that transfer data between the storage system and the network client. The data mover operating system is DART (Data Access in Real Time). You do not manage a data mover directly. You use the control station to send commands to the Data Mover.

A Celerra can have from 1 to 14 data movers. Normally, the data movers are named server_n, from server_2 to server_15.

A data mover can be active (or primary) or, in the case of a Celerra Network Server model that supports multiple data movers, can be a standby for other data movers. How-ever you must configure one or more data movers to support high availability.

FlexFrame Architecture NAS System


A data mover communicates with the data storage systems using Fibre Channel connec-tions. In the Celerra, this is done by a Fibre Channel HBA card, that connects to the host through a Fibre Channel switch. In the Celerra NS series, storage processing is provided by two storage processors.

2.9.2.3 Cluster Architecture

The EMC Celerra product family have distributed clustered architectures that give these products competitive level of scalability. The EMC Celerra NS family supports one to four data movers in a single system, and the EMC Celerra CNS can support up to 14 data movers. This scalability allows customers to grow their EMC NS and CNS environments by adding data movers to enhance bandwidth, processing power and cache memory. The EMC Celerra implementation of the NFS (Networked File System) allows sharing of the same data files between multiple hosts. No additional product (e.g. cluster file system) is required.

The Celerra Network Server protects against failures (service interruption or data loss) with an additional control station and one or more data movers that can take over the operations of a failed component.

Each data mover is a completely autonomous file server with its own operating system. During normal operations, the clients interact directly with the data mover, not only for NFS access, but also for control operations such as mounting and unmounting file sys-tems. Data mover failover protects the Celerra Network Server against total or partial hardware failure of an individual data mover. You can define one data mover as standby for the others (see Installation Guide “Configuration of Standby Data Mover”). This standby data mover has no own MAC address and IP address, no hostname. In the case of a failover it takes all these from the failed data mover, so that the client cannot decide, whether he is connected with the original or the standby data mover. In the same time of the failover, the control station alerts the EMC service center via modem, and an EMC technician can diagnose the problem and do corrective actions.

2.9.2.4 Snapshots

Snapshot images of a file system are used to minimize backup windows and to enable quick restores of accidentally deleted or corrupted files. The EMC product for this pur-pose is SnapSure™. It refers to these snapshot images as checkpoints. The amount of storage consumed for each checkpoint is only determined by the amount of data that was written since the last checkpoint was taken. Checkpoint images can be created on de-mand or scheduled according to a time or change based policy.

As an example, consider a SAP database. A checkpoint at midnight is used to quickly create a consistent point in time image of the file system for backup to tape. Checkpoints during business hours as well are used to enable quick restores of accidentally deleted or corrupted files and minimize the window of possible data loss compared to recovery from last night’s tape backup.

SAN Support FlexFrame Architecture


2.10 SAN Support This section provides an architectural overview on the integration of SAN-based storage in the FlexFrame environment. It describes the level of integration provided with the 4.0A00 release of FlexFrame and the components which are considered supported and those which are not explicitly supported, but can be used due to the approach described in here. Reading this section requires a basic technical understanding of the different SAN components (e.g. fibre channel adapter, fibre channel switches and storage compo-nents).

2.10.1 SAN Basic Layers Configuring a SAN environment for one or multiple SAP databases can be a very com-plex task. As with many complex tasks they become easier if you break it down into little pieces. In order to make you understand the various layers of a SAN environment we divide it along the path as seen from the disk to the database:

Level Description

DISK The real disk(s) contained in the storage subsystem (aka storage system).

RAID-GROUP One or multiple disks are grouped together based on RAID mechanisms provided by the storage subsystem.

ARRAY-LUN LUNs as seen by the storage subsystem.

SAN-FABRIC A SAN fabric to connect one or multiple storage subsystems with one or multiple hosts. It consists of fibre channel cabling and switches.

HBA The HBA (host bus adapter) which connects the host to the SAN fabric.

OS-DRIVER Basic OS drivers to make the HBA(s) work.

HOST-LUN LUNs as seen by the host (Application Node).

MULTIPATHING OS specific multipathing software to make multiple HBAs to work as a single virtual interface.

VOLUME MANAGER A OS specific volume manager to group multiple LUNs from the same or different storage subsystems into virtual volume groups.

FILE SYSTEM OS specific file systems created on top of the virtual volumes provided by the volume manager.

DATABASE The database files contained on the file system(s).

FlexFrame Architecture SAN Support


Each of the layers above can be configured in many different ways using lots of varia-tions. Therefore not all of those layers are controlled by FlexFrame.

2.10.2 Scope of the FlexFrame SAN Integration This section describes the scope of the FlexFrame SAN integration. The decisions were made based on a couple of customer projects.

In order to broaden the set of ‘’usable’’ SAN components in the FlexFrame environment the scope of the project was not to integrate the full stack of hardware and software lay-ers described in 2.10.1, but rather take care of the configuration as off a certain level.

Thus, installation and configuration of storage systems, SAN switches/fabrics, volume managers and even driver software is by no means automated, but left to the responsibil-ity of the user. Still, some predefined sets of the software stack components have been tested and explicitly went through a Quality Assurance process.

As a baseline the SAN fabric is ‘’out of scope’’ for all administrative tools within the Flex-Frame environment. Configuration of the storage system, such as configuration of disks into RAID groups, provisioning of LUNs, zoning on FC switches is completely left to the responsibility of the customer. However, some rules do apply to the outcome of a set-up, such as ‘’all servers in one FlexFrame group (NOT pool) have to be able to access the LUNs which are relevant for the SAP databases in question at the same time’’. This rule and further ones are described in detail later in this document.

On the Application Nodes, a set of adapters with pre-defined drivers is supported, in a way that those set-ups are fully described in the documents and went through a quality assurance cycle. Specific software components mentioned later are required in order to support certain features (such as multi-pathing). Some software packages (esp. for Linux), which are generally available, may explicitly be excluded from usage in a Flex-Frame environment, but this is usually due to SAP’s strict rules concerning the use of none-GPL drivers in Linux systems with their software on-top.

FlexFrame 4.0 comes with the utilities to optionally use a set of volume managers, how-ever customer specific software can easily be hook into the environment, if necessary.

The SAN integration is limited to the use for database files and logs only. Meaning every-thing else will still be placed on NAS (NFS) storage. This explicitly includes the database software, SAP applications and the OS. Even the database files do not necessarily have to be placed onto a SAN based storage. One might choose to place large, productive databases with IO requirements onto the SAN based storage while smaller databases will be on NAS storage. The choice is based on SID level. One SID might be on SAN another SID on NAS.

SAN Support FlexFrame Architecture


The following pictures illustrate the components which can be placed onto NAS and SAN storage:

FlexFrame Components on NAS

FlexFrame Components on SAN

2.10.3 Rules and Restrictions In order to make things work correctly with FlexFrame some rules upon each layer must be configured. This chapter lists some rules and restrictions which must be observed.

1. All Application Nodes within one pool group must be configured equally. This in-cludes the number and type of HBAs. They must also use the same OS image.

2. All Application Nodes within one pool-group must have access to the same set of LUNs of each involved storage subsystem. Each Application Node of a pool group must be able to access the same LUNs without reconfiguring the access settings of the storage subsystem.

FlexFrame Architecture SAN Support


3. Host based mirroring can be used on volume manager layer to mirror one or multiple LUNs from one storage subsystem to another (e.g. for a two-site-concept). Copy mechanisms of the storage subsystems can only be used if the failover to a different storage subsystem will not require a change in the WWN and host LUN addressing of the fail-over LUN.

4. A volume group has a N:1 relationship to a SID. There may be multiple volume groups for one SID, but a volume group must not contain volumes for more than one SID.

The appropriate volume groups are switched from one Application Node to the next if a database is moved.

5. Access to all file systems of a SID must be tested on each Application Node of its pool group before it can be monitored and used (“watch”).

6. A database which was installed using SAN based database files in one pool group cannot be started in a different pool group. If this is required the database must be installed on NAS.

Reactivating Application Nodes FlexFrame Basic Administration


3 FlexFrame Basic Administration

3.1 Accessing a FlexFrame Landscape (Remote Administration)

A FlexFrame landscape can be accessed through Secure Shell (ssh, scp) connections to the Control Node. Any other remote administration tools like rsh or telnet have been disabled for reasons of security.

3.2 Powering up the FlexFrame Landscape If the complete FlexFrame landscape was powered off, the following power-on sequence is recommended:

FlexFrame Basic Administration Powering up the FlexFrame Landscape


Before an Application Nodes can be booted, the NTP server must be set up correctly. This can be verified by runing the following command:

control1:~ # ntpq -p remote refid st t when poll reach delay offset jitter ================================================================= *LOCAL(0) LOCAL(0) 5 l 45 64 377 0.000 0.000 0.004 control2-se control1-se 7 u 248 1024 377 2.287 2.587 0.815

Powering off the FlexFrame Landscape FlexFrame Basic Administration


This command needs to be repeated until an asterisk (*) is displayed at the begining of one of the data lines. This character indicates that the NTP server is now ready. If you do not wait and continue with booting, the Application Nodes may work with a different time than the Control Nodes and may (among other possible side effects) create files which may have wrong time stamp information.

If this sequence is not used and all servers are powered on at the same time, the Application Nodes will try to boot while the Control Nodes are not ready to receive the Application Node’s boot request. If this is the case, manual interven-tion is required to re-initiate the boot process of the Application Nodes.

3.3 Powering off the FlexFrame Landscape If you need to power-off the complete FlexFrame landscape (e.g. to move it to a different location) we recommend following the steps as outlined below:

Before shutting down all SAP and DB services, check if users, batch jobs, print jobs and RFC connections to other SAP systems have finished working.

FlexFrame Basic Administration Reactivating ANs after Power Shutdown by FA Agents


To shutdown the SAP and DB services, use the following command for each pool:

control1:~ # stop_all_sapservices <pool_name>

Before you can stop the NAS-System, you need to stop all processes on the Control Nodes. Since those processes are under control of PRIMECLUSTER you can use a single command:

control1:~ # hvshut -a

To stop the NAS system, use the following command.

For a NetApp Filer:

control1:~ # rsh <filer_name> halt

There’s no explicit power-off sequence for the switches. Assuming there are no other devices connected to the switches, they may simply be plugged-off after all components were powered down.

If you do not send an explicit halt command to the Filer, the backup-battery of the Filer maybe drained since the Filer assumes a power-loss and tries to pre-serve the contents of its NVRAM. If the Filer is powered-off for too long the re-sult can be a loss of NVRAM data and a long waiting period during next startup of the Filer.

For an EMC Celerra (where cel-co is the control LAN address of the control station of Celerra):

control1:~ # ssh <cel-co> -l root cel-co: # init0

3.4 Reactivating ANs after Power Shutdown by FA Agents

FlexFrame Autonomy places an Application Node out of service if it is confronted with a problem it cannot solve. The reactions, messages and alarms which take place in this case are described in the “FA Agents - Installation and Administration” manual in the context of the switchover scenarios. It is the responsibility of the administrator to analyze why the node could not be used any longer. We recommend analyzing the FA log and work files as these may be able to supply valuable information. A node which is to start operating as a Spare Node after switchover must be validated using suitable test scenar-ios.


4 Displaying the Current FlexFrame Configuration State

To obtain a general overview of an active FlexFrame system, use the FA WebGUI.

In principle, the FA WebGUI can be used with every browser with SUN JAVA Plugin V1.4.1 or higher which has access to the page on the Control Node. The WebGUI can always be accessed when the Apache Tomcat service is running. This service is normally started by PRIMECLUSTER.

The WebGUI is described in detail in chapter 5.

The login mask expects a user name and password for authentication purposes. You can only use the WebGUI with a valid combination of user name and password. For details on the configuration of the users, see the myAMC documentation for the WebGUI.

Displaying the Current FlexFrame Configuration State Networks


The FA WebGUI provides a presentation of all elements of a FlexFrame system. On the left-hand side the pools, groups, nodes and the active SAP services on the individual nodes are shown in a TreeView.

The TreeView of the FA WebGUI can show either the physical view or the application-related view of the active SAP systems and their instances or a mixed view.

The panels derived from the TreeView (Application Server Panel, Application System Panel, ServiceView and MessageView) always show the objects in relation to the se-lected hierarchical level in the TreeView.

The FA WebGUI is thus the central cockpit for displaying the static configuration of a FlexFrame infrastructure, but as well it is the display for the active SAP systems and their instances. Here, all user interactions such as startup and shutdown are shown directly. All reactions initiate by the FA Agents are displayed as well.

If the FA messenger component has been configured and activated, all traps are stored in a support database. This permits the temporal process for the traps to be displayed very simply at pool, group or node level.

4.1 Networks The network of FlexFrame is its backbone. Here are some tips to get an overview of the current situation on the various networks:

To double-check the network addresses, their names and pool assignment you can use the getent command:

control1:~ # getent networks loopback 127.0.0.0 control 192.168.20.0 storage_pool1 192.168.10.0 server_pool1 192.168.3.0 client_pool1 10.1.1.0 storage_pool2 192.168.11.0 server_pool2 192.168.4.0 client_pool2 10.1.2.0 The loopback network is local for each host and always has the IP address 127.0.0.0.

The control network is the Control LAN network segment for the complete FlexFrame landscape.

In the example we have configured two pools called pool1 and pool2. For each pool there are the three dedicated and distinct segments storage, server and client. The building rule of the network name is <segment>_<pool_name>.

Networks Displaying the Current FlexFrame Configuration State


On the Control Nodes you can see the relation of each pool specific segment to its inter-face using the netstat -r command like this:

control1:~ # netstat -r Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 192.168.100.0 * 255.255.255.252 U 0 0 0 cip0 server_pool2 * 255.255.255.0 U 0 0 0 vlan42 control * 255.255.255.0 U 0 0 0 bond0 control * 255.255.255.0 U 0 0 0 eth1 control * 255.255.255.0 U 0 0 0 eth2 server_pool1 * 255.255.255.0 U 0 0 0 vlan32 storage_pool2 * 255.255.255.0 U 0 0 0 vlan41 client_pool1 * 255.255.255.0 U 0 0 0 vlan30 storage_pool1 * 255.255.255.0 U 0 0 0 vlan31 client_pool2 * 255.255.255.0 U 0 0 0 vlan40 default gw216p1 0.0.0.0 UG 0 0 0 vlan30 The cip0 interface is used for PRIMECLUSTER communication between the two Control Nodes, this network does not have a name associated with it. Here you can quickly see that the Server LAN segment of pool2 (server_pool2) is using the VLAN ID 42 on interface vlan42. Note that the control Control LAN segment is shown on the three interfaces bond0, eth1 and eth2) which is because bond0 is the combination of eth1 and eth2 grouped together for redundancy purposes.

4.1.1 Script: ff_netscan.sh To get more detailed information on each network component, FlexFrame provides a tool ff_netscan.sh. The tool probes all known hosts (derived from LDAP) and those who respond to a broadcast ping in each network (except for the Client LANs).

Gathering the information via SNMP can take some time.

The following outputs are sections of a single ff_netscan.sh command. It was split into several sections for explanation purposes only.

control1:~ # ff_netscan.sh Netscan version 1.28 started Copyright (C) 2004, 2005 Fujitsu Siemens Computers. All rights reserved. SNMP @ 10.1.1.10 (an_0600 ):ok. SNMP @ 10.1.1.20 (an_0700 ):ok. SNMP @ 10.1.1.30 (an_0800 ):ok. SNMP @ 10.1.1.40 (an_0900 ):Not reached. SNMP @ 10.1.1.60 (an_0510 ):ok. SNMP @ 10.1.1.61 (an_0511 ):ok.



Each line represents a request to a certain IP address. Since Application Nodes have multiple IP addresses they are queried multiple times, which is required to detect wrong configurations.

The Application Node an_0900 was “Not reached“ means that this IP address (read from LDAP database) did not respond to an ICMP request. A possible cause may be that this server is powered off.

SNMP @ 10.1.2.205 (control2 ):ok. SNMP @ 10.1.2.254 (gw216p2 ):No SNMP info. The message No SNMP info. is printed if the IP address is available but did not respond to the SNMP request. A possible cause is that the server may not be part of the Flex-Frame landscape. In our example it is a gateway which does not have an SNMP-daemon running.

SNMP @ 192.168.3.10 (an_0600-se ):ok. SNMP @ 192.168.3.12 ( ):ok. SNMP @ 192.168.3.20 (an_0700-se ):ok. SNMP @ 192.168.3.22 ( ):ok.

There are three IP addresses listed which do not have a host name associated with, therefore the names in brackets are missing. In our case this is caused by the test ad-dresses for Solaris Application Nodes (IPMP).

SNMP @ 192.168.20.252 (Cis-Sum ):ok. Scanning switches...

After the last server was probed with SNMP, we scan for available switches.

Now each Control Node or Application Node is listed. There are different types of compo-nents which produce slightly different output.

Here is a sample of a Control Node:

FlexFrame Node information: Nodename OS OS Version Model control1 cn SUSE SLES-9 FSC In the header of the block for this Control Node you see its name and the OS as cn which denotes a Control Node. The OS Version indicates that this Control Node is running SUSE SLES-9. The Model FSC (Fujitsu Siemens Computers) is displayed if no detailed information on the servers exact model name can be found.

Now, each network device is listed along with information on the associated IP address, the MTU size (maximum transfer unit), and MAC address and to which switch and which port the device is connected.



Device IP Addr MTU MAC Switch Port lo 127.0.0.1 16436 n.a. n.a. n.a. eth0 not up 1500 003005406ab8 !undef! !undef! eth1 192.168.20.201 1500 003005406ab9 Cisco1 Gi1/0/3 eth2 no ip 1500 003005406ab9 Cisco1 Gi1/0/3 bond0 no ip 1500 003005406ab9 Cisco1 Gi1/0/3 vlan30 10.1.1.201 1500 003005406ab9 Cisco1 Gi1/0/3 vlan31 192.168.10.201 1500 003005406ab9 Cisco1 Gi1/0/3 vlan32 192.168.3.201 1500 003005406ab9 Cisco1 Gi1/0/3 vlan40 10.1.2.204 1500 003005406ab9 Cisco1 Gi1/0/3 vlan41 192.168.11.204 1500 003005406ab9 Cisco1 Gi1/0/3 vlan42 192.168.4.204 1500 003005406ab9 Cisco1 Gi1/0/3 cip0 192.168.100.1 1500 00c0a8640100 !undef! !undef! cip1 not up 1500 !undef! !undef! cip2 not up 1500 !undef! !undef! cip3 not up 1500 !undef! !undef! cip4 not up 1500 !undef! !undef! cip5 not up 1500 !undef! !undef! cip6 not up 1500 !undef! !undef! cip7 not up 1500 !undef! !undef! sit0 not up 1480 !undef! !undef!

There are several abbreviations which are explained in this table:

Abbreviation Explanation

n.a. not applicable. Since the device lo (loopback) is a logical inter-face with no connection to the outside world, no information on MAC, switch or port can be displayed.

!undef! Marks an undefined status. The device is not configured or in an undefined status. Please check these interfaces. As for a Control Node, the eth0 is used for IPMI and hence not used for the OS. The devices sit0 and cip1 through cip7 are not used.

no ip There is no IP address associated whit this device.

not up This device or interface is not up.



Abbreviation Explanation

Gi<s>/0/<p> This denotes a port on a switch. Gi is short for “GigabitEthernet” and <s> is the switch-number of the stack of switches. <p> is the port number where this particular port is connected to.

The example Gi1/0/3 tells us that the eth1 interface is connected to port 3 of switch #1 of this stack of switches.

Since the “bonding” driver of Linux shares the same MAC address between both lines, only the active port of the re-dundant interfaces is seen.

Po<c> If Po<c> is shown as a port it is a PortChannel (a combination of multiple ports). To see further details of a port channel, you need to read the switch configuration using the command show ether-channel details.

The bond0 interface is built of eth1 and eth2. Unfortunately, Linux does not deliver the correct information in such a case. The bond0 interface is actually up and using the IP address shown for eth1.

The name of the switch and the short notation of the port are shown if possible. The in-formation is derived from ARP caches which are very volatile. There maybe cases where !undef! is listed, but a port is associated. Furthermore, only the active interface of eth1 and eth2 is seen.

The following output is from a NetApp Filer:

FlexFrame Node information: Nodename OS OS Version Model filpool1-co ONTAP 7.0 FAS940 Device IP Addr MTU MAC Switch Port e0 192.168.20.203 1500 00a098011e92 Cisco1 Gi1/0/18 e9a not up 1500 0007e9391884 !undef! !undef! e9b not up 1500 0007e9391885 !undef! !undef! e10a not up 1500 02a098011e92 Cisco1 Po5 e10b not up 1500 0007e93ef757 !undef! !undef! e11a not up 1500 0007e93919d8 !undef! !undef! e11b not up 1500 0007e93919d9 !undef! !undef! e4a not up 1500 02a098011e92 Cisco1 Po5 e4b not up 1500 0007e93ef3e1 !undef! !undef! lo 127.0.0.1 9188 00000000 !undef! !undef! vh not up 9188 00000000 !undef! !undef! storage not up 1500 02a098011e92 Cisco1 Po5 storage-31 192.168.10.203 1500 02a098011e92 Cisco1 Po5 storage-41 192.168.11.206 1500 02a098011e92 Cisco1 Po5



Here a sample for a BX600 or BX300 Linux blade server:

FlexFrame Node information: Nodename OS OS Version Model an_0504 Linux SUSE SLES-9 FSC Device IP Addr MTU MAC Switch Port lo 127.0.0.1 16436 n.a. n.a. n.a. eth0 192.168.11.54 1500 00c09f3a874e Cisco1 Po4 eth1 no ip 1500 00c09f3a874e Cisco1 Po4 bond0 no ip 1500 00c09f3a874e Cisco1 Po4 vlan40 10.1.2.54 1500 00c09f3a874e Cisco1 Po4 vlan42 192.168.4.54 1500 00c09f3a874e Cisco1 Po4

The blade servers are connected to switch blades. The related port is shown as a PortChannel. This represents the “view” from the Cisco switch.

A sample switch blade configuration is shown below:

FlexFrame Node information: Nodename OS OS Version Model bx600-2-swb2 BX_SwitchBlade 1.0.0.2 Switch Blade Device IP Addr MTU MAC Switch Port port:1 192.168.20.52 1522 0030f1c42f81 !undef! !undef! port:2 no ip 1522 0030f1c42f82 !undef! !undef! port:3 no ip 1522 0030f1c42f83 !undef! !undef! port:4 not up 1522 0030f1c42f84 !undef! !undef! port:5 not up 1522 0030f1c42f85 !undef! !undef! port:6 not up 1522 0030f1c42f86 !undef! !undef! port:7 not up 1522 0030f1c42f87 !undef! !undef! port:8 no ip 1522 0030f1c42f88 !undef! !undef! port:9 no ip 1522 0030f1c42f89 !undef! !undef! port:10 no ip 1522 0030f1c42f8a !undef! !undef! port:11 no ip 1522 0030f1c42f8b Cisco1 Po4 port:12 no ip 1522 0030f1c42f8c !undef! !undef! port:13 not up 1522 0030f1c42f80 Cisco1 Po4 port:14 no ip 1522 0030f1c42f8b Cisco1 Po4 port:15 no ip 0 000000000000 !undef! !undef! port:1001 no ip 0 0030f1c42f80 Cisco1 Po4 port:1008 no ip 0 0030f1c42f80 Cisco1 Po4 port:1030 no ip 0 0030f1c42f80 Cisco1 Po4 port:1031 no ip 0 0030f1c42f80 Cisco1 Po4 port:1032 no ip 0 0030f1c42f80 Cisco1 Po4 port:1040 no ip 0 0030f1c42f80 Cisco1 Po4 port:1041 no ip 0 0030f1c42f80 Cisco1 Po4 port:1042 no ip 0 0030f1c42f80 Cisco1 Po4



The management blade of a blade server chassis is displayed as follows:

FlexFrame Node information: Nodename OS OS Version Model "BX600" frame BX 1.42 --- Slot # 1 serial#: XXXX000000 --- Slot # 2 serial#: XXXX000001 --- Slot # 3 serial#: XXXX000002 --- Slot # 4 serial#: XXXX000003 --- Slot # 5 serial#: XXXX000004 --- Slot # 6 serial#: XXXX000005 --- Slot # 7 serial#: XXXX000006 --- Slot # 8 serial#: XXXX000007 --- Slot # 9 serial#: XXXX000008 --- Slot #10 serial#: XXXX000009 Device IP Addr MTU MAC Switch Port eth0 192.168.20.50 1500 00c09f37e6a8 Cisco1 Gi2/0/22

You can see if there is a blade server in a given slot. If so, its serial number is listed.

Only one interface (eth0) of the two available management blades is active at a given time.

Sample output for a Solaris Application Node:

FlexFrame Node information: Nodename OS OS Version Model an_0800 Solaris 5.8. PW250 Device IP Addr MTU MAC Switch Port fjgi1 192.168.10.31 1500 00e000c51a0b Cisco1 Gi1/0/9 lo0 127.0.0.1 8232 n.a. n.a. n.a. fjgi0 192.168.10.32 1500 00e000a6d1d8 Cisco1 Gi2/0/9 fjgi0:1 192.168.10.30 1500 00e000a6d1d8 Cisco1 Gi2/0/9 fjgi30000 not up 1500 00e000a6d1d8 Cisco1 Gi2/0/9 fjgi30000:1 10.1.1.30 1500 00e000a6d1d8 Cisco1 Gi2/0/9 fjgi30001 not up 1500 00e000c51a0b Cisco1 Gi1/0/9 fjgi32000 not up 1500 00e000a6d1d8 Cisco1 Gi2/0/9 fjgi32000:1 192.168.3.30 1500 00e000a6d1d8 Cisco1 Gi2/0/9 fjgi32001 not up 1500 00e000c51a0b Cisco1 Gi1/0/9 XSCF board @ 192.168.20.30 FlexFrame Node information: Nodename OS OS Version Model an_0800-co xscf 04040001 unknown Device IP Addr MTU MAC Switch Port lo0 127.0.0.1 1536 000000000000 !undef! !undef! scflan0 192.168.20.30 1500 00e000c59a0b Cisco1 Gi1/0/20



PRIMEPOWER 250 and 450 models have a built-in XSCF which is listed after the Application Node itself. Here, the XSCF is connected to port Gi1/0/20.

The following example shows some error messages. This error messages indicates that one NIC of this Application Node is not connected to the port where it should be (accord-ing to the LDAP database). If, e.g. two ports of two Application Nodes of the same pool are mixed up, the functionality is just the same. However, if one Application Node gets moved into a different pool, the ports are changed. This would lead to a malfunction of both ports.

If this error message is displayed check the cabling and run ff_netscan.sh again.

FlexFrame Node information: Nodename OS OS Version Model an_0600 Solaris 5.8. PW250 Device IP Addr MTU MAC Switch Port Error: Node "an_0600" is connected to port=Gi1/0/6, but should be Gi1/0/5 or Gi2/0/5. fjgi1 192.168.10.11 1500 00e000c51941 Cis-Sum-1 Gi1/0/6 lo0 127.0.0.1 8232 n.a. n.a. n.a. fjgi0 192.168.10.12 1500 00e000a6d13a Cis-Sum-1 Gi2/0/5 fjgi0:1 192.168.10.10 1500 00e000a6d13a Cis-Sum-1 Gi2/0/5 fjgi30000 not up 1500 00e000a6d13a Cis-Sum-1 Gi2/0/5 fjgi30000:1 10.1.1.10 1500 00e000a6d13a Cis-Sum-1 Gi2/0/5 Error: Node "an_0600" is connected to port=Gi1/0/6, but should be Gi1/0/5 or Gi2/0/5. fjgi30001 not up 1500 00e000c51941 Cis-Sum-1 Gi1/0/6 fjgi32000 not up 1500 00e000a6d13a Cis-Sum-1 Gi2/0/5 fjgi32000:1 192.168.3.10 1500 00e000a6d13a Cis-Sum-1 Gi2/0/5 Error: Node "an_0600" is connected to port=Gi1/0/6, but should be Gi1/0/5 or Gi2/0/5. fjgi32001 not up 1500 00e000c51941 Cis-Sum-1 Gi1/0/6 XSCF board @ 192.168.20.10 FlexFrame Node information: Nodename OS OS Version Model an_0600-co xscf 04040001 unknown Device IP Addr MTU MAC Switch Port lo0 127.0.0.1 1536 000000000000 !undef! !undef! scflan0 192.168.20.10 1500 00e000c59941 Cis-Sum-1 Gi1/0/21

Similar error messages are:

Error: Node <node_name> NIC <nic_name>: IP address <ip_address> is in wrong VLAN (<number>)

This error message indicates that the IP address shown does not belong to the interface where it is configured on. It has to be in a different VLAN (network segment).



Not all combinations can be verified this way.

Error Node <node_name> unknown MAC (<mac_addr>) instead of (<MAC1> or <MAC2>)

This indicates that the two MAC addresses (MAC1 and MAC2) which are known to the LDAP database are not found on this Application Node (<node name>). One possible cause is that a NIC was replaced but the LDAP database was not updated using the appropriate tool.

If the known NIC fails and the Application Node tries to boot again it will not get an IP address and cannot boot.

Error: Node <node_name> does not have IP address <ldap_an_ip> configured! The IP address ldap_an_ip could not be found on this Application Node. Maybe there was a problem during configuration of the network interfaces.

There may be still wrong settings which cannot be detected by ff_netscan.sh.

At the end of the ff_netscan.sh output you should find the line:

Netscan finished.

For further details on options refer to the online manual page of ff_netscan.sh using the following command:

control1:~ # man ff_netscan.sh

4.1.2 Script: ff_pool_dnssrv.sh

Description

This command adds or removes a DNS server for a pool. LDAP entries for DHCP con-figuration and pool information are adjusted. Each Solaris Application Node will get an /etc/resolv.conf file with the DNS server and DNS domain name given in the com-mand line. Each Linux node will get an entry in /FlexFrame/vollFF/pool-<pool name>/pooldata/config/etc/resolv.conf, which is a symbolic link from the /etc/resolv.conf file of the Application Node.

Synopsis

ff_pool_dnssrv.sh [-d -f] -p <pool_name> -r <dns_server>

-u <action> -n <domain_name>

State of Pools Displaying the Current FlexFrame Configuration State


Options

-d Debugging information will be logged to /tmp/ff_pool_dnssrv.DEBUGLOG.

-p <pool_name> Name of the pool. Note: The pool name is case sensitive.

-r <dnssrvr> IP address for the DNS serverof the given pool. If the IP address do not match any of the local pool networks (Client VLAN, Storage VLAN or Server VLAN) and no default router is installed, processing is aborted.

-u <action> Possible actions are add or remove.

-n <domain_name> DNS domain name of the given pool. If the domain name do not match the one given in LDAP and the -f option is not used, processing is aborted.

-f Use force option to change the domain name when adding a DNS server.

Debugging

/tmp/ff_pool_dnssrv.DEBUGLOG will hold debugging information. In case of prob-lems please provide this file.

4.2 State of Pools The active configured pools are displayed on the FA WebGUI. The nodes or systems belonging to a pool are displayed in the FA WebGUI TreeView. Each node element in the tree which represents a pool is identified by the prefixed keyword Pool.

4.3 State of Application Nodes Each Application Node with a running FA Application Agent is shown in the FA WebGUI. It is shown in the Nodes TreeView with its host name (Linux) resp. node name (Solaris) and also in the Application Server Panel. In the Application Server Panel the data dis-played depend on the hierarchical level selected in the TreeView.

4.4 State of SAP Systems The active SAP system IDs can be displayed very easily in the Systems TreeView of the FA WebGUI and also in the SAP System Panel. The SAP System Panel is shown in parallel to the Application Server Panel. In the SAP System Panel, the data displayed depend on the hierarchical level selected in the TreeView.

Displaying the Current FlexFrame Configuration State State of SID Instances


4.5 State of SID Instances The active instances of a SID can be displayed very simply in the InstancesView of the FA WebGUI by clicking on a SID in the SAP System Panel.

For each view, information is provided in tabular form specifying the service’s current pool, group, node, priority and status.


5 Web Interfaces

5.1 FlexFrame Web Portal The FlexFrame Control Nodes provide a web portal with links to Web interfaces of sev-eral FlexFrame components, i.e;

● FA Autonomous Agents

● PRIMECLUSTER Administration

● ServerView S2

To access this portal, start Mozilla and enter the Control Node’s IP address in the location bar. If you are directly on the Control Node you want to configure, just call mozilla from the shell:

mozilla localhost

You will see an overview of all Web interfaces installed on the Control Nodes.

Web Interfaces FA Autonomous Agents


5.2 FA Autonomous Agents Use this tool to manage the virtualized services in your FlexFrame environment.

For information on usage, please refer to the FA Autonomous Agents manuals.

You can access the FlexFrame Autonomous Agents WebGUI directly from the active Control Node by entering the following URL:

http://localhost:8080/FAwebgui

This only works if the Jakarta Tomcat web server is running. If it is not running, check the PRIMECLUSTER configuration.

http://localhost:8080/FAwebgui

PRIMECLUSTER Administration Web Interfaces


5.3 PRIMECLUSTER Administration The PRIMECLUSTER administration interface allows to manage the PRIMECLUSTER components of FlexFrame.

To start, click on the “PRIMECLUSTER Administration” link on the FlexFrame web portal.

You can directly access the PRIMECLUSTER Administration interface from any Control Node by accessing the following URL:

http://localhost:8081/Plugin.cgi

This even works if the Apache web server is not running and therefore the Flex-Frame web portal is down.

You see the logon window.

Enter user name and password of the Linux root user.

http://localhost:8081/Plugin.cgi

Web Interfaces PRIMECLUSTER Administration


After successful login you will see the Web-Based Admin View tool:

The Web-Based Admin View is a framework for integration of different administration interfaces running on dedicated http daemons, independent of other installed web serv-ers.



Click on Global Cluster Services and then click on Cluster Admin.

This will start the PRIMECLUSTER administration interface JAVA applet.



To run the Cluster Admin JAVA applet, you have to accept the security certificate by Fujitsu Siemens Computers. The following warning is displayed:

Select Yes or Always. Selecting Always will automatically accept the certificate the next time you call this applet.

Next, you will be asked to which management server Cluster Admin should connect:

Select a server and click on Ok. For the PRIMECLUSTER management of FlexFrame it does not matter which one you select.



5.3.1 Cluster Foundation (CF) In the first window you will see the state of Cluster Foundation:

This part of PRIMECLUSTER is the base for Cluster interconnectivity and node monitor-ing. Both Control Nodes should be displayed in green colour.

In some cases, PRIMECLUSTER cannot detect the correct state of the partner node and will display LEFTCLUSTER and/or DOWN. For example this will happen if you remove all network cables to the remote node including the IPMI interface.

In case of a fault:

● Check the connectivity and restart CF by selecting Tools - Stop CF and Tools - Start CF. If this does not work, reboot at least one Control Node.

● If a node is really down and cannot be rebooted, select Tools - Mark Node Down from the pull down menu.

Do not try to change CF configuration and do not unconfigure CF unless in-structed to do so by Fujitsu Siemens Computers support!



5.3.2 Reliant Monitor Services (RMS) RMS is a part of PRIMECLUSTER which is used to control “userApplications”.

In the following window you can see a properly working FlexFrame environment (in this example cn1_pool1RMS is the first Control Node, cn2_pool1RMS the second one).

Do not change configuration parameters of RMS unless instructed by Fujitsu Siemens Computers support!

Select the tab labeled rms at the bottom of the left panel to proceed to PRIMECLUSTER RMS administration.

For a description of how PRIMECLUSTER is implemented in FlexFrame, refer to section PRIMECLUSTER on page 38.



5.3.3 Inconsistent and Faulted Applications Inconsistent applications (see example 1) may occur when a sub application is started or stopped manually and therefore the whole application including its children is neither completely online nor completely offline.

If ff_manage is online on the first Control Node and Apache is started manually on the second one, PRIMECLUSTER will mark ff_manage as “Inconsistent” on this node. To see and understand this inconsistency, expand the tree view of the application to view its children by clicking on the leftmost symbol.

Example 1: Inconsistent application

To clear this state, right click on ff_manage on the second Control Node and select Offline or Clear fault.

You could also select Online in this case; PRIMECLUSTER would then shutdown ff_manage on the first Control Node and start it on the second one (this may take some time). The application will be coloured yellow (Wait state) until the switch-over was fin-ished.

Faulted applications (see examples 2 and 3) may occur when a sub application crashes and cannot be restarted or when a sub application cannot be started during PRIME-CLUSTER startup.

Before clearing a fault, you should look into the logfile to find out the reason for the fault.

To view the logfile, right click on the faulted application and select View logfile.



Example 2: Faulted application

Example 3: Inconsistent faulted application

Common causes for faults are misconfigured services, a broken network connection to the NAS system or a damaged mySQL database which is used by myAMC Messenger, to name just a few examples.

To clear the fault state, right click on the application and select Clear fault to tell PRIMECLUSTER that it should do offline processing on the application.

You have to switch the application online on one node to make sure that Flex-Frame will work correctly. This can be established by using the Online (hvswitch) or Switch (hvswitch) entry of the application context menu.



5.3.4 Switching Applications This section describes how to shutdown, start and switch-over applications including their services.

Switch application offline Right click on the application and select Offline (hvutil -f) from the context menu.

Switch application online Right click on the application and select Offline (hvutil -f) from the context menu.

Switch over application to other Control Node Right click on the application and select Switch (hvswitch) from the context menu, then select the Node which the application should be switched to.

5.3.5 Application Maintenance Mode PRIMECLUSTER RMS provides a special maintenance mode. If activated, RMS will not take any corrective actions if the status of a service changes.

Whenever you need to manually stop a service which is controlled by PRIMECLUSTER (e.g. when installing updates and/or patches), you should switch the corresponding appli-cation into maintenance mode.

To switch an application into maintenance mode, right click on the application and select Enter Maintenance Mode.

To leave maintenance mode. Right click on the application and select Exit Mainte-nance Mode.

If you attempt to stop a service (for example LDAP by calling /etc/init.d/ldap stop) without entering maintenance mode or shutting down RMS, PRIMECLUSTER will try to recover the service after a few seconds by calling the start script.

You may damage your configuration if you ignore this!

It is possible to set all applications into maintenance mode at once by right-clicking on the cluster name (FLEXFRAME) and selecting Enter Maintenance Mode and respectively Exit Maintenance Mode in this popup menu.

For further information how to use PRIMECLUSTER refer to the PRIMECLUSTER manuals.

Web Interfaces ServerView S2


5.4 ServerView S2 ServerView S2 allows you to query information of PRIMERGY servers in the FlexFrame environment.

This tool is not needed for administration of FlexFrame but is required by Fujitsu Siemens Computers support.

If you would like to monitor the hardware status of PRIMERGY servers in your Flex-Frame™ environment, click on ServerView and on the next page, click on the button labeled start:

Please note that SSL connections are not supported.

Beginning with FlexFrame 4.0, the ServerView database for configuration, traps and serverlists are no longer shared across both Control Nodes. Each Control Node runs its own instance with its own configuration.

ServerView S2 Web Interfaces


An overview of all configured servers (initially only the local host) will be shown on the main screen:

To add more servers, run the Server Browser by either clicking on Administration / Server Browser on the top navigation bar or right clicking anywhere in the list and selecting New Server from the context menu.



The Server Browser window will open:



The easiest way to add servers is to scan a subnet for each pool, for example the “Server” subnet for Application Nodes and the “Control” subnet for management blades.

To scan a subnet, enter the first three bytes of the network address in the Subnet input field on the bottom left of the screen and click on Start Browsing:

The browsing process is finished when the button Stop Browsing changes its caption to Start Browsing.

Please do not interrupt the browsing process as it may hang the JAVA applet or the browsing processes. If this happens, the Server Browser or even ServerView S2 must be restarted.

To restart ServerView S2, run the following commands on the console:

cn1:~ # /etc/PRIMECLUSTER/bin/srvst.sh stop

PRIMECLUSTER will detect the “faulted” application and try to restart it after about 10 seconds. Verify this by calling:

cn1:~ # /etc/PRIMECLUSTER/bin/srvst.sh status



To add all manageable servers, right click anywhere on the list and click on Select Manageables.



After clicking on Select Manageables, all servers with known type will be selected:

To finally add those selected servers, click on Apply in the upper right corner of the Server Browser window.

Note that the Control Nodes are accessible from all subnets, so a message about already existing servers will be shown which should be confirmed by clicking on OK:



If ServerView S2 detects blade servers, it asks to add the whole Blade Center instead of each individual blade. This makes management easier and is highly recommended.

So in the following dialogue you should reply with Yes:

If ServerView can not detect the host name of the Management Blade, it will ask for it:

You may enter anything you want here.



After closing the Server Browser, the main window containing the server list will show the added servers. Further servers can be added by repeating the recent steps using a dif-ferent subnet address.

To view the state of a single server, click on the blue server name. To view the state of a Server Blade, click on the Management Blade / Blade Center, then select the right blade and click on ServerView on the bottom navigation bar.

Events such as SNMP Traps can be viewed by navigating to Event Management on the top navigation bar. This will open ServerView AlarmService, which will not be described in detail. To monitor SNMP Traps, we recommend using FA WebGUI / myAMC Messenger.


6 Hardware Changes

6.1 Changing BIOS Settings for Netboot For detailed information on setting BIOS parameters, please refer to the hardware docu-mentation. Look at some of the configuration examples below. The exact procedure may differ depending on the BIOS version of your hardware.

BX300 / 600

● While booting, press F2

● Menu: Advanced -> PCI Configuration -> OnBoard LAN1 + LAN2 [Enable]

● Reboot

● Menu: Boot 2 x MBA …. on top

● Menu: Exit -> Saving Changes

BX600 S2


● Menu: Advanced -> PCI Configuration -> OnBoard LAN1 + LAN2 [Enable]

● Reboot


● Menu: Boot -> Boot Device Priority -> 1st Boot Device

● Select IBA GE Slot 0420 …

● Menu: Boot -> Boot Device Priority -> 2nd Boot Device

● Select IBA GE Slot 0421 …


Hardware Changes Changing BIOS Settings for Netboot


RX300


● Menu: Advanced -> LAN Remote Boot A+B [PXE]

● Reboot


● Menu: Boot 2 x MBA … on top


RX300 S2

First, you have to make the PCI-X NICs net bootable.

Download proboot.exe from the INTEL website:

http://downloadfinder.intel.com/scripts-df-external/Product_Filter.aspx?ProductID=412

Extract proboot.exe and move it to a bootable CD-Rom.

Power on the RX300 S2.

After booting start IBAutil from the DOS Prompt:

<LW>:\ cd <IBAutil-DIR> <LW>:\ IBAutil.exe -all -flashenable

Reboot.

After booting start IBAutil from the DOS Prompt:

<LW>:\ cd <IBAutil-DIR> <LW>:\ IBAutil.exe -all -install pxe

Repeat the following steps for each NIC:

1. Answer the first question with “No”.

2. Answer the second question with “Yes”

3. At the end you will find a list of all NICs with the corresponding MAC addresses

4. Write down the MAC address of NIC1.

Remove the CD and reboot.


● Menu: Advanced -> Peripheral Configuration

● Select LAN Controller [Channel A&B]


Changing BIOS Settings for Netboot Hardware Changes


● Select LAN Remote Boot Ch. A: [Disabled]

● Select LAN Remote Boot Ch. B: [PXE]

● Menu: Exit, Save Changes + Exit

● Select Yes

● Reboot


● Menu: Main -> Boot Option -> Boot Sequence

● Select BootManage PXE, SLOT 0500 (Onboard LAN B)

● Move it behind CDROM, Diskett by pressing +

● Select IBA GE Slot 0320 v1228 extension LAN card NIC 1)

● Move it behind BootManage PXE, SLOT 0500 by pressing +

● Select IBA GE Slot 0321 v1228 (extension LAN card NIC 2)

● Press space to deactivate (!)

● Menu: Exit, Save Changes + Exit

● Select Yes

RX600


● Menu: Advanced -> Ethernet on Board -> Rom Scan [Enable]

Enable Master [Enable]

● Menu: Advanced -> Embedded SCSI Bios Scan Order [LAST]

● Menu: Main -> Boot Options 2 x MBA …. on top


RX600 S2

First, configure the PCI-E NICs as network-bootable.

Download proboot.exe from the INTEL website:

http://developer.intel.com/design/network/products/ethernet/linecard_ec.htm

Extract proboot.exe and move it to a bootable CD-Rom. Insert it into the drive to boot the node from CD.

http://developer.intel.com/design/network/products/ethernet/linecard_ec.htm



● Power on the RX600 S2

● While booting you will be requested to Press any key …

● Select Boot Manager

● Select Primary Master CD-ROM

After booting, start IBAutil from the DOS Prompt:

<LW>:\ cd <IBAutil-DIR> <LW>:\ IBAutil.exe -all -install pxe

Repeat the following steps for each NIC:

1. Answer the first question with “No”.

2. Answer the second question with “Yes”

3. At the end you will find a list of all NICs with the corresponding MAC addresses

4. Write down the MAC address of NIC1 and NIC3.

Remove the CD from the drive and reboot.

● While booting you will be requested to Press any key …

● Select Maintenance Manager

● Select Boot Options

● Select Legacy BEV Order

● Select BEV Drive #00

● Select NIC1 from card1 e.g IBA GE Slot 0220 v1220

● Select BEV Drive #01

● Select NIC3 from card2 e.g IBA GE Slot 0920 v1220

● Select Apply Changes

● Go Back To Main Page

● Select Boot Option

● Select Change Boot Order

● Check if NIC1 is in 2nd position

● Press ESC 3 x to go back to Main Page

● Select Continue Booting



RX800


● Menu: Configuration/Setup Utility

● Select Menu Devices and I/O Ports

● Set Planar Ethernet to [Enabled]

● Press ESC to go back

● Select Menu Start Options

● Select Menu Startup Sequence Options

● Set First Startup Device to [Network]


● Menu: Start Options

● Set Planar Ethernet PXE/DHCP to [Planar Ethernet 2]


● Select Menu Advanced Setup

● Select Menu CPU Options

● Set Hyper-Threading Technology to [Enabled]

● Press 2 x ESC to go back

● Select Save settings

● Press ENTER to save

● Select Exit Setup

● Select Yes, exit the Setup Utility



RX800 S2/S3


● Menu Configuration/Setup Utility

● Select Menu Start Options

● Select Menu Startup Sequence Options

● Set First Startup Device to [Network]

● Press ESC to go back to menu Start Options



● Set Planar Ethernet PXE/DHCP to [Planar Ethernet 2]


● Select Save settings

● Press ENTER to save

● Select Exit Setup

● Select Yes, exit the Setup Utility

Hardware Changes Replacing Network Cards


6.2 Replacing Network Cards

6.2.1 Replacing a Network Card – Control Node If the motherboard or the PCI network card in a Control Node is replaced, no further ac-tions are required since there is no reference to the replaced MAC addresses.

6.2.2 Replacing a Network Card – Application Node If a network card on an Application Node fails and is replaced, further actions are neces-sary to ensure proper operation.

During their boot processes, both PRIMEPOWER and PRIMERGY nodes need to re-treive some information through lower level protocols which are based on hardware ad-dresses (MAC addresses). The PRIMERGY's BIOS utilizes DHCP (Dynamic Host Con-figuration Protocol) for that purpose, the PRIMERPOWER's OBP (Open Boot Prom) uses RARP (Reverse Address Resolution Protocol).

The MAC address is the build-in, physical address which is unique to that network card. Since there are two network cards configured for booting you may not even notice the MAC address change of a single one, because the other card will be used for booting. Once the server is up, operation is normal. For full redundancy – even for booting – we recommend to adjust the configuration accordingly.

● On PRIMEPOWER systems, use OBP to determine all MAC addresses, including the new one.

● On PRIMERGY blade systems, use the management blade to determine the server blade MAC addresses.

● On PRIMERGY non-blade systems, use BIOS or network card labels to determine the MAC addresses.

To change the MAC addresses for booting, use the command below. Replace the node name and MAC addresses of the sample with your Application Node’s name and noted addresses:

control:~ # ff_an_adm.pl --op mac --name appnode5 --mac 00:e0:00:c5:19:41,00:e0:00:c5:19:41 MAC addresses changed.

The program changed the MAC addresses of the node within DHCP or RARP configura-tion and LDAP. No further intervention is necessary.

Some software licenses are derived from information based on server specific numbers, such as MAC addresses of network cards. On Linux, double-check your SAP licenses after you have replaced a network card.

Replacing Power Control Hardware Hardware Changes


6.3 Replacing Power Control Hardware This section describes what to do after the replacement of power control hardware, de-pending on your Application Nodes and Control Nodes hardware.

For instructions on replacing the power control hardware, please refer to the documenta-tion delivered with your hardware.

Description of the Power Shutdown Function

The power shutdown is necessary for FlexFrame Autonomous Agents to switch off Appli-cation Nodes in case of a switchover scenario to make sure that a system in an unde-fined state will be switched off. It is also necessary for the PRIMECLUSTER Shutdown Facility on the Control Nodes.

The power shutdown will be implemented by the Shutdown Agents (SA) of the PRIME-CLUSTER Shutdown Facility and the ff_xscf.sh script.

Each Shutdown Agent (SA_blade, SA_ipmi, SA_rsb, SA_rps and SA_scon) has its own config file which will be configured automatically by the FlexFrame Autonomous Agents and the PRIMECLUSTER Shutdown Facility on the Control Nodes.

The PRIMEPOWER XSCF (eXtended System Control Facility) is a built-in component for PRIMEPOWER 250 and 450 systems. It enables either serial or LAN access to the server’s console or power-on, power-off and reset commands.

Only the hardware and software which is used by the Shutdown Agents and XSCF must be prepared with IP address, user, password etc.

For details how to configure the power control hardware, please refer to chapter “Power-Shutdown Configuration” in the “Installation of a FlexFrame Environment” manual.

More information about the functionality of the SA tools can be found in section “Power Management (On/Off/Power-Cycle)” on page 289.

6.4 Exchanging a Control Node If a Control Node must be replaced for any reason, attention has to be paid to the fact that the replacing hardware has to be the same as the replaced component. Otherwise, problems with missing Linux drivers might occur. Problems due to hard disk failures should usually not occur, since the Control Node's hard disks are mirrored through hard-ware-based RAID-1.

Hardware Changes Replacing Switch Blades


6.4.1 Hardware Failed, Hard Disk and Installed OS are not Affected

In case that other hardware components than one of the HDs are broken (e.g. memory, CPU, etc.) and the installed OS image is still operational, these hardware components have to be replaced with equivalent ones. Even the entire hardware can be replaced - excluding the HDs. The HDs can be removed from the existing hardware, plugged into the new hardware and booted, after checking the RAID controller's BIOS for hardware RAID settings being enabled. Even if one of the hard disks is broken, it can be replaced with the same model and synchronized with the working hard disk through the RAID controller's BIOS.

In this case the Control Node hardware must be replaced by the same model with the same disk controller and the same network interface controllers. The approach is very simple, plug in your hard disks in the new Control Node in the same order as the old one, power on the Control Node, enter the RAID controller BIOS and check the parameters; hardware raid should be enabled as before.

See also the manual “Installation of a FlexFrame Environment“, chapter “Control Node Host RAID Configuration”.

6.4.2 One Hard Disk is Defect, the Other One is Undamaged The Control Nodes OS installation must be on a RAID-1 disk array.

Replace the defect hard disk with the same model and synchronize it with the undam-aged one via the RAID controller BIOS. This approach is dependent from the RAID con-troller model and its BIOS.

6.4.3 The Control Nodes OS is Damaged If, for any case, the operating system of a Control Node should be damaged, it has to be installed anew with the original Control Node installation DVD. The configuration has to be recovered from a previous backup. See section “Backup/Restore of FlexFrame Con-trol Nodes” on page 304.

6.5 Replacing Switch Blades To reconfigure a replaced switch blade, see chapter Restore of Switch Configuration on page 310.


7 Software Updates

7.1 Upgrading the Entire FlexFrame Landscape Updating the entire FlexFrame to the 4.0 release is a complex process. It depends on the release you are starting from. Therefore the following paragraphs refer to special docu-mentations.

7.1.1 Upgrading from FlexFrame 3.0 or Lower Version to FlexFrame 4.0

Please contact the Competence Center SAP (CCSAP). There exists a process, which has to be adapted on project specific needs.

7.1.2 Upgrading from FlexFrame 3.1 or 3.2 to FlexFrame 4.0 See the document “Upgrading FlexFrame 3.1 or 3.2 to FlexFrame 4.0”.

7.2 Software Upgrade on the Control Node This section describes FlexFrame software updates. Control Nodes are not designed to run any third party software (except saprouter).

Installing third party software on Control Nodes or Application Nodes may cause functional restrictions and other malfunctions of the FlexFrame system software. Fujitsu Siemens Computers shall have no liability whatsoever whether of a di-rect, indirect or consequential nature with regard to damages caused by the third party software or its erroneous installation.

For further information on third party software within FlexFrame, please refer to section “Third Party Software” on page 117

Nevertheless we will describe here the general approach to update/install software on the Control Nodes, on a few examples. In your special case this approach can differ to the ones described here.

New FlexFrame software should be updated/installed as RPMs. Only the RPMs will be documented in the RPM database. If there is no RPM package available (because the vendor does not deliver this piece of software not as an RPM and SUSE does not provide this version as RPM as well) you may also install this software from a tar archive. In this case you should document this in the directory /FlexFrame/volFF/FlexFrame/documentation/ as a plain text file.

Software Updates Software Upgrade on the Control Node


Normally, an already installed software package must be updated with the rpm command rpm -U <package_name>.rpm.

Don’t forget to install the same software package on the other Control Node, too.

7.2.1 ServerView Update via RPM Download the newest ServerView packages from Fujitsu Siemens Computers, e.g. here version 4.50-02 for SLES9. Copy the RPMs to your software stage directory and install the RPMs.

In this example, the software stage is the directory /FlexFrame/volFF/FlexFrame/stage/SLES9/SV.

Copy the RPMs to your software stage:

control1: # cp -p <somewhere>/srvmagt-mods_src-4.50-02.suse.rpm /FlexFrame/volFF/FlexFrame/stage/SLES9/SV control1:~ # cp <somewhere>/srvmagt-eecd-4.50-02.suse.rpm /FlexFrame/volFF/FlexFrame/stage/SLES9/SV control1:~ # cp <somewhere>/srvmagt-scs-4.50-02.suse.rpm /FlexFrame/volFF/FlexFrame/stage/SLES9/SV control1:~ # cp <somewhere>/srvmagt-agents-4.50-02.suse.rpm /FlexFrame/volFF/FlexFrame/stage/SLES9/SV

Check for all srvmagt- RPMs and remove them in the correct order:

1. scs and agents 2. eecd 3. mods_src

control1:~ # rpm -qa | grep srvmagt srvmagt-eecd-4.40-08 srvmagt-agents-4.40-08 srvmagt-scs-4.40-08 srvmagt-mods_src-4.40-08 control1:~ # rpm –e srvmagt-scs ... control1:~ # rpm –e srvmagt-agents ... control1:~ # rpm –e srvmagt-eecd ... control1:~ # rpm –e srvmagt-mods_src ...

Software Upgrade on the Control Node Software Updates


Verify that all packages have been removed successfully:

control1:~ # rpm -qa | grep srvmagt control1:~ #

Now the new packages can be installed, the correct order should be automatically de-tected by the rpm command:

control1:~ # cd /FlexFrame/volFF/FlexFrame/stage/SLES9/SV control1:/FlexFrame/volFF/FlexFrame/stage/SLES9/SV # ls srvmagt-*4.50-02* srvmagt-agents-4.50-02.suse.rpm srvmagt-mods_src-4.50-02.suse.rpm srvmagt-eecd-4.50-02.suse.rpm srvmagt-scs-4.50-02.suse.rpm control1:/FlexFrame/volFF/FlexFrame/stage/SLES9/SV # rpm -ivh srvmagt-*4.50-02* Preparing... ########################################### [100%] 1:srvmagt-mods_src ########################################### [ 25%] Compiling ServerView modules for kernel 2.6.5-7.282-bigsmp ... done Loading compiled ServerView modules ... copa unused smbus done ipmi done 1 ServerView module(s) unused. See /var/log/log.eecd_mods_src for details! 2:srvmagt-eecd ########################################### [ 50%] AgentX socket is currently /var/agentx/master Starting eecd done 3:srvmagt-agents ########################################### [ 75%] Shutting down snmpd: done Starting snmpd done AgentX socket is currently /var/agentx/master Waiting for eecd sockets ....................................... Waiting for eecd ready ......................... Starting: sc sc2 bus hd unix ether bios secur status inv thr vv done 4:srvmagt-scs ########################################### [100%] Starting SVRemoteConnector done

Repeat this procedure on the second Control Node.

Software Updates Software Upgrade on the Control Node


7.2.2 Updating/Installing a New Linux Kernel The following sections describe how to install a new Linux kernel for Control Nodes in FlexFrame. This only has to be done if instructed to do so by Fujitsu Siemens Computers support.

The process will be described on examples accomplished on a development system. The version numbers here in this document are not generally binding; they must be replaced with the correct information for your system.

This approach describes the kernel installation without the recommended server-start-CD.

7.2.2.1 Software Stage

To install and update any of the software packages in FlexFrame, it is useful to mount your NAS system or jukebox or any other software stage on a local mount point.

For example, your software stage is /FlexFrame/volFF/FlexFrame/stage/.

Copy the delivered software packages to the software stage:

control1: # cd /FlexFrame/volFF/FlexFrame/stage control1:/FlexFrame/volFF/FlexFrame/stage # mkdir -p CN/kernel

Kernel:

control1: # cp <somewhere>/kernel-bigsmp-2.6.5-7.283.i586.rpm /FlexFrame/volFF/FlexFrame/stage/CN/kernel

Kernel source:

control1: # cp <somewhere>/kernel-source-2.6.5-7.283.i586.rpm /FlexFrame/volFF/FlexFrame/stage/CN/kernel

Special Fujitsu Siemens Computers driver package:

control1: # cp <somewhere>/SUSE-SLES_9-drivers.zip /FlexFrame/volFF/FlexFrame/stage/CN/kernel

For official SLES Update kernels this driver package can be downloaded from the http://support.fujitsu-siemens.com website.

For special PTF Updates this package will be delivered by the Fujitsu Siemens Com-puters support.

http://support.fujitsu-siemens.com/

Software Upgrade on the Control Node Software Updates


7.2.2.2 Install the New Kernel

To install the new kernel, execute the following command:

control1: # cd /FlexFrame/volFF/FlexFrame/stage/CN/kernel control1:/FlexFrame/volFF/FlexFrame/stage/CN/kernel # rpm -ivh kernel-bigsmp-2.6.5-7.283.i586.rpm

Install the special Fujitsu Siemens Computers drivers:

control1:/ # cd /FlexFrame/volFF/FlexFrame/stage/CN/kernel control1:/FlexFrame/volFF/FlexFrame/stage/CN/kernel # unzip SUSE-SLES_9-drivers.zip

This will unzip all drivers to a “SUSE” subdirectory which must be installed using the primesetup utility:

control1:/FlexFrame/volFF/FlexFrame/stage/CN/kernel # primesetup –b –m

Finally, create new initial ramdisks:

control1:/FlexFrame/volFF/FlexFrame/stage/CN/kernel # mkinitrd

Optionally, configure the old kernel as a fall-back in the boot loader. Create a new entry for the previous kernel by copying the original “Linux” entry and appending “.previous” to the vmlinuz and initrd filenames:

control1: # cd /boot/grub control1:/boot/grub # cp -p menu.lst menu.lst.ORIG control1:/boot/grub # vi menu.lst color white/blue black/light-gray default 0 timeout 8 gfxmenu (hd0,0)/boot/message ###Don't change this comment - YaST2 identifier: Original name: linux### title Linux kernel (hd0,0)/boot/vmlinuz root=/dev/sda3 vga=0x317 selinux=0 splash=verbose resume=/dev/sda6 elevator=cfq showopts initrd (hd0,0)/boot/initrd [...] title Previous Kernel kernel (hd0,0)/boot/vmlinuz.previous root=/dev/sda3 vga=0x317 selinux=0 splash=verbose resume=/dev/sda6 elevator=cfq showopts initrd (hd0,0)/boot/initrd.previous

Software Updates Installation of a New FA Agent Version


Adjusting the menu.lst is only needed if you want to have a persistent option in the grub menu for booting the old kernel. You may also edit the grub configuration on-the-fly dur-ing the boot process by pressing Esc in the boot menu and entering the character based menu. Once there it is possible to edit the existing entries using the e key (once for enter-ing edit mode and again to edit each configuration line), for example append .previous to the /boot/vmlinuz and /boot/initrd filenames, then press b to boot.

Those changes will be temporary for one boot and allow you to boot the system using the previously installed kernel.

7.2.2.3 New Kernel Source for PRIMECLUSTER and ServerView

The kernel source is necessary to check and recompile the PCL cf.o module and ServerView modules while rebooting the Control Node.

control1: # cd /FlexFrame/volFF/FlexFrame/stage/CN/kernel control1:/FlexFrame/volFF/FlexFrame/stage/CN/kernel # rpm -ivh kernel-source-2.6.5-7.283.i586.rpm

When updating a new Kernel it is very important to install the corresponding kernel-source package, otherwise PRIMECLUSTER will not work anymore!

7.2.2.4 Reboot the Control Node

Reboot the Control Node and test the functionality of the updated Control Node, PCL, network connectivity, and so on.

The uname -r command should display the new kernel version:

control1:~ # uname -r 2.6.5-7.283-bigsmp control1:~ #

7.3 Installation of a New FA Agent Version The following sections describe how to update the FA Agents. Below, you find a brief description of what is to be done. For details on how it is to be done, please refer to the “FA Agents - Installation and Administration” documentation.

In accordance to the FlexFrame concept, migration of FlexFrame Autonomy is flexible and involves considerably less effort than for conventional systems:

● Installation of multiple agent versions or revision levels in parallel

● Separation of the installation process from the activation process

● Parameter data and agent versions are only loosely linked

Installation of a New FA Agent Version Software Updates


● Pool-by-pool activation of the new agent versions

● Conversion and saving of old parameter files with support of the CLI

● Option of reverting to the previous FA Agent version in the event of an unsatisfactory update

Versioning of the FlexFrame Autonomous Agents is linked to the FlexFrame versions via a matrix. In other words, it is possible that multiple agent versions can exist for operation with one FlexFrame version. What is decisive then is the scope of functions and rules that is required. In concrete terms this means that the FlexFrame Autonomous Agent Version 2.0 can, for example, also be used in compatibility mode for FlexFrame 3.0. In this way you can profit from enhanced detector and rule characteristics of a new FlexFrame Autonomous Agent without at the same time having to update the entire FlexFrame.

As soon as new functions of a release are used, functional compatibility only applies to a limited degree. A matrix will show you which functions are compatible with the release in the FlexFrame Agents or which are upward- or downward-compatible to a FlexFrame version.

7.3.1 Migration at Pool Level Version 2.0 of the FlexFrame Autonomous Agents enables multiple independent pools to be set up.

Installation and startup of the agents must be regarded as two separate processes. In principle, multiple versions of the agents can be installed. This permits, for example, a patch for the FA Agents to be tested in one pool while a previous version is used in an-other pool.

The version or revision level of the FA Agents must be entered or modified in the .info file with an editor (such as vi). The .info pool is contained in the pool directory /opt/myAMC/vFF/vFF_<pool_name>. The example below shows an .info file:

# Version V20K16 pool.release.current=V20K16 pool.release.base=V20K16

To switch to another version or revision level you have to adjust the value of the pool.release.current entry. The syntax is

V<version_number>K<revision_number>

This syntax is mandatory. Ensure that the installed version is entered.

For further information on migrating an FA Agent version please refer to section 7.3.3 ff.

Software Updates Installation of a New FA Agent Version


7.3.2 FlexFrame Autonomy Command Line Interface FlexFrame Autonomy contains a command line interface (PGTool.sh) to provide various information about FlexFrame.

The command line interface is an autonomous program which is a component of every FlexFrame Autonomy installation package.

The FlexFrame Autonomy command line interface supplies information about

● the pool

● the FA Agent version running

● the hardware of the node,

where the tool is executed.

7.3.3 Migration of FA Agent Versions on Pool Level The FlexFrame Autonomous Agents offer a whole raft of strategies and functionalities for installing and activating patches and new release versions for a wide range of security, test and release scenarios.

The administrator can use the update and activation functionality provided by the agents in line with the specific requirements. The following basic functions are available:

1. Reading and observing update, patch and release notes.

2. Installation of a new FA Agent version parallel to an operating FA Agent version. All data and configuration information for the operating FA Agent version are retained.

3. Taking over the configuration data for the new FA Agent version using the FA migra-tion tool.

4. Pool-by-pool configuration/parameterization and activation of the new FA Agent version.

5. Testing a new FA Agent version, e.g. in a separate test pool, if required by deactivat-ing the autonomous reactions for test operation.

The following activities are required to install or update a FlexFrame Autonomous Agent patch or a newer release version.

1. Read the update, patch and release notes and observe any required modifications and special features, particularly when updating FlexFrame and operating system versions and patches simultaneously.

2. Installation of a patch or the new release (FA Control Agent and FA Application Agent).

Installation of a New FA Agent Version Software Updates


3. Parameterization/configuration, possibly using the FA migration tool.

– Copying the parameters from the active agent version to the migration configu-ration directory using the FA migration tool.

– Testing, parameterizing and configuring the parameters specified in the confi-gration files.

– Testing any new parameters and, if necessary, modifying the default values.

– Parameterizing and configuring of FlexFrame/operating system version de-pendencies if the FlexFrame basis is updated at the same time.

– Check the modifications made by the migration tool, according to the file MI-GRATION-INSTRUCTIONS.txt in the target directory of the migration.

4. Pool-specific deactivation of the active FA Agent.

– Stopping the FA CtrlAgent for the pool whose agents are to be updated.

– Stopping the FA AppAgents on all nodes of the pool whose agents are to be updated.

5. Pool-specific activation of the new FA Agents.

– Modifying the active agent version in the .info file in the associated pool direc-tory.

– Transferring the migrated configuration to the configuration directory.

– Starting the FA AppAgents on all Application Nodes of the updated pool.

– Starting the FA CtrlAgent.

6. Checking the new active FA Agent version.

– Checking the agent processes.

– Checking agent messages at startup.

– Diagnosis and checking if the shown data are correct.

– Performing FlexFrame Autonomy tests (restart, reboot, etc.).

Steps 1 - 3 can take place while FlexFrame Autonomy is active. The FA Autonomy func-tions are not available only for a brief period between deactivation of the active agent version and activation of the new agent version.

Only version-compatible FA CtrlAgents, FA AppAgents and FlexFrame versions can be used. Compatibility of the agent versions with various FlexFrame ver-sions results in dependencies which must be taken into account.

For further information on the migration of FA Agent versions on pool level, please see documentation "FA Agents - Installation and Administration", section 4.8.

Software Updates FA Migration Tool


7.4 FA Migration Tool The FA migration tool is used to migrate configurations of a selected pool to and from a particular FA Agent version. The FA migration tool also enables you to merge configura-tion files.

7.4.1 Pool Mode Pool mode generates a migrated configuration in the subdirectory Migra-tion.<Version>_<timestamp>, including the backup of the current files. To enable the migrated configuration to be used it must be copied into the relevant con-figuration directory of the pool concerned.

Required/useful parameters:

-p/--migrate-pool=<pool> -r/--target-release=<release> -b/--backup [-V/--verbose] (optional) [-d/--pools-basedir=<dir>] (optional) [-c/--clean] (optional) [-s/--source-release=<release>] (optional)

See section “Parameters of the FA Migration Tool” on page 115 for a description of the parameters.

Example:

MGRTool.sh --migrate-pool=<pool> --target-release=<release> --backup

7.4.2 File Mode

File mode merges two files which are in myAMC configuration format. The two files are defined with the parameters merge-file and template.

File mode can only be used on files which are in myAMC configuration format (e.g. myAMC_FA.xml). The myAMC_Pools.xml and myAMC_FA_Groups.xml files are not in this format. These files can therefore not be migrated using file mode, but only in pool mode.

Required / useful parameters:

-m/--merge-file=<file> -t/--template=<template> -o/--out-file=<file> [-V/--verbose] (optional) [-c/--clean] (optional)

FA Migration Tool Software Updates


See section “Parameters of the FA Migration Tool” on page 115 for a description of the parameters.

Example:

MGRTool.sh --merge-file=File1.xml --template=File1-default.xml --out-file=File-out.xml

7.4.3 Usage of Help The usage of the FA migration tool can be displayed using the /opt/myAMC/FA_CtrlAgent/MGRTool.sh --help command:

7.4.4 Parameters of the FA Migration Tool

Parameter Description

-m/--merge-file=<file> Merges the <file> file with the template.

-t/--template=<template> Specifies the template to be used for the merge.

-o/--out-file=<file> Write merged results into the <file> file (use '-' for standard output).

-p/--migrate-pool=<pool> Specifies the <pool> pool to be migrated.

-s/--source-release=<release> Migrates from specified version (parameter is optional). If parameter is not specified, the version of the pool to be converted is used.

-r/--target-release=<release> Migration to the specified version.

-b/--backup Generates a backup of all files.

-d/--pools-basedir=<dir> Basic directory of the pools (default: /opt/myAMC/vFF).

-l/--list-releases Lists all available (installed) versions.

-c/--clean Removes unnecessary files and configuration settings.

-V/--verbose Detailed output during migration.

-lf/--logfile <file> Writes log messages to log file <file>.

-lp/--logpath <path> Generates the log file in the specified directory.

-h/--help Prints usage.

Software Updates Installing NAS Patches


7.5 Installing NAS Patches

7.5.1 Installing ONTAP Patches Before applying a new patch to the Filer’s ONTAP make sure that it is supported for the FlexFrame concept. If in doubt contact Fujitsu Siemens Computers support.

Download the tar file(s) for the new ONTAP onto one of your Control Nodes.

Create a temporary directory e.g. in /FlexFrame/volFF/FlexFrame/stage/ontap (create the directory if it does not exists) and extract the tar file there.

Proceed as described in the manual or README information of that patch.

Before you activate the new patch by rebooting the Filer, make sure that no SAP system or Database is busy. You do not need to reboot the Application Nodes, however we rec-ommend double-checking the mount points after the Filer is up again. If an Application Node should get stuck, reboot it.

7.5.2 Installing DART Patches for Celerra Data Movers Download the patch (default name nas.exe) into the directory /nas/dos/bin on the Celerra. If you choose the default name nas.exe for your patch, you have to do nothing. If you take a different name for it, you will have to modify the files /nas/dos/slot_<ID>/boot.bat with this name, where ID is the data mover ID, eg. 2 for server_2 or 3 for server_3.

Then you have to reboot all affected data movers by the reboot command on the Cel-erra control station:

server_cpu ALL –reboot now server_version ALL (to see, if the new patched OS version is active).

Before activating the new patch by rebooting the data movers, make sure that no SAP system or database is busy. You do not need to reboot the Application Nodes, however we recommend double-checking the mount points after the data movers are up again. If an Application Node should get stuck, reboot it.

Third Party Software Software Updates


7.6 Third Party Software If the customer needs to install third party software on the Application Nodes, the main-tenance procedures described in this manual must be followed. Installation of software or patches may have to be performed again after installing a maintenance image shipped by Fujitsu Siemens Computers.

If you have installed additional 3rd party software to your FlexFrame landscape, please make sure to back up this software components and their configuration data for later restore. Please consider: If you install a new FlexFrame image, the whole file system and directories are overwritten by a complete new root file system image. Afterwards, any 3rd party software and configurations have to be re-installed or restored.

Installing third party software on Control Nodes or Application Nodes may cause functional restrictions and other malfunctions of the FlexFrame system software. Fujitsu Siemens Computers shall have no liability whatsoever whether of a di-rect, indirect or consequential nature with regard to damages caused by the third party software or its erroneous installation.


8 Administrating Application Nodes Manual Application Node maintenance would be very complex and and error-prone. The script ff_an_adm.pl does the major changes and supports adding, changing, removing and listing of Application Nodes. Below, each action is described in detail.

In this document, you often will find console output, configuration data and installation examples which are based on earlier FlexFrame versions. Please keep in mind that these are examples and may look slightly different on the new operating systems introduced in FlexFrame 4.0.

8.1 Listing Application Nodes To list details of an Application Node use the maintenance script /opt/FlexFrame/bin/ff_an_adm.pl.

8.1.1 Displaying Information on a Specific Application Node This section describes how to list detailed information on a specific Application Node.

Synopsis

ff_an_adm.pl --op list --name <node_name>

Options

--op list Lists the configuration details of an Application Node.

--name <node_name> The name of the Application Node to be listed. Use operation mode list-all to get a list of all configured Application Nodes with their node names (see 8.1.2).

The output is structured in sections: hardware, software, network, assigned pool and group, switch ports.

● Hardware

This section contains information about system type, rack ID, device rack name, shutdown facility with IP address and host name, mac addresses and on blade serv-ers or partition servers the chassis and slot/partition ID.

● Software

This section contains information on OS type, vendor and version and the root im-age path.

Administrating Application Nodes Listing Application Nodes


● Network

This section lists the VLAN ID, IP address and host name of all configured networks, sorted by LAN segments.

● Pool and Group

This section lists the names of the assigned pool and group.

● Switch ports

In case link aggregates are configured, this section identifies the aggregate and its ports. Each used switch port is shown with the switch group ID, switch ID and port ID (cabinet ID, switch blade ID and port on blade servers) for the common LANs (Stor-age, Server and Client LAN) and the Control LAN (if used).

Definition of Switch Group:

A number of Cisco Catalyst 3750G switches within one system cabinet. The switches are connected as a loop with Cisco StackWise cables at the rear of each switch. With connected cables, the switches form a stack that behaves like a virtual switch including all ports of connected switches. To identify a switch port within the entire FlexFrame environment, three items are used:

• the number of the switch group (it is like a numbering of the virtual switches) starting with 1.

• the number of the switch within the switch group starting with 1.

• the number of the switch port starting with 1.

Definition of Switch Port:

A switch has a number of ports where other network devices and the host net-work interfaces are connected. The port is identified by a number starting at 1. Within a switch group, the port number is prefixed by the switch number (the identification number of the switch within the switch group).

Output examples

The command displays detailed information on the selected Application Node. The output differs between blade servers and all others.

Example of a PRIMERGY blade server output:

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op list --name bx2-6 Configuration details of node bx2-6 Hardware System: BX620S2 RackID: 2 AN Name: AN6 Shut.Facil.: Mgmt Blade bx2-co (195.40.224.78)

Listing Application Nodes Administrating Application Nodes


MAC Addr.: 00:C0:9F:99:E9:F4 00:C0:9F:99:E9:F5 IDs: 2 / 6 (System Cabinet / Slot|Partition) Software OS: SUSE / Linux / SLES-9.X86_64 (Vendor / Type / Version) OS Path: f1-pool2-st:/vol/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/root_img Network VlanID Host IP Hostname Storage LAN: 890 195.40.231.98 bx2-6-st Server LAN: 741 195.40.231.2 bx2-6-se Client LAN: 740 172.28.26.2 bx2-6 Pool and Group Pool: pool2 Group: bx600_o Switch ports Cabinet SwBlade Port Common LANs: 2 1 6 Common LANs: 2 2 6

Example of a PRIMEPOWER output:

cn1:~ # ff_an_adm.pl --op list --name pw651 Configuration details of node pw651 Hardware System: PW650 RackID: 1 AN Name: AN5 Shut.Facil.: RPS pw651-co (192.168.10.171) MAC Addr.: 00:e0:00:a6:0a:10 00:e0:00:a6:da:0c Software OS: Sun / SunOS / 5.8 (Vendor / Type / Version) OS Path: filer-p1-st:/vol/volFF/os/Solaris/FSC_5.8_202_20050530/bi_FJSV,GPUZC-M_PW-CMZ/root/pw651-st Network VlanID Host IP Hostname Storage LAN: 200 192.168.21.171 pw651-st 200 192.168.21.181 pw651-stt1 200 192.168.21.191 pw651-stt2

Administrating Application Nodes Listing Application Nodes


Server LAN: 300 192.168.31.171 pw651-se 300 192.168.31.181 pw651-set1 300 192.168.31.191 pw651-set2 Client LAN: 400 192.168.41.171 pw651 400 192.168.41.181 pw651-clt1 400 192.168.41.191 pw651-clt2 Pool and Group Pool: p1 Group: group15 Switch ports SW Grp Switch Port Common LANs: 1 1 9 Common LANs: 1 2 9 Control LAN: 1 1 21

8.1.2 Displaying Information on all Application Nodes This section describes how to list information on all configured Application Nodes.

To list all configured Application Nodes of entire FlexFrame or of a specific pool use the operation mode list-all.

Synopsis

ff_an_adm.pl --op list-all [--pool <pool_name>]

Options

--op list-all Lists all configured Application Nodes.

--pool <pool_name> The name of the pool of which the Application Nodes have to be listed.

Output example

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op list-all Nodes sorted by pool, group and name Pool pool1 Pool Group bx600_a bx1-1 Node Type: BX600 Rack/Cabinet/Slot|Partition ID: 1/1/1

Listing Application Nodes Administrating Application Nodes


OS: SUSE / Linux / SLES-9.X86_64 (Vendor / Type / Version) OS Path: f1-pool1-st:/vol/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/root_img Host IP Hostname Storage LAN: 195.40.224.222 bx1-1-st Server LAN: 195.40.224.30 bx1-1-se Client LAN: 143.161.72.30 bx1-1 MAC Addr.: 00:c0:9f:95:5f:ac 00:c0:9f:95:5f:ad bx1-2 Node Type: BX600 Rack/Cabinet/Slot|Partition ID: 1/1/2 OS: SUSE / Linux / SLES-9.X86_64 (Vendor / Type / Version) OS Path: f1-pool1-st:/vol/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/root_img Host IP Hostname Storage LAN: 195.40.224.223 bx1-2-st Server LAN: 195.40.224.31 bx1-2-se Client LAN: 143.161.72.31 bx1.2 MAC Addr.: 00:c0:9f:95:5f:8a 00:c0:9f:95:5f:8b bx2-1 Node Type: BX600 Rack/Cabinet/Slot|Partition ID: 2/2/1 OS: SUSE / Linux / SLES-9.X86_64 (Vendor / Type / Version) OS Path: f1-pool1-st:/vol/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/root_img Host IP Hostname Storage LAN: 195.40.224.227 bx2-1-st Server LAN: 195.40.224.35 bx2-1-se Client LAN: 143.161.72.35 vies1pyx MAC Addr.: 00:c0:9f:95:60:60 00:c0:9f:95:60:61 bx2-2 Node Type: BX600 Rack/Cabinet/Slot|Partition ID: 2/2/2 OS: SUSE / Linux / SLES-9.X86_64 (Vendor / Type / Version) OS Path: f1-pool1-st:/vol/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/root_img Host IP Hostname Storage LAN: 195.40.224.228 bx2-2-st Server LAN: 195.40.224.36 bx2-2-se Client LAN: 143.161.72.36 bx2-2 MAC Addr.: 00:c0:9f:93:7f:cc 00:c0:9f:93:7f:cd Pool pool2 Pool Group bx600_o bx1-6

Administrating Application Nodes Adding Application Nodes


Node Type: BX620S2 Rack/Cabinet/Slot|Partition ID: 1/1/6 OS: SUSE / Linux / SLES-9.X86_64 (Vendor / Type / Version) OS Path: f1-pool2-st:/vol/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/root_img Host IP Hostname Storage LAN: 195.40.231.97 bx1-6-st Server LAN: 195.40.231.1 bx1-6-se Client LAN: 172.28.26.1 bx1-6 MAC Addr.: 00:C0:9F:99:E6:CC 00:C0:9F:99:E6:CD bx2-6 Node Type: BX620S2 Rack/Cabinet/Slot|Partition ID: 2/2/6 OS: SUSE / Linux / SLES-9.X86_64 (Vendor / Type / Version) OS Path: f1-pool2-st:/vol/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/root_img Host IP Hostname Storage LAN: 195.40.231.98 bx2-6-st Server LAN: 195.40.231.2 bx2-6-se Client LAN: 172.28.26.2 bx2-6 MAC Addr.: 00:C0:9F:99:E9:F4 00:C0:9F:99:E9:F5

The output of list-all is less detailed than the list output. It is used to get an over-view. It shows the Application Nodes sorted by pool and group in alphabetical order. For each node the system type, the cabinet and slot/partition ID (if node is a blade or partition server), the OS type, vendor, version, the root image path, the main IP addresses, host names and the MAC addresses are listed.

8.2 Adding Application Nodes This section describes how to provide the required information for adding a new AN to an existing FlexFrame environment. See also sections “Changing BIOS Settings for Netboot” on page 95 and “Power Management (On/Off/Power-Cycle)” on page 289.

To add an Application Node, use the maintenance script /opt/FlexFrame/bin/ff_an_adm.pl. You have to define some parameters at com-mand line. They are used to configure switch ports, to create the boot information and the OS image.

Adding an Application Node changes the exports file on the common volFF NAS system. Temporary exports on this NAS system will be gone after running ff_new_an.sh. Be sure not to have temporary exports.

Adding Application Nodes Administrating Application Nodes


Synopsis

ff_an_adm.pl --op add --type <system_type> --name <node_name>

--pool <pool_name> --group <group_name> --swgroup <switch_group_id> --mac <mac_addresses>

--ospath <path_to_os_image>

[--host <ip_host_number>[,<test1_host_number>, <test2_host_number>]]

[--slot <BXxxx_cabinet/slot>]

[--hba <list_of_HBA_names] [--sw <multipath_software_name>]

Options

--op add Adds an Application Node and displays some information about processing steps.

--type <system_type> Specifies the product name and type like “PW250” or “RX300S2”. These are the common system type (family) terms. More detailed product identifiers are not neces-sary. See usage (call ff_an_adm.pl without any parameter) to get a list of sup-ported system types.

--name <node_name> The name of the Application Node. This name has to be unique for the entire Flex-Frame system. All interface names are based on this node name. We recommend using lower case names if possible.

--pool <pool_name> The name of the pool this node should belong to. See usage (call ff_an_adm.pl without any parameter) to get a list of currently configured pools.

--group <group_name> The name of the pool group this node is a member of. A group must consist of Appli-cation Nodes of the same OS image version and should be of the same capacity (CPU, Memory etc.). There should be at least one spare Node in a group. Otherwise, take-over of failing services will not be possible. Use command ff_pool_adm.pl with op mode list or list-all to get the pool groups.

--swgroup <switch_group_id> Defines the switch group the Application Node is connected to. This information is necessary to assign and configure switch ports. The switch group was numbered during installation with the FlexFrame PlanningTool. Use this number here too. See usage (call ff_an_adm.pl without any parameter) to get a list of currently config-ured switch group IDs.

Administrating Application Nodes Adding Application Nodes


--mac <mac_addresses> Add here the both MAC addresses of the data NICs used with booting. Use the dou-ble colon separated hex notation for each MAC address. Concatenate them with a comma. The MAC address syntax is six colon separated hex values, eg. 00:e0:00:c5:19:41.

--ospath <path_to_os_image> Defines the OS image to be used. Add the relative path to /FlexFrame/volFF/os/ as seen from the Control Node. See usage (call ff_an_adm.pl without any parameter) to get a list of available OS pathes.

--host <ip_host_number>[,<test1_host_number>,<test2_host_number>] Host part to be used to build IP addresses for the three networks. On Solaris systems (PRIMEPOWER) you have to give three different host numbers, separated with a comma. If this option is omitted, the script uses free host numbers to calculate the IP addresses.

--slot <BXxxx_cabinet/slot> With PRIMERGY server blades use this option to define the cabinet and slot ID of the server blade. New cabinets have to be defined with the /opt/FlexFrame/bin/ff_bx_cabinet_adm.pl command.

--hba <list_of_HBA_names> Specifies a list of symbolic SAN HBA names. The HBA name may consist of alpha-numerical characters, dashes and underscores. Separate HBA names with a comma.

--sw <multipath software name> Specifies the name of the SAN multipath software to be used. See usage for a list of known and acepted software names.

Output examples

Output for a blade server:

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op add --type BX620S2 --name bx1-6 --pool pool1 --group bx600_o --ospath Linux/FSC_4.0A00-000.SLES-9.X86_64 --host 1 --slot 1/6 --mac 00:C0:9F:99:E6:CC,00:C0:9F:99:E6:CD update swblade 1/1 configuration Notice: Update will take about 1 minute. update swblade 1/2 configuration Notice: Update will take about 1 minute. If not reported any error all precautions are done to create application nodes os image. To do this call: ff_new_an.sh -n bx1-6 Creating and customizing an image may take some minutes. Don't get anxious.

Removing Application Nodes Administrating Application Nodes


Output for a non-blade server:

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op add --name pw250-3 --type PW250 --pool p1 --group Solaris --swgroup 1 --mac 00:e0:00:c5:42:ab 00:e0:00:a6:eb:9f --ospath Solaris/FSC_5.8_202_20050530 update switch 1/1 configuration Notice: Update will take about 1 minute. Connect your systems LAN interfaces to named switch ports: SwitchGroup / Switch / Port LAN Interface 1 / 2 / 14 data NIC-1 1 / 1 / 14 data NIC-2 1 / 2 / 15 mgmt NIC-1 If not reported any error all precautions are done to create application nodes os image. To do this call: ff_new_an.sh -n pw250-3 Creating and customizing an image may take some minutes. Don't get anxious.

The script first checks all arguments and aborts with error messages in case of errors. Then it fetches free IP addresses and switch ports. The switch ports are reconfigured to match requirements, the LDAP data is created and a netboot file is written. The netboot file is used by ff_new_an.sh to create Application Node images and extend the NAS system’s exports list.

At the end you get a cabling advice and instructions how to call ff_new_an.sh script to finish the Application Node creation.

8.3 Removing Application Nodes To remove an Application Node use the maintenance script /opt/FlexFrame/bin/ff_an_adm.pl. You only have to give the node name to be removed at the command line. All switch ports will be unconfigured and the boot informa-tion and OS image are deleted.

Removing an Application Node results in direct deletion of its image, removal of its LDAP entries as well as disabling the respective switch ports.

Please make sure you really want to remove the Application Node (AN) when calling the script, the script does not ask for further confirmation.

Administrating Application Nodes Renaming Application Nodes


Synopsis

ff_an_adm.pl --op rem --name <node_name>

Options

--op rem Removes an Application and displays only errors and warnings.

--name <node_name> The name of the Application Node to be removed. Use operation mode list-all to get all configured Application Nodes and their names (see 8.1.2).

Output example

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op rem --name pw250-3

8.4 Renaming Application Nodes Changing of node names may be nescessary for various reasons, eg. Solaris Image Maintenance Cycle. Removing and adding the node may result in changes of network cabling while renaming does not.

With renaming a node an alternate os image path may be given.

Synopsis

ff_an_adm.pl --op rename --name <node_name>

--newname <new_node_name>

[--ospath <path_to_os_image>] [--remove_image]

Options

--op rename Changes the node name of given application node.

--name <node_name> Current name of node to be changed.

Removing an Application Node changes the exports file on the common volFF NAS system. Temporary exports on this NAS system will be gone after running ff_an_adm.pl.

Please make sure not to have temporary exports.

Application Nodes and SAN Administrating Application Nodes


--newname <new_node_name> New name of the node.

--ospath <path_ to_os_ image> Os image path the renamed node should use.

--remove_image Removes the old unused image of node.

Output example

cn1:~ # ff_an_adm.pl --op rename --name node4 --newname node5 update LDAP ................ If not reported any error all precautions are done to create a new application node os image. To do this call: ff_new_an.sh -n node5 Creating and customizing an image may take some minutes. Don't get anxious.

8.5 Application Nodes and SAN For adding or changing the entire list of SAN HBA (Host Bus Adapter) names use opera-tion mode hba-change. The names are symbolical names and may consist of alphanu-merical characters, dashes and underscores. The names have to be separated by comma. For redundancy at least two names for two HBAs are needed.

ff_an_adm.pl --op hba-change --name <node_name> --hba <hba-list>

To remove the entire list of HBA names use operation mode hba-rem. It removes the list from nodes LDAP data. To remove only a single HBA use operation mode hba-change.

ff_an_adm.pl --op hba-rem --name <node_name>

To define the name of the available SAN multipath software use operation mode sansw-change. The usage for a list of known software names.

ff_an_adm.pl --op sansw-change --name <node_name> --sw <multipath_software>

To remove the SAN multipath software name use operation mode sansw-rem. It re-moves the name from nodes LDAP data.

ff_an_adm.pl --op sansw-rem --name <node_name>

Administrating Application Nodes Administrating Blade Server Cabinets


8.6 Administrating Blade Server Cabinets Some network settings have to be changed to add or remove a blade server cabinet. The script ff_bx_cabinet_adm.pl will simplify the administration by doing LDAP changes automatically and preparing configurations to be done manually. The script supports adding, removing and listing of blade server cabinets. Each action is described in detail below.

8.6.1 Listing Blade Server Cabinets Lists known cabinets from LDAP database with maintenance script /opt/FlexFrame/bin/ff_bx_cabinet_adm.pl.

8.6.1.1 Displaying Information on a Specific Blade Server Cabinet

This section describes how to list detailed information on a specific blade server cabinet.

Synopsis

ff_bx_cabinet_adm.pl --op list --name <cabinet_name>

Options

--op list Lists the configuration details of a blade server cabinet.

--name <cabinet_name> The name of the blade server cabinet to be listed.

The output is structured in sections: hardware, software, network, assigned pool and group, switch ports.

Output example

Primergy Cabinet 1 (cab1) System Type: BX600 Management Blade Hostname / IP Address: cab1-co 195.40.224.75 Integrated LAN Switch Ports: SwitchGroup SwitchID PortID 1 1 8 1 2 8 Switch Blade SwitchID Type Switch name Hostname IP Address 1 Quanta bx600-2-swb1 bx600-2-swb1 195.40.224.76

Administrating Blade Server Cabinets Administrating Application Nodes


2 Quanta bx600-2-swb2 bx600-2-swb2 195.40.224.77 Switch Blade Port Integrated LAN Switch Port Switch Blade ID PortID <--> SwitchGroup SwitchID PortID 1 11 1 2 11 1 12 1 1 11 2 11 1 2 12 2 12 1 1 12

As seen from the sample above, the cabinet ID and name, the cabinet system type, the management blade and the switch blades are listed.

For the management blade the host name, the IP address and both FlexFrame integrated LAN switch ports are displayed.

The switch blade information shows the switch and host name, the IP address and the switch blade port to FlexFrame integrated LAN switch port connections, structured by switch blade ID.

8.6.1.2 Displaying Information on all Configured Blade Server Cabinets

This section describes how to list information on all configured blade server cabinets.

Synopsis

ff_bx_cabinet_adm.pl --op list-all

Option

--op list-all Lists all configured blade server cabinets.

Output example

Primergy Cabinets 1 (cab1) BX600 Management Blade: cab1-co / 195.40.224.75 Switch Group ID: 1 Server Blades (by slot id) 1 (blade1) BX600 Pool / Group: pool1 / PROD 2 (blade2) BX600 Pool / Group: pool1 / PROD 3 (blade3) BX600 Pool / Group: pool1 / PROD



4 (blade4) BX600S2D Pool / Group: pool2 / DEV 5 (blade5) BX600 Pool / Group: pool2 / DEV

For each cabinet the ID, the cabinet name, the management host name and IP address and the server blades are displayed.

Each server blade is shown with its slot ID and name, the system type and the pool and group it belongs to.

8.6.2 Adding Blade Server Cabinets This section describes how to provide the required information for adding a new blade server cabinet to an existing FlexFrame environment.

To add a blade server cabinet, use the maintenance script

/opt/FlexFrame/bin/ff_bx_cabinet_adm.pl. You have to define some parame-ters at the command line. They are used to configure switch ports and to create the switch blade configurations.

Synopsis

ff_bx_cabinet_adm.pl --op add --type <system_type>

--name <cabinet_name>

--swgroup <switch_group_id> [--swblades <type_of_switch_blades>]

Options

--op add Adds a blade server cabinet.

--type <system_type> PRIMERGY blade system type e.g. BX300 or BX600. Call ff_bx_cabinet_adm.pl without any parameter to get a list of supported sys-tem types.

--name <cabinet_name> Name of the subsystem (cabinet). It is used to generate a new name for the man-agement blade (has to be unique within entire FlexFrame).

--swgroup <switch_group_id> Switch group number (starts with 1) the cabinet has to be connected to (physically). See usage (call ff_bx_cabinet_adm.pl without any parameter) to get a list of currently configured switch group IDs.



--swblades <type_of_switch_blades> The type of switch blades. Defaults to switch blade type Quanta for cabinet type BX600 and Accton for cabinet type BX300. Valid types for BX600 are Cisco, Quanta, Accton and Pass-Through. Valid type for BX300 is Accton. For the default switch blades this option may be omitted. Else give the type for the switch blade like "Accton,Accton", or any combination of valid types for all used switch blades.

Output example

At the end of the output, the command displays further instructions.

A new Primergy BX600 cabinet numbered 1 connected at SwitchGroup 1 was added to LDAP. You have to configure your ManagementBlade with the control lan settings: control lan IP address: 195.40.224.75 control lan name: cab1-co to interoperate correctly with the FA Agents. Interconnect the ManagementBlades and SwitchBlades with the switches of SwitchGroup 1 as noted below: SwitchID/Port Mgmt/SwitchBlade 1 / 8 slave ManagementBlade 1 / 11 SwitchBlade 1 Port 12 1 / 12 SwitchBlade 2 Port 12 2 / 8 master ManagementBlade 2 / 11 SwitchBlade 1 Port 11 2 / 12 SwitchBlade 2 Port 11 Upload of initial SwitchBlade configurations have to be done manualy. See installation guide for details. For a quick instruction see below. The files to be uploaded are named: SwitchBlade Blade Type File Path 1 Quanta /tftpboot/swblade-2-1.config 2 Accton /tftpboot/swblade-2-2.config Quick instruction for switch blade type "Accton" Plug only port 11 to integrated LAN switches to prevent disabling of portchannel at integrated LAN switches. Use the management blade console redirection (telnet cab1-co 3172) to get the console of the switch blade.



See below a session snippet as sample how to upload the configuration. Console> enable Console # configure Console(config)# interface ethernet 1/11 Console(config-if)# switchport allowed vlan add 200 Console(config-if)# end Console(config)# interface vlan 200 Console(config-if)# ip address 195.40.224.77 255.255.255.0 Console(config-if)# end Console # copy tftp startup-config TFTP server ip address: 195.40.224.3 Source configuration file name: swblade-2-2.config Startup configuration file name [startup-config]: \Write to FLASH Programming. -Write to FLASH finish. Success. Console # reload System will be restarted, continue <y/n>? y Plug port 12 to integrated LAN switches. Now the switch blade should be fully operational. Quick instruction for switch blade type "Quanta" Plug only port 11 to integrated LAN switches to prevent disabling of portchannel at integrated LAN switches. Use the management blade console redirection (telnet cab1-co 3172) to get the console of the switch blade. See below a session snippet as sample how to upload the configuration. (Vty-0) #configure (Vty-0) (Config)#vlan database (Vty-0) (Vlan)#vlan 200 (Vty-0) (Vlan)#exit (Vty-0) (Config)#interface 0/11 (Vty-0) (Interface 0/11)#switchport allowed vlan add 200 (Vty-0) (Interface 0/11)#switchport tagging 200 (Vty-0) (Interface 0/11)#exit (Vty-0) (Config)#interface vlan 200 (Vty-0) (if-vlan 200)#ip address 195.40.224.76 255.255.255.0 (Vty-0) (if-vlan 200)#exit



(Vty-0) (Config)#end (Vty-0) #copy tftp://195.40.224.3/swblade-2-1.config script config.scr Mode....................................... TFTP Set TFTP Server IP......................... 195.40.224.3 TFTP Path.................................. TFTP Filename.............................. swblade-2-1.config Data Type.................................. Config Script Destination Filename....................... config.scr Management access will be blocked for the duration of the transfer Are you sure you want to start? (y/n) y Validating configuration script... File transfer operation completed successfully. (Vty-0) #script apply config.scr Are you sure you want to apply the configuration script? (y/n)y The system has unsaved changes. Would you like to save them now? (y/n) y a lot of lines ... Configuration script 'config.scr' applied. (bx600-klaus-swb1) #copy running-config startup-config Configuration Saved! (swblade-2-1) # Plug all other uplink ports to integrated LAN switches. Now the switch blade should be fully operational. Unless any errors are reported follow instructions above to solve all precautions needed to create new application nodes. Look at "/opt/FlexFrame/network/wiring-BX600-cab1.txt" to get a copy of this message.



Set up the management blade initially with name and IP address listed by the output as seen above. Use console redirection of management blade to connect to the console of the switch blades, and upload configuration as described by FlexFrame Installation Guide.

Finally, plug in the network cables according to the wiring plan given by the command output.

8.6.3 Removing Blade Server Cabinets To remove a blade server cabinet, use the maintenance script /opt/FlexFrame/bin/ff_bx_cabinet_adm.pl. You only have to give the ID of the cabinet that is to be removed, at the command line. All FlexFrame integrated LAN switch ports will be unconfigured.

Removing a blade server cabinet requires removing of all of its server blades.

Synopsis

ff_bx_cabinet_adm.pl --op rem --id <cabinet_id>

Options

--op rem Removes a blade server cabinet.

--id <cabinet_id> Specifies the subsystem (cabinet) ID of the cabinet to be removed. Use the list-all option to get the ID (see section 8.6.1.2).

Output examples

If there are any server blades configured for this cabinet, an error message is displayed.

ERROR: there are server blades configured for this cabinet. To remove the cabinet, remove application nodes (server blades) first. Use command ff_an_adm.pl to do this.

Use the list operation mode to list the configured server blades. You have to remove them before you can remove the cabinet they are in.

If no server blades are configured for this cabinet, the command displays a summary at the end.

If not reported any warnings or errors the cabinet was removed from LDAP and integrated LAN switches.



The cabinet has been removed successfully from LDAP and the FlexFrame integrated LAN switch ports used by the cabinet have been reconfigured to default.

8.6.4 Changing Switch Blade Type Within service cases it may be necessary to change the type of a switch blade due to a defective part replacement. Only switching blades can be used for a type change.

The proper program for this action is /opt/FlexFrame/bin/ff_bx_cabinet_adm.pl. To change the switch blade type the cabinet, the switch blade id and the switch blade type have to be specified.

Synopsis

ff_bx_cabinet_adm.pl --op swb-change --id <cabinet_id> --swbid <switch_blade_id>

--swbtype <switch_blade_type>

Options

--op swb-change Selects the operation mode. Change the type of a switch blade.

--id <cabinet_id> Specifiesd the subsystem (cabinet) ID of the cabinet. Use the list-all option to get the ID.

--swbid <switch_blade_id> Specifies the switch blade ID. The ID is the slot number of the selected switch blade.

--swbtype <switch_blade_type> Defines the new type of the switch blade. See usage for the currently supported types.

Output example

Switch type of switch blade 2 was successfully changed from "Accton" to "Quanta" at LDAP database.

The switch blade type was changed at LDAP database. To get the initial configuration use operation mode swb-config of this program. It will display instructions how to up-load the configurations too.



8.6.5 Changing Switch Blade Name Adding a new cabinet names the switch blades like the cabinet with a slot extension. In some cases the name of the switch blades have to be changed to match naming conven-tions. To do this, use /opt/FlexFrame/bin/ff_bx_cabinet_adm.pl.

Synopsis

ff_bx_cabinet_adm.pl --op swb-name --id <cabinet_id> --swbid <switch_blade_id>

--swbname <switch_blade_name>

Options

--op swb-name Selects the operation mode. Change the name of a switch blade.

--id <cabinet_id> Specifies the subsystem (cabinet) ID of the cabinet. Use the list-all option to get the ID.

--swbid <switch_blade_id> Specifies the ID of the switch blade. The ID is the slot number of the selected switch blade.

--swbname <switch_blade_name> Defines the new name of the switch blade.

Output example

If not reported any warnings or errors the hostname was successfully changed at switch blade, hosts files and LDAP.

As noted by the program the name of the switch will be changed at /etc/hosts of both control nodes, the LDAP database and at least the hostname and, if possible, the SNMP sysname at the selected switch blade itself.



8.6.6 Changing Switch Blade Password The default password used with adding a new cabinet is password. This is not secure. To change the password to a secure one use /opt/FlexFrame/bin/ff_bx_cabinet_adm.pl.

Synopsis

ff_bx_cabinet_adm.pl --op swb-passwd --id <cabinet_id>

--swbid <switch_blade_id> --swbpwd <password>

Options

--op swb-passwd Selects the operation mode. Change the login password of a switch blade.


--swbid <switch_blade_id> Specifies the switch blade ID. The ID is the slot number of the selected switch blade.

--swbpwd <password> Defines the new login and enable password of the switch blade.

Output example

If not reported any warnings or errors the password was successfully changed at switch blade and LDAP.

As noted by the program the password will be changed at the LDAP database and the selected switch blade. At the switch blade the login password and theenable password are changed and have to be the same.



8.6.7 Getting Switch Blade Initial Configuration In case of a service issue it may be nescessary to get an initial switch blade configura-tion, with have to be uploaded manually. To create such a configuration use /opt/FlexFrame/bin/ff_bx_cabinet_adm.pl.

Synopsis

ff_bx_cabinet_adm.pl --op swb-config --id <cabinet_id> --swbid <switch_blade_id>

Options

--op swb-config Selects the operation mode. Create the initial switch blade configuration.


--swbid <switch_blade_id> Specifies the ID of the switch blade. The ID is the slot number of the selected switch blade.

Output example

If not reported any warnings or errors the switch configuration was suc-cessfully created and stored into /tftpboot/swblade2.config. To upload the initial switch blade configuration see detailed description at "FlexFrame(TM) Installation Guide" chapter "SwitchBlade Configuration". For a quick guide see below. Depending on switch blade type there are different upload procedures. The sample below shows the procedure according to this switch bladetype (Accton). Plug only port 11 to integrated LAN switches to prevent disabling of portchannel at integrated LAN switches. Use the management blade console redirection (telnet cab1-co 3172) to get the console of the switch blade. See below a session snippet as sample how to upload the configuration.

Maintenance of Linux Application Node Images Administrating Application Nodes


Console> enable Console # configure Console(config)# interface vlan 200 Console(config-if)# ip address 195.40.224.77 255.255.255.0 Console(config-if)# end Console # copy tftp startup-config TFTP server ip address: 195.40.224.3 Source configuration file name: swblade2-2.config Startup configuration file name [startup-config]: \Write to FLASH Programming. -Write to FLASH finish. Success. Console # reload System will be restarted, continue <y/n>? y Plug port 12 to integrated LAN switches. Now the switch blade should be fully operational. This hint is additionaly stored at: /tmp/swb-config_bx_cabinet/todo.txt

The configuration file is stored directly to /tftpboot. Upload of configuration is de-scribed using TFTP which uses /tftpboot as top level directory. The program displays a switch blade type depending short description how to upload the configuration to the switch blade. This short description uses actual parameters like file names and ip ad-dresses to be able to copy&paste if possible. A more detailed instruction can be found in the „Installation Guide“.

8.7 Maintenance of Linux Application Node Images This section describes how to …

● install an Application Node image from the installation media.

● create a new customer specific Linux Application Nodes image.

● install Linux service packs.

● update/install a new Linux kernel.

● update ServerView.

● upgrade RPM packages on an Application Node.

● upgrade the application software.

● migrate remaining Application Nodes to the new Application Node image.

Administrating Application Nodes Maintenance of Linux Application Node Images


Choose an Application Node for the maintenance and isolate it. This means all running Applications like SAP services must be moved to another Application Node i.e. a Spare Node. In this document, the Application Node “an_0504” will be used for the mainte-nance.

8.7.1 Installing an Application Node Image from Installation Media

This section describes how to install an Application Node image delivered by Fujitsu Siemens Computers. This Application Node image contains two components, one root image which is shared for all Application Nodes and one var image as a template for the individual Application Node var images.

All images delivered by FSC are named FSC_<FF version>.<OS ver-sion>.<OS architecture>. For SLES8 images, OS version and architec-ture may be missing due to backward compatibility. The prefix FSC_ is reserved for originals delivered from FSC and must not be used for customer specific modified images.

The following approach is similar to the Installation Guide. These newly delivered Images must be installed in a new separate image directory tree besides the old one.

For example, the old image directory tree is FSC_4.0A00-000.SLES-9.X86_64, contain-ing an SLES9 (64bit) Image. The new one is FSC_3.2.x.

control1:/FlexFrame/volFF/os/Linux # ls -l total 20 drwxr-xr-x 5 root root 4096 Apr 29 16:33 . drwxr-xr-x 4 root root 4096 Mar 15 15:10 .. drwxr-xr-x 4 root root 4096 Apr 22 13:41 FSC_4.0A00-000.SLES-9.X86_64 drwxr-xr-x 4 root root 4096 Apr 22 14:15 pool_img

8.7.1.1 Installing the Application Node Image

Install the Application Node image from installation media with the installation tool ff_install_an_linux_images.sh on the Control Node.

Synopsis

ff_install_an_linux_images.sh [-p <path_to_images>]



Option

-p <path_to_images> Each image version has its own path to images i.e. FSC_3.2.x. If this path already exist you have to specify a new image path with the -p option.

Example for destination path FSC_3.2.x:

control1:/ # mount /media/dvd control1:/ # /media/dvd/ff_install_an_linux_images.sh -p /FlexFrame/volFF/os/Linux/FSC_3.2.x .. .. control1:/FlexFrame/volFF/os/Linux # ls -l total 20 drwxr-xr-x 5 root root 4096 Apr 29 16:33 . drwxr-xr-x 4 root root 4096 Mar 15 15:10 .. drwxr-xr-x 4 root root 4096 Apr 22 13:41 FSC_4.0A00-000.SLES-9.X86_64 drwxr-xr-x 4 root root 4096 Apr 29 17:10 FSC_3.2.x drwxr-xr-x 4 root root 4096 Apr 22 14:15 pool_img

The location for the Linux Application Node images always has to be /FlexFrame/volFF/os/Linux/.

8.7.1.2 Creating the New Netboot Configuration

Create the new netboot configuration for one Application Node with ff_create_an_cfg.pl.

Synopsis

ff_create_an_cfg.pl --name <app node name>

--lin_os_path <directory>

Example:

control1:/ # ff_create_an_cfg.pl --name an_0504 --lin_os_path /FlexFrame/volFF/os/Linux/FSC_3.2.x

8.7.1.3 Creating the New var Image

Create the new var image for one Application Node. The following steps will be done automatically:

● create the new var images for the Application Nodes



● create the new entries for the NAS system exports

● create the new pxelinux configuration

● update the LDAP database

Synopsis

ff_new_an.sh -n <app_node_name>

Example:

control1:/ # ff_new_an.sh -n an_0504

You now have a new Application Node Image tree with one newly configured Application Node. The newly configured Application Node will use this image tree upon the next re-boot and can be used for maintenance purposes.

8.7.2 Creating a New Linux OS Image for Application Nodes This section describes how to build a custom Application Node image for test purposes or small software updates. Usually, new Application Node images will be delivered from Fujitsu Siemens Computers, see section “Installing an Application Node Image from In-stallation Media” on page 142.

Software maintenance for Application Nodes must be done in the right order:

● Choose an appropriate Application Node for the maintenance.

● Copy the productive image FSC_<fsc_version> to a maintenance image <Cus-tom_Image>, this includes the root-image and the var_template.

● Isolate the maintenance Application Node. FA Agents have to be disabled.

● Create the new netboot configuration for one Application Node with ff_create_an_cfg.pl.

● Create the new var image for one Application Node.

● Install or update the Software on the maintenance Application Node. New software parts in the var image must be back-ported to the new template var image var_template, e.g. copy the maintenance var image to the var_template and delete all files in /FlexFrame/volFF/os/Linux/<Custom_Image>/var_img /var_template/tmp/.

● Test all applications on the maintenance Application Node

● Migrate the remaining Application Nodes to the new image tree.



8.7.2.1 Schematic Overview of the Maintenance Cycle

CreateCustom Image Tree

CreateCustom Image Tree

Change exportson Filer to

Read-Writefor root Image

Change exportson Filer to

Read-Writefor root Image

Disable FA Agentsin Custom root ImageDisable FA Agents

in Custom root Image

StartMaintenance Actions

StartMaintenance Actions

CreateNetboot Configuration

for MaintenanceApplication Node


for MaintenanceApplication Node

Remountroot Filesystemas Read-Write

Remountroot Filesystemas Read-Write

Isolate oneApplication Nodefor Maintenance

Isolate oneApplication Nodefor Maintenance

Boot or RebootMaintenance

Application Node

Boot or RebootMaintenance

Application Node

Remountroot Filesystemas Read-Only

Remountroot Filesystemas Read-Only

Create and ConfigureMaintenance

Application NodeImage

Create and ConfigureMaintenance

Application NodeImage

Create var_templatefrom Customized

var Image

Create var_templatefrom Customized

var Image

Change exportson Filer toRead-Only

for root Image

Change exportson Filer toRead-Only

for root Image


for RemainingApplication Nodes


for RemainingApplication Nodes

Create and ConfigureRemaining

Application NodeImages

Create and ConfigureRemaining

Application NodeImages

Boot or RebootRemaining

Application Nodes

Boot or RebootRemaining

Application Nodes

Reboot and TestMaintenance

Application Node

Reboot and TestMaintenance

Application Node

Enable FA Agentsin Custom root Image

Enable FA Agentsin Custom root Image

Software Update,Kernel Update, ...Software Update,Kernel Update, ...

FinalizeMaintenance Actions

FinalizeMaintenance Actions

test failed?

next maintenance cycle



8.7.2.2 Creating a Custom Image Tree for Linux Application Nodes

In this example, the customer specific image tree is CustomImage_4.0.

Copy the original image directory tree FSC_4.0A00-000.SLES-9.i686 to CustomImage_4.0:

control1:/FlexFrame/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64 # find root_img var_img/var_template | cpio -pdum ../CustomImage_4.0 control1:/FlexFrame/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64 # cd .. control1:/FlexFrame/volFF/os/Linux # ls -l total 20 drwxr-xr-x 5 root root 4096 Apr 29 16:33 . drwxr-xr-x 4 root root 4096 Mar 15 15:10 .. drwxr-xr-x 4 root root 4096 Apr 22 13:41 FSC_4.0A00-000.SLES-9.X86_64 drwxr-xr-x 4 root root 4096 Apr 29 17:10 CustomImage_4.0 drwxr-xr-x 4 root root 4096 Apr 22 14:15 pool_img

This image directory tree contains all necessary images, the root image and all var im-ages:

control1:/FlexFrame/volFF/os/Linux # ls -l CustomImage_4.0 total 16 drwxr-xr-x 4 root root 4096 Apr 29 17:10 . drwxr-xr-x 5 root root 4096 Apr 29 16:33 .. drwxr-xr-x 37 root root 4096 May 3 11:07 root_img drwxr-xr-x 9 root root 4096 Apr 22 14:15 var_img

8.7.2.3 Disabling the FA Agents

The FA Agents must be disabled for the maintenance cycle before booting or rebooting the maintenance Application Node. In order to do this, deactivate the service by using the insserv -r command. You can do this directly in the maintenance image without influ-ence on the productive images

Execute in the maintenance root image on the Control Node:

control1:/FlexFrame/volFF/os/Linux/CustomImage_4.0/root_img/etc/ init.d # insserv -r ./myAMC*

8.7.2.4 Modifying the Netboot Configuration

Modify the netboot configuration for the maintenance Application Node with ff_an_adm.pl:

control1:/ # ff_an_adm.pl --op os --name an_0504 --ospath Linux/CustomImage_4.0



8.7.2.5 Creating the New var Image

Create the new var image for the maintenance Application Node on the Control Node. The following steps will be done automatically:






You now have a new Application Node image tree with one newly configured Application Node.

The newly configured Application Node will use this image tree upon the next reboot and can be used for maintenance purposes.

8.7.2.6 Enabling Read-Write Access to the New Root Image

Modify the new image on the NAS system, permit read-write access in the Storage LAN to this Application Node for the root image.

Get the Storage LAN IP address:

control1 # getent hosts an_0504-st 192.168.11.53 an_0504-st

Modify the exports of the NAS system.

NetApp Filer:

control1:/FlexFrame/<filer>/vol0/etc # vi exports .... /vol/volFF/os/Linux/CustomImage_4.0/root_img -sec=sys,rw=192.168.11.53,anon=0

Activate the permissions on the Filer:

control1:/ # rsh <filer>-st exportfs -a

EMC Celerra:

control1 # ssh <cel-co> -l nasadmin .... # server_export <datamover> –Protocol nfs –ignore –option ro,rw=192.168.11.53,anon=0 /vol/volFF/os/Linux/CustomImage_4.0/root_img



NetApp Filer and EMC Celerra:

Reboot this Application Node and remount the root image as read-write:

an_0504:/ # mount -o remount,rw /

Check the write access:

an_0504:/ # cd / an_0504:/ # touch delme an_0504:/ # rm delme

8.7.2.7 Maintaining the Application Node Image

The image maintenance can now go on. Normally RPMs will be installed. Examples for this will be described below.

8.7.2.8 Creating the New var_template

If this new Application Node’s root image and var image are fully operative, then it is time to copy this var image to the new template var image var_template. Obsolete files should be removed.

Copy:

control1:/FlexFrame/volFF/os/Linux/CustomImage_4.0/var_img # mv var_template var_template.OLD control1:/FlexFrame/volFF/os/Linux/CustomImage_4.0/var_img # cp -pdR var-coa80b35 var_template

Cleanup:

control1:/FlexFrame/volFF/os/Linux/CustomImage_4.0/var_img/var_template /tmp # rm -rf * control1:/FlexFrame/volFF/os/Linux/CustomImage_4.0/var_img/var_template /FlexFrame/etc/sysconfig/network # rm ifcfg-vlan*

Do not remove files in CustomImage_4.0/var_img/var_template/log/ because some files are required by several Linux components.



8.7.2.9 Enabling the FA Agents

The FA Agents must be re-enabled on the new root-image:

an_0504:/etc/init.d # insserv ./myAMC*

or, from the Control Node:

control1:/FlexFrame/volFF/os/Linux/CustomImage_4.0/root_img/etc/ init.d # insserv ./myAMC*

8.7.2.10 Disabling Read-Write Access to the New Root Image

Modify the new image on the NAS system, permit only read access to all Application Nodes for the root-image.

NetApp Filer:

control1:/FlexFrame/<filer>-st/vol0/etc # getent hosts an_0504-st 192.168.11.53 an_0504-st control1:/FlexFrame/<filer>-st/vol0/etc # vi exports .... /vol/volFF/os/Linux/CustomImage_4.0/root_img -sec=sys,ro,anon=0 control1:/ # rsh <filer>-st exportfs -a

EMC Celerra:

control1 # ssh <cel-co> -l nasadmin .... # server_export <datamover> –Protocol nfs –ignore –option ro,anon=0 /vol/volFF/os/Linux/CustomImage_4.0/root_img


Remount the root image as read-only:

an_0504:/ # mount -o remount,ro /

8.7.2.11 Migrating the Remaining Application Nodes

Migrate the remaining Application Nodes as described in section “Migrating Remaining Application Nodes to the New Application Node Image” on page 159.



8.7.3 Service Packs It is not recommended to install any Service Packs on Application Node images manually. The only way to obtain a new service pack for Application Nodes is to install new Applica-tion Node images delivered from Fujitsu Siemens Computers. Service Packs contain a lot of changes that can have influences to the FlexFrame functionality. Service Packs can also contain initial configuration files that will substitute the FlexFrame configurations.

8.7.4 Updating/Installing a New Linux Kernel This section describes how to install an additional (alternative) Linux kernel into an exist-ing Application Node image.

For new Linux kernels it is not necessary to do the upgrade in a maintenance image. Every kernel version has its own module directory, its own initial ramdisk initrd and its own kernel file.

The netboot configuration /tftpboot/pxelinux.cfg/<hw_class>_<pool_name>_<image_tree_name> will determine which Linux kernel and initial ramdisk need to be used for a subset of Ap-plication Nodes. The image tree name of original images delivered by FSC begins with FSC_and follows the scheme FSC_<FF version>.<OS version>.<OS architecture>.

8.7.4.1 Software Stage

To install and update any software packages in FlexFrame, it is useful to mount your NAS system or jukebox or any other software stage on a local mount point. The same software stage must be mounted on an Application Node which is appointed to install the new kernel-RPMs.

Our standard software stage in FlexFrame is /FlexFrame/volFF/FlexFrame/stage.

Create a subdirectory for the delivered software:

control1:/FlexFrame/volFF/FlexFrame/stage # mkdir -p SLES9/kernel

Copy the delivered software packages to the software stage.

SMP kernel RPM:

control1:~ # cp <somewhere>/kernel-bigsmp-2.6.5-7.283.i586.rpm /FlexFrame/volFF/FlexFrame/stage/SLES9/kernel



Kernel source RPM:

control1:~ # cp <somewhere>/kernel-source-2.6.5-7.283.i586.rpm /FlexFrame/volFF/FlexFrame/stage/SLES9/kernel

Application Node initrd:

control1:~ # cp <somewhere>/initrd_SLES-9.i686.283.gz /FlexFrame/volFF/FlexFrame/stage/SLES9/kernel

This is not the original initrd as created while installing the kernel RPM. This initrd already contains all necessary drivers for the Application Nodes while net-booting.

Special Fujitsu Siemens Computers driver package:

control1:~ # cp <somewhere>/SUSE-SLES_9-drivers.zip /FlexFrame/volFF/FlexFrame/stage/SLES9/kernel

For official SLES update kernels this driver package can be downloaded from the http://support.fujitsu-siemens.com website.

For special PTF updates this package will be delivered by the Fujitsu Siemens Com-puters support.

Wrapper script to install only GPL compatible driver modules from the driver package:

control1:~ # cp <somewhere>/primesetup_gpl.sh /FlexFrame/volFF/FlexFrame/stage/SLES9/kernel

This script will be delivered together with the updated kernel and initrd files.

8.7.4.2 Installing a New Linux Kernel

Logon to the appointed Application Node:

control1:/ # ssh -X an_0504

Check the currently installed Linux kernels:

an_0504:/ # rpm -qa|grep kernel kernel-source-2.6.5-7.282 kernel-update-tool-0.9-20.4 kernel-bigsmp-2.6.5-7.282

To install any software on the Application Nodes it is necessary to get write permissions to the mounted root file system. Mount the root file system as read-write (example).

Export the root image as read-write only for the maintenance Application Node 192.168.12.19.

http://support.fujitsu-siemens.com/



NetApp Filer:

Edit vol0/etc/exports on the Filer:

control1:/FlexFrame/<filer>/vol0/etc # vi exports ... /vol/volFF/os/Linux/CustomImage_4.0/root_img -ro,anon=0,rw=192.168.12.19

Export the temporary permissions via rsh on the Filer. Telnet to the Filer is also possible:

control1:/ # rsh <filer> exportfs -a

EMC Celerra:

control1:~ # ssh <cel-co> -l nasadmin .... # server_export <datamover> –Protocol nfs –ignore –option ro,rw=192.168.12.19,anon=0 /vol/volFF/os/Linux/CustomImage_4.0/root_img


Don’t forget to export the root image as read-only after the maintenance!

On the Application Node:

Remount the Application Nodes root file system with write permissions:


Mount the software stage:

an_0504:/mnt # mkdir /mnt/stage an_0504:/mnt # mount -t nfs <filer>:/vol/volFF/FlexFrame/stage /mnt/stage

Install the new Linux kernel:

an_0504:/mnt/stage/SLES9/kernel # export YAST_IS_RUNNING=instsys an_0504:/mnt/stage/SLES9/kernel # rpm -ivh kernel-bigsmp-2.6.5-7.283.i586.rpm

Extract the driver package:

an_0504:/mnt/stage/SLES9/kernel # unzip SUSE-SLES_9-drivers.zip

Install GPL compatible drivers from the driver package:

an_0504:/mnt/stage/SLES9/kernel # sh primesetup_gpl.sh

The messages about modprobe.conf, mkinitrd etc. can be ignored.



8.7.4.3 New initrd for Application Nodes

This section describes how to use a new initrd for Application Nodes within the Flex-Frame environment.

First, copy the new initrd file to the /tftpboot directory:

control1:/FlexFrame/volFF/FlexFrame/stage/SLES9/kernel # cp initrd_SLES-9.i686.283.gz /tftpboot/

8.7.4.4 New Kernel and Netboot Configuration

Copy the new kernel from the root_img/boot directory to /tftpboot:

control1:/FlexFrame/volFF/os/Linux/CustomImage_4.0/root_img/boot # cp -p vmlinuz-2.6.5-7.283-bigsmp /tftpboot/vmlinuz-2.6.5-7.283-bigsmp.SLES-9.i686

The suffix .<os>.<arch> is not necessary but helps differencing between different operating systems. Take a look at the default files located in /tftpboot for some file naming examples.

8.7.4.5 Netboot Configuration for the First Test

Change the kernel and initrd filenames to the new ones, in our example -283:

control1:/tftpboot/pxelinux.cfg # vi rx300s2_pool1_CustomImage_4.0 ... KERNEL vmlinuz-2.6.5-7.283-bigsmp.SLES-9.i686 APPEND initrd=initrd_SLES-9.i686.283.gz root=...

ServerView will try to generate (compile) new modules for the new Linux Kernel during the next reboot. The root image has to be writable for this, too.

While rebooting the new kernel, this message will appear on the console:

Compiling ServerView modules for kernel 2.6.5-7.283-bigsmp ... failed

Because of the mount option ro on the Application Node, automatic module compilation fails. For that reason this has to be done manually by mounting the root image read/write and restarting the ServerView agent framework:

an_0504:~ # mount -o remount,rw / an_0504:~ # sh /etc/srvmagt/srvmagt.sh restart ... Compiling ServerView modules for kernel 2.6.5-7.283-bigsmp ... done Loading compiled ServerView modules ... ...



8.7.4.6 Removing Write Permissions for Maintenance Application Node on the Root Image

Mount the root file system as ro:

an_0504:~ # mount -o remount,ro /

Export the root image as ro for all Application Nodes.

NetApp Filer:

Edit the vol0/etc/exports on the Filer:

control1:/FlexFrame/<filer>/vol0/etc # vi exports .. /vol/volFF/os/Linux/CustomImage_4.0/root_img -ro,anon=0

Export the regular permissions via rsh on the Filer. Telnet to the Filer is also possible:


EMC Celerra:


Finally, reboot this Application Node and verify if it is fully functional.

8.7.4.7 Changing the default Netboot Configuration to the New Linux Kernel

In case the kernel tests were run successfully, the netboot configuration for creating new Application Nodes using the custom image should be switched to the new configuration.

This can be achieved by setting the variabled PXE_KERNEL and PXE_INITRD in the file ff_pxe.conf which can be found in the image path /FlexFrame/volFF/os/Linux/<image>.

During the Application Node image installation process, an example template file will be generated which can modified easily.

Edit the ff_pxe.conf as follows, uncomment the variables when using the template:

control1:/FlexFrame/volFF/os/Linux/CustomImage_4.0 # vi ff_pxe.conf ... PXE_KERNEL=vmlinuz-2.6.5-7.283-bigsmp.SLES-9.i686 PXE_INITRD=initrd_SLES-9.i686.283.gz



From now on, every time ff_new_an.sh is called to create a new Application Node for the custom image, these options will be used for the new pxe configuration files.

8.7.5 ServerView Update Prepare the custom Application Node image as described in section “Creating a New Linux OS Image for Application Nodes” on page 144.

Note that the ServerView modules install some parts into the var image (/etc/srvmagt is symlinked to /var/FlexFrame/etc/srvmagt), so do not forget to migrate the main-tenance node var image to the generic var template.

To update or install new software or patches we need write permission for the root image on the NAS system. This should be done for this maintenance Application Node only. For this purpose we temporary remount the root image read-writable.

NetApp Filer:

On the Control Node:

control1:/FlexFrame/<filer>/vol0/etc # vi exports .. #/vol/volFF/os/Linux/CustomImage_4.0/root_img -ro,anon=0 /vol/volFF/os/Linux/CustomImage_4.0/root_img -ro,anon=0,rw=192.168.11.53

Activate the changes:


EMC Celerra:


control1 # ssh <cel-co> -l nasadmin .... # server_export <datamover> –Protocol nfs –ignore –option ro,rw=192.168.11.53,anon=0 /vol/volFF/os/Linux/CustomImage_4.0/root_img


On the Application Node:


Mount your software stage:

an_0504:/mnt # mkdir /mnt/stage an_0504:/mnt # mount -t nfs <filer>:/vol/volFF/FlexFrame/stage /mnt/stage



Check for all srvmagt- RPMs and remove them in the correct order:

1. scs and agents 2. eecd 3. mods_src

an_0504:~ # rpm -qa | grep srvmagt srvmagt-eecd-4.40-08 srvmagt-agents-4.40-08 srvmagt-scs-4.40-08 srvmagt-mods_src-4.40-08 an_0504:~ # rpm –e srvmagt-scs ... an_0504:~ # rpm –e srvmagt-agents ... an_0504:~ # rpm –e srvmagt-eecd ... an_0504:~ # rpm –e srvmagt-mods_src ...

Verify that all packages have been removed successfully:

an_0504:~ # rpm -qa | grep srvmagt an_0504:~ #

Before installing the new RPMs, make sure the link to the var image is being re-created:

an_0504:~ # rm -rf /etc/srvmagt an_0504:~ # ln -s /var/FlexFrame/etc/srvmagt /etc/srvmagt an_0504:~ # rm -rf /etc/srvmagt/*

Now the new packages can be installed, the correct order should be automatically de-tected by the rpm command:

an_0504:~ # cd /mnt/stage/SLES9/SV control1:/mnt/stage/SLES9/SV # ls srvmagt-*4.50-02* srvmagt-agents-4.50-02.suse.rpm srvmagt-mods_src-4.50-02.suse.rpm srvmagt-eecd-4.50-02.suse.rpm srvmagt-scs-4.50-02.suse.rpm an_0504:/mnt/stage/SLES9/SV # rpm -ivh srvmagt-*4.50-02* Preparing... ########################################### [100%] 1:srvmagt-mods_src ########################################### [ 25%] Compiling ServerView modules for kernel 2.6.5-7.282-bigsmp ... done Loading compiled ServerView modules ... copa unused smbus done ipmi done 1 ServerView module(s) unused. See /var/log/log.eecd_mods_src for details!



2:srvmagt-eecd ########################################### [ 50%] AgentX socket is currently /var/agentx/master Starting eecd done 3:srvmagt-agents ########################################### [ 75%] Shutting down snmpd: done Starting snmpd done AgentX socket is currently /var/agentx/master Waiting for eecd sockets ....................................... Waiting for eecd ready ......................... Starting: sc sc2 bus hd unix ether bios secur status inv thr vv done 4:srvmagt-scs ########################################### [100%] Starting SVRemoteConnector done

For security reasons, we now remount our root image as read-only again.

NetApp Filer:


control1:/FlexFrame/<filer>/vol0/etc # vi exports ... /vol/volFF/os/Linux/CustomImage_4.0/root_img -ro,anon=0 #/vol/volFF/os/Linux/CustomImage_4.0/root_img -ro,anon=0,rw=192.168.11.53 control1:/ # rsh <filer>-st exportfs -a

EMC Celerra:



NetApp Filert and EMC Celerra:

On the maintenance Application Node:

an_0504:/ # mount -o remount,ro /

The ServerView Agent update is now finished.



8.7.6 Upgrading RPM Packages on an Application Node Best practice to update RPM packages is “Creating a New Linux OS Image for Applica-tion Nodes” - see section 8.7.2. Alternatively, the deployment of RPM content belonging the var images can be done manually.

This is not recommended!

Anyway, the necessary steps are described below.

Application Nodes do not have a local Linux installation. The installation is divided into a shared root image and a private var image.

The contents of an RPM package are distributed on a package-specific basis to the file system. Therefore, it may occur that the RPM package is located partly in the root image and partly in the var image. If an RPM package is to be loaded, the root image must be temporarily mounted as read-writable.

Possible approaches:

● The RPM package is loaded separately on each Application Node; the part in the shared root image is always overwritten.

● The RPM package is only loaded on one Application Node, and the part in the var image, including the RPM database, is then distributed manually to the other Appli-cation Nodes.

The RPM database is linked from the var image to the root image.

8.7.7 Upgrading the Application Software The rules for installation of application software (SAP/Oracle) apply here as well.

An Application Node should be available exclusively for upgrading the application soft-ware, so that the physical host name can be set to be identical with the virtual host name of the application.

Provided this has been done, upgrading can be carried out in accordance with the soft-ware vendor’s standard guides.

All parts of the application software must reside on the network files system mounted from the NAS system.

For more details see the SAP Installation Guides.



8.7.8 Migrating Remaining Application Nodes to the New Application Node Image

Deploy the new image to other Application Nodes with the administration tools.

8.7.8.1 Modifying the Netboot Configuration

Modify the netboot configuration for each Application Node with ff_an_adm.pl:

Synopsis

ff_an_adm.pl –op os --name <app_node_name> --ospath <directory>

To modify the netboot configuration files for a single Application Node, i.e. Application Node an_0506, invoke:

control1:/ # ff_an_adm.pl –op os --name an_0506 --ospath Linux/CustomImage_4.0

8.7.8.2 Creating the New var Images

Create the new var images for the Application Nodes. The following steps will be done automatically:





Synopsis

ff_new_an.sh {-n <app_node_name>|-s <search_criteria>}

To create a new var image for one Application Node, i.e. an_0506 invoke:


To create new var images for all Application Nodes belonging the new Application Node image CustomImage_4.0, invoke:

control1:/ # ff_new_an.sh -s 'PATH_TO_IMAGES=/FlexFrame/volFF/os/Linux/CustomImage_4.0'

This will create all var-images for the new Application Node image at once.

The newly configured Application Nodes will use their new images upon the next reboot.

Administrating Application Nodes Installation / Activation of New Solaris Images


8.8 Installation / Activation of New Solaris Images

8.8.1 Introduction

8.8.1.1 General Notes

Seen from the view of a Control Node the Solaris Application Node images are located in a folder on the NAS storage at /FlexFrame/volFF/os/Solaris/<identifier>.

<identifier> denotes the version and origin of the images below. For images created at Fujitsu Siemens Computers the identifier is built as follows:

<identifier> = FSC_<SunOS_version>_<version_details>

Names for customer specific Solaris images must not start with “FSC”. It is rec-ommended to use a similar naming convention.

In this directory you will find one or more boot images (bi*) in the form

bi_<server_class>

<server class> denotes groups of PRIMEPOWER servers within the same server class which share the same basic technology.

As of the time this document was written, the following server classes are supported:

FJSV,GPUZC-M_PW-P (PW 250 and PW 450) FJSV,GPUZC-M_PW-CMZ (PW 650 and PW 850)

The classification in different hardware platforms is necessary because of the ESF (En-hanced Shutdown Facility) software, which has to be installed on each PRIMEPOWER system.

The next directory level holds a structure similar to the Solaris diskless client concept:

Solaris_<version> exec root

In exec /Solaris_<version>_sparc.all/usr you will find the read-only area (/usr). This folder is shared between all Application Nodes of the same server class.

The root folder holds the clone and directories named like the Application Nodes using this image (including -st suffix for Storage LAN). The clone is the basis for creating an Application Node specific image.

Installation / Activation of New Solaris Images Administrating Application Nodes


8.8.1.2 How to Access the Console of a PW 250 / PW 450

The PRIMEPOWER server models 250 and 450 have a built-in XSCF (eXtended System Control Facility) which allows the administrator to access the console of the server via LAN interface. During the first boot of a PRIMEPOWER the console is redirected to this LAN interface.

Here’s how you access the console (e.g. of Application Node an_0800):

control1:~ # telnet an_0800-co Trying 192.168.20.30... Connected to an_0800-co. Escape character is '^]'. an_0800 console login:

Now you have direct access to the Application Node’s console. We recommend keeping the consoles of the Application Nodes open during boot and maintenance work.

If the console is not accessible (locked by another administrator or the Application Node is not responding, etc.) you can connect directly to the XSCF itself using port 8010:

control1:~ # telnet an_0800-co 8010 Trying 192.168.20.30... Connected to an_0800-co. Escape character is '^]'. SCF Shell login:root Password: SCF Version 03190001 ALL RIGHTS RESERVED, COPYRIGHT (C) FUJITSU LIMITED 2003 ff.ff.ff[192.168.20.30] SCF>

To hang-up a telnet console connection enter:

SCF> hangup 23

If the Application Node got stuck, check the console logs using the following command:

SCF> show-console-logs

To initiate a hardware-reset enter the following.

The Application Node will unconditionally enter the OBP prompt and processes running on the Application Node will be interrupted without the chance of a con-trolled stop!



SCF> xir Are You sure to Reset(XIR)? [y/n] y

On the console you will see:

Externally Initiated Reset {1} ok

Now you can now reboot the Application Node.

8.8.1.3 OBP Flag: Local-mac-address?

The following option must be set:

{0} ok setenv local-mac-address? true

If this setting is not done, strange network behaviour may be the result. Some network switches cannot handle the same MAC address at different ports/VLANs.

The address reported by banner may differ! banner typically reports hme0's mac address, which is not used in FlexFrame.

8.8.1.4 OBP Flag: Auto-boot?

For testing purposes, it may be useful to disable auto-boot:

{0} ok setenv auto-boot? false {0} ok printenv auto-boot? auto-boot? = false

You can enable it again after testing with

{0} ok setenv auto-boot? True

8.8.1.5 What Happens When the Solaris Client Boots

A freshly installed and not yet booted Solaris has a flag file. At the first boot, rc scripts run depending on this flag file. Be sure to have at least the line reading e.g.

network_interface=PRIMARY {protocol_ipv6=no hostname=pw250f-st ip_address=192.168.1.100 netmask=255.255.255.0}

properly configured in /etc/sysidcfg. The host name must match the network name of the NIC in the Storage LAN you are booting from.

Installation / Activation of New Solaris Images Administrating Application Nodes


On the very first boot /etc/sysidcfg is read by a couple of tools, which in turn edit e.g. /etc/hosts, /etc/default/init etc. After the next reboot /etc/sysidcfg is no longer used. After having mounted its root file system, the client mounts /usr as defined in /etc/vfstab. The system should now go to multiuser level as usual.

We start the boot process from the console of the PRIMEPOWER (from the OBP).

More information is displayed with the options -dv.

Fujitsu Siemens PRIMEPOWER250 2x SPARC64 V, No Keyboard OpenBoot 3.10.1-1, 4096 MB memory installed Ethernet address 0:e0:0:c4:9a:b, Host ID: 80f39a0b. XSCF Version: 3.19.1 {0} ok boot -dv Boot device: /pci@83,2000/FJSV,pwga@1 File and args: -dv Timeout waiting for ARP/RARP packet 2b600 Using RARP/BOOTPARAMS... Requesting Internet address for 0:e0:0:c5:1a:b Internet address is: 192.168.10.30 hostname: an_0800-st whoami: Router ip is: 192.168.10.201 Found 192.168.10.201 @ 0:30:5:40:6a:b9 root server: <filer>-st(192.168.10.203) root directory: /vol/volFF/os/Solaris/FSC_5.8_202_20050211/bi_FJSV,GPUZC-M_PW-P/root/an_0800-st Found 192.168.10.203 @ 2:a0:98:1:1e:92 Size: 376880+105005+158015 Bytes SunOS Release 5.8 Version Generic_117350-18 64-bit Copyright 1983-2003 Sun Microsystems, Inc. All rights reserved. Ethernet address = 0:e0:0:c4:9a:b ...

Explanation of this screen:

The message Timeout waiting... may occur a few times. If it does not continue, check if the rarpd process is running on the Control Node and if there is an appropriate entry in /etc/ethers on the Control Nodes.

Another cause could be that the network interface of the Application Node is not in the network segment where the rarpd of the Control Node is listening on. In this case check the network cabling and switch configuration.

The 2b600 is given as an example, this value may vary depending on the file size. It may be a different count on your installation. It is the size (in hex) of the TFTP kernel file we have linked to the Application Node’s IP address in Hex in the /tfttpboot directory of the Control Node.



The Router ip is: ... denotes the Control Node, for the boot process, the Control Node is the router (which is not really serving with a routing function).

The root server and root directory must point to the NAS system and the direc-tory where the root file system of the Application Node is located. If the boot process stops right after this message, the root file system may not be found or the access per-missions in the exports of the NAS system are not set correctly. Another possibility could be network segment issues.

The boot phase should run until you see a login prompt.

The system is ready. an_0800 console login:

8.8.2 Solaris Image – rc scripts If the Application Node does not come up (e.g. “hangs” during execution of an rc script), it is useful to know which rc script is currently being executed.

To get this information you need to create a file /etc/.trace.sd into the root file sys-tem of the Application Node (e.g. from the Control Node), e.g. like:

control1:~ # touch /FlexFrame/volFF/os/Solaris/FSC_5.8_202_20050211/ bi_ FJSV,GPUZC-M_PW-P/root/an_0800-st/etc/.trace.sd

The messages look like:

... ##### rc2: /etc/rc2.d/S01SMAWswapmirror start ##### rc2: /etc/rc2.d/S05RMTMPFILES start ##### rc2: /etc/rc2.d/S06log3 start Starting the logging daemon Started! ##### rc2: /etc/rc2.d/S10lu start ##### rc2: /etc/rc2.d/S19FJSVdmpsnap start ##### rc2: /etc/rc2.d/S20sysetup start ...

If you want turn of the messages again, remove the file .trace.sd, e.g. like:

control1:~ # rm /FlexFrame/volFF/os/Solaris/FSC_5.8_202_20050211/ bi_ FJSV,GPUZC-M_PW-P/root/an_0800-st/etc/.trace.sd

Solaris Image Maintenance Cycle Administrating Application Nodes


8.9 Solaris Image Maintenance Cycle This section describes how to run a maintenance cycle for an existing Solaris image.

It does not describe the creating of a new Solaris image. If you want to create a new Solaris image please refer to the manual “Installation of a FlexFrame Environment”, sec-tion “Preparation for Solaris Application Nodes".

The Solaris image maintenance cycle has been redesigned and is now easier than in former FlexFrame releases. Please read carefully this section.

8.9.1 Introduction Running a Solaris image maintenance cycle means maintaining an existing Solaris im-age: You might want to apply patches, install or upgrade software, modify several con-figurations or change other settings on a Solaris image where your Solaris Application Nodes run on.

In order to run a Solaris image maintenance cycle you will need a

● Solaris image ● Solaris Application Node running on that Solaris image ● Solaris Application Node console port access ● time slot of min. 2 hours ● the software to install and/or a plan what you want to maintain

The Solaris Application Node selected for running the Solaris image mainte-nance cycle won't be visible to FlexFrame i.e. in particular won't serve as a spare node during that time.

Only one Solaris Application Node can run the Maintenance Cycle at the same time!

When running in Solaris Maintenance mode the Application Node is only acces-sible via console port. In this case there is no remote access like rsh, ssh or telnet to the Application Node.

We would like to point out that changes due to 3rd party software or patches or customer modifications may have heavy (negative) side effects to the Flex-Frame functionality.

Before installing any 3rd party software, please see section “Third Party Soft-ware” on page 117.

We would also like to point out that there is software on the market which can not be installed on NFS or is not ready for Adaptive Computing (e.g. moving SAP services from one to another node). If this is the case, contact your 3rd party software vendor. It is usually not an issue of FlexFrame.

Administrating Application Nodes Solaris Image Maintenance Cycle


8.9.2 Overview This section lists an example in order to give an idea how the Solaris image maintenance cycle works. Basically the Solaris image maintenance cycle comprises of just three steps:

1. Transfer a Boot Image into a Maintenance Image.

2. Boot a Solaris Application Node from it and maintain it: install or upgrade software, modify, apply.

3. Transfer the Maintenance Image into a Boot Image and boot all of the Solaris Appli-cation Nodes of that group from the maintained Boot Image.

Example:

Assuming you run a couple of Solaris Application Nodes on a certain Image. To make it short let's call it GOODIMAGE.

Now you decide that it is needed to have a certain piece of software SOFTWARE on these Solaris Application Nodes.

You know that you can't install software directly on the Solaris Application Nodes:

● On the one hand the Solaris Application Nodes of a group share the same /usr file system. This is only read-only for all of them.

● On the other hand you don't like to install software to all of them (via the shared file system) without a careful test procedure.

● Beside that you must prevent FlexFrame from taking over this Solaris Application Node in case of failures of another Solaris Application Node during the installation of your software SOFTWARE.

Therefore you have to select one of your Solaris Application Nodes and bring it into a so called maintenance state. This comprises of:

1. Choose one Solaris Application Node which is running on the Image you like to maintain.

2. Save the configuration data of this Solaris Application Node - you might want to save the configuration data for documentation purposes.

3. Create a so-called Maintenance Image out of the Solaris image GOODIMAGE the chosen Solaris Application Node run on.

4. Rename this Solaris Application Node: The Application Node will get only a new name.

5. Change configuration information for this Solaris Application Node: assign the newly created Maintenance Image to the renamed Application Node.

6. Configure this Solaris Application Node.



7. Boot this Solaris Application Node.

Now this Solaris Application Node runs on a Maintenance Image which is separated from the original GOODIMAGE. All the other Solaris Application Nodes on the GOODIMAGE run without getting touched. The Maintenance Image allows software installation since it has in particular it's own writable /usr file system and the FlexFrame (FA Agents) don't control it. That's why you can continue now:

8. Install the software SOFTWARE on the Solaris Application Node and test it. Configure, reboot, change settings of SOFTWARE until you are satified.

The result is a modified GOODIMAGE which contains the software SOFTWARE. To make it short let's call it BETTERIMAGE. In order to have the software SOFTWARE available on all the Solaris Application Nodes in the group you have to:

9. Shut down the Solaris Application Node.

10. Rename the Solaris Application Node you used in Maintenance back to the former name.

11. Transfer the Maintenance Image BETTERIMAGE back to a Boot Image (now recog-nizable by ff_an_adm.pl).

12. Create the Solaris Application Node Image based on the Boot Image created in #11 - i.e. on the BETTERIMAGE instead of on the GOODIMAGE.

13. Save the configuration data of this Solaris Application Node - you might want to save the configuration data for documentation purposes.

14. Boot this Solaris Application Node.

Now the Maintenance cycle is done but we have to switch the Solaris Application Nodes onto the new maintained Image.

Now set by help of ff_an_adm.pl and ff_new_an.sh all of the Solaris Application Nodes of the group to the BETTERIMAGE (see 8.9.4). Just reboot them and all of them contain the modifications you made during maintenance.

Switching from GOODIMAGE to BETTERIMAGE costs only a reboot! In case of problems you can switch back in the same way.

But please bear in mind:

It is very important that all the Solaris Application Nodes of one group have the same configuration (software, patches) since each one must be able to take over if another one fails. That's why all of the Solaris Application Nodes of a group must run on the same image.



8.9.3 Running the Cycle This section shows in detail how the process works. It includes a real life example you easily can follow. The numbering refers to the "Overview” section.

Please log in onto a FlexFrame Control Node and change the directory to /opt/SMAW/SMAWnbpw/bin.

1. Define the Boot Image we want to run a maintenance cycle on. The Boot Image could be any Image suitable for running clients on. Normally it should be a proven Boot Image where clients run successful on. Our Solaris Boot Image is /FlexFrame /volFF/os/Solaris/FSC_5.9_904_20060607/bi_FJSV,GPUZC-M_PW-P. Choose a Solaris Application Node running on that image. This Solaris Application Node has now to be brought down and will not be available during the maintenance cycle for normal FlexFrame operation. Our Solaris Application Node has the name kid20_4:

control1: ssh kid20_4 init 0

2. Save the configuration data of this Solaris Application Node. This is only for docu-mentation purposes – you may skip it:

control1: ff_an_adm.pl --op list --name kid20_4 --cmdline > kid20_4.config

3. Create a so called Maintenance Image out of the Boot Image. The command to do that is nb_boot2maintenance (for details and example output refer to the respec-tive chapter in the User Guide). The nb_boot2maintenance needs two parame-ters to work: The Boot Image we like to maintain and a target directory. The latter will get the Maintenance Image. Please note that it must not exist and should get a meaningful name: It must not start with FSC but should start with MAINTENANCE and reflect the date or version or other tags we need to identify it. We choose /FlexFrame/volFF/os/Solaris/MAINTENANCE_5.9_904_20060704.

control1: ./nb_boot2maintenance -s /FlexFrame/volFF/os/Solaris/FSC_5.9_904_20060607/ bi_FJSV,GPUZC-M_PW-P -d /FlexFrame/volFF/os/Solaris/MAINTENANCE_5.9_904_20060704

The command will need some time to finish.

4. Rename this Solaris Application Node. The Application Node will get only a new name; all the other data stored in LDAP are the same. This allows the use of the same configuration data of kid20_4 for the node during maintenance cycle.

control1: ff_an_adm.pl --op rename --name kid20_4 --newname maintenance



The name maintenance is mandatory! Please ignore exceptionally the instruction ff_new_an.sh -n maintenance given by ff_an_adm.pl!

The command writes a file /tftpboot/config/netboot_pool1_maintenance.cfg we will need later.

5. Change configuration information for the Solaris Application Node during its life in the Maintenance cycle: Assign the newly created Mainztenance Image (see #3) to the renamed Application Node (see #4):

control1: ff_an_adm.pl --op os --name maintenance --ospath Solaris/MAINTENANCE_5.9_904_20060704

Please ignore exceptionally the instruction ff_new_an.sh -n maintenance given by ff_an_adm.pl!

The command writes a file /tftpboot/config/netboot_pool1_maintenance.cfg we will need later.

6. Configure this Solaris Application Node in order to make it ready to boot. The com-mand to do this is nb_configure_client. This command reads the configuration file we've got on the fly by the last ff_an_adm.pl call. Don't forget to type in the -c parameter: it will run the command in cluster mode and therefore not harm the clus-ter consistency monitored by RMS. Don't forget to type in the -M parameter: it will run the command in maintenance mode resulting in writable /usr file systems later.

control1: ./nb_configure_client -M -c -r /tftpboot/config/netboot_pool1_maintenance.cfg

7. Now we are able to boot the Solaris Application Node.

{0} ok boot

You may want to specify an option to enable a more verbose output: for Solaris 8 and Solaris 9 use boot -v, for Solaris 10 use boot -m verbose.

8. Since the Solaris Application Node runs on a separate Image with writable file sys-tems the real maintenance can start. We can apply patches, install or upgrade soft-ware, modify several configurations or change other settings on the system. We should carefully check the system after applying the changes and respect the warn-ing we read in the “Introduction” section.

Any change we do will remain in the image i.e. any Solaris Application Node which will run on that image later on will "inherit" everything. This is good for sure in case of intended changes. But keep an eye on unwanted log files, notes, and so on. Clean them up.



9. After the test run of the Solaris Application Node on the Maintenance Image we have to shut it down again. Please do this using the console of the Solaris Application Node:

maintenance-st console # init 0

10. Rename this Solaris Application Node. Now we will bring the former Solaris Applica-tion Node kid20_4 back to life: This is the counterpart to the rename operation in #4.

control1: ff_an_adm.pl --op rename --name maintenance --newname kid20_4

Please ignore exceptionally the instruction ff_new_an.sh -n kid20_4 given by ff_an_adm.pl!

The command writes a file we will need later: /tftpboot/config/netboot_pool1_ kid20_4.cfg

11. Now we have to transfer the Maintenance Image into a Boot Image. The command to do that is nb_maintenance2boot (for details and example output refer to the re-spective chapter in the User Guide). This is the counterpart to nb_boot2maintenance and transfers the Maintenance Image into a Boot Image by moving appropriate contents and cleaning up the Image. The command reads the configuration file created in #10:

control1: ./nb_maintenance2boot -r /tftpboot/config/netboot_SOLUX_kid20_4.cfg

12. Create the Solaris Application Node Image as requested in #10 and based on the Boot Image BETTERIMAGE created in #11:

control1: ff_new_an.sh -n kid20_4

13. Save the configuration data of this Solaris Application Node. This is only for docu-mentation purposes - you may skip it:

control1: ff_an_adm.pl --op list --name kid20_4 --cmdline > kid20_4.maintain.config

Comparing this kid20_4.maintain.config with the kid20_4.config created in #2 reveals the differences of the Application Node before and after the Mainte-nance cycle: Only the Image path has changed.

14. Now we are able boot the Solaris Application Node but we have to consider: The test run we did was a test run without the specific FlexFrame software. The ff_new_an.sh -n kid20_4 call enabled the FlexFrame functions. Therefore after coming up the FlexFrame (FA Agents) will recognize the Application Node. In order to prevent this new Application Node from being treated as a spare node it is wise to hide the Node. The procedure to do this depends on the Solaris release.



On Solaris 8 and 9: Remove the respective link:

control1: cd /FlexFrame/volFF/os/Solaris/MAINTENANCE_5.9_904 _20060704/bi_FJSV,GPUZC-M_PW-P/root/kid20_4-st/etc/rc3.d control1: mv S20myAMC.FA_AppAgent _S20myAMC.FA_AppAgent

On Solaris 10 wait until the Solaris Application Node is up and after logged in type:

kid20_4 # svcadm disable -st smawfa-agents

Now we boot the Solaris Application Node:

{0} ok boot

You may want to specify an option to enable a more verbose output: For Solaris 8 and Solaris 9 use boot -v, for Solaris 10 use boot -m verbose.

The kid20_4 system will come up and will contain all the changes we made during maintenance. It runs in the FlexFrame environment without serving as a spare node. If this works now without any problems as well we are done. The Maintenance cycle is actually finished but we have to switch the Solaris Application Nodes onto the new maintained Image.

8.9.4 Create Solaris Application Nodes on the Maintained Image

The very last step is to assign the new - maintained - Image to all the other Solaris Appli-cation Nodes. E.g. we have the Nodes kid20_1, kid20_2, and kid20_3 in our envi-ronment. Since we only have to change their Boot Image path it is easy to do that:

control1: ff_an_adm.pl --op os --name kid20_1 --ospath Solaris/MAINTENANCE_5.9_904_20060704 control1: ff_an_adm.pl --op os --name kid20_2 --ospath Solaris/MAINTENANCE_5.9_904_20060704 control1: ff_an_adm.pl --op os --name kid20_3 --ospath Solaris/MAINTENANCE_5.9_904_20060704 After that launch ff_new_an.sh as in #12.

control1: ff_new_an.sh -n kid20_1 control1: ff_new_an.sh -n kid20_2 control1: ff_new_an.sh -n kid20_3

Up to this point the Solaris Application Nodes can still operate on the previous version of the Solaris image. Finally we need only to reboot the Solaris Applica-tion Nodes to change their Image from the previous to the new maintained Im-age.

Administrating Application Nodes Image Customization for Experts


control1: ssh kid20_1 init 6 control1: ssh kid20_2 init 6 control1: ssh kid20_3 init 6 And now we can activate the kid20_4 as a fully functional FlexFrame Solaris Application Node by moving back the link and rebooting the system.

On Solaris 8 and 9:

control1: cd /FlexFrame/volFF/os/Solaris/MAINTENANCE_5.9_904 _20060704/bi_FJSV,GPUZC-M_PW-P/root/kid20_4-st/etc/rc3.d control1: mv _S20myAMC.FA_AppAgent S20myAMC.FA_AppAgent control1: ssh kid20_4 init 6

On Solaris 10 you don't have to do anything. Because the disable command was a tem-porary one (svcadm disable -t) the service will start when the system reboots.

Because all the Solaris Application Nodes of a group must have the very same software it is recommended to bring up the Solaris Application Nodes on the new maintained Image in a short period of time..

8.10 Image Customization for Experts If new software has been brought into the image, it may have to be configured individually for each Application Node. Here is the process how to achieve this:

In /opt/SMAW/SMAWnbpw/bin/nbrc.FF31_00 the customer may create scripts on both Control Nodes in the form nb_customn_<NR>.

Where <NR> is a number with two digits. There are sample scripts in the same folder.

The pool name can be read from $POOL_NAME, but there's currently no variable denoting the group name within a pool.

The scripts must be backed up carefully since a new Control Node image or Solaris im-age may remove the complete folder to install new scripts of the software package.

For detailed information, please take a look at the “FlexFrame Installation Guide”, section “Script: nb_customize_client”.

Troubleshooting Administrating Application Nodes


8.11 Troubleshooting

8.11.1 Solaris Image – Traces of Solaris rc-scripts If the Application Node does not come up (e.g. “hangs” during execution of an rc-script), it is useful to know which rc-script is currently being executed. To get this information you need to create a file /etc/.trace.sd into the root file system of the Application Node (e.g. from the Control Node), e.g. like:

control1:~ # touch /FlexFrame/volFF/os/Solaris/FSC_5.8_202_20050211/ bi_ FJSV,GPUZC-M_PW-P/root/an_0800-st/etc/.trace.sd

The messages look like:

##### rc2: /etc/rc2.d/S01SMAWswapmirror start ##### rc2: /etc/rc2.d/S05RMTMPFILES start ##### rc2: /etc/rc2.d/S06log3 start Starting the logging daemon Started! ##### rc2: /etc/rc2.d/S10lu start ##### rc2: /etc/rc2.d/S19FJSVdmpsnap start ##### rc2: /etc/rc2.d/S20sysetup start

The messages can be turned off by removing the file .trace.sd, e.g. like:

control1:~ # rm /FlexFrame/volFF/os/Solaris/FSC_5.8_202_20050211/ bi_FJSV,GPUZC-M_PW-P/root/an_0800-st/etc/.trace.sd

8.11.2 Problems With /usr During Maintenance Cycle Please verify that the /usr file system is mounted read-writable. You can test this by reading the output of the mount command, or by trying to create a file on /usr:

an_Solaris# touch /usr/testfile

If an error message like

touch: /usr/testfile cannot create

appears, the /usr file is still read-only.

Make sure to remove the testfile to avoid confusion.

an_Solaris# rm /usr/testfile

Administrating Application Nodes Troubleshooting


8.11.3 Boot Hangs With “Timeout waiting…” If the boot process does not continue after the message Timeout waiting..., check if the rarpd process is running on the Control Node and if an appropriate entry in /etc/ethers is listed. Another cause could be that the network interface of the Applica-tion Node is not in the network segment where the rarpd of the Control Node is listening on. The file /etc/ethers should not be modified manually, but by using ff_new_an.sh instead.

8.11.4 Boot Hangs After “router IP is…” If the boot process stops right after this message, the root file system may have not been found or the access permissions in the exports of the NAS system are not set correctly. Another possibility could be network segment issues.

8.11.5 Boot Stops Complaining About /usr Check the client's /etc/vfstab.

8.11.6 Boot Hangs When Trying to Mount /usr Hostname: pw250f-1a nfs mount: file1a: RPC: Rpcbind failure - RPC: Success nfs mount: retrying: /usr

Check /usr in the vfstab and the NAS system’s exports. Check that it is actually ex-ported via exportfs. Check /etc/hosts.

8.11.7 Boot Asks for Date, Time and/or Locale The boot phase should run until you see a login prompt. If the Application Node asks for input (like time and date etc), something went wrong. The settings in ../etc/sysidcfg (AN) file could not be interpreted correctly.

Pools and Groups Administrating Application Nodes


8.12 Pools and Groups

8.12.1 Adding a Pool A new pool can be added using the maintenance tool ff_pool_adm.pl. Some parame-ters have to be defined on the command line. They are used to configure switch VLANs and ports, to create the NAS system volume folder structures, to create LDAP pool sub-tree and to configure Control Nodes.

Synopsis

ff_pool_adm.pl

--op add --name <pool_name> --storage <vlan_id>,<network_ip>,<netmask>

--server <vlan_id>,<network_ip>,<netmask>

--client <vlan_id>,<network_ip>,<netmask> --dns <domain_name>,[<dns_server_ ip>]

[--sapdata <nas_name>[,<volume_path>]]

[--saplog <nas_name>[,<volume_path>]] [--volff <nas_name>[,<volume_path>]]

[--volff_common <nas_name>[,<volume_path>]]

[--defrouter <default_router_ip>] [--switchgrp <id>[,<id>]]

Options

--op add Adds a pool and displays information about processing steps, errors and warnings.

--name <pool_name> Name of new pool (has to be unique within entire FlexFrame). We recommend using short lowercase names for <pool_name>.

--storage <vlan_id>,<network_ip>,<netmask> Pool specific storage network segment. The option is followed by a comma sepa-rated list of the VLAN ID, the network IP and the netmask of the network IP address.

--server <vlan_id>,<network_ip>,<netmask> Pool specific server network segment. The option is followed by a comma separated list of the VLAN ID, the network IP and the netmask of the network IP address.

Adding a pool changes the exports file on all given NAS systems. Temporary exports on these NAS systems will be gone after running ff_pool_adm.pl. Be sure not to have temporary exports.

Administrating Application Nodes Pools and Groups


--client <vlan_id>,<network_ip>,<netmask> Pool specific client network segment. The option is followed by a comma separated list of the VLAN ID, the network IP and the netmask of network IP.

--dns <domain_name>,[<dns_server_ip>] DNS domain name and servers to be used for this pool. More than one DNS server IP address may be given. Keep in mind to use the default router option if the server IP addresses do not match any of the pool networks. A DNS option may look like this: my.domain.com,192.168.251.17,192.168.251.18

--sapdata <nas_name>[,<volume_path>] Optional NAS name and volume path the pool should use for sapdata. A missing vol-ume path is auto-filled with the default (/vol/sapdata/<pool_name>). e.g. filer1,/vol/sapdata/pool1. The entire option defaults to common NAS name with default path. <nas_name> is the NAS system's node name for this pool (without -st suffix).

--saplog <nas_name>[,<volume_path>] Optional NAS name and volume path the pool should use for saplog. A missing vol-ume path is auto filled with the default (/vol/saplog/<pool_name>). e.g. filer1,/vol/saplog/pool1. The entire option defaults to common NAS name with default path. <nas_name> is the NAS system's node name for this pool (without -st suffix).

--volff <nas_name>[,<volume_path>] Optional NAS name and volume path the pool should use for volFF. A missing vol-ume path is auto filled with the default (/vol/volFF/pool-<pool_name>). e.g. filer1,/vol/volFF/pool-pool1. The entire option defaults to common NAS name with default path. <nas_name> is the NAS system's node name for this pool (without -st suffix).

--volff_common <nas_name>[,<volume_path>] Optional NAS name and volume path the pool should use for common volFF data. A missing volume path is auto filled with the default (/vol/volFF). e.g. filer1,/vol/volFF. The entire option defaults to common NAS name with default path. <nas_name> is the NAS system's node name for this pool (without -st suffix).

Currently it has to be the first NAS system of the FlexFrame landscape and /vol/volFF/pool-<pool_name>.

--defrouter <default_router_ip>

The default router is a gateway to route IP data to other, non-pool local networks. All IP data that can not be addressed to a local network will be sent to the default router to be forwarded to the destination network. The option parameter is an IP address of this default router. Use a default router IP address matching one of the local pool networks, because otherwise it will not be accessible by Application Nodes.



--switchgrp <id>[,<id>] The switch group ID(s) the Client LAN to corporate LAN ports should be configured. If not given, the client VLAN is assigned to the existing trunk ports or a new port pair at first both switch groups. Not more than two switch group IDs are accepted.

Output example

cn1:~ # ff_pool_adm.pl --op add --name pool4 --storage 30,192.168.30.0,255.255.255.0 --server 31,192.168.31.0,255.255.255.0 --client 32,192.168.32.0,255.255.255.0 --sapdata filer --saplog filer --volff filer --volff_common filer --dns my.domain.com --defrouter 192.168.32.254 update LDAP .......................... .......................... .......................... .......................... .......................... .......................... ..... update switch 1/1 configuration Notice: Update will take about 1 minute. vlan: storage-30 has been created restart cluster service ldap_srv1 Notice: restart will take up to 1 minute. stop and wait until service is offline start and wait until service is online restart cluster service ldap_srv2 Notice: restart will take up to 1 minute. stop and wait until service is offline start and wait until service is online restart cluster service netboot_srv Notice: restart will take up to 2 minutes. stop and wait until service is offline start and wait until service is online If not reported any warnings or errors all precautions are done and the pool was successfully created. Use ff_poolgroup_adm.pl to define the host groups of this pool to be able to add application nodes. See /tmp/pool-pool4/ff_pool_adm.errlog for complete error and warning log.



8.12.2 Removing a Pool A pool can be removed using the maintenance tool ff_pool_adm.pl. Some parame-ters have to be defined on the command line. Switch VLANs will be removed and the affected ports reconfigured. The LDAP pool subtree will be removed and Control Node configurations rewritten.

A pool may not be removed if any Application Node or SID is defined. The fist pool may not be removed due to system requirements.

Removing a pool changes the exports on all NAS systems used by this pool. Use operation mode list or list-all to get the storage configuration of pool to be removed. Temporary exports (not written to the exports file /vol0/etc/exports) on this Filers will be gone after running ff_pool_adm.pl.

Be sure not to have temporary exports.

Synopsis

ff_pool_adm.pl --op rem --name <pool_name>

Options

--op rem Removes a pool and displays only errors and warnings.

--name <pool_name> Name of pool to be removed. Use ff_pool_adm.pl --op list-all to get a list of currently configured pools (see 8.12.4).

Output example

cn1:~ # ff_pool_adm.pl --op rem --name pool4 update LDAP .......................... .......................... .......................... .......................... .......................... .......................... update switch 1/1 configuration Notice: Update will take about 1 minute. restart cluster service ldap_srv1 Notice: restart will take up to 1 minute. stop and wait until service is offline



start and wait until service is online restart cluster service ldap_srv2 Notice: restart will take up to 1 minute. stop and wait until service is offline start and wait until service is online restart cluster service netboot_srv Notice: restart will take up to 2 minutes. stop and wait until service is offline start and wait until service is online If not reported any warnings or errors the pool was successfully removed. Keep in mind, the volumes and their data were not harmed. It's on you to remove them. See /tmp/pool-pool4/ff_pool_adm.errlog for complete error and warning log.

8.12.3 Listing Pool Details To list the configurations details of a pool like used networks, pool groups, SIDs and Application Nodes, the maintenance tool ff_pool_adm.pl can be used. The pool name has to be defined on the command line.

Synopsis

ff_pool_adm.pl --op list --name <pool_name>

[--list <part>[,<part>]]

Options

--op list Displays pool configuration details

--name <pool_name> Name of pool to be listed. Use ff_pool_adm.pl -op list-all to get a list of cur-rently configured pools (see 8.12.4).

--list <part>[,<part>] To reduce the output to the interesting parts, use this option. The parameters to this option are the display sections. Add them as a comma separated list. The default sections are: network,storage,dns,group,sid,an,cn,NAS system. You may also use a two character abbreviation instead the full section name like ne for network.



Output examples

cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list --name p1 Pool configuration details of pool p1 Networks Client-LAN Network: 10.10.10.0 Netmask: 255.255.255.0 VLAN ID: 100 Server-LAN Network: 192.168.10.0 Netmask: 255.255.255.0 VLAN ID: 110 Storage-LAN Network: 192.168.20.0 Netmask: 255.255.255.0 VLAN ID: 120 Def.Router: 192.168.10.254 Storage Volumes sapdata fas01-p1-st:/vol/sapdata/p1 saplog fas01-p1-st:/vol/saplog/p1 volFF fas01-p1-st:/vol/volFF/pool-p1 volFF shared fas01-p1-st:/vol/volFF DNS data Domain Name: my.domain.com Pool Groups Linux OS: SUSE Linux SLES-9.X86_64 Solaris OS: Sun SunOS 5.8 SIDs and their instances D01 SAP Version: SAP-6.20 DB Version: Oracle-9 Instances Type db ID 0 Server-LAN: dbd01-se 192.168.10.110 Type ci ID 9 Client-LAN: cid01 10.10.10.111 Server-LAN: cid01-se 192.168.10.111 Type app ID 10 Client-LAN: app10d01 10.10.10.112 Server-LAN: app10d01-se 192.168.10.112 ID 11 Client-LAN: app11d01 10.10.10.113 Server-LAN: app11d01-se 192.168.10.113 P01



SAP Version: SAP-6.20 DB Version: Oracle-9 Instances Type db ID 0 Server-LAN: dbp01-se 192.168.10.100 Type ci ID 0 Client-LAN: cip01 10.10.10.101 Server-LAN: cip01-se 192.168.10.101 Type app ID 1 Client-LAN: app01p01 10.10.10.102 Server-LAN: app01p01-se 192.168.10.102 ID 3 Client-LAN: app03p01 10.10.10.103 Server-LAN: app03p01-se 192.168.10.103 ID 4 Client-LAN: app04p01 10.10.10.104 Server-LAN: app04p01-se 192.168.10.104 ID 5 Client-LAN: app05p01 10.10.10.105 Server-LAN: app05p01-se 192.168.10.105 Q01 SAP Version: SAP-6.20 DB Version: Oracle-9 Instances Type db ID 0 Server-LAN: dbq01-se 192.168.10.106 Type ci ID 6 Client-LAN: ciq01 10.10.10.107 Server-LAN: ciq01-se 192.168.10.107 Type app ID 7 Client-LAN: app07q01 10.10.10.108 Server-LAN: app07q01-se 192.168.10.108 ID 8 Client-LAN: app08q01 10.10.10.109 Server-LAN: app08q01-se 192.168.10.109 Application Nodes PW250-1 Type: PW250 OS: Sun SunOS 5.8 Group: Solaris Client-LAN PW250-1 10.10.10.33 local test1 PW250-1-clt1 10.10.10.34 local test2 PW250-1-clt2 10.10.10.35 Server-LAN PW250-1-se 192.168.10.33 local test1 PW250-1-set1 192.168.10.34 local test2 PW250-1-set2 192.168.10.35 Storage-LAN PW250-1-st 192.168.20.33 local test1 PW250-1-stt1 192.168.20.34 local test2 PW250-1-stt2 192.168.20.35 blade01



Type: BX600 Cabinet ID: 1 Slot/Partition ID: 1 OS: SUSE Linux SLES-9.X86_64 Group: Linux Client-LAN blade01 10.10.10.23 Server-LAN blade01-se 192.168.10.23 Storage-LAN blade01-st 192.168.20.23 blade02 Type: BX600 Cabinet ID: 1 Slot/Partition ID: 2 OS: SUSE Linux SLES-9.X86_64 Group: Linux Client-LAN blade02 10.10.10.24 Server-LAN blade02-se 192.168.10.24 Storage-LAN blade02-st 192.168.20.24 blade03 Type: BX600 Cabinet ID: 1 Slot/Partition ID: 3 OS: SUSE Linux SLES-9.X86_64 Group: Linux Client-LAN blade03 10.10.10.25 Server-LAN blade03-se 192.168.10.25 Storage-LAN blade03-st 192.168.20.25 pw250-2 Type: PW250 OS: Sun SunOS 5.8 Group: Solaris Client-LAN pw250-2 10.10.10.36 local test1 pw250-2-clt1 10.10.10.37 local test2 pw250-2-clt2 10.10.10.38 Server-LAN pw250-2-se 192.168.10.36 local test1 pw250-2-set1 192.168.10.37 local test2 pw250-2-set2 192.168.10.38 Storage-LAN pw250-2-st 192.168.20.36 local test1 pw250-2-stt1 192.168.20.37 local test2 pw250-2-stt2 192.168.20.38 rx801 Type: RX800 OS: SUSE Linux SLES-9.X86_64 Group: Linux Client-LAN rx801 10.10.10.2 Server-LAN rx801-se 192.168.10.2 Storage-LAN rx801-st 192.168.20.2 Control Nodes cn1 Client-LAN cn1-p1 10.10.10.10 Server-LAN cn1-p1-se 192.168.10.10



Storage-LAN cn1-p1-st 192.168.20.10 cn2 Client-LAN cn2-p1 10.10.10.11 Server-LAN cn2-p1-se 192.168.10.11 Storage-LAN cn2-p1-st 192.168.20.11 NAS Nodes fas01-p1 Storage-LAN fas01-p1-st 192.168.20.14

A sample with a reduced output:

cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list --name p1 --list ne,gr Pool configuration details of pool p1 Networks Client-LAN Network: 10.10.10.0 Netmask: 255.255.255.0 VLAN ID: 100 Server-LAN Network: 192.168.10.0 Netmask: 255.255.255.0 VLAN ID: 110 Storage-LAN Network: 192.168.20.0 Netmask: 255.255.255.0 VLAN ID: 120 Def.Router: 192.168.10.254 Pool Groups Linux OS: SUSE Linux SLES-9.X86_64 Solaris OS: Sun SunOS 5.8

8.12.4 Listing all Pools To display an overview of all pools with their used networks, pool groups, SIDs and Con-trol Node and NAS system interfaces, the maintenance tool ff_pool_adm.pl can be used. No arguments except the operation mode have to be defined on the command line.

Synopsis

ff_pool_adm.pl --op list-all [--list <part>[,<part>]]



Options

--op list-all Displays the configuration details of all configured pools.

--list <part>[,<part>] To reduce output to interesting parts use this option. The parameters to this option are the display sections. Add them as a comma separated list. The default sections are: network,storage,group,sid,cn,NAS system. You may also use a two character abbreviation instead the full section name like ne for network.

Output examples

cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list-all Pool configurations p1 Pool Networks Client-LAN Network: 10.10.10.0 Netmask: 255.255.255.0 VLAN ID: 100 Server-LAN Network: 192.168.10.0 Netmask: 255.255.255.0 VLAN ID: 110 Storage-LAN Network: 192.168.20.0 Netmask: 255.255.255.0 VLAN ID: 120 Pool Storage Volumes sapdata fas01-p1-st:/vol/sapdata/p1 saplog fas01-p1-st:/vol/saplog/p1 volFF fas01-p1-st:/vol/volFF/pool-p1 volFF shared fas01-p1-st:/vol/volFF Pool Groups Linux OS: SUSE Linux SLES-9.X86_64 Solaris OS: Sun SunOS 5.8 Pool SIDs D01 SAP Version: SAP-6.20 DB Version: Oracle-9 P01 SAP Version: SAP-6.20 DB Version: Oracle-9 Q01 SAP Version: SAP-6.20 DB Version: Oracle-9 Pool Control Node Interfaces



cn1 Client-LAN cn1-p1 10.10.10.10 Server-LAN cn1-p1-se 192.168.10.10 Storage-LAN cn1-p1-st 192.168.20.10 cn2 Client-LAN cn2-p1 10.10.10.11 Server-LAN cn2-p1-se 192.168.10.11 Storage-LAN cn2-p1-st 192.168.20.11 Pool NAS Node Interfaces fas01-p1 Storage-LAN fas01-p1-st 192.168.20.14

A sample with a reduced output on a single pool configuration:

cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list-all --list sid,group Pool configurations p1 Pool Groups Linux OS: SUSE Linux SLES-9.X86_64 Solaris OS: Sun SunOS 5.8 Pool SIDs D01 SAP Version: SAP-6.20 DB Version: Oracle-9 P01 SAP Version: SAP-6.20 DB Version: Oracle-9 Q01

8.12.5 Changing Pool DNS Domain Changing or adding a pools DNS domain can be done using maintenance tool ff_pool_adm.pl. Some parameters to specify the pool and the domain name have to be given. If only DNS domain server ips should be added or changed the domain name has to be given too.

Synopsis

ff_pool_adm.pl --op dns --name <pool_name>

--dns <dns_domain>[,<dns_server_ip>]



Options

--op dns Adds or changes DNS configuration of a pool.

--name <pool_name> Name of pool the DNS configuration has to be changed. Use ff_pool_adm.pl –op list-all to get a list of currently configured pools (see 8.12.4).

--dns <dns_domain>[,<dns_server_ip>] DNS domain name and servers (servers only with default router) to be used for this pool.

Output example

cn1:~ # ff_pool_adm.pl --op dns --name pool3 --dns pool3.wdf.fsc.net,192.168.200.17,192.168.200.57 DNS domain and servers successfully changed at LDAP and AN images.

8.12.6 Changing Pool Default Router Changes or adds a pools default router. The default router my be nescessary to reach external DNS servers. Within all cases a default router has to have an IP address match-ing one of pools IP networks (in detail the server or client LAN). The default router is used to transfer network traffic between the local IP networks and external IP networks.

Synopsis

ff_pool_adm.pl --op defrouter --name <pool_name> --defrouter <default_router_ip

Options

--op defrouter Adds or changes default router configuration of a pool.

--name <pool_name> Name of pool the default router configuration has to be changed. Use ff_pool_adm.pl –op list-all to get a list of currently configured pools (see 8.12.4).

--defrouter <default_router_ip> The default router to be used for this pool to communicate with other, non local net-works. The IP address has to match to client or server pool network.



Output example

cn1:~ # ff_pool_adm.pl --op defrouter --name pool3 --defrouter 192.168.100.200 Defaultrouter successfully changed at LDAP and AN images.

8.12.7 Adding a Group to a Pool A pool may have more than one group. To add a group to a pool, the maintenance tool /opt/FlexFrame/bin/ff_poolgroup_adm.pl can be used. To associate a group to a pool, some parameters have to be defined on the command line. Groups are used with the FlexFrame Autonomous Agents.

The group is added to a pool in the LDAP database. Since the FlexFrame Autonomous Agents are not able to read pool and group configurations from LDAP, groups have to be configured manually in the FA Agent configurations to take effect.

Synopsis

ff_poolgroup_adm.pl --op add --pool <pool_name>

--group <group_name> --ostype {Linux|SunOS} --osversion <version_string>

--osvendor {Sun|SUSE}

Options

--op add Adds a group to a pool.

--pool <pool_name> Name of pool the group should be added. We recommend using short lowercase names for <pool_name>.

--group <group name> The name of the group to be added.

--ostype {Linux|SunOS} Type of operating system (OS) the systems of this group work with. Currently only Linux and SunOS (Solaris) are valid choices.

--osversion <version_string> The version of the OS. Can be omitted if only one image version (Linux or Solaris) is installed.

--osvendor {Sun|SUSE} The vendor of the OS. Currently only Sun (Solaris) and SUSE (Linux) are supported. Can be omitted if only one image version (Linux or Solaris) is installed.



8.12.8 Removing Pool Group To remove a group of a pool, use the maintenance tool /opt/FlexFrame/bin/ff_ poolgroup_adm.pl.. You have to define the pool and the group name on the command line.

The group is removed from a pool in the LDAP database. Since the FlexFrame Autono-mous Agents are not able to read pool and group configurations from LDAP, groups have to be configured manually in the FA Agent configurations to take effect.

Synopsis

ff_poolgroup_adm.pl --op rem --pool <pool_name> --group <group_name>

Options

--op rem Removes a group from a pool.

--pool <pool_name> Name of the pool the group should be removed from.

--group <group_name> The name of the group to be removed.

8.12.9 Changing Group Assignment of Application Nodes Change the assigned pool group of an Application Node with /opt/FlexFrame/bin/ff_an_adm.pl. Command line arguments are the Application Node name for which the pool group should be changed and the name of the new pool group. The command changes the pool group of Application Node in the LDAP database. The configuration of FA Agents currently has to be changed manually.

Synopsis

ff_an_adm.pl --op group --name <node_name> --group <group_name>

Options

--op group Changes pool group of Application Node.

--name <node_name> Name of the Application Node the pool group has to be changed.

Hosts Database Administrating Application Nodes


--group <group_name> The name of the pool group the Application Node should be assigned to. Use ff_pool_adm.pl --op list-all to get available pool groups (see 8.12.4).

8.12.10 Changing Group and Pool Assignment of Application Nodes

There is currently no maintenance tool to do this. The recommended way is to remove an Application Node and add it with the new pool and group name.

8.13 Hosts Database It may become necessary to have additional entries in the hosts database. Those entries may be required by 3rd-party products installed on customized Application Node images.

The hosts database is stored in LDAP. To maintain the additional host entries use the command ff_hosts.sh.

You cannot remove names or addresses which are essential to the FlexFrame landscape. Each pool has its own hosts database. Therefore you have to maintain each pool individually.

8.13.1 Script: ff_hosts.sh This tool allows the administrator to list, add and delete host names or aliases to the LDAP database. Note: When adding host names with an IP address that matches one of the pool's network segments or the Control LAN segment, the list of IP addresses for that segment gets extended by the IP address of this host name to prevent automatic alloca-tion of the same IP address by other FlexFrame tools.

Only one option of -l, -a or -r can be used at one time.

Synopsis

ff_hosts.sh [-d] -p <pool_name> [{-l|-a <ip> -n <name>|-r <name>}]

Options

-d debug. This option will log debug information which can be found in the file /tmp/ff_hosts.DEBUGLOG

Administrating Application Nodes Hosts Database


-l List all the hosts entries for the pool as provided by option -p.

-p <pool_name> Pool where the host name should be added to.

-a <ip> Add the IP addess <ip>. Has to be used togehter with option -n. If an entry with the IP address <ip> already exists, the name provided will be added as an alias.

-n <name> Host name <name> will be added to the list.

-r <name> Deletes the host name or alias of name. The host name cannot be deleted if it has additional aliases. Remove the aliases first.

Debugging

/tmp/ff_hosts.DEBUGLOG will hold debugging information. In case of problems, please provide this file.

Examples:

The following example will list all additional hosts entries for pool poolname created using this tool:

cn1:~ # ff_hosts.sh -l -p poolname

The following example will add a host newhost with the IP address 1.2.3.4 to the pool poolname:

cn1:~ # ff_hosts.sh -p poolname -a 1.2.3.4 -n newhost

The following example will remove the hosts entry for newhost:.

cn1:~ # ff_hosts.sh -p poolname -r newhost

Rebooting All Application Nodes Administrating Application Nodes


8.14 Rebooting All Application Nodes There may be desaster situations where the Application Nodes remain in an undefined or uncertain state. This may be caused by a loss of the NFS storage system. Once the NFS storage is back online again, the Application Nodes need to be restarted. This can be done using the tool ff_rebot_all_an.sh. Note that this tool will not shut down the systems gracefully. If the command is used during regular operations, it can lead to loss of data.

The purpose of this tool is to support the administrator in a desaster scenario. In case all Application Nodes need to be rebooted, this command can be used to issue the power-off and power-on commands to the Application Nodes. The user has to enter Yes (capital Y) on all questions. Otherwise the program will not continue and will not send any com-mands to the Application Nodes. The program will display the affected Application Nodes as read from the LDAP database. Then, the power-off commands are sent to each Appli-cation Node. After a delay of 5 seconds, the power-on commands are sent to each Appli-cation Node.

Currently only servers of the following type are supported

● PRIMEPOWER 250

● PRIMEPOWER 450

● PRIMERGY BX* series (blades)

Synopsis

ff_reboot_all_an.sh

Debugging

/tmp/ff_reboot_all_an.DEBUGLOG will hold debugging information. In case of prob-lems, please provide this file.


9 User Administration

9.1 Create, Modify, Delete, or List User(s) for App Nodes

Synopsis

/opt/FlexFrame/bin/ff_user_adm.pl

--op add –-user <user_name> --pool <pool_name>

[–-group {<group_name> | <group_id>},...] [–-uid <uid_number>]

[--home <home_directory>] [–-pass <passwd>]

--shell <login_shell> -–gecos <text> [--shadowmin <number_of_days>] [--shadowmax <number_of_days>]

[--shadowwarn <number_of_days>] [--project <project name>]

/opt/FlexFrame/bin/ff_user_adm.pl --op mod –-user <user_name> --pool <pool_name>

[–-pass <passwd>] --shell <login_shell> [-–gecos <text>]

[--shadowmin <number_of_days>] [--shadowmax <number_of_days>] [--shadowwarn <number_of_days>] [--project <project name>]

[--projdel] [--home <home_directory>]

[--group {<primary_group_name> | <primary_group_id>}] [--uid <uid_number>]


--op rem –-user <user_name> --pool <pool_name> [--shell <login_shell>]


--op list –-user <user_name> --pool <pool_name> [--shell <login_shell>]


--op list-all --pool <pool_name>

User Administration Create, Modify, Delete, or List User(s) for App Nodes


Options

--op add –-user <user name> --pool <pool_name> Creates a new user in a given pool.

--op mod –-user <user_name> --pool <pool_name> Modifies a user in a given pool.

--op rem –-user <user_name> --pool <pool_name> Deletes a user in a given pool.

--op list –-user <user_name> --pool <pool_name> Displays information of a user in a given pool.

--op list-all --pool <pool_name> Displays information of all users in a given pool.

--group <group_name> | <group_id>,... A comma separated list of group names/ids belonging to the user. The first group is taken as primary group. Default: gid number 1 (=other).

--group <primary_group_name> | <primary_group_id> The affiliation to the primary group can be changed with this option.

--uid <uid_number> A certain uid number, if desired. Default: The first free number >= 100. In connection with the mod option the existence of the uid is not checked.

--home <home_directory> Home directory of the user. If it doesn’t exist yet, it will be created. Default: /home/<user_name>. In connection with the mod option the existence of the home directory is not checked.

The home directory is created on the base /vol/volFF/pool-<pool_name>/pooldata.

--pass <passwd>

Login password for the user. Default: password.

--shell <login_shell> Login shell of the user. Default: /bin/csh. In connection with the mod option the existence of the login shell is not checked.

--gecos <text> Some comment used for the user’s gecos attribute. Default: Normal user. This replaces the previous used option comment.

--shadowmin <number_of_days> Min. number of days until the password can be changed.

Create, Modify, Delete, or List Project(s) for Solaris 10 App Nodes User Administration


--shadowmax <number_of_days> Max. number of days until the password have to be changed.

--shadowwarn <number_of_days> Number of days you get a warning message before the password expires.

--project <project_name> Name of Solaris 10 project assigned to <user>.

--projdel Requests deletion of project reference in user entry (only available with –op mod). Using this option requires also option —project. The script tries to remove the specified user from the corresponding project entry, too. If it fails, please use ff_project_adm.pl to remove it manually.

9.2 Create, Modify, Delete, or List Project(s) for So-laris 10 App Nodes

The setting of the operating system parameters changed with Solaris 10. Instead of set-tings made in /etc/system so-called Resource Controls are administered in project specifications. For further information please have a look into corresponding sites at SUN or get an overview reading SAP note 724713 (https://service.sap.com/sap/support/notes/724713) ).

With FlexFrame the project settings are assigned to a specific user. So the usage of projects mean you can find the project definitions in a pool specific leaf of the LDAP da-tabase and an corresponding reference within the user entry.

To keep the information consistent you use ff_project_adm.pl and ff_user_adm.pl. The system administrator is responsible to keep this consistence.

The script tries to interact with ff_user_adm.pl as far as possible if you create, modify or delete a project entry. If the script itself cannot prepare the corresponding user entry please use ff_user_adm.pl to keep the project specifications consistent manually.

Synopsis

/opt/FlexFrame/bin/ff_project_adm.pl --op add --pool <pool_name>

–-project <project_name> –-pid <project_identifier>

{--attr ‘<attr=value:...>’ | --attrfile <file_name>} [--users <user_name,...>]

https://service.sap.com/sap/support/notes/724713)

User Administration Create, Modify, Delete, or List Project(s) for Solaris 10 App Nodes


/opt/FlexFrame/bin/ff_project_adm.pl

--op mod --pool <pool_name>

{–-project <project_name> | –-pid <project_identifier>} [--attr ‘<attr=value:...>’ | --attrfile <file_name>]

[--users <user_name,...>]

/opt/FlexFrame/bin/ff_project_adm.pl --op rem --pool <pool_name>

{–-project <project_name> | –-pid <project_identifier>}

[--attr ‘<attr=value:...>’] [--users <user_name,...>]


--op list --pool <pool_name>

{–-project <project_name> | –-pid <project_identifier>}


--op list-all --pool <pool_name>

Options

--op add Creates a new project in a given pool. The project reference is automatically written into the user entries.

--op mod Modifies a project specification in a given pool.

--op rem Deletes a project specification in a given pool. The project reference is automatically removed from the user entries.

--op list Displays information of a project entry.

--op list-all Displays information of all projects in a given pool.

--pool <pool_name> Name of the pool the group should belongs to.

--project <project_name> Name of the project.

--pid <project_identifier> Numeric value for the project identifier.

Create, Modify, Delete, or List Project(s) for Solaris 10 App Nodes User Administration


--attr ‘<attr=value:...>’ A colon separated list (bordered by quotation) of resource controls. No validation takes place. In case of mod you can change attribute values in an existing project specification. In case of rem you can delete specific attributes from the project specification.

--attrfile <file_name> A file containing the specifications of resource controls. Each attribute has to be writ-ten in a separate line. No validation takes place. Alternative instead of using --attr.

--users <user_name,...> A comma separated list of users which references this project. In case of add the project reference is written to the corresponding user entry. In case of mod you can add users to the project member list.The project reference is written into the user entries. Existing references in the user entries are overwritten. In case of rem you can delete specific users from the project specification.The refer-ence to the project in the corresponding user entries are also removed.


10 SAN Configuration in a FlexFrame Environment

10.1 Setting Up the SAN Configuration The SAN configuration for a FlexFrame landscape needs careful planning and design. A team of specialists for storage systems, servers and SAN infrastructure must work out a concept that will be a guideline for the complete setup.

Furthermore, it is required to consult the actual recommendations from the manufacturer of the used storage and SAN components and from the involved partners as well as the MatrixEP from Fuitsu Siemens Computers. These documents can contain important in-formation that must be taken into account when planning the configuration and when configuring the individual components.

10.1.1 Configuring Storage This section does not intend to give a detailed description of how to configure SAN stor-age for usage in a FlexFrame environment. This task will usually be done by a storage systems expert in cooperation with a member of the FlexFrame implementation team.

Furthermore, the SAN storage system is not necessarily an integral part of the FlexFrame system; you can use LUNs from a storage system which is also used for other purposes in your computing environment. Therefore, the LUN creation and administration is a task outside the scope of FlexFrame.

Before starting with this task, a detailed storage concept should be worked out.

10.1.1.1 General Remark for the Use of Navisphere

When using a CLARiiON storage subsystem within a FlexFrame environment (SAN- or NAS-attached) and the Control Node is used for administrating and/or configuring the CLARiiON via Navisphere Web GUI it may happen that caused by a switchover of an Application Node the Servlet Container tomcat is started and requires the port 8080 used by the Navisphere client. In that case the java_vm and the web browser, e.g. the Nav-isphere client are terminated. To avoid this situation use a non-FlexFrame server for the Navisphere client.

SAN Configuration in a FlexFrame Environment Setting Up the SAN Configuration


10.1.1.2 Storage System Access

The FlexFrame software itself does not need an administrative access to the SAN stor-age system. However, if an administrative access to the storage system from inside the FlexFrame environment is desired, you can add it either via the Planning Tool (External Connectivity), or later with the ff_swport_adm.pl tool.

A special case is when the SAN storage system is used for NAS access too. If it is a NetApp Filer, the administrative access for SAN administration is the same as the access used for NAS administrative tasks and will be usually created as a part of the initial setup via the Planning Tool (Controlling, NAS details). In case of a Celerra gateway that uses a CLARiiON/FibreCAT as backend, the Celerra control station needs access to the Storage Processors (SPs) of the CLARiiON/FibreCAT; as the Celerra control station is in the FlexFrame control LAN, the SPs must be connected to the control LAN too. For details see “Configuration of NAS Systems” in the manual “FlexFrame™ for SAP® – Installation of a FlexFrame Environment”.

Be aware that to be able to use the SAN functionality of the NetApp Filer, you have to license the fcp service (license add <license_code> - may require a reboot, depending on filer model), start the fcp service (fcp start) and bring up the target HBA(s) (fcp config <adapter> up). If you are using a clustered NetApp Filer, it is recommended to use a cfmode setting of single_image.

10.1.1.3 LUN Creation

For usage by the FlexFrame SAP databases, several LUNs must be created on the stor-age system.

The term LUN refers here to a storage device exported from a storage system that can be seen as a disk device by a host. It may be a LUN from a FibreCAT CX or NetApp system or a Symmetrix device.

The number and size of LUNs depends on the number and type of the SAP systems to be installed on SAN storage, as well as on the fact if a host based volume management software will be used. Based on these LUNs, all file systems needed for the database data and log files will be created.

A host based volume management software might be used for different reasons, such as more administrative flexibility, or the ability to create volumes with mirrors on different storage systems (host-mirroring). Supported volume managers are the native volume managers from Linux (LVM2) and Solaris (SVM) as well as Veritas Volume Manager (VxVM) on Solaris. For FlexFrame specific details on creating volumes and file systems see section 10.1.4.

It have to be ensured that a LUN is used by only one SAP system. Never use a partition/slice of a LUN for one SAP system and another parti-tion/slice of the same LUN for another system, nor use a volume group for more than one SAP system.

Setting Up the SAN Configuration SAN Configuration in a FlexFrame Environment


When using a host based volume management software, at least two LUNs -preferably more - should be used for each SAP system, one for sapdata vol-umes and one for saplog volumes. Only in test environments with low require-ments concerning database performance, you can place saplog and sapdata volumes on the same LUN.

A similar recommendation also applies to the usage of partition/slices of a LUN in cases where no host based volume management software is used.

For productive systems, you should also place the LUNs for sapdata and saplog volumes on physically separated disks, e.g. by using different RAID groups of a FibreCAT CX, or different aggregates of a NetApp system.

Example:

Creation of two LUNs on a NetApp system for usage with Solaris, without special per-formance considerations. Be aware that if performance is an issue, it may be advisable to create the LUNs for sapdata in a separate volume that is also located in a separate ag-gregate.

> vol create SAN aggr0 120g Creation of volume 'SAN' with size 120g on containing aggregate 'aggr0' has completed. > vol options SAN nosnap on > qtree create /vol/SAN/Sol > lun create -s 50g -t solaris /vol/SAN/Sol/dataSA1 > lun create -s 7g -t solaris /vol/SAN/Sol/logsSA1 > lun show /vol/SAN/Sol/dataSA1 50g (53687091200) (r/w, online) /vol/SAN/Sol/logsSA1 7g (7516192768) (r/w, online) > > lun show -v /vol/SAN/Sol/dataSA1 /vol/SAN/Sol/dataSA1 50g (53687091200) (r/w, online) Serial#: hpDnSJ8y2Uj0 Share: none Space Reservation: enabled Multiprotocol Type: solaris > lun show -v /vol/SAN/Sol/logsSA1 /vol/SAN/Sol/logsSA1 7g (7516192768) (r/w, online) Serial#: hpDnSJ8y1eiR Share: none Space Reservation: enabled Multiprotocol Type: solaris

For more information on how to create RAID groups, aggregates, volumes, qtrees and LUNs on a NetApp storage system, please refer to the documentation from http://now.netapp.com.



To create a LUN on a FibreCAT CX, you may use the Bind LUN command of the Nav-isphere Manager GUI as shown in the following example:

For more information on how to manage storage and create LUNs on EMC storage sys-tems, please refer to the documentation from http://powerlink.emc.com.

10.1.1.4 Recording LUN Information

For each LUN to be used in a FlexFrame pool for the database of a SAP system, you can create a VLUN section in the SAN configuration file as described in section 10.2.2.

For a NetApp LUN, use the LUN path as arrayLUN and determine the LUNGUID from the LUN’s Serial# by means of the tool ff_san_info.sh on a FlexFrame Node.

For a FibreCAT CX, use the LUN ID as arrayLUN and the LUN’s Unique ID as LUNGUID. The following example shows how to find these informations using the Navisphere Man-ager GUI:



The hostLUN value will be determined and added later, when mapping the LUNs to hosts (see section 10.1.3.4).

10.1.2 Configuring Application Nodes All Application Nodes that will run database instances of SAP systems which are to be configured for SAN usage must be equipped with Host Bus Adapters (HBAs). To ensure that the correct type of Fibre Channel HBA is used, please consult the FlexFrame Sup-port Matrix. The supported types include Qlogic and Emulex HBAs for Linux nodes and Emulex HBAs for Solaris nodes.

Starting with version 4.0, the Application Node Images for Solaris 10 as well as for SLES 9 and SLES 10 already contain the driver software required for these HBAs.

For Solaris 9 Application Nodes, the required driver software (SFS driver, also called Leadville or emlxs) is not included in the Image creation procedure. It must be downloaded from the Sun website and installed using the Solaris image maintenance cycle.

To run the cycle, carefully read and follow the instructions in the section



Solaris Image Maintenance Cycle on page 165. Be sure to select one of the Application Nodes that is equipped with Fibre Channel HBAs. The installation itself is done during step 8 of this cycle.

For all types of Application Node images, depending on the used combination of OS/HBA/storage system, specific driver settings may be needed. These must be also set using the image maintenance cycle. For details see section 10.1.2.1.

For performance reasons as well as to protect against hardware failure, it is strongly recommended to use more then one HBA or a at least a dual ported HBA on each Appli-cation Node. To manage the multiple pathes between host and storage system, a multi-path software is needed.

The SLES 9 and SLES 10 Application Node Images already contain a ready-to-use installation of the Linux native multipath software DM-MPIO.

For Solaris 9 and 10, multipathing is provided as an additional functionality of the SFS driver; this driver is contained in Solaris 10 and must be installed for Solaris 9, as de-scribed in section 10.1.2.1. If you decide to use this multipathing option (MPxIO, also called Sun StorEdge Traffic Manager, STMS), follow the steps described in section 10.1.2.2. On Solaris Application Nodes using EMC storage, you can alternatively use EMC Power-Path. This software must also be installed using the Solaris image maintenance cycle (see above). For details on the installation of PowerPath see 10.1.2.3.

As described in section 10.1.1, Veritas Volume Manager (VxVM) can be used as a host based volume management software on Solaris hosts. This software also includes its own multipathing component (DMP) that can be used as an alternative to MPxIO or PowerPath, if VxVM is used for volume management. Veritas software must also be installed using the Solaris image maintenance cycle (see above). For details on the installation of Veritas software see section 10.1.2.4.

After you have installed and tested all software components needed in your Solaris im-age, complete the final steps of the Solaris image maintenance cycle. On the end of the cycle, all Application Nodes of the pool group must run on the same image.

Be aware that the DB-Instance of a SAP system configured for SAN usage will be able to run only on Application Nodes with the correct image. Therefore spe-cial care must be taken not to start this instance on an Application Node outside the pool group (of course it is assumed that all nodes run on the same image, as generally required by a correct FlexFrame installation).



10.1.2.1 Installing the SFS driver for Solaris 9 Application nodes

To get the SFS software, go to http://www.sun.com/download -> Downloads A-Z and search for StorageTek SAN 4.4 or StorEdge SAN 4.4. You need a Sun Online Account for this download. If you aren’t a registered user, the web page offers you the possibility to register. Download the Install_it Script SAN 4.4.9 (package SAN_4.4.9_install_it.tar.Z) and put it on a place where it can be accessed from the Application Node that will be used to run the Solaris image maintenance cycle.

On the download page you will also find a link to a detailed documentation of the SFS software.

For the next steps, carefully read and follow the instructions in the section Solaris Image Maintenance Cycle on page 165.

The installation itself is done during step 8 of the Solaris image maintenance cycle.

Navigate to the directory where you have put the package SAN_4.4.9_install_it.tar.Z. Then uncompress it and extract the files from the archive. Finally call the install_it script and reboot your system as requested by the install_it script.

To check that the driver is correctly installed, search for its messages in the system logfile /var/adm/messages.

Example:

Get the driver messages on a Solaris 9 system with an Emulex LP10000DC Fibre Chan-nel HBA. Note the Driver attach messages, which report the driver name and version, the HBA type and firmware version as well as the WWNN and WWPN of the HBA ports.

# grep emlxs: /var/adm/messages Jun 29 11:27:46 tom2pw4 emlxs: [ID 349649 kern.info] [B.125A]emlxs0: NO-TICE: 100: Driver attach. (Emulex-S s9-64 sparc v1.10g (2006.02.15.12.43)) Jun 29 11:27:46 tom2pw4 emlxs: [ID 349649 kern.info] [B.1260]emlxs0: NO-TICE: 100: Driver attach. (LP10000DC Dev_id: fa00 Firmware: 1.91a1 Fcode: none Boot: none) Jun 29 11:27:46 tom2pw4 emlxs: [ID 349649 kern.info] [B.1266]emlxs0: NO-TICE: 100: Driver attach. (WWPN:10000000C94942D2 WWNN:20000000C94942D2) Jun 29 11:27:50 tom2pw4 emlxs: [ID 349649 kern.info] [B.125A]emlxs1: NO-TICE: 100: Driver attach. (Emulex-S s9-64 sparc v1.10g (2006.02.15.12.43)) Jun 29 11:27:50 tom2pw4 emlxs: [ID 349649 kern.info] [B.1260]emlxs1: NO-TICE: 100: Driver attach. (LP10000DC Dev_id: fa00 Firmware: 1.91a1 Fcode: none Boot: none) Jun 29 11:27:50 tom2pw4 emlxs: [ID 349649 kern.info] [B.1266]emlxs1: NO-TICE: 100: Driver attach. (WWPN:10000000C94942D1 WWNN:20000000C94942D1)

http://www.sun.com/download



If you plan to install further SAN software, such as EMC PowerPath or Veritas Storage Foundation, you may decide to continue with the installation of these products before you proceed with step 9 of the Solaris image maintenance cycle.

10.1.2.2 Enabling Solaris Native Multipathing

Solaris native multipathing (MPxIO) is an optional feature of the Solaris SFS driver, that have to be enabled separately. If you decide to use this multipath option, you have to enable it in the image used by the Application Nodes of the pool group.

For Solaris 9, this can be done as a part of the maintenance cycle for the driver installa-tion on the maintenance Application Node.

1. Open the /kernel/drv/scsi_vhci.conf file in a text editor

2. To enable MPxIO, change the value of mpxio-disable to no. By default, the value of mpxio-disable is yes.

3. To enable automatic failback, change the value of auto-failback to enable.

4. For symmetric storage devices, an entry like this have to be created:

device-type-scsi-options-list ="EMC SYMMETRIX", "symmetric-option"; symmetric-option = 0x1000000;

Please note that the first part of the strings in the list is a vendor id that has to be specified with exactly 8 characters, so it must be filled up with blanks as necessary.

5. Save your changes

For Solaris 10, you have to modify two files:

– In the /kernel/drv/fp.conf file change the value of mpxio-disable to no.

– In the /kernel/drv/scsi_vhci.conf file, create an entry as specified at point 4 above.

These changes must also be done using the maintenance cycle for Solaris Application Nodes.

After changing this configuration files, a reconfiguration reboot is needed.

# touch /reconfigure # shutdown -g0 -y -i6



10.1.2.3 Installing EMC PowerPath

PowerPath software can be used as an multipathing option for EMC storage systems on Solaris Application Nodes. It is not an integral part of FlexFrame and must be purchased and licensed separately. Check the FlexFrame Support Matrix for the correct version supported with FlexFrame.

Please note that EMC PowerPath is supported for Solaris Application Nodes only.

The installation has to be done using the maintenance cycle for Solaris images as de-scribed above. For the installation itself follow the steps described in the EMC documen-tation.

Note that if you have purchased host specific PowerPath registration keys, you must enter a separate key on each Application Node of the pool group.

10.1.2.4 Installing Veritas Software

Veritas Volume Manager (VxVM) and Veritas File System (VxFS) software can be in-stalled on Solaris Application Nodes. It also have to be purchased and licensed sepa-rately. This software is part of the Veritas Storage Foundation. Currently FlexFrame sup-ports the version 4.1 with MP1.

Please note that Veritas Volume Manager (VxVM) and Veritas File System (VxFS) is supported for Solaris Application Nodes only.

The installation has to be done using the maintenance cycle for Solaris images as de-scribed above.

The installation procedure of the product cannot be executed directly, as the product installation scripts from the VERITAS CD fail if the system has no space on locally mounted file system.

Therefore, after copying the CD contents to your software, locate the file CPI41SOL.pm in the subdirectory scripts/install of the product you wish to install; for VxVM it is volume_manager/scripts/install, for VxFS it is file_system/scripts/install. Make the following change in each of these files:

$CMD{DFK}="/usr/bin/df -kl";

-> $CMD{DFK}="/usr/bin/df -k";

To install VxVM, run the installation script installvm from the volume_manager direc-tory, where you have changed the scripts/install/CPI41SOL.pm file.

To install VxFS, run the installation script installfs from the file_system directory, where you have changed the scripts/install/CPI41SOL.pm file.



For the installation of the maintenance pack MP1, a similar procedure have to be fol-lowed.

This workaround has been discussed with and approved by Symantec and may no longer be needed in a newer version of the software.

If using the DMP component of VxVM with a CLARiiON/FibreCAT array, you must also install a so-called APM and ASL. This packages are contained in a tar archive that can be downloaded from the Symantec web page at http://support.veritas.com/docs/275736 . A similar package for NetApp storage is located at http://support.veritas.com/docs/281855, that should be installed if using a NetApp filer as a SAN device with VxVM. A detailed description of how to install these packages can be found if you follow the given links.

10.1.3 Connecting the Storage to the Application Nodes Before connecting your Application Nodes to the SAN Storage System, please ensure that your configuration including server, HBA, Fibre Channel switch and storage system is supported according to the FlexFrame Support Matrix. Also check that all used compo-nents are correctly configured according to the actual recommendations from the respec-tive manufacturer or partner and from the MatrixEP.

Fibre Channel switches used to connect the Application Nodes with the Storage Systems are not considered an integral part of the FlexFrame landscape, and no automatic con-figuration of these switches is done. For more information on setup and configuration of the Fibre Channel switches, refer to the documentation of your switch. For high availabil-ity, a redundant fabric should be set up, so use two separate fabrics if possible.

It is not necessary to connect the LAN ports of the Fibre Channel switches to one of the FlexFrame LANs. However, if you wish to administrate the switches from the FlexFrame Control Node, you can create connections to the control LAN either via the Planning Tool (External Connectivity), or later with the ff_swport_adm.pl command.

It is also supposed that the storage system is already connected to the Fibre Channel switches. If not yet done, follow the guidelines of your storage system’s documentation.

To connect an Application Node to the storage system and the LUNs created on the system for usage by the SAP systems that will run on this Application Node, several preparations have to be done.

You should go through this procedure for one Application Node first, then create volumes and file systems on this Application Node as described in section 0. Finally repeat it for all other nodes. Optionally, some steps can be done in parallel for all nodes (e.g. create zones for all Application Nodes at the same time).

During the preparation of an Application Node it must be ensured the FlexFrame Autonomous Agents are deactivated for this node.

http://support.veritas.com/docs/275736

http://support.veritas.com/docs/281855



1. Connect the Fibre Channel ports of the Application Node to the Fibre Channel switches; for redundancy purposes, connect one port to one switch and the other port to the other switch.

2. Create zones on the Fibre Channel switch, so that the Application Node can see the storage system(s). For some hints see section 10.1.3.1.

3. Check the visibility of the storage system on the Application Node. An example for this action is shown in section 10.1.3.2.

4. Check the visibility of the Application Node’s HBAs (host initiators) on the storage system. On a FibreCAT CX, register the connection of the host initiators (see section 10.1.3.3).

5. On the storage system, implement the actions required to assign the LUNs that will be accessed by this Application Node to this node (LUN masking/mapping). Some details on how this can be done are shown in section 10.1.3.4.

6. Check the visibility of the LUNs on the Application Node (see section 0).

10.1.3.1 Creating Zones on the Fibre Channel Switches

Zones are created on Fibre Channel switches to protect servers and storage systems. There is no specific requirement how zoning is to be done for a FlexFrame environment; you may use hard or soft zoning, port based or WWN-based.

We recommend to use WWN-based zoning, and create zones that contain two WWPNs: one from a host and one from the storage system.

If you want to configure Brocade Fibre Channel switches, you can use the tool ff_fc_gen_sw.sh on the FlexFrame Control Node to provide you with some assis-tance. Consult the man page of this tool for details.

To get the WWPNs of the Application Nodes, you can use the command ff_san_info.sh –i on each Application Node.

Example on a Solaris Application Node:

# ff_san_info.sh -i HBA WWNN WWPN fp0 20:00:00:00:c9:49:41:5c 10:00:00:00:c9:49:41:5c fp1 20:00:00:00:c9:49:41:5b 10:00:00:00:c9:49:41:5b

Example on a Linux Application Node:

# ff_san_info.sh -i HBA WWNN WWPN host1 20:00:00:c0:9f:c6:45:22 21:00:00:c0:9f:c6:45:22 host2 20:00:00:c0:9f:c6:45:23 21:00:00:c0:9f:c6:45:23



For a NetApp system, the WWPNs can be found as shown in the following example:

> fcp config 0c: ONLINE <ADAPTER UP> Loop Fabric host address 280000 portname 50:0a:09:81:86:28:13:0c nodename 50:0a:09:80:86:28:13:0c mediatype ptp speed 2Gb 0d: ONLINE <ADAPTER UP> PTP Fabric host address 280400 portname 50:0a:09:82:86:28:13:0c nodename 50:0a:09:80:86:28:13:0c mediatype ptp speed 2Gb

You can find the WWNs of a FibreCAT CX in the Navisphere Manager GUI as shown in the following screenshot:

10.1.3.2 Checking Visibility of the Storage System on the Application Node

If the correct zones have been created on the Fibre Channel switch, the Application Node should be able to see the storage system. As no LUNs have been mapped yet, the storage system will present a pseudo LUN 0 to the host. You can verify now that the correct number of pathes between storage system and Application Node is available.



Example on a Solaris Application Node Connected with a CLARAiiON/FibreCAT CX

Get the host ports of the fabric-connected fibre channel HBAs:

# cfgadm -l | grep fc-fabric c2 fc-fabric connected unconfigured unknown c3 fc-fabric connected unconfigured unknown

Configure each port, if not yet done:

# cfgadm -c configure c2 # cfgadm -c configure c3

The output of the luxadm probe command should now show a node with the storage system’s WWN and a list that contains each path from the server to the storage system:

# luxadm probe No Network Array enclosures found in /dev/es Found Fibre Channel device(s): Node WWN:50060160b0601f31 Device Type:Disk device Logical Path:/dev/rdsk/c2t5006016030601F31d0s2 Logical Path:/dev/rdsk/c2t5006016830601F31d0s2 Logical Path:/dev/rdsk/c3t5006016930601F31d0s2 Logical Path:/dev/rdsk/c3t5006016130601F31d0s2

10.1.3.3 Registering Host Initiators with a CLARiiON/FibreCAT CX

Use Navisphere Manager’s Connectivity Status window for the storage system to register each HBA connection with the storage system:

● In the Connectivity Status window, select the connection for the WWN of the HBA, click Register to open the Register Initiator Record window.

● Select the following:

- Initiator Type of CLARiiON Open - ArrayCommPath - Unit Serial Number of Array - Failover Mode of 1

● In the Host Information box, select New Host and enter the Application Node’s host name and IP address for the first HBA connection of the host. For all other con-nections of the host, select Existing Host and choose the appropriate host name in the drop-down box.

● Click OK.



After a few seconds the registration status of the HBA connection should change from No to Yes.

If you have a host with the Navisphere CLI package installed and LAN access to the SPs, you can alternatively use the following command to register the HBA connections of an Application Node as shown in the following example:

# /opt/Navisphere/bin/navicli -h 10.10.13.51 storagegroup -setpath -hbauid 20:00:00:00:c9:49:42:d2:10:00:00:00:c9:49:42:d2 -sp a -spport 0 -failovermode 1 -arraycommpath 1 -o -host tompw4 # /opt/Navisphere/bin/navicli -h 10.10.13.51 storagegroup -setpath

-hbauid 20:00:00:00:c9:49:42:d1:10:00:00:00:c9:49:42:d1 -sp a

-spport 1 -failovermode 1 -arraycommpath 1 -o -host tompw4

# /opt/Navisphere/bin/navicli -h 10.10.13.51 storagegroup -setpath

-hbauid 20:00:00:00:c9:49:42:d2:10:00:00:00:c9:49:42:d2 -sp b


# /opt/Navisphere/bin/navicli -h 10.10.13.51 storagegroup -setpath

-hbauid 20:00:00:00:c9:49:42:d1:10:00:00:00:c9:49:42:d1 -sp b


Note that the hbauid above is a concatenation of the WWNN and the WWPN of the Application Node’s HBA.

10.1.3.4 Mapping LUNs to the Application Nodes

For a FibreCAT CX, this means that you have to connect the Application Node to the storage group that contains all LUNs that will be used by the Application Node’s pool group. If the storage group does not exist yet, create it and add the LUNs to the storage group. When adding a LUN to a storage group, a Host LUN Id is automatically assigned. For ease of administration, you may decide to assign this number by yourself and choose the same number as the LUN ID. The following example shows how you can do this:

You may also decide to use storage groups that contain only one Application Node, but in this case you have to take special care to add the same LUNs to all storage groups that



contain hosts from the FlexFrame pool group, and that the LUNs also have the same host LUN number in all groups. Therefore we do not recommend this scheme.

Example on a NetApp host:

> igroup add igroupsol 10000000c949415c 10000000c949415b > lun map /vol/SAN/Sol/dataSA1 igroupsol lun map: auto-assigned igroupsol=0 > lun map /vol/SAN/Sol/logsSA1 igroupsol lun map: auto-assigned igroupsol=1

The number reported behind the igroup name is the hostLUN to be used in the VLUN section of the SAN configuration file described in section 10.2.2.

If you decide to assign this number by yourself, you can add a Lun ID parameter to the lun map command.

You can display all existing LUN mappings on a NetApp storage system as shown in the following example:

> lun show -m LUN path Mapped to LUN ID Protocol ------------------------------------------------------------------ /vol/SAN/Sol/dataSA1 igroupsol 0 FCP /vol/SAN/Sol/logsSA1 igroupsol 1 FCP

However, if you decide to assign the host LUN ID by yourself, it is advisable to arrange for a host LUN ID of 0 for each storage group or igroup, to avoid some known problems.

10.1.3.5 Checking Visibility of the LUNs on the Application Node

Example on a Solaris Application Node with MPxIO enabled

To make the LUNs visible on the Application Node, you have to do either a reconfigura-tion reboot of the Node, or to enter the command cfgadm –al.

# cfgadm -al Ap_Id Type Receptacle Occupant Condition c1 scsi-bus connected configured unknown c1::dsk/c1t0d0 CD-ROM connected configured unknown c2 fc-fabric connected configured unknown c2::5006016030601f31 disk connected configured unknown c2::5006016830601f31 disk connected configured unknown c3 fc-fabric connected configured unknown c3::5006016130601f31 disk connected configured unknown c3::5006016930601f31 disk connected configured unknown



# luxadm probe No Network Array enclosures found in /dev/es Found Fibre Channel device(s): Node WWN:50060160b0601f31 Device Type:Disk device Logical Path:/dev/rdsk/c4t600601603E851800C65FF7869F39DB11d0s2 Node WWN:50060160b0601f31 Device Type:Disk device Logical Path:/dev/rdsk/c4t600601603E8518003749F962FEFADA11d0s2 Node WWN:50060160b0601f31 Device Type:Disk device

Note that the logical path contains the LUNGUID as noted when creating the LUN (see last paragraph of section 10.1.1).

Example on a Linux SLES 9 Application Node

Make the LUNs visible by doing a reboot. Optionally you can use the ff_qlascan.sh script for a host using QLA 2xxx HBA, followed by the command multipath –v2. The visibility can be checked at various levels, starting from driver, SCSI layer up to multipath layer.

# cat /proc/scsi/qla2xxx/1 QLogic PCI to Fibre Channel Host Adapter for FCI/O-CARD2Gb/s: Firmware version 3.03.18 IPX, Driver version 8.01.02-sles [...] SCSI LUN Information: (Id:Lun) * - indicates lun is not registered with the OS. ( 0: 0): Total reqs 26, Pending reqs 0, flags 0x0, 0:0:81 00 ( 0: 1): Total reqs 27, Pending reqs 0, flags 0x0, 0:0:81 00 [...] # multipath -ll [...] 3600601603e851800917bab561d38db11 [size=3 GB][features="1 queue_if_no_path"][hwhandler="1 emc"] \_ round-robin 0 [prio=2][active] \_ 1:0:2:9 sdad 65:208 [active][ready] \_ 2:0:1:9 sdbn 68:16 [active][ready] \_ round-robin 0 [enabled] \_ 1:0:1:9 sdl 8:176 [active][ready] \_ 2:0:0:9 sdav 66:240 [active][ready] 3600601603e8518003a9fa75d1d38db11 [size=3 GB][features="1 queue_if_no_path"][hwhandler="1 emc"] \_ round-robin 0 [prio=2][active]



\_ 1:0:2:6 sdaa 65:160 [active][ready] \_ 2:0:1:6 sdbk 67:224 [active][ready] \_ round-robin 0 [enabled] \_ 1:0:1:6 sdi 8:128 [active][ready] \_ 2:0:0:6 sdas 66:192 [active][ready] [...]

10.1.4 Creating Volumes and Filesystems for a SAP System If you have decided to place the sapdata and saplog data of a SAP System on SAN stor-age, you have to create file systems on the LUNs that have been created and made ac-cessible to an Application Node.

Usually, these file systems are not created directly on the LUNs, but on volumes created on top of these LUNs with the help of a host based volume management software (the native volume managers of Linux/Solaris and Veritas VxVM on Solaris are supported only), or on partitions/slices of the LUNs. For more information on how to create volume groups, add physical volumes to the volume group and create logical volumes inside a volume group, refer to the documentation of the used volume management software.

The type of the used volume manager, the names of the volume groups as well as the used volumes have to be documented in the SAN configuration file as described in sec-tion 10.2.2. Here you can also add some options to be used when importing or deporting a volume group (e.g. forceimport; waitforce=5). Note that each volume name is recorded as file SystemMountpointSource in the MOUNTPATH section for the file system that is created on top of this volume. Keep in mind that a volume group have to be exclu-sively used for the database of one SAP system only. For some special considerations on creating a volume group (also called a disk set) with Solaris native Volume Manager for usage by FlexFrame see section 10.1.4.1. For some notes that must be taken into account when creating a volume group with Linux native Volume Manager (LVM2) for usage by FlexFrame see section 10.1.4.2.

The number and size of the needed file systems depends on the type of the SAP System to be installed and is determined as a part of the detailed storage concept that has been defined in advance. As an example for an Oracle based installation you may need 6 sap-data file systems - sapdata1 to sapdata6 and 8 saplog file systems: saparch, oraarch, sapbackup, sapreorg, origlogA, origlogB, mirrlogA, mirrlogB. Depending on the storage concept, it is also possible to define a lower or higher number of file systems. For more information refer to the appropriate SAP Installation Guide. For each file system that will be used by a SAP system, you will have to add a MOUNTPATH entry in the SAN configuration file as described in section 10.2.2. Consult the SAP notes applicable for the database and file system type to get the correct attributes to be used for the creation of the file systems and the recommended mount options.

If you have entered the description of the volume groups and file systems for a SAP sys-tem ID (SID) in your SAN configuration file, it is time to add these information to the Flex-



Frame database (LDAP) and test their usability. The needed steps are detailed in section 10.1.4.3.

10.1.4.1 Creating a Solaris VM Volume Group for FlexFrame Usage

If you decide to use Solaris native Volume Manager to manage the SAN volumes for a SAP database, you have to make some additional preparations.

If a volume group (called disk set in Solaris VM terms) is to be used by a SAP database instance, all Application Nodes of the pool group have to be connected as hosts to this disk set. To be able to do this, the rpc.metad daemons of the involved hosts must be able to communicate with each other. Therefore, you need to edit the .rhosts file on each Application Node of the pool group to allow access for all hosts of the group. Then you can connect the hosts to the disk set, as shown in the following example, where we create two disk sets for a pool group that contains the Application Nodes pw1 and pw2:

root@pw1# metaset -s saplog_vi2 -a -h pw1 root@pw1# metaset -s sapdata_vi2 -a -h pw1 root@pw1# luxadm probe No Network Array enclosures found in /dev/es Found Fibre Channel device(s): Node WWN:500a0980833103fb Device Type:Disk device Logical Path:/dev/rdsk/c4t60A9800056716D70426F38414C503476d0s2 Node WWN:500a0980833103fb Device Type:Disk device Logical Path:/dev/rdsk/c4t60A9800056716D70426F38414C524175d0s2 root@pw1# metaset -s saplog_vi2 -a c4t60A9800056716D70426F38414C524175d0 root@pw1# metaset -s sapdata_vi2 -a c4t60A9800056716D70426F38414C503476d0 root@pw1# metaset -s saplog_vi2 -a -h pw2 root@pw1# metaset -s sapdata_vi2 -a -h pw2 root@pw1# metaset Set name = saplog_vi2, Set number = 1 Host Owner pw1 Yes pw2 Drive Dbase /dev/dsk/c4t60A9800056716D70426F38414C524175d0 Yes Set name = sapdata_vi2, Set number = 2 Host Owner pw1 Yes



pw2 Drive Dbase /dev/dsk/c4t60A9800056716D70426F38414C503476d0 Yes

During the lifecycle of this SID, the preparation made as described here must be adjusted on several occasions:

1. If adding a new Application Node to the pool group, adjust the .rhosts file on all Nodes of the group. Then connect the new host to the disk set by entering the ap-propriate metaset command on the host that currently owns the disk set.

2. When creating a new customized Boot Image for an Application Node, the prepara-tions for this host must be reapplied.

3. After upgrading all Application Nodes of the pool group to a new Image, the disk set

is no longer known and must be reimported on one host, as shown in the following example:

# metaimport -r Drives in regular diskset including disk c4t60A9800056716D70426F38414C524175d0: c4t60A9800056716D70426F38414C524175d0 More info: metaimport -r -v c4t60A9800056716D70426F38414C524175d0 Import: metaimport -s <newsetname> c4t60A9800056716D70426F38414C524175d0 Drives in regular diskset including disk c4t60A9800056716D70426F38414C503476d0: c4t60A9800056716D70426F38414C503476d0 More info: metaimport -r -v c4t60A9800056716D70426F38414C503476d0 Import: metaimport -s <newsetname> c4t60A9800056716D70426F38414C503476d0 # metaimport -s saplog_vi2 c4t60A9800056716D70426F38414C524175d0 [...] # metaimport -s sapdata_vi2 c4t60A9800056716D70426F38414C503476d0 [...]

Then you can add the other hosts to the set using the same commands as during initial setup.



10.1.4.2 Creating a Linux LVM2 Volume Group for FlexFrame Usage

The FlexFrame images for Linux Application Nodes contain a configuration file for LVM (/etc/lvm/lvm.conf) that is prepared so that LVM volumes can be set up on DM-MPIO multipath nodes.

Furthermore, if you want to use an LVM volume group with FlexFrame, it is necessary to mark this volume group with an “FF4S” tag as shown in the following example:

bx1:~ # pvcreate /dev/disk/by-name/3600601603e851800681c1bad1d38db11 Physical volume "/dev/disk/by-name/3600601603e851800681c1bad1d38db11" success-fully created bx1:~ # vgcreate sapdata_s12 /dev/disk/by-name/3600601603e851800681c1bad1d38db11 Volume group "sapdata_s12" successfully created bx1:~ # vgchange --addtag FF4S sapdata_s12 Volume group "sapdata_s12" successfully changed

10.1.4.3 Completing the configuration and testing usability of SAN for an SID

After you have entered the description of the volume groups and file systems for a SAP system in the SAN configuration file, you should add these informations to the Flex- Frame database (LDAP) and test their usability. You can do this by performing the follow-ing steps:

1. Add the SAP SID to the FlexFrame configuration using the script ff_sid_adm.pl as described in the section Adding / Removing SAP SIDs and Instances on page252. Alternatively, you can also use an SID that has been defined with the Planning Tool.

2. Tell the LDAP database that this SID is using SAN volumes for its database as shown in the following example:

control1:~ # ff_sid_mnt_adm.pl --op=add --pool=pool2 --sid=T13 --san=/tmp/tftpboot/config/sancfgT13

Alternatively you can do this by using the script ff_san_ldap_conf.pl as shown in section 10.2.1.

3. Before you can install SAP with its database on the SAN volumes, some folders for the SID have to be created in advance. This also creates links to the mount points for the SAN volumes. To do so, run the script ff_setup_sid_folder.sh as shown in the following example:

control1:~ # ff_setup_sid_folder.sh pool2 T13

After running this script, you should also check that in the database directory for this SID(/oracle/T13 according to the example shown above) the required links to /var/FlexFrame/SAN/... have been created and there are no links into the sapdata/saplog volumes on the NAS system. If some of these links are still here



(this can happen if not all standard areas are needed), delete them, as otherwise a mixed SAN/NAS installation could result from this setup, if not taking care during SAP installation. This must be avoided, as mixing SAN and NAS for the database of one SID is not supported.

4. Mount the file systems defined for this SID by using the script ff_san_mount.sh on the Application Node where you have prepared the file systems, as shown in the following example:

# ff_san_mount.sh pre sapdb T13 start /FlexFrame/scripts/ff_san_mount.sh: SAN tool. (SID=T13, PHASE=pre, ACTION=start) Checking availability of disk group "dtaT13" Importing disk group "dtaT13" Recovering disk group "dtaT13" Checking availability of disk group "logT13" Importing disk group "logT13" Recovering disk group "logT13" file system is clean - log replay is not required /dev/vx/dsk/dtaT13/sapdata1 mounted at /oracle/T13/sapdata1 file system is clean - log replay is not required /dev/vx/dsk/dtaT13/sapdata2 mounted at /oracle/T13/sapdata2 file system is clean - log replay is not required /dev/vx/dsk/dtaT13/sapdata3 mounted at /oracle/T13/sapdata3 file system is clean - log replay is not required /dev/vx/dsk/dtaT13/sapdata4 mounted at /oracle/T13/sapdata4 file system is clean - log replay is not required /dev/vx/dsk/dtaT13/sapdata5 mounted at /oracle/T13/sapdata5 file system is clean - log replay is not required /dev/vx/dsk/dtaT13/sapdata6 mounted at /oracle/T13/sapdata6 file system is clean - log replay is not required /dev/vx/dsk/logT13/saparch mounted at /oracle/T13/saparch file system is clean - log replay is not required /dev/vx/dsk/logT13/oraarch mounted at /oracle/T13/oraarch file system is clean - log replay is not required /dev/vx/dsk/logT13/sapreorg mounted at /oracle/T13/sapreorg file system is clean - log replay is not required /dev/vx/dsk/logT13/sapbackup mounted at /oracle/T13/sapbackup file system is clean - log replay is not required /dev/vx/dsk/logT13/origlogA mounted at /oracle/T13/origlogA file system is clean - log replay is not required /dev/vx/dsk/logT13/origlogB mounted at /oracle/T13/origlogB file system is clean - log replay is not required /dev/vx/dsk/logT13/mirrlogA mounted at /oracle/T13/mirrlogA file system is clean - log replay is not required /dev/vx/dsk/logT13/mirrlogB mounted at /oracle/T13/mirrlogB

SAN Configuration in a FlexFrame Environment FlexFrame SAN Configuration


5. Check that the mounted file systems have the correct ownership required by the database installation and change it as appropriate.

6. Unmount the file systems on this Application Node using the ff_san_mount.sh script as shown in the following example:

# ff_san_mount.sh post sapdb T13 stop

Repeat the steps 4 and 6 on the other Application Nodes of the Pool Group to test that each one can access the file systems if needed.

Before you start the SAP installation, mount the file systems again on the Application Node that has been selected for this task.

After the SAP installation is finished, unmount the file systems again using the ff_san_mount.sh script. For more details on using this script refer to its man page. Please note in particular that this script is usually implicitly called during the start and stop of a database instance and should not be used directly, except in situations where you have assured that no other host is using the volumes and file systems configured for the database instance. Not taking care of this may result in severe data corruption, up to making the respective database unusable.

10.2 FlexFrame SAN Configuration

10.2.1 Script: ff_san_ldap_conf.pl ff_san_ldap_conf.pl is used to administrate the FlexFrame SAN configuration of Application Nodes.

The arguments given with the command line differ with the operation mode.

To add new information to the FlexFrame database it is necessary to define a lot of data. These data are stored in a configuration file which is specified as parameter of the add operation.

Synopsis

ff_san_ldap_conf.pl --op=add [--conffile=<filename>]

[--outfile=<filename>] [--sid={<sid>|*}]

[--pool={<poolname>|*}] [--vlun={<vlunname>|*}]

[--node={<nodename>|*}] [--dryrun] [--verbose]

FlexFrame SAN Configuration SAN Configuration in a FlexFrame Environment


ff_san_ldap_conf.pl --op=del [--outfile=<filename>]

[--sid={<sid>|*}] [--pool={<poolname>|*}]

[--vlun={<vlunname>|*}] [--node={<nodename>|*}] [--dryrun]

ff_san_ldap_conf.pl --op=list [--sid={<sid>|*}]

[--pool={<poolname>|*}] [--vlun={<vlunname>|*}]

[--node={<nodename>|*}]

ff_san_ldap_conf.pl –-help

Options

--op=add Adds new SAN information to the FlexFrame database.

--op=del Deletes SAN information from the FlexFrame database.

--op=list Displays SAN information from the FlexFrame database.

--conffile=<filename> Name of the SAN configuration file. The default for conffile is /tftpboot/config/san.cfg.

--outfile=<filename> Name of the workfile for writing data to LDAP. The default for outfile is /tmp/ff_san_ldapdata.

--sid=<sid>|* The data for the given SID or all sids (*) of the specified pool(s) are selected.

--pool=<poolname>|* The data for the given pool or all pools (*) are selected.

--vlun=<vlunname>|* The data for the given vlun or all vluns (*) of the specified pool(s) are selected.

--node=<nodename>|* The data for the given node or all nodes (*) are selected.

--dryrun Performs the actions of the script but don't write the data to LDAP. The result is writ-ten to outfile.



--verbose All LDAP messages are displayed.

--help Displays usage.

10.2.2 FlexFrame SAN Configuration File

10.2.2.1 Syntax

# definition of a node NODE <node_name> { availableMultipathSW = {MPxIO|Powerpath|DM-MPIO|DMP|USER|NONE} listOfHBA = <value> ... } ... # definition of an SID SID <pool_name> <sid_name> <instance_name> { VOLUMEGROUP <volume_group_name> { usedMultipathSW = {MPxIO|Powerpath|DM-MPIO|DMP|USER|NONE} volumeManagerType = {Veritas|SolarisVM|LVM|USER|NONE} volumeManagerUserScript = <value> volumeManagerOptions = <value> usage = {SAPDATA|SAPLOG} virtualLUN = <value> MOUNTPATH <file_system_mountpoint_destination> { fileSystemType = {UFS|VxFS|ext3|USER} fileSystemUserScript = <value> fileSystemMountOptions = <value> fileSystemCheckOptions = <value> fileSystemMountpointSource = <value> partitionOfLUN = <value> virtualLUN = <value> } ... } ...



} ... # definition of a VLUN VLUN <vlun_name> <pool_name> { hostLUN = <value> arrayLUN = <value> arrayID = <value> LUNGUID = <value> } ...

10.2.2.2 Description

The file /opt/FlexFrame/etc/ff_san_ldap_conf.template contains a template of the configuration file.

The configuration file consists of several lines:

– Comment lines are marked with a “#” in the first column of the line.

– Empty Lines or lines consisting only of whitespaces are handled like comment lines.

– All other lines describes the entries of the configuration file.

The configuration file has three entries.

– NODE entry

– SID entry

– VLUN entry

Each entry consist of the header line, the opening curly bracket line, the data lines and the closing curly bracket line.

NODE entry

The NODE entry describes the properties of an Application Node and is identified by its host name (as defined in the Planning Tool and returned by uname -n). It consists of the following parameters:



availableMultipathSW This parameter is required and defines the multipath software which is available on the node. The possible values are:

Value Meaning Available on

MPxIO Native multipath software of SOLARIS SOLARIS

Powerpath Multipath software of EMC SOLARIS

DM-MPIO Native multipath software of LINUX LINUX

DMP Multipath software delivered with Veritas Volume Manager

SOLARIS

USER User defined multipath software LINUX/SOLARIS

NONE No multipath software used LINUX/SOLARIS

The special value USER means that a non-standard multipath software is used. The functionality of this multipath software is not guarantied.

listOfHBA This parameter is required and assigned a logical name to each available HBA The values of the logical names are not restricted by FlexFrame.

SID entry

The SID entry describes the SAN properties of an SID’s database instance and is identi-fied by the name of the FlexFrame pool, the SID and the database instance. It consists of one or more VOLUMEGROUP entries.

VOLUMEGROUP entry

A VOLUMEGROUP entry consists of parameters describing the properties of the volume group and one or more MOUNTPATH entries.

Each VOLUMEGROUP entry is identified by the name of the volume group. The syntax of the volume group name depends on the used volume manager. If no volume manger is used then the value is not restricted by FlexFrame.



usedMultipathSW This parameter is required and defines the used multipath software. The possible values are:


MPxIO Native multipath software of SOLARIS SOLARIS

Powerpath Multipath software of EMC SOLARIS

DM-MPIO Native multipath software of LINUX LINUX

DMP Multipath software delivered with Veritas Volume Manager

SOLARIS

USER User defined multipath software LINUX/SOLARIS

NONE No multipath software used LINUX/SOLARIS

The special value USER means that a non-standard multipath software is used. The functionality of this multipath software is not guarantied.

volumeManagerType This parameter is required and defines the used volume manager. The possible values are:


SolarisVM Native volume manager of SOLARIS SOLARIS

Veritas Veritas Volume Manager SOLARIS

LVM Native volume manager of LINUX LINUX

USER User defined volume manager LINUX/SOLARIS

NONE No volume manager used LINUX/SOLARIS

The special value USER means that a non-standard volume manager is used. The functionality of this volume manager is not guarantied.

volumeManagerUserScript If the value of the parameter volumeManagerType is USER then this parameter specifies the name of a user script for handling the volume manger’s start and stop actions. The file name must be full qualified.

The call by FlexFrame is performed with the following parameters:

– Actioncode volume_on for actions performed during start of a SID or volume_off for actions performed during start of an SID

– SID of the SAP system



– Service type

– Service number

– DN of the LDAP entry which caused the user script to be called

volumeManagerOptions This parameter is optional and contains parameters for the used volume manager. The possible values depends on the used volume manager. For details see the de-scription of the used volume manager. Examples for values are listed in the following table:

Value Meaning

forceimport Force the import (this value is mandatory)

waitforce=5 Waittime is five seconds

The different values are separated by a semicolon.

usage This parameter specifies the usage of the volume group. The allowed values are SAPDATA, SAPLOG, or both.

virtualLUN This parameter specifies the name(s) of the virtual LUN(s) which are contained in the volume group and is only required if the SOLARIS Volume Manager is used.

MOUNTPATH entry

A MOUNTPATH entry consists of parameters describing the properties of the mountpath and is identified by the absolute pathname of the destination of the mountpath. The syn-tax of this name must be conform with the syntax of the Application Node’s operating system. The pathname must start with /oracle/<SID> or /sapdb/<SID>. This path-name will be prefixed with /var/FlexFrame/SAN.

fileSystemType This parameter is required and defines the used file system. The possible values are:


UFS Universal file system SOLARIS

VxFS File system delivered with Veritas Volume Manager SOLARIS

ext3 Extended file system version 3 LINUX

USER User defined file system LINUX/SOLARIS



The special value USER means that a non-standard file system is used. The function-ality of this file system is not guarantied.

fileSystemUserScript If the value of the parameter fileSystemType is USER then this parameter specifies the name of a user script for handling the file system start and stop actions. The file name must be full qualified.

The call by FlexFrame is performed with the following parameters.

– mount / umount specify that a mount or an umount action is required

– SID of the SAP system

– Service type

– Service number

– Mountpoint (full qualified filename).

– Volume name

– Volume group name

fileSystemMountOptions This parameter is optional and defines options used during the mount of the file sys-tem. The possible values depends on the used file system. For details see the de-scription of the file system. Examples for values are listed in the following table:

Value Meaning

forceumount Force the umount

mountopts=largefiles=yes Options for the mount command

The different values are separated by a semicolon.

fileSystemCheckOptions This parameter is optional and defines options used during the check of the file sys-tem. The possible values depends on the used file system. For details see the de-scription of the file system. The special value nofsck indicates that no check of the file system is to be performed. Examples for values are listed in the following table.

Value Meaning

nofsck Skip file system check

fsckopts=-ya Options for the fsck command



fileSystemMountpointSource This parameter is required and specifies the source of the data which are used for this mountpath. The source is a relative file name. For example, if using a volume manger this is the volume name. The complete file name is constructed by flexframe depending on the used software stack.

virtualLUN This parameter specifies the name of the virtual LUN which is used by this mount-path and is only required if no volume manager is used.

partitionOfLUN This parameter specifies the partition of the LUN and is only required if no volume manager is used.

VLUN entry

The VLUN entry consists of data describing the properties of the virtual LUN and is identi-fied by the name of the virtual LUN. For this name there is no syntax restriction by Flex-Frame.

The specification of VLUN entries is only required if the SOLARIS volume manager or no volume manager is used (see section 10.1.1.4 and 10.1.3.4 for getting the required in-formation). It consists of the following parameters:

hostLUN This parameter is required and specifies the ID of the storage LUN as it is visible by the application node.

arrayLUN This parameter is required and specifies the ID of the storage LUN as it is visible by the SAN storage subsystem.

arrayID This parameter is required and specifies the ID of the SAN storage subsystem.

LUNGUID This parameter is required and specifies the global unique ID of the LUN.

SAN Support Scripts SAN Configuration in a FlexFrame Environment


10.3 SAN Support Scripts

10.3.1 Script: ff_san_mount.sh Mounting of SAN attached file systems

Synopsis

ff_san_mount.sh {pre|post} <service> <SID>

{start|stop|prestart|post-stop}

Description

This script is used on Application Nodes (Linux or Solaris) during start or stop of a data-base service. During installation of a database this script must be called if the datafiles are to be placed onto SAN based file systems. It is assumed that the information for all mountpoints is mainted in the configuration database using ff_san_ldap_conf.pl.

Caution must be applied when using this script directly. The configured file sys-tems and logical volumes must not be used by other hosts. Not taking care of this may cause data corruption and make the data on these file systems unus-able. In particular you must be aware that if the forceimport option has been speci-fied in the configuration database for the affected volume group, the volume group will be made accessible on this host by forcing import with the mecha-nisms of the used volume manager software.

Once the database is properly installed the sapdb or saplc script is calling ff_san_mount.sh during the start or stop phase implictly. The prestart and prestop options are used with SAP ACC. However, their functionality is identical to start and stop respectively.

To mount a specific set of file systems for the database SID use

ff_san_mount.sh pre sapdb SID start

To unmount the file systems (after all services have stopped) use

ff_san_mount.sh post sapdb SID stop

This script acts upon information stored in the LDAP database. Refer to ff_san_ldap_conf.pl to configure this information.

Debugging

/tmp/ff_san_mount.sh.DEBUGLOG

SAN Configuration in a FlexFrame Environment SAN Support Scripts


This file is located at the Application Node and will hold debugging information. In case of problems provide this file.

10.3.2 Script: ff_san_info.sh SAN information utility

Synopsis

ff_san_info.sh {-g lunid|-t lunspec|-w guid|-n serno|-e|-s path|-i}

Description

This utility can be used to get various SAN information and convert them into other for-mats.

Options

-g lunid (Linux only) Shows GUID of LUN(s) with LUN-ID.

-t lunspec (Linux only) Shows WWNN of target by lunspec (format <host_no>:0:<target_no>:<lun_no>).

-w guid (Linux only) Shows WWNN of target by GUID.

-n serno Shows GUID of a NetApp LUN based on its serial number (see lun show -v <path> on filer).

-e Shows instructions to find GUID of EMC systems.

-s path (Solaris only) Shows details of a SFS device path.

-i Shows information on FC hosts and targets which are seen from this node.

Usage example on a SLES 9 Application Node:

bx2:~ # ff_san_info.sh -i HBA WWNN WWPN 1 20:00:00:c0:9f:c6:45:08 21:00:00:c0:9f:c6:45:08 2 20:00:00:c0:9f:c6:45:09 21:00:00:c0:9f:c6:45:09 Target WWNN WWPN 1:0:0 50:06:01:60:b0:60:1f:31 50:06:01:60:30:60:1f:31 1:0:1 50:06:01:60:b0:60:1f:31 50:06:01:68:30:60:1f:31 2:0:0 50:06:01:60:b0:60:1f:31 50:06:01:61:30:60:1f:31

SAN Support Scripts SAN Configuration in a FlexFrame Environment


2:0:1 50:06:01:60:b0:60:1f:31 50:06:01:69:30:60:1f:31 bx2:~ # ff_san_info.sh -g 1 LUN(s): 1:0:1:1 2:0:1:1 1:0:0:1 2:0:0:1 GUID: 3600601603e8518008570a1841d38db11 tom1bx2:~ # ff_san_info.sh -t 1:0:1:1 LUN 1:0:1:1: Target WWNN: 5006016830601f31 Corresponding line from 'lsscsi': [1:0:1:1] disk DGC RAID 5 0219 /dev/sdy tom1bx2:~ # ff_san_info.sh -w 3600601603e8518008570a1841d38db11 GUID 3600601603e8518008570a1841d38db11: LUN 1:0:1:1: Target WWNN: 5006016830601f31 Corresponding line from 'lsscsi': [1:0:1:1] disk DGC RAID 5 0219 /dev/sdy

10.3.3 Script: ff_qlascan.sh Scanning for new LUNs on LINUX using a QLA 2xxx HBA

Synopsis

ff_qlascan.sh [-d] -c <a-b> -l <x-y>

Description

This script can be used to query for new LUNs on a Linux Application Node, if the IO subsystem is connected using QLA host bus adapters (HBA).

Options

-d Turn on debugging. Currently not supported - use bash -x ff_qlascan.sh ... instead.

-c <a-b> Scan controllers from a to b (e.g. "1-2").

-l <x-y> Scan for LUN-IDs from x to y (e.g. "10-20").


11 Security

11.1 Requested Passwords and Password Settings During Installation

11.1.1 Requested Passwords of the Control Nodes Entering all passwords is mandatory. For IPMI and LDAP, passwords have to be equal on both Control Nodes.

Please note that the passwords are displayed on the screen – the input will not be hidden!

Security Requested Passwords and Password Settings During Installation


11.1.2 Setting Passwords of a NAS System

NetApp Filer:

Setting the administrative (root) password for <FILER-NAME> ... New password: Retype new password:

EMC Celerra

You can set a new password for root and nasadmin on Celerra as you do it for any Linux system.

11.1.3 Initial SSH Configuration In order to initialize ssh authorization between control nodes, call

/opt/FlexFrame/bin/ff_ssh_tool.sh –i

You will be asked to enter the passwords of both Control Nodes.

cn1:~ # ff_ssh_tool.sh -i Copyright (C) 2005 Fujitsu Siemens Computers. All rights reserved. Would you like to (re)initialize SSH authorization of the Control Nodes? Warning: If you enter "yes", current SSH authorization will be overwritten. That means you will lose your authorized_keys and known_hosts files. Note that /etc/hosts from this node will be transferred to the other Con-trol Node if it is different. It is recommended to backup theses files on both Control Nodes before you continue. Following files will be affected: /root/.ssh/authorized_keys /root/.ssh/known_hosts /etc/hosts Continue? yes You are on first Control Node (cn1) Enter password for "cn1" (this node): Enter password for "cn2" (leave empty for same password): ~~~ Initial test of ssh connection between Control Nodes ~~~ Testing ssh to this Control Node (cn1) SSH connection to "cn1" works

Password Settings During Operation, Update and Upgrade Security


Testing ssh to other Control Node (cn2) SSH connection to "cn2" works ~~~ ssh connection works, configuring authorization ~~~ root@cn2's password: ~~~ authorization complete, testing all addresses... ~~~ SSH connection to "..." works SSH connection to "..." works SSH connection to "..." works ... ... ~~~ transferring known_hosts file to other Control Node ~~~ SSH initialization is complete ~~~ syncing /etc/hosts with other Control Node ~~~ Copying hosts to "cn2" ssh initialization finished successfully.

11.2 Password Settings During Operation, Update and Upgrade

11.2.1 User Administration In order to change user and/or group IDs as well as passwords, several tools can be utilized.

For changing user passwords on Solaris-based Application Nodes the standard passwd tool can be used. On Linux based Application Nodes this tool will not work and ff_ldappasswd.sh should be used instead.

ff_ldappasswd.sh will retrieve the current user, will prompt the user for a new pass-word and change it accordingly.

Do not use ldappasswd, it will leave your LDAP password entries unusable.

The default password for all Unix users is password.

In order to change Unix user passwords from a Control Node use

ff_ldappasswd.sh -l <login_name> -p <pool_name> [-r]

Security Password Settings During Operation, Update and Upgrade


The -r option enables the root user to change a password without authentication through the rootdn of the LDAP server, i.e. the tool will retreive the rootdn and its password for authorization from the configuration files.

In order to change the password for the LDAPadmins use -p LDAPadmins instead of providing a pool name, and provide admins or replicate as the user option's value. This will change passwords of admins or replicate LDAP users.

In cases where existing SAP systems are transfered into FlexFrame or recovered from an earlier installation, it might be necessary to change user or group IDs to different values than the ones provided by the installation tools. In order to do that, ff_change_id.pl can be utilized:

ff_change_id.pl --pool <pool_name>

[--uid <uid> <group_name> | --gui <gid> <user_name>]

should be run on a Control Node. It will change IDs according to the values given, as long as those values are not occupied by other entries.

11.2.2 Password Management on Control Nodes

11.2.2.1 Passwords for Root and Standard Unix Users

The root password and the user passwords are located in the file /etc/shadow on both Control Nodes and can be changed with the standard passwd command.

cn1:~ # vi /etc/shadow cn1:~ # passwd Changing password for root. New password: Password will be truncated to 8 characters Re-enter new password: Password changed

The user passwords of the Control Nodes may (but should not) differ and can be changed at any moment. A change has no other impact.

11.2.2.2 Password for Root of LDAP Server and Replica

The password is stored in the configuration file /etc/openldap/slapd.conf as value of the entry rootpw on both Control Nodes and within the configuration inside the LDAP server itself under cn=root,ou=LDAPadmins resp. cn=replica,ou=LDAPadmins.

Modify the password with an editor in /etc/openldap/slapd.conf on both Control Nodes.



Modify the password on both, the root and the replica of the LDAP server.



The passwords can be changed at any moment. A change has no other impact.

If the LDAP server and its replica are not rebooted both the old password and the new password will be valid. After rebooting both only the new password will be valid.

The LDAP servers should be rebooted from the PRIMECLUSTER Interface (ldap_srv1 and ldap_srv2) .



11.2.2.3 Password for LDAPadmins

The password is stored in cn=sysclient,ou=LDAPadmins,ou=<pool>,ou=Pools for each pool inside the LDAP server. Modify the password with an LDAP editor. The password have to be match the password given to the Solaris LDAPcachemgr.

11.2.2.4 Password for SNMP Community

The password is stored on both Control Nodes in the /etc/snmpd.conf and /opt/FlexFrame/etc/ff_misc.conf configuration files. Modify this password with an editor. The password can be changed at any moment. A change has no other impact.

11.2.2.5 Key for Access to the Name Server (named)

The key is stored on both Control Nodes in the file /etc/rndc.key. Modify this key with an editor. The key can be changed at any moment. A change has no other impact.

11.2.2.6 Password for myAMC WebGUI

The password is stored on both Control Nodes in the configuration file /opt/myAMC/config/FA_WebGui.conf. Modify this password with an editor. The password can be changed at any moment. A change has no other impact.



11.2.2.7 Password for mySQL Database

This password is stored in the configuration file /opt/myAMC/config/amc.conf and inside the mySQL server on both Control Nodes.

Step 1: Password for the connection of myAMCMessenger to database

Modify the password in the following command:

/etc/init.d/myAMC.MessengerSrv connect <user> <password>

Step 2: Password inside the database

Login to mySQL database:

cn1:~ # mysql -u myAMC -p FlexFrame mysql> SET PASSWORD FOR 'myAMC'@'localhost' = PASSWORD('<new_password>');

The password can be changed at any moment. A change has no other impact.

11.2.2.8 Password for Power Shutdown

1. The password is stored inside the server BIOS. For detailed description see section “Configuring User, Password and Community” on page 291.

2. The password is stored on both Control Nodes in the configuration file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml. Modify this password with an editor or the WebGUI.

3. When modified with the WebGUI the password is automatically changed in the PCL configuration file /etc/opt/SMAW/SMAWsf/SA_ipmi.cfg.


11.2.2.9 Password for SAPDB/MaxDB in FlexFrame Start/Stop Scripts

The password is stored aside the user control in /FlexFrame/scripts/* for SAPDB/MaxDB on both control nodes. The default value is control.

cn1:~ # cd /FlexFrame /scripts cn1:~ # grep control, * Modify this password with an editor. The user control must not be changed. This password must be identical for user control in all SAPDBs/MaxDBs across the site.



11.2.3 Password Management on Linux Application Nodes Refer also to the section “User Administration” on page 235.

11.2.3.1 Password for Admin on BX(3,6)00

For detailed information how to change see section “Configuring User, Password and Community” on page 291.

11.2.3.2 Password for PowerShutdown on BX(3,6)00, RXxxx

For detailed information how to change the password for SNMP community see section “Configuring User, Password and Community” on page 291.

1. The password is located in the myAMC configuration file myAMC_FA_SD_Sec.xml on both Control Nodes. Modify this password with an editor or the WebGUI

2. When modified with the WebGUI the password is automatically changed in the PCL configuration files /etc/opt/SMAW/SMAWsf/SA_*.cfg.

3. For RX300 application nodes the file is /etc/opt/SMAW/SMAWsf/SA_ipmi.conf. The configuration can be tested with

cn1:~ # ipmipower -s <ip_address> -u <user> -p <password>


11.2.3.3 Passwords for Root / Standard Unix Users and SAP Users

See section “User Administration” on page 235.

11.2.3.4 Changing Root Password for Linux Application Nodes

The script ff_passwd_linux.pl can be used to change passwords for all users in /etc/shadow.

Synopsis

/opt/FlexFrame/bin/ff_passwd_linux.pl

--user <user_name>

{[–-scope host] –-host <host_ip_address>| --scope group –-group <group_names/numbers>|

--scope pool} -–pool <pool_name>

--ospath <path_to_image> [--pass <passwd>]



Description

You can change the password for

● a certain Application Node (you can omit the option –-scope host because host is the default):

--user <user_name> –-host <host_ip_address> –-pool <pool_name>

● a group within a pool:

--user <user_name> --scope group --group <group_name/number>

-–pool <pool_name>

● all Linux nodes within a pool:

--user <user_name> --scope pool –-pool <pool_name>

If the option --pass is not given, it is asked later by prompting.

All changes are related to a concrete image, giving by the –ospath option.

All possible images are shown by ff_passwd_linux.pl –help.

11.2.4 Password Management on Solaris Application Nodes Refer also to the section “User Administration” on page 235.

11.2.4.1 Password for PowerShutdown

For detailed information how to change password for community see section “Configuring User, Password and Community” on page 291.

1. The password is stored on both Control Nodes in the configuration file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml. Mod-ify the password with an editor or the WebGUI.

2. When modified with the WebGUI the password is automatically changed in the PCL configuration files /etc/opt/SMAW/SMAWsf/SA_*.cfg.


11.2.4.2 Passwords for Root / Standard Unix Users and SAP Users

See section “User Administration” on page 235.



11.2.5 Passwords for Networking Components

11.2.5.1 Password for Cisco Switch Enable/Login and SNMP_Community

Change password by calling ff_swgroup_adm.pl as described in section 2.5.4.

11.2.5.2 Admin Password for SwitchBlade BX(3,6)00

Change password by calling ff_bx_cabinet_adm.pl as described in section 8.6.6.

11.2.5.3 Admin and SNMP_Community Storage Passwords

Passwords can be modified with ONTAP WebGUI or by telnet session to every Filer.

11.2.6 Preparation for Linux Application Nodes

11.2.6.1 Script: ff_install_an_linux_images.sh

Synopsis

ff_install_an_linux_images.sh [-v] [-p <path_to_images>]

Description

● Setting up the root password in the root image

You need a password to log onto the Application Nodes as root. This password is valid for Application Nodes with the same shared root Image. The script asks for the password twice. If both strings are the same, the password will be accepted.

Enter password: ******** Enter password again: ******** Password accepted.

If you plan to use more root images, for example a separate root image for every pool with different root passwords, you must copy these root images manually after this installation procedure.

● Setting up ssh in the root image

To log on from the Control Nodes to the Application Nodes without entering the root password, set up ssh as follows:

– Generate host keys for the root image

– Deploy authorized keys from the Control Nodes into the root image

Security Passwords Stored in Initialization Files


11.2.7 Preparation for Solaris Application Nodes

11.2.7.1 Script: nb_unpack_bi

Synopsis

nb_unpack_bi [-x] [-q|-v] | [-h]

Description

Please enter the password for your Solaris Application Nodes. It applies for all Solaris Application Nodes.

INFO: A root password is required for the Application Nodes You will now be asked for the root password (valid for all Application Nodes using this image). Enter password: Enter password again: Password accepted.

11.3 Passwords Stored in Initialization Files

11.3.1 Switch Group Switch Definition Files The script ff_wiring.pl creates (amongst others) switch group switch definition files (used with ff_switch_conf.pl).

/opt/FlexFrame/network/switchgroup*.def Switch group switch definition files. These may contain data of more than one switch if the switch type is stackable and usable as one virtual switch (as supported by the CISCO Catalyst 3750 switch family). Contains switch type, host name, snmp, ntp, syslog, vlan and port definitions. The file content may look like this:

SWITCHTYPE=cat3750g-24ts HOSTNAME=sw1-1 IP=100.100.13.18;255.255.255.0;14 SNMP=public;ro SYSLOG=100.100.13.13;local0 PASSWORD=passwort;root NTP=100.100.13.12;server NTP=100.100.13.13;server

Configuration of the SCON Shutdown Agent on the Control Nodes Security


11.4 Configuration of the SCON Shutdown Agent on the Control Nodes

For this purpose the rcsd.cfg file was supplemented under /etc/opt/SMAW/SMAWsf and the SA_scon.cfg file created.

cn1:~ # more /etc/opt/SMAW/SMAWsf/SA_ipmi.cfg # FlexFrame(TM) # Generated file at Mon Nov 29 11:53:03 CET 2004 # A backup of a previous version (if existed) my be found # in directory: /opt/FlexFrame/etc/backup_20041129115223 # # DO NOT EDIT MANUALLY! # CN1-1 :OEM:passwort cycle CN2-1 :OEM:passwort cycle # end of configuration file

11.5 FA Agents

Security FA Agents


The password is stored in configuration file /opt/myMAC/config/FA_WebGui.conf in the entries messengerdb.jdbc.username and messengerdb.jdbc.password.

The default entries are:

messengerdb.jdbc.password messengerdb.jdbc.url=jdbc:mysql://localhost:3306/messenger messengerdb.jdbc.username=myAMC messengerdb.jdbc.password=FlexFrame

For modification(s) use a text editor on the Control Nodes.

11.5.1 Power-Shutdown Configuration The power shutdown is necessary for FlexFrame Autonomous Agents to switch off Appli-cation Nodes in the case of a switchover scenario to make sure that a system in an unde-fined state will be switched off. This is also necessary for the PRIMECLUSTER Shutdown Facility on the Control Nodes.

The power shutdown will be implemented by the Shutdown Agents (SA) of the PRIME-CLUSTER Shutdown Facility and the ff_xscf.sh script.

Each Shutdown Agent (SA_blade, SA_ipmi, SA_rsb, SA_rps and SA_scon) has its own config file which will be automatically configured by the FlexFrame Autonomous Agents and the PRIMECLUSTER Shutdown Facility on the Control Nodes.

The PRIMEPOWER XSCF (eXtended System Control Facility) is a built-in component for PRIMEPOWER 250 and 450 systems. It enables either serial or LAN access to the server’s console or power-on, power-off and reset commands.

Only the hardware and software which is used by the Shutdown Agents and XSCF must be prepared with IP address, user, password, etc.

You must therefore perform the following steps in accordance with the Application Node and Control Node hardware.

Detailed information on the power-shutdown configuration is available in the "FA Agents - Installation and Administration" manual. The following sections just show where user-names and passwords are stored.

11.5.1.1 BX300/600

Default: <Username> = root, <Password> = root

Use the same user name, password and community string in config file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml, config section Security_default or use a separate config section entry for this server. The SNMP community string should be “read-write”.

FA Agents Security


11.5.1.2 RX300/RX300 S2

IPMI configuration:

● LAN A onboard should only be used for IPMI on the Control LAN

● LAN B onboard and the upper NIC of the separate LAN card should be used for bonding.

Use the same user name and password in config file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml, config section Security_default or use a separate config section entry for this server.

11.5.1.3 RX600

RSB configuration:

● RX600: RSB1 onboard

Connect the Control LAN cable with the LAN port for the remote management controller.

Use the same user name and password in config file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml, config section "Security_default" or use a separate config section entry for this server.

Use the same community string in config file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml, config section Security_default or use a separate config section entry for this server.

11.5.1.4 RX800

ASM, RSB and the connection between must be configured.

Set user name and password (same as in /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml, config section Security_default or separate entry for this server).

Set Permissions for Reset/SwitchOff to 1 and then press <F1>.

Use the same user name and password in config file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml, config section Security_default or use a separate config section entry for this server.

Create new SNMP Community 2 for tcc and then press F1.

Use the same community string in config file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml, config section Security_default or use a separate config section entry for this server.

Security FA Agents


11.5.1.5 PRIMEPOWER 250/450

The PRIMEPOWER XSCF (eXtended System Control Facility) is a built-in component to enable either serial or LAN access to the server’s console or power-on, power-off and reset commands.

The FlexFrame images for Solaris will automatically configure most of the XSCF settings (serial/lan, IP address etc.) during system startup. The /etc/default/SMAW_xscf configuration file of each server contains the settings.

For security reasons, password information is not stored in this configuration file and must be configured manually. Log on to the server and call the program /opt/FJSVmadm/sbin/madmin.

Set the same user name and password in config file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml, config section Security_default or use a separate config section entry for this server.

You can test the user and password by accessing the port 8010 of the server’s control LAN interface using telnet:

cn1:~ # telnet server-co 8010

The initial values are root (username) and fsc (password).

11.5.1.6 PRIMEPOWER 650/850

To power down servers with RPS (PRIMEPOWER 650/850), only one RPS per server is used controlling one (PRIMEPOWER 650) or two (PRIMEPOWER 850) protection boxes.

When the RPS has been powered down it has to be ensured that the associated protec-tion boxes and the server are also powered down so that a consistent status is achieved.

Prerequisite: The RSB inside the RPS must have the firmware version 1.0.3.105 or higher.

Set the same user and password in the configuration file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml, config section Security_default or use a separate config section entry for this server.

The initial values are root (username) and fsc (password).

FA Agents Security


11.5.2 SNMP Traps The FA Agents are able to send SNMP traps to configured trap destinations. The traps contain status information resp. changes of services in the FlexFrame environment. For example there is an SNMP trap if a service is starting. SNMP traps can be used to con-nect an external system monitoring application to FlexFrame.

Detailed information on the SNMP trap configuration is available in the "FA Agents - In-stallation and Administration" manual. The following sections just show where SNMP community strings are stored.

11.5.2.1 General

The TrapTargets.xml file contains all the trap destinations, i.e. information which is needed to send SNMP traps. Two parameters are required for each target:

● Host name or IP address

● SNMP community

The community roughly corresponds to a password.

Generally public is configured as default.

11.5.2.2 Configuring User, Password and Community

To use agent power shutdown, user, password and community must be defined in the configuration of the FA Agent. This configuration is specified in the pool-specific configu-ration file /opt/myAMC/vFF/vFF_<pool_name>/config/myAMC_FA_SD_Sec.xml. The entries for user, password and community must be the same as those configured in the Application Nodes.

11.5.2.3 Configuring Management Blades

The management blades have to be configured. This is done in the Managementblades configuration section of the myAMC_FA.xml configuration file.

Security PRIMECLUSTER


11.6 PRIMECLUSTER

Read access with the PCL Web-based Admin View is possible with every Unix user de-fined on the control nodes. Write access is only possible with root

See section “User Administration“on page 235.


12 Administrating SAP Systems This chapter describes the management of SAP System IDs (so-called SIDs) and their respective instances within the FlexFrame environment. It further describes how to clone SIDs as well as their instances for a different pool than the one they were installed to.

The tools described below only maintain the LDAP database entries, rather than adding or removing the data and binaries of the respective SAP systems. These steps need to be performed manually.

Listing, adding, removing and cloning the above entities in the LDAP server is supported by two tools, ff_sid_adm.pl and ff_clone_sid.pl. Both scripts will take care of keeping the data accessed by the operating system’s naming service mechanism in sync with the FlexFrame internal configuration data, both of which reside in the LDAP server.

This data should not be manipulated manually.

Preventing port conflicts

Please take into account that depending on the instance number ports are reserved for the SAP application. That could mean that you conflict with other non-SAP applications on your nodes (e.g. SAP’s ICM uses ports 80nn which maybe conflicts with an application which uses port 8081). Please refer to SDN document “TCP/IP Ports Used by SAP Applications” to prevent conflicts.

12.1 Listing SAP SIDs and Instances The script ff_sid_adm.pl is used to list existing SAP System IDs and their instances (for adding and removing SIDs please see 12.2).

Synopsis

ff_sid_adm.pl –-op list –-pool <pool_name> [--sid <SAP_system_id>]

Options

--op list Determines the list operation.

--pool <pool_name> Determines the FlexFrame pool to which the operation should be applied.

--sid <SAP_system_id> Determines the SID being used.

Administrating SAP Systems Adding / Removing SAP SIDs and Instances


Examples:

%> ff_sid_adm.pl –-op list –-pool Pan OL1 OL4 OL2 SHT

%> ff_sid_adm.pl –-op list –-pool Pan –-sid SHT 01 00 DB0

12.2 Adding / Removing SAP SIDs and Instances The script ff_sid_adm.pl is used for adding and removing SAP SIDs and/or their in-stances (for listing configured SID and instance see 12.1).

Synopsis

ff_sid_adm.pl –-op add –-pool <pool_name> --sid <SAP_system_id> --sapversion {4.6|6.20|6.40|7.0} --db {ORACLE9|ORACLE10|SAPDB73|SAPDB74|MAXDB75|MAXDB76} --lc {MAXDB75|MAXDB76} --group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,... --user <username1>:<uidnumber1>,<username2>:<uidnumber2>,... <db loghost>:{<db_loghost_ip>|*} --sap {ci|app|jc|j|scs|ascs|ers}:<SYSNR>: <loghost-client>:{<loghost_client_ip>|*} <loghost-server>:{<loghost_server_ip>|*}

ff_sid_adm.pl –-op del –-pool <pool_name> --sid <SAP_system_id> [--sysnr <SYSNR>]

Options

--op add Determines the add operation.

--op del Determines the del operation.

--pool <pool_name> Determines the FlexFrame pool to which the operation should be applied.

--sid <SAP_system_id> Determines the SID being used.

Adding / Removing SAP SIDs and Instances Administrating SAP Systems


--sapversion {4.6|6.20|6.40|7.0} Specifies the SAP basis version being used.

--db {ORACLE9|ORACLE10|SAPDB73|SAPDB74|MAXDB75|MAXDB76} Specifies the database type as well as the respective version being used.

--lc {MAXDB75|MAXDB76} Specifies the database type as well as the respective version being used.

--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,... --user <username1>:<uidnumber1>,<username2>:<uidnumber2>,...

user and group enable specially selected user numbers and group numbers to be assigned to SAP users and SAP groups respectively. In this case a check is made to see whether the user or group has already been defined for the DB system involved. A user or group is created only if they do not already exist. For example, a group dba which already exists cannot be assigned a group number which deviates from the de-fault value.

<db loghost>:{<db_loghost_ip>|*}} The logical host name is used for the database as well as the IP address for that host name. Use an asterisk if you want it to be chosen automatically. All the entries need to be specified in a colon separated format.

<lc loghost>:{<lc_loghost_ip>|*}} The logical host name is used for the database as well as the IP address for that host name. Use an asterisk if you want it to be chosen automatically. All the entries need to be specified in a colon separated format.

--sap {ci|app|jc|j|scs|ascs|ers}:<SYSNR>: <loghost>-client:{<loghost client ip>|*} <loghost>-server:{<loghost server ip>|*} Specifies an SAP instance (optionally multiple of those) through its type (ci, app, jc, j, scs, ascs, ers), its SAP system number, the logical host name in the client net-work, the respective IP address, the logical host name in the server network and the respective IP address. Again, the IP addresses can be replaced with asterisks in or-der to have them chosen automatically. LiveCache is only supported with --sapversion 7.0. It does not need an instance of type ci. ERS is only supported with --sapversion 7.0. For Enqueue Repli-cated Servers (ERS) the asterisk should be used because we do not need an IP ad-dress from the client and server LAN (given IP addresses will be ignored). All the entries need to be specified in a colon separated format.

--sysnr <SYSNR> Removes a specific SAP instance instead of the entire system (SID).

Administrating SAP Systems Cloning a SAP SID into a Different Pool


Examples:

Adding an SID with one Central Instance:

control1:~ # ff_sid_adm.pl –-op add –-sid SHT –-pool Otto --sapversion 6.40 –-db ORACLE9:dbsht:192.168.1.1 --sap ci:00:sht00-client:\*:sht00-server:\*

Adding an instance to an existing SAP System:

control1:~ # ff_sid_adm.pl –-op add –-sid SHT –-pool Otto --sapversion 6.40 --sap app:01:sht01-client:\*:sht01-server:\*

Adding SID for LiveCache:

control1:~ # ff_sid_adm.pl --op add --pool test --sid LCA --sapversion 7.0 --lc MAXDB76:dblca-se:\*

Adding a SID with ERS support:

control1:~ # ff_sid_adm.pl --op add --pool pool1 –sid S04 --sapversion 7.0 --db ORACLE10:dbs04-se:10.10.12.100 --sap ers:12:ers12s04:\*:ers12s04-se:\* --sap ci:13:cis04:10.10.11.102:cis04-se:10.10.12.102

Removing an entire SID (including its instances):

%> ff_sid_adm.pl –-op del –-sid SHT –-pool Otto

Removing an Application Server:

%> ff_sid_adm.pl –-op del –-sid SHT –-pool Otto –-sysnr 01

12.3 Cloning a SAP SID into a Different Pool

12.3.1 Script: ff_clone_sid.pl The script ff_clone_sid.pl allows users to clone (basically copy) an entire SID from one pool to another. It needs to be clearly understood that only the FlexFrame-specific administrational data in the LDAP server as well as the required information for the oper-ating system’s naming services are copied and/or added to the LDAP database. Any additional work (like copying SAP/database binaries and database content) will not be performed by this tool and needs to be performed, as well.

Cloning a SAP SID into a Different Pool Administrating SAP Systems


Also, if the script determines conflicts in userIDs, groupIDs, service entries etc., it will not perform further steps, in order to avoid potential damage done to the target pool. Users would then have to add those data to the target pool manually, which clearly requires some knowledge of the LDIF format and usage of tools such as ldapsearch and ldap-modify. In addition to that, userIDs, groupIDs and other entries might have to be adapted, accordingly.

Synopsis

ff_clone_sid.pl --sid=<SID_name> --srcpool=<pool_name>

--trgtpool=<pool_name>

12.3.2 Script: ff_change_id.pl This script allows changing user IDs or group IDs in LDAP. This is meant to support the cloning of SAP systems to other pools. In some cases - when IDs are already in use - it may be required to change them, after cloning a system into a different pool.

Synopsis

ff_change_id.pl --pool=<pool_name>

[{--uid=<id> <user_name>|--gid=<id> <group_name>}]

Example:

ff_change_id.pl --pool=pool1 --uid=501 orasht

Data Transfer when Cloning an SID

After cloning the information in the LDAP database with the script ff_clone_sid.pl, it is necessary to copy SAP database binaries and SAP database content. The copy will be done from the Control Node as user root. If the database size is greater than 50 GB it is recommended to use the ONTAP command ndmpcopy on the NetApp Filer (see Exam-ple 3 below) or the DART command server_archive on the EMC Celerra to copy the database content without using the network connections.

Please check carefully that no data will be overwritten in the target pool when copying data from the source pool!

The following directories have to be copied from the source pool to the target pool:

/FlexFrame/volFF/pool-<pool_name>/<dbtype>/<OS>/<SID> /FlexFrame/volFF/pool-<pool_name>/sap/sapmnt/<SID> /FlexFrame/volFF/pool-<pool_name>/sap/usr_sap/<SID> /FlexFrame/volFF/pool-<pool_name>/sap/home_sap/<sid>adm

Administrating SAP Systems Cloning a SAP SID into a Different Pool


/FlexFrame/sapdata/<pool_name>/<SID> /FlexFrame/saplog/<pool_name>/<SID> /FlexFrame/volFF/pool-<pool_name>/oracle/<OS>/client/* (Only necessary if the Oracle client software is not already installed in the target pool.) You have to edit the file /FlexFrame/scripts/ora_listener_names.

This file is pool-dependent. Add a new line as shown below:

<ora_listener_name> DE2:LISTENER_DE2 QA2:LISTENER_QA2 PR2:LISTENER_PR2 PRD:LISTENER_PRD # <-- add this line If SAPDB is used, please copy also /FlexFrame/volFF/pool-<pool_name>/sap/home_sap/sqd<sid>.

<pool name> Name of source or target pool

dbtype oracle or sapdb

OS Linux or SunOS

SID The upper-case three-digit system ID

sid The lower-case three-digit system ID Example 1 (Oracle-DB on Solaris):

SID = OSI Source pool = p1 Target pool = p2

control1:/ # cd /FlexFrame/volFF/pool-p1/oracle/SunOS control1:/FlexFrame/volFF/pool-p1/oracle/SunOS # cp -r OSI ../../../pool-p2/oracle/SunOS control1:/FlexFrame/volFF/pool-p1/oracle/SunOS # cd ../../sap control1:/FlexFrame/volFF/pool-p1/sap # cp -r sapmnt/OSI ../../pool-p2/sap/sapmnt control1:/FlexFrame/volFF/pool-p1/sap # cp -r usr_sap/OSI ../../pool-p2/sap/usr_sap control1:/FlexFrame/volFF/pool-p1/sap # cp -r home_sap/osiadm ../../pool-p2/sap/home_sap control1:/FlexFrame/volFF/pool-p1/sap # cd /FlexFrame/sapdata/p1 control1:/FlexFrame/sapdata/p1 # cp -r OSI ../p2 control1:/FlexFrame/sapdata/p1 # cd ../../saplog/p1 control1:/FlexFrame/saplog/p1 # cp -r OSI ../p2

Cloning a SAP SID into a Different Pool Administrating SAP Systems


Example 2 (SAPDB-DB on Linux):

SID = MLI Source pool = p1 Target pool = p2

control1:/ # cd /FlexFrame/volFF/pool-p1/sapdb/Linux control1:/FlexFrame/volFF/pool-p1/sapdb/Linux # cp -r MLI ../../../pool-p2/sapdb/Linux control1:/FlexFrame/volFF/pool-p1/sapdb/Linux # cd ../../sap control1:/FlexFrame/volFF/pool-p1/sap # cp -r sapmnt/MLI ../../pool-p2/sap/sapmnt control1:/FlexFrame/volFF/pool-p1/sap # cp -r usr_sap/MLI ../../pool-p2/sap/usr_sap control1:/FlexFrame/volFF/pool-p1/sap # cp -r home_sap/mliadm ../../pool-p2/sap/home_sap control1:/FlexFrame/volFF/pool-p1/sap # cp -r home_sap/sqdmli ../../pool-p2/sap/home_sap control1:/FlexFrame/volFF/pool-p1/sap # cd /FlexFrame/sapdata/p1 control1:/FlexFrame/sapdata/p1 # cp -r MLI ../p2 control1:/FlexFrame/sapdata/p1 # cd ../../saplog/p1 control1:/FlexFrame/saplog/p1 # cp -r MLI ../p2

Example 3 using ndmpcopy (Oracle-DB on Solaris) for NetApp Filers only:

Filer IP = 192.168.10.203 (Source and Target identical) SID = OSI Source pool = p1 Target pool = p2

control1:/ # rsh 192.168.10.203 ndmpcopy -da root:password \ /vol/volFF/pool-p1/oracle/SunOS/OSI \ 192.168.10.203:/vol/volFF/pool-p2/oracle/SunOS/OSI control1:/ # rsh 192.168.10.203 ndmpcopy -da root:password \ /vol/volFF/pool-p1/sap/sapmnt/OSI \ 192.168.10.203:/vol/volFF/pool-p2/sap/sapmnt/OSI control1:/ # rsh 192.168.10.203 ndmpcopy -da root:password \ /vol/volFF/pool-p1/sap/usr_sap/OSI \ 192.168.10.203:/vol/volFF/pool-p2/sap/usr_sap/OSI control1:/ # rsh 192.168.10.203 ndmpcopy -da root:password \ /vol/volFF/pool-p1/sap/home_sap/osiadm \ 192.168.10.203:/vol/volFF/pool-p2/sap/home_sap/osiadm control1:/ # rsh 192.168.10.203 ndmpcopy -da root:password \ /vol/sapdata/pool-p1/OSI 192.168.10.203:/vol/sapdata/pool-p2/OSI control1:/ # rsh 192.168.10.203 ndmpcopy -da root:password \ /vol/saplog/pool-p1/OSI 192.168.10.203:/vol/saplog/pool-p2/OSI

Administrating SAP Systems Multiple NAS Systems and Multiple Volumes


12.3.3 Changing User and Group IDs after Cloning The ff_change_id.pl script allows changes of user IDs or group IDs in LDAP. This utility is meant to support the cloning of SAP systems to other pools. In some cases - when IDs are already in use - it may be required to change them, after cloning a system into a different pool.

Synopsis

ff_change_id.pl --pool=<pool_name>

[{--uid=<id> <user_name>|--gid=<id> <group_name>}]

Example:

# ff_change_id.pl --pool=pool1 --uid=501 orasht

12.4 Multiple NAS Systems and Multiple Volumes During the installation process, FlexFrame assumes that there is one NAS system with sapdata and saplog volumes. Larger installations may require more than one NAS system or multiple volumes on the same NAS system.

It is possible to distribute SAP databases across multiple NAS systems and multiple vol-umes under the following conditions:

1. All NAS systems where entered in the FlexFrame Planning tool, prior to installation of the FlexFrame landscape.

2. The software for SAP and the database are always located in a centralized volume volFF of the first NAS system. No distribution here.

3. For each SID you can assign a sapdata volume for the database's data files. This sapdata volume can be shared with other SIDs or solely for this SID.

4. For each SID you can assign a saplog volume for the database's online redolog files. This saplog volume can be shared with other SIDs or solely for this SID.

Multiple NAS Systems and Multiple Volumes Administrating SAP Systems


12.4.1 NetApp Filer The volumes must be created manually on the Filer, e.g. like:

filer2> vol create dataC11 10 filer2> vol create logC11 4

Here, dataC11 and logC11 are the new names of the volumes and 4 and 10 are the numbers of disks to be used.

We recommend using the volume names sapdata and saplog (if on a different Filer) or data<SID> and log<SID> if SID specific volumes on the same Filer.

You may use FlexVols (ONTAP 7G) or regular volumes.

The following options must be set for the volume:

filer2> vol options dataC11 nosnap on filer2> vol options dataC11 nosnapdir on filer2> vol options dataC11 minra on filer2> vol options dataC11 no_atime_update on filer2> vol options logC11 nosnap on filer2> vol options logC11 nosnapdir on filer2> vol options logC11 minra on filer2> vol options logC11 no_atime_update on

Next, create qtrees for each FlexFrame pool which will store data and logs in those vol-umes:

filer2> qtree create /vol/dataC11/pool1 filer2> qtree create /vol/logC11/pool1

If you use more than the first Filer, make sure it is reachable using, e.g.:

control1:~ # ping filer2-st PING filer2-st (192.168.10.203) from 192.168.10.201 : 56(84) bytes ofdata. 64 bytes from filer2-st (192.168.10.203): icmp_seq=1 ttl=255 time=0.117 ms 64 bytes from filer2-st (192.168.10.203): icmp_seq=2 ttl=255 time=0.107 ms 64 bytes from filer2-st (192.168.10.203): icmp_seq=3 ttl=255 time=0.103 ms

Now the LDAP database has to be told that this SID is not using the default (first) Filer and sapdata/saplog volumes:

control1:~ # ff_sid_mnt_adm.pl --op=add --pool=pool1 --sid=C11 --sapdata=filer2:/vol/dataC11/pool1/C11 --saplog=filer2:/vol/logC11/pool1/C11

Now the volumes on the Control Nodes need to be mounted. To do so you should add the following lines to each Control Node's /etc/fstab:

Administrating SAP Systems Multiple NAS Systems and Multiple Volumes


filer2-st:/vol/vol0 /FlexFrame/filer2-st/vol0 nfs nfsvers=3,rw,bg,udp,soft,nolock,wsize=32768,rsize=32768

filer2-st:/vol/dataC11/pool1 /FlexFrame/filer2-st/pool1/dataC11 nfs nfsvers=3,rw,bg,udp,soft,nolock,wsize=32768,rsize=32768

filer2-st:/vol/logC11/pool1 /FlexFrame/filer2-st/pool1/logC11 nfs nfsvers=3,rw,bg,udp,soft,nolock,wsize=32768,rsize=32768

Repeat the sapdata and saplog-lines for each pool, if there's more than one pool for those volumes.

Use the volume name for the last directory in the mount point.

Now we need the mount points:

control1:~ # mkdir -p /FlexFrame/filer2-st/vol0 control1:~ # mkdir -p /FlexFrame/filer2-st/pool1/dataC11 control1:~ # mkdir -p /FlexFrame/filer2-st/pool1/logC11

(Again, sapdata and saplog for each pool)

Before mounting the files we need to tell the Filer to export the volumes appropriately:

control1:~ # mount /FlexFrame/filer2-st/vol0 control1:~ # ff_exports.pl --op add --nas filer2-st --path /vol/dataC11/pool1 --option "-sec=sys,rw=192.168.10.0/24,anon=0" control1:~ # ff_exports.pl --op add --nas filer2-st --path /vol/logC11/pool1 --option "-sec=sys,rw=192.168.10.0/24,anon=0"

Save the file.

The network 192.168.10.0/24 must match the Storage LAN segment of pool pool1.

Now you need to make the file re-read its configuration file:

control1:~ # rsh filer2-st exporfs -a

Now we can mount the volumes:

control1:~ # mount -a

Before you can install SAP and the database on those volumes, some folders for the SID in question have to be created in advance. To do so, run the following command for each SID (replace "pool1" with your pool name and "C11" with your SID:

control1:~ # ff_setup_sid_folder.sh pool1 C11

Now you can continue with the SAP installation.

Multiple NAS Systems and Multiple Volumes Administrating SAP Systems


12.4.2 EMC Celerra Regard a Celerra with at least two active data movers: What is to be done, if volFF is on one data mover, and the SAP volumes should be on the other ?

– Define the volumes as described in the Installation Guide, but you have to export the corresponding file systems on the data mover 2 (=server_3), e.g.:

server_export server_3.../dataC11 server_export server_3.../logC11

– For the data mover, which is to acess the SAP volumes, you will have to define the storage VLANs of all FlexFrame pools, that should have access to these volumes. In the example it is pool1 and VLAN-ID 11.

Now the LDAP database has to be told that this SID is not using the default NAS and sapdata/saplog volumes:

control1:~ # ff_sid_mnt_adm.pl --op=add --pool=pool1 --sid=C11 --sapdata=filer2:/vol/dataC11/pool1/C11 --saplog=filer2:/vol/logC11/pool1/C11

Now the volumes on the Control Nodes need to be mounted. To do so you should add the following lines to each Control Node's /etc/fstab:

<Datamover2-st>:/vol/dataC11/pool1 /FlexFrame/<Datamover2-st>/pool1/dataC11 nfs nfsvers=3,rw,bg,udp,soft,nolock,wsize=32768,rsize=32768

<Datamover2-st>:/vol/logC11/pool1 /FlexFrame/<Datamover2-st>/pool1/logC11 nfs nfsvers=3,rw,bg,udp,soft,nolock,wsize=32768,rsize=32768

Repeat the sapdata and saplog lines for each pool, if there's more than one pool for those volumes.

Use the volume name for the last directory in the mount point.

Now we need the mount points:

control1:~ # mkdir -p /FlexFrame/<Datamover2-st>/pool1/dataC11 control1:~ # mkdir -p /FlexFrame/<Datamover2-st>/pool1/logC11

(sapdata and saplog for each pool again)

Now we can mount the volumes:

control1:~ # mount -a

Administrating SAP Systems Upgrading a SAP System


Before you can install SAP and the database on those volumes, some folders for the SID in question have to be created in advance. To do so, run the following command for each SID (replace pool1 with your pool name and C11 with your SID:

control1:~ # ff_setup_sid_folder.sh pool1 C11

If you have several Celerras, the FlexFrame administrator will have to configure the time service and the SSH keys for each of them (see Installation Guide).

Now you can continue with the SAP installation.

12.5 Upgrading a SAP System

12.5.1 Service Port If you plan to upgrade your SAP release, you have to add a special service port (called shadow instance) to LDAP.

SAP shadow service ports are required during an SAP release upgrade. To list, add or remove the service ports in the LDAP database for service entries, you can use this tool.

Synopsis

ff_sap_shadowport.sh [-d] -l -p <pool_name> [-s <sid>]

[-o <port_no>]

ff_sap_shadowport.sh [-d] {-a|-r} -p <pool_name> -s <sid>

[-o <port_no>]

Options

-d Writes debugging information to a log file (see below).

-l Lists all SAP shadow service ports of the pool provided with the -p option. If the option -s is used, only the port of the specified SID is displayed.

-a Adds an entry.

-r Removes an entry.

-p <pool_name> Specifies the name of the pool (e.g. pool1).

-s <sid> Specifies the SAP System ID (SID) by a 3 character string (e.g. C11).

SAP Kernel Updates and Patches Administrating SAP Systems


-o <port_no> Specifies the service port number. The default is 3694 (optional).

Debugging

/tmp/ff_sap_shadowport.DEBUGLOG holds debugging information if option -d was used. In case of problems, please provide this file.

12.5.2 FA Agents Please make sure that the FA Application Agents are stopped on the hosts while you are installing, updating or removing any SAP or database software:

Stop the FA Agent:

/etc/init.d/myAMC.FA_AppAgent stop

Check the status:

/etc/init.d/myAMC.FA_AppAgent status

There should be no running processes listed.

12.6 SAP Kernel Updates and Patches For an SAP kernel update (binary patches), logon to the Application Node with the CI of the SAP system that is to be updated. Please make sure that the FA Application Agents are stopped on the host while you are updating the SAP kernel (see also section “FA Agent” on page 263).

SAP’s OSS note 19466 describes where to find kernel patches and how to handle the installation.

For the installation of SAP ABAP patches or similar, please refer to the SAP documenta-tion. There are no FlexFrame specific changes.


13 Administrating SAP Services In a FlexFrame environment, any type of SAP instance, i.e. database (DB), SAP Central Services (SCS), Java central instance (JC), Central Instance (CI), application instance (APP) or Java-only stack (J), is called an SAP service. The management can be done either web based with the FA WebGUI or script based with the adapted start/stop scripts.

If the SAP ACC is running, SAP services can be managed by ACC (see chapter “SAP ACC” on page 275”).

13.1 Displaying Status of SAP Services

13.1.1 myAMC.FA WebGUI The myAMC.FA WebGUI can be used to display the states of SAP services in a Flex-Frame environment. To see all active SAP services and nodes in the FA WebGUI, the FA Application Agent should be started on all Application Nodes. The displayed active SAP services can be shown system-related or node-related.

The following table shows the node and service states of FlexFrame Autonomy and the corresponding colors of these states in the FA WebGUI:

white

inactive or no further informa-tion

green

normal, everything ok

yellow

warning

red

critival

black

critical

Node states

RUNNING

SWITCH INT

SWITCH EXT

PowerOff

Service states

SHUTDOWN DOWN

WATCH

NOWATCH

UNKNOWN

NULL

RUNNING

Administrating SAP Services Displaying Status of SAP Services


white

inactive or no further informa-tion

green

normal, everything ok

yellow

warning

red

critival

black

critical

STOPPING

STARTING

REBOOT

RESTART RESTARTING

REBOOT REBOOTING

RBGET

SWITCH SWITCHOVER

SWGET

ERROR

If you don't want to configure the parameters in the plain xml files, you may use the FA WebGUI to do this more conveniently. Details are provided in the “FA Agents – Installa-tion and Administration“ manual.

13.1.2 Script: ff_list_services.sh The script ff_list_services.sh lists the status of all installed SAP services.

Synopsis

ff_list_services.sh [-ivcCH] [<pool> ...]

Options

-i Shows inactive services

-v Verbose mode

-c Force use of colors for status information

-C Suppress colors for status information

-H Suppress headers

-h or -? Shows usage

If no pools are specified, services of all pools will be shown.

Starting and Stopping Application Services Administrating SAP Services


13.2 Starting and Stopping Application Services Virtualization of the applications and services demands special measures for starting, stopping, restarting them etc. These measures are cared for by an SAP service script for each service type.

The application or service must not be started directly, e.g. for an SAP in-stance as <sid>adm with startsap, since in this case the interfaces are neither supplied with IP addresses, nor is the service control file maintained. The started application will not work due to the lack of a network connection.

13.2.1 Starting and Stopping SAP Services without Root Privileges

In order to enable user to start and stop SAP services without having root priviliges a sudo (see man sudo) configuration is shipped with the Application Nodes. This configura-tion allows the SAP and the DB administration users to start their respective services (including their virtual hostname’s IP addresses). The user command line requires a pre-fix of sudo in addition to the standard command line (see example below):

> sudo /FlexFrame/scripts/sapdb SHT start

The sudo mechnisms will check against its etc/sudoers if the user (or the responsible group) is allowed to run the command and run it with root privileges.

13.2.2 SAP Service Scripts Each service type has its own script. The service type is part of the script name:

sapascs Start and stop an ABAP-SAP Central Services Instance

sapdb Start and stop Oracle, SAPDB etc

sapci Start and stop an ABAP Central Instance

sapscs Start and stop a SAP Central Services Instance

sapjc Start and stop a Java Central Instance

sapapp Start and stop an ABAP Application Instance (application server)

sapj Start and stop a Java Application Instance

sapers Start and stop ERS instance. The ERS should not run on the same host as the (A)SCS instance

saplc Start and stop LiveCache instance

Administrating SAP Services Starting and Stopping Application Services


The call syntax for sapdb, sapci, sapscs, sapascs, sapjc and saplc is:

<script_name> <sid> <action> The call syntax for sapapp, sapj and sapers is:

<script_name> <id> <sid> <action> The call parameters are:

<id> Distinction of several similar instances of a service type of an SID; 2-digit numerical.

<sid> System ID (SID), 3-digit, in lower case.

<action> The action to be performed with the application or service. Actions are start, stop, restart, status, cleanup, watch and nowatch.

Call the following from the Control Node, using ssh [-t] with reference to the example of sapapp:

ssh <application_node_name> sapapp <ID> <SID> <action> Without the -t option, the standard output is redirected to the logfile /FlexFrame/scripts/log/<script_name>_[<ID>_]<SID>_<action>.log.

Example:

The application server of the system CB1 with the ID 01 is to be started on the node blade3; let us assume that the Control Node is control1.

Execute the following as root in the command shell of control1:

control1:~ # ssh blade3 sapapp 01 cb1 start

13.2.3 SAP Service Script Actions In the following, the SAP service script actions are desribed:

Start This checks whether the application or service in question is already running. If it is running, it is not restarted. It also checks whether required applications or services are running. The required virtual IP addresses are assigned to the relevant interfaces for Client LAN and Server LAN (ifconfig <ifc> <ip-adr> netmask <netmask> up), the application is started and the service control file is written.

Starting and Stopping Application Services Administrating SAP Services


Stop This checks whether the application or service in question is running. If it is not, it is not stopped. The application is terminated, the service control file is deleted and the virtual IP addresses are brought down again (ifconfig <ifc> down).

Status This checks the logical status of the application or service. The functional availability is not tested.

Restart This merges the actions stop, cleanup and start in one call. restart is in-tended for restarting of a malfunctioning application.

Cleanup This kills application processes that are still running and deletes occupied resources such as shared memory, semaphores and the message queue.

Note that this action may only be performed after stopping the application has failed.

Nowatch This removes the application from monitoring by the high-availability software (FA Agent) without the application having to be restarted. The application itself retains its current status.

Watch This includes the application again into monitoring by the high-availability software (FA Agent) without the application having to be restarted. The application itself re-tains its current status.

13.2.4 SAP Service Scripts User Exits The FlexFrame start-/stop-scripts for SAP Services (sapdb, sapci, sapapp etc.) provide a user exit as a shell script /FlexFrame/scripts/user_script. If this user_script exist, the FlexFrame start-/stop-scripts will call it twice, 1st in the begin-ning, 2nd at the end of the start-/stop-scripts. This means two phases, a pre-phase and a post-phase. The pre-phase will be run before any action, the post-phase will run after all action. Because of plausibility errors while executing the FlexFrame start-/stop-script, it is possible that the post-phase will be omitted.

A sample user_script is delivered as user_script.TEMPLATE.

To achieve a pool depedent function of user_script, put the script into the pool image and create a symbolic link to it in FlexFrame/scripts. Seen from the Control Node:

Control1: # ln -s /FlexFrame/pooldata/config/scripts_config/ user_script /FlexFrame/scripts/user_script

If no pool dependent function is needed, the script user_script can be located directly in /FlexFrame/scripts.

Administrating SAP Services Return Code of the SAP Service Scripts


13.3 Return Code of the SAP Service Scripts The SAP service scripts issue a return code (exit code). The meaning of this code can be looked up in the sapservice_functions file:

#================================================================ # common exit codes for service scripts #================================================================ no_error=0 # no error wrong_parameter_count=1 # Bit 0, wrong number of parameters plausibility_error=2 # Bit 1, plausibility error interface_server_lan_error=4 # Bit 2, error at server lan interface up/down interface_client_lan_error=8 # Bit 3, error at client lan interface up/down service_start_stop_error=16 # Bit 4, error at service start/stop/status/... any_error=32 # Bit 5, any other error san_mount_error=64 # Bit 6, san mount error user_script_error=128 # Bit 7, user_script error

For actions like start, stop, nowatch and watch, the exit code should be no_error normally. For the action status, the exit code is dependent from various factors:

● The installation check against the SAP service may have failed (plausibil-ity_error)

● the SAP service is possibly not running or not running on this host (any_error)

● the SAP service is running on this host (no_error).

In addition to these exit codes, the SAP service scripts print out messages as described in the section “Start/Stop Script Errors” on page 324.

Starting and Stopping Multipe SAP Services Administrating SAP Services


13.4 Starting and Stopping Multipe SAP Services The following scripts are provided for starting and stopping multiple applications and services:

Script name Application Place of execution

start_all_sapservices Initial start of all configured applications

Only on a Control Node

stop_all_sapservices Stopping all running appli-cations


stop_all_sapservices_SID Stopping all running appli-cations of one SID


stop_all_sapservices_local Stopping all running appli-cations on the local node

Only on an Applica-tion Node

Call syntax for pool-dependent scripts:

stop_all_sapservices [<pool>] stop_all_sapservices_SID <SID> [<pool>]

Functionality of the SAP service scripts for multiple services:

start_all_sapservices Initial start script for customer defined sap service landscapes. This can be the whole FlexFrame landscape, a pool, a SAP system landscape with a productive system, a quality assurance system and a development system, or a single SAP system with database, central instance and application instances. This script can be duplicated to create various initial start scripts with different names, e.g. start_all_sapservices_pool2. A sample start_all_sapservices script is delivered as start_all_sapservices.TEMPLATE.

stop_all_sapservices Shutdown script for all running sap services in a dedicated pool. The advantage of this script is that sap services have not to be stopped one-by-one.

stop_all_sapservices_SID Shutdown script for a singe SAP system (one SID) with all belonging sap services like application instances, central instances und database instance in the right order.

stop_all_sapservices_local Shutdown script for all running sap services on one Application Node. This script will be integrated via runlevels. The information base for these stop_all_sapservices* scripts are the /FlexFrame/scripts/log/*_host files.

Administrating SAP Services Removing an Application from Monitoring by FA Agents


13.5 Removing an Application from Monitoring by FA Agents

If the applications or services are started with the scripts for virtualization, they are moni-tored by the FA Agents. If you do not want this, be it for tests, installation or upgrades, you have to inform the high-availability software of this, using the additional parameter nowatch when the application is started.

Example:

The central instance of BW1 is to be started without monitoring by the high-availability software:

blade1 # sapci bw1 nowatch blade1 # sapci bw1 start

or from a Control Node:

control1 # ssh blade1 sapci nowatch control1 # ssh blade1 sapci start

If a running application is to be excluded from monitoring by the high-availability software without being restarted, this is possible, using the nowatch option. The application then retains its current status.

Example:

The central instance of BW1 is to be excluded from monitoring by the high-availability software while running:

blade1 # sapci bw1 nowatch

Alternatively, call the following from a Control Node:

control1 # ssh blade1 sapci bw1 nowatch

If a running application is to be included (again) into monitoring by the high-availability software without being restarted, this is possible using the watch option. The application then retains its current status.

Example:

The central instance of BW1 is to be included into monitoring by the high-availability soft-ware while running:

blade1 # sapci bw1 watch

or from a Control Node:

control1 # ssh blade1 sapci bw1 watch

Service Switchover Administrating SAP Services


13.5.1 Stopping and Starting an Application for Upgrades Using r3up

The r3up upgrade control program starts and stops the central instance or application server in various upgrade phases, using the conventional startsap and stopsap. The start and stop scripts for visualization are therefore left out.

Suitable measures must be taken to ensure that the virtual host name for <sid>adm is always available and that the interfaces are supplied with the correct IP addresses. In addition, the application has to be removed from monitoring by the high-availability soft-ware.

Note that for the following solution steps, the application must have been started correctly beforehand with start and stop scripts for virtualization, so that the matching virtual host name is set by default in the $HOME/hostname_default script.

● Remove the application from monitoring by the high-availability software and run it.

● r3up can now shut down and start the application directly with stopsap and startsap; the IP addresses remain assigned to the interfaces and the host name is retained for <sid>adm without any changes on the correct virtual host name.

● Include the application again into monitoring by the high-availability software after the upgrade.

13.6 Service Switchover There is no command line tool to enforce a direct switchover. To switch over an SAP service like a Central Instance, Database Instance, Application Instance etc., it is required to first stop this SAP service on the actual Application Node and to start this SAP service on the destination Application Node thereafter.

Examples:

The central instance of OL4 from pool pool1 is running on klinge4 and should switch-over to klinge5.

First, check whether the Central Instance OL4 is running on klinge4:

For a first overview to see which SAP service should be running on which Application Node in the pool pool1, run

control1 # view_hosts pool1 app_44_ol4 should be running on klinge4 app_56_ml4 should be running on Baby_4 ci_ml4 should be running on RX300-01 ci_ol4 should be running on klinge1

Administrating SAP Services Service Switchover


ci_osm should be running on Baby_3 db_osm should be running on Baby_3 db_ml4 should be running on RX300-01 db_ol4 should be running on klinge4

The output should be means that the script view_hosts does not really check the state of the SAP services. It only checks the FlexFrame/script/log/*_host files which are created immediately after a successfully start of the corresponding SAP ser-vice.

For a detailed status information you can use:

control1 # ssh klinge4 sapci ol4 status

The exit code from our SAP service script will be “0” in this case. For any other errors while executing our SAP service script, it will return an exit code not equal “0”.

control1:/ # ssh klinge1 sapci ol4 status Central-Instance OL4 should be running on this host. Central-Instance OL4 has really running processes on this host. client_lan interface vlan2002:1 10.1.7.151 is already up. server_lan interface vlan2006:2 172.16.4.151 is already up. ciol4: klinge1 /FlexFrame/scripts/sapci done! ciol4: exit_code=0

Another Example on solaris:

control1:~ # ssh Baby_3 sapci osm status Central-Instance OSM should be running on this host. Central-Instance OSM has really running processes on this host. client_lan interface fjgi2002000:2: 10.1.7.106 is already up. server_lan interface fjgi2006000:3: 172.16.4.106 is already up. ciosm: Baby_3 /FlexFrame/scripts/sapci done! ciosm: exit_code=0

Stop the running Central Instance OL4 on klinge4:

control1 # ssh klinge4 sapci ol4 stop

And now start the stopped Central Instance OL4 on the new destination klinge5:

control1 # ssh klinge5 sapci ol4 start

The manual switch over should now be complete.


14 SAP ACC This chapter describes the administration and use of ACC in a FlexFrame environment.

Below, you find a brief description of what is to be done. For detail on how it is to be done, please refer to the following documents:

● ACCImplementation.pdf ● ACCSecurity.pdf ● ACCCustomizing.pdf

14.1 Integration of New Servers, Pools and SAP Services

This section summarizes how to integrate new hosts and SAP services into ACC. De-tailed information how to do this can be found in the Installation Guide for SAP Adaptive Computing Controller.

The following sections are referring to the Installation Guide for ACC from FSC.

Assumption

Running ACC and SolMan services.

14.1.1 Integration of New ACC Pools (=FF groups) For information on creating a new ACC pool, please see chapter “Configure ACC Pools”.

14.1.2 Integration of New Servers For information on integrating new servers in a FlexFrame environment with ACC Agents started, please see chapter “ACC Agents”.

Integration of a new Server to ACC

For information on making a new registered server adaptive, please see chapter “Adap-tive Enabled Computing Nodes”.

SAP ACC User Administration


14.1.3 Integration of new SAP Services There are different procedures to integrate new SAP services into ACC, depending on ABAP or Java services.

14.1.3.1 Moving SAP Services into SLD

See chapter “Register managed Application Services into SLD”.

SolMan

Transaction smsy_setup started (see chapter “Activate and check Transfer from SLD to Solution Manager”).

Making services adaptive (see chapter “Integration of SAP Services”).

ACC

Click on refresh button.

New services should be visible in the ACC WebGUI.

14.2 User Administration ACC is a single point of control component which needs to be secured against unauthor-ized access. Details are described in the “ACCSecurity.pdf”.

For administration of ACC users see the manual chapter “User Administration for ACC Users”.

14.3 Usage of ACC

The ACC WebGUI has a Help menu integrated on the right side of the status bar. For detailed information use the Help menu.

14.3.1 Displaying Status of SAP Services ● After starting the ACC WebGUI see chapter “Start of ACC”. ● The Logical Landscape appears. ● Click on the desired pool or select “Expand All” in Option menue of tray. ● You should see the status of each integrated SAP service.

Usage of ACC SAP ACC


14.3.2 Starting SAP Services ● After starting of the ACC WebGUI see chapter “Start of ACC”. ● The Logical Landscape appears. ● Click on the desired pool or “Expand All” in Option menue of tray. ● Choose an SAP service by clicking on the instance name. ● Select the desired service by clicking on the blue quad. ● Select a server from the Selected Server assortment. ● Click on the Start Application Service button to start. ● Confirm the upcoming window and check the Status.

14.3.3 Stoping SAP Services ● After starting the ACC WebGUI, see the chapter “Start of ACC”. ● The Logical Landscape appears. ● Click on the desired pool or “Expand All” in Option menue of tray ● Choose a SAP service by clicking on the instance name. ● Select the desired service by clicking on the blue quad. ● Click on the “Stop Application Services” button to stop. ● Confirm the upcoming window and check the status.

14.3.4 Relocating SAP Services ● After starting the ACC WebGUI, see the chapter “Start of ACC”. ● The Logical Landscape appears. ● Click on the desired pool or “Expand All” in Option menue of tray ● Choose a SAP service by clicking on the instance name ● Select the desired service by clicking on the blue quad. ● Select a server from the “Selected Server” assortment ● Click on the “Relocate Application Service” button to relocate ● Confirm the upcoming window and check the Status

14.3.5 Archiving the ACC Log It is also possible to archive the logging information: Navigate in the ACC WebGUI to “Technical Settings” => “Archive Log”

Delete, Archive and Retrieve are the possible functions. For more information use the Help button in the status bar.


15 Configuring FA Agents Operation of the FA Agents does not necessarily require individual parameterization. Usable parameter files are available after installation has taken place. For productive use, the values must have been tested and, if necessary, adjusted to the requirements and the start, stop, ping and restart times on the customer system for the services monitored in myAMC.FA.

The FlexFrame Autonomy solution is configured to use files contained in the directory /opt/myAMC/vFF/vFF_<pool_ name>/config on the Control Node.

The files are available in XML format.

TrapTargets.xml Trap target (pool-dependent). In this file you can configure the trap receivers to which messages are sent.

myAMC_FA_Groups.xml Groups (pool-dependent). The group affiliation is configured in this file.

myAMC_FA.xml FlexFrame Autonomy (pool-dependent), settings for the autonomous reactions.

myAMC_FA_ACC.xml ACC Connector (pool-dependent), settings for the interface to SAP ACC.

myAMC_FA_GUI.xml FA WebGUI (pool-dependent), settings for the FlexFrame WebGUI.

myAMC_FA_SD_Sec.xml FA shutdown security (pool-dependent), settings for the power shutdown.

After the installation, the default files for all configurations are available as <xxxx>-default.xml.

Example:

In its as-supplied status, the myAMC_FA_default.xml file is identical to the myAMC_FA.xml file. It can be used to restore a modified or destroyed myAMC_FA.xml file.

Details are provided in the “FA Agents – Installation and Administration“ manual.

Configuring FA Agents Groups


15.1 Groups

15.1.1 General FlexFrame offers advanced functions for partitioning a FlexFrame environment into ser-vice or customer specific server pools and groups. This may be interesting for large installations or application service providers.

Server pools

On FlexFrame 3.2, a pool is a number of Application Nodes belonging to the same de-partment (or customer) with exclusive hardware requirements. FlexFrame systems can be divided into pools. Each FlexFrame system consists of at least one pool. Within a pool, all servers may communicate with each other, but not with the systems of other pools.

Servers of different pools can use different copies of the OS and can be separated into different network segments.

The autonomous reactions of FlexFrame Autonomy are always pool-related. Pools can be configured in LDAP by using the FF administration tools. Details about the pooling within FA Agents are provided in the “FA Agents – Installation and Administration“ man-ual. The FA configuration file myAMC_FA_Pools.xml is used as a cache file in the case LDAP is not responding.

Server groups

Within a server pool, various types of hardware can be used with different characteristics, such as operating system and architecture, number of CPUs and RAM size. The bulk of servers can be divided into groups (pool groups) of servers with similar operating sys-tems and hardware performance. This may be very useful for groups of high-performance database servers or groups of medium-performance application servers. Each pool con-sists of at least one group.

Each SAP application running in a pool can use one or more servers in one or more groups of servers in the same pool. Each instance of this application runs in a selected group. In case of a failure, switchover to a spare server is possible in the same group.

For example, a pair of servers can be divided into a high-performance database server and a smaller application server. The group configuration makes sure that database in-stances run in the group of high-performance servers, while application instances stay on groups of smaller servers without interfering with each other.

All servers in a pool share the same VLAN network segments, even if they belong to different groups. They do not share VLAN segments with other pools, except the Control VLAN.

Groups Configuring FA Agents


The autonomous reactions of FlexFrame Autonomy are always group-related. Groups can be configured in LDAP by using the FF administration tools. Details about the group-ing within FA Agents are provided in the “FA Agents – Installation and Administration“ manual. The FA configuration file myAMC_FA_Groups.xml is used as a cache file in the case LDAP is not responding.

After modifying the group configuration in LDAP, the FA agents must be stopped and restarted in the affected pool. The new group configuration is now used.

15.1.2 Service Classes SAP systems and their services can be classified. Classification on system or service level permits various reaction scenarios.

Systems and services are classified in the myAMC_FA_Groups.xml file. The classifica-tion is pool-related.

The service classes are defined in the group configuration file of a virtual FlexFrame pool.

A service class is defined by the following variables:

System ID ("P46", "O20", ...) Service type ("db", "app", "ci", ...) Service ID ("00", ...) The following attributes are defined in accordance with these variables:

service-priority

service-powervalue This value is currently only provided for information purposes.

15.1.3 Service Priority A priority can be assigned to all services of a service class.

The highest service priority is 1. By default each service is assigned this priority, i.e. if no service classes are defined, all services have priority 1. The higher the number, the lower is the priority of a service. The highest possible number is 1000.

Priority 0 is a special case. Setting priority to 0 for a service class enables the autono-mous functions for a service to be deactivated.

The service priority is evaluated for all autonomous reactions. If, for example, a service of a productive system and a service of a test system are running on the same node and the test system service has been assigned priority 5, this reboot is not executed because the service of the productive system which is running without error has priority 1 and thus a higher priority.

Configuring FA Agents Traps


15.1.4 Service Power Value For this service, the service power value attribute defines a performance value which specifies the maximum performance (SAPS) this service requires.

This attribute is intended for use in future versions of the FA agents in the field of load distribution and load transfer.

15.1.5 Class Creation Rules A service either belongs to the default class which always exists or it can be assigned unambiguously to another class by evaluating the aforementioned variables.

Details are provided in the “FA Agents – Installation and Administration“ manual.

15.2 Traps The FA Agents are able to send SNMP traps to configured trap destinations. The traps contain status information respectively changes of services in the FlexFrame environ-ment. For example there is an SNMP trap if a service is starting. SNMP traps can be used to connect an external system monitoring application to FlexFrame.

15.2.1 General The TrapTargets.xml file contains all the trap destinations, i.e. information which is needed to send SNMP traps. Two parameters are required for each target:

● Host name or IP address ● SNMP community

The community roughly corresponds to a password.

Generally public is configured as the default value.

Details are provided in the ”FA Agents – Installation and Administration“ manual.

15.2.2 Changing the Trap Destinations In a FlexFrame configuration, at least the two Control Nodes of the pool involved must be entered as trap destinations. Other systems may also be entered as trap destinations, however they need to be part of a network reachable from the FF environment.

After the trap destinations have been modified or extended, the FA Control Agent and FA Application Agents must be restarted to apply the modification.

FlexFrame Autonomy Configuring FA Agents


15.3 FlexFrame Autonomy The behavior of the FlexFrame Autonomous Agents can be influenced by a number of parameters.

The FlexFrame Autonomous Agents’ parameters are described in detail in the FlexFrame Autonomy documentation. You should always check that the default values can be used in the configuration involved.

In particular the parameters for startup time, restart time, etc. should be checked, for example, with relation to the database size or memory size as very great differences to the default values are possible here. Every operator has to measure the dynamic values of his/her configuration here and correlate it with the parameters set.

Only when the parameters are set correctly the FlexFrame Autonomous Agents can per-form their tasks securely and reliably. The information to be configured relates to the following components:

● General parameters ● Node-related parameters ● Service-related parameters ● Path configurations ● Power shutdown configuration

15.3.1 General Parameters The list below describes a selection of the parameters which influence the behavior of the FlexFrame Autonomous Agents. Further information on parameterization with corresponding sample scenarios is provided in the manual ”FA Agents – Installation and Administration“.

CheckCycleTime This parameter defines the cycles in which the internal detector modules of the FA Agents supply results and the rule modules evaluate the status derived from these. The parameter value may not be less than the minimum processing time which the detector modules, rule modules and reaction modules require to process a cycle. The default value in the as-supplied status is 10 seconds. The parameter value must also always be at least 1/3 of the lifetime of the MonitorAlerts. In the FlexFrame standard installation the lifetime of the MonitorAlerts is 30 seconds.

LivelistWriterTime This parameter defines the intervals at which the FA Agents must generate a Livelist. It is specified in seconds.

ControlAgentTime This parameter specifies how often the Control Agent checks the Livelists of the Ap-plication Agents. The parameter should thus be about the same as the LiveList-WriterTime.

Configuring FA Agents FlexFrame Autonomy


MaxHeartbeatTime This parameter specifies the maximum time which may elapse between two Livelist entries of an Application Agent before the Control Agent intervenes. The intervention is a check if the Application Node is alive. If not, there will be an external switchover to another Application Node. The MaxHeartbeatTime must therefore always be greater than the ControlAgentTime and the LivelistWriterTime. In practice the factor of 3 between LivelistWriterTime and MaxHeartbeatTime has proved practical.

MaxRebootTime This parameter specifies the maximum time which may elapse between two Livelist entries of an Application Agent before the Control Agent intervenes during a reboot of this Application Node.

MaxFailedReachNumber Number of tries by the FA Control Agent to reach the Application Node after the ap-plication has exceeded the MaxHeartbeatTime. After this number of tries, an ex-ternal switchover is initiated. This parameter specifies how often the Control Agent attempts to reach a node after the MaxHeartbeatTime has been exceeded before an external switchover is initiated.

15.3.2 Node-Related Parameters Node_MaxRebootNumber

This parameter specifies how many consecutive reboots may be performed to re-store a service. If 3 is specified, the Application Agent attempts to make the system available again with up to three reboots. Keep in mind that a reboot is also unsuc-cessful if the system could not be restored within the MaxRebootTime set. In the event of reboot problems, the MaxRebootTime parameter must therefore also al-ways be checked and compared with the reboot time actually needed.

Node_MaxSwitchOverNumber This parameter specifies how many consecutive switchovers may be performed to restore a service.

Node_SwitchOverServiceStartDelayTime This parameter specifies the waiting time for starting of services on the target Appli-cation Node during an internal switchover. This waiting time is necessary, to avoid er-rors caused by switch IP caching. Therefore, the described parameter has to be lar-ger than the caching time of the used network switch, otherwise the starting of a ser-vice could fail.

Node_PowerDownTime This parameter specifies the maximum time a Control Agent will wait for a node to shut down before the services are started on a spare node by means of a switchover.



Node_CheckAvailabilityTime This parameter specifies the maximum time a Control Agent will wait for complete execution of the Node_CheckAvailabilityCommand to be completed. If no posi-tive acknowledgment from the Application Agent is received in that time, the node is regarded as unavailable.

Node_SendTrapsAllowed This parameter releases or blocks the sending of node traps.

Node_RebootCommand This parameter specifies which command is executed when the Application Agent ini-tiates a reboot. Normally this is a shutdown with a subsequent reboot.

Node_ShutdownCommand This parameter specifies which command is executed when the Application Agent ini-tiates a switchover. Normally this is a shutdown without a subsequent reboot.

Node_PowerDownCommand This parameter specifies which command is executed by the Control Agent before an external switchover is initiated. In this way it is ensured that the services on the node being switched over are really stopped and can be taken over without any problem by other nodes. The Control Agent waits at most for the period specified with the Node_PowerDownTime parameter before it continues with the switchover.

Node_CheckAvailabilityCommand This parameter specifies which command is executed by the Control Agent to check the availability of a node. A return value of 0 is interpreted as a positive result, every other return value as negative. The Control Agent waits at most for the period speci-fied with Node_CheckAvailabilityTime. If the command has not been executed completely by then, it is assumed that the test is negative, i.e. the node is no longer available, resulting in an external switchover.

Node_RemoteExecutionCommand This parameter specifies which command the Control Agent puts ahead of a com-mand to be executed on another node. This is used, for example, to start or stop a service remotely on an Application Node. Normally ssh is used here.



15.3.3 Service-Related Parameters The following parameters can be set individually for each service type or for multiple services simultaneously. This is also a result of the hierarchical structure of the parameter file. In the parameter file there is also an option for configuring the values for the DB, CI, APP, SCS, JC and J services individually. The value of the default service is used for any value which is not service-specific.

● Service_EnableMonitoring ● Service_SendTraps ● Service_MaxRestartNumber ● Service_TrapSendDelayTime ● Service_ReactionDelayTime ● Service_MaxStartTime ● Service_MaxStopTime

The dynamic behavior of the FA Application and Control Agents depends very much on the values in the configuration file and the physical conditions. You should therefore check very carefully that the relation between certain values is secure and application-oriented.

Service_EnableMonitoring This parameter defines whether monitoring is enabled or disabled for the service type in question.

Service_SendTraps This parameter releases or blocks the sending of service traps.

Service_MaxRestartNumber This parameter defines how many attempts are made to restart a failed service. This value can be configured individually for each service type. The value is typically in the range 1 to 10. The value 0 means that no attempt is made to restart a failed ser-vice. If reboots are permitted on the node, failure of a service leads directly to a re-boot.

Service_TrapSendDelayTime This parameter defines the send delay time for the service traps.

Service_ReactionDelayTime This parameter interacts directly with CheckCycleTime. It can be set individually for each service type. This time defines how long the triggering of a reaction is delayed after a failure has been detected.

Examples:

CheckCycleTime = 10 sec; ServiceReactionDelayTime = 30 sec

In this example a failed service is detected in a cycle. However, the reaction only takes place after 30 seconds. The failure must therefore have been identified as a



failure over at least three detection cycles. This allows preventing a detection error resulting in an incorrect reaction.

CheckCycleTime = 10 sec; ServiceReactionDelayTime = 0 sec

In this example the required reaction takes place immediately in the cycle in which the problem was detected.

Service_MaxRestartTime This parameter defines the maximum time which may be required for a service type in the event of a restart. If this time is exceeded, a second or further attempt is made in accordance with Service_MaxRestartNumber. Thus if the time is selected to short for the service to be monitored and the hardware used, i.e. the service requires longer to restart than permitted by Service_MaxRestartTime, a problem situation is triggered incorrectly.

Service_MaxStartTime This parameter defines how long a service may take to start up. If this time is ex-ceeded, the Agent interprets the service as not started and initiates further reactions.

Service_MaxStopTime This parameter defines how long a service may take to stop. If this time is exceeded, the Agent interprets the service as not stopped and initiates appropriate reactions.

Service_PingVirtualServiceInterface This parameter defines whether the associated virtual FlexFrame service interface is pinged to determine the availability of a service. If it is set to 0, the virtual LAN inter-faces of the client and server network are not queried. Interface availability then has no influence on the status change of a service.

15.3.4 Path Configuration The path configuration is used to define the directories in which the FlexFrame Autonomy components store their various work files.

A FlexFrame Autonomy solution stores a range of information, such as files with display information for the WebGUI and logging information to be used for support when this is required, in various files.

To ensure that performance and clarity are retained even in larger configurations, we recommend that you do not modify these settings!

If the suggested path configuration is changed, though, make sure that clarity is still re-tained and no problems arise regarding performance and accessibility.

LiveListLogFilePath This parameter specifies the directory in which the Livelist is stored.



LiveListXmlFilePath This parameter specifies the directory in which the XML representation of the Livelist is stored. This file is required by the FA WebGUI. The parameter should contain the same path as ServicesXmlFilePath.

ServicesXmlFilePath This parameter specifies the directory in which the XML representation of the ser-vices list is stored. These files are required by the FA WebGUI. The parameter should contain the same path as LiveListXmlFilePath.

ServicesListFilePath This parameter specifies the directory in which the services list files are stored.

ServicesLogFilePath This parameter specifies the directory in which the services log files are stored.

RebootListFilePath This parameter specifies the directory in which the reboot files are stored. These files contain a list of all services which must be restored after a reboot.

SwitchOverListFilePath This parameter specifies the directory in which the switchover files are stored. These files contain a list of all services which must be restored on another node after a switchover.

PerformanceFilePath This parameter specifies the directory in which the performance files are stored. These files contain measured values for performance data.

SAPScriptFilePath This parameter specifies the directory in which the start and stop scripts for the SAP services (sapdb, sapci, sapapp etc.) can be found. The default path (/opt/myAMC/scripts/sap) is normally a symbolic link to the actual script direc-tory.

ControlFilePath This parameter specifies the directory of the control files (<service type><service ID><service SID>_host) generated by the start/stop scripts.

BlackboardFilePath This parameter specifies the directory in which the BlackBoard file can be found. Commands can be entered in it that are executed by the FA Application Agents.

GroupConfigFile This parameter specifies the file in which the group affiliation is configured.

Power Management (On/Off/Power-Cycle) Configuring FA Agents


15.4 Power Management (On/Off/Power-Cycle) Details are provided in the “FA Agents – Installation and Administration“ manual.

15.4.1 General The power shutdown concept of FlexFrame Autonomy provides an easy-to-configure method for implementing secure shutdown of various hardware platforms. Various blade, PRIMERGY and PRIMEPOWER systems (Midrange, Enterprise) can be installed simul-taneously in a FlexFrame. Each of these systems has different requirements which must be taken into consideration for the power shutdown.

The FlexFrame Autonomy Control Agents make direct use of the Shutdown Agents from the PRIMECLUSTER shutdown facility. These are part of the high-availability solution PRIMECLUSTER (PCL) which is used on all Control Nodes of a FlexFrame system.

Normally, these agents are provided with their configuration information in the course of the PCL configuration. However, in a FlexFrame solution only the Control Node PCLs are configured; no configuration information on any Application Nodes exists in PRIME-CLUSTER.

The FlexFrame Autonomous Agents ascertain this lack of configuration information automatically at runtime and then generate the configuration information required for the agents. Different configuration information is generated in accordance with the type of system for which the power shutdown is performed.

The FlexFrame Autonomous Agents consequently create an agent-specific configuration file:

SA_blade.cfg for Blade configuration

SA_ipmi.cfg for PRIMERGY configuration

SA_rsb.cfg for PRIMERGY configuration

SA_xscf.cfg for PRIMEPOWER configuration

SA_rps.cfg for PRIMEPOWER (Midrange) configuration

SA_scon.cfg for PRIMEPOWER (Enterprise) configuration

Configuring FA Agents Power Management (On/Off/Power-Cycle)


15.4.2 Architecture The figure below provides an overview of the components involved and how these inter-act in a FlexFrame environment.

The PRIMECLUSTER software runs on the two Control Nodes in a FlexFrame solution. The FA Control Agent runs on the active Control Node defined by PCL. The FA Applica-tion Agents provide the Control Agent with information on the computer type and further information which is required for generic creation of configuration files, insofar as this is technically possible and the information is unambiguous and can be ascertained se-curely.

Information which cannot be ascertained generically must be entered manually in the BrutForceShutdown config section of the myAMC_FA.xml file.

For further information on configuring the power shutdown manually, please see the "FA Agents - Installation and Administration" manual.

FlexFrame Autonomy ACC Integration external Switch Over

SA_SCON Console

Application Node PRIMERGY

Application Node PRIMEPOWER Midrange

w/o XSCF

Control Node

Control Node

Application Node PRIMERGY

Application Node PRIMERGY Midrange

with XSCF

PRIMECLUSTERFlexFrame AutonomyControl Agent

SA_Blade Agent

Application Node PRIMEPOWER Enterprise

SA_IPMI Agent

SA_XSCF Agent

CFGBlade

CFGPRIMERGY

CFGPRIMEPOWER

XSCF

CFGPRIMEPOWER

SCONMidrange

CFGPRIMEPOWER

SCONEnterprise

SA_SCON Agent

Linux Kernel Crash Dump (LKCD) Utilities Configuring FA Agents


15.4.3 Configuring User, Password and Community To use agent power shutdown, user, password and community must be defined in the configuration of the FA Agent. This configuration is specified in the pool-specific configu-ration file myAMC_FA_SD_Sec.xml. The entries for user, password and community must be the same as those configured in the Application Nodes.

Details are provided in the manual “FA Agents - Installation and Administration”.

15.4.4 Configuring Management Blades The management blades have to be configured. This is done in the Managementblades configuration section of the myAMC_FA.xml configuration file. For further information on Management Blades configuration, please see the “FA Agents - Installation and Admini-stration“ manual.

15.5 Linux Kernel Crash Dump (LKCD) Utilities In case of kernel panics on a Control Node, the Linux Kernel Crash Dump Utilities (lkcdu-tils) can be configured to dump the memory to a crash dump file in the local file system of the Control Node.

To activate this functionality, edit the /etc/sysconfig/dump file and set DUMP_ACTIVE = “1”. Then run lkcd config from the shell or reboot the Control Node.

However, there is a conflict between PRIMECLUSTER’s Shutdown Facility and the crash dump itself: During a crash dump on one Control Node, PRIMECLUSTER on the other Control Node can’t query the status of its partner and assumes that it has crashed.

Therefore it will issue an IPMI kill command to the Control Node that is currently writing its crash dump. The IPMI power cycle will be done immediately and interrupts the crash dump which will be unusable.

If you expect kernel panics, you have two possibilities:

Disabling the Shutdown Facility

● This can easily be done by unplugging LAN A of the affected Control Node(s).

● Disadvantage: PRIMECLUSTER will not recover any Control Node faults. A kernel panic may result into an inconsistent and/or faulted cluster without application take-over.

This solution is ideal if kernel panics occur regularly or can be reproduced easily.

Configuring FA Agents Linux Kernel Crash Dump (LKCD) Utilities


Adjust the interconnect timeout value.

● On both Control Nodes edit the file /usr/opt/reliant/bin/hvenv.local and set HV_CONNECT_TIMEOUT=360.

● Also edit /etc/sysconfig/dump and set PANIC_TIMEOUT=0 to prevent an automatic reboot after the dump.

● Reboot both Control Nodes one after another.

With this configuration, PRIMECLUSTER will wait 10 minutes and 45 seconds before sending the IPMI kill command. This might not be enough, depending on the memory size of your Control Node.

You can provoke a crash dump by doing the following

control1: # echo 1 > /proc/sys/kernel/sysrq

Press ALT GR + SYSRQ(=PRINTSCREEN) + c to start a dump.

To see what happens, you should change to the syslog console by pressing CTRL + ALT +F10 before starting the dump.

If you like to measure the time needed for a crash dump. Remember to disable the Shut-down Facility before testing. Otherwise it may be interrupted if it takes too much time.

You should see increased activity of the local disks during the dump.

On the next reboot, the crash dump will be saved from the swap devices to /var/log/dump.

To analyze this dump, change to this directory and call lcrash -n <number>, where <number> has to be the suffix of the dump files, for example map.0, dump.0 and kerntypes.0.


16 Data Protection Data protection means backing up data and being able to recover it. You protect the data by making copies of it so that it is available for restoration even if the original is no longer available.

Reasons that businesses need data backups and protection systems include the follow-ing:

● To protect data from accidentally deleted files, application crashes, data corruption, and viruses

● To archive data for future use

● To recover from a disaster

16.1 Backup/Restore for NetApp Filer Depending on your data protection and backup needs, Data ONTAP from Network Appli-ance offers a variety of features and methods to ensure against accidental, malicious, or disaster-induced loss of Filer data. The following list describes the Data ONTAP online features to protect data. For detailed information see the “Network Appliance Online Backup and Recovery Guide”.

Data protection features:

● SnapShotTM

Backup within a volume.

This feature allows you to manually or automatically create, schedule, and maintain multiple backups of data on a volume. Snapshots use only a minimal amount of ad-ditional volume space on your Filer and do not reduce performance significantly.

If a user accidentally modifies or deletes crucial data on a volume with SnapShot enabled, these data can be easily and quickly restored from one of the last snap-shots taken.

● SnapRestore® (license required)

Fast, space efficient restoration of large volumes of data backed up to snapshots.

The SnapRestore feature performs on-request snapshot recovery from snapshots on an entire volume.

Data Protection Backup/Restore for NetApp Filer


● SnapMirror® (license required)

Volume-to-volume and qtree-to-qtree replication.

This feature enables you to periodically create snapshots of data on one volume or qtree, replicate that data to a partner volume or qtree, usually on another Filer, and archive one or more iterations of that data as snapshots.

Replication on the partner volume or qtree ensures quick availability and restoration of data, from the point of the last snapshot, in case the Filer of the original volume or qtree should be unavailable.

If you conduct tape backup and archival operations, you can carry them out on the data that were already backed up to the SnapMirror partner Filer, thus freeing the original Filer of this time-consuming, performance-degrading chore.

● SyncMirrorTM (cluster configuration required)

Continuous mirroring of data to two separate Filer volumes.

This feature allows you to mirror Filer data in real-time to matching volumes physi-cally connected to the same Filer head. In case of unrecoverable disk errors on one volume, the Filer automatically switches access to the mirrored volume.

Filer cluster configuration is required for this feature.

● MetroCluster

SyncMirror functionality, enhanced to provide continuous volume mirroring across distances from 500 meters to 10 kilometers.

16.1.1 Backup of Filer Volumes with NetApp SnapShot

16.1.1.1 Filer Volumes

A snapshot is a frozen, read-only image of the entire Data ONTAP file system that re-flects the state of the file system at the time the snapshot was created. Data ONTAP maintains a configurable snapshot schedule that creates and deletes snapshots auto-matically. Snapshots can also be created and deleted manually. It is possible to store up to 255 snapshots at one time on each Filer volume. You can specify the percentage of disk space that snapshots can occupy. The default setting is 20% of the total (both used and unused) space on the disk.

For the following volumes in a FlexFrame 3.2 environment a snapshot schedule should be configured:

vol0 Root volume with Data ONTAP.

Backup/Restore for NetApp Filer Data Protection


VolFF FlexFrame specific volume with the OS images, FA Agents, scripts and pool specific data (e.g. SAP and DB executables, profiles etc.).

Sapdata FlexFrame specific volume with all data files of the SAP databases.

Saplog FlexFrame specific volume with all log files of the SAP databases.

16.1.1.2 Snapshot Schedules

When you install Data ONTAP on a Filer, it creates a default snapshot schedule. The default snapshot schedule automatically creates one nightly snapshot Monday through Saturday at midnight, and four hourly snapshots at 8 a.m., noon, 4 p.m., and 8 p.m. Data ONTAP retains the two most recent nightly snapshots and the six most recent hourly snapshots, and deletes the oldest nightly and hourly snapshots when new snapshots are created.

For newly created volumes, the default snapshot schedule is not activated. In FlexFrame environments this has to be done for the volumes volFF, sapdata and saplog. Please use the FilerView GUI or the snap command to activate it.

There are three types of schedules that you can set up to run automatically using the snap sched command:

Weekly Data ONTAP creates these snapshots every Sunday at midnight.

Weekly snapshots are called weekly.n, where n is an integer. The most recent weekly snapshot is weekly.0, and weekly.1 is the next most recent weekly snap-shot.

Nightly Data ONTAP creates these snapshots every night at midnight, except when a weekly snapshot is scheduled to occur at the same time.

Nightly snapshots are called nightly.n, where n is an integer. The most recent nightly snapshot is nightly.0, and nightly.1 is the next most recent nightly snapshot.

Hourly Data ONTAP creates these snapshots on the hour or at specified hours, except at midnight if a nightly or weekly snapshot is scheduled to occur at the same time.

Hourly snapshots are called hourly.n, where n is an integer. The most recent hourly snapshot is hourly.0, and hourly.1 is the next most recent hourly snap-shot.



To display the snapshot schedule for one or all volumes on a Filer, enter the following command:

filer> snap sched [volume_name]

Example:

filer> snap sched volFF Volume volFF: 2 6 8@8,12,16,20

The result means that for volume volFF weekly (max. 2), nightly (max. 6) and hourly (max. 8) snapshots will be created. The hourly snapshots will be at 8:00, 12:00, 16:00 and 20:00 h.

You can create different snapshot schedules for different volumes on a Filer. On a very active volume like SAPLOG, schedule snapshots every hour and keep them for just a few hours. For example, the following schedule creates a snapshot every hour and keeps the last three:

filer> snap sched saplog 0 0 3

This schedule does not consume much disk space, and it lets users recover files in re-cent snapshots as long as they notice their mistake within a couple of hours.

16.1.2 Backup of SAP Databases SAP backup and recovery presents several challenges:

● Performance impact on the productive SAP system

Backups typically have a significant performance impact on the productive SAP sys-tem because there is a high load on the database server, the storage system, and the storage network during backups.

● Shrinking backup windows

Since conventional backups have a significant performance impact on the productive SAP system, backups can be made only during times with low dialog or batch activi-ties on the SAP system. It becomes more and more difficult to define an appropriate backup window when the SAP system is used 24x7.

● Rapid data growth

Databases are growing. Rapid data growth together with shrinking backup windows results in ongoing investments in the backup infrastructure – more tape drives, new tape drive technology, faster storage networks etc. Growing databases also require more tape media or disk space for backups. Incremental backups can address these issues, but result in a very slow restore process that usually is not acceptable.



● Increasing cost of downtime and decreasing mean time to recover

The mean time to recover (MTTR) is the time needed to recover from a database failure (logical or physical error). The MTTR cuts into two areas — the time that is necessary to restore the database and time that is necessary to do the forward re-covery of the database. The forward recovery time depends on the number of redo logs that need to be applied after a restore. Unplanned downtime of a SAP system will always have a financial impact on the business process. A significant part of the unplanned downtime is the time that is needed to restore and recover the SAP sys-tem in the case of a database failure. The backup and recovery architecture has to be designed according to the maximum acceptable unplanned downtime.

● Backup and recovery time included in SAP upgrade projects

The project plan for an SAP upgrade always includes at least three backups of the SAP database. The time needed to perform these backups will reduce the total available time for the upgrade process.

A backup and recovery solution using a Network Appliance Filer will always consist of two parts:

1. Backup and restore/recovery using SnapShot and SnapRestore

2. Backup and restore to/from a second location, which can be disks or tape

A backup to a second location will always be based on snapshot copies created on the primary storage. Therefore, the data will be directly read from the primary storage system without generating load on the SAP database server. Several options to back up the data to a second location are possible.

Snapshots can be used to create an online/offline backup of Oracle databases or an online backup of SAPDB/MaxDB databases. With an online backup the Oracle database has to be put in hot backup mode before the snapshot copy is created. With an offline backup the database is shut down before the snapshot copy is created. SAPDB/MaxDB databases are always consistent during online backups, but it is recommended to use the GUI-based Database Manager from SAPDB/MaxDB to create backups.

Backups using the snapshot technology directly are not visible in the SAP sys-tems (using transaction DB24 for database operations). This will cause at least warnings during SAP Early Watch sessions. By using the SAP standard backup tools brbackup and brarchive the backups will be visible in the SAP system.



Example for an Oracle backup script using snapshot (only parts visible)

################################################################## # Name: sapsnap # Create a snapshot # Turning tablespaces in hot backup mode or shut down DB # # Syntax sapsnap <online|offline> SID # # Parameter: online Turns all tablespaces in hotbackup mode before # doing the snapshot. Ends hotbackup mode after # creating the snapshot # offline Shuts down the DB before doing the snapshot # Starts the DB after creating the snapshot # ################################################################## # functions ################################################################## # Sending Database tables in BEGIN BACKUP hotbackup_start (){ echo "sqlplus \"/ as sysdba\" @/tmp/prebackup_${SID}.sql" > /tmp/prebackup_${SID}.sh; echo "set feedback off;" > /tmp/prebackup_${SID}.sql; echo "set pagesize 0;" >> /tmp/prebackup_${SID}.sql; echo "alter system switch logfile;" >> /tmp/prebackup_${SID}.sql;



echo "spool /tmp/hotbackup${SID}start.sql;" >> /tmp/prebackup_${SID}.sql; echo "select 'alter tablespace '||tablespace_name||' begin backup;' from dba_tablespaces;" >> /tmp/prebackup_${SID}.sql; echo "spool off;" >> /tmp/prebackup_${SID}.sql; echo "@/tmp/hotbackup${SID}start.sql;" >> /tmp/prebackup_${SID}.sql; echo "exit;" >> /tmp/prebackup_${SID}.sql; chmod 777 /tmp/prebackup_${SID}.sh su - ora${sid} -c /tmp/prebackup_${SID}.sh rm /tmp/prebackup_${SID}.sql rm /tmp/prebackup_${SID}.sh rm /tmp/hotbackup${SID}start.sql }

# Set the Database Tables back in online mode hotbackup_end (){ echo "sqlplus \"/ as sysdba\" @/tmp/postbackup_${SID}.sql" > /tmp/postbackup_${SID}.sh; echo "set feedback off" > /tmp/postbackup_${SID}.sql echo "set pagesize 0" >> /tmp/postbackup_${SID}.sql echo "spool /tmp/hotbackup${SID}end.sql" >> /tmp/postbackup_${SID}.sql echo "select 'alter tablespace '||tablespace_name||' end backup;' from dba_tablespaces;" >> /tmp/postbackup_${SID}.sql echo "spool off" >> /tmp/postbackup_${SID}.sql echo "@/tmp/hotbackup${SID}end.sql;" >> /tmp/postbackup_${SID}.sql echo "exit;" >> /tmp/postbackup_${SID}.sql chmod 777 /tmp/postbackup_${SID}.sh su - ora${sid} -c /tmp/postbackup_${SID}.sh rm /tmp/postbackup_${SID}.sh rm /tmp/hotbackup${ORACLE_SID}end.sql rm /tmp/postbackup_${ORACLE_SID}.sql }

online_snap (){ # Performing Snapschot from Database only rsh ${DFILER} snap delete ${DATAVOL} sap_online_${SID}_old2 rsh ${DFILER} snap rename ${DATAVOL} sap_online_${SID}_old1 sap_online_${SID}_old2 rsh ${DFILER} snap rename ${DATAVOL} sap_online_${SID}_new sap_online_${SID}_old1 rsh ${DFILER} snap create ${DATAVOL} sap_online_${SID}_new }



For more information about backup of SAP systems, see the NetApp Technical Library report 3365 about “SAP Backup and Recovery with NetApp Filers”:

http://www.netapp.com/tech_library/ftp/3365.pdf

16.1.3 Restore SnapShot You might need to restore a file from a snapshot if the file was accidentally erased or corrupted. If you have purchased the SnapRestore license, you can automatically restore files or volumes from snapshots with one command.

To restore a file from a snapshot, execute the following steps.

1. If the original file still exists and you don’t want it to be overwritten by the snapshot file, then use your UNIX client to rename the original file or move it to a different di-rectory.

2. Locate the snapshot containing the version of the file you want to restore.

3. Copy the file from the .snapshot directory to the directory in which the file originally existed.

Example with SnapRestore license:

filer> snap restore -t file /vol/vol0/etc/testfile -s nightly.0 filer> WARNING! This will restore a file from a snapshot into the active filesystem. If the file already exists in the active filesystem, it will be overwritten with the contents from the snapshot. Are you sure you want to do this? Y You have selected file /vol/vol0/etc/testfile, snapshot nightly.0 Proceed with restore? y

Example with copying the same file from the .snapshot directory using a Control Node (no SnapRestore license required):

control1> df -k Filesystem 1K-blocks Used Available Use% Mounted on filer:/vol/vol0/ 50119928 620012 49499916 2% /FlexFrame/vol0 control1> cd /FlexFrame/vol0/etc control1> cp /FlexFrame/vol0/.snapshot/etc/testfile testfile

http://www.netapp.com/tech_library/ftp/3365.pdf

Backup/Restore for EMC Celerra Data Protection


Copying will allocate new space while snap restore will use the old data blocks.

The data on volume volFF is shared among multiple SAP systems. Restoring the complete volume has an effect on all SAP systems of the complete Flex-Frame landscape.

16.2 Backup/Restore for EMC Celerra Depending on the customers data protection and backup needs, EMC Celerra offers several features and methods to ensure against accidentally deleted or corrupted files. The following list describes mainly the DART online features for snapshots.

16.2.1 Backup of Celerra with EMC SnapSure The Celerra network server’s SnapSure feature creates a read-only, logical image (checkpoint) of a Production File System (PFS) that reflects the state of the PFS at the point in time that the checkpoint is created. SnapSure can support up to 64 checkpoints of the PFS.

You can use a checkpoint to restore a PFS or part of a file system (for example, a file or directory) to the state in which it existed at the time when the checkpoint was created. SnapSure copies when modifications are done. A production file system is build up of blocks. When a block within the PFS is modified, a copy containing the original contents of that block is saved to a separate volume called SavVol. Following changes made to the same block in the PFS are not copied into the SavVol. The original blocks from the PFS (in the SavVol) and the unchanged PFS blocks (that remain in the PFS) are read by SnapSure and these blocks together result in a complete point-in-time file system image, which is called a checkpoint.

By default, when you create the first checkpoint of a PFS, SnapSure creates a SavVol, which is a checkpoint-specific volume. It uses this SavVol for all additional checkpoints you create of the same PFS.

To create a checkpoint using SnapSure’s default storage and checkpoint name assign-ments, use fs_ckpt <fs_name> -Create, where <fs_name> is the file system of which you want to create a checkpoint.

EMC recommends that you allow at least 15 minutes between the crea-tion/refresh of SnapSure checkpoints of the same PFS. This includes check-points created via the CLI, and those created/refreshed in an automated sched-ule, or between schedules running on the same PFS.

Example:

To create a checkpoint of production file system pfs1, type:

$ fs_ckpt pfs1 -Create

Data Protection Backup/Restore for EMC Celerra


16.2.1.1 Volumes on Celerra

For the following volumes in a FlexFrame environment a snapshot schedule should be configured:

VolFF FlexFrame specific volume with the OS images, FA Agents, scripts and pool specific data (e.g. SAP and DB executables, profiles etc.).

Sapdata FlexFrame specific volume with all data files of the SAP databases.

Saplog FlexFrame specific volume with all log files of the SAP databases.

16.2.1.2 SnapSure Schedules

An automated checkpoint refresh solution can be configured using the Celerra Manager or a Linux cron job script. You can get to the Celerra Manager by a GUI, for example via Mozilla by addressing the control station. There is no NAS CLI command equivalent to the checkpoint scheduling functionality of the Celerra Manager.

Creating and managing automated checkpoint schedules has been simplified by en-hancements to the Celerra Manager. Using the Checkpoints->Schedules tab of the Celerra Manager, you can schedule checkpoint creation and refreshes on arbitrary, mul-tiple hours of a day, days of a week, or days of a month. You can also specify multiple hours of a day on multiple days of a week to further simplify administrative tasks.

16.2.2 Backup of SAP Databases See section 16.1.2 on page 296.

16.2.3 Restore Snapshots From an active checkpoint, you can restore an online PFS back to the point-in-time in which it existed when the checkpoint was created. Before it begins the restore operation, SnapSure creates a new checkpoint of the PFS in case you do not want the restored image and it uses a 75% HWM for the SavVol when the new checkpoint is created. Be-fore you attempt a restore, ensure that SnapSure has enough SavVol space available to create the new checkpoint.

The restore operation may fail the first time if SnapSure determines there is insufficient SavVol space and extends the SavVol. The next restore attempt succeeds because the needed space is available.

If you have file systems mounted on the nested PFS mountpoint, you have to unmount all of them before attempting to restore the PFS from a checkpoint.

FlexFrame Backup with Tape Library Data Protection


To restore a PFS from a checkpoint, use /nas/sbin/rootfs_ckpt <fs_name> -Restore, where <fs_name> is the checkpoint from which you want to restore the PFS.

Example:

To restore the PFS from checkpoint ufs_ckpt, type:

$ /nas/sbin/rootfs_ckpt ufs_ckpt –Restore

16.3 FlexFrame Backup with Tape Library

16.3.1 NetWorker A dedicated backup server will be used for maintaining NetWorker. As a backup tool NetWorker is used including the database module NetWorker Module PLUS for Ora-cle.

In the case of a NetApp Filer as NAS device the database module NetWorker Module PLUS for Oracle is used. This concept is based on snapshots and uses the NDMP protocol for transferring data from the NetApp Filer directly to the tape library as shown in the example below.

Configuration Example for Oracle on NetApp Filer:

In the graphic the * means: NetWorker Module PLUS for Oracle for NDMP NetApp

Data Protection Backup/Restore of FlexFrame Control Nodes


Detailed information on FlexFrame for SAP Backup with NetWorker such as

● Implementation and configuration of the NetWorker Backup Solution for Oracle (at great length)

● Slidesets for different target groups like marketing and sales

● White papers

will be provided by Fujitsu Siemens Computers Storage Consulting and is available at

1http://extranet.fujitsu-siemens.com/com/ep/storage/solutions/FlexFrame_backup/ff4mySAP_backup

16.3.2 Arcserve Detailed information on a backup solution with Arcserve is available at

http://extranet.fujitsu-siemens.com/com/ep/storage/management/ brightstor/arcserve_backup/general/arcserve_with_flexframe

as well as

http://extranet.fujitsu-siemens.com/flexframe

at the bottom.

16.4 Backup/Restore of FlexFrame Control Nodes

16.4.1 Backup of a Control Node Most of the files on a Control Node are shipped with the installation DVD. Necessary data can be backed-up using ff_cn_backup.sh.

Synopsis

ff_cn_backup.sh [-dr] [-f <file_name>] [-l <list>]

Options

-d Debug messages to /tmp/ff_cn_backup.DEBUGLOG

-r Restore

-f <file_name> Use file name for restore

-l <list> Blank separated list of files (quoted)


http://extranet.fujitsu-siemens.com/com/ep/storage/management/�brightstor/arcserve_backup/general/arcserve_with_flexframe

http://extranet.fujitsu-siemens.com/com/ep/storage/management/�brightstor/arcserve_backup/general/arcserve_with_flexframe

http://extranet.fujitsu-siemens.com/flexframe

Backup/Restore of FlexFrame Control Nodes Data Protection


The default behavior (without any options) is to backup the configuration files in a zipped tar file at /FlexFrame/volFF/FlexFrame/backup/<name_of_cn><date>.tgz.

Additional files can be backup up using the -l option.

The Control Nodes are not intended to have additional software installed on. Therefore backup in the regular sense is not requried. Installation from the DVD is much faster than any restore.

16.4.2 Restore of a Control Node Restore can be done after installation using the Instalation DVD along with the configura-tion media. Make sure to install the same patches and RPM packages of the FlexFrame tools as before.

After the installation, ff_cn_backup.sh must be called using the option -r.

By default, the restore will pick the latest backup it can find. If this should not be the ap-propriate backup file, use the option -f <file_name>.

In detail, follow these steps to restore a Control Node:

1. Boot the Control Node Installation DVD.

2. Provide the configuration files (*.conf) with the USB stick or floppy. They may be obtained manually from the /opt/FlexFrame/etc folder of the remaining Control Node or from a backup file.

You will get a warning message about wrong checksums if you reuse con-figuration files which have had been installed on a Control Node. This warn-ing message is only intended to give you a hint to double-check the configu-ration files. It can be ignored safely.

3. Abort the installation at the first “<YES> <NO>” screen with the title "FlexFrame(TM) Setup for Control Nodes ".

4. Enter s to get into a subshell.

5. Install the latest patch set.

6. Reboot the Control Node using reboot -f

7. Again the setup screen will appear.

8. If the passwords are requested, enter the current passwords.

9. Enter the correct Control Node (first or second) for this Control Node during installa-tion.



10. Once the system is booted execute the following command:

control1:~# ff_cn_folders.pl -notrans 11. Check the access to /FlexFrame/volFF/FlexFrame/backup

12. Use ff_cn_backup.sh with the option -r and optionally -f <file_name> to restore the original contents of the Control Node's configuration files.

13. Compare the two Control Nodes using the command:

control1:~# ff_cn_cmp.sh

If there are differences in files listed, copy them over from the other Control Node us-ing:

scp -p control2:/<path> /<path>

14. Reboot the Control Node using the init 6 command.

15. The Control Node operating system should now be ok, proceed with restoring PRIMECLUSTER.

16.4.3 Restore PRIMECLUSTER The tool ff_cn_backup.sh does not backup the PRIMECLUSTER configuration, there-fore after restoration of a Control Node, PRIMECLUSTER has to be recovered manually.

If you restored both Control Nodes at once and there is no RMS running on any node, simply run ff_pcl_tool.sh -i as described in the “FlexFrame Installation Guide”.

In most situations, the left-over Control Node is still running RMS and its services, the running configuration has to be distributed to the recovered node which can be briefly described in three steps:

1. Configure and start Cluster Foundation (CF) on the recovered node.

2. Shut down the running RMS and start activation process, then restart RMS.

3. Activate automatic startup and start PRIMECLUSTER on the recovered node.

In detail, follow these steps carefully.

1. Get the current configuration of Cluster Foundation (CF) from the backup Control Node where PRIMECLUSTER is still running:

control2:~ # cfconfig -g control2 FLEXFRAME bond0

Backup/Restore of FlexFrame Control Nodes Data Protection


2. Configure the Cluster Foundation on the restored Control Node using the configura-tion determined in step 1:

control1:~ # cfconfig -s FLEXFRAME bond0 configuration set to: control1 FLEXFRAME bond0

3. Start the Cluster Foundation and check if it is up and running:

control1:~ # /etc/init.d/cf start Starting cf exit 0 control1:~ # cftool -n Node Number State Os Cpu control1 1 UP Linux Pentium control2 2 UP Linux Pentium

4. Verify the ssh interconnection between both RMS hostnames and add their key to the known_hosts file if needed:

control1:~ # ssh control2RMS The authenticity of host 'control2rms (1.2.3.5)' can't be established. RSA key fingerprint is 9b:ca:f7:b5:76:c0:4d:ec:3e:a0:62:e2:09:f3:19:a3. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'control2rms,1.2.3.5' (RSA) to the list of known hosts. control2:~ # ssh control1RMS The authenticity of host 'control1rms (1.2.3.4)' can't be established. RSA key fingerprint is 6d:22:1d:67:9a:e1:69:fc:f4:70:1e:57:07:d9:db:44. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'control1rms,1.2.3.4' (RSA) to the list of known hosts.

If ssh asks for a password, something went wrong during restoration of the ssh keys using ff_cn_backup.sh. This is a serious problem and must be fixed before the RMS configuration can be distributed!

5. Stop RMS on the node where PRIMECLUSTER and RMS are still running.

Active services – for example the ldap server – will not be interrupted when using the capital L option with hvshut:

control2:~ # hvshut -L WARNING ------- The '-L' option of the hvshut command will shut down the RMS software without bringing down any of the applications.



In this situation, it would be possible to bring up the same application on another node in the cluster which *may* cause data corruption. Do you wish to proceed ? (yes = shut down RMS / no = leave RMS run-ning). yes NOTICE: User has been warned of 'hvshut -L' and has elected to pro-ceed.

6. Start the configuration activation and distribution process which may take a while:

control2:~ # pcstool -a FlexFrame SYSNODE_LIST=control1RMS control2RMS UNREACHABLE= REACHABLE=control1RMS ccontrol2RMS DISTRIBUTION_LIST=control1RMS control2RMS hvbuild using /opt/SMAW/SMAWpcs/Config/FlexFrame/runtime/FlexFrame.us Activation succeeded

Please verify if both Control Nodes are listed under REACHABLE and DISTRIBU-TION_LIST. If this command failed, something is wrong with the ssh configuration (see step 4).

7. Restart RMS:

control2:~ # hvcm

8. Finally, enable automatic startup of PRIMECLUSTER and load all services on the recovered Control Node:

control1:~ # ff_pcl_tool.sh -e Copyright (C) 2004-2007 Fujitsu Siemens Computers. All rights re-served. Enabling rc scripts: cf SMAWsf SMAWpcs SMAWRrms control1:~ # ff_pcl_tool.sh -l Copyright (C) 2004-2007 Fujitsu Siemens Computers. All rights re-served. Loading SMAWsf... Loading SMAWpcs... Loading SMAWRrms...

The recovered Control Node and PRIMECLUSTER should now be fully functional.

Backup of Switch Configurations Data Protection


16.5 Backup of Switch Configurations The switch configuration may be different to that at the time of installation due to the operation of a FlexFrame landscape. New Application Nodes and switches may be added, ports configured and so on. For system backup, there are a lot of well known products and programs.

To simplify backing up the configuration of all switches used within the FlexFrame land-scape the program /opt/FlexFrame/bin/ff_save_switch_config.pl should be used. It is designed to be run by cron(8) and recommended to call it once a day.

The program scans the LDAP database for switches and switch blades. It connects the switches and the switch blades and stores the running or startup configuration via TFTP into /tftpboot/<switch host name>.config on the Control Nodes. The switch host name is derived from LDAP, not from the switch backed up. The program is able to backup all (this is the default) switches or the switch given by its LDAP known host name.

Synopsis

ff_save_switch_config.pl [--silent] [--running] [--name <switch_name>]

[--ip <control_node_ip>]

Options

--silent The silent mode supresses any message, even error messages.

--running In normal case the configuration, which the switch reads on booting, is backed up, the so called startup-config. Using the --running option the current running switch configuration is used instead of startup-config. For a Quanta BX600 switch blade (has six external ports) the running switch configuration is backed up only.

--name <switch_name> Backup only the switch with given name. Keep in mind to use the name known at LDAP database. It may differ from current switch name if it was not synchronized.

--ip <control_node_ip> Use control node with given control lan ip address to save configurations to. In nor-mal case the program is able to detect the control node with running netboot_srv ser-vice itself. If it is not able to detect the active node or can not determine the control lan ip of the control node control, the program requests to use the --ip option.

Data Protection Restore of Switch Configurations


16.6 Restore of Switch Configurations To restore a switch configuration select the proper configuration file from /tftpboot and follow the instructions in the chapter „Switch Configuration” of the „Installation Guide. For switch blades the configuration may alternatively be created with /opt/FlexFrame/bin/ff_bx_cabinet_adm.pl using operation mode swb-config (see section 8.6.7).


17 Error Handling & Trouble Shooting

17.1 Log Files For easy gathering of log file information on all Application Nodes in FlexFrame, it is not necessary to login on each Application Node because all log files are directly readable at the Control Nodes on the mounted NAS system.

To elevate the admin to read these log files it is useful to create symbolic links on the Control Nodes in the directory /var like this:

control1:/ # cd /var control1:/var # ls -ld log* drwxr-xr-x 20 root root 1440 May 2 14:01 log lrwxrwxrwx 1 root root 58 Apr 29 17:28 log_pool1_klinge1 -> /FlexFrame/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/var_img/var-ac10020e/log lrwxrwxrwx 1 root root 58 Apr 29 17:29 log_pool1_klinge2 -> /FlexFrame/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/var_img/var-ac10020f/log lrwxrwxrwx 1 root root 58 Apr 29 17:30 log_pool1_klinge3 -> /FlexFrame/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/var_img/var-ac100210/log lrwxrwxrwx 1 root root 58 Apr 29 17:30 log_pool1_klinge4 -> /FlexFrame/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/var_img/var-ac100211/log lrwxrwxrwx 1 root root 58 Apr 29 17:30 log_pool1_klinge5 -> /FlexFrame/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/var_img/var-ac100212/log lrwxrwxrwx 1 root root 58 Apr 29 17:31 log_otto_RX300-01 -> /FlexFrame/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/var_img/var-ac100113/log lrwxrwxrwx 1 root root 58 Apr 29 17:33 log_otto_RX300-02 -> /FlexFrame/volFF/os/Linux/FSC_4.0A00-000.SLES-9.X86_64/var_img/var-ac100114/log control1:/var #

To read the /var/log/messsages from Application Node klinge5 in pool pool1 invoke this:

control1:/ # cd /var control1:/var # less log_pool1_klinge5/messages

Error Handling & Trouble Shooting Network Errors


Or enter the log directory from this Application node and look for logfiles:

control1:/ # cd /var/log_pool1_klinge5 control1:/var/log_pool1_klinge5 # ls -lrt|tail -rw-r--r-- 1 root root 278 May 4 16:23 log.scagt -rw-r--r-- 1 root root 174 May 4 16:24 log.vvagt -rw-r--r-- 1 root root 257 May 4 16:24 log.statusagt -rw-r--r-- 1 root root 612 May 4 16:28 ntp -rw-r--r-- 1 root root 5542 May 4 16:35 warn -rw-r--r-- 1 root root 1696 May 4 16:35 auth.log -rw-r--r-- 1 root root 484 May 4 18:16 log.busagt drwxr-xr-x 2 root root 4096 May 4 18:19 sa -rw-r--r-- 1 root root 27622 May 4 18:19 messages prw------- 1 root root 0 May 4 18:24 psadfifo control1:/var/log_pool1_klinge5 #

17.2 Network Errors In most cases, network problems are most times configuration mistakes. All switches except the switch blades are configured to send SNMP traps and log via syslog to Control Nodes. To see what happened, look at the Control Nodes’ /var/log/messages. Any conditions reported by the switches can be found here. For SNMP traps look at the FA Agents support database. The FA Agents collect all kinds of SNMP traps for the entire FlexFrame environment.

17.3 NFS Mount Messages During start/stop procedures of SAP instances, NFS mount failures for /usr/sap/SYS and /oracle/db_sw can be seen in /var/log/messages. These messages are not FlexFrame-specific failures, but may occur on any system that has /usr/sap and /oracle in its automount maps. These messages are caused by the way SAP is linking binaries during software development process.

17.4 LDAP Error Codes and Messages LDAP failures may have various reasons. To determine the reason, first have a look at the Control Node’s /var/log/messages. If slapd does not report anything you may have to increment the loglevel configured with /etc/openldap/slapd.conf. Log levels are additive, and available levels are:

LDAP Error Codes and Messages Error Handling & Trouble Shooting


Log level Meaning

1 Trace function calls

2 Debug packet handling

4 Heavy trace debugging

8 Connection management

16 Print out packets sent and received

32 Search filter processing

64 Configuration file processing

128 Access control list processing

256 Stats log connections/operations/results

512 Stats log entries sent

1024 Print communication with shell backends

2048 Entry parsing Most LDAP problems are related to access control and connection management.

If you have changed the loglevel send a -HUP to the slapd to advice it to reread configu-ration files (use pkill -HUP slapd).

Another reason may be the LDAP client configuration. Check the following issues:

● Does the client use the correct server address?

● Does it use the correct user and password (Solaris only)?

Look at the client messages file to get this information.

If ldapsearch -x on Linux and ldaplist on Solaris works properly the problem may be the nsswitch configuration. These are the top most problems.

When reporting problems with the administration scripts, please provide the following information:

● The error message of the admin tool

● A dump of the LDAP database (ldapsearch -x > <file name>)

The configuration files /etc/openldap/slapd.conf and /FlexFrame/volFF/FlexFrame/ldap/common/slapd.acl.conf

Error Handling & Trouble Shooting FA Agents Error Diagnosis


17.5 FA Agents Error Diagnosis The FA Agents offer a large number of diagnostic options for detecting and diagnosing problems on the FA Agents themselves or other components.

Problems concerning FA Agents can be assigned to one of the following categories:

● FlexFrame installation and configuration errors

● Parameter errors

● Configuration errors

● Detection, reaction errors, start, stop, maintenance errors

● Power shutdown errors

Typical consequences of installation and configuration errors are:

● FA Agents fail to start

● Error messages during startup of FA Agents

Error: Mount points missing

Diagnosis:

In the case of missing mount points monitored by FA Autonomy, traps are sent to the central trap consoles. With other mount points which are absolutely essential for the operation of the node in question, it can happen that the agents cannot be started as the directories required are not available.

Response:

Provide the mount points required with the appropriate mount options.

Error: Mount points without “File Locking”

Diagnosis:

The FA Agents log this situation both in the operating system’s Syslog and in special files (/opt/myAMC/vFF/log/log_syslog*).

Response:

Provide the mount points required with the appropriate mount options (lock).

FA Agents Error Diagnosis Error Handling & Trouble Shooting


Error: Rights for the directories/files are not sufficient

Diagnosis:

In the case of files monitored by FA Autonomy, traps are sent to the central trap consoles. With other directories/files which are absolutely essential for the operation of the node concerned, it can happen that the agents cannot be started as the direc-tories/files required are not available.

Response:

Provide the directories/files with the required rights.

Error: Agents do not have the authorization to write to the directories assigned

Diagnosis:

The FA Autonomy production and log files are not written.

Response:

Provide the directories/files with the required permissions.

Error: Version incompatibility

Diagnosis:

FlexFrame installation and FlexFrame Autonomy installation are not directly com-patible. This can always be the case when older FlexFrame installations are updated with new FlexFrame Autonomous Agents.

Response:

For diagnosis and troubleshooting, the mount points, the directory structure and the access rights to the directories used by the agents must be checked.

Use the migration tool, check that the parameters used in the FA config files are compatible with the version and syntactically correct.

Error: Pool assignment not found

Displayed in the FA WebGUI or in the agent’s start trap and on an event console.

Diagnosis:

A node is assigned to the wrong pool or to the default pool.

Response:

Check the LDAP configuration parameters, call the PGTool Pool.sh and check the pool name returned. Check the pool membership for each node.

Error Handling & Trouble Shooting FA Agents Error Diagnosis


Error: Group assignment is not correct

Display on the FA WebGUI or in the agent’s start trap and display on an event console.

Response:

Check the group configuration in the group configuration file. Check the group mem-bership for each node.

Error: Service priority not recognized

Display on the FA WebGUI or in the agent’s start trap and display on an event console.

Response:

Check the configuration of the service class and service priority in the group configu-ration file. Check the values for each node.

Error: Availability problem not rectified by autonomous reaction

Diagnosis:

Services are discontinued (possibly due to hardware fault) and are not made avai-lable again by FlexFrame Autonomy.

Response:

Check whether nodes are available for taking over the services (Spare Nodes). Check whether the FA Agents on the nodes involved have been started.

Error: Services do not start - Constant reboot, Permanent switchover

Diagnosis:

SAP services which are started do not enter run mode but are repeatedly restarted or, if the problem escalates, the node is rebooted or an internal switchover takes place. Possible causes:

– The MaxRestart time for the service is too short. This parameter can be ad-justed in the FA configuration.

– The virtual interfaces cannot be reached.

– There is a permanent problem which prevents a service being started (e.g. necessary database recovery).

Response:

Stop the FA Agents to interrupt escalation of the reaction and check whether the service can be started manually.

FA Agents Error Diagnosis Error Handling & Trouble Shooting


If the service cannot be started manually, this problem must be corrected by the ad-ministrator.

If the service can be started manually, the time required for this must be matched to the MaxStart time and MaxRestart time in the configuration and the configura-tion must be adjusted, if necessary.

If the virtual interfaces cannot be reached from the Application Agent, the network configuration must be checked.

Error: Service cannot be stopped

Diagnosis:

An active SAP service is repeatedly restarted after a manual stop command.

Response:

The problem could result of not using the FlexFrame SAP scripts to stop the service manually. The FlexFrame SAP scripts must be used for every action concerning FlexFrame services.

If the FlexFrame SAP scripts were used, the Monitor Alert Script might not be avail-able or does not have the required rights.

However, it is also possible that the Monitor Alert Time and CycleTime are config-ured incorrectly. The agents’ CycleTime is too long in relation to the Monitor Alert Time. The Monitor Alert Time must be at least 3 times the CycleTime.

Error: Maintenance activities are interrupted by autonomous reactions

Diagnosis:

Unwanted autonomous reactions during maintenance.

Response:

Set NoWatch for the service in question or stop the Application Agents for the node concerned and restart them after maintenance has been completed.

Error: Incorrect Display on the FA WebGUI

Diagnosis:

The state checked manually does not match the display.

Response:

Check whether the Application Agents in question, the Control Agents concerned and the web server are running properly for the WebGUI. The log files of the Flex-Frame Autonomous Agents can be used for the diagnosis.

Error Handling & Trouble Shooting FA Agents Operation and Log Files


The FlexFrame Autonomous Agents write detailed log files. The functions of the FA Agents are documented in their own files. These files are created dynamically during ongoing operation and may not be modified manually as this can impair fault-free operation of the FA Agents or lead to erroneous reactions. Deleting these files re-sults in a state in which the Autonomous Agents reorganize themselves and, from this point on, analyze the situation from their current viewpoint without any previous information.

17.6 FA Agents Operation and Log Files

17.6.1 General The activities and dynamic states of the FA Agents are documented in various files. These files may not be changed manually as this can impact error free operation of the FA Agents or result in incorrect reactions.

These files are created dynamically during ongoing operation. Deleting these files leads to a status in which the Autonomous Agents reorganize themselves, and from this point they re-evaluate the situation from the current viewpoint without any previous knowledge.

17.6.2 Overview, important Files and Directories This section shows a number of FF specific files and directories and their respective content.

Base directory is /opt/myAMC/.

Version numbers: In the table below, V<v>K<r> corresponds to V<version number>K<revision_number>.

Subdirectories Content

./scripts Scripts for various tasks

./scripts/sap Link to the FlexFrame scripts

./scripts/acc Scripts for the SAPACC Interface

./scripts/PowerMng Scripts for the power management blades.

./scripts/ShutDown_Node Scripts to shut down a node.

./config General configuration data

FA Agents Operation and Log Files Error Handling & Trouble Shooting



./config/FA_WebGui.conf General settings for the WebGUI (directo-ries, cycle times, database settings)

./config/amc-users.xml User management

./FA_AppAgent Installation path of myAMC.FA_AppAgent and of diverse scripts.

./FA_AppAgent/myAMC.FA_AppAgent Start/stop scripts Application Agent

./FA_AppAgent/PGTool_Pool.sh Determination of pool membership

./FA_AppAgent/PGTool_Version.sh Determination of pool version

./FA_AppAgent/PVget.sh Determination of the SAPS number of a node

./FA_AppAgent/BBTool.sh BlackBoard control

./FA_AppAgent/BBT_dialog.sh BlackBoard dialog mode control

./FA_AppAgent/bin_Solaris_V<v>K<r>

./FA_AppAgent/bin_Linux_V<v>K<r>

./FA_AppAgent/bin_Linux_SLES9_V<v>K<r>

./FA_AppAgent/lib_Solaris_V<v>K<r>

./FA_AppAgent/lib_Linux_V<v>K<r>

./FA_AppAgent/lib_Linux_SLES9_V<v>K<r>

Binaries and libraries for Solaris and Linux for each version.

./FA_AppAgent/config myAMC.FA_AppAgent-specific configura-tion data

./FA_AppAgent/log empty

./FA_CtrlAgent Installation path of myAMC.FA_CtrlAgent and of scripts

./FA_CtrlAgent/myAMC.FA_AppAgent Start/stop scripts Control Agent

./FA_CtrlAgent/PGTool_Pool.sh Determination of pool membership

./FA_CtrlAgent/PGTool_Version.sh Determination of pool version

./FA_CtrlAgent/PVget.sh Determination of the SAPS number of a node

./FA_CtrlAgent/BBTool.sh BlackBoard control

./FA_CtrlAgent/BBT_dialog.sh BlackBoard dialog mode control




./FA_CtrlAgent/bin_Solaris_V<v>K<r>

./FA_CtrlAgent/bin_Linux_V<v>K<r>

./FA_CtrlAgent/bin_Linux_SLES9_V<v>K<r>

./FA_CtrlAgent/lib_Solaris_V<v>K<r>

./FA_CtrlAgent/lib_Linux_V<v>K<r>

./FA_CtrlAgent/lib_Linux_SLES9_V<v>K<r>

Binaries and libraries for Solaris and Linux for each version.

./FA_CtrlAgent/config myAMC.FA_CtrlAgent-specific configu-ration data

./FA_CtrlAgent/log empty

./vFF Pool-specific (vFF) data

./vFF/log Pool-specific log files

./vFF/Common/myAMC_FA_Pools.xml

./vFF/Common/myAMC_FA_Pools-default.xml Pools configuration file and its default version This is used as LDAP cache

./vFF/Common/.vFF_template.V<v>K<r> Template of pool-specific data for each version

./vFF/Common/.vFF_template.V<v>K<r>/ config

Configuration

./vFF/Common/.vFF_template.V<v>K<r>/ config/TrapTargets.xml

Trap targets

./vFF/Common/.vFF_template.V<v>K<r>/ config/myAMC_FA.xml ./vFF/Common/.vFF_template.V<v>K<r>/ config/myAMC_FA-default.xml

myAMC.FA configuration and default

./vFF/Common/.vFF_template.V<v>K<r>/ config/myAMC_FA_ACC.xml ./vFF/Common/.vFF_template.V<v>K<r>/ config/myAMC_FA_ACC-default.xml

myAMC.FA ACC configuration and default

./vFF/Common/.vFF_template.V<v>K<r>/ config/myAMC_FA_GUI.xml ./vFF/Common/.vFF_template.V<v>K<r>/ config/myAMC_FA_GUI-default.xml

myAMC.FA GUI configuration and default




./vFF/Common/.vFF_template.V<v>K<r>/ config/myAMC_FA_Groups.xml ./vFF/Common/.vFF_template.V<v>K<r>/ config/myAMC_FA_Groups-default.xml

myAMC.FA groups configuration and default

./vFF/Common/.vFF_template.V<v>K<r>/ config/myAMC_FA_SD_Sec.xml ./vFF/Common/.vFF_template.V<v>K<r>/ config/myAMC_FA_SD_Sec-default.xml

myAMC.FA shutdown security configura-tion and default

./vFF/Common/.vFF_template.V<v>K<r>/log

./vFF/Common/.vFF_template.V<v>K<r>/log/AppAgt ./vFF/Common/.vFF_template.V<v>K<r>/log/CtlrAgt

Logfiles of myAMC.FA_AppAgent and myAMC.FA_CtrlAgent for each pool

./vFF/Common/.vFF_template.V<v>K<r>/data

./vFF/Common/.vFF_template.V<v>K<r>/data/FA

Work files

./vFF/Common/.vFF_template.V<v>K<r>/data/FA/livelist

Livelist livelist.log

./vFF/Common/.vFF_template.V<v>K<r>/data/FA/xmlrepository

XML repository for the web interface livelist.xmlServices_<node name>.xml

./vFF/Common/.vFF_template.V<v>K<r>/data/FA/servicelists

Service lists Services_<node name>.lst

./vFF/Common/.vFF_template.V<v>K<r>/data/FA/servicelogs

Service logs (history) Services_<node name>.log

./vFF/Common/.vFF_template.V<v>K<r>/data/FA/reboot

Reboot files Reboot_<node name>.lst

./vFF/Common/.vFF_template.V<v>K<r>/data/FA/switchover

SwitchOver files SwitchOver_<node name>.lst

./vFF/Common/.vFF_template.V<v>K<r>/data/FA/blackboard

BlackBoard blackboard.txt

./vFF/Common/.vFF_template.V<v>K<r>/data/FA/performance

Measured performance data




./vFF/vFF_Cust_1

./vFF/vFF_Cust_1/config

./vFF/vFF_Cust_1/log/…

./vFF/vFF_Cust_1/data

./vFF/vFF_Cust_1/data/FA/….

Pool-specific data for pool “Cust_1” (example). See above for the description of the subdi-rectories and files.

./vFF/vFF_Cust_2

./vFF/vFF_Cust_2/config

./vFF/vFF_Cust_2/log/…

./vFF/vFF_Cust_2/data

./vFF/vFF_Cust_2/data/FA/…

Pool-specific data for pool “Cust_2” (example). See above for the description of the subdi-rectories and files.

17.6.3 Special Files The write cycle for the entries (with the exception of reboot, switchover and Black-Board) and the storage location of the files described in the following are defined using a parameter in the configuration file myAMC_FA.xml.

17.6.3.1 Livelist

Each FA Application Agent regularly enters itself in this list. Through these entries the Control Agent recognizes whether the various Application Agents are available and func-tioning without error.

17.6.3.2 Services List

This file (testament) exists for each FA Application Agent on a node-specific basis. In this file, the FA Application Agent logs the detected services and their actual status, which are detected by the internal detector. So the status information in this file is service related. The content is updated with every detector cycle.

17.6.3.3 Services Log

The contents of this file are identical to those of the Services-List file, with the differ-ence that the history is contained in this file. This enables status changes and reaction decisions to be detected and replicated.

17.6.3.4 Reboot

The contents of this file are identical to those of the Services-List file. The file serves as information storage when a reboot takes place. It is written only for the autonomous reaction reboot and is deleted again after the reboot has been completed and the ser-vices have been started up.



17.6.3.5 Switchover

The contents of this file are identical to those of the Services-List file. The file serves as information storage (testament) when a switchover takes place. It is written only for the autonomous reaction switchover and is deleted again after the services were taken over.

17.6.3.6 XML Repository

In terms of contents, the files in the XML Repository are the same as those in the Livelist and Services List. By contrast, the contents are written in XML notation and can thus be visualized directly with the associated FA WebGUI. The write cycle for the entries and the storage location of the file are defined using a parameter in the configuration file myAMC_FA.xml.

17.6.3.7 BlackBoard

The BlackBoard is an input interface for the FA Agents. Commands can be entered here which are executed by the FA Application Agents. The commands have a specific validity period and are secured against manipulation. The file is written manually using a tool which guarantees, among other things, protection against manipulation.

17.6.4 FA Autonomy Diagnostic Tool Manual diagnosis of the log files can be very time-consuming. The Fujitsu Siemens Com-puters support organization works with specialized diagnostic tools which can analyze even large quantities of data very quickly and efficiently. This service can be utilized when required and if a corresponding service agreement exists.

To utilize this service, either individual log files or the entire virtual FA directory of a pool can be sent to the support department, e.g. as a compressed and protected zip archive.

17.6.5 Data for Diagnosis in the Support Department If support is required, there is special data needed by the FlexFrame Support. This infor-mation is required to analyze problems with FlexFrame and the Autonomous Agents.

● Error description, as precise as possible What is the problem or error? On which nodes does it occur?

● Version of the FA Agents installed Run rpm -qa|grep myAMC on the Control Node.

● Configuration, work and log files of the FA Agents

Error Handling & Trouble Shooting Start/Stop Script Errors


The following script creates an achive with the desired information:

/opt/myAMC/FA_CtrlAgent/SAVE_FA_files_for_diag.sh

This script has to be invoked from a Control Node!

cd /opt/myAMC/FA_CtrlAgent ./SAVE_FA_files_for_diag.sh

The functions of the FA Agents are documented in various files. These files may not be changed manually as this can impair error-free operation of the FA Agents or result in incorrect reactions. These files are created dynamically during ongoing operation. Delet-ing these files leads to a status in which the Autonomous Agents reorganize themselves, and from this point they re-evaluate the situation from the current viewpoint without any previous knowledge.

For further information on collecting diagnosis data see “FA Agents - Installation and Administration“ manual, section 4.7.3.

17.7 Start/Stop Script Errors While executing the SAP service start/stop scripts, several errors may occur for arbitrary reasons. They can be grouped in error classes like

● wrong parameters invoked

● installation errors of the SAP service (e.g. virtual host names, instance profiles, service entries)

● functionality errors with interface configuration

● logical errors (e.g. with start_flags)

The $-variables in the messages shown below are dependent from the system ID, in-stance number and instance type.

17.7.1 Common Error Messages for all Start/Stop Scripts This section lists common error messages delivered by the start/stop service scripts:

$SERVICE_SCRIPT_PATH/sapservice_config does not exist;

you are possibly not on an Application Node!

All start/stop scripts use a common configuration file.in aFlexFrame 3.2 envi-ronment that is pool-specific.

Start/Stop Script Errors Error Handling & Trouble Shooting


Wrong parameter count

To less or to many parameters invoked, please attend the usage.

For clean up:

$v_service_text ${ID} ${SID} have still running processes. For sure to kill running processes on $v_service_text ${ID} ${SID}? (yes/no):

Clean up is the ultimative method to kill still running processes, freeing occupied shared memory and semaphores caused by this SAP service. You should never invoke this if you are not really sure!

homedir from $v_user not found in passwd.

For the OS user $v_user there is no home directory in /etc/passwd or getent passwd.

homedir $v_home from $v_user not exist.

The home directory from $v_user does not exist.

Interface errors

no interface defined for $v_lan_type.

In /FlexFrame/scripts/sapservice_config must be defined which inter-face will be used for which LAN type.

no IPMP interface found for $v_interface.

No IPMP interface is configured for Solaris. Please check with ifconfig -a.

no netmask defined for $v_lan_type.

The netmasks to be used for each LAN type have to be defined in /FlexFrame/scripts/sapservice_config.

$v_lan_type interface $v_interface:$I ypcat/getent failure for host $v_vhost.

getent hosts deliveres no entry for this virtual host name $v_vhost.

$v_lan_type interface $v_interface_n $MY_IP is already up.

This is a warning only, not an error.



my ${v_lan_type} ip-addr $MY_IP is already in use, ping gets an-

swers from a foreign interface.

While configuring a virtual interface to start a SAP service, it is checked if the vir-tual IP is not in use.

$v_lan_type interface $v_interface:$I $MY_IP is not up.

The newly configured interface seems not to work properly for ping/arping.

$v_lan_type interface $v_interface: no free interface found.

This message occurs when all 64 virtual interfaces are in use. There may be a maximum of 64 virtual interfaces on one physical interface.

$v_interface_n $MY_IP is already down.

This warning occurs while trying to shut down an already deconfigured virtual in-terface.

$v_interface_n $MY_IP is not down.

Shutting down a virtual interface has failed.

start_flag errors

$v_service_text $ID $SID is possibly running on another host:

$v_vhost.

$v_service_text $ID $SID is possibly not started.

$v_service_text $ID $SID is already running on this host $v_vhost.

$v_service_text $ID $SID is already running on another host: $v_vhost, MY_HOST: $MY_HOST.

The situation expected does not accord to the situation documented in the start flag files *_host.

announce_start_flag errors

$v_service_text $ID $SID is possibly already started.

$v_service_text $ID $SID is possibly already starting.

A concurrent start situation for one SAP service has occurred. Only one attempt to start it can be successful, the other one will fail with this message.



inet port 8081 is already used

CHECK_PORTS: 3281 4781 3381 4881 8081

Please take into account that depending on the instance number ports are reserved for the SAP application. That could mean that you conflict with other non SAP application on your nodes (e.g. SAP’s ICM uses ports 80nn which maybe conflicts with an application which uses port 8081). Please refer to SDN document “TCP/IP Ports Used by SAP Appli-cations” to prevent conflicts..

17.7.2 SAPDB Specific Error Messages This section lists common error messages delivered by the start/stop service scripts.

we have neither a ABAP CI nor a Java JC instance profile for

$vhostname_ci!

To gather the dbms_type (ORA, ADA) from the $<sid>adm environment, the script is looking for instance profiles to get the appropriate virtual host name to set the belonging environment. If no instance profiles exists or if none is reach-able via NFS, this message will occur.

dbms_type is not set!

The dbms_type could not be determined.

Database /sapdb/$SID not found!

or Database /oracle/$SID not found!

This directory does not exist or is not reachable via NFS.

Unknown Database Type !

The dbms_type is neither ORA nor ADA. The sapdb script supports only these two database types.

$SIDADM_HOME/.dbenv_${vhostname_ci}.csh not found for SID $SID

Instance $ID !

and $SIDADM_HOME/.sapenv_${vhostname_ci}.csh not found for SID $SID

Instance $ID !

The database is started as ${sid}adm, hence the environment files for this have to exist.



For MaxDB or SAPDB Databases (i.e. dbms_type ADA):

Database /sapdb/$SID has no appropriate XUser found!

xuser list as ${sid}adm deliveres neither “c” or “c_J2EE” key. Please check /home/${sid}adm/.XUSER.62 for a correct installation.

For Oracle Databases (i.e. dbms_type ORA):

tns-listener ${SID} is possibly not started.

TNS-listern process is not running. Please check the TNS listener configuration.

For Oracle Databases (i.e. dbms_type ORA):

tns-listener is possibly not correct configured for ${SID}.

tnsping has failed. Please check the TNS listener configuration.

$DB_SERVICE ${SID} not started.

startdb as ${sid}adm has failed, the reason can be found in the original start logfile os4adm/startdb.log.

$DB_SERVICE ${SID} is possibly not stopped.

stopdb as ${sid}adm has failed, the reason can be found in the original stop logfile os4adm/stopdb.log.

For dbms_type ADA:

vserver ${SID} is possibly not stopped.

For dbms_type ORA:

tns-listener ${SID} is possibly not stopped, we kill them.

17.7.3 Sapci-specific Error Messages This section lists common error messages delivered by the start/stop service scripts.

System /usr/sap/$SID not found!

The directory /usr/sap/$SID does not exist or is not reachable via NFS.



No startprofile found for SID $SID $CI_SERVICE ! and No instance profile $v_profile found for SID $SID $CI_SERVICE !

The profiles in directory /sapmnt/${SID}/profile/ must be reachable.

$CI_SERVICE ${ID} ${SID} could not determine the System Nr

(SAPSYSTEM) in $v_profile !

The instance number is the value of the instance parameter SAPSYSTEM. It has to be found. The profile name is represented by $v_profile here.

$SIDADM_HOME/.dbenv_${vhostname}.csh not found for SID $SID In-stance $v_sysnr ! and $SIDADM_HOME/.sapenv_${vhostname}.csh not found for SID $SID In-

stance $v_sysnr !

The SAP instance is started as ${sid}adm, hence the environment files for this must exist. The instance number is $v_sysnr here.

One or more /etc/services entries sapms$SID sapgw$v_sysnr sapgw${v_sysnr}s sapdp$v_sysnr

sapdp${v_sysnr}s for SID $SID Instance $v_sysnr missed !

All entries in /etc/services (getent services) for the SAP instance must exist. $v_sysnr represents here the instance number.

$DB_SERVICE ${SID} host $vhost is not answering to ping.

There is no ping to the dependent service e.g. database (Oracle or SAPDB/MaxDB) is answering.

$DB_SERVICE ${SID} is not running, please start it first.

Connecting via R3trans -d to the database (Oracle or SAPDB/MaxDB) was impossible.

$CI_SERVICE ${SID} not started.

startsap r3 as ${sid}adm has failed, the reason can be found in the original start logfile: os4adm/startsap_DVEBMGS67.log.

$CI_SERVICE ${SID} is possibly not stopped.

stopsap r3 as ${sid}adm has failed, the reason can be found the original stop logfile: os4adm/stopsap_DVEBMGS67.log

Error Handling & Trouble Shooting SAP ACC Troubleshooting


17.7.4 Sapscs-specific Error Messages The same messages as for sapci are used, instead of $CI_SERVICE or $SCS_SERVICE is used.

17.7.5 Sapascs-specific Error Messages The same messages as for sapci are used, instead of $CI_SERVICE or $ASCS_SERVICE is used.

17.7.6 Sapjc-specific Error Messages The same messages as for sapci are used, instead of $CI_SERVICE or $JC_SERVICE is used.

17.7.7 Sapapp-specific Error Messages The same messages as for sapci are used, instead of $CI_SERVICE or $APP_SERVICE is used.

17.7.8 Sapj-specific Error Messages The same messages as for sapci are used, instead of $CI_SERVICE or $J_SERVICE is used.

17.8 SAP ACC Troubleshooting See also chapter “Troubleshooting” in the ACC Installation Guide.

17.8.1 ACC Logging The ACC stores all logging information. To access the log information, navigate in the WebGUI to Controller Log

For detailed information, click on the suitable line and the information will be shown.

It is also possible to archive the logging information. Navigate to the Technical Set-tings-> Archive Log

Delete, Archive and Retrieve are possible functions. For more information use the Help link in the status bar.

SAP ACC Troubleshooting Error Handling & Trouble Shooting


17.8.2 Missing Server in the ACC Physical Landscape ● Check if the server is already up and reachable via network

● Start the ACCagents:

Server # /usr/sap/adaptive/ACCagents start

● ACC: Press the Refresh button after a while and check if the server is visible in the

Physical Landscape.

17.8.3 Reset of Service Status in Case of Failures If the service has the status failed, some steps have to be done to reset the status depending on where the failure happened:

● Shut down the application service (App, CI, DB, …) by hand using the FF

start/stop scripts.

● Reset the status of the application service: Navigate to the detailed view of the ap-plication service, click on the underlined status message and change the status.

17.8.4 Hanging Locks For a safe handling of the different adaptive hosts, services and configurations, locks are set to the currently used components. In some error situations it is possible that those locks are not deleted correctly.

To delete those locks by hand:

● Login to Visual Administrator,

● Navigate to Server 0…. -> Services -> Locking Adapter.

● Type * into the field Name and press Refresh button.

Choose the right lock and press Delete selected locks.

Error Handling & Trouble Shooting PRIMECLUSTER


17.9 PRIMECLUSTER

17.9.1 Problem Reporting When reporting PRIMECLUSTER problems, always provide the logfiles of both Control Nodes (CN1 and CN2).

17.9.2 Removing “Ghost Devices” from RMS GUI To remove non-existing „Ghost Devices“ that may be diplayed in the RMS GUI, run the following command:

rcqconfig -c -a CN1 CN2

Eventually, you may need to reboot afterwards.

17.10 Script Debugging

17.10.1 Shell Scripts For many FlexFrame scripts, debugging is activated by default. If debugging is not active, it can be activated with the option "-d".

Example:

ff_netscan.sh -d

Further on, shell scripts can be traced using the option "-x".

For details, see the man page of the shell in use (man sh; man bash; etc.)

sh -x ff_netscan.sh

17.10.2 Perl Scripts Before debugging, activate the logging functions of the shell by calling "script" (For de-tails, see man script).

The debugger is called using

perl -d <script_name>

Functions:

h Help

Debugging the Linux Kernel Error Handling & Trouble Shooting


x <expression> Evaluate an expression (hash/array) and display the result

p <string> Returns strings

s Execute next command; follow subroutines

n Execute next command; skip subroutines

b <line_number> Set break point at line <line number>

q Quit.

For further information, see the man page of perldebug.

17.11 Debugging the Linux Kernel For the Linux Kernel, debugging is disabled by default. There are several approaches to collect debugging information. Everything is only described briefly; please contact Fujitsu Siemens support for further instructions.

17.11.1 Netconsole If the Linux kernel detects an internal error, it tries to write some information about the so called “Oops” into the messages file.

If an error completely crashes the kernel (which is known as a kernel panic), this informa-tion may be generated but cannot be written to the disk or the NFS mount.

To get this information anyway, activate the netconsole feature to redirect kernel logging over the network.

If the netconsole output does not provide the required information, you should configure the crash dump functionality.

17.11.2 Capturing Crash Dumps

17.11.2.1 Common Restrictions for Taking Crash Dumps

If taking crash dumps is configured and activated, the process of writing the dump must not be interrupted, so the PRIMECLUSTER and myAMC agent power shutdown features must be disabled for this specific node, for example by pulling the IPMI cable or setting appropriate timeouts.

Amongst others that is one reason for not activating crash dump capturing by default.

Error Handling & Trouble Shooting Debugging the Linux Kernel


17.11.2.2 SLES9: Linux Kernel Crashdump Utilities (LKCD)

For SLES9 Control Nodes and Application Nodes, the LKCD feature is already preconfi-gured, but not activated. On the Control Node you just jeed to activate this feature in the configuration file, on the Application Nodes it is also necessary to create and configure a dump partition on one of the local disks.

● on Application Nodes, create a partition using fdisk and create a symlink /dev/vmdump -> <dump_partition>

● Enable DUMP_ACTIVE in /etc/sysconfig/dump

● run lkcd config

In case of a Linux kernel panic, information about the running kernel, memory usage, and processes is collected and written to the dump partition.

During the next reboot, the contents of this dump partition will be written to a file in /var/log/dump.

17.11.2.3 SLES10: Kdump Kernel Crash Dump Capturing

In SLES10, the implementation how to take crash dumps has changed to a new method. If a kernel panic occurs, a special kernel will be directly started and during the boot pro-cess of this kernel, the dump will be written to a sparse file in /var/log/dump which contains all information that is being kept in memory.

This feature is well integrated in FlexFrame SLES10 Application Nodes and may be acti-vated by specifying a kernel command line option in the PXE configuration file:

● Open the corresponding file in /tftpboot/pxelinux.cfg

● Append crashkernel=64M@16M to the APPEND line

● Reboot the correspronding Application Node(s)

17.11.2.4 Forcing a Crash Dump

There may be situations where the kernel does not initiate a crash dump automatically.

You can manually start a crash dump by using the sysrq shortcuts.

Activate sysrq as follows:

cn1:~ # echo 1 >/proc/sys/kernel/sysrq

To make this setting persistent, set ENABLE_SYSRQ="yes" in /etc/sysconfig/sysctl.

Now a manual’s crash dump capture can be manually initiated by pressing Alt+SysRq+c altogether on the keyboard. On most keyboards, the SysRq key is equal to the PrintScrn key.


18 Abbreviations ABAP Advanced Business Application Programming

ACC Adaptive Computing Controller

ACI Adaptive Computing Infrastructure

ACPI Advanced Configuration and Power Interface

APM Advanced Power Management

APOLC Advanced Planner & Optimizer Life Cache

CCU Console Connection Unit

CIFS Common Internet File System

DART Data Access in Real Time

DHCP Dynamic Host Configuration Protocol

DIT Domain Information Tree

ERP Enterprise Resource Planning

ESF Enhanced System Facility

EULA End User License Agreement

FAA FlexFrame Autonomous Agent

FC Fiber Channel

FTP File Transfer Protocol

IP Internet Protocol

IPMP IP Multipathing

LAN Local Area Network

LDAP Lightweight Directory Access Protocol

LUN Logical Unit Number

MAC Media Access Control

MINRA Minimal Read Ahead

NAS Network Attached Storage

NDMP Network Data Management Protocol

NFS Network File System

Abbreviations


NIC Network Interface Card

NVRAM Non-Volatile Random Access Memory

OBP Open Boot Prom

OLTP On-Line Transaction Processing

ONTAP Open Network Technology for Appliance Products

OSS Open Source Software

POST Power-On Self Test

PCL PRIMECLUSTER

PFS Production File System (on Celerra)

PW PRIMEPOWER

PXE Preboot Execution Environment

PY PRIMERGY

QA Quality Assurance

QS Quality of Service

RAID Redundant Array of Independent (or Inexpensive) Disks

RARP Reverse Address Resolution Protocol

RDBMS Relational Database Management System

RHEL Red Hat Enterprise Linux

RSB Remote Service Board

SCS System Console Software

SAP BW SAP Business Warehouse

SAPGUI SAP Graphical User Interface

SAPOSS SAP Online System Service

SID System Identifier

SLD System Landscape Directory

SLES SUSE Linux Enterprise Server

SMB Server Message Block

SMC System Management Console

SNMP Simple Network Management Protocol

Abbreviations


SPOC Single Point Of Control

TELNET Telecommunications Network

TFTP Trivial File Transfer Protocol

UDP User Datagram Protocol

UPS Uninterruptible Power Supply

VLAN Virtual Local Area Network

VTOC Virtual Table Of Contents

WAN Wide Area Network

WAS Web Application Server

WAFL Write Anywhere File Layout

XSCF Extended System Control Facility


19 Glossary Adaptive Computing Controller

SAP system for monitoring and controlling SAP environments.

Advanced Business Application Programming Proprietary programming language of SAP.

Advanced Power Management Advanced Power Management defines a layer between the hardware and the operat-ing system that effectively shields the programmer from hardware details.

Application Agent A software program for monitoring and managing applications.

Application Node A host for applications (e.g. SAP instances db, ci, agate, wgate, app etc.). This definition includes Application Servers as well as Database Servers.

Automounter The automounter is an NFS utility that automatically mounts directories on an NFS client as they are needed, and unmounts them when they are no longer needed.

Autonomous Agent Central system management and high availability software component of FlexFrame.

Blade A special form factor for computer nodes.

BladeRunner The working title for the solution part of SAP for FlexFrame.

BOOTPARAM Boot time parameters of the kernel.

BRBACKUP SAP backup and restore tools.

Celerra NAS system of EMC.

Checkpoint Restore On EMC Celerra a SnapSure feature that restores a PFS to a point in time using checkpoint information. As a precaution, SnapSure automatically creates a new checkpoint of the PFS before it performs the restore operation.

Client LAN Virtual network segment within FlexFrame, used for client-server traffic.

Glossary


Common Internet File System A protocol for the sharing of file systems (same as SMB).

Computing Node From the SAP ACI perspective: A host that is used for applications.

Control Agent A software program for monitoring and managing nodes within FlexFrame.

Control LAN Virtual network segment within FlexFrame, used for system management traffic.

Control Node A physical computer system, controlling and monitoring the entire FlexFrame land-scape and running shared services in the rack (dhcp, tftp, ldap etc.).

Control Station A Control Node in an SAP ACI environment.

DART Operating system of Celerra data movers (Data Access in Real Time).

Dynamic Host Configuration Protocol DHCP is a protocol for assigning dynamic IP addresses to devices on a network.

Dynamic Host Configuration Protocol server A DHCP server provides configuration parameters specific to the DHCP client host, required by the host to participate on the Internet.

EMC NAS Network attached storage for file systems of EMC.

Enterprise Resource Planning Enterprise Resource Planning systems are management information systems that in-tegrate and automate many of the business practices associated with the operations or production aspects of a company.

Ethernet

A Local Area Network which supports data transfer rates of 10 megabits per second.

Fiber Channel Fiber Channel is a serial computer bus intended for connecting high-speed storage devices to computers.

Filer Network attached storage for file systems of NetApp.

FlexFrame A joint project in which the main partners are SAP, Network Appliance, Intel and Fu-jitsu Siemens Computers.

Glossary


FlexFrameTM for SAP® FlexFrame for SAP® is a radically new architecture for SAP environments. It exploits the latest business-critical computing technology to deliver major cost savings for SAP customers.

FlexFrame internal LAN Switch Cisco network switches which are integral part of the FlexFrame for SAP hardware configuration and which are automatically configured by the FlexFrame for SAP soft-ware.

Gigabit Ethernet A Local Area Network which supports data transfer rates of 1 gigabit (1,000 mega-bits) per second.

Host name The name of a node (assigned to an interface) that is resolved to a unique IP ad-dress. One node can have multiple host names (cf. node name). In SAP environ-ments host names are currently limited to 13 alphanumeric characters including the hyphen (“ - “). The first character must be a letter. In the SAP environment host names are case-sensitive.

Image In the FlexFrame documentation, “Image” is used as a synonym for “Hard Disk Im-age”.

Internet Protocol Address A unique number used by computers to refer to each other when sending information through networks using the Internet Protocol.

Lightweight Directory Access Protocol Protocol for accessing on-line directory services.

Local Area Network A computer network that spans a relatively small area. Most LANs are confined to a single building or group of buildings. However, one LAN can be connected to other LANs over any distance via telephone lines and radio waves. A system of LANs con-nected in this way is called a Wide Area Network (WAN).

Local host name The name of the node (physical computer); it can be displayed and set using the command /bin/hostname.

Logical Unit Number An address for a single (SCSI) disk drive.

MAC address Device identifier number of a Network Interface Card. In full: "media access control address".

Glossary


MaxDB A relational database system from mySQL (formerly ADABAS and SAPDB).

Media Access Control address An identifier for network devices, usually unique. The MAC address is stored physi-cally on the device.

NAS system Network Attached Storage of any vendor (in our context: EMC NAS or NetApp Filer).

NDMPcopy NDMPcopy transfers data between Filers using the Network Data Management Pro-tocol (NDMP).

Netboot A boot procedure for computers where the operating system is provided via a net-work instead of local disks.

Netweaver

SAP NetWeaver is the technical foundation of SAP solutions.

Network Appliance Filer See “Filer”.

Network Attached Storage A data storage device that is connected via a network to one or multiple computers.

Network File System A network protocol for network-based storage access.

Network Interface Card A hardware device that allows computer communication via networks.

Node A physical computer system controlled by an OS.

Node name The name of a physical node as returned by the command uname -n. Each node name within a FlexFrame environment must be unique.

Non-Volatile Random Access Memory A type of memory that retains its contents when the power is turned off.

On-Line Transaction Processing Transaction processing via computer networks.

OpenLDAP An Open Source LDAP Service Implementation.

Open Network Technology for Appliance Products The operating system of Network Appliance Filers.

Glossary


Open Source Software Software that is distributed free of charge under an open source license, such as the GNU Public License.

Oracle RAC A cluster database by Oracle Corporation.

Physical host Name of a physical computer system (node).

Power-On Self Test Part of a computer's boot process; automatic testing of diverse hardware compo-nents.

Preboot Execution Environment An environment that allows a computer to boot from a network resource without hav-ing a local operating system installed.

PRIMECLUSTER Fujitsu Siemens Computer’s high-availability and clustering software.

PRIMEPOWER

Fujitsu Siemens Computer's SPARC-based server product line.

PRIMERGY

Fujitsu Siemens Computer's i386-based server product line.

Red Hat Enterprise Linux Linux distribution by Red Hat, Inc., targeting business customers.

Reverse Address Resolution Protocol A protocol allowing resolution of an IP address corresponding to a MAC address.

SAP Service In FlexFrame: SAP Service and DB Services.

SAP service script An administration script for starting and stopping an SAP application on a virtual host.

SAP Solution Manager Service portal for the implementation, operation and optimization of an SAP solution.

SAPLogon Front-end software for SAPGUI.

SAPRouter Router for SAP services like SAPGUI or SAPTELNET.

SavVol A Celerra volume to which SnapSure copies original point-in-time data blocks from the PFS before the blocks are altered by a PFS transaction.

Glossary


Server A physical host (hardware), same as node.

Service A software program providing functions to clients.

Service type The type of an application or service (db, ci, app, agate, wgate etc.).

Single Point of Control In FlexFrame: One user interface to control a whole FlexFrame environment.

Storage LAN A virtual LAN segment within a FlexFrame environment, carrying the traffic to NAS systems.

SUSE Linux Enterprise Server A Linux distribution by Novell, specializing in server installations.

Telecommunications Network A terminal emulation program for TCP/IP networks such as the Internet.

Trivial File Transfer Protocol A simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram Protocol (UDP) and provides no security features. It is often used by servers to boot diskless workstations, X-terminals, and routers.

TFTP server A simple FTP implementation.

Virtual host The name of the virtual host on which an application runs; it is assigned to a physical node when an application is started.

Virtual Local Area Network A VLAN is a logically segmented network mapped over physical hardware according to the IEEE 802.1q standard.

Virtualization Virtualization means the separation of hardware and processes. In a virtualized envi-ronment (FlexFrame), a process can be moved between hardware nodes while stay-ing transparent to the user and application.

.


20 Index A

abbreviations 335

ACC

integration of new servers 275

integration of pools 275

integration of SAP serevices 275

usage of 276

user administration 276

Application

removing from monitoring by FA Agents 272

stopping ans starting for upgades using 273

application maintenance mode 85

Application Nodes 10

adding 124

administrating 119

and SAN 129

create new Linux OS image 144

displaying information on a specific 119

displaying information on all 122

installing image update 142

Linux 10

listing 119

reactivating after power shutdown by FA Agents 62

removing 127

renaming 128

Solaris 11

state of 73

upgrading RPM packages 158

application services

starting and stopping 267

application software

upgrading 158

automounter concept 47

B

Blade Server Cabinets

adding 132

administrating 130

listing 130

removing 136

C

cloning of SIDs 258

cluster file system, built 52

Cluster Foundation 81

Cluster Foundation (CF) 38

common network configuration parameters

display and change 17

configuration

power-shutdown 246

Control Node

defect hardware 104

exchanging 103

failed hardware 104

Index


OS damaged 104

software updates 105

Control Nodes

backup and restore 304

Control Nodes 10

SSH configuration 234

D

DART patches

installing 116

data protection 293

Debugging

Linux kernel 333

script 332

desaster repair 44

document history 2

Domain Information Tree (DIT) 36

E

EMC Celerra


backup with EMC SnapSure 301

EMC SnapSure 301

error handling 311

F

FA Agent version

install new 110

migration on pool level 112

FA Agents

configuring 279

error diagnosis 314

FlexFrame Autonomy 283

groups 280

operation and log files 318

power management 289

traps 249, 282

FA Autonomous Agents 76

FA Autonomy

diagnostic tool 323

FA migration tool 114

file mode 114

parameters 115

pool mode 114

ff_change_id.pl 258

ff_install_an_linux_images.sh 243

ff_list_services.sh 266

ff_netscan.sh 65

ff_pool_dnssrv.sh 72

Filer cluster 52

Filer volumes

backup with NetApp SnapShot 294

FlexFrame

architecture 3

backup with tape library 303

basic administration 59

general notes 3

network 64

FlexFrame Autonomous Agents

migration at pool level 111

FlexFrame Autonomy

Index


command line interface 112

FlexFrame configuration state, displaying 63

FlexFrame landscape

accessing 59

power-off 61

power-on 59

FlexFrame Web portal 75

G

glossary 339

H

hardware 4

hardware changes 95

host

database 189

I

image customization for experts 172

inconsistent and faulted applications 83

J

Jakarta Tomcat 76

L

LDAP 36, 258

FlexFrame structure in 37

working with 37

LDAP error codes and messages 312

Linux Application Nodes

preparation 243

Linux images

installing new 141

Linux kernel

update/install new 108

update/install new 150

log files 311

N

NAS

add new 26

adding a blade 34

disk free 35

display all configured 29

display configuration 30

remove 35

NAS configuration 11

NAS patches

installing 116

NAS systems

multiple 258

support 52

nb_unpack_bi 244

NetApp Filer


NetApp Filers

configuring SNMP traps 29

NetApp SnapShot 294

network 45

automounter concept 47

LAN failover 45

segments 45

switches 47

Index


Network Appilance Filer 51

network cards

replacing 102

network errors 312

NFS mount messages 312

node failures 40

notational conventions 1

O

ONTAP patches

installing 116

P

password management

Control Nodes 236

Linux Application Nodes 241

Solaris Application Nodes 242

passwords

requested during installation 233

setting during installation 234

setting during operation, update and upgrade 235

pool

add to a NAS 31

remove from a NAS 33

pool default router

changing 186

pool DNS domain

changing 185

pool groups

changing group and pool assignment of Application Nodes 189

changing group assignment of Application Nodes 188

removing 188

pools

adding 175

adding a group 187

listing details 179, 183

removing 178

state of 73

pools and groups 175

power control hardware

replacing 103

power-shutdown configuration 246

PRIMECLUSTER 38, 76

administration 77

CLI commands 41

log files 44

problems 332

PRIMECLUSTER components 38

R

related documents 2

Reliant Monitor Services 82

Reliant Monitor Services (RMS) 38

remote administration 59

requirements 1

RMS configuration

FlexFrame specific 38

schematic overview 39

S

SAN

Index


basic layers 55

configuration in a FlexFrame environment 199

rules and restrictions with FlexFrame 57

scope of the FlexFrame integration 56

support 55

SAP ACC 275

hanging locks 331

logging 330

missing serverin ACC physical landscape 331

reset of service status in case of failures 331

troubleshooting 330

SAP databases

backup 296, 302

SAP kernel

updates and patches 263

SAP service

scripts 267

starting and stopping without root privileges 267

SAP service scripts

actions 268

return code 270

user exits 269

SAP services

administrating 265

display status 265

list status 266

starting and stopping multiple 271

SAP SIDs

adding instances 252

removing instances 252

SAP SIDs

adding 252

cloning 254

data transfer when cloning 255

listing 251

listing instances 251, 254

removing 252

SAP system

upgrading 262

Sap systems

administrating 251

SAP systems

state of 73

Sapapp

specific error messages 330

Sapascs


Sapci


SAPDB


Sapj


Sapjc

Index



Sapscs


scripts

ff_install_an_linux_images.sh 243

ff_list_services.sh 266

ff_netscan.sh 65

ff_pool_dnssrv.sh 72

nb_unpack_bi 244

security 233

ServerView

update 155

ServerView S2 86

ServerView update via RPM 106

service packs 150

service switchover 273

shared operating system 8

shared operating system, boot concept 8

Shutdown Facility (SF) 38

SID instances

state of 74

snapshot 52

SnapShot

restore 300

Snapshots

restore 302

software 5

software updates 105

Solaris Application Nodes

preparation 244

Solaris images

install new/activate 160

maintenance cycle 165

rc script 164

troubleshooting 173

special files 322

SSH configuration 234

start/stop script errors 324

start/stop scripts

common error messages 324

state

of Applicatrion Nodes 73

of pools 73

of SAP systems 73

of SID instances 74

support deartment

diagnosis data 323

switch

add to a switch group 11

remove from a switch group 13

switch application

usage example 42

switch applications 85

switch blade

changing name 138

changing password 139

changing type 137

getting initial configuration 140

Index


switch blades

replacing 104

switch configuration 11

backing up 309

restoring 310

switch group

adding 19

change host name 16

change password 16

list configuration 14

removing 22

switch port

add configuration 22

display configuration 25

remove configuration 24

T

Third Party software 117

U

user administration 193, 235

V

volume layout 52

volumes

multiple 258

W

Web interfaces 75

Information on this document On April 1, 2009, Fujitsu became the sole owner of Fujitsu Siemens Compu-ters. This new subsidiary of Fujitsu has been renamed Fujitsu Technology So-lutions.

This document from the document archive refers to a product version which was released a considerable time ago or which is no longer marketed.

Please note that all company references and copyrights in this document have been legally transferred to Fujitsu Technology Solutions.

Contact and support addresses will now be offered by Fujitsu Technology So-lutions and have the format …@ts.fujitsu.com.

The Internet pages of Fujitsu Technology Solutions are available at http://ts.fujitsu.com/... and the user documentation at http://manuals.ts.fujitsu.com.

Copyright Fujitsu Technology Solutions, 2009

Hinweise zum vorliegenden Dokument Zum 1. April 2009 ist Fujitsu Siemens Computers in den alleinigen Besitz von Fujitsu übergegangen. Diese neue Tochtergesellschaft von Fujitsu trägt seit-dem den Namen Fujitsu Technology Solutions.

Das vorliegende Dokument aus dem Dokumentenarchiv bezieht sich auf eine bereits vor längerer Zeit freigegebene oder nicht mehr im Vertrieb befindliche Produktversion.

Bitte beachten Sie, dass alle Firmenbezüge und Copyrights im vorliegenden Dokument rechtlich auf Fujitsu Technology Solutions übergegangen sind.

Kontakt- und Supportadressen werden nun von Fujitsu Technology Solutions angeboten und haben die Form …@ts.fujitsu.com.

Die Internetseiten von Fujitsu Technology Solutions finden Sie unter http://de.ts.fujitsu.com/..., und unter http://manuals.ts.fujitsu.com finden Sie die Benutzerdokumentation.

Copyright Fujitsu Technology Solutions, 2009