Upload
-
View
135
Download
0
Tags:
Embed Size (px)
Citation preview
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 1 of 156
IBM SAP Technical Brief
Implementing and Using
SAP NetWeaver Landscape Virtualization Management
on IBM Power Systems and IBM Flex System POWER Nodes
IBM SAP International Competence Center
Walldorf, Germany
Version: 2.0
Status: December 2013
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 2 of 156
Preface Edition Notice (December 2013)
SAP NetWeaver Landscape Virtualization Management software (SAP LVM) is the
successor and extension of the SAP Adaptive Computing Controller. This document is
complement to the ISICC Implementation Guide “SAP’s Adaptive Computing on Power
Systems”, Version 3.0, August 2011. The paper focuses on the new features and uses-cases
provided with SAP LVM in conjunction with IBM Systems Director managed Power
Systems. It describes the IBM infrastructure specifics required to implement and operate an
on-premise Cloud Solution for SAP landscapes.
The paper covers a solution stack tested during a Proof-of-Concept at the ISICC. This
includes the following components:
SAP Business Suite 7 with IBM DB2 UDB LUW 9.7/10.1 databases
IBM PureFlex Systems
AIX 6 and AIX 7 operating systems
IBM Flex System Manager
IBM Tivoli Storage FlashCopy Manager
IBM SAN Volume Controller
IBM XIV Storage System
At the time of writing, SAP LVM V2.0 SP3 is generally available.
Scope and Audience
People reading this document should understand core components and pre-requisites of the
SAP and Systems Software solution stack in an IBM POWER environment including their
installation and handling. They should understand the typical use-cases and their execution
and effects on a POWER/AIX based SAP infrastructure.
The document is intended for IBM (practitioners from Pre- & Post-Sales and Services,
GBS/GTS, ATS, eTS,) and customer personnel involved in the implementation of SAP
Landscape Virtualization Management software on IBM Power Systems. Since we do not
repeat existing documentation and technical information from both IBM and SAP the readers
are expected to have a solid knowledge in AIX, PowerVM virtualization, IBM Systems
Director and its plug-ins, IBM Flex System Manager, IBM Tivoli Storage FlashCopy
Manager and SAP Basis administration.
Authors
Dr. Edmund Haefele, IBM Technical Sales for SAP
Dr. Thomas Hebert, IBM Technical Sales for SAP
Maik Gasterstaedt, IBM SAP International Competence Center
Walter Orb, IBM SAP International Competence Center
Matthias Koechl, IBM SAP International Competence Center
Arnold Beilmann, IBM Research & Development
Christoph Langer, IBM Research & Development
Feedback
We are interested in any feedback you have. Please send your comments to
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 3 of 156
Disclaimer
This document is subject to change without notification and will not comprehensively cover
the issues encountered in every customer situation. It should be used only in conjunction with
the product literature accompanying the products listed above. The information contained in
this document has not been submitted to any formal IBM test and is distributed AS IS.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 4 of 156
1 INTRODUCTION ....................................................................................................................................... 9
1.1 BACKGROUND AND BUSINESS CONTEXT ................................................................................................... 9 1.2 ENABLING END-TO-END MANAGEMENT THROUGH INTEGRATION ............................................................ 9 1.3 SCOPE OF THIS COOKBOOK ...................................................................................................................... 10 1.4 SAP NETWEAVER LANDSCAPE VIRTUALIZATION MANAGEMENT SOFTWARE (SAP LVM) .................... 11 1.5 SAP LVM VERSION 2.0 UPDATE............................................................................................................. 13
2 OVERVIEW AND PLANNING ............................................................................................................... 15
2.1 ARCHITECTURE OVERVIEW ...................................................................................................................... 15 2.2 COMBINING THE IBM AND SAP LAYERS ................................................................................................. 19 2.3 PLANNING PROJECT RESOURCES AND TIMING ......................................................................................... 25 2.4 TESTED CONFIGURATION ......................................................................................................................... 27 2.5 SAP LVM ENVIRONMENT ....................................................................................................................... 28
3 INSTALLATION ...................................................................................................................................... 37
3.1 NAMING CONVENTIONS ........................................................................................................................... 38 3.2 INSTALL STORAGE INFRASTRUCTURE ...................................................................................................... 39 3.3 INSTALL MANAGEMENT ENVIRONMENT .................................................................................................. 49 3.4 INSTALL SYSTEM LANDSCAPE ................................................................................................................. 57
4 INITIAL LVM CONFIGURATION ....................................................................................................... 78
4.1 ENGINE SETTINGS .................................................................................................................................... 78 4.2 VIRTUALIZATION MANAGER CONFIGURATION IN SAP LVM .................................................................. 78 4.3 STORAGE MANAGER CONFIGURATION .................................................................................................... 80 4.4 CONFIGURE SAP LVM POOLS ................................................................................................................ 82 4.5 CONFIGURE NETWORKS .......................................................................................................................... 83 4.6 DISCOVERY OF HOSTS (LPARS) .............................................................................................................. 84 4.7 INSTANCE CONFIGURATION/ DISCOVERY IN SAP LVM .......................................................................... 86 4.8 ADDITIONAL CONFIGURATION STEPS ...................................................................................................... 93 4.9 IBM FLEX SYSTEM MANAGER CONFIGURATION ..................................................................................... 94
5 DETAILED DESCRIPTION / EXECUTION OF USE CASES ........................................................... 99
5.1 LVM STANDARD EDITION USE CASES .................................................................................................... 99 5.2 SAP LVM ENTERPRISE EDITION USE CASES ........................................................................................ 109
6 TROUBLESHOOTING HINTS AND TRICKS ................................................................................... 151
6.1 MANUAL CLEANUP OF SAP LVM COPY PROCESS .................................................................................. 151 6.2 DELETE THE SYSTEM IN SAP LVM ....................................................................................................... 153
7 RESOURCES ........................................................................................................................................... 154
7.1 IBM DOCUMENTATION ......................................................................................................................... 154 7.2 SAP DOCUMENTATION AND SAP NOTES .............................................................................................. 154
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 5 of 156
List of figures
Figure 1 Logical connections of IBM and SAP components ................................................... 10
Figure 2 Generic SAP NetWeaver LVM capabilities and LVM 2.0 updates (green) .............. 11 Figure 3 SAP LVM Block Diagram (Source: SAP) ................................................................ 16 Figure 4 IBM Systems Director and Plug-Ins .......................................................................... 18 Figure 5 Software structure to integrate IBM IT layers ........................................................... 20 Figure 6 Generic SAP System Copy Workflow ....................................................................... 21
Figure 7 Architecture overview diagram ................................................................................. 28 Figure 8 Redundant virtual fibre channel attachment .............................................................. 33 Figure 9 System configuration in SAP LVM with distributed file system layout ................... 39 Figure 10 Example configuration of an SAP system using GPFS ........................................... 44
Figure 11 Volume Group and Filesystem layout ..................................................................... 46 Figure 12 Volumes of SAP system CL6 .................................................................................. 48 Figure 13 SAP System CL6 in LVM and mount points .......................................................... 49 Figure 14 Overview of the FlashCopy Manager Installation ................................................... 51
Figure 15 Sample LDAP Directory Information Tree for AIX user management .................. 54 Figure 16 Start IPsec ................................................................................................................ 60 Figure 17 Define the Host as “Isolation Ready” in SAP LVM ................................................ 60 Figure 18 Filesystem for Oracle Redo Logs ............................................................................ 72
Figure 19 LVM Engine Settings .............................................................................................. 78 Figure 20 Virtualization Manager Configuration ..................................................................... 78
Figure 21 Virtualization Manager: Virtualization Manager Types .......................................... 79 Figure 22 Virtualization Manager: Virtualization Manager Properties ................................... 79
Figure 23 Virtualization Manager: Summary .......................................................................... 80 Figure 24 Storage Manager Configuration ............................................................................... 80 Figure 25 Storage Manager: Storage Manager type ................................................................. 80
Figure 26 Storage Manager: Storage Manager Properties ....................................................... 81 Figure 27 Storage Manager: Storage Systems ......................................................................... 82
Figure 28 SAP LVM Pools: Pool Overview ............................................................................ 83 Figure 29 SAP LVM Network Configuration .......................................................................... 83 Figure 30 SAP LVM Network: Basic Configuration ............................................................... 84 Figure 31 SAP LVM Network: Advanced Configuration ....................................................... 84
Figure 32 SAP LVM Network: Summary ................................................................................ 84 Figure 33 SAP LVM hosts: Discover Using Host and Instance Agent ................................... 85 Figure 34 SAP LVM Hosts: Add Hosts ................................................................................... 85
Figure 35 SAP LVM Hosts: Edit host properties ..................................................................... 86 Figure 36 SAP LVM Systems: Discover new SAP instances .................................................. 86 Figure 37 SAP LVM Systems: Find Instances/Hosts .............................................................. 87 Figure 38 SAP LVM Systems: Add Instances – Modify Host Names .................................... 88
Figure 39 SAP LVM Systems: Add Instances – Auto Assign to New or Existing System ..... 88 Figure 40 SAP LVM Systems: Basic Configuration ............................................................... 89 Figure 41 SAP LVM Systems: Provisioning & RFC ............................................................... 89 Figure 42 SAP LVM Systems: Network Isolation ................................................................... 90 Figure 43 SAP LVM Systems: Network Isolation - Add allowed communications ............... 90
Figure 44 Database Configuration: Instance Details ................................................................ 91 Figure 45 Database Configuration: Basic configuration .......................................................... 91
Figure 46 Database Configuration: Instance properties ........................................................... 92 Figure 47 Database Configuration: Mount points .................................................................... 92 Figure 48 Central Services: Mount points ................................................................................ 93 Figure 49 Overview of systems and instances ......................................................................... 93
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 6 of 156
Figure 50 System and AS Provisioning Configuration ............................................................ 94 Figure 51 System Copy Configuration: Release configuration type ........................................ 94 Figure 52 System Copy Configuration: Basic properties ......................................................... 94 Figure 53 System Copy Configuration: Installation master ..................................................... 94
Figure 54 IBM Flex System Manager Security: Group membership ...................................... 96 Figure 55 IBM Flex System Manager Security: Create new roles .......................................... 96 Figure 56 IBM Flex System Manager Security: Modify role permissions .............................. 97 Figure 57 IBM Flex System Manager Security: Define static groups ..................................... 98 Figure 58 IBM Flex System Manager Security: Assign groups and roles to user group ......... 98
Figure 59 IBM PowerVM Relocate ....................................................................................... 101 Figure 60 SAP LVM Relocate ............................................................................................... 101 Figure 61 Prepare SAP Instance ............................................................................................. 103
Figure 62 Start SAP system: Mass Operations ...................................................................... 104 Figure 63 Logs of a 'Start SAP system' Operation ................................................................. 104 Figure 64 Relocate SAP System: Mass Operation ................................................................. 105 Figure 65 Shutdown Host ....................................................................................................... 106
Figure 66 SAP LVM Dashboard ............................................................................................ 107 Figure 67 SAP LVM Infrastructure Visualization ................................................................. 108 Figure 68 SAP LVM Performance Monitoring ...................................................................... 109 Figure 69 SAP System Copy Scenarios Supported by SAP LVM (Source SAP) ................. 110
Figure 70 Enable SAP System for SAP System Rename ...................................................... 112 Figure 71 Start SAP System Clone ........................................................................................ 114 Figure 72 SAP System Clone: Basic Data of Target System ................................................. 114
Figure 73 SAP System Clone: Host Selection of Target System ........................................... 115
Figure 74 SAP System Clone: Virtual Host Names and Networks ....................................... 117 Figure 75 SAP System Clone: Storage Volumes ................................................................... 117 Figure 76 SAP System Clone: Database consistency ............................................................ 118
Figure 77 SAP System Clone: Define Allowed Outgoing Connections for System Isolation
................................................................................................................................................ 118
Figure 78 SAP System Clone: Summary ............................................................................... 119 Figure 79 Define RFC Destination for PCA Execution ......................................................... 121 Figure 80 Start SAP System Copy ......................................................................................... 121
Figure 81 SAP System Copy: Basic Data of Target System .................................................. 122 Figure 82 SAP System Copy: Host Selection of Target System ............................................ 122
Figure 83 SAP System Copy: Virtual Host Names and Networks ........................................ 123 Figure 84 SAP System Copy: SAP Instance Numbers .......................................................... 123
Figure 85 SAP System Copy: Storage Volumes .................................................................... 124 Figure 86 SAP System Copy: SVC CLI Warning Message .................................................. 124 Figure 87 SAP System Copy: Database consistency ............................................................. 125 Figure 88 SAP System Copy: User and Group Management ................................................ 125 Figure 89 SAP System Copy: SAPinst Release Configuration .............................................. 126
Figure 90 SAP System Copy: Define Allowed Outgoing Connections for System Isolation 126 Figure 91 SAP System Copy: Post Copy Automation ........................................................... 127 Figure 92 SAP System Copy: Select Task List ...................................................................... 127 Figure 93 SAP System Copy: Summary View ...................................................................... 128 Figure 94 SAP System Refresh: Start System Refresh .......................................................... 129
Figure 95 SAP System Refresh: Basic Data of System to be Refreshed ............................... 129 Figure 96 SAP System Refresh: Host Selection of Target System ........................................ 130
Figure 97 SAP System Refresh: Virtual Host Names and Networks .................................... 130 Figure 98 SAP System Refresh: Storage Volumes ................................................................ 131 Figure 99 SAP System Refresh: Database consistency ......................................................... 131
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 7 of 156
Figure 100 SAP System Refresh: User and Group Management .......................................... 132 Figure 101 SAP System Refresh: SAPinst Release Configuration ........................................ 132 Figure 102 SAP System Refresh: Define Allowed Outgoing Connection for System Isolation
................................................................................................................................................ 133
Figure 103 SAP System Refresh: Post Copy Automation ..................................................... 133 Figure 104 SAP System Refresh: Storage Volumes .............................................................. 134 Figure 105 SAP System Refresh: Summary .......................................................................... 135 Figure 106 Start SAP System Rename ................................................................................... 136 Figure 107 SAP System Rename: Basic Data of Target System ........................................... 136
Figure 108 SAP System Rename: Virtual Hostnames and Networks .................................... 137 Figure 109 SAP System Rename: SAP Instance numbers ..................................................... 137 Figure 110 SAP System Rename: Mount Data ...................................................................... 138
Figure 111 SAP System Rename: User and Group Management .......................................... 139 Figure 112 SAP System Rename: SAPinst Release Configuration ....................................... 139 Figure 113 SAP System Rename: Define Allowed Outgoing Connections for System Isolation
................................................................................................................................................ 140
Figure 114 SAP System Rename: Post Copy Automation .................................................... 140 Figure 115 SAP System Rename: Summary .......................................................................... 141 Figure 116 Provider Implementation Definition (I) ............................................................... 144 Figure 117 Provider Implementation Definition (II) .............................................................. 144
Figure 118 Provider Implementation Definition (III) ............................................................ 145 Figure 119 Provider Implementation Definition (IV) ............................................................ 145 Figure 120 Custom Hook Definition (I) ................................................................................. 145
Figure 121 Custom Hook Definition (II) ............................................................................... 146
Figure 122 Custom Hook Definition (III) .............................................................................. 146 Figure 123 System State in Operations View ........................................................................ 147 Figure 124 Monitoring View .................................................................................................. 148
Figure 125 Mass Stop and Unprepare Completed ................................................................. 148 Figure 126 Destroy System .................................................................................................... 149
Figure 127 System Destroy: Delete Storage Volumes ........................................................... 149 Figure 128 System Destroy: Host Names .............................................................................. 150 Figure 129 System Destroy: Summary .................................................................................. 150
Figure 130 Remove System in SAP LVM ............................................................................. 153
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 8 of 156
List of tables
Table 1 Core Functionalities in SAP LVM .............................................................................. 13
Table 2 Supported SAP releases for the SAP LVM scenario .................................................. 15 Table 3 Overview of the Scenarios .......................................................................................... 24 Table 4 Project Phases and estimated time .............................................................................. 27 Table 5 Software Stack ............................................................................................................ 28 Table 6 Operational Model ...................................................................................................... 37
Table 7 Sample directory structure for a distributed file system implementation ................... 39 Table 8 Entries in /etc/services ................................................................................................. 61 Table 9 Local Filesystem layout .............................................................................................. 62 Table 10 XIVGUI Install directory .......................................................................................... 63
Table 11 Minimum filesystem layout (DB2 UDB) .................................................................. 69 Table 12 Extended filesystem layout (DB2 UDB) ................................................................... 70 Table 13 Minimum filesystem layout (Oracle) ........................................................................ 70 Table 14 Extended filesystem layout (Oracle) ......................................................................... 71
Table 15 Storage Connection Methods .................................................................................... 81 Table 16 Storage Connection user credentials ......................................................................... 82 Table 17 Required Setup Procedures ..................................................................................... 113 Table 18 Examples for IP addresses ...................................................................................... 115
Table 19 List of SAP Notes .................................................................................................... 155
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 9 of 156
1 Introduction
1.1 Background and Business Context
In the era of worldwide business relations and processes, the flexible and non-disruptive
operation of business applications and their underlying middleware and hardware becomes an
essential factor for success. Even unforeseen fluctuations in transaction volumes and changing
business processes must be reflected immediately by the supporting system capacities. On the
other hand, planned interventions such as hardware or application maintenance and testing
must not disrupt SAP Business Suite availability to worldwide end users, partners and
consumers. In order to accelerate innovation and fast adoption of new business processes,
non-production stages of SAP systems need to be deployed on demand, without a long lead-
time for procurement and installation.
Consequently, manually operating and administrating a growing number of individual
systems or landscape components is no longer an option. The concepts of cloud computing
promise to save costs and will increase flexibility, elasticity, and automation of system
operations to efficiently serve the needs of the business.
This paper describes how to design and implement an on-premise cloud environment by
combining IBM platform technologies, namely IBM Flex System Manager, IBM Systems
Director, and Tivoli Flash Copy Manager with the SAP NetWeaver® Landscape
Virtualization Management (SAP LVM) environment. Finally, the paper shows how all the
use-cases, including cloning, copying, and refreshing of SAP systems, are managed via the
SAP LVM administrator user interface.
1.2 Enabling End-to-End Management through Integration
The major advantage of the solution stack described herein is the enablement of end-to-end
administration scenarios. This covers both aspects of IT management and SAP basis
administration by:
1. Providing a unified administrator interface for SAP topology monitoring and
launching automated tasks
The SAP basis administrator is familiar with SAP terminologies and user interfaces.
Hence, infrastructure complexity and platform specialties are masked and abstracted to
that level. Via the SAP LVM Administrator GUI IT tasks are provided to the SAP
administrator as self-services. At the same time, the IT administrator keeps control
over the infrastructure when enabling the self-services using IBM Systems Director or
IBM Flex System Manager.
2. Programmatic integration of IBM and SAP software layers for automation
SAP knows best how to manage SAP components and applications. IBM knows best
how to manage the infrastructure components consisting of IBM servers, storage,
virtualization technology and operating systems. For this reason, IBM platform
capabilities and software are used for all infrastructure related tasks. For the same
reason, SAP NetWeaver Landscape Virtualization Management complements these
for all management and automation functions related to the SAP software stack.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 10 of 156
Figure 1 Logical connections of IBM and SAP components
The combination of IBM platforms’ native virtualization and automation capabilities with the
SAP management products establishes an excellent foundation for deploying and operating
SAP landscapes in an on-premise cloud environment. Consumers of the cloud services are the
administrators, not the end-users, though. Line of Business users benefit from faster
availability of systems, more flexible testing environments, and demand driven capacity of
server and storage resources.
1.3 Scope of this Cookbook
This cookbook describes the specifics and installation steps for implementing the SAP
NetWeaver Landscape Virtualization Management software V2.0 on an IBM PureFlex
System infrastructure. It also provides how-to’s for using the resulting administration
environment for launching and monitoring the most popular SAP “cloud use cases”.
The installation sequence has been tested in a Proof-of-Concept including
SAP Business Suite 7
DB2 UDB LUW
IBM POWER
AIX 6 and AIX 7
IBM Flex System Manager
IBM Tivoli Storage FlashCopy Manager
SAN attached disk storage using
o IBM System Storage SAN Volume Controller
o IBM Storwize V7000
o IBM XIV Storage System
Because of the technical affinity of Flex System Manager (FSM) and IBM Systems Director,
the content applies to standalone Power systems too. This has been verified in a lab
environment and in various customer proof of concept projects.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 11 of 156
SAP LVM V1.0 became available in June 2012 and was baseline for the initial version of this
document. Since October 2013, SAP LVM version 2.0 is generally available. Our team
participated in the SAP LVM 2.0 ramp-up program since April 2013. Therefore, the current
document is based on SAP LVM 2.0 already.
The fact that we describe a POWER based implementation does in no way mean other
hardware and virtualization platforms (including IBM System x) do not offer a comparable
degree of SAP LVM integration.
The cookbook does not replace a solid technical knowledge of any of the involved layers and
is no substitute for existing literature by IBM and SAP. We will point to applicable literature
where needed.
1.4 SAP NetWeaver Landscape Virtualization Management software (SAP LVM)
SAP NetWeaver® Landscape Virtualization Management software allows customers to
monitor and manage entire SAP software landscapes and their associated heterogeneous IT
infrastructures. SAP LVM provides a single point of control, which allows visualizing,
monitoring, and managing data center tasks. The managing aspect includes resource
allocation and de-allocation on IT infrastructure level, but also on SAP Business Suite
application level.
Figure 2 Generic SAP NetWeaver LVM capabilities and LVM 2.0 updates (green)
SAP LVM is an evolution of the previously established SAP Adaptive Computing Controller
(ACC 7.3). As such SAP LVM inherits functions like “Landscape-wide Visualization and
Monitoring” and “Automated Capacity Management” from the former ACC.
In addition, SAP LVM provides new and more powerful functions like “SAP System
Cloning” and “SAP System Copy / Refresh”, which are frequently performed operations at
SAP sites. They combine numerous steps on infrastructure and SAP application level
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 12 of 156
including Post Copy Automation (PCA) sequences. SAP PCA is only available with the SAP
LVM “Enterprise” Edition.
SAP LVM Standard Edition is included in a SAP NetWeaver 7.x license without additional
fees. The SAP LVM Enterprise Edition is a separately licensed product with additional SAP
license fees.
SAP LVM provides all the administrative functionalities as listed in Table 1. Functions that
require SAP LVM Enterprise Edition are highlighted in blue and italic font.
The Standard Edition establishes the basic framework and includes administrative functions
that have been available with SAP Adaptive Compute Controller (SAP ACC) in the past.
These are related to landscape monitoring and automated startup, stop, and relocate of SAP
instances.
The Enterprise Edition extends these core capabilities with the advanced functions including/
involving SAP Post Copy Automation sequences. One can mostly automate complex and
frequently applied SAP system management tasks. This includes generation of isolated clones
of a SAP production system for testing purposes, but also replication of a production system
into a QA or pre-production-system that stay fully integrated within the existing transport
stages. For latter purpose, business data and system settings need to be replicated, while at all
times avoiding any impact on the online production environment.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 13 of 156
SAP LVM Functionalities
End-to-End SAP System Clone / Copy / Refresh for ABAP
End-to-End SAP System Clone / Copy / Refresh for JAVA
Post-Copy Automation (PCA) for ABAP and JAVA
Automatic Capacity Management (ACM)
Application Server Installation / Un-installation
Mass Operations (start / stop / relocate)
User Configurable Dashboards
Reporting
Landscape Visualization
Custom Operations and Custom Hooks (Extensibility)
Custom Services (Management of Non-SAP Components)
Table 1 Core Functionalities in SAP LVM
1.5 SAP LVM Version 2.0 Update
SAP LVM version 2.0 introduced a number of new features. Evolutionary enhancements have
been made regarding architecture and functionality, and new features had been implemented.
The affected areas are highlighted in Figure 2. The most important changes are described in
the following three categories:
SAP LVM Administrator GUI Changes
The user interface allows for more customization. In addition, visualization of SAP
landscapes has been improved. Here it is important to note that SAP has changed terminology
of managed entities:
LVM 1.0 SAP 2.0
Service Instance or System
Resource Host
Controller Log Log
Customer scripts can extend the default SAP LVM functionality. For these custom operations,
new parameters can be defined and applied to managed systems.
A search facility allows you to easier identify and navigate to certain instances.
System Provisioning Changes
In general, LVM 2.0 has introduced some application aware functions for provisioning tasks.
E.g., the near-zero downtime (nZDM) cloning scenario for SAP NetWeaver Portal (in SP8)
provides a pre-canned procedure for the implementation of support package stacks on a
Portal system at minimum business downtime. This minimizes business downtime for all
users accessing SAP backend systems through this portal.
Further, end-to-end copy and refresh workflows have been extended respectively introduced
for SAP ERP/ECC, CRM, SCM and SAP NetWeaver BI. Optionally, these can include the
customizable post-copy automation (PCA) phase. PCA can also be launched as standalone
sequence on a target system provisioned by other means than SAP LVM.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 14 of 156
When executing default LVM sequences enhanced custom hooks increase customization
capabilities for platform partners and customers. These hooks are either executed pre-, post,
or on-error of operations.
Platform Support and Integration Changed
Cluster awareness is an interesting point here. However, it does not mean that we now can
fully manage PowerHA clusters with LVM 2.0. Currently, this statement means detection and
monitoring of multi-node DBs like DB2 LUW DPF or Oracle RAC is supported. So
administrators are alerted when manipulating a DB-node. The same applies to shadow DBs,
which are mainly used for D/R purposes.
For programmatic use and integration into an enterprise monitoring and management system
extended outbound interfaces make LVM 2.0 an information provider to any of these 3rd
party
solutions. In addition, LVM can actively emit notifications about certain events or activities to
such solutions.
Changes Concerning Supported IBM Storage Solutions
With SAP LVM 2.0 the matrix of supported IBM storage solutions was extended. SAP LVM
2.0 supports all IBM storage solutions already supported in SAP LVM 1.0 (IBM XIV, IBM
SAN Volume Controller and IBM Storwize V7000) and introduces support for IBM System
Storage DS8000.
SAP LVM 2.0 improves the execution of SAP system clone, copy, and refresh operations. It
provides is more flexibility for specifying characteristics of storage volumes for SAP target
systems.
The IBM storage solutions now allow to
define names for AIX volume groups and logical volumes for the target systems
specify the storage pool in the storage system where the target storage volumes will be
allocated
create target SAP system on remote storage using a pre-configured storage mirroring
setup
The setup and configuration for SAP system clone, copy, refresh operations with IBM storage
solutions was simplified. No additional IBM specific scripts are required anymore. Support
for SMI-S communication with storage systems was added. Other communication methods,
used in SAP LVM 1.0 before, are now deprecated, but still supported.
Changes Concerning Supported IBM Virtualization Solutions
With SAP LVM 2.0 the integration with IBM Flex System Manager and IBM Systems
Director was improved. Server System Pools in IBM Systems Director VMControl are now
supported and may be used as a target for relocation or deployment of logical partitions. For
deployment of logical partitions, you may specify multiple network adapters and use IBM
Shared Storage Pools as image repository. You may capture logical partitions as image and
delete captured images directly from SAP LVM user interface.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 15 of 156
2 Overview and Planning
2.1 Architecture overview
This section provides an introduction where and how individual functions are implemented.
2.1.1 SAP NetWeaver Landscape Virtualization Management
2.1.1.1 Technical Implementation
SAP LVM is an Java application that runs on add-on to SAP NetWeaver Application Server
Java. The required base version for SAP AS Java is 7.30, 7.31 or 7.40. Recommended
minimum support packages are SAP NetWeaver Release 7.30 Support Package 08, SAP
NetWeaver release 7.31 support package 04 or SAP NetWeaver release 7.4 support package
04.
The SAP LVM software is available for any platform that supports those SAP NetWeaver
releases.
SAP LVM can manage SAP systems running with older releases too. Table 2 lists the
supported releases for the managed SAP systems in a SAP LVM scenario. SAP note 1783702
describes in details the restrictions that may apply to some of the older SAP releases.
Supported Managed SAP systems (See SAP Note 1783702)
NetWeaver 7.40 NetWeaver 7.30 NetWeaver 7.20 NetWeaver 7.11 NetWeaver 7.10 NetWeaver 7.00 Web AS 6.40 Web AS 6.20 Web AS 6.10 R/3 4.6D R/3 4.6C
Table 2 Supported SAP releases for the SAP LVM scenario
Figure 3 illustrates the communication between all the different components. SAP LVM
software (both the SAP NetWeaver AS Java and the SAP LVM Add-on) is installed on a
single system designated as “Management Server. SAP LVM communicates with the
managed nodes via the SAP Host Agent (a process running on each managed node) for
landscape-wide monitoring and execution of tasks on the systems. Additionally there are
direct interfaces (black boxes in the diagram) to the IBM PowerVM virtualization layer and
the IBM Storage Layer. The IBM solution described here uses of all those paths.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 16 of 156
Figure 3 SAP LVM Block Diagram (Source: SAP)
SAP LVM provides a set of APIs as interface. In order to support specific hardware,
virtualization, OS and DB layers, each SAP partner needs to supply a set of libraries that
translate the generic SAP LVM operations to platform specific commands and executes them
as integrated solution on the appropriate component. SAP Note 1783702 describes all the
supported platforms for SAP LVM 2.0. Not all platform providers have implemented specific
libraries for all of their platforms, or partly they support just a subset of the functionality.
Thus, the supported use case for the SAP LVM solution may vary by platform.
2.1.2 IBM Systems Management Components
In today’s complex and demanding environment, the IT infrastructure needs to react quickly
and adapt to the changing needs of the business. More and more of the IT infrastructure is
being virtualized and administrators must manage complex interactions between
virtualization, compute, storage, and network platforms. System administrators face
extraordinary demands as they try to plan, document, and roll out IT infrastructures, identify
capacity needs, get the most out of current assets, and contend with constant budget pressure.
IBM’s Systems Management Components free-up the IT administrators from repetitive, time-
consuming tasks allowing them to pursue higher value projects and accelerate innovation.
With more automation and integrated management across infrastructure elements, IT
administrators can focus a greater portion of their time on programs that drive innovation and
business advantage instead of daily routine tasks.
IBM Flex Systems Manager (FSM) and IBM Systems Director are platform management
tools. FSM builds upon IBM Systems Director code and its advanced plugins, but both were
enhanced with specific elements.
They have different specifications and support different types of hardware:
Flex Systems Manager (FSM) is a specialized tool for management of IBM Flex
Systems and IBM PureFlex Systems
IBM Systems Director is a general purpose systems management tool
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 17 of 156
2.1.2.1 IBM Systems Management Components for IBM PureFlex Systems
IBM Flex Systems Manager (IBM FSM) is a pre-installed appliance on a specialized compute
node for Flex and PureFlex systems. It is optimized for managing the Flex and PureFlex
systems. IBM FSM has a deep integration with hypervisors and system patterns for the
configuration of the environment on the Flex Chassis. The FSM Explorer user interface (UI)
of IBM FSM contains additional functionality and allows a graphical view to the components
in the Flex Chassis, the operating system provisioning engine and the configuration patterns
for Flex System components.
IBM FSM provides a single point of control for management of physical and virtual compute,
storage and networking resources from a single management console. IBM FSM enables fast
deployment operations of new partitions by simple and repeatable hardware configuration and
bare metal provisioning for server nodes, and five click creation and zoning of storage
volumes. IBM FSM allows the administrator to monitor the infrastructure from anywhere
using iOS, Android and Blackberry mobile devices. IBM FSM is designed to help to get the
most out of the IBM PureFlex System or IBM Flex System by automating repetitive tasks,
and providing visibility and control across compute, storage, network, and virtualization
functions within the data center.
IBM FSM management capabilities include:
Server management
IBM FSM auto-discovers hardware and manages inventory that supports up to 16
managed chassis, 224 compute nodes and 5,000 managed elements. IBM FSM
monitors hardware’s overall health status to proactively resolve issues. IBM FSM
detects and recovers potential problems in the PureFlex environment through an event
setup that triggers alerts and actions.
Virtualization management (VMControl Express)
o Create, edit, manage, and relocates virtual machines for rapid deployment
o Discover virtual machines, storage and network resources and visualize the
physical-to-virtual relationships
Advanced virtualization management (VMControl Standard and Enterprise):
o Automate the virtualized environment with system pools
Storage management (Storage Control)
o Storage provisioning for image creation, deployment, and cloning
o Shows relationships between storage and server resources
o Policy based storage placement and provisioning.
Fabric management (IBM Fabric Manager)
o Simplify and manage assignments of Ethernet MAC and Fibre Channel WWN
addresses using an enhanced UI
o Enables quick and easy deployment, configuration and recovery of compute
nodesI
o Monitors the health of compute nodes, and automatically without user
intervention replaces a failed compute node from a designated pool of spare
compute nodes
Network management (Network Control)
Network resource virtualization and graphical topology view of network resources and
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 18 of 156
connectivity end to end in a single tool.
Energy management (Active Energy Manager)
This feature is activated and monitors chassis by default. With simple real-time power
and thermal management across compute node, storage, networking and facility
providers it can set thresholds on hardware components to drive lower energy costs.
Security management
Provide administrative capabilities, such as setting up users within profile groups,
assigning security level of policies and security governance. FSM can view the access
state of managed resources in a web interface display.
The integration with SAP LVM leverages the virtualization management of
VMControl and Storage Control.
2.1.2.2 IBM Systems Management Components for IBM Power Systems
IBM Systems Director® provides similar functionality in an IBM Power Systems
environment compared to IBM FSM in an IBM PureFlex System landscape. In contrast to
IBM FSM (which is delivered as an appliance), IBM Systems Director must be manually
installed on a physical or virtual server.
IBM Systems Director is the platform management backbone, providing all the building
blocks for integrated services management. It provides the IT administrator with the features:
Unified management of physical and virtual resources for IBM servers, storage, and
networks.
Automated data center operations by implementing cloud-ready virtual infrastructures.
Figure 4 IBM Systems Director and Plug-Ins
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 19 of 156
Key value of IBM Systems Director is its ability to work in diverse IT environments, so it
reduces the number of required management tools and interfaces. IBM Systems Director can
manage rack, tower, and blade servers across all IBM brands. IBM Systems Director can
manage Flex Systems also, but only in case that no FSM is installed in the Flex Chassis.
Optional plug-ins can extend IBM Systems Director. The SAP LVM integration exploits:
IBM® Systems Director VMControl™ is a multi-platform virtualization management
solution. This solution – available as a plug-in for IBM Flex System Manager or for IBM
Systems Director – provides the following features that are also relevant in any SAP
landscape:
Create and manage virtual machines
Relocate virtual machines
Import, edit, create and delete virtual images
Deploy new virtual machines from virtual images
The VMControl plug-in is used by SAP LVM to perform the partition and AIX provisioning
steps along the overall processing sequence.
IBM® Systems Director Storage Control is another plug-in for IBM Flex System Manager
or for IBM Systems Director. It extends the management of systems to include the storage
layer. It is based on IBM Tivoli Storage Productivity and enables functions as:
Extends storage management of IBM Systems Director to cover most IBM storage
systems
Storage device discovery and coverage in integrated physical and logical topology views
Show relationships between storage and server resources
Ability to configure logical and physical configuration
Ability to view controller and volume status and to set notification alerts
Integration with IBM Systems Director VMControl storage provisioning for image
creation, deployment, and cloning
Storage Control is used in the end-to-end provisioning sequence to allow “bare-metal”
LPAR creation and AIX installation: During the provisioning process, Storage Control
takes care that the storage for the OS is provisioned and attached to the LPAR.
2.1.2.3 Storage Management (for Snapshots/FlashCopies)
IBM Tivoli® Storage FlashCopy® Manager. This software provides fast application-aware
backups and restores by exploiting snapshot technologies in IBM storage systems.
In the context of SAP landscape management, IBM Tivoli FlashCopy Manager is used to
create instantaneously consistent copies of online SAP systems. Instead of using these copies
as source for backups, SAP LVM uses the flashes as source for further processing system
replica by subsequent post-processing steps.
2.2 Combining the IBM and SAP Layers
2.2.1 SAP LVM Interfaces to Management Components
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 20 of 156
Figure 3 shows how the SAP LVM architecture is implemented for IBM POWER and IBM
storage platforms. The integration consists of a set of libraries, which transfer generic SAP
LVM commands into the specific syntax for the IBM environment. IBM provides these
libraries as part of the SAP LVM software stack.
Different communications paths exist between the managing server and the managed nodes:
Communication path via the SAP Host Agent
The SAP Host agent can execute OS commands on each managed SAP node. SAP
LVM communicates with this agent, which calls the vendor libraries in order to map
the LVM commands to an OS, DB, or storage specific language and syntax.
Communication path using the SAP LVM virtualization adapter
The managing system interfaces directly with the IBM Power HMC or with the
VMControl plug-in of IBM Flex System Manager/IBM Systems Director to control
PowerVM and OS layers.
Communication path using the SAP LVM storage adapter (IBM Storage Adapter)
For non-disruptive copies of running SAP instances, SAP LVM interacts with the
Tivoli Storage Flash Copy Manager, which interfaces to the storage system for the
cloning, and handles then handles all storage activities required on the managed nodes
also. This implementation guarantees a consistent replication of SAP instances by
synchronizing the required steps among the involved infrastructure layers (storage,
AIX, DB).
Figure 5 Software structure to integrate IBM IT layers
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 21 of 156
2.2.2 Example Workflow “SAP System Copy”
In this section, a walk-through through a sample “SAP System Copy” use case will
demonstrate how the integration works together. The workflow is composed of two phases
(See Figure 6):
System Cloning Phase
The system cloning phase begins by creating host name mappings on a name server
and is finished when the clones SAP systems is ready to be started in an isolated
network segment. These steps are the same for the system copy and system cloning
scenarios. At the end of this phase, a 1:1 clone of the source SAP system exists which
per default cannot be accessed from other system systems from outside that fenced
segment.
System Copy Phase
After the system-cloning phase is completed, SAP LVM triggers additional automated
tasks that make up the system copy phase. This includes renaming the SAP system
clone. The Post Copy Automation (PCA-) sequence adjusts the complete SAP
configuration so that finally a unique SAP system (SID) is established.
Figure 6 Generic SAP System Copy Workflow
We assumes the following landscape example for a detailed description of the workflow:
The source system is an SAP application server ABAP system
It is deployed in an AIX LPAR
The SAP system is configured with virtual IP addresses
The operating system, SAP system, and database are installed on separate AIX volume
groups
The physical disks for these volume groups are allocated on SAN storage systems.
The source system is configured in SAP NetWeaver Landscape Virtualization
Management
The administrator navigates to the “Provision view” in the SAP LVM console and selects the
source SAP system (All the detailed steps are shown later in chapter 5.2). After pressing the
Copy button, SAP LVM guides the administrator through dialogue steps defining the end-to-
end process. SAP LVM knows all the configuration settings of the registered source system;
however, the administrator has to specify configuration parameters for the target system.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 22 of 156
Reasonable values for some parameters are determined automatically and pre-set as default
during the configuration dialogue. Following parameters can be changed (this is a non-
exhaustive list):
Target host: SAP NetWeaver Landscape Virtualization Management provides
the option to choose an already existing AIX LPAR or to create a new AIX
partition as target for the system. (For the latter the virtualization manager
configured in SAP NetWeaver Landscape Virtualization Management needs to
be set-up for provisioning and configuring such an operating system partition).
Host names: The copied SAP system is setup with a new virtual IP address
(and IP label) for each of the SAP instances. SAP NetWeaver Landscape
Virtualization Management is able to create new addresses and update the
domain name server automatically, otherwise the addresses for the target SID
need to be pre-configured in DNS.
Storage volumes: SAP NetWeaver Landscape Virtualization Management
uses the registered storage adapter to gather the data about the volumes
attached to the source system. It collects this information at runtime directly
from the storage management system. The administrator can specify target
volume names and change mount points, if required.
Database consistency: SAP NetWeaver Landscape Virtualization
Management asks whether the SAP source system is to be stopped during the
cloning process or whether the system should stay up and running (for
example, in case of a production system). With the online mode, the
administrator decides whether SAP NetWeaver Landscape Virtualization
Management is responsible for database consistency. Otherwise, there is no
guarantee that the cloned database can be recovered and brought online. When
selecting database consistency, SAP NetWeaver Landscape Virtualization
Management switches the source database into online backup mode (Oracle) or
into write suspend I/O mode (DB2). This ensures a consistent database image
during the actual cloning step.
Target isolation: On the target system network, network fencing has to be
established to ensure that the cloned system can be started without interfering
with other systems in the data center. This is achieved by using the IPsec
feature of AIX. SAP NetWeaver Landscape Virtualization Management allows
the definition of permitted outgoing network connections. All other outbound
connections are blocked while incoming connections are permitted.
SAP copy parameters: SAP NetWeaver Landscape Virtualization
Management queries information about the copied SAP system, such as the
new SAP system identifier (SAP SID) and master password.
SAP copy users: SAP NetWeaver Landscape Virtualization Management
allows new users to be created and defined for the copied SAP system on a
central user repository.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 23 of 156
SAP post copy automation: SAP NetWeaver Landscape Virtualization
Management allows the selection of task lists and task list variants to be used
for post copy automation. This is possible for multiple clients in the SAP
system (such as basis tasks in client „000‟ and BDLS in client „100‟).
After all the configuration parameters have been defined, the administrator reviews the input
and then starts the SAP system copy workflow.
There is a narrated ScreenCam video available on TechDocs, in case you want get a real
impression about the look and feel of the SAP LVM administrator console.
2.2.3 Supported Use-Cases and involved components
The use-cases for
Landscape-wide overview and system monitoring incl. virtualization layers (physical
servers and LPARs)
Activation, deactivation, and relocation of SAP LVM hosts (LPARs)
Provision of new SAP LVM hosts (LPARs)
are included in the standard edition of SAP LVM and will leverage one of the virtualization
adapters during execution. These use-cases are not dependent on special storage, as long as
the storage satisfies the requirements of the virtual environment.
IBM Tivoli Storage FlashCopy Manager is not required for execution of these use-cases.
Enhanced functionalities (e.g. customized dashboards, generation of reports) will require the
enterprise edition.
The use-cases for
Attaching, detaching, relocating of SAP systems
Cloning, copying, refresh of SAP systems
have some dependencies on the storage infrastructure. SAP LVM manages the storage
volumes belonging to SAP systems during the execution of these operations.
The implementation of attach, detach, relocate operations of SAP systems depends on the
actual storage type.
The functionality for NAS and GPFS is included in the IBM storage library. For SAN-based
IBM storage, the IBM Storage Adapter is also required in addition to the IBM storage library.
Clone, copy, and refresh operations require SAN based IBM storage, IBM Storage Adapter,
and IBM storage and platform libraries.
Integration with other, non-IBM storage (like EMC or NetApp) is possible but not covered in
this paper.
Table 3 summarizes the different use case families in SAP LVM and illustrates which Edition
for SAP LVM is required, which SAP LVM adapters needs to be invoked, and which
boundary conditions need to be fulfilled for the execution.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 24 of 156
Use Case Landscape
Overview System monitoring
VM (LPAR) Activate Deactivate Relocate
VM (LPAR) Provision new LPARs
SAP System Attach Detach Relocate
SAP System Clone Copy Refresh
SAP LVM Adapter
Virtualization Adapter
Not required for NFS and GPFS, Storage Adapter for SAN based storage
Storage Adapter
IBM libraries Not required
IBM storage and platform libraries
IBM storage and platform libraries
SAP LVM Edition
Standard Edition
(advanced scenarios
require Enterprise
Edition)
Standard Edition
Standard Edition
Standard Edition for NFS and GPFS Enterprise Edition for SAN based storage
Enterprise Edition
Virtu
aliz
ation M
an
age
me
nt IBM Power
Systems with IBM Systems Director
Yes, requires VMControl Express Edition
Yes, requires VMControl Express Edition
Yes, requires VMControl Enterprise Edition and
Storage Control
Not required Not required
IBM PureSystems with IBM Flex System Manager
IBM Power Systems with HMC
Yes Yes No
Sto
rage M
an
age
ment IBM Tivoli
Storage FlashCopy Manager
Not required Not required Not required Not required Required
Supported Storage
Any Any Any
NAS (NFS) GPFS IBM XIV IBM SVC IBM V7000 IBM DS8000
IBM XIV IBM SVC IBM V7000 IBM DS8000
Table 3 Overview of the Scenarios
2.2.4 Custom Cloning Capability and custom hooks/ operations
SAP LVM 2.0 introduced a new “custom cloning” feature. This feature allows creating a
system clone based on existing cloning procedures and integrating those into SAP LVM. One
can replace the pre-defined sub-steps in the “standard” workflow of the pre-clone, clone, and
post-clone steps by custom defined web services or scripts executed on the source and target
systems.
The web services/scripts need to be registered as a “Provider Implementation Definition” and
references the script/web services execution triggered by the SAP Host Agent on the source or
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 25 of 156
target system during the workflow step. The provider implementation definition specifies
“what” is executed on the host or on the SAP instance.
When the provider implementation definitions is done, the new custom provisioning process
then maps each provider implementations instead of the “standard” step of the built-in cloning
process. It specifies “when” and “under which circumstances” the provider implementation
definition is executed.
Provider implementation definitions can be used in the context of custom hooks or custom
operations too:
Custom hooks allow to add additional pre-, post-, or error processing steps to the
standard workflows defined in SAP LVM.
Custom operations allow triggering of user-defined activities directly from the SAP
LVM GUI or as part of mass operations. For example, the SAP Basis administrator
could run certain custom-defined scripts with privileged permissions on the hosts
without granting him “root” access in general.
2.3 Planning Project Resources and Timing
Table 4 summarizes the activities in order to implement a SAP LVM managed POWER
landscape comparable to our proof of concept setup described in the following section 2.4.
The estimated duration per phase provides a rough guideline only. In detail, the times are
dependent on the complexity of the SAP landscape (IT and SAP systems) and its operation as
well as on the availability of all the required skills for all involved disciplines.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 26 of 156
Project Phase Included Tasks / Accomplishments Estimated time
Planning workshop, Verification of landscape prerequisites in client environment
Network and disk design requirements, SW prerequisites
Validate that customer has obtained the necessary SAP and AIX software, validate that dedicated IBM hardware infrastructure is available
SAP Landscape needs to support virtual host names, and a proper range of IP addresses needs to be available
2-3 Day workshop with customer
Plan and Prepare setup of storage infrastructure
Storage LUNs
Network, SAN zoning
Eventually map SVC to existing storage
Plan and create filesystems
3-4 days working with customer
Installation and configuration steps for the management environment (IBM Systems Director, SAP NetWeaver Landscape Virtualization Management)
Define LPARs on the managed systems, and install AIX OS on the LPARs
Set-up virtualization environment o Install and configure IBM Systems
Director including VMControl and Storage Control
o Integrate NIM server into System Director VMControl environment
o Set-up Storage Control for all storage entities (SAN, Storage System)
Install and configure storage management software (FlashCopy Manager, Storage CLI) on the management system
Plan and configure shared infrastructure services
o Install and configure LDAP (if not to be integrated with existing LDAP environment
o Plan for DNS
Install SAP LVM system
Two weeks working with customer
Installation and configuration steps of managed environment incl. basic testing Installation from scratch for one SAP System
Define AIX “golden image” content
Install first AIX LPAR with golden image
Install and configure storage management software (FlashCopy Manager, SVC CLI)
Install first “source” SAP system
LVM base configuration (Virtualization Manager, Storage Manager configuration, discover hosts, SAP systems, users)
Capture image in Systems Director VMControl for further deployment
Additional LVM use case specific configuration (storage, post processing, steps per use case) Note: configuration of PCA post processing requires skilled SAP specialist with both SAP Basis and Application/Business Process knowledge)
Two weeks with customer
Installation and configuration steps of managed environment incl. basic testing Migrate/ Customize one already existing SAP system
Adapt existing SAP system to virtualised environment
Eventually Upgrade SAP Host Agent
Install and configure agents and storage management
Install and configure storage management software (FlashCopy Manager, SVC CLI)
Discovery and setup of LPARs and SAP
At least 2 weeks with customer
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 27 of 156
Project Phase Included Tasks / Accomplishments Estimated time
systems in IBM Systems Director and SAP LVM
Additional LVM use case specific configuration (storage, post processing, steps per use case) Note: configuration of PCA post processing requires skilled SAP specialist with both SAP Basis and Application/Business Process knowledge)
Verification Steps using the customer’s SAP system
Perform functional verification
Execute and document each use case o Start / Stop / Relocate of LPARs and SAP
systems o Clone SAP Systems (target LPAR available) o Copy SAP Systems (target LPAR available) o Refresh SAP system o Clone SAP Systems (create target LPAR
from scratch) o Copy SAP Systems (create target LPAR
from scratch)
At least 2 weeks working with customer
Acceptance Tests
Support customer in LVM acceptance testing from basis or application level
At least 1 week
Document Solution or Handover
at least 2 days for handover, documentation at least 1 week
Table 4 Project Phases and estimated time
2.4 Tested configuration
The installation hints in this document are based on a proof of concept installation in the
ISICC demo landscape using IBM PureFlex Systems with Power nodes and IBM SAN
Volume Controller Storage as well as IBM XIV Storage Systems.
Figure 7 illustrates the environment in the proof of concept.
In the management environment:
Virtualization Manager is IBM Flex System Manager 1.3.0. including VMControl
2.4.3.1 and Storage Management 6.3.3.1.
SAP NetWeaver Landscape Virtualization Management software V2.0 SP02 is
deployed on top of a SAP NetWeaver Java AS 7.31 engine in a second partition.
Additional infrastructure servers like DNS, LDAP server, and NIM server are
available and configured.
In the managed environment
at least one SAP source system is existing
all hosts (AIX LPARs) for both the SAP source system(s) and SAP target system(s)
are registered within SAP LVM. The hosts are attached via virtual fibre channel to an
IBM XIV Storage system.
On each host the software components
SAP Host Agent package
IBM Tivoli Storage FlashCopy Manager
IBM XIV XCLI
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 28 of 156
are installed and configured.
Figure 7 Architecture overview diagram
Table 5 reflects the detailed software stack tested in the scenario:
Software Version
SAP Landscape Virtualization Management
Software 2.0 SP02
IBM Flex System Manager / VMControl /
Storage Management 1.3.0 / 2.4.3.1 / 6.3.3.1
IBM Script Package
SAP Host Agent 7.20 Patchlevel 164
SAPINST (Rename)1
SAP Software Provisioning Manager
(SWPM) V1.0 SP4_2
AIX 7.1 TL2 SP2 7100-02-02-1316
IBM Tivoli Storage FlashCopy Manager 3.2.0.1
IBM XIV 10.2.4.a
XIVGUI XCLI 3.1 build 18
DB2 UDB DB2 v10.1.0.2 (Fixpak 2)
Table 5 Software Stack
2.5 SAP LVM Environment
2.5.1 Storage Virtualization
A SAP LVM infrastructure requires a centralized storage solution. The storage infrastructure
in the ISICC sample environment is based on IBM System Storage™ solutions using a
Storage Area Network (SAN).
1 Starting with SAP NetWeaver Landscape Virtualization Management 2.0 Support Package 03 the delivery of
software component "SAP SYSTEM RENAME 1.0" is part of the "SOFTWARE PROVISIONING MANAGER
1.0" package.
IBM Flex System Manager
NIM Server AIX
Management Environment
IBM XIV SVC / Storwize
SAP LVM AIX
LDAP IBM Tivoli Directory Server
Source Target
VIOS
vFC (NPIV)
VM Control, Storage Manager
DNS
SAP NetWeaver Landscape Virtualization Management SAP NetWeaver Java AS
SAP Source AIX SAP Source
AIX SAP Source AIX SAP Source
AIX SAP Source AIX SAP Source
AIX SAP Component - SAP NetWeaver
Managed Environment
VIOS
vFC (NPIV)
SAP Target AIX
• Storage connected via vFC (NPIV)
• Compliant Storage Layout
IBM Tivoli FlashCopy Manager Storage Manager native Interface e.g. IBM XIV XCLI
IBM Tivoli FlashCopy Manager Storage Manager native Interface e.g. IBM XIV XCL
IBM Tivoli FlashCopy Manager Storage Manager native Interface e.g. IBM XIV XCLI
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 29 of 156
IBM System Storage virtualization products achieve the abstraction from physical volumes of
data storage to a logical level. It addresses the increasing complexity of managing storage,
while reducing the associated costs. Its main purpose is the full exploitation of the benefits
promised by a SAN. Virtualization enables data sharing, ensuring higher availability,
providing disaster tolerance, improving performance, allowing for consolidation of resources,
providing policy-based automation, and much more besides, which do not automatically result
from the implementation of today’s SAN hardware components. Storage virtualization is
possible on several levels of the storage network components, meaning that it is not limited to
the disk subsystem. Virtualization separates the representation of storage to the operating
system and its users, from the actual physical components.
Storage virtualization accumulates the storage into storage pools, which are independent of
the actual layout of the storage (that is, the overall file system structure). Because of this
independence, new disk systems can be added to a storage network, and data migrated to
them, without causing disruption to applications. Since the storage is no longer controlled by
individual servers, it can be used by any server as needed. In addition, it can allow capacity to
be added or removed on demand without affecting the application servers. Storage
virtualization will simplify storage management, which has been an escalating expense in the
traditional SAN environment.
IBM SAN Volume Controller (SVC), IBM Storwize V7000, IBM System Storage DS8000, or
IBM XIV System Storage is not mandatory for all the use-cases in an adaptive computing
environment, but they help in the overall management of the storage landscape. For the SAP
System Copy use-cases however the Storage Adapter and Storage Library require one of those
systems.
2.5.1.1 SAN Volume Controller System Storage
The IBM (SVC) is an in-band, block-based virtualization product that minimizes the
dependency on unique hardware and software, decoupling the storage functions expected in a
SAN environment from the storage subsystems and managing storage resources. SVC
combines software and hardware into a comprehensive, modular appliance that uses
symmetric virtualization.
Symmetric virtualization is achieved by creating a pool of managed disks (MDisks) from the
attached backend storage systems. Those storage systems are then mapped to a set of volumes
for use by attached host systems. System administrators can view and access a common pool
of storage on the storage area network (SAN). This functionality helps administrators to use
storage resources more efficiently and provides a common base for advanced functions.
Key feature exploited for the SAP System Copy scenarios is the ability to create and manage
FlashCopies of existing (SAP application) volumes. IBM FlashCopy is a part of the SVC
Copy Services and provides a point-in-time copy function. This includes thin-provisioned
FlashCopy to make multiple targets affordable.
2.5.1.2 IBM Storwize V7000 Storage System
The IBM Storwize V7000 is a storage server with internal disks, providing the same storage
virtualization functions and features like SAN Volume Controller.
2.5.1.3 IBM System Storage DS8000
The IBM System Storage DS8000 is a high-performance, high-capacity, secure storage
system designed to deliver the highest levels of performance, flexibility, scalability, resiliency
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 30 of 156
and total overall value for the most demanding, heterogeneous storage environments. SAP
LVM 2.0 supports this IBM storage solution since service pack 3.
2.5.1.4 IBM XIV System Storage
The IBM® XIV® Storage System is a high-end disk storage series designed to address
storage challenges across the application spectrum, including for virtualization, email,
database, and analytics and data protection solutions. The IBM XIV Storage System is a fully
virtualized system designed to eliminate the need for performance tuning and numerous other
storage management activities. IBM XIV provides consistent, hotspot-free enterprise
performance and exceptional ease of use. As virtualized storage that meshes tightly with
hypervisors, XIV offers optimal agility for cloud and virtualized environments.
Key feature exploited for the SAP System Copy scenarios is the ability to create and manage
snapshots of existing (SAP application) volumes. The IBM XIV Storage System has taken
this concept one step further, offering a completely innovative approach to snapshot creation
and management. The XIV system offers clear advantages, including the following:
Unlimited number of snapshots in the system.
Snapshot creation in virtually zero time, regardless of the size of replicated volumes.
Unaffected performance levels in a system that supports snapshots, regardless of the
number of snapshots currently defined in the system.
2.5.1.5 General Parallel File System (GPFS)
Exploiting GPFS is the easiest and most convenient way to implement the „classic“ adaptive
scenarios for relocating SAP instances between different hosts.
IBM's General Parallel File System (GPFS) provides file system services to parallel and serial
applications. GPFS allows parallel applications simultaneous access to the same files, or
different files, from any node, which has the GPFS file system mounted, while managing a
high level of control over all file system operations. GPFS is particularly appropriate in an
environment where the aggregate peak need for data bandwidth exceeds the capability of a
distributed file system server.
GPFS allows users shared file access within a single GPFS cluster and across multiple GPFS
clusters. A GPFS cluster consists of:
AIX nodes, a node may be:
o An individual operating system image on a single computer within a cluster
o A system partition containing an operating system
Network shared disks (NSDs) created and maintained by the NSD component of
GPFS
A shared network for GPFS communications allowing a single network view of the
configuration (a single network is used for GPFS communication, including the NSD
communication)
All disks utilized by GPFS must first be given a globally accessible NSD name. On AIX
server running GPFS, an existing virtual shared disk or physical disk may be given an NSD
name.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 31 of 156
2.5.2 SAP NetWeaver Landscape Virtualization Management
All the base considerations are described in the SAP Notes Note 1783702 - SAP NetWeaver
Landscape Virtualization Management 2.0 and Note 1833980 - Integration of IBM
Components into SAP LVM 2.0.
SAP LVM distinguishes in its configuration between:
Instances
An instance is a part of an SAP System (database instance, central services instance,
application server instance). The SAP LVM use-cases typically are executed on instances
or whole SAP systems.
Hosts
A host describes an operating system installed on a physical or virtual server. SAP
instances may be relocated between hosts, or a host may be the target for a new SAP
system created via SAP system copy or clone. A new host may be deployed (via SAP
LVM and VMControl) so that it can be used as a target for a relocate operation or a SAP
system copy, clone, or refresh. SAP LVM monitors hosts through SAP Host Agents
running on the operating systems.
2.5.2.1 SAP LVM Virtualization Management
SAP LVM interacts with the virtualization managers to
extend host monitoring data available via SAP Host Agents with information about
virtualization layers (physical hosts and virtual machines)
provision new virtual machines
perform load balancing by starting, stopping and relocating virtual machines
SAP LVM includes two virtualization managers (adapters) for IBM solutions
IBM Flex Systems Manager/ VMControl Adapter
IBM Hardware Management Console for IBM Power
Both of them enable
Monitoring of IBM Power physical servers and logical partitions (LPARs)
Management of LPARs (Power on / Power off / Shutdown OS)
Relocation of LPARs using Live Partition Mobility
Additionally the “IBM Flex System Manager/VMControl” adapter provides provisioning
capabilities for both AIX OS and Storage.
See SAP 1833980 - Integration of IBM Components into SAP LVM 2.0 for detailed
description of supported configurations and features.
2.5.2.2 SAP LVM Storage Management
SAP LVM interacts with the storage managers (adapters) to
clone, copy, or refresh SAP systems
prepare, un-prepare, and relocate of SAP instances and systems
SAP LVM also includes an “IBM Storage Adapter” which supports invocation of
FlashCopies/SnapShots within the storage system to provide a fast, efficient, and application-
aware cloning of the SAP source system storage volumes.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 32 of 156
These functionalities are used in SAP system clone, copy, and refresh operations. In addition,
this adapter provides the required functionality to attach and detach storage volumes and
filesystems to and from the LPARs during SAP system prepare, unprepared, and relocate
activities.
“IBM Storage Adapter” in SAP LVM 2.0 supports following storage system types:
IBM XIV Storage System
IBM Storwize V7000 storage system
IBM SAN Volume Controller (incl. any supported backend storage system)^
IBM System Storage DS8000
See SAP Note 1833980 - Integration of IBM Components into SAP LVM 2.0 for detailed
description of supported configurations and features.
In this proof of concept we used the following configuration for SAP System Copy use cases:
IBM SAN Volume Controller 7.1.0.4
IBM XIV Storage System, Version 10.2 with XIV CLI 3.0
2.5.2.2.1 SAN Zoning Requirements
As a prerequisite for all the SAP System Copy use cases (SAP System clone, copy, refresh)
SAN based storage is required for usage with the “IBM Storage Adapter”.
SAN zoning on the target LPAR must be pre-configured. This can be easily fulfilled if the
volume(s) for the AIX operating system (rootvg) of the target LPAR is located on the same
storage like the SAP systems in scope of the System Copy.
Then the existing access to the rootvg on the target LPAR provides SAN access for the SAP
application volumes too. In this case, the “host” and “adapter id” definitions in the storage
system exist already, and the LPAR has an already zoned access in the SAN to the storage
system. The SAP application volumes are assigned to the LPAR by “LUN masking”
commands from the SAP LVM storage adapter.
The storage system should be zoned according to “single initiator zones”. Each zone set
should contain one (active) virtual fibre channel host port only, but can include multiple
storage adapter ports. (e.g. between three and six adapter ports for a storage system).
Although not mandatory it is best practise to define the name of the host in the storage system
identical to the name of the LPAR or its hostname. If virtual fibre channel adapters are used,
then all the WWPNs need to be included in port definition on the storage system, as well as in
the SAN zoning definition. This includes all “active” WWPNs via the VIOS pair, as well as
the secondary WWPNs used during a Live Partition Mobility relocation event.
2.5.3 Considerations for the SAP LVM hosts
Several storage attachment considerations need to be obeyed, and additional software
components need to be installed on the LPARs used as SAP LVM hosts. All these are
described briefly in this section.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 33 of 156
2.5.3.1 VIOS / PowerVM and SAN attachment
For all the SAP System clone, copy, and refresh scenarios that invoke storage-based cloning,
all the storage volumes for both the source and the target systems need to be attached either
via virtual Fibre Channel (vFC) adapters with NPIV or via physical fibre channel adapters
attached directly to these systems.
In the proof of concept described in this paper we used NPIV attachment via two redundant
VIO servers. Each client LPAR is defined with two virtual fibre channel adapters. On each
VIO server a virtual fibre channel server adapters is created and assigned to the corresponding
client LPAR. Two pairs of WWPNs are created (WWPN1, WWPN2, plus the additional
WWPNs for the LPM scenario). We use single initiator zones. Each zone contains one
(active) virtual fibre channel host port and between three and six storage system ports. The
maximum number of paths that can be zoned to an AIX LPAR is 32. However there is no
benefit in having more than 24 paths.
Figure 8 illustrates the redundant attachment of LPARs with IBM XIV storage system via two
redundant VIO Servers. Each of the VIO servers has one or more physical dual-port fibre
channel adapters assigned to the partition. The two ports of the fibre channel adapter card are
connected to two different SAN fabrics. Also, the storage system is connected to both fabrics.
If an LPAR has only one fibre channel adapter, then zone it to three IBM XIV modules. Zone
it to six modules only in case the LPAR has very high throughput requirements.
Figure 8 Redundant virtual fibre channel attachment
2.5.3.2 AIX operating system specifics
The AIX operating system on the managed systems needs to fulfil certain criteria:
IPsec
IPsec is required for network fencing on target hosts for SAP system clone, copy or
refresh scenarios and needs to be activated in the OS.
Domain Name System
For the sake of simplicity, all the virtual hostnames of the SAP instances and systems
LPAR
IBM XIV Storage
connected via vFC
(NPIV)
vFC Client
WWPN1
vFC Client
WWPN2
vFC Client
WWPN3
vFC Client
WWPN4
VIOS #1 VIOS #2
vFC ServervFC Server vFC ServervFC Server
FC adapter
port 1
FC adapter
port 0
FC adapter
port 1
FC adapter
port 0
FC adapter
port 1
FC adapter
port 0
FC adapter
port 1
FC adapter
port 0
Fabric #1
Fabric #2
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 34 of 156
should be provided by DNS. One may also configure DNS update functionality in
SAP LVM. Local hostname resolution using “/etc/hosts” must be configured as
alternative path.
Central User Management is recommended
In the test environment, LDAP is used for that purpose. The LDAP filesets need to be
installed, and the OS needs to be configured as LDAP client.
Services entries
We used LDAP as a central repository for the SAP related service entries too.
Otherwise an /etc/services file need to be prepared and distributed to all hosts.
2.5.3.3 IBM Tivoli Storage FlashCopy Manager
IBM Tivoli Storage FlashCopy Manager software provides fast application-aware backups
and restores, so leveraging advanced snapshot technologies in IBM storage systems. The
software provides following features in general:
Near-instant application-aware snapshot backups, with minimal performance impact
for IBM DB2, Oracle, SAP, Microsoft SQL Server, and Exchange
Improvement in application availability and service levels through high-performance,
near-instant restore capabilities that reduce downtime
Integration with IBM Storwize V7000, IBM System Storage DS8000, IBM System
Storage SAN Volume Controller and IBM XIV Storage System on AIX, Solaris,
Linux, and Microsoft Windows
Fulfilment of advanced data protection and data reduction needs with optional
integration with IBM Tivoli Storage Manager
In the context of SAP landscape management, IBM Tivoli FlashCopy Manager is utilized to
create instantaneously a consistent copy of an online SAP system.
2.5.3.4 Storage Management Interfaces
2.5.3.4.1 XCLI
The IBM XIV Storage System command-line interface (XCLI) provides a mechanism for
issuing commands to manage and maintain the XIV storage systems. XCLI commands are
entered on the XCLI client either by a user or by another application. The XCLI client and the
XIV system communicate using Secure Sockets Layer (SSL) over TCP/IP. . Authentication is
based on username/password.
2.5.3.4.1 SAN Volume Controller Command Line Interface (SVC CLI)
The SAN Volume Controller command-line interface (SVC CLI) provides a mechanism for
issuing commands to manage and maintain the IBM SVC storage system. Communication is
achieved by SSH using SSL over TCP/IP: The SAN Volume Controller cluster acts as the
SSH server in this relationship. The SSH client provides a secure environment to connect to
the remote machine based on public and private keys for authentication.
2.5.3.4.2 SMI-S
The IBM SVC’s CIMOM interface is supplied with the SVC Master Console and is
automatically installed as part of the SVC Master Console installation. For the IBM DS8000,
the CIMOM interfaces is supplied with the HMC. The CIMOM interface translates an SMI-S
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 35 of 156
(Storage Management Initiative Specification) command into a proprietary command that the
device understands and then convert the proprietary response back into the SMI-S based
response.
2.5.3.5 SAP Host Agent
The SAP Host Agent package contains all the required elements for centrally monitoring of
any hosts. It is used by SAP LVM to monitor and manage SAP instances and hosts. The SAP
Host Agent package needs to be installed and configured on each host containing an SAP
component managed by the SAP NetWeaver Landscape Virtualization Management software.
The SAP Host Agent package is automatically installed as part of the “SAPinst” installation
procedure for any components of SAP NetWeaver 7.0 EHP2 and higher. However it may
need to be updated to a higher patch level to exploit all SAP LVM management features. It is
also required to install SAP Host Agent manually on operating systems without any SAP
systems.
In our case these are operating systems that act as targets in SAP system clone, copy, refresh,
or relocate operations.
2.5.4 Considerations for the SAP LVM Instances and Systems
All SAP systems need to be installed compliant to a SAP LVM environment. This includes
different considerations for user management, network, and storage. Some best practises are
discussed in the following.
2.5.4.1 User Management
Ensure that the administrative users are available with same UID and GID on all the LPARs
involved in the scenario. A central user management solution provides this requirement and
we used LDAP for this purpose.
2.5.4.2 Network Considerations
The following guidelines for the network setup for the SAP system to be managed by SAP
LVM needs to be considered:
Provide a dedicated virtual hostname for each SAP instance
The various instances of an SAP system (SAP database, SAP Central Services, SAP
instances) shouldn’t share an IP address. Each instance should own a dedicated IP
address.
Provide a dedicated IP address for each virtual host name
Do not use multiple virtual hostnames (IP alias names) referring to one and the same
IP address.
Ensure that all the virtual host names can be resolved on the SAP LVM system and on
all hosts
Ensure that the reverse lookup on the SAP LVM server and all other hosts provides a
one-to-one mapping
2.5.4.3 Storage Layout
Separate all storage volumes for the different instances
During the prepare, unprepared, and relocate activities is must be possible to handle
the storage independent of the instances:
o Create different AIX volume group for the filesystems related to the SAP
Central Instance and the Database
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 36 of 156
o If a Solution Manager Diagnostics agent exists for the SAP System, place its
filesystems to the Central Instance.
o If there is a dedicated Solution Manager diagnostics agent for the Database,
then its filesystems need to be part of the volume group of the database
o If you intend to use FlashCopy backup for the production database also, then
split the database content into three VGs: database data (tablespaces), database
online logs, and other database files (instance directory, executables, archive
logs etc …)
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 37 of 156
3 Installation
The following table provides an overview on the operational model. In the test environment,
both the management environment and the managed environment are hosted on IBM AIX
partitions. Most of the deployment units for the managed environments were directly added to
the AIX operating system image. They were part of the NIM deployment process for a new
SAP LVM host and automatically available on new systems.
Management Environment
Node Deployment Unit
Management LPAR SAP LVM
AIX
LDAP Client (IBM Tivoli Directory Server Client)
SAP NetWeaver 7.3 Java System / DB2 UDB
SAP NetWeaver Landscape Virtualization Management 2.0 SP02
IBM Tivoli Storage FlashCopy Manager [acsd] (including GSKit 8 package)
XIVGUI XCLI
SAP Host Agent package and SAP Adaptive Computing Extensions (SAPACEXT)
Management LPAR IBM Flex System Manager
AIX
LDAP Client (IBM Tivoli Directory Server Client) optional
IBM Flex System Managers incl. VMControl plug-in
IBM Flex System Manager Storage Management (including embedded Storage Control plug-in)
NIM Server
AIX, setup as NIM Master
LDAP Client (IBM Tivoli Directory Server Client) optional
IBM Flex System Manager VMControl subagent for NIM
LDAP Server AIX
IBM Tivoli Directory Server
DNS Server AIX bind
Managed Environment
Node Deployment Unit Part of “AIX
image template”
on all SAP LVM hosts
AIX Yes
LDAP Client (IBM Tivoli Directory Server Client) Yes
SAP Host Agent package and SAP Adaptive Computing Extensions
Yes
IBM Tivoli Storage FlashCopy Manager [acsgen] (including GSKit 8 package)
Yes
XIVGUI XCLI (for IBM XIV Storage Systems)
Yes
SAP System Rename (required for SAP System Copy on target host)
No (accessed
from a central NFS share)
SAP Software Provisioning Manager (required for automated provisioning of SAP Application Server)
on one SAP LVM host
SAP ECC 6.0 source system DB2 UDB
SAP PostCopyAutomation tool (required for SAP System Copy and Refresh)
Table 6 Operational Model
This chapter describes the installation of the components for both the management and
managed environments for an AIX/ POWER environment. Not all the installation steps will
be described in all details; it is assumed that the reader has sufficient background knowledge
to perform these steps.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 38 of 156
In principle the SAP NetWeaver Landscape Virtualization Management server could be
deployed on another platform, e.g. Linux/Intel or Microsoft Windows/Intel. Then the
deployment unit “IBM Tivoli Storage FlashCopy Manager [acsd]” would need to be moved to
another AIX server (e.g. to the NIM server). XIVGUI XCLI, SAP Host Agent package, and
SAPACEXT libraries then would need to be installed on this server too.
3.1 Naming Conventions
In a SAP LVM environment SAP systems can potentially run on any SAP LVM enabled
server. For example, one could install all SAP system relevant data (database, binaries, and
working directories) on a distributed file system. In this case, we recommended putting this
data under a common root directory on that distributed file system.
A careful planning is required to avoid the collision of various names like IP names, user
names, user ids and group ids, directory structures, etc.
One method is to include the SAP System ID (SID), which typically is unique in a customer
environment, in the required SAP system IP names and root directories of the SAP system
data directories.
In addition, if you want to run multiple SAP systems or instances on a single compute node,
you have to plan for unique SAP instance numbers as well, as you cannot run two SAP
instances with the same instance number (but different SIDs) within the same OS image.
3.1.1 Sample Instance IP Names for SID C01
The following is one possible naming convention to build SAP systems IP names. There are
no specific requirements for IP names in the SAP LVM infrastructure, however choosing and
sticking to one convention will help to simplify the administration.
c01dbs IP name for database server
c01scs IP name for SAP central services instance
c01pas IP name for SAP primary application server instance
c01as1 IP name for additional application server instance
3.1.2 Sample directory structure for SID C01 on a Distributed File System
Each SAP system should have a common high level subdirectory name to provide a standard
entry point to all data that belongs to the same system. For example if the distributed file
system is mounted at /sapfs, then all data belonging to SAP system C01 could be allocated
under the subdirectory /sapfs/sapC01. The “root” directory for the SAP system C01 would
then be /sapfs/sapC01. In that root directory, all instance relevant subdirectories are stored.
The mappings between the physical locations on the storage subsystem and the directory
locations as they are expected by the SAP kernel have to be defined in SAP LVM. The
controller will then create the required mapping between the new data location and the
required standard path before starting an instance on a server. Table 7 shows a sample
directory structure for SID C01:
Directory Content Path in standard
installation
<root>/home/c01adm Home directory for SAP admin user /home/c01adm
<root>/db2 <root>/oracle
Database server, client, instance /db2 /oracle
<root>/sapmnt/C01 Directory for executables, profiles, shared files /sapmnt/C01
<root>/usr/sap/c01scs Instance directories for central services instance /usr/sap/C01
<root>/usr/sap/c01as1 Instance directories for application server 1 /usr/sap/C01
/sapfs/saptrans01 Common transport directory /usr/sap/trans
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 39 of 156
Table 7 Sample directory structure for a distributed file system implementation
The screenshot in Figure 9 shows the SAP instance configuration screen in SAP LVM, which
illustrates the mapping between physical layout and the required layout for the SAP database
instance EC1.
The storage type is Distributed File System (DFS) with a subtype of “gpfs", which refers to a
General Parallel File System installation.
The entries with mount option dirmount define the required mapping between the physical
location and the mount point that is expected by the SAP kernel.
Figure 9 System configuration in SAP LVM with distributed file system layout
3.1.3 Sample directory structure using Network Attached Storage (NAS)
The same concept as in the previous section can be used to layout the physical data locations
on a NAS device. A typical organization unit with NAS is called a volume where you store
directories and files. Those resources then have to be exported to the clients that need to
access them. The root directory for SAP system C01 could for example look like
nas_server:/vol/sapvol1/sapC01, where nas_server is the IP name of the NAS device. Again
the mappings between physical locations and expected locations have to be defined in the
SAP LVM Controller. The controller will then create the required mappings (in this case
using client NFS mount instead of creating symbolic links) before starting a instance on a
server.
3.2 Install Storage Infrastructure
3.2.1 Exploit GPFS as “adaptive storage”
This section provides a brief summary of setup tasks to get started with a GPFS cluster using
the ISICC SAP LVM landscape implementation as an example.
Using GPFS it is possible to carry out the use cases of the standard edition of LVM including
start, stop, and relocate of SAP systems. The use-cases for System Clone, Copy, and Refresh
are currently not supported with GPFS.
We do not cover in detail typical GPFS administration tasks like GPFS installation and
administration, the creation and management of storage pools, definition of file placement
policies, etc. For those tasks you need to refer to the standard product documentation and
available Redbooks. Especially to configure data availability and disaster recovery scenario,
you should carefully review the GPFS Base documentation (e.g. the “GPFS Concepts,
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 40 of 156
Planning, and Installation Guide” and the “GPFS Advanced Administration Guide”).
Additional information can be found in the GPFS Wiki
3.2.1.1 Planning for GPFS
3.2.1.1.1 Quorum Nodes
The first task is to decide on a node quorum strategy. As described in the “GPFS Concepts,
Planning, and Installation Guide”, GPFS enforces a node quorum rule to prevent multiple
nodes from assuming the role of the file system manager in the event of a network
communication problem in the cluster. A majority of quorum nodes must remain active in
order for the cluster to sustain normal file system usage (multi-node quorum).
The ISICC implementation uses a new node quorum mechanism that was introduced with
GPFS V2.3 – node quorum with tiebreaker disks, which allows you to run with as little as one
quorum node available as long as you have access to a majority of the quorum disks. Please
refer to Chapter 2 in the “GPFS Concepts, Planning, and Installation Guide” for a detailed
description of the quorum concept and guidelines to select quorum nodes. In the ISICC setup,
we use two quorum nodes and one tiebreaker disk.
3.2.1.2 Install and Setup GPFS
3.2.1.2.1 Install GPFS Software
You need to install the following filesets:
gpfs.base
gpfs.msg.en_US
This is best accomplished by including these filesets in a NIM installp bundle.
3.2.1.2.2 Create GPFS Cluster
The mmcrcluster is used to create a GPFS cluster. Our setup uses two quorum nodes: siccps10
and siccps12. The same nodes are also used as primary (option –p) and secondary (option –s)
configuration server to store the GPFS cluster configuration data. For cluster communication
we use ssh and scp instead of rsh and rcp, which has to be specified with the –r and –R
options. The –A option specifies that GPFS daemons are to be automatically started when
nodes come up. The default is not to start daemons automatically. Our cluster is named
siccfs.isicc (-C option).
root> mmcrcluster –N siccps12:manager-quorum,siccps10:manager-quorum –p siccps12 –s siccps10 –
r /usr/bin/ssh –R /usr/bin/scp –C siccfs.isicc –A
After a successful creation of the cluster, you can verify your setup using the mmlscluster
command.
root> mmlscluster
GPFS cluster information
========================
GPFS cluster name: siccfs.isicc
GPFS cluster id: 691765743288319456
GPFS UID domain: siccfs.isicc
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
GPFS cluster configuration servers:
-----------------------------------
Primary server: siccps12
Secondary server: siccps10
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 41 of 156
Node Daemon node name IP address Admin node name Designation
----------------------------------------------------------------------------------------------
1 siccps12 9.153.165.71 siccps12 quorum-manager
2 siccps10 9.153.165.69 siccps10 quorum-manager
Then you need to startup GPFS on all nodes and you should check the state of the cluster:
root> mmstartup -a
root> mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
------------------------------------------------------------------------------------
1 siccps12 2 2 8 active quorum node
2 siccps10 2 2 8 active quorum node
3.2.1.2.3 Define Network Shared Disk (NSD) Devices
GPFS uses Network Shared Disk (NSD) devices to provide access to the disks. A NSD device
can either be physically attached (using a SAN infrastructure) to a node or it can be accessed
using a virtual connection (through the network) using a primary and, if specified, backup
NSD server. GPFS determines if a node has physical or virtual connectivity to an underlying
NSD through a sequence of commands invoked from the GPFS daemon. This determination
is called disk discovery and occurs at both initial GPFS startup as well as whenever a file
system is mounted.
In the ISICC proof of concept landscape, we used the following configuration file to create
the NSD devices:
/var/mmfs/conf/nsd.disks:
#Description of disk attributes
#<disk name>:<primary NSD server>:<2ndary NSD server>:<disk usage>:<failure group>:<NSD name>
#Tiebreaker disk
hdisk3:::descOnly:-1:nsd_tbrk_01:
#Data and metadata disk for /siccfs
hdisk4:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_01:
hdisk5:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_02:
The first disk (hdisk3) holds file descriptor data only and is used as a tiebreaker disk. hdisk4
and hdisk5 are designated as data disks for our LVM cluster file system. We use our quorum
nodes as primary and backup NSD servers as well, although all of our nodes do have physical
access to the disks. This provides some automatic fallback capability should the SAN access
become unavailable.
Create the NSD devices using the mmcrnsd command and verify your setup using the
mmlsnsd command (the option -v yes specifies that the NSD are to be created only if the disk
has not been formatted by a previous invocation of the mmcrnsd command):
root> mmcrnsd -F /var/mmfs/config/nsd.disks -v yes
root> mmlsnsd -aL
File system Disk name NSD volume ID Primary node Backup node
---------------------------------------------------------------------------------------------
siccfs nsd_siccfs_01 0999A547447D7EFE siccps12 siccps10
siccfs nsd_siccfs_02 0999A547447D7F00 siccps12 siccps10
(free disk) nsd_tbrk_01 0999A547447D7DA6 (directly attached)
The specified disk file is rewritten during NSD creation. Each hdisk entry is preceded by a
comment (# sign) and followed by the equivalent NSD entry. The new file can then be reused
for file system creation during subsequent configuration steps. The file system create
command uses NSD devices and not AIX hdisks.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 42 of 156
3.2.1.2.4 Activate Node Quorum using Tiebreaker Disks
You need to shutdown the GPFS cluster and use the mmchconfig to activate the tiebreaker
disks. Use the following commands to activate the NSD device nsd_tbrk_01 as a tiebreaker
disk:
root> mmshutdown –a
root> mmchconfig tiebreakerDisks="nsd_tbrk_01"
root> mmstartup –a
You can use the mmlsconfig and mmgetstate commands to check your configuration after the
switch to tiebreaker mode:
root> mmlsconfig
Configuration data for cluster siccfs.isicc:
-------------------------------------------
clusterName siccfs.isicc
clusterId 691765743288319456
clusterType lc
autoload no
useDiskLease yes
maxFeatureLevelAllowed 903
tiebreakerDisks nsd_tbrk_01
[siccps12]
takeOverSdrServ yes
File systems in cluster siccfs.isicc:
------------------------------------
/dev/siccfs
root> mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
------------------------------------------------------------------------------------
1 siccps12 1* 2 8 active quorum node
2 siccps10 1* 2 8 active quorum node
In the mmlsconfig output you can see that device nsd_tbrk_01 is used as tiebreaker disk. The
quorum information in the mmgetstate output is displayed as “1*”, which indicates that this is
a two node tiebreaker disk cluster.
3.2.1.2.5 Create GPFS File System
The last step is to create the file system itself. We used the following excerpt of the disk file
that was created by the mmcrnsd command as input to create file system command:
/var/mmfs/config/disks.siccfs:
#Description of disk attributes
#<disk name>:<primary NSD server>:<2ndary NSD server>:<disk usage>:<failure group>:<NSD name>
#Data and metadata disk for /siccfs
# hdisk4:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_01:
nsd_siccfs_01:::dataAndMetadata:1::
# hdisk5:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_02:
nsd_siccfs_02:::dataAndMetadata:1::
The mmcrfs command was used to create the /siccfs cluster file system:
root> mmcrfs /siccfs /dev/siccfs -F /var/mmfs/config/disks.siccfs –A automount -B 256K -v yes
We used the –A automount option to specify that the file system is automatically mounted
when its first accessed.
3.2.1.2.6 Mount GPFS File System
If you did not use the automount option during file system creation, you can use the standard
AIX mount command to mount the cluster filesystem:
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 43 of 156
root> mount /siccfs
3.2.1.3 Considerations for SAP Instance Installation on GPFS
A SAP system has to be up and running before it can be registered and configured in the
Landscape Virtualization Management software. SAP LVM is not able to create the required
environment (mount target directories) for a new installation, this has to be done manually
(see also section 3.4.5). You should first create the directory structure on the GPFS file
system according to your naming conventions and then mount the subdirectories at the mount
points that are expected by the SAP installation routines.
The following commands create a sample directory structure for system C01:
mkdir /siccfs/sapC01
mkdir /siccfs/sapC01/oracle
mkdir /siccfs/sapC01/sapmnt
mkdir /siccfs/sapC01/usrsap
mkdir /siccfs/sapC01/trans
mkdir /siccfs/sapC01/c01adm
Then you need to create the required mapping before you can start the installation. This is
done with a normal mount command using the option –v mmfs (you may have to create some
of the directories for the mount points first, e.g. mkdir /home/c01adm).
mount -v mmfs /siccfs/sapC01/oracle /oracle
mount -v mmfs /siccfs/sapC01/sapmnt /sapmnt/C01
mount -v mmfs /siccfs/sapC01/usrsap /usr/sap/C01
mount -v mmfs /siccfs/sapC01/trans /usr/sap/trans
mount -v mmfs /siccfs/sapC01/c01adm /home/c01adm
After the installation is finished and the new SAP instance is registered at the Solution
Manager, you should unmount all directories and proceed to the SAP LVM configuration
tasks.
3.2.1.4 Instance Configuration in SAP LVM for GPFS
When you enable SAP systems in SAP LVM, you have to configure the data storage on the
System tab in the Configuration menu (for further details see chapter 4.7).
Data residing on a GPFS file system must be defined with Storage Type DFS. The Partner Id
is ibm and FS/SRID Type is gpfs. Please note that these values are case sensitive and have to
be entered using lower case.
The Export Path designates the directories as they are stored in the GPFS file system and the
Mount Point/SRID is the mount point as expected by the SAP kernel. The last field is Mount
Options, the available options for GPFS file systems are fsmount and dirmount.
The first one can be used to mount the GPFS file system itself (in case you don’t use the
GPFS automount option) and the second option to specify the subsequent subdirectory
mounts.
For example to mount subdirectory /usr/sap/EC1 on a GPFS file system that does not use the
automount option, you would need the following two entries:
Storage Type DFS
Mount Point/SRID /siccfs
Export Path /dev/siccfs
Mount Options fsmount
FS/SRID Type gpfs
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 44 of 156
Partner ID ibm
Storage Type DFS
Mount Point/SRID /usr/sap/EC1
Export Path /siccfs/sapEC1/usrsap
Mount Options dirmount
FS/SRID Type gpfs
Partner ID ibm
Figure 10 shows an example configuration of a GPFS based SAP system within the ISICC
SAP LVM landscape. The database used in this example is Oracle. For DB2 based systems,
the filesystems need to be created and mounted accordingly to the DB2 file system structure.
Note that a separate pool for SAP systems using the GPFS file systems was created in SAP
LVM.
Figure 10 Example configuration of an SAP system using GPFS
3.2.1.5 Add a Node to GPFS Cluster
You need to add an AIX node to the GPFS cluster before you can mount the cluster file
system. This is done with the mmaddnode command after the GPFS software installation is
completed:
root> mmaddnode –N siccps18
You can remove a node from the cluster using the mmdelnode command and we recommend
that you remove the node before you start a NIM re-installation of a specific node.
You can list the state of cluster nodes with the mmgetstate command:
root> mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
------------------------------------------------------------------------------------
1 siccps12 1* 2 8 active quorum node
2 siccps10 1* 2 8 active quorum node
3 siccps17 1* 2 8 active
5 siccps13 1* 2 8 active
6 siccps14 1* 2 8 active
7 siccps15 1* 2 8 active
8 siccps16 1* 2 8 active
9 siccps18 1* 2 8 active
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 45 of 156
3.2.2 SAN Storage
The SAP NetWeaver Landscape Virtualization Management allows the direct management of
natively attached storage without having the need of a shared file system like GPFS.
IBM provides a deep integration of IBM storage functionality with SAP LVM.
This includes, but is not limited to,
creation and mapping of storage volumes to hosts or LPARs
discovering the volumes within the operating system
create file systems
creation of snapshots of volumes
mapping the snapshots to hosts / LPARs
re-mapping of volumes
This allows automated and fast moving, cloning and copying of SAP systems.
This chapter briefly describes the storage components and setup used in this proof of concept.
You can find the detailed description about how the components were installed and
configured in chapter 3.4.4.
3.2.2.1 Storage Components
Storage systems:
IBM SAN Volume Controller
IBM XIV Storage System
Firmware Version: 10.2.4a
79 TB usable space
A separate pool ‘ISICC_cloud’ was created for the volumes of the test environment
Storage Management Software:
XIV CLI version (XCLI) version: 3.1 build 18
IBM Tivoli FlashCopy Manager: 3.2.0.1
3.2.2.2 SAP LVM SAN Configuration and File Systems
You need to install the SAP instances to be managed with SAP LVM according to the SAP
installation guides.
The following file system layout is recommended, and required if IBM Tivoli® Storage
FlashCopy® Manager is used to create a FlashCopy backup or clone of a SAP DB instance;
typically this applies to SAP production systems. The number of sapdata file systems (see
chapter 3.4.5.1) might vary, the default is four. A minimum of three different volume groups
need to be configured, as shown in Figure 11.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 46 of 156
Figure 11 Volume Group and Filesystem layout
List of file systems used: root> mount
node mounted mounted over vfs date options
-------- --------------- --------------- ------ ------------ ---------------
/dev/hd4 / jfs2 Jul 02 09:47 rw,log=/dev/hd8
/dev/hd2 /usr jfs2 Jul 02 09:47 rw,log=/dev/hd8
/dev/hd9var /var jfs2 Jul 02 09:47 rw,log=/dev/hd8
/dev/hd3 /tmp jfs2 Jul 02 09:47 rw,log=/dev/hd8
/dev/hd1 /home jfs2 Jul 02 09:48 rw,log=/dev/hd8
/dev/hd11admin /admin jfs2 Jul 02 09:48 rw,log=/dev/hd8
/proc /proc procfs Jul 02 09:48 rw
/dev/hd10opt /opt jfs2 Jul 02 09:48 rw,log=/dev/hd8
/dev/livedump /var/adm/ras/livedump jfs2 Jul 02 09:48 rw,log=/dev/hd8
siccserv /siccfs /home/siccfs nfs4 Jul 02 09:48 ro,bg,hard,intr,vers=4,sec=sys
/dev/lv10 /sapmnt/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv00
/dev/lv11 /usr/sap/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv00
/dev/lv13 /home/cl6adm jfs2 Jul 02 11:02 rw,log=/dev/loglv00
/dev/lv12 /sapinst jfs2 Jul 02 11:02 rw,log=/dev/loglv00
/dev/lv15 /usr/sap/DAA/SMDA97 jfs2 Jul 02 11:02 rw,log=/dev/loglv00
/dev/lv09 /db2 jfs2 Jul 02 11:02 rw,log=/dev/loglv01
/dev/lv16 /db2/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv01
/dev/lv01 /db2/CL6/db2cl6 jfs2 Jul 02 11:02 rw,log=/dev/loglv02
/dev/lv03 /db2/CL6/log_dir jfs2 Jul 02 11:02 rw,log=/dev/loglv03
/dev/lv04 /db2/CL6/log_archive jfs2 Jul 02 11:02 rw,log=/dev/loglv01
/dev/lv02 /db2/CL6/db2dump jfs2 Jul 02 11:02 rw,log=/dev/loglv01
/dev/lv05 /db2/CL6/sapdata1 jfs2 Jul 02 11:02 rw,log=/dev/loglv02
/dev/lv06 /db2/CL6/sapdata2 jfs2 Jul 02 11:02 rw,log=/dev/loglv02
/dev/lv07 /db2/CL6/sapdata3 jfs2 Jul 02 11:02 rw,log=/dev/loglv02
/dev/lv08 /db2/CL6/sapdata4 jfs2 Jul 02 11:02 rw,log=/dev/loglv02
/dev/lv14 /var/db2 jfs2 Jul 02 11:02 rw,log=/dev/loglv01
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 47 of 156
List of volume groups used:
root> lsvg -l vg00 vg01 vg02 vg03
vg00:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv00 jfs2log 1 1 1 open/syncd N/A
lv12 jfs2 256 256 1 open/syncd /sapinst
lv10 jfs2 251 251 1 open/syncd /sapmnt/CL6
lv11 jfs2 376 376 1 open/syncd /usr/sap/CL6
lv13 jfs2 8 8 1 open/syncd /home/cl6adm
lv15 jfs2 62 62 1 open/syncd /usr/sap/DAA/SMDA97
vg01:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv01 jfs2log 1 1 1 open/syncd N/A
lv14 jfs2 1 1 1 open/syncd /var/db2
lv02 jfs2 7 7 1 open/syncd /db2/CL6/db2dump
lv04 jfs2 254 254 1 open/syncd /db2/CL6/log_archive
lv09 jfs2 178 178 1 open/syncd /db2
lv16 jfs2 1 1 1 open/syncd /db2/CL6
vg02:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv02 jfs2log 1 1 1 open/syncd N/A
lv01 jfs2 1 1 1 open/syncd /db2/CL6/db2cl6
lv05 jfs2 313 313 1 open/syncd /db2/CL6/sapdata1
lv06 jfs2 313 313 1 open/syncd /db2/CL6/sapdata2
lv07 jfs2 313 313 1 open/syncd /db2/CL6/sapdata3
lv08 jfs2 313 313 1 open/syncd /db2/CL6/sapdata4
vg03:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv03 jfs2log 1 1 1 open/syncd N/A
lv03 jfs2 497 497 1 open/syncd /db2/CL6/log_dir
List of hdisks used:
root> lsvg -p vg00 vg01 vg02 vg03
vg00:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk2 active 1019 65 00..00..00..00..65
vg01:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk1 active 1019 577 77..00..92..204..204
vg02:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk3 active 1279 25 00..00..00..00..25
vg03:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk4 active 1019 521 113..00..00..204..204
The list of volumes defined on the XIV storage system for SID CL6 is shown in Figure 12.
It also shows the snapshots created during the cloning of SAP system CL6 as source system.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 48 of 156
Figure 12 Volumes of SAP system CL6
The volume wwn is part of the hdisk unique_id.
This attribute is used to identify the hdisk on the storage subsystem, as highlighted in yellow
and green:
XIV 1300360>>vol_list vol=CL6_datavg_1 -t wwn
WWN
00173800016800CA
root> lsattr -l hdisk3 -E -a unique_id
unique_id 2611200173800016800CA072810XIV03IBMfcp
Figure 13 shows the database instance of SAP system CL6 in LVM and the according mount
points.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 49 of 156
Figure 13 SAP System CL6 in LVM and mount points
We recommend that host names defined on the storage system to which the volumes are
mapped exactly match the real host/node name. The primary mapping of operating system to
the host name on the storage system is done by using port IDs (WWPNs). In case this is not
possible for some reason, mapping is identified by comparing the real host name of the
operating system with the host name on the storage system.
IBM Tivoli FlashCopy Manager (FCM) is used to manage the snapshots on the storage
systems for SAP system cloning and copying purposes and must be also installed on all SAP
LVM managed systems.
The installation and configuration steps of FCM are described in detail in chapter 3.4.4).
3.3 Install Management Environment
3.3.1 Install SAP NetWeaver Landscape Virtualization Management Software
The current SAP NetWeaver Landscape Virtualization Manager 2.0 software is a Java
application that is deployed on a SAP NetWeaver Application Server Java system of usage
type “Basic”.
Please download the appropriate installation guide for the target operating system and
database platform combination and follow the instructions in that guide to install a SAP
NetWeaver Java System:
http://service.sap.com/instguidesNW73
http://service.sap.com/instguidesNW74
As always please pay particular attention to the SAP notes mentioned in the “SAP Notes for
the Installation” section of the installation guide.
Download and deploy the SAP NetWeaver Landscape Virtualization Management software
after the installation and configuration of the SAP Java System is complete.
Please download the latest version of the installation guide for your licensed version (standard
edition or enterprise edition):
http://service.sap.com/instlvm
Again it is important to check all SAP notes mentioned on the download page and in the
installation guide itself. The installation guide describes the necessary steps to install LVM
after the initial setup of SAP NetWeaver AS Java.
At a high level, the installation of a SAP LVM management system requires:
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 50 of 156
1. Install SAP NetWeaver AS Java with usage type “BASIC”
2. Download the required SCA files for SAP LVM from the SAP Service Marketplace
http://service.sap.com/swdc
Software DownloadsBrowse our Download CatalogSAP NetWeaver and
complementary productsSAP NW LANDSC VIRT MGT ENT
3. Deploy the downloaded SCA files using one of the supported implementation tools
(SWPM or telnet)
4. Perform the initial setup using the configuration wizard as described in the post-
installation section
The last step is described in detail on the SAP Help Portal at:
http://help.sap.com/lvmstd20
http://help.sap.com/lvment20
Initial SetupCarrying Out the Initial Setup Using the Configuration Wizard
Follow the procedure documented in the help portal to step through the configuration wizard.
After the wizard finished successfully, you should be able to access SAP LVM using the
following URL: http://<host>:<port>/lvm.
3.3.2 Install IBM Tivoli Storage FlashCopy Manager
SAP LVM exploits IBM Tivoli Storage FlashCopy Manager to perform snapshot backups for
file systems related to the SAP Systems. The FCM setup is done according to the FlashCopy
backup of "Custom application environments" documentation. The snapshots are managed as
backup versions using the IBM Tivoli Storage FlashCopy Manager management policies. The
storage volumes for the target system are acquired and mounted on the target LPAR.
The management agent (acsd) of Tivoli Storage FlashCopy Manager is installed centrally on
one of the management LPARs. The "acsd" agent coordinates the snapshot backup operation
and controls the backup flow and mediates between the other agents. The management agent
provides access to the snapshot backup repository, which contains information about the valid
snapshot backups and their relationships to the snapshot capable storage devices.
On each host an “acsgen” agent is started during the SAP cloning/ SAP system copy process.
This agent performs the storage related tasks on the source and target hosts.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 51 of 156
Figure 14 illustrates the environment:
Figure 14 Overview of the FlashCopy Manager Installation
The Tivoli FlashCopy Manager installation will follow the same steps as described in chapter
3.4.4.8 for the setup for a "Custom application environments". Additionally, a profile template
needs to be created on the management node.
The following shows a sample profile for the IBM XIV Storage:
>>> GLOBAL
# ACS_DIR /home/sapadm/acs
ACSD siccplvm 57328
# TRACE NO
<<<
>>> ACSD
ACS_REPOSITORY /home/sapadm/acs/acs_rep
# REPOSITORY_LABEL TSM
<<<
>>> DEVICE_CLASS STANDARD
COPYSERVICES_HARDWARE_TYPE XIV
# CLONE_DATABASE NO
#STORAGE_SYSTEM_ID 3984
PATH_TO_XCLI /opt/IBM/xiv/XIVGUI3.1
COPYSERVICES_SERVERNAME siccxiv
COPYSERVICES_USERNAME saplvm
BACKUP_HOST_NAME siccps98
# RECON_INTERVAL 12
# GRACE_PERIOD 24
# USE_WRITABLE_SNAPSHOTS AUTO
# USE_CONSISTENCY_GROUPS YES
<<<
>>> CLIENT
LVM_FREEZE_THAW NO
ENHANCED_PARTITIONING NO
<<<
VIOS
vFC (NPIV)
VIOS
vFC (NPIV)
Management Environment
SAP LVM AIX
Storage Manager native Interface e.g. IBM XIV XCLI 3.1 /opt/IBM/xiv/XIVGUI3.1/xcli
Managed Environment
IBM Tivoli FlashCopy Manager acsd /home/sapadm/acs
SAP Host Agent /usr/sap/hostctrl
SAP Source AIX
Storage Manager native Interface e.g. IBM XIV XCLI /opt/IBM/xiv/XIVGUI3.1/xcli
IBM Tivoli FlashCopy Manager acsgen /home/sapadm/acs
SAP Host Agent /usr/sap/hostctrl
SAP Target AIX
Storage Manager native Interface e.g. IBM XIV XCLI /opt/IBM/xiv/XIVGUI3.1/xcli
IBM Tivoli FlashCopy Manager acsgen /home/sapadm/acs
SAP Host Agent /usr/sap/hostctrl
IBM XIV SVC / Storwize Source
Target
SAN Storage • IBM XIV, IBM SVC, or IBM
Storwize V7000 Storage System
• Storage connected via vFC (NPIV)
• Compliant Storage Layout
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 52 of 156
SAP LVM takes this template and generates “runtime” templates during the actual execution.
The “ACSD” entry in the GLOBAL section of the template refers to the hostname of the
management system running the FCM acsd agent.
SAP LVM will replace the host name and port number (2nd argument) dynamically during
the actual run. IBM Tivoli Storage FlashCopy Manager traces can be enabled in this section
(set parameter “TRACE YES”).
The ACS_REPOSITORY in the ACSD section refer to the repository directory and will be
dynamically replaced too.
In the DEVICE_CLASS section, ensure that the following settings are included for IBM XIV
storage:
COPYSERVICES_HARDWARE_TYPE needs to be set to XIV
PATH_TO_XCLI needs to point to the installation directory of the XIV XCLI (see also
chapter 3.4.4.7)
COPYSERVICES_SERVERNAME needs to include the hostname / IP address of the XIV
system
COPYSERVICES_USERNAME needs to contain an administrative user on the IBM XIV
system.
IBM Tivoli Storage FlashCopy Manager uses this user to communicate with the IBM
XIV System. The user needs to be created on the IBM XIV system first, and the
password needs to be entered later during FCM setup.
In the CLIENT section, two entries need to be specified:
LVM_FREEZE_THAW NO
ENHANCED_PARTITIONING NO
Tivoli Storage FlashCopy Manager 3.2 uses IBM Global Security Kit (GSKit) for secure
communication between the distributed processes running on different LPARs. GSKit is part
of the Tivoli Storage FlashCopy Manager 3.2 software package and needs to be installed on
all LPARs.
After the configuration of Tivoli Storage FlashCopy Manager there are additional files (server
key database, client certificate) which need to be distributed across all the LPARs involved in
the scenario (central system, source system, target system).
3.3.3 LDAP Server
3.3.3.1 User Management
A SAP LVM infrastructure requires a method to synchronize naming services on all enabled
servers. This includes information on users, groups, protocols, services, hosts, etc.
Possible solutions are NIS/NIS+ or the synchronization of local files (typically stored in /etc)
using tools like rdist.
However, development of NIS has stopped and the recommended solution for centralized user
management is LDAP (Lightweight Directory Access Protocol). LDAP is widely used and
has become the standard protocol for access to information registries, optimized for read
mostly access.
AIX clients can work with any LDAP server as long as it is V3 conform. In our sample
implementation, we implemented the IBM Tivoli Directory Server V6.3 as LDAP server.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 53 of 156
More information is available at:
http://www-01.ibm.com/software/tivoli/products/directory-server/
The following introduces some of the LDAP terminology and discusses briefly some
important steps for planning and running LDAP on AIX. This is no replacement for the
product documentation. Especially an enterprise wide usage of LDAP sharing directory
information for other services requires careful planning and you need to consider issues like
namespaces, secure authentication using SSL, and implementing high availability using server
replication. These topics are fully covered in the product documentation and the available
Redbooks and whitepapers.
3.3.3.2 Schema
A schema defines which objects and attributes can be stored in a Directory Information Tree
(DIT). It defines the names of objects and attributes, syntax, and possible values as well as
whether attributes are required or optional for an object. For example, an object could be
people or server and attributes of the objects could be name, homedirectory, uid or ip-
address, hostname.
The following schemas are available for AIX user management and authentication:
AIX The AIX schema includes the aixAccount and aixAccessGroup object classes. This
schema offers the entire AIX user and group attributes. This schema is included to support
legacy LDAP installations prior to Version 5.2.
RFC2307 The RFC2307 schema includes the posixAccount, posixGroup and other NIS
related object classes. This experimental RFC defines a schema that allows NIS maps to be
imported into LDAP. RFC2307 only defines a subset of the AIX user and group attributes.
This schema supports any RFC2307 compliant platforms and AIX 5L Version 5.2.
RFC2307AIX The RFC2307AIX schema includes the RFC2307 schema plus the AIX
specific object classes, aixAuxAccount and aixAuxGroup. The AIX specific object classes
provide attributes to store additional attributes not defined by the RFC2307 standard.
The recommended schema for AIX5.2 and higher is RFC2307AIX and has been used for the
SAP LVM reference implementation. This has several advantages such as RFC2307 is
considered a standard for UNIX systems including Linux.
3.3.3.3 Namespace
The LDAP naming model defines how entries are identified and organized. Entries are
organized in a tree-like structure called the Directory Information Tree (DIT) based on their
distinguished name (DN). A DN is a unique name that unambiguously identifies a single
entry. Distinguished Names are made up of a sequence of relative distinguished names
(RDN). Each RDN in a DN corresponds to a branch in the DIT leading from the root of the
DIT to the directory entry.
A RDN typically has the form <attribute name> = <value>. A DN is composed of a sequence
of relative distinguished names separated by commas.
The following DIT shows a sample DIT with a suffix o=isicc,c=de:
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 54 of 156
Figure 15 Sample LDAP Directory Information Tree for AIX user management
3.3.3.4 LDAP Server Installation and Configuration
For LDAP server implementations please review and follow the instructions in the mentioned
references.
3.3.3.5 LDAP Client Installation
AIX LDAP client is available in a 32-bit and a 64-bit version. Their filesets are shipped with
the AIX CDs and should be included in the install images for the SAPL LVM client nodes.
3.3.3.6 LDAP Client Configuration
mksecldap is an AIX command for IBM Directory server and client setup. Please check the
man pages or the online documentation for the full syntax description.
The mksecldap command performs the required updates in various configuration files, starts
the secldapclntd daemon, and puts an entry in /etc/inittab so that the seclpdapclntd daemon
will be automatically started after reboot.
The following command is used in our demo environment to configure an AIX LDAP client:
/usr/sbin/mksecldap -c -h siccldap1,siccldap2 -a cn=root -p ldappw
There are a number of commands available to control the client daemon:
start-secldapclntd start ldap client daemon
stop-secldapclntd stop ldap client daemon
restart-secldapclntd restart the ldap client daemon
ls-secldapclntd show the status of the ldap client daemon
flush-secldapclntd clear the cache of the ldap client daemon
3.3.3.7 User Authentication
The last step required to setup user login through LDAP is to change a user’s authentication
mechanism. User authentication is controlled by the SYSTEM attribute in the
/etc/security/user file. For example to enable LDAP user c01adm to login to the local system,
you could execute the following command;
c=de
o=isicc
cn=user
uid=c01adm
uid=orac01
cn=group
gid=sapsys
gid=dba
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 55 of 156
chuser SYSTEM=LDAP registry=LDAP c01adm
To allow all LDAP users to login to the system, one can set each LDAP user's SYSTEM
attribute to LDAP, or simply set the default stanza's SYSTEM attribute to LDAP. This has to
be done manually in the /etc/security/user file, where you need to change the following two
attributes in the default stanza.
Default:
SYSTEM = "LDAP or compat"
registry = LDAP
We recommend using local authentication for the root user so that you can still login to the
server in case there is something wrong with the LDAP setup. With local authentication the
root user’s stanza should look like:
root:
SYSTEM = "compat"
registry = files
In our sample environment the complete client setup is automated in a NIM script. The script
uses mksecldap to configure the client and then runs a Perl command to automatically change
the default and root user’s stanza to the appropriate:
Sample Customizing Script for Automated LDAP Client Setup #Setup LDAP Client
#AIX users are managed by LDAP
/usr/sbin/mksecldap -c -h siccldap1,siccldap2 -a cn=root -p ldappw
perl -p -i -e 'if ( /default:/ .. /SYSTEM =/ ) {
s/"compat"/"LDAP or compat"\n\tregistry = LDAP/;
} elsif ( /root:/ .. /SYSTEM =/ ) {
s/"compat"/"compat"\n\tregistry = files/;
}' /etc/security/user
3.3.3.8 Create Users and Groups
The AIX mkuser command is able to create a user on the LDAP server using the load module
option “-R LDAP”. Unfortunately the “smit mkuser” panel doesn’t allow for this option. You
should create a new user using the mkuser command. Once a user is created, you can use the
“smit chuser” panel to change user attributes. For example to create a user c01adm, use the
command
mkuser –R LDAP registry=LDAP c01adm
Then use the “smit chuser” panel to change the other required attributes (of course, you could
also set them directly using the appropriate attribute options with the mkuser command).
The problem also applies to the mkgroup command, so you have to create the required groups
manually as well:
mkgroup –R LDAP sapsys
Consider using some naming conventions for the numerical user id, for example by reserving
certain ranges for the SAP and database administration users:
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 56 of 156
mkgroup -R LDAP registry=LDAP id=400 dbqa1adm
mkgroup -R LDAP registry=LDAP id=401 dbqa1ctl
mkgroup -R LDAP registry=LDAP id=402 dbqa1mnt
mkgroup -R LDAP registry=LDAP id=403 dbqa1mon
mkgroup -R LDAP registry=LDAP id=404 sapsys
mkgroup -R LDAP registry=LDAP id=405 sapinst
mkuser -R LDAP registry=LDAP id=400 pgrp=sapsys
groups='sapsys,dbqa1ctl,dbqa1mon,sapinst' home='/home/qa1adm' gecos='SAP System
Administrator' shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151
qa1adm
mkuser -R LDAP registry=LDAP id=401 pgrp=dbqa1adm groups='dbqa1adm'
home='/db2/db2qa1' gecos='SAP Database Administrator' shell='/usr/bin/csh'
fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 db2qa1
mkuser -R LDAP registry=LDAP id=402 pgrp=dbqa1mnt groups='dbqa1mnt'
home='/home/sapqa1' gecos='ABAP Database Connect User' shell='/usr/bin/csh'
fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 sapqa1
mkuser -R LDAP registry=LDAP id=403 pgrp=dbqa1mon groups='dbqa1mon'
home='/home/sapqa1db' gecos='Java Database Connect User' shell='/usr/bin/csh'
fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 sapqa1db
3.3.4 Other Central Management Components
3.3.4.1 OS Management
We highly recommend a centralized OS software management concept.
There are two common solutions to manage the installation and maintenance of AIX images
in a server landscape. The first one is using the AIX built-in Network Installation
Management (NIM) features. The second one is to use the VMControl plugin of IBM Flex
System Manager or IBM Systems Director.
NIM allows the installation and maintenance of AIX base operating system and additional
software from a central repository. As images are sent across the network to the partition to be
installed, the installation steps can be performed fully remotely without needing physical
access to the server. During and after base installation, you can customize the machines by
running specific customizing scripts. Later on, you can deploy fixes and new software images
to the server partitions from a central administration point.
NIM supports several different machine types: diskless, dataless and standalone. A diskless
machine has no own disks, all file systems and other resources are installed remotely and
accessed via the network. A dataless machine has local paging and dump devices, the file
systems for / and /usr are located on a remote server, all the other file systems are located
either local or remote. On a standalone machine, all file systems and resources are installed
on local disks. Due to performance and availability reasons, we recommend using local (or
SAN attached disks) on the computing nodes and install them as standalone machines. It is
possible to use diskless or dataless machines for test and development systems. They boot of
the network and mount the OS from the NIM server.
The following two Redbooks describe in detail the concepts of centralized OS management
using NIM or VMcontrol:
NIM from A to Z in AIX 5L
http://www.redbooks.ibm.com/abstracts/sg247296.html?Open
IBM Systems Director VMControl Implementation Guide on IBM Power Systems
http://www.redbooks.ibm.com/abstracts/sg247829.html?Open
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 57 of 156
3.3.4.2 Network Infrastructure
There are no special network infrastructure requirements for an SAP LVM landscape.
Common standard today and recommended environment is a switched Gigabit Ethernet
network infrastructure.
The major difference compared to a traditional SAP implementation is the consistent usage of
virtual IP names and addresses assigned to a SAP instance. These addresses need to be
implemented in the customer’s name services infrastructure, typically based on DNS (please
refer to the section 3.1. for some recommendations about naming conventions for virtual IP
names).
If virtual network addresses come from new subnets, one might have to update router tables,
saprouter definitions, etc. to ensure that the end user can access the SAP instance using the
virtual IP name. You also have to change SAP logon groups to reflect the virtual IP names.
Note 962955 - Use of virtual TCP/IP host names
3.4 Install System Landscape
The following section provides an overview about the preparation steps required for the
environment. This includes considerations about
PowerVM and VIOS
IBM Flex System Manager or IBM Systems Director
IBM HMC
OS setup for the hosts
SAP Installation
3.4.1 PowerVM and VIOS
Base assumption is that the volume(s) for the AIX operating system (rootvg) is/are located on
the same storage system as the SAP system, and that the “rootvg” storage volumes are also
attached via virtual fibre channel (vFC). In this case, the “host” and “adapter” definition in the
storage system required for the access to the “rootvg” volumes will ensure the required
connectivity to the SAP application related volumes in the storage system also.
3.4.2 IBM Systems Director Setup
The following describes high-level configuration steps required for the integration of either
IBM Systems Director or IBM Flex System Manager with SAP LVM. Further details are
described in SAP Note 1728222 - Configuration of IBM Systems Director VMControl for
SAP LVM
The term “virtualization manager” is applied for both integration scenarios. IBM Systems
Director and IBM Flex System Manager both require the identical steps for the setup. The
tasks on the virtualization manager include:
Discover the IBM HMCs on the virtualization manager and configure access to them.
Perform a "collect inventory" task on the virtualization manager for IBM HMC, Power
servers, and for all logical partitions (LPARs).
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 58 of 156
Discover operating systems running on partitions, configure access, and perform
"collect inventory" task on virtualization manager for all the discovered operating
systems.
Define a user for SAP LVM access on the virtualization manager
The default administrator user can be used as user for SAP LVM. The administrators
in SAP LVM then have the same credentials and get data for all the IBM Power
Systems configured in the virtualization manager.
To restrict the access to the specific set of IBM Power Systems managed by SAP
LVM, a specific user can be created in the virtualization manager. This user does not
have the credentials for all Power systems available in the virtualization manager, but
only to those in scope of the SAP LVM environment. The configuration is described
in chapter 4.9.
Create a security certificate on the virtualization manager
Per default, the virtualization manager uses Port 8422 for communication.
Communication via this port must be allowed in the network and firewall settings for
the environment.
o Open the URL “https://<virtualization manager host>:8422" in a browser
window. During the first connection attempt the browser displays a notification
window to accept the certificate. This virtualization manager certificate is
stored in the internet browser configuration locally, and can be exported to a
certificate file.
o The virtualization manager certificate needs to be imported into the SAP
NetWeaver system with the deployed SAP LVM software.
The certificate store must be "TrustedCAs".
Check that the hostname in the certificate is valid.
For more information on this topic see also the SAP documentation "Using the
AS Java Key Storage".
Using the virtualization manager for AIX OS provisioning requires to discover, to
access and to perform "collect inventory" tasks for NIM, VIO servers, fibre channel
switches and storage managers in addition.
It must be possible to
Capture and deploy an AIX operating system via NIM (4.9)
Select a particular IBM Power server as target for deployment
Select a storage system for deployment
within the different layers of the virtualization manager (VMControl, Storage Manager
or Control). Then similar provisioning operations can be triggered out of SAP LVM.
Using the virtualization manager for LPAR relocation (Live Partition Mobility)
requires the configuration of “server pools” or “virtual farms”. If it is intended to
migrate LPAR "X" hosted on IBM Power system "A" to IBM Power system "B", then
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 59 of 156
both Power system "A" and Power system "B" must be members of the same server
pool or virtual farm. If the virtualization manager is able to relocate a partition, the
same operation can be triggered in SAP LVM too.
3.4.3 HMC Setup
The recommended setup for SAP LVM is to use IBM Flex System Manager or IBM Systems
Director as virtualization manager. However, the management via IBM Power HMC is
possible too.
This is described in detail in SAP Note 1728293 - Configuration of IBM Power HMC for
SAP LVM.
3.4.4 Install Hosts
This section describes the base configuration steps on an AIX operating system level for the
compute nodes. All the required steps need to be executed at least on the first system image.
This image can then be captured (NIM, VMControl) and can be used as a "master image”
template for the deployment of all additional compute nodes.
The required steps include
Activate IPsec
Check/ update DNS configuration
Configure the LDAP client
Adapt /etc/services base configuration
Check/adapt filesystem sizes
Install SAP Host Agent package and SAPACEXT libraries
Install XIVGUI (IBM XIV Storage systems)
Configure ssh (IBM SVC or Storwize systems)
Install IBM Tivoli Storage FlashCopy Manager
3.4.4.1 Activate IPsec
Each of the hosts participating in a system provisioning use case for SAP system cloning or
SAP system copy needs to be prepared for network isolation.
In an AIX environment, the network isolation is provided via IPsec. IP security, commonly
known as IPsec, is a protocol developed by the Internet Engineering Task Force (IETF),
designed to provide "end-to-end" cryptographically based security for IP network
connections. IPsec implementations include a method of restricting connections to various
services, based on their origin and destination.
This feature is known as packet filtering. All packets on an IP network originate from an IP
address and a port, and are destined for another IP address and port. A packet filter is a
physical or virtual device that sits between the endpoints of a connection and determines
whether the packet should be permitted to continue to its destination. The decision is made by
comparing various attributes of the packet to rules that are defined by the administrator of the
packet filter. Those attributes include source address and subnet, source port, destination
address and subnet, destination port, protocol, direction of the connection, and fragmentation
of the packet.
Make sure that APAR IZ98741 "DYNAMIC FILTER RULE ID NOT MODIFIED BY
RMFILT -N ALL" is installed in the AIX OS image
IPsec is activated via the AIX SMIT tool using fast path “ips4_start”:
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 60 of 156
# smitty ips4_start
Figure 16 Start IPsec
Alternatively, IPsec can be started directly via the command line:
# /usr/sbin/mkdev -c ipsec -t 4
No manual definitions are required for packet filters. SAP LVM defines or releases
automatically the packet filters for system isolation during the execution of the System Copy
or System Clone workflows.
Figure 17 illustrates an additionally required step during the definition of a managed host. The
checkbox “Isolation Ready” must be enabled so that the LPAR can participate as a target host
in a System Copy/System Clone scenario:
Figure 17 Define the Host as “Isolation Ready” in SAP LVM
3.4.4.2 DNS Name Resolution
Ensure that the search order for DNS name resolution is first based on the local /etc/hosts file,
and then on the external DNS nameserver.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 61 of 156
# vi /etc/netsvc.conf
hosts = local,bind4
In our test environment, the content of the /etc/hosts file contains only the IP label for the
hostname of the partition. All external addresses are resolved via DNS. In this case,
communication to the DNS server needs to be allowed during the “isolated”, “network
fenced” state within the System Clone / System Copy workflow. The required settings to
allow DNS communication are illustrated in chapter 4.7, Figure 42.
3.4.4.3 Configure LDAP Client
As prerequisite for the use of LDAP the IBM Tivoli Directory Server Client needs to be
installed on each LPAR.
idsldap.clt32bit61
idsldap.clt64bit61
The LDAP client needs to be defined on the LPAR in the same way as described in chapter
3.3.3.5 and 3.3.3.6.
3.4.4.4 Adapt /etc/services Entries
Table 8 illustrates the ports to be predefined for ABAP instances. For simplicity all ports
covering the full instance number range from 00 to 99 are added to /etc/services.
In total 400 ports for sapdpXX, sapgwXX, sapdpXXs and sapgwXXs are added. If only a
subset of the ports is required in the organization (e.g. due to SAP instance number standards)
then it is sufficient to add only those specific ports to the /etc/services file.
Port Description /etc/services content
sapdpXX
XX { 00, 01, 02, … 99 }
SAP System Dispatcher Ports sapdp00 3200/tcp
…
sapdp99 3299/tcp
sapgwXX
XX { 00, 01, 02, … 99 }
SAP System Gateway Ports sapgw00 3300/tcp
…
sapgw99 3399/tcp
sapdpXXs
XX { 00, 01, 02, … 99 }
SAP System Dispatcher Security Ports sapdp00s 4700/tcp
…
sapdp99s 4799/tcp
sapgwXXs
XX { 00, 01, 02, … 99 }
SAP System Gateway Security Port sapgw00s 4800/tcp
…
sapgw99s 4899/tcp
Table 8 Entries in /etc/services
3.4.4.5 Adapt File System Sizes
Several software components need to be available on each of the LPARs in the managed
environment. If all the software components are installed locally on the LPAR, then sufficient
free space is required in certain filesystems for the management components. Table 9 gives an
example about required sizes for the components in the “local” filesystems/directories:
Directory Size
(approx.)
MByte
VG Content
/usr/sap/hostctrl 200 rootvg or saplocalvg
SAP Host Agent package, together with the SAPACEXT libraries
/opt/IBM/xiv/XIVGUI3.1 (example)
350 rootvg XIV XCLI Installation
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 62 of 156
Directory Size (approx.)
MByte
VG Content
/usr/tivoli/tsfcm 700 rootvg Installation directory for IBM Tivoli Storage
/home/sapadm/acs (example) 700 rootvg or saplocalvg
Work directory for IBM Tivoli Storage FlashCopy Manager (SAP LVM context)
Table 9 Local Filesystem layout
Increase the size of the /usr filesystem to allow to install the SAP Host Agent package
and the IBM Tivoli Storage FlashCopy Manager software (FCM)
Increase the size of the /home filesystem for the copy of the FCM package in
/home/sapadm/acs
Increase the size of the /opt filesystem for the XIVGUI package
# chfs -a size=+1500M /usr
# chfs –a size=+1G /home
# chfs -a size=+1G /opt
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 63 of 156
3.4.4.6 Install SAP Host Agent and SAP Adaptive Computing Extensions
The runtime user for the SAP Host Agent is user “sapadm”. SAP LVM uses this user id to
authenticate with the SAP Host Agent on the managed system. The user “sapadm” must be
available on a compute node before the SAP Host Agent installation.
The SAP Host Agent package requires “root” user credentials during installation.
The group “sapsys” must exist on the host.
The user “sapadm” must exist and be a member of the “sapsys” group.
Create a home directory for sapadm. The FCM setup on the managed LPAR is
performed in the home directory of user sapadm
# mkdir /home/sapadm
# chown sapadm.sapsys /home/sapadm
Download the recent SAPHOSTAGENT.SAR archive and unpack it using the
SAPCAR tool.
The archive contains the “saphostexec” program. The SAP Host Agent package is
installed/upgraded by executing “saphostexec –install”, respectively “saphostexec –
upgrade” as root user. This installs the SAP Host Agent package in directory
/usr/sap/hostctrl.
Some SAP Host Agent runtime executables have the setuid bit set allowing for the
execution of privileged commands during SAP LVM prepare and unprepare
operations.
Download the SAPACEXT.SAR archive and unpack it to the operations.d
subdirectory of the SAP Host Agent installation.
# cd /usr/sap/hostctrl/exe/operations.d
# SAPCAR –xvf /tmp/SAPACEXT.SAR
Additional considerations for the SAP Host Agent are found in the SAP Note
Note 1292836 - Logfiles of the SAP Host Agent
3.4.4.7 Install XIVGUI
In the proof of concept environment XIVGUI 3.1 build 18 is installed. The XIVGUI image is
deployed to the directory /opt/IBM/xiv (XIVGUI is part of the OS image).
Parameter Value
XCLI Installation Directory /opt/IBM/xiv/XIVGUI3.1
Table 10 XIVGUI Install directory
At first create the target directories for the XIVGUI image
# mkdir –p /opt/IBM/xiv/XIVGUI3.1
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 64 of 156
Install additional rpm packages for the XIVGUI image.
Although this is a prerequisite this is not explicitly mentioned in the XIV XCLI
installation guide: Required rpm packages include:
o libgcc
o libstdc
o readline-5
readline-5 additionally requires
o bash
o info
# rpm -i libgcc-4.2.0-3.aix6.1.ppc.rpm
# rpm -i libstdc++-4.2.0-3.aix6.1.ppc.rpm
# rpm -i gcc-4.2.0-3.aix6.1.ppc.rpm
# rpm -i bash-4.2-6.aix5.1.ppc.rpm
# rpm -i info-4.6-1.aix5.1.ppc.rpm
# rpm -i readline-5.2-3.aix5.1.ppc.rpm
Adjust the library settings and the LIBPATH to include these shared libraries
# cd /opt/freeware/lib
# ln -s /opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/4.2.0/libgcc_s.a
# ln -s /opt/freeware/lib/gcc/powerpc-ibm-
aix6.1.0.0/4.2.0/libstdc++.a
# export LIBPATH=$LIBPATH:/opt/freeware/lib
Extract the XIVGUI software from the Software Image Repository
# ./xivgui-3.1-build18-aix.bin
Adapt the xcli script
There is a bug in the xcli script: xcli can be started only out of the installation path.
xcli needs to be patched to be able to start it from any location.
Edit the xcli script and add the line
prg="$0"
in the upper section of the xcli script
Validate the XCLI by a test connection to the IBM XIV storage system
# /opt/IBM/xiv/XIVGUI3.1/xcli
User Name: ...
Password: ...
Machine IP/Hostname: ...
connecting.
XIV 1300360>>
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 65 of 156
3.4.4.8 Install IBM Tivoli Storage FlashCopy Manager
SAP LVM exploits IBM Tivoli FlashCopy Manager for the backup of "Custom application
environments". On the managed systems, the “acsgen” agent is used to invoke the different
storage related activities on the LPAR during the System Copy/ System Cloning process.
Base Installation of IBM Tivoli Storage FlashCopy Manager
./3.2.0.1-TIV-TSFCMFTP-AIX.bin
Parameters:
- Choose Locale: 2- English
- PRESS <ENTER> TO CONTINUE
- Press Enter to continue viewing the license agreement, or enter "1" to
accept the agreement
- Choose the Install Set to be installed by this installer: 4 - IBM
Tivoli Storage FlashCopy (R) Manager (Custom Applications)
- ENTER AN ABSOLUTE PATH: /usr/tivoli/tsfcm/acs_3.2.0.1
- IS THIS CORRECT? (Y/N): Y
- PRESS <ENTER> TO CONTINUE
- PRESS <ENTER> TO EXIT THE INSTALLER
Check the license file (contained in IBM Tivoli FlashCopy Manager base installation).
If the FCM 3.2.0.1 package was downloaded as an update package, it will not include
the license file. The license file included in the FCM 3.2.0.0 base package needs to be
copied to the directory:
o Copy license file “tsmacs.lic” from the FCM 3.2.0.0 base package to the FCM
install directory /usr/tivoli/tsfcm/acs_3.2.0.1
Create Directory for FCM profile / environment:
# mkdir /home/sapadm
# chown sapadm:sapsys /home/sapadm
# mkdir /home/sapadm/acs
# chown sapadm:sapsys /home/sapadm/acs
Run the setup for FCM custom application
# cd /usr/tivoli/tsfcm/acs_3.2.0.1
# /setup_gen.sh
Script not started from 'acs' sub-directory of the Home directory of
a custom application user. Trying to locate the correct Home
directory or install IBM Tivoli Storage FlashCopy(R) Manager in a new
Home directory ...
Please enter the Home directory:
/home/sapadm
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 66 of 156
This is a new installation of IBM Tivoli Storage FlashCopy(R) Manager
into this Home directory.
Please enter the name of the user of this Home directory:
sapadm
Please enter the group name of the user:
sapsys
Installing/Updating IBM Tivoli Storage FlashCopy(R) Manager
installation for /home/sapadm/acs
Checking environment
- - - - - - - - - - -
+ OK: installation directory exists.
+ OK: setup_gen.sh runs in IBM Tivoli Storage FlashCopy(R) Manager
installation directory '/usr/tivoli/tsfcm/acs_3.2.0.1'
+ OK: IBM Tivoli Storage FlashCopy(R) Manager installation will be
installed to '/home/sapadm' for the first time.
Installation messages
- - - - - - - - - - -
+ OK: New installation !
+ OK: IBM Tivoli Storage FlashCopy(R) Manager directory
'/home/sapadm/acs' created successfully.
+ OK: Files successfully copied.
checking /home/sapadm/acs/acsgen ...
OK
Select one of these configurations:
(1) On-Site Production System configuration with optional remote
Backup System configuration
(2) On-Site Backup System configuration
Enter '1' to configure IBM Tivoli Storage FlashCopy(R) Manager on the
production server with the option to remotely synchronize the
configuration of one or multiple backup systems using Secure Shell.
Enter '2' to configure IBM Tivoli Storage FlashCopy(R) Manager on the
backup system as a 'separate installation'.
2
Going to install on backup server
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 67 of 156
Please enter the directory where the profile will be stored:
[/home/sapadm/acs]
Deactivating IBM Tivoli Storage FlashCopy(R) Manager management
daemon.
Are you going to perform off-loaded backups to Tivoli Storage
Manager? [y|n]
n
Profile configuration
=====================
Special commands:
'?' ==> Show help for current parameter
'!d' ==> Delete current parameter instance (only applicable
to multi-instance parameters)
FMM8422I Creating new profile '/home/sapadm/acs/profile' for
application 'GENERIC' ...
****** Profile parameters for section GLOBAL : ******
Path of the IBM Tivoli Storage FlashCopy(R) Manager directory
{ACS_DIR} = [/home/sapadm/acs]
Hostname and port of machine running Management Agent {ACSD}
(<hostname> <port>) = [] siccplvm 57328
Low level tracing {TRACE} (YES|NO) = [NO]
FMM8426I Saving profile '/home/sapadm/acs/profile' ...
FMM1555I Profile successfully created.
Please enter the password for authentication with the ACS daemon:
Please re-enter password for verification:
Creating password file at /home/sapadm/acs/shared/pwd.acsd.
A copy of this file needs to be available to all components that
connect to acsd.
Enter the device classes to use for this Backup System. Make sure
each device class is separated by a comma.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 68 of 156
STANDARD
IBM Tivoli Storage FlashCopy(R) Manager requires at least two daemon
processes to be running at all times.
IBM Tivoli Storage FlashCopy(R) Manager can add the necessary daemon
processes to the inittab.
Alternatively you may choose to start and stop these processes
yourself, for instance if you want to include the processes in your
HA policy.
Do you want IBM Tivoli Storage FlashCopy(R) Manager to create the
inittab entries for you? [y|n]
n
Please run the following daemon processes:
/home/sapadm/acs/acsgen -D -M -s STANDARD
Please make sure that the command '/home/sapadm/acs/tsm4acs' is
started by a scheduler or manually.
No special customization for the FCM profile is required on the host. SAP LVM will
derive the required content for the profile out of the master profile on the FCM
management LPAR (“acsd”), and copy the profile to the source and target hosts during
the execution of the SAP System Cloning/SAP System Copy workflow.
No special customization for the FCM profile is required on the host.
Install Global Security Kit
IBM Global Security Kit (GSKit) needs to be installed on all LPARs participating in
the scenario. GSKit can be installed using the setup script:
# cd /home/sapadm/acs
# ./setup_gen.sh -a install_gskit -d /home/sapadm
Afterwards, check if the filesets are properly installed:
lslpp -l GSKit8.gskcrypt64.ppc.rte
lslpp -l GSKit8.gskssl64.ppc.rte
Distribute certificate files
Distribute the files for the Server Key DB
o fcmcert.crl
o fcmcert.rdb
o fcmcert.kdb
o fcmcert.sth
and the Client Certificate
o fcmselfcert.arm
across all LPARs (directory /home/sapadm/acs in our case).
Tivoli Storage FlashCopy Manager password file
The FlashCopy Manager configuration on the central node (acsd) generates a
password file in /home/sapadm/acs/shared/pwd.acsd. The password file needs to be
copied to all the FlashCopy Manager directories on the target hosts. It contains the
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 69 of 156
connection password to the acsd, and also the connection password to the Storage
System.
A NFS-shared directory for the “ACS” directory would simplify the setup for the
password file and the certificates, but was not realized in the lab environment.
3.4.5 SAP System
In the following section, base considerations for the installation of a new SAP system, as well
as the eventually required modifications of an existing SAP system are given. These
considerations need to be taken into account, before the SAP system then can be registered
with SAP LVM. The definition of the new system in the SAP LVM context is described in
chapter 4.7
3.4.5.1 Filesystem considerations for the SAP Systems (SAP System Copy)
3.4.5.1.1 DB2 UDB LUW
The two tables below illustrate two different storage layouts for a SAP System based on DB2
UDB. The first example (Table 11) includes a simple layout, having a few different
filesystems only and just separating between SAP application and database filesystems. The
filesystem layout fulfils the requirements for the relocation and SAP system copy scenarios
with SAP LVM.
Place all DB2 UDB Files into one common AIX Volume Group for the database: Filesystem Volume Group Content
/db2 vg01 Contains the instance directory /db2/db2<sid> and all other database related directories. All DB2 database specific data is stored in a directory structure below the filesystem /db2
/var/db2 vg01 Filesystem for DB2 global registry
/home/<sid>adm vg00 Filesystem for home directory of SAP admin user
/sapmnt/<SID> vg00 Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source
/usr/sap/<SID> vg00 Filesystem for SAP Instance directories
/usr/sap/DAA/SMDA97 vg00 Filesystem for SMD agent
Table 11 Minimum filesystem layout (DB2 UDB)
In the second example (Table 12) the filesystem layout is more complex. The database layout
is split-up into different volume groups. The filesystem layout fulfils the requirements for the
relocation and SAP system copy scenarios with SAP LVM as well as the capability for
performing SnapShot/FlashCopy backups for the database with IBM Tivoli Storage
FlashCopy manager.
Separate DB2 datafiles, active logfiles, and all other DB2 files into three different AIX
Volume Groups
Make sure that the database directory /db2/<SID>/db2<sid> is part of the volume
group with the data files
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 70 of 156
Filesystem Volume Group Content
/db2/ vg01 Contains the instance directory /db2/db2<sid>.
/db2/<SID> vg02 Contains the database directory. In case that regular SnapShot backups are intended for the database, the content needs to be included in the SnapShot image together with the DB2 database containers/ storage paths
/db2/<SID>/log_archive vg01 Separate filesystem for DB2 inactive logs
/db2/<SID>/log_dir vg03 Separate filesystem for DB2 active logs: In case SnapShot backups are regularly taken and should be used for a fast restore, this filesystem *must* be separate from database data.
/db2/<SID>/db2dump vg01 Separate filesystem for the db2diag.log for convenience
/db2/<SID>/sapdataX vg02 Contains the database containers / storage paths
/home/<sid>adm vg00 Filesystem for home directory of SAP admin user
/sapmnt/<SID> vg00 Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source
/usr/sap/<SID> vg00 Filesystem for SAP Instance directories
/usr/sap/DAA/SMDA97 vg00 Filesystem for SMD agent
/var/db2 vg01 Filesystem for DB2 global registry
Table 12 Extended filesystem layout (DB2 UDB)
If FlashCopy backups for the database are a requirement, then a layout similar to the second
example has to be implemented.
3.4.5.1.2 Oracle Database
The two tables below illustrate two different storage layouts for a SAP System based on an
Oracle database. The first example (Table 11) includes a simple layout, having a few different
filesystems only and just separating between SAP application and database filesystems. The
filesystem layout fulfils the requirements for relocation and SAP system copy scenarios with
SAP LVM.
Place all ORACLE Files into one common AIX Volume Group for the database: Filesystem Volume Group Content
/oracle vg01 Contains all database related directories.
/home/<sid>adm vg00 Filesystem for home directory of SAP admin user
/sapmnt/<SID> vg00 Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source
/usr/sap/<SID> vg00 Filesystem for SAP Instance directories
/usr/sap/DAA/SMDA97 vg00 Filesystem for SMD agent
Table 13 Minimum filesystem layout (Oracle)
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 71 of 156
In the second example (Table 12) the filesystem layout is more complex. The database layout
is split-up into different volume groups. The filesystem layout fulfils the requirements for
relocation and SAP system copy scenarios as well as the capability for performing
SnapShot/FlashCopy backups for the database with IBM Tivoli Storage FlashCopy manager:
Separate Oracle datafiles, online logfiles and other Oracle files into three different
AIX volume groups
Relocate all „non“ datafiles (e.g. Oracle control files) from the AIX volume group for
the Oracle datafiles to a new location
o The SAP DB installation per default stores one control file below
/oracle/<SID>/sapdata1
Example for „new“ control file locations:
/oracle/<SID>/origlogA/cntrl/cntrl<SID>.dbf
/oracle/<SID>/origlogB/cntrl/cntrl<SID>.dbf
/oracle/<SID>/mirrlogA/cntrl/cntrl<SID>.dbf
An additional, new filesystem is required for the redo-logs written during execution of
a SAP System Copy or Clone (for the rollforward recovery of the database). This
filesystem is either exported via NFS and mounted on the target host, or is also Flash-
copied and mounted on the target.
either exported via NFS and gets mounted on the target host, or is also Flash-copied
and mounted on the target.
Filesystem Volume Group Content
vg01 Oracle binaries
/oracle/<SID> vg02 Contains the database directory. In case that regular SnapShot backups are intended for the database, the content needs to be included in the SnapShot image together with the Oracle data files
/oracle/<SID>/oraarch/ vg01 Separate filesystem for Oracle archive logs
/oracle/<SID>/origlogA /oracle/<SID>/origlogB /oracle/<SID>/mirrlogA /oracle/<SID>/mirrlogB
vg03 Separate filesystem for DB2 online logs: In case SnapShot backups are regularly taken and should be used for a fast restore, this filesystem *must* be separated from database data.
/oracle/<SID>/saptrace vg01
/oracle/<SID>/sapreorg vg01
/oracle/<SID>/sapdataX vg02 Contains the database containers/storage paths
/home/<sid>adm vg00 Filesystem for home directory of SAP admin user
/sapmnt/<SID> vg00 Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source
/usr/sap/<SID> vg00 Filesystem for SAP Instance directories
/usr/sap/DAA/SMDA97 vg00 Filesystem for SMD agent
Table 14 Extended filesystem layout (Oracle)
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 72 of 156
If FlashCopy backups for the database are a requirement, then a layout similar to the second
example has to be implemented.
3.4.5.1.2.1 Filesystem for Oracle Redo Logs for System Clone and System Copy
During the SAP LVM Step “Clone Volumes with DB backup mode” (execution of the storage
based copy), the Oracle database is set into “begin backup” mode. For the duration of this
mode data of changed database blocks (with full “before” and “after” information) is written
to the Oracle database logs.
This redo log information is required on the target system to start the database clone.
When the storage based copy is complete, SAP LVM copies
an Oracle DB configuration file
a backup of the Oracle control file
all (archived) logs written during the time frame “BEGIN BACKUP” until “END
BACKUP”
to the filesystem for Oracle Redo Logs.
The filesystem is then accessed on the target system for the Oracle rollforward recovery to the
last committed transaction of the source system.
The filesystem is part of the SAP system configuration in LVM:
Figure 18 Filesystem for Oracle Redo Logs
If the filesystem is shared via NFS, the following settings need to be maintained:
Usage Type Oracle Redo Logs for Online Clone
Storage Type NETFS
Mount Point (1) JFS2 filesystem of „cross-mount“
(2) n/a (but needs to be specified, e.g. /tmp/VCM)
Export Path NFS exported fs (<NFS Server>:<NFS export> )
Mount Options hard mount, background, interrupt, read and write size 32768
(The settings are Oracle requirements for a BACKUP CONTROLFILE
command to a NFS-mounted filesystem)
In case of NFS sharing, SAP LVM will mount the exported directory to
/tmp/VCM/<TGT_SID> on the hosts.
A stanza for /tmp/VCM/<TGT_SID> is required (with all mount options) in
/etc/filesystems
The NFS export needs to allow „root“ access for both the source and target host (no
„root squash“)
If multiple NFS client addresses (e.g. virtual IP addresses of DB and CI) are present in
the sub-network for the NFS server access, then all of them need to be granted access
in the export list on the NFS server
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 73 of 156
3.4.5.2 Install a new SAP system
For the SAP system installation, select one host of the SAP LVM environment that has no
active SAP instances.
The filesystem layout needs to be compliant to chapter 3.4.5.1
For the SAP system installation the virtual IP addresses for the database instance and
the central instance/central services need to be active. Add the virtual IP addresses to
the network adapter with the “ifconfig” command. These addresses must not be
defined as persistent, as the virtual IP addresses for the SAP instances later on will be
managed exclusively by SAP LVM during the prepare/unprepare workflows.
The ifconfig syntax to define an alias on an existing interface is:
# ifconfig <interface> alias <virtual IP> netmask <netmask>
For example: # ifconfig en0 alias c01scs netmask 255.255.255.0
# ifconfig en0 alias c01dbs netmask 255.255.255.0
During the SAP system installation, use the SAPINST_USE_HOSTNAME
parameter to specify the virtual hostname for the instance to be installed.
Users and Groups should be defined centrally in the LDAP environment. As SAPinst
or Software Provisioning Manager (SWPM) currently cannot handle LDAP users
properly, create users and groups for SAP and DB administration manually before the
SAP system installation.
The detailed installation procedure needs to follow all the steps described in the SAP
installation documentation and the respective SAP notes.
3.4.5.3 Adapt an already existing SAP system
File System Layout
The filesystem layout needs to be compliant to chapter 3.4.5.1. It may be necessary to
copy or move the content of the standard installation directories into a SAP LVM
compliant storage layout.
Virtual IP Address and Hostname
Within the SAP LVM environment the database server and the central instance/central
services can be operated on separate hosts. Two different virtual IP addresses are
required for them. If one of the virtual IP addresses was defined as base address of a
network adapter, redefine the network adapter to a new address. Define the virtual IP
addresses as IP alias to the network adapter using the ifconfig command:
If the hostname for the SAP instance is changed to a new virtual hostname, make sure
to adapt all occurrences of the name in the filenames and contents of DEFAULT,
START, and instance profiles. Create and update the following parameters in the
instance profile to the virtual hostname too.
SAPLOCALHOST = <virtual hostname CI>
SAPLOCALHOSTFULL = <full qualified virtual hostname CI>
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 74 of 156
icm/host_name_full = <full qualified virtual hostname CI>
For example, in the default profile DEFAULT.PFL, the SAPDBHOST has to reflect
the virtual IP label of the DB server, and the virtual IP label of the CI will be part of
the mshost, vbname, enqname, btcname, and gateway parameters:
SAPDBHOST = c01dbs
rdisp/mshost = c01scs
rdisp/vbname = c01pas_C01_10
rdisp/enqname = c01pas_C01_10
rdisp/btcname = c01pas_C01_10
Dependent on your individual environment, you may have to adapt settings in further
interfaces/SAP systems too.
(Re-) create the Users in LDAP
The administrative users for Database and SAP system have to be created with the
same numeric UID in the LDAP directory. If the UID for the users violate uniqueness
within the SAP LVM environment, the ownership of all files needs to be changed to
an unique UID.
Adapt user environment files
During shell startup, several files are sourced to create the proper environment for
SAP admin and DB admin users. During installation time, these files are created
containing the hostname within their name. Rename them in the home directory of
both SAP admin and DB admin user.
# mv .sapenv_<virtual hostname>.sh .sapenv.sh
# mv .apoenv_<virtual hostname>.sh .apoenv.sh
# mv .dbenv_<virtual hostname>.sh .dbenv.sh
# mv .sapenv_<virtual hostname>.csh .sapenv.csh
# mv .apoenv_<virtual hostname>.csh .apoenv.csh
# mv .dbenv_<virtual hostname>.csh .dbenv.csh
You also need to check the shell start profiles (.cshrc) that it sources all required
environment files correctly. Each entry should look like
# RDBMS environment
if ( -e $HOME/.dbenv_`hostname`.csh ) then
source $HOME/.dbenv_`hostname`.csh
else if ( -e $HOME/.dbenv.csh ) then
source $HOME/.dbenv.csh
endif
This means the shell first searches for a file containing the hostname, will not find it
and then uses the corresponding file without the hostname. Sometimes the “else if”
clause is missing and you need to create it manually.
Adapt database configuration files to virtual IP address
o ORACLE: Adapt configuration files tnsnames.ora and listener.ora (ADDRESS =
(COMMUNITY = SAP.WORLD)
(PROTOCOL = TCP)
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 75 of 156
(HOST = c01dbs)
(PORT = 1537)
)
The stanzas in tnsnames.ora and listener.ora have to reflect the virtual
hostname of the DB instance. The communication port should be unique within
the environment.
o DB2 UDB: Update db2cli.ini to the virtual DB server address
; Comment lines start with a semi-colon.
[C01]
Database=C01
Protocol=tcpip
Hostname=c01dbs
Servicename=5912
[COMMON]
Diagpath=/usr/sap/C01/SYS/global/db6/db2dump
The “Hostname” in the db2cli.ini file needs to reflect the virtual hostname of
the DB instance. The service name reflects the communication port and should
be unique within the environment.
Perform a test connection to the database: Start the database (and, in case of DB
ORACLE, the listener process). As user <sid>adm, perform a test connection to the
database using R3trans:
c01adm> R3trans –d –v
This is R3trans version 6.22 (release 720 - 03.06.11 - 17:57:00).
unicode enabled version
R3trans finished (0000).
R3trans should return 0 for a successful DB connection.
3.4.5.4 Additional SAP profile parameters
See also SAP Note 1438774 - New profile parameter system/uuid and system/description:
SAP LVM will verify the system/uuid profile parameter against its configuration prior to
any operation (start, stop, relocate) on any instance of a (cloned) system, to ensure that the
operation is triggered on the correct instances. The system/description profile parameter
contains a verbal description of the system.
The parameter can be added to the SAP system later on also during the LVM configuration
phase. However, the SAP system then needs to be restarted to activate the parameter.
3.4.5.5 SAP license considerations
The license key for an SAP system is checked on the partition on which the message server is
running. If the message server is moved during a relocation event to another host, a license
key for the new environment may be required. Request and import in advance license keys for
all the systems on which the message server may run. The license key depends on the
hardware key: The hardware key is different for different physical servers. (All the LPARs
within one physical IBM Power server have the same hardware key).
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 76 of 156
The license key is checked during the (re-) start of the message server:
If the relocation is initialized virtualization-based (Live-Partition Mobility), then the
message server keeps running in the same state as before.
o Even if there is no license key for the target server available the SAP system
continuous its operation. The attempt to restart the SAP system on the target
server while having no valid license for the hardware key of the target server
would then fail.
If the relocation is initialized adaptive-based (stop and unprepare on the source host/
prepare and start on the target host), then the message server is restarted on the target
host, and a valid license key for the hardware key of the target server is required.
SAP has introduced a new license key feature called Flexible License Mechanism to simplify
the administration of SAP license keys in system landscapes where the message server can
move between several physical servers. With this method the license key is no longer tied to
the hardware key of the message server. The flexible license mechanism uses a separate ID
generator, which creates a unique network ID for each message server. This network ID is
hardware independent and therefore it is possible to move the message server to a different
host and retain the unique ID. It is possible to configure multiple ID generators to eliminate
any potential single point of failure.
Refer to the current SAP NetWeaver help documentation for a detailed description and
instructions of how to set up this new Flexible License Mechanism:
SAP NetWeaver 7.0 EHP2 and SAP NetWeaver 7.3
3.4.6 Install SAP LVM Prerequisites on the Management Node
3.4.6.1 Storage System Connectivity using SMI-S
As of November 2013 SMI-S connectivity between SAP LVM and storage system is
supported for SAN Volume Controller, IBM Storwize V7000, and IBM System Storage
DS8000. SMI-S connectivity is currently not supported for IBM XIV Storage Systems.
Because of simplicity, SMI-S is the preferred communication method between SAP LVM and
storage systems. SMI-S is already preconfigured on SAN Volume Controller and IBM
Storwize V7000stem and can be used with default users. On IBM System Storage DS8000
one has to configure SMI-S. Please refer to the storage system documentation for more
information.
3.4.6.2 XIV XCLI
The XIV CLI is used for the communication of LPARs with the storage system. It needs to be
available on the management as well as on all managed LPARs.
3.4.6.3 SAN Volume Controller Connectivity
SMI-S is the recommended communication method for SAN Volume Controller.
Nevertheless, in the lab setup we still used the SVC CLI. SAP LVM then communicates with
the IBM SVC storage systems directly using “ssh” connectivity.
The communication is initiated from the central host (the host where the “acsd” component of
IBM Tivoli Storage FlashCopy is installed) only. First a ssh key pair has to be created for the
“root” user on the central system. To allow the execution of CLI commands from SAP LVM
the created key pair must not contain a key passphrase. The private key of the key pair is
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 77 of 156
stored in a common directory. This location is referred as certificate later on in the SAP LVM
storage manager configuration (see Figure 27).
Create a RSA keypair for user root and store the keys e.g. /home/sapadm/.ssh:
> ssh-keygen -t rsa
Generating public/private rsa key pair
Enter file to save the key: /home/sapadm/.ssh/id_rsa
Enter passphrase (empty for no passphrase):
Enter passphrase again:
Your identification has been saved in /home/sapadm/.ssh/id_rsa
Your public key has been saved in /home/sapadm/.ssh/id_rsa.pub
An administrative user needs to be created in the SVC system. This user needs to be member
of the IBM System Storage SAN Volume Controller user group "Administrator".
Transfer the public key of the key pair to the IBM SVC and assign it to the administrative
user.2
Perform a connectivity test after the public key is attached to the SVC admin user:
> ssh -i /home/sapadm/.ssh/id_rsa -l <SVC admin user> <SVC hostname>
2 See the IBM documentation IBM System Storage SAN Volume Controller and IBM Storwize V7000 Version
6.3.0 Command-Line Interface User's Guide (GC27-2287-02): Chapter 1. Preparing the SSH client for details.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 78 of 156
4 Initial LVM Configuration
4.1 Engine Settings
The SAP LVM engine settings are normally configured when running the initial configuration
wizard. A number of the settings (for example the default user name or password for the Host
Agents) can be changed again under SetupSettingsEngine.
Figure 19 LVM Engine Settings
4.2 Virtualization Manager Configuration in SAP LVM
The IBM Virtualization Manager settings (IBM Systems Director/ VMControl) are defined on
the InfrastructureVirtualization Managers panel. SAP Note 1728222 – Configuration of
IBM Systems Director VMControl for SAP LVM describes further background information
for this step. The screen shot in Figure 20 shows the start panel. Push the “Add” button to
define a new virtualization manager in SAP LVM.
Figure 20 Virtualization Manager Configuration
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 79 of 156
Select “IBM Systems Director / IBM Flex System Manager Adapter” and push the “Next”
button.
Figure 21 Virtualization Manager: Virtualization Manager Types
On the next panel enter a label for the virtualization manager, the user name, the password of
the “administration” user that was created in chapter 4.9, the URL to access the virtualization
manager, and the monitoring interval (how often SAP LVM tries to retrieve monitoring data
from virtualization manager)
Press “Test Connection” to validate all the settings and then proceed to the next panel.
Figure 22 Virtualization Manager: Virtualization Manager Properties
Verify the data on the summary screen and press the “Save” button to store the configuration.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 80 of 156
Figure 23 Virtualization Manager: Summary
4.3 Storage Manager Configuration
The IBM Storage Manager settings (IBM Tivoli Storage FlashCopy Manager) are defined via
the InfrastructureStorage Managers panel. The start panel is illustrated in Figure 24.
Navigate to this panel, and press “Add” to define a Storage Manager in SAP LVM.
Figure 24 Storage Manager Configuration
In the next screen (Figure 25) the Storage Manager Type can be selected. Select Vendor
“IBM”, Product “IBM Storage Adapter” and click “Next”.
Figure 25 Storage Manager: Storage Manager type
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 81 of 156
In the next panel (Figure 26) the IBM Tivoli Storage FlashCopy Manager related settings are
specified (See also chapter 3.3.2, as a lot of these parameters need to be provided from the
FCM installation):
The “FCM CLI Host Name” field specifies the node on which the IBM Tivoli Storage
FlashCopy Manager Management agent is installed. The management agent (acsd) controls
the backup flow and mediates between the other agents running on the hosts. The
management agent also provides access to the snapshot backup repository, which contains
information about the valid snapshot backups and their relationships to snapshot capable
storage devices. During a system cloning/copy process SAP LVM starts the management
agent on the “FCM CLI Host Name” node.
The “FCM CLI Path” specifies the working directory for the IBM Tivoli Storage FlashCopy
Manager on all of the nodes. This must be the path that was specified during the installation
customization of the IBM Tivoli Storage FlashCopy Manager (FlashCopy backup for generic
applications in “setup_gen.sh”).
The “IBM FCM template” name is the filename of the profile template located in the “FCM
CLI Path”.
Select “AIX” for the platform type.
The “Storage Connection Method” specifies the storage access method from SAP LVM to the
storage system.
Storage System Storage Connection Method
(SAP LVM) FlashCopy Manager communication to Storage
SAN Volume Controller (V7000), or Storwize V7000
SMI-S (recommended) or Native CLI (SVC CLI via ssh)
SMI-S
IBM XIV Native CLI (XCLI) XCLI
IBM DS8000 SMI-S SMI-S
Table 15 Storage Connection Methods
Figure 26 Storage Manager: Storage Manager Properties
When all required entries are defined, press “Test Configuration”. SAP LVM will initiate a
communication test to FlashCopy Manager and the storage subsystem CLI. The configuration
test should return “Connection successful: SAP LVM successfully gets data from storage
managers through IBM Tivoli Storage FlashCopy Manager”.
After a successful connection test, select “Next” to get to a user and password configuration
screen. Here you can enter the user credentials for the configured storage system. Dependent
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 82 of 156
on the storage system and the storage connection method you either need to provide a
username and password or the path to a ssh certificate:
Storage System Storage Connection Method
(SAP LVM) User credentials
SAN Volume Controller (V7000), or Storwize V7000
SMI-S Username/ password
Native CLI (SVC CLI) ssh certificate (see 3.4.6.3)
IBM XIV Native CLI (XCLI) Username/ password
IBM DS8000 SMI-S Username/ password
Table 16 Storage Connection user credentials
Figure 27 Storage Manager: Storage Systems
If the user or password is changed in the storage subsystem, you have to update the LVM
Storage Manager configuration too.
Click “Next” and a summary screen appears. Verify the entered settings and click “Save” to
save the configuration.
4.4 Configure SAP LVM Pools
LVM uses pools to separate the landscape components logically and physically. Hosts and
SAP systems are assigned to pools. SAP LVM operations on SAP systems assigned to a
specific pool can only be targeted to hosts that are assigned to the same pool.
Pools are configured on the ConfigurationPools tab. One can define multiple levels of
containers and pools to reflect a hierarchical level of the logical and physical separation of the
landscape entities. The screenshot below shows a container “AIX Pools” groups all pools with
AIX based systems and hosts together. On the next level we separated the pools according to
some physical attributes, all systems and hosts assigned to one pool are allocated or have
physical access to the same storage subsystem.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 83 of 156
Figure 28 SAP LVM Pools: Pool Overview
4.5 Configure Networks
SAP LVM requires some network definitions to enable the activation and deactivation of
virtual host names as aliases of existing network interfaces in a host. Navigate to
InfrastructureNetwork ComponentsNetwork and click the “Add” button to define a new
network (
Figure 32).
Figure 29 SAP LVM Network Configuration
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 84 of 156
Enter a name, the subnet mask, and the broadcast address of the physical network on the
“Basic Configuration” panel.
Figure 30 SAP LVM Network: Basic Configuration
In the next step, “Advanced Configuration” settings are specified. They include the network
gateway address, DNS Zone, Reverse DNS Zone, and Update IP Range.
The “Update IP Range” can restrict the range for new IP addresses that are provisioned by
SAP LVM if automatic DNS update is configured in the environment. If the "Auto Assign IP
Address" option is selected in the "Host Names" step during a clone or copy operation, a
random free IP address is selected from within this range. If an IP address is specified
manually, it must still be within this range.
Figure 31 SAP LVM Network: Advanced Configuration
Click “Next” to proceed to the Summary panel, verify the configuration and push the “Save”
button to store the network configuration.
Figure 32 SAP LVM Network: Summary
4.6 Discovery of Hosts (LPARs)
To discover a new host in SAP LVM, the SAP Host Agent must be installed and running on
the target partition. Select ConfigurationHostsDiscover to get to the “Find
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 85 of 156
Instances/Hosts” panel. There are various options to perform a host detection, in this example
we select “Detect Using Host and Instance Agent” as Source and “Hosts only” under Find.
Figure 33 SAP LVM hosts: Discover Using Host and Instance Agent
Enter the host name of the partition you want to discover. You can use the default credentials
for the SAP Host Agent user id as configured in the engine settings or choose other
authentication types like specific user/password settings or a X.509 client certificate. Click the
Detect button and you should get a message that new hosts are found.
On the next panel select the pool you want to assign this partition to and click the next button.
Click the Save button on the summary screen to add this newly discovered host.
Figure 34 SAP LVM Hosts: Add Hosts
The next step is to change some properties of this new host. Select the host on the “Hosts
Overview” panel and click “Edit”. The “Basic configuration” step should already have the
correct settings, click Next to get to the “Host properties” panel.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 86 of 156
Figure 35 SAP LVM Hosts: Edit host properties
Select the AC-Enabled flag and add a network interface that is used to bind the virtual IP
address of a SAP instance. You also have to provide a capacity figure (SAPS Provided) for
this host. If the partition is used as a target host for cloning and system copy operations, the
“Isolation Ready” flag has to be selected too. Press the “Save” button to store the changes.
4.7 Instance Configuration/ Discovery in SAP LVM
The discovery for a new system is done in several steps. In the “base discovery” the new SAP
system is discovered and added to the SAP LVM configuration. Then the configuration of the
newly discovered instances is completed. This includes the detailed storage configuration for
database and for the central instance.
4.7.1.1 Discovery of a New SAP System
The base discovery of new SAP systems is started on the “Systems” tab in the
“Configuration” view by using the “Discover” button (Figure 36):
Figure 36 SAP LVM Systems: Discover new SAP instances
The new system needs to meet the following prerequisites:
the host is already discovered in SAP LVM
the SAP Host Agent is installed and active on the host
the SAP system is installed in an “adaptive” configuration
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 87 of 156
o virtual hostnames are defined for both database and central instance
o the storage layout supports separation of database and central instance
the SAP system is active
all the user credentials (e.g. <sid>adm) and their passwords are known to the SAP
LVM admin
For older SAP releases (without having enhanced security measures implemented) the SAP
Host Agent is able to discover both database and SAP instances running on the host. For
recent SAP releases the SAP Host Agent may not be able to detect the SAP instance. Due to
enhanced security settings, the “sapadm” user is not authorized any more to retrieve all the
instance parameters. Discovery of the SAP instance requires to contact the instance agent
using the <sid>adm credentials to retrieve all relevant instance data.
Specify the connection data to the SAP Host Agent (IP address of the host), as well as the
connection details to the instance agent (Figure 37):
Figure 37 SAP LVM Systems: Find Instances/Hosts
The discovery is then started by clicking the “Detect Button”. The database and SAP
instances (central services, application server, diagnostic agent) are identified. Push the
“Next” button to go to the “Add instances” view.
On this screen, the instances are assigned to the appropriate pool.
Verify that the “Host Name” field for each instance matches with the desired virtual name. If
an incorrect host name is suggested, select the action “Modify Host Names”, select the
instance to be changed, enter the correct host name in the “New Host Name” field and click
the “Modify” button to perform the change.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 88 of 156
Figure 38 SAP LVM Systems: Add Instances – Modify Host Names
Select action “Auto Assign to New or Existing Systems” once the host names are verified.
Select all instances and choose a target pool using the selection menu in the “Pool for New
Systems”. Click the “Auto Assign” button to assign a System name and the pool to each
instance. Auto assign will generate a new system name for this system
Then click the “Next” button to go to the “Summary” screen and “Save” to save the
configuration.
Figure 39 SAP LVM Systems: Add Instances – Auto Assign to New or Existing System
The base system discovery is now complete. The next steps are to configure additional details
for the SAP system, database, and SAP instances. These additional configuration steps are
required to enable the system for SAP LVM.
4.7.1.2 System Configuration
Select the new system in the “Systems” view and switch to “Edit” mode. On the “Basic
Configuration” screen one can enter a description for the new system, assign a Solution
Manager System and enable E-mail and custom notifications.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 89 of 156
Figure 40 SAP LVM Systems: Basic Configuration
Click the “Next” button to go to the “Provisioning & RFC” screen. On this panel, one can
enable the flags for various provisioning scenarios.
If either one of the copying or renaming flags are enabled, a RFC destination (target client(s)
for post copy automation) needs to be defined. The customization panel for a RFC destination
will open automatically by checking the “Configure RFC Destination” flag.
Use the “Test Connection” buttons to verify the data before moving to the next panel with the
“Next” button.
Figure 41 SAP LVM Systems: Provisioning & RFC
ACM was not evaluated in the test installation; use the “Next” button again to go to the next
“Network Isolation” panel.
Network Isolation
In the network isolation step, all network connections which are still allowed in a “Network
fenced” state are configured. The settings can still be changed during the SAP System
Copy/System cloning definition steps. The settings defined here will appear as default
allowed connections during the System Copy / System Cloning step (Figure 42).
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 90 of 156
Figure 42 SAP LVM Systems: Network Isolation
In the following example, the network connections to “siccserv2” (DNS server for test
environment and the “ssh” protocol were explicitly added as allowed default to the network
fencing configuration.
Figure 43 SAP LVM Systems: Network Isolation - Add allowed communications
Push the “Save” button to save the configuration.
4.7.1.3 DB Configuration
The next step is to configure the detail settings for the SAP database instance. Expand the new
SAP system in the “Systems” tab and select the database.
Press the “Edit” button to open the database configuration panel.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 91 of 156
Figure 44 Database Configuration: Instance Details
Basic Configuration
Keep (or change) the settings in the basic configuration view (Figure 45) and then use the
“Next” button to proceed to the “Instance properties” panel.
Figure 45 Database Configuration: Basic configuration
Instance Properties
Set the “AC-Enabled” flag in the Adaptive Enablement section of the screen (Figure 46) and
select the appropriate network. In the “Requirements” section of the screen enter the required
SAPS and Memory figures for the system. These attributes will be used during Start- and
Relocate actions to determine an appropriate target host candidate for the system. The
definition of the target host needs to satisfy the conditions defined in this section. Select the
allowed Operating System levels for the system and proceed with the “Next” button.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 92 of 156
Figure 46 Database Configuration: Instance properties
Mount Points
In this step, the storage configuration for the system is defined. This step depends on the
storage provider. For the IBM storage environment, some data needs to be gathered on the
host.
After un-checking the “OS Managed Mounts /Automounter)” flag, the screen changes its
appearance and one can push the “Retrieve Mount List” to gather the currently mounted file
systems from the host.
During a prepare operation the file systems are mounted in the sequence as defined in this
panel. Remove the file systems that are not relevant for the database instance and make sure
that the file systems are defined in the correct order.
Push the “Save” button to store the database configuration.
Figure 47 Database Configuration: Mount points
4.7.1.4 Configuration of SAP Central Services Instance
After the configuration of the database instance is done, the next step will configure details
for the SAP central services instance. Select the “Central services” instance on the “Systems”
tab and push the “Edit” button. Then follow the same steps as in the database configuration
section to configure the attributes of the central services instance.
This time make sure to choose the appropriate file systems required for the central service
instance on the “Mount points” panel.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 93 of 156
Figure 48 Central Services: Mount points
4.7.1.5 Configuration of SAP Primary Application Server Instance
Follow the same steps once more to configure the primary application server instance for this
system (SAP LVM still uses the term “Central Instance” to designate the primary application
server instance).
At this time all configuration steps for a new system are complete and the system should show
up as operational under OperationsSystems tab.
Figure 49 Overview of systems and instances
4.8 Additional Configuration Steps
4.8.1 Configuration for SAP System Copy
During a SAP System Copy and a SAP System Rename activity SAP LVM starts the SAPInst
rename tool on the target LPAR to change the SAP System ID of the system. SAP LVM
expects that the SAPInst tool is available in an installation folder accessible on the target
LPAR. In the test environment a central NFS server is used as repository, and the directories
are mounted on all LPARs. This information is defined in ConfigurationExtended
SettingsSystem and Provisioning for each different target OS type. Click the “Add” button
to add a new “System Copy” definition.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 94 of 156
Figure 50 System and AS Provisioning Configuration
Select “System Copy” as provisioning type and proceed with the “Next” button.
Figure 51 System Copy Configuration: Release configuration type
On the next screen enter a descriptive name and select the release version and operating
system type.
Figure 52 System Copy Configuration: Basic properties
Enter the path to the System Rename tool and proceed to the Summary panel. Verify the
configuration on the Summary screen and click “Save” to store the configuration.
Figure 53 System Copy Configuration: Installation master
4.9 IBM Flex System Manager Configuration
IBM Flex System Manager is delivered as a pre-installed appliance node for an IBM PureFlex
System. FSM integrates the virtualization management functionality of VMControl, which
simplifies the management of virtual resources (server, storage, network, virtual appliance
images) and pools of virtual resources.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 95 of 156
The following shows the basic steps required to enable VMControl for virtual appliance
image management:
Discover storage and configure storage system pools
Discover servers and configure server system pools
Configure fabric management for SAN switches (a deploy operation might have to
update the SAN zoning)
Discover, request access, and collect full inventories on each object used by
VMcontrol (server pools, storage pools, physical server, VIO servers, NIM server, etc)
Install the IBM System Director Common Agent, the NIM subagent, and common
repository subagent on the NIM server
Create a common repository on the NIM server
Capture or import a virtual appliance
Please refer to the redbook “IBM Systems Director VMControl Implementation Guide on
IBM Power Systems” (http://www.redbooks.ibm.com/abstracts/sg247829.html?Open) and the
VMControl documentation for a detailed description about implementing VMControl and its
image deployment capabilities.
Once VMControl is configured and the Flex System Manager is defined in SAP LVM, a
subset of the available tasks (like capture or deploy an image) can be started directly from the
SAP LVM user interface too.
4.9.1 Setup of IBM System Management Security
It is possible to use IBM Flex System Manager or IBM Systems Director administrator users
for integration with SAP LVM. Some virtualization use cases (OS provisioning and LPAR
relocation using Live Partition Mobility) require administration credentials. For virtualization
monitoring and management, it is also possible to use credentials that are more restricted.
Administrators may allow monitoring and management of particular elements only (e.g. of
certain physical servers, LPARs, operating systems, etc.) and thus restrict the view of SAP
LVM users.
Please check sap note SAP note 1728222 for the latest information about the definition of
virtualization manager users and roles.
Below we describe how to extend IBM Flex System Manager roles and groups for integration
with SAP LVM.
1. Logon to the FSM console and navigate to AdministrationView or modify user
accounts. Switch to the Groups tab and use the Create button to create a new group.
2. Go back to the User tab and create a new user. On the “User groups” panel select the
previously created group.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 96 of 156
Figure 54 IBM Flex System Manager Security: Group membership
3. Go to AdministrationManage Roles, select the SMUser role and use the “Create Like”
button to create a copy of the role, for example create a new copy called LVM_User.
Perform the same step to create a copy of the SMManager role. We named this copy
LVM_Manager.
Figure 55 IBM Flex System Manager Security: Create new roles
4. Select the new LVM_User role, click “Edit” and extend the copied role with following
permissions (select the permission under “Available permissions” and click “Add >” to
move it to “Selected permissions”):
GeneralCreate Group
GeneralEdit… (Description: Provides an editor for changing an existing group)
System Status and HealthMonitors
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 97 of 156
Figure 56 IBM Flex System Manager Security: Modify role permissions
5. Use the same steps again to extend the copied LVM_Manager role with following
permissions:
Power On/OffPower Off Now
Power On/OffPower On
Power On/OffShut Down
Power On/OffShut down and power off
6. Navigate to Plug-insDiscovery ManagerResource Explorer and create two static
resource groups for monitoring and management.
The monitoring group should include physical machines, virtual appliances, storage pools,
virtual servers, and all other resources that should be visible in SAP LVM.
The management group should include only the virtual servers that will be manageable
through SAP LVM.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 98 of 156
Figure 57 IBM Flex System Manager Security: Define static groups
7. Assign the new roles and resource groups directly to the user or the user group created in
step 1 and 2. In this example, we use the user group.
Navigate back to AdministrationView or modify user accounts. Switch to the “Groups”
tab, select the SAP LVM group and click “Assign Role”.
Select the LVM_User role under “User Role” and the “LVM_Monitoring” group under
“Selected resource groups”. Click the “Add >” button to add this combination to the
“Assigned Roles”.
Perform the same step with the copy of the SMManagement role and the
“LVM_Management group”.
Figure 58 IBM Flex System Manager Security: Assign groups and roles to user group
The configured user is now available for the integrating IBM Flex System Manager as a
virtualization manager in SAP LVM.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 99 of 156
5 Detailed Description / Execution of Use Cases
5.1 LVM Standard Edition Use Cases
This section describes the use cases included in the standard version of SAP LVM. Basically
the SAP LVM Standard Edition allows to start, stop, and relocate SAP Systems.
5.1.1 Description of the LVM Standard Edition Use Cases
Following use cases are possible:
Prepare/unprepare SAP instances and systems
Start/stop SAP instances and systems
Relocate SAP instances and systems
Start/stop hosts
Monitoring
5.1.1.1 Prepare/Unprepare SAP Instances and Systems
The Standard Edition of SAP LVM is able to prepare and unprepare SAP systems and
instances. In case of a prepare operation, the SAP instance or system is made ready to be
started on a host. During the prepare phase, the storage volumes belonging to an SAP system
are mapped to the LPAR, the volume groups are imported, the file systems are mounted, the
network isolation on the LPAR is being setup, and the virtual hostnames are assigned to a
network interface.
During an unprepared operation, file systems volumes belonging to a SAP system are
unmounted, the volume groups are exported, the storage volumes are unmapped, and the
network isolation and virtual hostnames are being removed.
Prerequisite is that the SAP instance or system to be unprepared is stopped. This allows to
map and start the SAP system to/on the same host, or to/on another host.
The prepare/unprepare can be run on a SAP instance or on a complete SAP system including
all instances belonging to an SAP system. If the complete SAP system is affected, then the
function is called mass operation.
Prepare/unprepare triggers following calls:
Storage CLI commands in order to attach or detach storage volumes containing SAP
systems to / from certain OS hosts
OS specific storage libraries for handling the storage mapping
OS specific libraries for handling the SAP instance service
OS specific libraries for handling the network setup
5.1.1.2 Start/Stop SAP Systems and Instances
In order to start a SAP instance or system, it needs to be successfully prepared. After a SAP
instance or system has been stopped, it can be unprepared.
The prepare/start and stop/unprepare tasks can be executed as a single workflows.
As for prepare/unprepare mass operations allow to start/stop complete SAP systems versus
single SAP instances.
Start / stop triggers following calls:
OS specific libraries for handling the SAP tasks start / stop
5.1.1.3 Relocate SAP Instances and Systems
The relocation of SAP instances and systems allows to move SAP instances or systems to
another host.
There are two flavors of SAP instance or system relocation:
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 100 of 156
IBM PowerVM based relocation
The complete LPAR including the SAP system and operating system is moved (also
called migrated) from one physical host to another physical host. In that case, LVM
uses IBM PowerVM Live Partition Mobility (LPM). The LPAR and SAP system stay
up and running during the execution of LPM. LPM is used e.g. for hardware
maintenance activities.
Hypervisor based relocation triggers following calls:
o IBM PowerVM specific libraries for handling Live Partition Mobility in the
HMC or Systems Director
SAP LVM relocation
This means that the SAP system is moved from one managed host to another managed
host. This causes a short SAP downtime as the SAP system needs to be stopped on the
source host and started on the target host. The process automates following functions:
o Stop SAP instance or system
o Unprepare SAP instance or system
o Prepare SAP instance or system on the ‘new’ LPAR
o Start SAP instance or system on the ‘new’ LPAR
As for prepare/unprepare or start/stop, mass operations allow to relocate complete
SAP systems versus single SAP instances.
SAP based relocation can be used for operating system maintenance.
SAP based relocation triggers following calls:
o Storage CLI commands in order to attach or detach storage volumes containing
SAP systems to/from certain OS hosts
o OS specific storage libraries for handling the storage mapping
o OS specific libraries for handling the SAP instance service
Figure 59 and Figure 60 illustrate the two relocation scenarios.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 101 of 156
Figure 59 IBM PowerVM Relocate
Figure 60 SAP LVM Relocate
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 102 of 156
5.1.1.4 Start/Stop Hosts
With SAP LVM hosts can be started or stopped.
Caution: Powering off hosts without stopping and unpreparing the SAP systems running
inside the LPAR may cause data loss or data corruption.
Starting or stopping LPARs triggers following calls:
IBM PowerVM specific libraries for handling Live Partition Mobility in the HMC or
Systems Director
5.1.1.5 Monitoring
SAP LVM provides several types and levels of monitoring, e.g. CPU consumption, memory
consumption, dashboard, history data etc.
SAP LVM communicates with the virtualization manager to retrieve monitoring data about
IBM Power physical servers and logical partitions. Optionally, SAP LVM tries to gather CPU
utilization data for discovered elements (physical servers and LPARs). SAP Notes 1728222
(for IBM Systems Director VMControl adapter) and 1728293 (for IBM HMC adapter)
provide configuration details of components involved in this scenario.
Following setups are supported:
Monitoring through IBM Systems Director VMControl adapter
Monitoring through HMC adapter
5.1.2 Executing the LVM Standard Edition Use Cases
5.1.2.1 Prepare/Unprepare SAP Instances and Systems
The following section describes the required steps to prepare a SAP instance. As the
unprepare process is very similar, it will not be shown separately.
As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with
correct storage volumes and mount points.
To prepare a SAP instance:
Log in to SAP LVM and navigate to OperationsSystems. Select the pool in which the SAP
system resides, then the SAP system you want to prepare. Select the instance and assign a
host in the Host Properties field. Click the small arrow in the right lower corner of the button
‘Prepare and Stop’ button and select ‘Prepare‘.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 103 of 156
Figure 61 Prepare SAP Instance
You can monitor the progress in the ‘Logs’ tab (see Figure 63).
5.1.2.2 Start/Stop SAP Systems and Instances
The following section describes the required steps to start a complete SAP system including
the prepare step. As the complete SAP system is started, this is a mass operation.
The stopping process is very similar, it will not be shown separately.
As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with
correct storage volumes and mount points.
To start a complete SAP instance including preparation:
Log in to SAP LVM and navigate to OperationsSystems. Select the pool in which the SAP
system resides, then the SAP system you want to start. Select or enter a host for each instance
to be started and select the operation ‘Mass Start (including prepare if possible)’. Click the
‘Execute’ button to start the operation.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 104 of 156
Figure 62 Start SAP system: Mass Operations
The logs for an operation are available under MonitoringLogs:
Figure 63 Logs of a 'Start SAP system' Operation
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 105 of 156
5.1.2.3 Relocate SAP Instances and Systems
The following section describes the required steps to relocate a complete SAP system using
SAP LVM relocation. That is, only the SAP system (not the host) is moved to another LPAR.
As the complete SAP system is relocated, this is a mass operation.
As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with
correct storage volumes and mount points, and it needs to be running.
To start the relocation of a complete SAP system:
Log in to SAP LVM and navigate to OperationsSystems. Select the pool in which the SAP
system resides, then the SAP system you want to relocate. Select the instances and select a
target host for each instance. Select the operation Select “Mass Relocate All in State
‘Running’’’ and click the ‘Execute’ button.
Figure 64 Relocate SAP System: Mass Operation
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 106 of 156
5.1.2.4 Start/Stop Hosts
The following section describes the required steps to stop a LPAR. As the start process is very
similar, it will not be shown separately.
Make sure that all SAP systems and instances are stopped and unprepared. Otherwise, you
risk a data loss or a database corruption. There are two options to stop a LPAR:
1. OS Shutdown: LVM triggers an operating system shutdown
2. Power Off: LVM just switches off the LPAR
Using the OS Shutdown is recommended. Use Power Off only in case the shutdown is no
longer working, e.g. the LPAR is not reacting to any commands. Power Off will not close
network connections, unmounts file system etc.
To shut down a host:
Log in to SAP LVM and navigate to OperationsHosts. Select the pool in which the host
resides, then the host you want to stop. Select the ‘Virtual Host’ tab, then ‘Deactivate’ and
select Deactivate (OS Shutdown).
Figure 65 Shutdown Host
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 107 of 156
5.1.2.5 Monitoring
The following section describes how to access the different monitoring functions.
Dashboard
The Dashboard is the first screen opened per default when you log in to LVM. It provides
information e.g. regarding SAP systems, physical resources, and virtual resources. By
selecting ‘Add Pool’ you can open a ‘window’ showing an additional subset of information.
Figure 66 SAP LVM Dashboard
Visualization
The ‘Visualization’ tab provides a visual view on the infrastructure and SAP systems and
instances running on the infrastructure.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 108 of 156
Figure 67 SAP LVM Infrastructure Visualization
Performance monitoring
Performance monitoring enables you to display metrics for specific entities (such as SAP
systems, specific instances, or virtual machines) for a specific time period. The historical
monitoring data is fed to automatic capacity management (ACM) to provide response time
data.
To open the performance monitor:
Log in to SAP LVM and navigate to MonitoringPerformance.
Select the pool in which the host resides, then the host you want to stop. Select the ‘Virtual
Host’ tab, then ‘Deactivate’ and select Deactivate (OS Shutdown). Select one or more entities
on the left (for example, application server) and click the metric you want to display (e.g CPU
usage) and the period (e.g. hour).
To add entities to the display of a metric, specify the entity in the Entity field (or choose the
entity using the value selector) and choose Add.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 109 of 156
To add the metric to the dashboard as a pod, click the ‘Add to Dashboard’ button. If you have
added entities to the display of the metric, the metric is displayed on the dashboard only for
the first entity.
Figure 68 SAP LVM Performance Monitoring
5.2 SAP LVM Enterprise Edition Use Cases
The execution of SAP LVM Enterprise edition use cases require that
LVM Enterprise Edition (VCM_ENT.SCA) is deployed to the managing SAP
NetWeaver Java Web AS system.
5.2.1 SAP System Copy Use Cases
The SAP LVM Enterprise Edition Use-Cases include the SAP System Copy use cases
Figure 69 illustrates the three different scenarios for SAP System copies in SAP LVM:
SAP System Clone
Initial SAP System Copy
SAP System Copy Refresh
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 110 of 156
Figure 69 SAP System Copy Scenarios Supported by SAP LVM (Source SAP)
5.2.1.1 SAP System Clone
In the SAP System Clone scenario, an exact duplicate of an existing SAP system is created.
Content and system configuration are 100% identical in all aspects to the original system. To
avoid conflicts, e.g. with interfaces dealing with the original system landscape, the clone SAP
system is fully network isolated from the rest of the environment before the SAP target
system can be started. Therefore, the clone system is established in an isolated networking
environment. AIX IPsec allows restricting the outbound TCP/IP network communication to
hosts/ports which don’t participate in any interface communication of the SAP system.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 111 of 156
SAP System cloning is realized in two different versions:
Storage based System Clone
For storage based cloning the system to be cloned must be “adaptive enabled”: This
includes
o virtual IP addresses for both database and central instance
o Storage Setup separating all the SAP and database entities from the operating
system image. The SAP and database entities must be deployed to storage
supported by the SAP LVM storage adapter.
The target host must match all prerequisites for a relocation target of adaptively
installed systems (e.g. users, services, etc).
Virtualization based
For virtualization based System Cloning the system to be cloned must be installed in
the rootvg of a LPAR that is managed by the SAP LVM virtualization manager, and
which is registered with SAP LVM. In addition, virtualization based cloning is
supported with an “offline database copy” only. During virtualization based System
Cloning, the SAP System is stopped, the LPAR is captured via IBM Systems Director/
VMControl, and is deployed to the new target LPAR afterwards.
5.2.1.2 Initial SAP System Copy
During a SAP System Copy a duplicate of an existing SAP system is created with a different
system name and a unique SAP system ID (SID). The target system is updated with the new
system information, and can run in the same network environment as original/source SAP
system.
As the first step, the system copy involves the creation of a system clone to create the initial
target system. Then this initial target system is modified during subsequent steps to update the
settings according to the new SAP System ID. A special version of SAPInst, “SAPInst
Rename” is used to change the SAP system ID during this process. Additional post
configuration tasks in the SAP System are achieved via the Post Copy Automation (PCA)
tool. PCA provides task lists, with a predefined sequence of configuration tasks to configure
the extensive technical scenarios automatically. In the SAP LVM environment the execution
of a PCA task list is embedded in the overall system copy workflow.
5.2.1.3 SAP System Refresh
Given are two systems 'SAP source' and 'SAP target' which exist and are configured in a
copy-relationship (target is copied from source). SAP LVM then allows to 'refresh' the 'target'
system by:
exporting configuration data from the 'target' system
SAP System Copy for copying the source system to the 'target' system
(same target SID, same hostnames as previous system copy)
importing the formerly exported configuration data into the new 'target' system
The System Refresh feature has the same requirements as the System Copy.
5.2.1.4 SAP System Rename
The SAP System Copy can be executed in two steps also:
Create a SAP System Clone in a first step
Then use this clone later on, and rename it to a new SAP System ID. Like in scenario
5.2.1.2, the SAPInst Rename tool is invoked for the SAP System ID change.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 112 of 156
At first, the SAP system to be renamed needs to be enabled for the SAP System Rename by
executing following steps:
Go to ConfigurationSystem. Select the new cloned System and click the “Edit” button.
In the configuration screen, activate the checkbox “Renaming” and click “Save”.
Figure 70 Enable SAP System for SAP System Rename
5.2.2 SAP ACM (Automatic Capacity Management) Use Cases
The SAP ACM use cases are SAP LVM enterprise use cases too. However, they will not be
discussed in further detail here. Just note that the following considerations need to be taken
into account for the ACM Use-Cases:
Both adaptively installed SAP systems and traditionally installed SAP systems are
supported for ACM
One dialog instance per host is supported
All ACM managed dialog instances must be in the same set of logon groups reserved
for use by ACM.
SAP CCMS must be installed and operational for each managed SAP system.
For ACM-managed VMs in the same LVM pool, all the OS users must be configured
identical and must be defined in the central user store (LDAP).
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 113 of 156
5.2.3 Required Preparation Steps
Table 17 includes an overview about the required initial setup step before a system can be
cloned, copied, refreshed or renamed with SAP LVM. For each of the required steps it
references to the section in this document where the step is described in more detail.
Setup Procedure in SAP LVM
Clone System Copy System Refresh System/ Database
Rename System
Configuring Networks
Required (See chapter 4.5)
Configuring Storage Providers
Required, if the source system is adaptively installed. (See chapter 4.3)
Not Required
Configuring User Management
Not part of the test environment: All users for the target systems had been pre-defined in LDAP
Configuring Name Servers
Not part of the test environment: All virtual hostnames and IP addresses had been pre-defined in DNS.
(See chapter 5.2.4.1 for additional information)
Configuring Remote Function Call Destinations
Not Required Required
(See chapter 5.2.5.2)
Setting the Infrastructure Assignment
Required (See chapter 4.7.1.1 and 4.7.1.2)
Configuring System Rename Configurations
Not Required Required
(See chapter 4.8.1)
Configuring Mount Points
Required, if the source system is adaptively installed (See chapter 4.7.1.3 and 4.7.1.4)
Configuring Java Post Copy Automation
Not Required Not part of the test environment:
SAP ABAP Stack tests only
Configuring a universally unique identifier (UUID)
Required (See chapter 3.4.5.4)
Not Required
Enabling the system for (See chapter 5.2.1.4, Figure 70)
cloning copying refreshing renaming
Table 17 Required Setup Procedures
5.2.4 Execution of a SAP System Clone Operation
The following section describes the required steps for starting the SAP system clone in SAP
LVM. As a prerequisite, a SAP source system need to be installed and discovered in SAP
LVM, and need to be enabled for SAP System cloning in the SAP LVM configuration
A SAP System Clone is started on the “System and AS Provisioning” tab in the
“Provisioning” view. Select the SAP System to be cloned and click the “Clone System”
button.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 114 of 156
Figure 71 Start SAP System Clone
SAP LVM starts the System Cloning dialog. In the first screen, the LVM Pool is shown, and a
short name and a description for the target system need to be specified. Accept the default
settings, and click “Next” to go to the next screen.
Figure 72 SAP System Clone: Basic Data of Target System
In the next screen, target hosts for the SAP system clone are specified:
1. In this example the default value “Use Existing Host” was chosen for the Central
Services instance and the Central instance. SAP LVM takes an existing LPAR out of
the pool of hosts. The instances will be cloned and attached to this already existing
host.
For the Database instance, “Provision of New Host” is selected. Here a new LPAR
will be created as the target host for the SAP System Clone. Once you click “Add”,
SAP LVM calls the virtualization manager (in our case IBM Flex System Manager
VMControl) to deploy the new target host as part of the cloning process.
2. Click “Next” to continue with the next selection screen
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 115 of 156
Figure 73 SAP System Clone: Host Selection of Target System
In the next screen, the virtual host names and the network for the SAP system clone are
defined. The virtual hostname per default is derived from naming conventions. The short
name specified in Figure 72 is inserted into the virtual hostname.
5.2.4.1 Excursus: DNS Name Resolution
DNS Name Server Considerations
In our test landscape all target IP addresses for the “clone” IP labels are predefined in DNS by
an appropriate address range.
Role IP Label (DNS server) IP Address
SAP Source cl6ci01 192.168.165.116
cl6db01 192.168.165.126
SAP Target cl6cloneci01 192.168.165.136
cl6clonedb01 192.168.165.146
Table 18 Examples for IP addresses
SAP Note 1572841 - Setup of name server update of SAP LVM describes an alternative
solution where SAP LVM automatically updates/creates virtual hostnames in the DNS server.
SAP LVM then creates the respective A (forward) and PTR (reverse) lookup records for IPv4
addresses in the DNS server at the runtime during the SAP System Clone operation. During a
SAP System Destroy operation these records previously added are removed. This mechanism
ensures that in the DNS server the new IP addresses are assigned to the new IP label. In the
test environment however all virtual hostnames and IP addresses had been pre-defined in
DNS and the automatic name server update was not tested.
DNS Behaviour on the Target Host
On the target host, things may look slightly different. SAP LVM acquires the new IP
addresses on the target host during the “prepare” operation, e.g.
On the target host:
root> netstat -in
Name Mtu Network Address
en0 1500 link#2 86.1f.b8.54.6b.2
en0 1500 9.153.164 9.153.164.137
en0 1500 192.168.164 192.168.165.146
en0 1500 192.168.164 192.168.165.136
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 116 of 156
However, the name resolution on the target host still states the “source” IP labels. This is
required for SAP system operation of the clone. The clone system is a 100% identical clone of
the source system and the virtual IP labels need to be unchanged on the target host (as e.g. the
virtual IP labels appear in the instance profiles, start scripts, etc.).
On the target host:
root> netstat -in
Name Mtu Network Address
en0 1500 link#2 86.1f.b8.54.6b.2
en0 1500 9.153.164 siccp137
en0 1500 192.168.164 cl6db01
en0 1500 192.168.164 cl6ci01
To achieve this behaviour, SAP LVM adds new entries to the local /etc/hosts file:
On the target host:
root> cat /etc/hosts
…
192.168.165.136 cl6ci01.isicc.de.ibm.com cl6ci01 #SAP-LM: Added to
modify local hostname resolution for cloning DO NOT MODIFY
192.168.165.146 cl6db01.isicc.de.ibm.com cl6db01 #SAP-LM: Added to
modify local hostname resolution for cloning DO NOT MODIFY
With “local” name resolution, the new “target” IP addresses are mapped to the “source” IP
labels. In order to ensure that the mechanism works, it is mandatory that the DNS search order
evaluates the local /etc/hosts first. This ensures that the local host file supersedes the
(external) nameserver. The nameserver gets contacted only in case the record to be resolved is
not found in the local hosts file.
root> cat /etc/netsvc.conf
# order of search: 1. /etc/hosts 2. nameserver
hosts = local,bind4
Back to the selection screen of the SAP system clones, see Figure 74.
1. The network name needs to be selected
2. Click “Next” to continue with the next screen
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 117 of 156
Figure 74 SAP System Clone: Virtual Host Names and Networks
In the next screen, all the storage definitions are made for the SAP system clone.
1. SAP LVM provides proposals based on built in naming conventions for the new
volume names on the storage system, and logical volume names on the operating
system. You can accept the defaults, or overwrite the values based on your own
naming conventions.
2. Selection of “Full Clone” allows to create a new, independent set of target volumes
(which no longer depend on the source volume)
3. SAP LVM also allows specifying the storage pool on the storage systems which
should be used the hold the target system. If you don’t specify it, then LVM will use
the same storage pool where the source system is located.
4. Click “Next” to get to the next selection screen.
Figure 75 SAP System Clone: Storage Volumes
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 118 of 156
On the next screen, specify the database consistency method.
By selecting “Online Copy/Suspend IO Mode”, SAP LVM will place the DB2 UDB database
into “write suspend” state before the FlashCopy/SnapShot is triggered in the storage system.
When the FlashCopy/SnapShot is finished, SAP LVM will initiate a “write resume” to bring
back the database into the operational state.
Figure 76 SAP System Clone: Database consistency
In the next screen, all the port definitions for the network fencing of the SAP system clone are
specified.
The highlighted item under allowed connections is the DNS name server. In our test
environment, the standard /etc/hosts file didn’t include entries for the virtual IP labels. All the
name resolution is done via DNS to the DNS server. So it is important that the LPAR is able
to communicate with the DNS server in a fenced state too. Therefore the DNS server (here:
siccserv2) was added as allowed communication already during the SAP system setup in SAP
LVM.
Click the “Next” button to get to the final confirmation screen.
Figure 77 SAP System Clone: Define Allowed Outgoing Connections for System Isolation
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 119 of 156
The final confirmation screen appears. It is possible to review all the parameters defined in
the steps before.
Click “Start System Cloning” to finally start SAP System Clone execution.
Figure 78 SAP System Clone: Summary
After starting the SAP System Cloning process, the Monitoring screen is launched and the
progress of the SAP System Clone can be tracked.
5.2.5 Execution of a SAP System Copy
The following section describes the required steps for starting the SAP system copy in SAP
LVM. As a prerequisite, a SAP source system needs to be available and discovered in SAP
LVM, and need to be enabled for SAP system copies in the SAP LVM configuration.
5.2.5.1 Preparations for Post Copy Automation (PCA)
Post Copy Automation (PCA) is part of SAP NetWeaver Landscape Virtualization
Management, enterprise edition solution. This section illustrates how to set-up PCA initially.
Several preparation steps before set-up of SAP Post Copy Automation need to be executed in
the source system first: Details are described in the document "ABAP Post Copy Automation
Guide for Enterprise Edition". See SAP NetWeaver Landscape Virtualization Management -
Installation & Upgrade GuidesSAP NetWeaver Landscape Virtualization Management 2.0
and in SAP Note 1614266 - System Copy: Post Copy Automation (PCA) / LVM for further
details.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 120 of 156
The file "Note_Overview.pdf" (which is attached to SAP Note 1614266) lists a set of
additional SAP notes which need to be implemented as a prerequisite for PCA in the source
system.
Dependent on the actual support package stack of the SAP system they may be already
included in the system, and corrections may need to be applied to the SAP system (via
transaction SNOTES). The ABAP program Z_SAP_NOTE_ANALYZER (which is also
attached to SAP Note 1614266) automates this procedure. The SAP system is checked, and
according to its actual support package stack the required SAP notes content is downloaded
directly during execution of the from Z_SAP_NOTE_ANALYZER report, the content is
transferred into the SNOTES notes assistant transaction and added to the queue.
If all the pre-requisites are met, then PCA is installed via the “Post Copy Automation
Installer” (PCAI) automatically into the managed source system.
See also the following notes and the referenced notes that are mentioned in there:
Note 1589145 - Task Manager for Technical Configuration,
Note 1589175 - System Copy: Task Content for Task Manager
Further details are described in chapter 3 and chapter 4 of the “ABAP Post Copy Automation
Guide for Enterprise Edition“.
PCA is part of the enterprise edition of the SAP NetWeaver Landscape Virtualization
Management solution. PCA is shipped with SAP NetWeaver packages, but is deactivated per
default. The License Enabler add-on PCAI_ENT 1.0 activates PCA and is required in the
system to enable the execution of PCA task lists. Install the add-on PCAI_ENT 1.0 before the
first system copy of the system using transaction SAINT. Details are described in chapter 5 of
the “ABAP Post Copy Automation Guide for Enterprise Edition“.
To grant the privileges to execute PCA activities for the administrator (dialog user) the
corresponding roles SAP_BC_STC need to be assigned to all relevant users. This includes the
system administrator user responsible for PCA activities, and the technical users that will be
used via remote access by SAP LVM. Details are described in chapter 7 of the “ABAP Post
Copy Automation Guide for Enterprise Edition“.
PCA provides task lists, with a predefined sequence of configuration tasks to configure
extensive technical scenarios automatically. To prepare the run of these task lists, the ABAP-
based “task manager for technical configuration” is used. In the SAP LVM environment, the
execution of such a task list is induced without calling the task manager for technical
configuration directly. Further details are described in chapter 8 and chapter 9 of the “ABAP
Post Copy Automation Guide for Enterprise Edition“.
5.2.5.2 Add the RFC destinations for PCA to the SAP System configuration
In the systems configuration, define the RFC destinations that will be used for the PCA
execution for the SAP system (see Figure 79).
Define the logon parameters for the main destinations, and click “Test Connection”. An
attempt to connect to the SAP system is made.
Press “Retrieve Clients” and the list will be populated with all clients of the existing system.
Different PCA activities may be executed specific for each client of the SAP system.
Adjust the Users accordingly, and define the passwords.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 121 of 156
Figure 79 Define RFC Destination for PCA Execution
5.2.5.3 Start the SAP System Copy
A SAP System Copy is started in the “System and AS Provisioning” tab in the “Provisioning”
view. Select the SAP system to be copied and click the “Copy System” button.
Figure 80 Start SAP System Copy
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 122 of 156
On the next screen, enter the SAP system ID and master password for the target system.
Both will be used later during the “SAPINST_RENAME” step to adapt the configuration. The
additional description may be adjusted additionally. This will be the description for the new
system in the SAP LVM configuration.
Click the “Next” button to go to the next step.
Figure 81 SAP System Copy: Basic Data of Target System
In the second step the target host(s) for the system copy is defined:
1. In this example the default value “Use Existing Host” was chosen. SAP LVM takes an
existing host out of the pool of hosts. The instances will be cloned and attached to this
already existing host.
2. Alternatively, you may select “Provision of New Host” for one or more instances. In
this case, a new LPAR would be created as the target host for the SAP instance(s).
SAP LVM would call the virtualization manager, here IBM Flex System Manager
VMControl, to deploy the new target host as part of the SAP system copy process.
3. Click the “Next” button to continue with the next selection screen
Figure 82 SAP System Copy: Host Selection of Target System
In the next step, the network configuration for the target system is specified.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 123 of 156
The IP addresses for the target system are proposed according to the naming conventions for
the virtual IP labels (<sid>pasXX, (<sid>scsXX, <sid>dbsXX). The addresses and labels are
either predefined in the DNS (respectively in the local host files of the SAP LVM system and
the involved resources) already.
Otherwise, SAP LVM may directly update the DNS domain name server for the new entries,
however that need to be defined and configured according to SAP Note 1572841 - Setup of
name server update of SAP LVM first.
A network name for the Network needs to be selected by using the drop-down list.
Click “Next” to continue with the next step.
Figure 83 SAP System Copy: Virtual Host Names and Networks
In Step 4, the Instance number(s) for the Instances of the target system are defined. Per
default, the target system will be installed using the same instance numbers as the source
system. Click “Next” to continue.
Figure 84 SAP System Copy: SAP Instance Numbers
In step 5 the storage configuration for the target system is defined. The storage image for the
target system is created as a SnapShot/FlashCopy of all source volumes to the target volumes.
The main structure like the overall number of AIX volume groups and their content cannot be
changed. Names for volume groups, logical volumes and filesystem names can be adjusted
however. The storage pool for the target volumes can be specified.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 124 of 156
Per default, all SAP System IDs in the filesystem names are updated to the new SAP System
ID of the target system.
Figure 85 SAP System Copy: Storage Volumes
In case the storage connection to the IBM SVC is configured to use ssh, a warning message is
shown, as the recommendation is to use SMI-S.
Click “Ignore warnings for the step” and continue with “Next” button.
Figure 86 SAP System Copy: SVC CLI Warning Message
In the next screen, specify the database consistency method.
By selecting “Online Copy/Suspend IO Mode”, SAP LVM will place the DB2 UDB database
into “write suspend” state before the FlashCopy/SnapShot is triggered in the storage system.
When the FlashCopy/SnapShot is finished, SAP LVM will initiate a “write resume” to bring
back the database into operational state.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 125 of 156
Figure 87 SAP System Copy: Database consistency
In step 7 the OS administration users are specified for the target SAP system. If LDAP is used
for central user management, then the users need to be created manually before the SAP
system Copy is started. “sapinst” is currently not able to handle LDAP users correctly.
However, SAP LVM is able to create local users as part of the SAPInst Rename step.
Click “Next” to continue the next step.
Figure 88 SAP System Copy: User and Group Management
In step 8 further additional settings for DB2 schema user and DB2 instance are specified for
the new SAP System ID. For DB2 UDB, the schema user cannot be changed during the
process and must be set identical to the schema user of the source database. The name of the
database instance for the target system is derived from the SAP System ID according to SAP
naming conventions.
The DB2 Security Administrator can be replaced (from db2<src-sid> to db2<tgt-sid>), which
will cause a re-assignment of all DB2 objects to the new security admin.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 126 of 156
Click “Next” to continue.
Figure 89 SAP System Copy: SAPinst Release Configuration
In step 9, the allowed ports/connections for the network fencing step are defined. Additional
ports/hosts allowed for communication can be selected out of the set of “Current connections
on host” from the source system, and then added to the “allowed outgoing connections on the
target hosts”.
To avoid additional manual steps it is best-practise to include all the necessary hosts/ports
required for outbound connections of the system already in the SAP system definition.
Click “Next” to go to the next step.
Figure 90 SAP System Copy: Define Allowed Outgoing Connections for System Isolation
In step 10, the SAP clients for executing the Post-Copy Automation (PCA) steps can be
defined.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 127 of 156
Figure 91 SAP System Copy: Post Copy Automation
All available tasks can be gathered from the source system, and can be selected accordingly.
Click the “Next” button to continue to the next selection screen.
Figure 92 SAP System Copy: Select Task List
Before actually starting the SAP System Copy, a summary screen is displayed. All the
parameter settings can be reviewed by expanding the different subsections, and eventually
adjust them once again.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 128 of 156
Figure 93 SAP System Copy: Summary View
Click “Start System Copy” to finally start the SAP System Copy. The Monitoring screen is
launched and the progress of the system copy can be tracked.
5.2.6 Execution of SAP System Refresh
Prerequisite for the SAP System Refresh is a SAP System Copy run that was previously
completed successfully.
The execution of the SAP System Rename is very similar to the steps before. Start with the
Provisioning view. Click the “Refresh System” button to start the parameter dialog.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 129 of 156
Figure 94 SAP System Refresh: Start System Refresh
In the first screen base configuration parameters can be adjusted.
1. Specify and confirm the Master Password
2. Click “Next” to continue with the next selection screen
Figure 95 SAP System Refresh: Basic Data of System to be Refreshed
Acknowledge the target host selection in the following screen and click “Next” again.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 130 of 156
Figure 96 SAP System Refresh: Host Selection of Target System
Acknowledge the virtual hostnames and network settings and click “Next”.
Figure 97 SAP System Refresh: Virtual Host Names and Networks
In the next screen, all the storage definitions are made for the SAP system clone:
1. SAP LVM provides proposals based on built in naming conventions for the new
Volume names on the storage system, and Logical Volume names on the operating
system. You can accept the defaults, or overwrite the values based on your own
naming conventions.
2. Selection of “Full Clone” allows to create a new, independent set of target volumes
(which no longer depend on the source volume)
3. SAP LVM also allows specifying the storage pool on the storage systems which
should be used the hold the target system. If you don’t specify it, then LVM will use
the same storage pool where the source system is located.
4. Click “Next” to get to the next selection screen.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 131 of 156
Figure 98 SAP System Refresh: Storage Volumes
In the next screen, specify the database consistency method.
By selecting “Online Copy/Suspend IO Mode”, SAP LVM will place the DB2 UDB database
into “write suspend” state before the FlashCopy/SnapShot is triggered in the storage system.
When the FlashCopy/SnapShot is finished, SAP LVM will initiate a “write resume” to bring
back the database into operational state.
Figure 99 SAP System Refresh: Database consistency
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 132 of 156
In the next step, acknowledge the user settings. As it is a SAP system refresh, the users should
already exist.
Click “Next” to continue.
Figure 100 SAP System Refresh: User and Group Management
In the next screen, enter the password for the schema user and click the “Next” button to
proceed to the next step.
Figure 101 SAP System Refresh: SAPinst Release Configuration
In the next screen, acknowledge or adjust the isolation settings during the network fencing.
Click “Next” to continue.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 133 of 156
Figure 102 SAP System Refresh: Define Allowed Outgoing Connection for System Isolation
Adjust or confirm the PCA settings and click “Next” to continue.
Figure 103 SAP System Refresh: Post Copy Automation
In the next screen, select whether to delete storage volumes at the end of the SAP System
refresh.
Chose the default value to not delete volumes, and click “Next” to get to the next selection
screen.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 134 of 156
Figure 104 SAP System Refresh: Storage Volumes
Finally review the settings and start the operation by clicking the “Start System Refresh”
button.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 135 of 156
Figure 105 SAP System Refresh: Summary
5.2.7 Execution of SAP System Rename
The execution of the SAP System Rename is very similar to the steps before. Start with the
Provisioning view. The system needs to be enabled for SAP System Rename. Only then the
appropriate button is shown.
Click “Rename system” to start the parameter dialog.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 136 of 156
Figure 106 Start SAP System Rename
In the following screen, enter the basic data for the SAP System Rename.
1. Enter and confirm the master password
2. Click “Next” to get to the next selection screen
Figure 107 SAP System Rename: Basic Data of Target System
For convenience, the virtual hostname can be changed on the next screen, as the one from the
system clone will not fit to the naming conventions.
1. Then, select a network
2. Click “Next” to proceed
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 137 of 156
Figure 108 SAP System Rename: Virtual Hostnames and Networks
SAP Instance number for the renamed system can be specified or, per default, use the same as
the source system. Click “Next” to continue.
Figure 109 SAP System Rename: SAP Instance numbers
On the next panel, the mount points are adjusted to the new SAP System ID. In most cases
you can accept the default names, except if you specifically want to update certain mount
points. Click “Next” to proceed.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 138 of 156
Figure 110 SAP System Rename: Mount Data
Select the administrative users for the new SAP System.
If you use LDAP, create users and groups manually in LDAP before, as SAPinst is currently
not able to create LDAP users correctly. However, SAPInst Rename is able to create the SAP
users locally in AIX.
If the users already exist in LDAP or locally in the operating system, then the checkbox “User
already exists” is automatically flagged.
Click “Next” to proceed to the following screen.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 139 of 156
Figure 111 SAP System Rename: User and Group Management
In the next step, the settings for the database connect user are specified. For DB2, the connect
user cannot be changed during the System Rename.
1. Specify the password for the connect user
2. Click “Next” to proceed
Figure 112 SAP System Rename: SAPinst Release Configuration
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 140 of 156
The next screen describes the allowed communications during the network fencing phase.
Ensure that all necessary connections (e.g. to the DNS nameserver) are allowed, but that all
potential dangerous communication is prohibited. Click “Next” to proceed to the following
screen.
Figure 113 SAP System Rename: Define Allowed Outgoing Connections for System Isolation
In the next screen, the SAP target clients for Post Copy Automation execution are specified.
In case you want to disable PCA, remove all the potential clients from the list. If the table is
empty, then the PCA won’t be invoked for any client.
Click “Next” to proceed to the final confirmation screen.
Figure 114 SAP System Rename: Post Copy Automation
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 141 of 156
In the final selection screen, the parameters can be review before the actual start. Then click
the “Start System Renaming” button to start the execution.
Figure 115 SAP System Rename: Summary
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 142 of 156
5.2.8 Additional Considerations for SAPInst Rename
See also SAP Note 1619720 - System Rename for SAP Systems based on SAP NetWeaver
for additional information.
In our lab scenario we identified a few issues in the context of SAP system copies. (Some or
all of them may be already fixed with the latest versions of SAP System Rename/SAP Kernel,
but this was not re-tested).
1. The SAP System Copy in SAP LVM procedure doesn’t handle DB2 UDB with
Automatic Storage Management (ASM) correctly.
o During the SAP System Copy, the SAP Kernel executable “brdb6brt” is called
on the source system to generate a parameter file “relocate.template”. The
content of the file is later used on the target system during the “relocatedb”
step: The db2relocatedb tool allows to move the location of an entire DB2
UDB database to the new file system structure (e.g. change of <Source SID>
to <Target SID> in both database and filesystem structure). In case of ASM,
additional parameters are required in the template file:
1. STORAGE_PATH
2. CONT_PATH
Those parameters were not provided from “brdb6brt”
o The ASM directory structure follows the conventions
/db2/<SID>/sapdataX/db2<sid>/NODE0000/<SID>
Not all the <SID> and <sid> entries are adapted by SAP LVM to
the “new” target <SID> / <sid> values of the target system
Resolution: Handled in Event script, see below
2. SAPinst and SAP LVM don’t handle user creation in LDAP correctly
Make sure that the ABAP Connect User on the target host has the
following group sets assigned groups=db<src-sid>mon,db<src-
sid>mnt,db<tgt-sid>mon,db<tgt-sid>mnt
Resolution: Handled in Event script, see below
3. SAPSYSTEM RENAME didn’t handle the update to db2nodes.cfg if a “full
qualified hostname” is set as hostname on the target host
If hostname and netname are specified in db2nodes.cfg, only the hostname
is changed. If only the hostname is specified, then it is translated
erroneously
Resolution: Handled in Event script, see below
4. An issue with “db2chgpath” was identified in the test environment (DB2 UDB
V9.7 FP5SAP)
Resolution: Fixed in DB2 UDB V9.7 FP6 (APAR IC80849).
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 143 of 156
5. Update to AIX 6.1 TL7 SP4 (6100-07-04-1216) introduced a new issue IV22062
on some of the nodes.
Resolution: Install eFIX on top of AIX 6.1 TL7 SP4
root> emgr -l
ID STATE LABEL INSTALL TIME UPDATED BY ABSTRACT
======================================================
1 S IV22062s04 06/11/12 12:42:43 Ifix [email protected]
Item 1.), 2.) and 3.) are addressed by a “Pre-Event” script, which is invoked before SAPInst
rename is started: This is realized via a “Pre-Hook” in SAP LVM. The following outlines how
to implement such a hook. See also SAP Note 1465491 - Provider Implementation Definition
for further details.
5.2.8.1 Create the Script and Register it with the Host Agent
The script is invoked for both SAP Central Instance and Database. It checks, if it is running
for the database, and then identifies the Source SID based on the environment. Target SID is
specified as argument. The script provides the following functionality
Check and eventually create/correct the LDAP users and groups required for the target
system
Handle DB2 UDB ASM
o Update relocate.template to include storage and container pathes in the
template
o Correct directory structure below the /db2/<SID>/sapdataX filesystems
Update db2nodes.cfg (in the directory of the source instance)
When the script is installed, a .conf file needs to be created in the directory
/usr/sap/hostctrl/exe/operations.d so that the Host Agent can start the script.
cat ISICC_PreSAPinst.conf
Name: ISICC_PreSAPinst
Command: /usr/sap/hostctrl/exe/operations.d/IBM/PreSAPinst.ksh
$[SYSTEM_HOST:#required] $[system_sid:#required] $[SERVICE_CLASS:#required]
$[service_name:#required] $[service_type:#required]
Workdir: /usr/sap/hostctrl/exe/operations.d/IBM/traces
ResultConverter: flat
Platform: Unix
In the .conf file, the script and its parameters are defined in the “Command:“ section.
“Workdir:” contains the actual work directory for the script. Make sure that the directory
exists.
Restart the SAP Host Agent to register the script.
5.2.8.2 Define the Provider Implementation Definition
For the Provider Implementation Definition, login to SAP LVM and navigate to
SetupExtensibilityCustom Operations, Hooks, and NotificationsProvider
Implementation Definition. Click the “Add” button to start a new definition.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 144 of 156
Figure 116 Provider Implementation Definition (I)
On the following screen:
1. Enter name of the hook and select Type “Script Registered with Host Agent”
2. The hostname running a SAP Host Agent where the script was registered
3. Click “Retrieve Registered Scripts”
Figure 117 Provider Implementation Definition (II)
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 145 of 156
Now, select the registered script from the pick list.
Figure 118 Provider Implementation Definition (III)
The registered script is a custom hook, and it shall be started for an instance (Database, SAP
Central Instance). Checkmark “Instance” in the row for “Hook”. Afterwards, save the new
definition.
Figure 119 Provider Implementation Definition (IV)
5.2.8.3 Define the Custom Hook Definition
The “Custom Hook Definition” then maps the script defined before to the “Pre-SAPInst”
Event. Change to the Custom Hook Definition Tab (SetupExtensibilityCustom
Operations, Hooks, and NotificationsCustom Hooks), and click “Add”.
Figure 120 Custom Hook Definition (I)
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 146 of 156
On the next panel:
1. Enter a name for the custom hook
2. Select the provider implementation definition defined before from the “Definition
Name” list.
3. Select “Instance” from the “Entity Type” list.
4. Select the operation “SAPinst” from the “Operation” pick list.
Figure 121 Custom Hook Definition (II)
5. Select the hook type “Pre Hook” from the list. The hook is called directly as pre-event
before starting the SAPInst rename step.
Add an “OS Type (static)” constraint in the table at the bottom to ensure the hook runs on
AIX hosts only.
Click “Save” to confirm the changes.
Figure 122 Custom Hook Definition (III)
5.2.9 Destroy a SAP System
During the SAP system destroy action
the storage configuration and all the actual volumes for the systems will be deleted in
the storage system
IP labels, which were created in the DNS server during SAP system creation by SAP
LVM, will be removed from the DNS server once again
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 147 of 156
The SAP system will be deleted in the SAP LVM repository.
All instances for the system needs to be in “unprepared” state before the “Destroy” action can
be started in SAP LVM. If the instances are still running, then the system needs to be stopped,
and an unprepare operation needs to be started for it first.
5.2.9.1 Stop and Unprepare a SAP System
Navigate to the „Operations“ view and check the status of the system. As a prerequisite to
start the “Destroy” operation, all the instances of the system need to be in state “Unprepared”.
In the example below the system is still active. A mass operation to “Stop and Unprepare” is
selected for both the database and the SAP Central Instance.
Figure 123 System State in Operations View
Click “Execute“ to start the mass operation. The instances get locked for other operations, and
the “Stop and Unprepare” operation is started.
The progress of the current action can be monitored in the “Monitoring” view. On the upper
section of the screen a status and progress overview is given. By selecting an operation from
the list, and choosing the “Steps” sheet the individual steps of the workflow can be monitored.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 148 of 156
Figure 124 Monitoring View
Wait until the “Stop and Unprepare“ operation completes for all the instances.
Figure 125 Mass Stop and Unprepare Completed
5.2.9.2 Destroy the SAP System
Navigate to the “Provisioning“ tab and select the system to be destroyed.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 149 of 156
Figure 126 Destroy System
Click „Destroy“ System: three additional selection/information steps are processed before the
final destroy operation is started.
During the destroy operation all the storage volumes which were deployed for the system
during an install/copy/clone process are removed on the storage system. In the first step, all
these storage volumes to be deleted later are shown.
Figure 127 System Destroy: Delete Storage Volumes
Click “Next“ to proceed. If SAP LVM is configured for DNS updates and has generated DNS
entries for the system during a SAP LVM deployment process then these entries will get
removed during the destroy operation. This feature was not configured in the test
environment. Instead all IP addresses and corresponding DNS entries were preconfigured in
the DNS before. So in the example case this list is empty.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 150 of 156
Figure 128 System Destroy: Host Names
Click “Next” to proceed to the summary screen. The parameters are shown for a final review.
Click “Start System Destroy” to start the actual destroy operation.
Figure 129 System Destroy: Summary
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 151 of 156
6 Troubleshooting hints and tricks
6.1 Manual cleanup of SAP LVM copy process
Manual cleanup may be required in case one of the SAP System clone, copy, or refresh steps
fails and the SAP LVM task “Destroy partially prepared system” does not work.
SAP Note 1925060 describes the required cleanup steps. Below we list the actions that
should be performed in case of cleanup. Check the above SAP note 1925060 for updates.
6.1.1 Retrieve information details about the process
On SAP source host
Identify the profile directory of the copy process. The directory is located within FCM
Directory as configured in SAP LVM Storage Manager configuration and begins
with "SAPLVM_<TARGET_HOST>_<SOURCE_HOST>_<STORAGE_ID>_".
On SAP source host
Go to the directory found in the previous step and check if "backup.log" is available.
o Check for lines with the following pattern:
"#SAVED <PREFIX>__<BACKUP_ID> <DIRECTORY>"
(e.g. "#SAVED SAP0__A0HCJ22U1M /usr/sap/L43")
Note the Backup ID.
o In case the last line of backup.log shows "FMM0024I Return code is: 2.", the
backup id is not relevant and the last step of this cleanup procedure can be
skipped.
o In case backup.log is not available, the last step of this cleanup procedure can
be skipped.
In SAP LVM UIMonitoringLogs
Open SAP LVM logs of PreProcessCloneVolumes step and note target volume names
from context.
In SAP LVM UIConfiguration:
o Go to SAP LVM mount configuration for each of the SAP target system
instances.
o Note mount points that should become available on SAP target host after
successful process.
o Check if the values for export path of the mount points are not empty. If the
export path contain comma separated IDs, note them.
6.1.2 Cleanup hosts
Stop SAP instances and all instance agents
Stop the SAP Host Agent
Clean up FCM process on SAP copy central management host
o Get corresponding "acsd" process and kill it. "ps -ef | grep acsd" will show a
process that uses profile directory identified in first step. The directory name
will end with "_acsd". (e.g. "kill -9 12345")
o Delete or move profile directory identified in first step. (e.g. "mv
/home/sapadm/acs/SAPLVM_* /usr/sap/hostctrl/exe/operations.d/IBM/traces")
Clean up FCM process on SAP copy source host
o Get corresponding "acsgen" process and kill it. "ps -ef | grep acsgen" will
show a process that uses profile directory identified in first step.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 152 of 156
o Delete or move profile directory identified in first step. (e.g. "mv
/home/sapadm/acs/SAPLVM_* /usr/sap/hostctrl/exe/operations.d/IBM/traces")
Clean up FCM process on SAP copy target host
o Get corresponding "acsgen" process and kill it. "ps -ef | grep acsgen" will
show a process that uses profile directory identified in first step.
o Delete or move profile directory we found previously. (e.g. "mv
/home/sapadm/acs/SAPLVM_* /usr/sap/hostctrl/exe/operations.d/IBM/traces"
Clean up firewall rules on SAP copy target host
o Check if SAP LVM related firewall rules are available. SAP LVM related
firewall rules begin with "SLM" in rule description.
"lsfilt -v4 -O" will show all currently active firewall rules.
Delete all SAP LVM related firewall rules
Delete the "deny" rules first, otherwise the SAP target host may become isolated.
"rmfilt -v4 -n xxx" will delete the firewall rule with ID xxx.
Activate updated firewall rule list afterwards
Use "mkfilt -v4 -u" to activate the changes.
Clean up storage file systems on SAP copy target host
o Check with "df -g" if any of file systems retrieved previously from SAP LVM
mount configuration of SAP target system instances are mounted.
o Unmount all of these file systems (e.g. "umount -f /usr/sap/CPY").
o Get available logical volume group names with "lsvg".
o Identify with "lsvg -l <VGNAME>" all the logical volume groups that include
file systems retrieved previously from SAP LVM mount configuration of SAP
target system instances.
o Execute "varyoffvg <VGNAME>" and "exportvg <VGNAME>".
Prerequisite: all the file systems listed in logical volume groups with "lsvg -l
<VGNAME>" are closed.
o Execute "lspv" and get all the physical disks on the system.
o Remove all the physical disks that do not belong to any logical volume groups
from the operating system by executing "rmdev -d -l hdiskX"
Clean-up /usr/sap/sapservices
Remove all entries set by SAP LVM on the target host
o /etc/hosts
o /etc/services
Restart SAP Host Agent
6.1.3 Cleanup storage
Identify storage volumes and/or snapshots that should be removed
In order to identify storage volumes / snapshots use target volume names and/or
Backup ID and/or IDs from export path in SAP LVM mount configuration retrieved
previously. Target volume names correspond to storage volume / snapshot names,
Backup ID will be a part of snapshot name (on XIV) and IDs from export path in SAP
LVM mount configuration correspond to storage volume / snapshot WWNs (on XIV,
V7000 or on SVC)
Ensure that the storage volumes / snapshots are NOT masked
This process is very specific and depends on the storage subsystem or storage cluster.
Below we describe only actions that must be performed but not the exact process
itself.
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 153 of 156
o Check if storage volumes / snapshots are attached to host alias / port IDs that
represent SAP target host.
o Ensure that you completed clean up storage file systems procedure on SAP
copy target host. Otherwise, you will need to restart SAP target host because of
AIX volume group errors.
o Detach storage volumes / snapshots from host alias / port IDs that represent
SAP target host.
On V7000 storage system or on SVC cluster: remove consistency groups and
FlashCopy relations that include storage volumes to be deleted:
o Remove FlashCopy relations from consistency group
o Remove consistency group
o Bring up FlashCopy relations into a final state (e.g. stop them if "Stop" action
is available)
o Delete FlashCopy relation. Set optional flag "Delete the FlashCopy mapping
even when the data on the target volume is inconsistent, or if the target volume
has other dependencies."
Delete storage volumes and/or snapshots.
6.2 Delete the System in SAP LVM
Go to SAP LVM UIConfigurationSystems
Select the appropriate system, and press “Remove Selected Instances and Systems” (Figure
130).
Confirm the deletion in the following conformation dialog.
Figure 130 Remove System in SAP LVM
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 154 of 156
7 Resources
7.1 IBM Documentation
7.1.1 Whitepapers
Flyer “Cloud-enabled SAP application management”
IBM Cloud Solution for SAP – Integrating IBM Infrastructure with SAP Landscape
Virtualization Management
Introduction to an Integral IBM / SAP Cloud Solution for SAP Landscape Management
IBM entry cloud configuration for SAP solutions on Power Basic Concepts and High Level
Design
7.1.2 Redbooks / Technotes
Deploying Cloud Components on POWER
IBM Flex System Manager
IBM PureFlex System and IBM Flex System Products and Technology
IBM Flex System p260 and p460 Compute Node
7.1.3 Product Documentation
Requirements and support for AIX using Network Installation Manager (NIM)
Requirements and support for AIX, IBM i, and Linux using storage copy services (SCS)
7.1.4 iRAM
IBM Systems Director VMControl Implementation Perform Guide
7.2 SAP Documentation and SAP Notes
7.2.1 SAP Documentation
SAP NetWeaver Landscape Virtualization Management
SAP Community Network (SCN) Virtualization and Cloud SAP Help Portal (SAP LVM)
SAP NetWeaver Landscape Virtualization Management, Enterprise
SAP NetWeaver Landscape Virtualization Management at a Glance
7.2.2 SAP Notes
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 155 of 156
SAP Note Title / Description
Note 962955 Use of virtual TCP/IP host names
Note 1292836 Logfiles of the SAP Hostagent
Note 1438774 New profile parameter system/uuid and system/description
Note 1465491 Provider Implementation Definition
Note 1527538 SAP NetWeaver Landscape Virtualization Management 1.0
Note 1572841 Setup of name server update of SAP LVM
Note 1589145 Task Manager for Technical Configuration
Note 1589175 System Copy: Task Content for Task Manager
Note 1614266 System Copy: Post Copy Automation (PCA) / LVM 1.0
Note 1619720 System Rename for SAP Systems based on SAP NetWeaver
Note 1644520 Integration of IBM Components into SAP LVM
Note 1728222 Configuration of IBM Systems Director VMControl for SAP LVM
Note 1728293 Configuration of IBM Power HMC for SAP LVM
Table 19 List of SAP Notes
IBM SAP International Competence Center
SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 Page 156 of 156
COPYRIGHT LICENSE
If this information contains sample application programs in source language, which illustrate
programming techniques on various operating platforms, you may copy, modify, and
distribute these sample programs in any form without payment to IBM, for the purposes of
developing, using, marketing or distributing application programs conforming to the
application programming interface for the operating platform for which the sample programs
are written. These examples have not been thoroughly tested under all conditions. IBM,
therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.
The information contained in this document has not been submitted to any formal IBM test
and is provided "AS IS" with no warranties or guarantees either expressed or implied.
All examples cited or described in this document are presented as illustrations of the manner
in which some IBM products can be used and the results that may be achieved. Actual
environmental costs and performance characteristics will vary depending on individual client
configurations and conditions.
IBM is not responsible for printing errors in this document that result in pricing or
information inaccuracies.
A full list of U.S. trademarks owned by IBM may be found at:
http://www.ibm.com/legal/copytrade.shtml
A full list of trademarks owned by SAP may be found at:
http://www.sap.com/company/legal/copyright/trademark.epx
SAP and other SAP products and services mentioned herein, as well as their respective logos,
are trademarks or registered trademarks of SAP AG in Germany and in several other countries
all over the world.
UNIX is a registered trademark in the United States, other countries or both.
Linux is a registered trademark of Linus Torvalds in the United States, other countries or
both.
Microsoft, Windows, Windows NT and the Windows logo are trademarks of Microsoft
Corporation in the United States, other countries or both.