18.08 Upgrade Experience

REMEDY ITSM/ SMART-IT

Carin SinclairPremier Support

Introduction...........................................................................................................................................3

Upgrade path.........................................................................................................................................3

18.08 Enhancements.............................................................................................................................6

Pre- Upgrade Steps................................................................................................................................6

Running Platform Installers -ARServer/CMDB/AI..............................................................................8

Secondary Server Installs.................................................................................................................11

Platform Tuning...................................................................................................................................11

Mid-Tier...............................................................................................................................................13

Mid-Tier Tomcat Tuning & SSL.............................................................................................................14

ITSM....................................................................................................................................................15

Service Request Management.............................................................................................................19

Service Level Management..................................................................................................................19

ITSM/SRM/SLM Post installation.........................................................................................................21

Deployment Console...........................................................................................................................22

Smart-IT...............................................................................................................................................23

Pre –Upgrade tasks..........................................................................................................................23

Post-Upgrade tasks..........................................................................................................................25

Run the migration utilities...............................................................................................................25

Migrating social data from MongoDB to the AR System database..................................................25

Stop and delete services..................................................................................................................25

Verify the Activity feeds...................................................................................................................26

Smart Reporting..................................................................................................................................26

Remedy Single Sign-On........................................................................................................................26

Zero Downtime Upgrade.............................................................................................................26

Migrate and validate Customizations..................................................................................................27

Apply all Patches and Hotfixes.............................................................................................................27

Known Errors.......................................................................................................................................27

IntroductionThis document is intended to give you an overall idea of the upgrade process. It is not a replacement for existing BMC Install documentation.

Upgrade path

Based on conversations between customer and BMC the following path is recommended for <Customer Name>

In Place upgrade

Reasons for this approach Reduce Risk of introducing mixed version issues with code line and Reduce impact/ outage to end customer Limit different lower environments to support a mixed release/swim-line support path

Components to be Upgrade and Timelines as identified in lower environment

Assumptions: Since Production database copy cannot be restored to lower lanes the assumption is that at

least code line is a match to production Code freeze is in place during lower lane upgrade process

Prerequisites 1. Project Plan for both lower environment and Production Upgrade- <customer name>2. Clear User Acceptance Test Cases with Business sign off - <customer name>3. Performance Testing plan - <customer name>4. Integration Testing with external and internal vendors - <customer name>5. Code Freeze communicated to business from start of Dev Upgrade - <customer name>6.

Upgrade Risks1. Customers need to be trained on new UI changes2. System will be unavailable for duration of Upgrade/ or portion 3. Some integrations cannot be validated in lower environment and will l need to be address

in After Prod window4. Some cases could might only be identified after Upgrade in Prod by end user

ComponentFoundation

MidTier 18.08ARServer 18.08CMDB 18.08Atrium Integrator 18.08

Applications

ITSM 18.05SRM 18.05SLM 18.05ITSM 18.08 SRM 18.08SLM 18.08

Smart-IT 18.08 DWP 18.05

DWP 18.08Smart Reporting 18.08RSSO 18.08

The following is basic process to complete

Component Duration NotesMid-Tier 18.08

Foundation Elements on Primary Server - ARSERVER- CMDB- AIDatabase backup / OS SnapshotFoundation Elements on all Secondary Servers- ARSERVER- CMDB- AI

This needs to be completed as ITSM will not install if Foundation tasks is not complete

Application Elements on Primary- ISTM 18.05- SRM 18.05- SLM 18.05

Suggestion to take additional backup after ITSM

Database backup / OS SnapshotApplication Elements on all Secondary- ISTM 18.05- SRM 18.05- SLM 18.05

Can be run in staged parallel since it is only binaries

Application Elements using D2P- ISTM 18.08- SRM 18.08- SLM 18.08

Primary and Secondary will be done with same deployment package and can be managed using the payload utility

Database backup / OS SnapshotSmart-IT 18.08 Smart-It will require new install

path and new database tablesDatabase backup / OS SnapshotDWP 18.05Database backup / OS SnapshotDWP 18.08RSSO 18.08Database backup / OS Snapshot18.08 Patch 001 deployment using D2P on all servers- ARSERVER- MT- CMDB- AI- ITSM- SLM

- SRM- Smart-IT- DWP18.08 Patch 001 latest hotfix or point fix deployment using D2P on all servers- ARSERVER- MT- CMDB- AI- ITSM- SLM- SRM- Smart-ITDWP

Note not all components will have hotfixes or point fixes- this will depend on GA release or customer testing and escalations

Smart Reporting 18.08 RSSO client deployments to all serversSmart-IT tablespace migration OptionalSmart-IT mongo dB migration OptionalDWP - mongo db migration Optional

Code Migration and Verification3 way reconciliationCustom code verificationPackage for deployment to prod

ValidationFunctional testing Sign offIntegration testing Sign offPerformance Testing Sign off

Perform the following steps:

Upgrading BMC Remedy AR System

Verify and complete the AR System upgrade

Completing the BMC Remedy ITSM upgrade Upgrade ITSM, SRM, SLM to 18.05:Apply the latest deployment packages of ITSM, SRM, SLM 18.08:

Upgrade to Smart IT 18.08:(Pre-Upgrade)(Optional) Run the social data migration utility for Smart IT.Verify social data migrationUpgrade to Smart IT 18.08.Verify the Smart IT upgrade

(Post-upgrade)(Optional) Run the migration utilities and perform the post-upgrade tasks.Upgrade Android and iOS mobile clients for Smart IT to 18.08.

Installing BMC Digital Workplace 18.08 .

NOTE: The Smart IT 18.08 installer provides you the option to upgrade only Smart-IT MUST have ITSM 18.08.The Smart IT 18.08 installer does not have an option to upgrade BMC Digital Workplace.

18.08 Enhancements

There have been many ITSM Suite enhancements which are documented here.

https://docs.bmc.com/docs/brid/1805/tuning-the-sql-server-database-805357823.html

Pre- Upgrade Steps

Run Configuration Checker Set JAVA_HOME =Open cmd windowRun configchecker-ui.bat

Click Run

When progress bar is complete

Select HTML Report to view and validate output

Running Platform Installers -ARServer/CMDB/AIMake sure you have at least 120GB Disk Space free

Clear out all unnecessary Log fileshttps://communities.bmc.com/docs/DOC-108153

Disable Object Modification Log

Tail log via PowerShell

Get-Content “<tempdir>installog.txt” -Wait -Tail 10

If you receive following error and you are on Admin ranked server

Verify server name in installer is same as server name in operational Ranking form

Recheck Configuration Checker

Note: Unselect Webservices and Registry

If Webservices was previously installed

Bmcuddi directory under /AriumCore/wsc must be removed prior to running upgrade

Verify Configuration Check

Secondary Server Installs

Run Installers for ARServer, CMDB and AI on all remaining ARServers to upgrade Foundation Stack

ALL Foundation Components need to be upgraded before starting on ITSM stack

Platform Tuning

Check the guidelines from the following links:

CONFIGURATION CHECKLIST from the BMC R&D PERFORMANCE TEAM

https://docs.bmc.com/docs/brid/1805/overall-system-tuning-805357748.html

Some general observations:

a) Check JAVA HOME is correct <installdir>/ARSystem/bin/arserverd.conf)

# JVM search paths (number indicates search order)jvm.search.path.1=FILE_SYSTEM=<JAVA_Instalpath>/jdk/jdk1.8.0_172/jre/bin

b) Update the “arserverd.conf” file.

Increase JVM from# JVM minimum heap sizejvm.minimum.heap.size=8589934592

# JVM maximum heap sizejvm.maximum.heap.size=8589934592

To 32GB for Indexing ServersTo 24GB for all other servers

Verify -XX:+DisableExplicitGC-XX:MaxMetaspaceSize=512m-XX:MetaspaceSize=384m-XX:+UseCompressedOops-XX:+UseConcMarkSweepGC-XX:+UseParNewGC-XX:NewRatio=2

c) ARmonitor

Check paths in the “armonitor are correct.

d) AR.conf

Added IP/Names to each of the Server Added “Next-ID-Block-Size: 100” Added “Server-Side-Table-Chunk-Size: 1000”

e) Pluginserver_config.xml (ARSystem/pluginsvr/pluginsvr_config.xml)

<numCoreThreads>30</numCoreThreads><numSelectorThreads>2</numSelectorThreads>

Create arsys_api.xml in the current plug-in server directory with the following content.

<?xml version="1.0" encoding="UTF-8" ?><javaapi_config> <maxProxiesPerServer>80</maxProxiesPerServer> </javaapi_config

f) Pluginserver_config.xml(…cmdb/plugins/shared/pluginsvr_config.xml)

<numCoreThreads>10</numCoreThreads><numSelectorThreads>2</numSelectorThreads

Updated the arsys_api.xml

<?xml version="1.0" encoding="UTF-8" ?> <javaapi_config> <maxProxiesPerServer>80</maxProxiesPerServer></javaapi_config>

g) Pluginserver_config.xml (..cmdb/plugins/ne/pluginsvr_config.xml)

<numCoreThreads>5</numCoreThreads> <numSelectorThreads>2</numSelectorThreads>

h) Create arsys_api.xml in the current plug-in server directory with the following content:

<?xml version="1.0" encoding="UTF-8" ?> <javaapi_config><maxProxiesPerServer>80</maxProxiesPerServer> </javaapi_config>

Mid-Tier

https://docs.bmc.com/docs/display/ars1808/Issues+table

Once the installation is successful, repeat on the secondary Mid-Tier Server and configure for clustering and proceed to check load balancer settings with your network team.

https://docs.bmc.com/docs/display/ars1808/Configuring+a+cluster

Mid-Tier Tomcat Tuning & SSLI used the following references to tune tomcat:CONFIGURATION CHECKLIST from the BMC R&D PERFORMANCE TEAMTuning the Remedy Mid-TierSome general settings:Tomcat/conf/server.xml

connectionTimeout="60000"

Tomcat/conf/logging.properties

Change all INFO to SEVERE.

Setenv.sh (tomcat/bin directory)export CATALINA_OPTS="-Xmx8192m -Xms8192m -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:MaxMetaspaceSize=512m -XX:NewRatio=2"MIDTIER/WEB-INF/classes/config.propertiesarsystem.pooling_max_connections_per_server=350

Configure SSL as per the following instructions:Configuring the Mid-Tier web server for SSL certificate

ITSMEnsure All Platform components in Server Group has been upgraded IF NOT You will receive the following

IF all the Platform components have been upgrade do the following Access Server Information Tab

NOTE: Server Group Upgrade Status is still: PendingAccess Central Configuration Select Component com.bmc.arsys.server.sharedsharedAdd entry Setting Name: Upgrade-Status Value: Pending Complete

Validate Configuration Checker

https://docs.bmc.com/docs/brid1808/bmc-remedy-itsm-checks-817165646.html

NOTE INSTALLER Completed with Warnings

Check folder C:\Program Files\BMC Software\BMCRemedyITSMSuite\Logs

Once successful, run the installer on the remaining servers making sure to flush the Mid-Tier cache.

Service Request Management

Service Level Management

ITSM/SRM/SLM Post installation

Run post install check from “/opt/bmc/BMCRemedyITSMSuite/Utilities”

Post install configuration

Run through standard configuration tasks

https://docs.bmc.com/docs/display/itsm1808/Configuring+Service+Desk+Optimization

Synchronise people information

https://docs.bmc.com/docs/display/itsm1808/Synchronizing+people+information+for+BMC+Remedy+ITSM+Suite

Create a Private queue for CAI

https://docs.bmc.com/docs/display/itsm1808/Generating+a+private+server+queuehttps://docs.bmc.com/docs/display/itsm1808/Configuring+the+CAIPluginRegistry+form

Change escalation Server thread counts

https://docs.bmc.com/docs/display/itsm1808/Configuring+the+BMC+Remedy+Notification+Engine+threads+count

Set up Archiving Policies

https://docs.bmc.com/docs/display/itsm1808/Archiving

Configure Submitter Mode

https://docs.bmc.com/docs/display/itsm1808/Configuring+the+Submitter+mode+for+BMC+Service+Request+Management

Configure Mid-Tier for SRM

https://docs.bmc.com/docs/display/itsm1808/Configuring+the+mid+tier+for+BMC+Service+Request+Management

Deployment Console

NOTE: Increase JAVA plugin from 1024 to 2048 for MainJavaPlugin to allow bigger files to import

a) Download the 18.08 ITSM, SLM, SRM packages from the EPD.b) Select “Import” and choose ITSM, SRM and SLM (In that order)

c) Deploy one package at a time (same order as above) and view the details and you’ll notice ITSM will get stuck so you’ll need to run the shell script on each of the AR Server that it reports (refer to step “To run the payload deployment utility”)

Once completed you should see a “Deployed” message:

Smart-ITPre –Upgrade tasks

Backup Smart_IT database Backup ARSystem Database Backup OS

ENSURE CURRENT INSTALL path is selected or Upgrade path would not be used

NOTE: Select NEW Install Dir

NOTE: New DB Schemas

NOTE: SMART-IT url will change after upgrade

http://vw-aus-cus-sp9u:9000/smartit/app/

http://vw-aus-cus-sp9u:9000/smartit/smart-it/#/

The following error will occur after upgrade IF there is an Overlay on HPD:WorkLog form for field

FieldID: 1000000170Field Name: Work Info Type Action: Remove Overlay from fieldValue does not fall within the limits specified for the field :(Field ID - HPD:WorkLog <1000000170>, {firstValue ... lastValue} - {1000 ... 37000}) (ARERR 306)

Post-Upgrade tasksAfter the upgrade is complete, perform the following tasks:

Run the migration utilities Run the Mongo migration utility in an incremental mode to migrate remaining social data.

Migrating social data from MongoDB to the AR System database

https://docs.bmc.com/docs/display/smartit1808/Migrating+social+data+from+MongoDB+to+the+AR+System+database

The mongo migration utility is available for download through BMC Communities.

Run the Customization and configuration utility and migrate your data to Centralized configuration. This step is required as you need to migrate your existing configuration data from the previous versions of Smart IT to the latest version of Smart IT. If you had selected the Remove leading zeroes option during installation of the previous version of

Smart IT, run the utility to remove leading zeroes from a ticket ID.

If you are not performing an in-place upgrade, run the data migration utility to copy data from old schemas to the new Smart IT schemas. Database changes in 1805 and 1808 due to new structure. That is why you must run the data migration utility to prevent data loss.

Stop and delete services Stop and delete the relevant service: (Windows) From the list of services, manually stop and delete MyITSocialService. (Linux) Run the following command at a command prompt:

service socialserviced stop Stop and delete the relevant MongoDB service:

(Windows) From the list of services, manually stop and delete MongoDB. (Linux) Run the following command at a command prompt:

service mongod stop From the pluginsvr_config.xml file, delete the SocialEventManager plug-in entry.

Verify the Activity feedsVerify the Activity feeds from the respective work log forms and from the SMT:Social_Events form.

Smart Reporting Upgrade for Smart Reporting was completed in same step as AR Server in mock labUsing AR System db as source

Remedy Single Sign-Onhttps://docs.bmc.com/docs/rsso1808/release-notes-and-notices-820496117.htmlhttps://docs.bmc.com/docs/rsso1808/known-and-corrected-issues-820494920.html

BMC recommends that you export the existing configurations prior to an upgrade or prior to making significant changes. If Remedy SSO upgrade is rolled back, the old configurations can be re-applied.

Upgrade in this sequence:o Upgrade the Remedy SSO servers.o Upgrade the Remedy SSO integration with BMC Remedy AR system, Remedy

MidTier, BMC MyIT or SmartIT, and BMC Analytics. Keep Remedy SSO servers and integrations at the same version. To prevent down time in a high availability environment, perform a rolling upgrade of all

Remedy SSO servers and integrations. Temporarily exclude the node that is being upgraded from load balancing, upgrade it, make sure it is functioning, and then include it back to balancing.

Zero Downtime Upgrade

Remove a Remedy SSO server node from the cluster. Upgrade the removed server node to the required version. Log in to the Admin console of Remedy SSO after the upgrade is complete. For each realm that is configured for SAML authentication, perform the following steps: Edit the realm to enter PEM encoded IdP signing certificate data in the IdP Signing

Certificate field or import the IdP metadata again. Ensure that the other SAML configuration is same as before except the IdP Signing Certificate field data.

Save the details. Bring the server node back to the cluster. Repeat Steps 1 to 5 for the remaining nodes in the cluster.

Digital Workplace Basic

Migrate and validate Customizations

Import payload via D2p for customizations created

Validate customizations

Apply all Patches and Hotfixes

Check EPD for available patches AT Time of document 18.08 patch 01 was released

Check BMC recommended hotfixes: ftp://ftp.bmc.com/pub/ARRecommendedFixes/Server/ for available hotfixesAlso check with Premier Support to validate that latest is published

Known Errors

Message in arerror.log on start-up of server in Server Group

DB id of current server is not matching with other servers, Please check Db-Host-Name, it should exactly match for all the servers in server group

Defect: SW00550497Solution: Apply patch 001NOTE: THIS IS AN INFO message and can be ignored