Upload
dangkien
View
215
Download
1
Embed Size (px)
Citation preview
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Brad McCuskerSoftware Concepts International, LLC30 Temple St, Suite 303Nashua, NH 03060, USA
Phone: 603-879-9022e-mail: [email protected]
www.sciinc.com
A World of Difference in Database Support
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
From external to in-house without knowing how to spell VMS
We sold off what? Need a copy – quick!
Hardware refresh and DR via virtualization
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS 3
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Managing OpenVMS systems and databases requiring the highest
levels of performance and availability – worldwide
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS 5
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Healthcare sector
Hosting provider exiting the business Also their application provider
Customer needed to: Host system Assume application maintenance Continue to run their business
6
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
ES-45, 4P, 32GB, VMS 8.3, EVA 3000s (Chicago) COBOL, MultiNet, DCPS, SSL and Ghostscript
~500GB data to be migrated
Single System supporting 8 customers Each customer had test and prod environments This customer was largest and had 5 environments
DR: Similar configuration (Denver) Home grown, host level data replication (transactional) Not complete replication
7
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Prod – Denver, CO: ES80, 6P, 48GB DR – Tacoma: ES80, 6P, 48GB Dev/Test - Tacoma: DS25, 2P, 16GB
IBM DS8000, Global Mirror between sites
OS, LP – same as source
Oversized for growth
8
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
IT team knew nothing of VMS
Subsidiary new to corporate infrastructure No control over resources, urgency not appreciated
Data Migration – HIPAA Difficult to determine data volume
DR requirement high priority
Plenty of managers – rarely clear who was in charge
9
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
January – first contact Contacted by 2 organizations (HP and PWC)
May – First detailed planning session June – Hardware received July – System buildout August – Dev/Test Migration November – Production Migration
10
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Network from Chicago? Struggled to understand how much data, rate &
reliability
Hand carry from Chicago? Time consuming
Hand carry across Denver? Concern over completeness of replication
11
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Hand carry across Denver Rely on replication for transactional data Push non transactional data from Chicago to Denver
12
Contingency Plan: Create full backup tape in Chicago Fly tape Chicago to Denver
This contingency saved the project!
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Corporate solution – CommVault No Alpha client
Solution: Backups copied to Linux, CommVault from Linux Create savesets to disk Compress savesets SFTP to Linux
13
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
100GB/day – time consuming
Used PIGZ to compress Parallel gzip
SFTP problems Multinet SFTP unstable, >2GB transfers didn’t work Split the .gz file into 2GB chunks prior to transfer
14
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
“warm standby” Global Mirror replicates LUNS to Tacoma Including system disk
ES80 booted with simple VMS system disk Replicated LUNS mounted RO on DR system DR Event: Stop replication Reboot ES80 from replicated system disk SCI added code to startup to recognize hardware – do
the right thing for network, etc.
15
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
We learn (late) that Global Mirror is new to their infrastructure
VMS systems first to significantly use it.
August 2014 – We learn it doesn’t work (2.5 months before “go live”) Fix targeted for late Sep/early October
Forced to re-evaluate “home grown” replication Ultimately, problem was resolved in early October
16
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Storage team promised they would monitor replication and recognize failures
Didn’t trust After all – Global Mirror was also brand new to them
SCI added replication monitoring On Prod, update timestamp in a file on each disk On DR, read the file; if time is not within 5 minutes of
system time – raise alert
And…. We caught replication failures that corporate missed!
17
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS 18
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Two AlphaServer systems must be migrated from physical hardware in one country to emulated systems in another country within weeks
SCI engaged to perform total system migration AlphaServer 1200 / OpenVMS V7.2 AlphaServer 4000 / Digital UNIX (aka Tru64) V4.0E All data transferred unchanged to emulation
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Complete systems content successfully migrated from physical hardware to CHARON-AXP emulated environment
Hardware refitted Operating system, data and application unaffected Systems IP address modified for new location Virtual disk and tape replace physical drives
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
We felt a little bad being involved with sending tshis old hardware out to pasture
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Need to transfer +/- 200GB business sensitive data securely over international borders Two 256GB memory sticks from different vendors Bitlocker AES-256 encryption with password known
only to customer
Limited downtime available No network Sites within 1 day drive of each other
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
1. Full physical system disk backups performed 2. Backups copied to encrypted devices 3. Transfer devices transported via automobile4. Emulator configured; matching disk drive layout5. Full system content restored from transfer devices
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
6. Operating systems reconfigured for network, disk and tape drive configurations
7. Data validated to compare old and new 8. Application tested for data and functionality9. Tested automated monthly / daily backup jobs 10. Data transfer devices securely erased
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
During initial data copy and validation phase, SCI laptop used as emulator platform
Network backup target
Validate systems functionality
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
HP ProLiant DL380p Gen8 2x E5-2680 v2 (10C, 25M Cache, 2.80/3.60T GHz) 32GB
RHEL 6 Update 5 (Santiago) Linux 2.6.32
CHARON-AXP V6.1
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Prior AlphaServer 1200 1 CPU 2GB memory 1 tape 2 disks 10/100mb NIC
New Emulated AlphaServer 1200 1 CPU 4GB memory 1 tape 2 disks 1gb NIC
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Prior AlphaServer 4000 1 CPU 512MB memory 3 tapes 4 disks 10/100mb NIC
New Emulated AlphaServer 4000 1 CPU 4GB memory 3 tapes 4 disks 1gb NIC
Interestingly enough, had been running at 10mb (presumably forever); set to 100 to make data copies tolerable
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Container files represent emulated disk and tape drives When tape image is dismounted by guest, it is disconnected from
CHARON-AXP and can be manipulated Can be replaced with different tape image retaining same name Tape images ("*.vtape") created automatically if not existing
as1200_vms /charon/vms/DKA0.*disk /charon/vms/DKA100.*disk
as4000_tru64 /charon/tru64/DKA0.*disk /charon/tru64/DKA100.*disk /charon/tru64/DKA200.*disk /charon/tru64/DKA300.*disk
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Once replacement system tested and validated
1. $ SET DEFAULT SYS$MANAGER:2. $ DISMOUNT DRA0:3. $ INIT/ERASE/SYSTEM DRA0: CLEANED4. $ SHOW DEV SYS$SYSDEVICE:5. $ COPY/ALLOC={freeblocks-1} NLA0: X.X6. $ SET FILE/END X.X7. $ DELETE/ERA/EXCL=*.DIR SYS$SYSDEVICE:[*…]*.*.*
! Ignore warnings and errors8. Power off system
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS 32
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Replace aging Alpha hardware
Provide Disaster Recovery capability 2 data centers, same corporate campus.
Utilize corporate NetApps SAN
33
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Semiconductor manufacturer – Highly available
OpenVMS V7.3-2 , Rdb, Workstream for MES
3 Node Cluster ES45, 1 GHz, 2P 1 node very lightly used
1 additional ES45 – Dev/Test
34
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
CHARON-AXP on VMware hosts 2 * Fujitsu RX300 S8 E5-2643v2 CPUs (6C/12T, 3.50GHz). 5 virtual cores presented to each host
128 GB RAM (32GB per Windows host) 2 * 10Gb NICS, 2 * 1Gb NICs
NetApp Metrocluster – 2 FAS3270 filers 8 * 1G Ethernet ports
Reduced cluster from 3 nodes to 2 nodes
35
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
No FibreChannel on NetApps. Customer standard is NFS to VMWare hosts
Introduce split site cluster for DR (1700’)
Minimize downtime Customer prefers none
Data migration – how?
No legitimate test environment
36
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Shared “vdisk” container files? Performance concerns with file shares Reliability & persistence of file shares
“vidisk” containers in each location with VMS HBVS between sites? Performance concerns using MSCP
Shared VMWare multi -writer .vmdk files? Not proven in production We chose this as first option
37
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMSGB network - NFS
VMWare leverages hosted .vmdk images as a Windows local diskESX allows multi-writer sharing of .vmdk images (VMWare KB 1034165)
\\.\PhysicalDriveN
(.vmdk)
ESX
NFS
NetApps Disk VolumesDisk Volumes
Building 1/ESX Host 1 Building 2/ESX Host 2
\\.\PhysicalDrive0
(.vmdk)… Disk controller
emulation: SCSI
Charon-AXP/DS20
VMWare Guest 2Windows
OpenVMS
DKA0
DKAn
Disk controlleremulation: SCSI
Charon-AXP/DS20
VMWare Guest 1Windows
OpenVMS
DKA0
DKAn
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMSGB network - NFS
\\.\PhysicalDriveN
(.vmdk)
ESX
NFS
NetApps Disk VolumesDisk Volumes
Building 1/ESX Host 1 Building 2/ESX Host 2
\\.\PhysicalDrive0
(.vmdk)…
Disk controlleremulation: SCSI
Charon-AXP/DS20VMWare Guest 1
OpenVMS
Production “A” DKA0
DKAn
Disk controlleremulation: SCSI
Charon-AXP/DS20VMWare Guest 2
OpenVMS
Test/Dev DKA0
DKAn
Disk controlleremulation: SCSI
Charon-AXP/DS20VMWare Guest 1
OpenVMS
Production “B” DKA0
DKAn
Normal Operations
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMSGB network - NFS
ESX
NFS
NetApps Disk VolumesDisk Volumes
Building 1/ESX Host 1 Building 2/ESX Host 2
\\.\PhysicalDriveN
(.vmdk)
\\.\PhysicalDrive0
(.vmdk)
…
Disk controlleremulation: SCSI
Charon-AXP/DS20VMWare Guest 1
OpenVMS
Production “A” DKA0
DKAn
Disk controlleremulation: SCSI
Charon-AXP/DS20VMWare Guest 2
OpenVMS
Development DKA0
DKAn
Disk controlleremulation: SCSI
Charon-AXP/DS20VMWare Guest 3
OpenVMS
Production “B” DKA0
DKAn
Disaster Strikes
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMSGB network - NFS
ESX
NFS
NetApps Disk VolumesDisk Volumes
Building 1/ESX Host 1 Building 2/ESX Host 2
\\.\PhysicalDriveN
(.vmdk)
\\.\PhysicalDrive0
(.vmdk)
…
Disk controlleremulation: SCSI
Charon-AXP/DS20VMWare Guest 1
OpenVMS
Production “A” DKA0
DKAn
Disk controlleremulation: SCSI
Charon-AXP/DS20VMWare Guest 2
OpenVMS
Development DKA0
DKAn
Disk controlleremulation: SCSI
Charon-AXP/DS20VMWare Guest 3
OpenVMS
Production “B” DKA0
DKAn
Disk controlleremulation: SCSI
Charon-AXP/DS20VMWare Guest 1
OpenVMS
Production “A” DKA0
DKAn
VMware Recover Guest 1 on Host 2
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
1. 2 CHARON emulators on 2 VMWare instances. Use multi-write .vmdk containers
2. Build “test” OpenVMS cluster Test shared .vmdk works (system and data disks) Test V -Motion between sites (for planned
maintenance) Test VMWare recovery between sites (for DR)
3. Present all LUNs to emulator4. Test environment is ready
42
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Remove 1 of 3 nodes (Remove NODE_A) Some processing had to be moved Some SCSNODE dependencies
Lots of node specific queue cleanup
Plan to boot an emulator into NODE_A’s root
43
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Boot 1 emulator into NODE_A’s root
Use HBVS to synch data between Alpha/EVA cluster and CHARON/NetApps
FAIL!!!
44
Despite documentation indicating Satellite boot supported by CHARON – it isn’t
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Create Heterogeneous cluster: 2 Alphas booting from EVA 1 Emulator booting from NetApps
Add network infrastructure for SCS traffic HBVS transfers data to NETApps (including Alpha
system disk) Shutdown CHARON node when copies complete Boot both CHARON nodes using replicated Alpha
cluster disk
45
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Add an emulator to the cluster, in place of a prod node Couldn’t boot as node B because of references to node
B’s local disk on node C Had to reboot C
46
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Repeat data transfer process Shutdown Alphas Shutdown CHARON nodes Boot CHARON node off replicated Alpha cluster
system disk
47
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
CHARON uses physical license token
What happens in a disaster?
Network USB hub? Rejected – single point of failure
48
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
CHARON Support Customers receive:“An additional CHARON license key, enabling 360 hours of execution time per supported product license, valid during the duration of the support contract. “
1 permanent license connected to each ESX host 1 “spare” license connected to each ESX host Specify license preference for each config
49
set session license_key_id[0]=nnnnnnn
set session license_key_id[1]=nnnnnnn
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Alphas used SLS backing up to tape
Emulators: Database backups to VMS disk Non-database disks (.vmdk files) all reside on
dedicated NetApp datastore Daily NetApp snapshot of datastore used for backups
50
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS 51
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Brad
Brad McCusker
www.sciinc.com
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
Located in Nashua, NH (USA) 30 years in business supporting OpenVMS!
International reputation as leading provider of Managed services for OpenVMS & databases OpenVMS performance and consulting services
Proven global track record Actively managing 100s of systems and databases At many sites 24x365 since 1995
Copyright 2015
SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.
World Class Managed Services For OpenVMS
System and database performance consulting
VAX/Alpha emulation – CHARON reseller
Migration consulting Specializing in minimal downtime migrations
Oracle’s worldwide provider of CODASYL DBMS training
54