VXVM Training 1

VERITAS Confidential

Welcome

To

VERITAS Presentation


Volume Manager for Solaris


Suite of Products

• VERITAS Volume Manager (VM)• VERITAS FlashSnap• Dynamic Multipathing Software (DMP)• VERITAS Volume Replicator (VVR) Support• VERITAS Cluster Server (VCS) Support• Microsoft Cluster Service (MSCS) Support


VERITAS Volume ManagerGeneral features are:• Centralized Storage Management across

Entire Domain– Supports concurrent local and remote client

connections to the server for browsing and update capabilities; thus, it’s possible to administer online storage remotely.

– Provides asynchronous notification to all connected clients whenever any change to the storage configuration takes place.


VM – General Features• Configuration Solutions

– Ability to configure and manage different volume layouts: concatenated, striped, mirrored, mirrored striped, and RAID-5 volumes. Supports up to 32-way mirrors on a mirrored volume.

– Automatic detection of failed disks and the ability to repair fault-tolerant volumes on those disks without interrupting applications using the volumes.


VM – General Features- Provides storage migration path with the ability

to upgrade existing Windows NT 4 or “basic disk” partitions to the new “dynamic” volumes based on VERITAS Volume Manager technology.

- Supports and manages FT (fault-tolerant) disks and volumes previously created in a Windows NT system with Disk Administrator.

- Supports online extending of all volume types.


VM – General Features- Supports a cluster environment that runs

under VERITAS Cluster Server software or Microsoft Cluster Server software.

- Supports mounting a volume without requiring the use of a drive letter.(windows only)

- Supports moving of storage between computers with the Import and Deport Dynamic Disk Group functions.

- Command line support for administrators who prefer this method over a GUI.


VM – General Features• Optimized System Performance

- I/O statistics are used to identify high-traffic areas, known as “hot spots.” You can use the Volume Manager Move Subdisk command to resolve these hot spots online.

- Data is assigned to physical drives to evenly balance the I/O load among the disk drives. This is known as storage load balancing.

- Event logging of errors and important information is provided.


GUI vs. Command Line

• Easy to use• Multi Platform• Makes hard tasks easy• Easy error detection• Alert logs

• Lots of options• Works remotely from

nearly any place.• Makes very hard tasks

possible.• Steep learning curve

GUI Command Line


VERITAS Enterprise AdministratorVERITAS Enterprise Administrator (VEA), provides access to applications on managed hosts.


Storage ObjectsStorage Objects• Disk Group

– A set of disks that are an administrative domain for creating and managing volumes. Volumes can’t span disk groups.

– Each disk stores a copy of the configuration data for the whole group. Dividing all disks into smaller groups reduces the amount of configuration data that have to be stored on each disk.

• Disk– Volume Manager disk objects persist even

when the physical disk they represent are removed from the system.


VEA – Disk View tab


Storage ObjectsStorage Objects• Volume

– A virtual disk without the size limitations, and with better performance and reliability than the underlying physical disks.

• Subdisk– A contiguous region on a disk that is part of the

physical storage for a volume. Subdisks can’t overlap.

• Plex– A component of a volume that contains one

complete copy of the data or a log.


VEA – Volume tab


VEA – Subdisks and Plex


Storage Object Relationships in General

Disk group

Disk

Plex

Subdisk

File System

Volume

1

NN

N

N N

1 1

1

1

1

1


VRTSexplorer• Allows support to check the health of the System

•Provides us with the understanding of the system, so that support can make right recommendation

•Gives support a true glance of the problem or issue being resolved.


How to get VERITAS VRTSexplorer• # ftp ftp.veritas.com• # Login: anonymous• # Passwd: <email address>• # cd /pub/support• # bin• # get vxexplore.tar.Z

• In Windows Explorer• ftp://ftp.veritas.com/pub/support/

vxexplore.tar.Z

• VERITAS Volume Manager 3.5 “Ships with vxexplorer and other tools”

• Trouble Shooting Tools are located ins package “VRTSspt”


How to Run VRTSexplorer#cd VRTSspt#cd VRTSexplorer./VRTSexplorerVRTSexplorer: Initializing.VRTSexplorer: Please enter case number, or just hit enter:100000001

VRTSexplorer: Please select a destination directory (default: /tmp): VRTSexplorer: Collecting system configuration information for SunOS system.VRTSexplorer: Collecting VERITAS package version information.VRTSexplorer: Collecting loadable module information.VRTSexplorer: Collecting ISIS configuration information.VRTSexplorer: Collecting SIG licensing information.VRTSexplorer: Determining current VxVM operating mode. VRTSexplorer: Collecting VxVM configuration information.

VRTSexplorer: Collecting DMP configuration information.

NOTICE: This section will stop and restart the VxVM Configuration Daemon,

vxconfigd. This may cause your VxVA, VMSA and/or VEA session to exit.

This may also cause a momentary stoppage of any VxVM configuration

actions. This should not harm any data; however, it may cause some

configuration operations (e.g. moving subdisks, plex

resynchronization) to abort unexpectedly. Any VxVM configuration

changes should be completed before running this section.

If you are using EMC PowerPath devices with VERITAS Volume Manager,

you must run the EMC command(s) 'powervxvm setup' (or 'safevxvm

setup') and/or 'powervxvm online' (or 'safevxvm online') if this

script terminates abnormally.

Restart VxVM Configuration Daemon? [y,n] (default: n)y

VRTSexplorer: Collecting VRAS configuration information.

VRTSexplorer: Script finished.

VRTSexplorer: Please ftp /tmp/VRTSexplorer_100000001_aisia.tar.Z to ftp.veritas.com:/incoming


What is in VRTSexplorer

arp_a ifconfig_a nfs uname_a cron iostat_En pam uptime dev isainfo_v pkginfo usr df_klaV isis pkginfo_l var df_klag kernel pkgparam_patchlist vexplore_version

df_klat license prtconf vmstateeprom modinfo prtdiag vmstat_s env mount_v ps_elf vras etc netstat_i reboot vxfshosted netstat_r showrev vxvm---

Directory


VXVM Directory

Dmp-----Directoryvxdctl_c_modevxdctl_modevxdg_listvxdg_list_debdgvxdg_list_deqvxdg_list_dg1vxdg_list_rootdgvxdisk_listvxdisk_list_c0t0d0s2

For Every Diskvxdisk_s_listvxkprintvxprint

vxprint_htvxprint_m_debdgvxprint_m_deqvxprint_m_dg1vxprint_m_rootdgvxprint_mpvshr_debdgvxprint_mpvshr_deqvxprint_mpvshr_dg1vxprint_mpvshr_rootdgvxstat_g_debdgvxstat_g_deqvxstat_g_dg1vxstat_g_rootdg


Common Issues on Solaris

DMP Device FailuresPackage Install/UpgradeDisaster Recovery

(Duplicate Disk Group Copies)


DMP• DMP Functionality• When is DMP installed• DMP Flow Chart• DMP Error Handling

– DMP Supported Arrays (Appendix A)– Array Types (Appendix B)


DMP Functionality• Provide a single path abstraction from each disk

device to VxVM– Handles I/O across these paths

• Load balancing of I/Os across the multiple path– Feature currently available only for Active/Active(A/A) DMP

Arrays

• Transparent fail over from one path to another


DMP Flow Chart


When is DMP installed• The DMP driver is installed during the VxVM

pkgadd• This can be verified by doing looking at the

modinfo file in the explorer– #more modinfo | grep VxVM

• 23 10202043 1188e8 263 1 vxio (VxVM 3.5s_p2.8 I/O driver)• 25 10305b3e 1b23c 262 1 vxdmp (VxVM 3.5s_p2.8 DMP Driver)• 26 1031e81a 80f 264 1 vxspec (VxVM 3.5s_p2.8 control/status)


Error Handling in DMP• Error daemon is awakened whenever I/O returns

with an error

• Error daemon then checks the status of the path in a disk array by doing SCSI inquiry on the path

• If path is not alive, error daemon disables the path and retries the I/O on the other path before returning error


Error Handling in DMP(contd)

• If path is alive, the error daemon retries the I/O, a default of five times. If I/O fails even after the retries DMP assumes it is a media


Disk Failure• Determine the disk(s) failing

– Use command vxdisk list

• Determine the disk group the failure is involved with

• Check the volume,plexes and sub-disk states – Use command vxprint –ht

• Check message log to determine cause of failure


DEVICE TYPE DISK GROUP STATUSc1t13d0s2 sliced testdg04 testdg onlinec1t14d0s2 sliced testdg03 testdg onlinec1t15d0s2 sliced - - error c2t10d0s2 sliced rootdisk rootdg onlinec2t13d0s2 sliced testdg01 testdg online- - testdg00 testdg failed was:c1t15d0s2

vxdisk list


vxprint –g testdg -ht

v vol02 - ENABLED ACTIVE 2097216 RAID - raid5pl vol02-01 vol02 ENABLED ACTIVE 2098272 RAID 4/32 RWsd testdg04-02 vol02-01 testdg04 524356 699426 0/0 c1t13d0 ENAsd testdg03-02 vol02-01 testdg03 524356 699426 1/0 c1t14d0 ENAsd testdg00-02 vol02-01 testdg00 524356 699426 2/0 - NDEVsd testdg01-02 vol02-01 testdg01 524356 699426 3/0 c2t13d0 ENA

/var/adm/messages• Sep 16 13:19:34 aisia disk not responding to selection• Sep 16 13:19:34 aisia vxdmp: [ID 997040 kern.notice] NOTICE:

vxvm:vxdmp: disabled path 32/0x160 belonging to the dmpnode 239/0x20

• Sep 16 13:19:34 aisia vxdmp: [ID 148046 kern.notice] NOTICE: vxvm:vxdmp: disabled dmpnode 239/0x20

• Sep 16 13:19:48 aisia vxio: [ID 686135 kern.warning] WARNING: vxvm:vxio: object testdg00-02 detached from RAID-5 vol02 at column 2 offset 0

• Sep 16 13:19:48 aisia vxio: [ID 354480 kern.warning] WARNING: vxvm:vxio: RAID-5 vol02 entering degraded mode operation

0x20 = 32 Decimal ---This is the DMP node made by Veritas DMP to match the node you must convert the Hex number

/dev/vx/rdmp & dmpbrw------- 1 root other 239, 32 Sep 14 21:18 c1t15d0s0


With VERITAS VRTSexplorer review the following

• /vxvm/vxprint_ht• /vxvm/vxdisk_list• /var/adm/messages

– search for degraded

• Solution (Appendix E)


Package Installation/Upgrade• Make sure packages are installed completely

– VXVM requires packages to be installed in correct order– If unsure of what order refer to the manual– VXVM3.5 provides the (installer) script to assist in installation

only

• Check modules are correctly loaded for the Solaris Version loaded on the system

• Check license keys are correctly listed


pkginfo –l VRTSvxvmPKGINST: VRTSvxvm NAME: VERITAS Volume Manager, Binaries CATEGORY: system ARCH: sparc VERSION: 3.5,REV=06.21.2002.23.14 BASEDIR: / VENDOR: VERITAS Software DESC: Virtual Disk Subsystem PSTAMP: VERITAS-3.5s:21-Jun-2002 INSTDATE: Aug 20 2003 10:47 HOTLINE: 800-342-0652 EMAIL: [email protected] STATUS: completely installed FILES: 603 installed pathnames 22 shared pathnames 9 linked files 76 directories 340 executables 158087 blocks used (approx)

pkgadd -d /cdrom/CD_name/volume_manager3.5/pkgs

1-VRTSvlic 2-VRTSvxvm 3-VRTSvmdoc4-VRTSvmman 5-VRTSob 6-VRTSobgui 7-VRTSvmpro 8-VRTSfspro


/uname_aSunOS aisia 5.9 Generic_112233-03 sun4u sparc SUNW,Sun-Blade-1000

/kernel# more /kernel/drv/sparcv9/ls_l_vx-rw-r--r-- 1 root sys 435312 Jun 21 2002 /kernel/drv/sparcv9/vxdmp-rw-r--r-- 1 root sys 412304 Jun 21 2002 /kernel/drv/sparcv9/vxdmp.SunOS_5.7-rw-r--r-- 1 root sys 427536 Jun 21 2002 /kernel/drv/sparcv9/vxdmp.SunOS_5.8-rw-r--r-- 1 root other 435312 Aug 20 10:48 /kernel/drv/sparcv9/vxdmp.SunOS_5.9-rw-r--r-- 1 root sys 2953480 Jun 21 2002 /kernel/drv/sparcv9/vxio-rw-r--r-- 1 root sys 2866112 Jun 21 2002 /kernel/drv/sparcv9/vxio.SunOS_5.7-rw-r--r-- 1 root sys 2908152 Jun 21 2002 /kernel/drv/sparcv9/vxio.SunOS_5.8-rw-r--r-- 1 root other 2953480 Aug 20 10:48 /kernel/drv/sparcv9/vxio.SunOS_5.9-rw-r--r-- 1 root sys 15336 Oct 31 2002 /kernel/drv/sparcv9/vxportal-rw-r--r-- 1 root sys 19216 Jun 21 2002 /kernel/drv/sparcv9/vxspec-rw-r--r-- 1 root sys 17840 Jun 21 2002 /kernel/drv/sparcv9/vxspec.SunOS_5.7-rw-r--r-- 1 root sys 18536 Jun 21 2002 /kernel/drv/sparcv9/vxspec.SunOS_5.8-rw-r--r-- 1 root other 19216 Aug 20 10:48 /kernel/drv/sparcv9/vxspec.SunOS_5.9


/opt/VRTSvlic/bin#./vxlicrep -e License Key =

528222690330627075168549872296 Product Name = VERITAS Volume Manager Lic Key Lib Ver = ELM Key = Valid License Type = PERMANENT_NODE_LOCK Node Lock Type = 3 (Hostid and Architecture ID)

Features := RAID = Enabled


Disaster Recovery• Disaster Recovery(DR)

– Any event that will require a rebuild of a Disk Group or Volume configuration database.

• Example Errors:Duplicate Disk Group RecordsCorrupted Disk Group ConfigurationNo Valid Disk Group Configuration Found

• When preparing for a disaster it is always a good plan to have the following:


Disaster Recover (contd)• Layouts of the Diskgroup and Volumes

– vxprint –ht or vxprint -htr– vxprint –g <diskgroup> -mpvshr

• If the above outputs cannot be provided due to a “duplicate disk record” error. It is important to gather the following before making any volume record changes. (Example in: Appendix F)– /etc/vx/diag.d/vxprivutil dumpconfig

/dev/rdsk/c#t#d#s<private> > dump.out


Tools for Windows

• VxExplorer – gathers most of the necessary information to troubleshoot a problem.

• Vxdisksetid – used to verify diskset id consistency.

• Vxtool – advanced tools used by support and engineering to diagnose and correct problems.


Disk Group ConfigurationDisk Group Configuration

• VxConfig stores a copy of a disk group’s configuration on each disk in the group.

• A disk group’s configuration is described in terms of configuration records.

• There is a record type for each object type.


Configuration Record TypesConfiguration Record Types•The types of configuration records

correspond to the VM object model– Disk group record– Volume record– Plex record– Subdisk record– Two kinds of disk records


Disk Media v.s. Disk Access RecordsDisk Media v.s. Disk Access Records• Disk Media Configuration Record

– Identifies a disk by disk id and gives it an administrative name, e.g. “disk1”.

– A disk group contains dm recs for all member disks.– Subdisk recs are associated with dm recs.– Dm recs can be disassociated from a failed physical

disk and re-associated with a replacement disk.• Disk Access Record

– Identifies an O/S access path to a disk,e.g. \device\Harddisk0.

– Not stored in the on-disk configuration database because it can easily change after rebooting.


Missing DisksMissing Disks

• If a disk group contains a disk media record for a disk that Windows doesn’t know about (i.e. no disk access record), the disk is a Missing Disk.


Missing Disk


VM Alert MessageAlert MessageAlert Message


Event Viewer Alert


Replace Disk


Foreign Disks•A disk is foreign if the hostid in it’s

configuration doesn’t match the server,

or

• If it’s disk group id matches an existing disk group but it’s diskset id doesn’t


Foreign Disk


Merge Foreign


Dynamic Disk HeaderDynamic Disk Header

•A disk header records the disk’s VM disk id, identifies the disk’s disk group, and describes the layout of the private region.


Disk Header ContentsDisk Header Contents• diskid: 8998ae08-f2a6-4264-97dc-8f5ceeda43de• group: name=NewGrp id=a51fb942-4fcd-4a0f-bc88-0dd0b07a64c9• flags: private autoimport• import: bootsig=130 hostid=d0120cc1-8d21-11d7-aade-806d6172696f• diskset: id=b95a984f-aeb0-4855-ab15-0b39e05145fe• version: 2.12• iosize: 512• public: slice=0 offset=63 len=35551782• private: slice=0 offset=35564430 len=2048• update: seqno=0.36 time=1635821074• headers: 2047 1856• configs: count=1 len=1481• logs: count=1 len=224


Configuration RecordsConfiguration Records• Disk: Disk2 rid=0.1029 updated=0.1155• assoc: diskid=8998ae08-f2a6-4264-97dc-8f5ceeda43de

• Group: NewGrp rid=0.1025 version=30 update=0.1154• id: dgid=a51fb942-4fcd-4a0f-bc88-0dd0b07a64c9• diskset: id=b95a984f-aeb0-4855-ab15-0b39e05145fe• • Volume: volB rid=0.1054 update=0.1151 mountname=F:• info: len=1536000 guid=dda5ab3f-ac13-4915-8353-51d1a3615fd9• state: state=ACTIVE• • Plex: volB-01 rid=0.1056 update=0.1151• type: layout=STRIPE columns=3 width=128• state: state=ACTIVE• • Subdisk: Disk2-01 rid=0.1049 updated=0.1155• info: disk=0.1029 offset=0 len=256000 hidden=0• assoc: plex=0.1041 (column=3 offset=0)


Diskset id mismatch

•Diskset id mismatch means the disk was removed from the disk group while the disk was missing.


Disketset Id• Diskset id is used to make sure that a disk that’s

removed and then reinserted is not modified.• A new diskset id is created when a disk is

missing.• Diskset id is stored in the disk header and also in

the dg record in the config database.• Each dm record also keeps track of the last

diskset id when its da is detached.


Diskset Id Checking• A disk cannot be onlined if the diskset id does

not match in its header and dg record in config.• A disk can be brought online automatically in

rescan operation if the diskset id in the header matches the “lastset” id in a detached dm record.

Disk: Disk2 rid=0.1029 updated=0.1158 assoc: diskid=8998ae08-f2a6-4264-97dc-8f5ceeda43de detached lastset: id=b95a984f-aeb0-4855-ab15-0b39e05145fe


Vxdisksetid output

harddisk4header diskset ID=47b91238-1db1-4943-9d63-6264c7b667b7config diskset ID=47b91238-1db1-4943-9d63-6264c7b667b7

harddisk5 has a mismatched diskset IDheader diskset ID=b45dd49a-10aa-43f8-aa21-7c3fc4b3e1ffconfig diskset ID=98a808a8-07cf-4175-8334-3419ed27b396


Vxtoolvxtool - VERITAS Volume Manager diagnostic operations

Usage: • vxtool disk <sub-operation> [-<options>] [<arguments>] • vxtool kernel <sub-operation> [-<options>] [<arguments>] • vxtool verify [-v] <volume-name> [[<offset>] <length>] • vxtool getioparms <driveletter>:


Vxtool disk setheaderFields that can be modified:• Diskid• Hostid• Dgid• Dg_name• Disksetid• newdisksetid


Vxtool disk setheader

The below command sets the diskset id entry in the header record to match that of the config record. After a rescan the disk can be brought back into the diskgroup successfully.

vxtool disk setheader harddisk5 diskset=98a808a8-07cf-4175-8334-3419ed27b396


DMP Supported Arrays Appendix A

ARRAY TYPE VID PIDA/A SUN AP_NODESA/A VERITAS

ATFNODESA/AECCS allA/A EMC SYMMETRIXA/A FUJITSU GR710, GR720,

GR730, GR740, GR820, GR840A/A HITACHI OPEN-*A/PG HITACHI DF350,DF400,

DF500A/P LSI INF-01-00A/A NEC DS1200,

DS1200F, DS3000SL,DS3000SM,DS3001

A/A NEC DS3001, DS3002, DS1000, DS1000F, DS1100,DS1100F, DS3011, DS1230, DS450, DS450F, iStorage 1000, iStorage 2000, iStorage 4000

A/P SUN T300A/A VERITAS

RDACNODESA/A SENA allA/AIBM 2105A/ASSA SSAA/A StorComp

OmniForceA/PF VERITAS allA/A HP OPEN-*


Array Types Appendix B

• A/A

• A/P

• A/PF

• A/PG

•Active/Active disk array; I/Os could be simultaneously active on more than one path to the device without any performance degradation

•Active/Passive disk array; I/Os can not be issued to more than one path to the device without severe performance penalty; when such arrays are con.figured in auto-trespass mode,the path failover occurs with a normal I/O command to the passive (standby) path

•Active/Passive disk array in explicit failover mode; A/P array that will require special failover command; support for such arrays can not be added dynamically

•Active/Passive disk array with LUN group failover. A special case of A/P array where a group of LUN’s fail-over


vxdg_list Appendix C


vxdisk_list_c1t15d0s2 Appendix D


Solution Appendix E

• Vxdiskadm (Appendix G)– Option 4 Remove a disk for replacement– Option 5 Replace a failed or removed disk

• Volume should start rebuilding parity on new device– Use vxtask list to view status

• Example: vxtask listTASKID PTID TYPE/STATE PCT PROGRESS171 171 R5RECOVER/R 32.09% 0/699426/224448

RECOV vol02 testdg00-02


vxprivutil Appendix F

• Appendix C shows what disk have enabled configuration copies

Example:./vxprivutil dumpconfig /dev/rdsk/c2t15d0s3 >

/tmp/dump.out

• Appendix D show what slice the private region is in.

• Verify the dump.out file with the following:cat /tmp/dump.out | vxprint –D - -ht


vxdiskadm Appendix G

Volume Manager Support OperationsMenu: Volume Manager/Disk

1 Add or initialize one or more disks 2 Encapsulate one or more disks 3 Remove a disk 4 Remove a disk for replacement 5 Replace a failed or removed disk 6 Mirror volumes on a disk 7 Move volumes from a disk 8 Enable access to (import) a disk group 9 Remove access to (deport) a disk group 10 Enable (online) a disk device 11 Disable (offline) a disk device 12 Mark a disk as a spare for a disk group 13 Turn off the spare flag on a disk

SO ON……….

Documents

VXVM Training 1