Upgrade Veritas Storage foundation 4.1 to 5.0 on Sun Solaris

All right, After a long time I finally back on blog. Little busy with my house shifting work and setting it up along with managing office work... It was real busy week... Today I have a task of Veritas Volume Manager/Veritas Storage Foundation upgrade work so thought having this information on the blog.

let me admit before I start - I am not an expert on this product and have very small experience with this product.

I am upgrading Veritas Storage foundation 4.1 to 5.0 on Sun Solaris 5.9, below steps guide on "how to?"

1. Save your existing configuration at some safe location. This document does not consider rootdg as we dont use rootdg at all. What we will concentrate on is datadg.

Lets see what is current Veritas Volume Manager version is -

# pkginfo -l VRTSvxvm
PKGINST: VRTSvxvm
NAME: VERITAS Volume Manager, Binaries
CATEGORY: system
ARCH: sparc
VERSION: 4.0,REV=12.06.2003.01.35
BASEDIR: /
VENDOR: VERITAS Software
DESC: Virtual Disk Subsystem
PSTAMP: VERITAS-4.0_MP2_RP7.2:2007-02-22
[.....]

or you can use command -

# pkginfo -l VRTSvxvm |grep VERSION |awk '{print $2}' |awk -F, '{print $1}'

Before upgrade make sure you save your current configuration like license file, vfstab file, df -kh O/P, mount command's O/P and few VxVM commands output like vxprint, vxdisk list, vxdg list etc... This information will help you in case of upgrade fail and backout process.

# cd /etc/vx/licenses/; tar -cvf /home/nilesh_rjoshi/vx_licenses_101109.tar ./* >>>> Take backup of VxVM licenses

# cp -p /etc/vfstab /home/nilesh_rjoshi/ >>>> Backup vfstab file.

# df -kh > /home/nilesh_rjoshi/df_kh_op_101109 >>>> df -kh output

# mount > /home/nilesh_rjoshi/mount_op_101109 >>>> mount command output

# vxprint -ht > /home/nilesh_rjoshi/vxprint_ht_op_101109 >>>> Save vxprint -ht output. vxprint utility displays complete or partial information from records in VERITAS Volume Manager (VxVM) disk group configurations.

# vxprint -g datadg -mvphsrqQ > /home/nilesh_rjoshi/vxprint-g_datadg_mvphsrqQ_op_101109 >>>> In detail O/P of vxprint command

# vxdisk list > /home/nilesh_rjoshi/vxdisk_list_op_101109 >>>> Save O/P of vxdisk, it shows all disks associated with or used by VERITAS Volume MAnager.

# vxdg list > /home/nilesh_rjoshi/vxdg_list_101109 >>>> Backup the data group list

Nice one, I guess now we have enough information backed up and can be used in case of backout if something goes wrong.

Let us start with the upgrade process.

Before proceeding ask application users to stop their applications or if you have authority and knowledge on how to stop applications then please disable Applications & Unmount filesystems.

Below three steps needs to be carried out -

- umount vxfs filesystems
- comment them out of the vfstab
- disable application startup in /etc/rc2.d and /etc/rc3.d

# df -kh
/dev/md/dsk/d0 3.0G 2.4G 481M 84% /
/proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
fd 0K 0K 0K 0% /dev/fd
/dev/md/dsk/d3 2.0G 1.6G 334M 83% /var
swap 11G 160K 11G 1% /var/run
dmpfs 11G 0K 11G 0% /dev/vx/dmp
dmpfs 11G 0K 11G 0% /dev/vx/rdmp
/dev/md/dsk/d6 992M 8.2M 925M 1% /home
/dev/md/dsk/d5 2.0G 362M 1.5G 19% /tmp
/dev/vx/dsk/datadg/app1_vol
5.0G 2.6G 2.3G 54% /app1
/dev/vx/dsk/datadg/app_db_vol
1.0G 452M 536M 46% /app_db
/dev/vx/dsk/datadg/appdata_vol
40G 10G 28G 28% /app_data
[.....]

# umount /app1; umount /app_db; umount /app_data

Verify if with df -kh to see everything is fine.

# vi /etc/vfstab >>>> Comment out veritas filesystems in vfstab file.

Now disable application startup in /etc/rc2.d and /etc/rc3.d

# mv S90app_tomcat K90app_tomcat

Make sure you have comment out entries in vfstab file for all veritas filesystems, application auto startup is stoped through rc directories, and all veritas filesystems are un-mounted. Next task is export the diskgroup.

# vxvol -g datadg stopall >>>> The vxvol utility performs VERITAS Volume Manager (VxVM) operations on volumes. This is Maintenance command for VERITAS.

# vxdg deport datadg >>>> Disables access to the specified disk group. A disk group cannot be deported if any volumes in the disk group are currently open. When a disk group is deported, the host ID stored on all disks in the disk group are cleared (unless a new host ID is specified with -h), so the disk group is not reimported automatically when the system is rebooted.

# vxdg -t import datadg >>>> Dissassocaite dg from host. Where -t for a temporary name. import option imports a disk group to make the specified disk group available on the local machine.When a disk group is imported, all disks in the disk group are stamped with the host's host ID.

# vxdg deport datadg

Fine, now we are done with all applicable operations on disk group.

Start the installer/upgrader:

Mount the CD/ISO file on the server and start the installation.

Quick note on mounting ISO file on Solaris -

# lofiadm -a /nilesh_rjoshi/VERITAS_Storage_Foundation_5.iso
/dev/lofi/1
# mount -F hsfs /dev/lofi/1 /mnt

Navigate to location
# cd /mnt/Solaris/Veritas/v5.0/sf_50_base

NOTE: BE SURE YOU'RE IN RUNLEVEL 3, THIS WILL NOT WORK IN RUNLEVEL 1

# ./installer

Storage Foundation and High Availability Solutions 5.0

Symantec Product Version Installed Licensed
===========================================================================
Veritas Cluster Server no no
Veritas File System 4.0 yes
Veritas Volume Manager 4.0 yes
Veritas Volume Replicator no no
Veritas Storage Foundation 4.0 yes
Veritas Storage Foundation for Oracle no no
Veritas Storage Foundation for DB2 no no
Veritas Storage Foundation for Sybase no no
Veritas Storage Foundation Cluster File System no no
Veritas Storage Foundation for Oracle RAC no no

Task Menu:

I) Install/Upgrade a Product C) Configure an Installed Product
L) License a Product P) Perform a Pre-Installation Check
U) Uninstall a Product D) View a Product Description
Q) Quit ?) Help

Enter a Task: [I,C,L,P,U,D,Q,?] I

Storage Foundation and High Availability Solutions 5.0

1) Veritas Cluster Server
2) Veritas File System
3) Veritas Volume Manager
4) Veritas Volume Replicator
5) Veritas Storage Foundation
6) Veritas Storage Foundation for Oracle
7) Veritas Storage Foundation for DB2
8) Veritas Storage Foundation for Sybase
9) Veritas Storage Foundation Cluster File System
10) Veritas Storage Foundation for Oracle RAC
11) Veritas Cluster Management Console
12) Web Server for Storage Foundation Host Management
13) Symantec Product Authentication Service
b) Back to previous menu

Select a product to install: [1-13,b,q] 5
Enter the system names separated by spaces on which to install SF: XXXXXX

Checking system licensing

Installing licensing packages
Permanent SF license registered on sdxapp1

Do you want to enter another license key for sdxapp1? [y,n,q] (n) >>>> press enter

Storage Foundation and High Availability Solutions 5.0

installer will upgrade or install the following SF packages:
VRTSperl Veritas Perl 5.8.8 Redistribution
VRTSvlic Veritas Licensing
VRTSicsco Symantec Common Infrastructure
VRTSpbx Symantec Private Branch Exchange
VRTSsmf Symantec Service Management Framework
VRTSat Symantec Product Authentication Service
VRTSobc33 Veritas Enterprise Administrator Core Service
VRTSob Veritas Enterprise Administrator Service
VRTSobgui Veritas Enterprise Administrator
VRTSccg Veritas Enterprise Administrator Central Control Grid
VRTSmh Veritas Storage Foundation Managed Host by Symantec
VRTSaa Veritas Enterprise Administrator Action Agent
VRTSspt Veritas Software Support Tools
SYMClma Symantec License Inventory Agent
VRTSvxvm Veritas Volume Manager Binaries
VRTSdsa Veritas Datacenter Storage Agent
VRTSfspro Veritas File System Management Services Provider
VRTSvmman Veritas Volume Manager Manual Pages
VRTSvmdoc Veritas Volume Manager Documentation

Press [Return] to continue:
[........]

I got below error -

The following errors prevent SF from being installed successfully on XXXXX:
827877 KB is required in the / volume and only 490085 KB is available on XXXXX


installer log files are saved at /opt/VRTS/install/logs/installer-kPOqLK

check which directory or file eating up more disk space -

# find . -name core -xdev -exec ls -l {} \; 2>/dev/null >>>> Find core files
# find . -size +20000 -xdev -exec ls -l {} \; 2>/dev/null

Do the appropriate clean-up and start installer once again.

So make sure your root filesystem has enough space available.

CPI WARNING V-9-1-1303 Installing Veritas Storage Foundation will leave only approximately 5107 KB of free space remaining
in the /var volume on XXXXX.
Do you want to continue? [y,n,q] (n) y
installer is now ready to upgrade SF.
All SF processes that are currently running must be stopped.

Are you sure you want to upgrade SF? [y,n,q] (y) y
Checking for any AP driver issues on XXXXXX ..................................................................... None

Storage Foundation and High Availability Solutions 5.0


Storage Foundation and High Availability Solutions 5.0

Stopping SF: 100%

Shutdown completed successfully on all systems


Storage Foundation and High Availability Solutions 5.0

Uninstalling SF: 100%

Uninstall completed successfully on all systems

Storage Foundation and High Availability Solutions 5.0

Installing SF: 100%

Installation completed successfully on all systems
Upgrade log files and summary file are saved at:

/opt/VRTS/install/logs/installer-XffXHU


CPI WARNING V-9-11-2246 You have completed upgrading VxVM on some or all of the systems. Reboot your systems at this time.

Reboot all systems on which VxFS was installed or upgraded.
shutdown -y -i6 -g0

See the Veritas File System Administrators Guide for information on using VxFS.

At least one package will require a reboot prior to configuration. Please reboot and run installer -configure on the
following systems after reboot has completed:

XXXXXX
Execute '/usr/sbin/shutdown -y -i6 -g0' to properly restart your systems.

Now let us install the patch.

# cd /mnt/Solaris/Veritas/v5.0/sf_50_mp1

# ./installmp

Veritas Maintenance Pack 5.0MP1 Installation Program

Copyright (c) 2007 Symantec Corporation. All rights reserved. Symantec, the Symantec Logo are trademarks or registered
trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their
respective owners.

The Licensed Software and Documentation are deemed to be "commercial computer software" and "commercial computer software
documentation" as defined in FAR Sections 12.212 and DFARS Section 227.7202.

Logs for installmp are being created in /var/tmp/installmp-tjSj6m.

Enter the system names separated by spaces on which to install MP1: XXXXX

Initial system check:



Veritas Maintenance Pack 5.0MP1 Installation Program

Checking installed packages on XXXXXX

The following patches will be installed on XXXXXX:
123996-06 to patch package VRTSsmf
123678-04 to patch package VRTSat
122632-02 to patch package VRTSobc33
122631-02 to patch package VRTSob
122633-02 to patch package VRTSobgui
123076-02 to patch package VRTSccg
123079-02 to patch package VRTSmh
123075-02 to patch package VRTSaa
[.......]

Press [Return] to continue:
installmp is now ready to install MP1.
Required 5.0 processes that are currently running will be stopped.

Are you sure you want to install MP1? [y,n,q] (y)

Installer will now stop the process and will install MP1. Basically this will install several Sun patches required for VERITAS packages. It is just a process of fixing bugs!

Add the mp1_rp3 [VRTSvxvm 5.0MP3RP2: Rolling Patch 02 for Volume Manager 5.0MP3]

# cd /mnt/Solaris/Veritas/v5.0/sf_50_mp1_rp3/

# patchadd 123742-04

# patchadd 124361-04

Once all this done the re-import the diskdg

# vxdg import datadg

# vxdg deport datadg

# vxdg import datadg

# vxvol -g datadg startall

Verify with

# vxprint -ht >>>> Verify that the volumes are ENABLED/ACTIVE

Final touchup

Renable filesstems in /etc/vfstab
Re-enable application startup in /etc/rc2.d/ /etc/rc3.d
verify you can mount all filesystems in vfstab with "mount -a"
reboot system one final time: init 6 or shutdown -y -i6 -g0

Here you done. This upgrade is pretty straight forward if everything is good like enough disk space on root, var and tmp, system is well patched etc...

Hope it will be a useful information for someone! Have a great weekend.