Add VERITAS Volume Manager Disk Group & Filesystem into VERITAS Cluster service group - from the scratch orientation

In this article we will see how to create a new disk group, create 4-way stripe VxVM filesystem & then will see how do we create a new VCS service group and incorporate the newly created Disk Group & VxFS filesystem to the service group.
So let's roll.

Well, what's the requirement? so, here we go!

• Create 3 diskgroups namely dapos1dg, dapos2dg, dapos3dg.
• Create three VxVM filesystems within above three DiskGroup /twm/dapos1, /twm/dapos2, /twm/dapos3  with 4/128 stripe layout.
• Create three VERITAS service groups dapos1_sg, dapos2_sg & dapos3_sg.
• Insert/attach/link the needed infra resources (DiskGroup, filesystem, ..) into the corresponding VERITAS service groups.

Solution Delivery –

In order to create 4/128 stripe layout volumes/filesystems we need to get storage disks in bunch of 4 hence I ordered for 12x15G LUNs. Each DiskGroup will have 4 disks.

Well, below are the LUNs that we have from storage –

c2t5006048C52A8CCC3d40
c2t5006048C52A8CCC3d41
c2t5006048C52A8CCC3d42
c2t5006048C52A8CCC3d43
c2t5006048C52A8CCC3d44
c2t5006048C52A8CCC3d45
c2t5006048C52A8CCC3d46
c2t5006048C52A8CCC3d47
c2t5006048C52A8CCC3d48
c2t5006048C52A8CCC3d49
c2t5006048C52A8CCC3d50
c2t5006048C52A8CCC3d51

Before formatting the disks verify if someone hasn’t used these disks for any other purpose. (it happens sometimes, some lazy frogs uses unused disks without taking much efforts and without knowing may someone has ordered these LUNs for some purpose)

# for disk in `cat disks`;do vxdisk -eo alldgs list | grep $disk;done
EMC0_223 auto - - online c2t5006048C52A8CCC3d40s2
EMC0_224 auto - - online c2t5006048C52A8CCC3d41s2
EMC0_225 auto - - online c2t5006048C52A8CCC3d42s2
EMC0_229 auto - - online c2t5006048C52A8CCC3d43s2
EMC0_226 auto - - online c2t5006048C52A8CCC3d44s2
EMC0_227 auto - - online c2t5006048C52A8CCC3d45s2
EMC0_228 auto - - online c2t5006048C52A8CCC3d46s2
EMC0_231 auto - - online c2t5006048C52A8CCC3d47s2
EMC0_222 auto - - online c2t5006048C52A8CCC3d48s2
EMC0_221 auto - - online c2t5006048C52A8CCC3d49s2
EMC0_220 auto - - online c2t5006048C52A8CCC3d50s2
EMC0_230 auto - - online c2t5006048C52A8CCC3d51s2

Good. Disks are not being used by any VERITAS DG so let's label the disks with SMI label.

 Once done with labelling the disks make sure you run vxdctl enable command & re-scan for new disks devices so that the disks are visible to VxVM.

# vxdctl enable

Now we are all set to get all new disks to VERITAS control. Before taking disks into VxVM control verify if those are visible to VxVM.

# for lun in `cat /tmp/lun-list`; do vxdisk -eo alldgs list | grep -i $lun; done
EMC0_223 auto - - online c2t5006048C52A8CCC3d40s2
EMC0_224 auto - - online c2t5006048C52A8CCC3d41s2
EMC0_225 auto - - online c2t5006048C52A8CCC3d42s2
EMC0_229 auto - - online c2t5006048C52A8CCC3d43s2
EMC0_226 auto - - online c2t5006048C52A8CCC3d44s2
EMC0_227 auto - - online c2t5006048C52A8CCC3d45s2
EMC0_228 auto - - online c2t5006048C52A8CCC3d46s2
EMC0_231 auto - - online c2t5006048C52A8CCC3d47s2
EMC0_222 auto - - online c2t5006048C52A8CCC3d48s2
EMC0_221 auto - - online c2t5006048C52A8CCC3d49s2
EMC0_220 auto - - online c2t5006048C52A8CCC3d50s2
EMC0_230 auto - - online c2t5006048C52A8CCC3d51s2

Now let's initialize the new disks and get them under VERITAS control.

Get the list of new disks visible to VxVM using below command.

# for lun in `cat lun-list`; do vxdisk -e list | grep -i $lun; done | awk '{print $1}' > /tmp/vxvm_disks && cat /tmp/vxvm_disks
EMC0_223
EMC0_224
EMC0_225
EMC0_229
EMC0_226
EMC0_227
EMC0_228
EMC0_231
EMC0_222
EMC0_221
EMC0_220
EMC0_230

Initialize new disks using below command.

# for disk in `cat /tmp/vxvm_disks`;do vxdisksetup -i $disk format=sliced; done

See if all the new disks are online and under VxVM control.

# for lun in `cat vxvm_disks`; do vxdisk list | grep -i $lun; done | sort
EMC0_220 auto:sliced - - online
EMC0_221 auto:sliced - - online
EMC0_222 auto:sliced - - online
EMC0_223 auto:sliced - - online
EMC0_224 auto:sliced - - online
EMC0_225 auto:sliced - - online
EMC0_226 auto:sliced - - online
EMC0_227 auto:sliced - - online
EMC0_228 auto:sliced - - online
EMC0_229 auto:sliced - - online
EMC0_230 auto:sliced - - online
EMC0_231 auto:sliced - - online

Now let's create diskgroups –

NOTE: As these are not the shared disk groups make sure you create them on individual systems where these diskgroups needs to be mounted or else create all three disks groups on one system and then deport –import them on respective systems. Either way!

# vxdg init dapos1dg cds=off s01c0_dapos1dg=EMC0_220 s01c1_dapos1dg=EMC0_221 s01c2_dapos1dg=EMC0_222 s01c3_dapos1dg=EMC0_223
# vxdg init dapos2dg cds=off s01c0_dapos2dg=EMC0_224 s01c1_dapos2dg=EMC0_225 s01c2_dapos2dg=EMC0_226 s01c3_dapos2dg=EMC0_227
# vxdg init dapos3dg cds=off s01c0_dapos3dg=EMC0_228 s01c1_dapos3dg=EMC0_229 s01c2_dapos3dg=EMC0_230 s01c3_dapos3dg=EMC0_231

# vxdg list |egrep 'dapos1dg|dapos2dg|dapos3dg'
dapos1dg enabled 1330089054.668.sys1
dapos2dg enabled 1330089075.670.sys1
dapos3dg enabled 1330089087.672.sys1

Now we will create STRIPE volumes within these newly created three diskgroups.

# vxassist -g dapos1dg make dapos1 3g layout=stripe s01c0_dapos1dg s01c1_dapos1dg s01c2_dapos1dg s01c3_dapos1dg
# vxassist -g dapos2dg make dapos2 3g layout=stripe s01c0_dapos2dg s01c1_dapos2dg s01c2_dapos2dg s01c3_dapos2dg
# vxassist -g dapos3dg make dapos3 3g layout=stripe s01c0_dapos3dg s01c1_dapos3dg s01c2_dapos3dg s01c3_dapos3dg

Well, the volumes are created. Let’s check them out.

# for i in dapos1dg dapos2dg dapos3dg; do vxprint -htg $i | grep -i dapos;done
dg dapos1dg default default 16000 1330089054.668.sys1
dm s01c0_dapos1dg EMC0_220 auto 17023 31420800 -
dm s01c1_dapos1dg EMC0_221 auto 17023 31420800 -
dm s01c2_dapos1dg EMC0_222 auto 17023 31420800 -
dm s01c3_dapos1dg EMC0_223 auto 17023 31420800 -
v dapos1 - ENABLED ACTIVE 6291456 SELECT dapos1-01 fsgen
pl dapos1-01 dapos1 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos1dg-01 dapos1-01 s01c0_dapos1dg 0 1572864 0/0 EMC0_220 ENA
sd s01c1_dapos1dg-01 dapos1-01 s01c1_dapos1dg 0 1572864 1/0 EMC0_221 ENA
sd s01c2_dapos1dg-01 dapos1-01 s01c2_dapos1dg 0 1572864 2/0 EMC0_222 ENA
sd s01c3_dapos1dg-01 dapos1-01 s01c3_dapos1dg 0 1572864 3/0 EMC0_223 ENA
dg dapos2dg default default 68000 1330089075.670.sys1
dm s01c0_dapos2dg EMC0_224 auto 17023 31420800 -
dm s01c1_dapos2dg EMC0_225 auto 17023 31420800 -
dm s01c2_dapos2dg EMC0_226 auto 17023 31420800 -
dm s01c3_dapos2dg EMC0_227 auto 17023 31420800 -
v dapos2 - ENABLED ACTIVE 6291456 SELECT dapos2-01 fsgen
pl dapos2-01 dapos2 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos2dg-01 dapos2-01 s01c0_dapos2dg 0 1572864 0/0 EMC0_224 ENA
sd s01c1_dapos2dg-01 dapos2-01 s01c1_dapos2dg 0 1572864 1/0 EMC0_225 ENA
sd s01c2_dapos2dg-01 dapos2-01 s01c2_dapos2dg 0 1572864 2/0 EMC0_226 ENA
sd s01c3_dapos2dg-01 dapos2-01 s01c3_dapos2dg 0 1572864 3/0 EMC0_227 ENA
dg dapos3dg default default 20000 1330089087.672.sys1
dm s01c0_dapos3dg EMC0_228 auto 17023 31420800 -
dm s01c1_dapos3dg EMC0_229 auto 17023 31420800 -
dm s01c2_dapos3dg EMC0_230 auto 17023 31420800 -
dm s01c3_dapos3dg EMC0_231 auto 17023 31420800 -
v dapos3 - ENABLED ACTIVE 6291456 SELECT dapos3-01 fsgen
pl dapos3-01 dapos3 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos3dg-01 dapos3-01 s01c0_dapos3dg 0 1572864 0/0 EMC0_228 ENA
sd s01c1_dapos3dg-01 dapos3-01 s01c1_dapos3dg 0 1572864 1/0 EMC0_229 ENA
sd s01c2_dapos3dg-01 dapos3-01 s01c2_dapos3dg 0 1572864 2/0 EMC0_230 ENA
sd s01c3_dapos3dg-01 dapos3-01 s01c3_dapos3dg 0 1572864 3/0 EMC0_231 ENA

Next thing is to create a mount point for these volume & create a file system on this volume.

On individual servers…

# for fs in 1 2 3;do mkdir -p /twm/dapos$fs;done && ls –ld /twm/dapos*
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos1
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos2
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos3

Not required to be done on individual servers as the storage disks are shared between cluster nodes.

# mkfs -F vxfs /dev/vx/rdsk/dapos1dg/dapos1
# mkfs -F vxfs /dev/vx/rdsk/dapos2dg/dapos2
# mkfs -F vxfs /dev/vx/rdsk/dapos3dg/dapos3

First time needed only on respective individual servers.

# mount -F vxfs /dev/vx/dsk/dapos1dg/dapos1 /twm/dapos1

Now, furthermore there is requirement to create new VCS service groups & Insert/attach the needed infra resources (DiskGroup, filesystem, ..) into the corresponding VERITAS service groups. Let’s on it!

Before doing anything, first thing to check if cluster configuration is read-only or in write mode.

# haclus -display |grep -i 'readonly'
ReadOnly 1

Where,
0 = write mode
1 = read only mode

If the cluster configuration is read-only make it write mode -

# haconf –makerw

# hagrp -add dapos1_sg
# hagrp -modify dapos1_sg SystemList sys1 0 sys3 1 sys4 2 sys2 3
# hagrp -modify dapos1_sg AutoStart 0
# hagrp -modify dapos1_sg AutoStartList sys1
# hagrp -modify dapos1_sg SourceFile "./main.cf"

# hares -add dapos1_dg DiskGroup dapos1_sg
# hares -modify dapos1_dg DiskGroup dapos1dg
# hares -modify dapos1_dg PanicSystemOnDGLoss 0
# hares -modify dapos1_dg StartVolumes 1
# hares -modify dapos1_dg StopVolumes 1
# hares -modify dapos1_dg MonitorReservation 0
# hares -modify dapos1_dg tempUseFence INVALID
# hares -modify dapos1_dg DiskGroupType private
# hares -modify dapos1_dg Enabled 1

# hares -add dapos1_EMCProc EMCProc dapos1_sg
# hares -modify dapos1_EMCProc Enabled 1
# hares -add dapos1_vol Volume dapos1_sg
# hares -modify dapos1_vol Volume dapos1
# hares -modify dapos1_vol DiskGroup dapos1dg
# hares -modify dapos1_vol Enabled 1

# hares -add dapos1_mnt Mount dapos1_sg
# hares -modify dapos1_mnt Critical 0
# hares -modify dapos1_mnt MountPoint "/twm/dapos1"
# hares -modify dapos1_mnt BlockDevice "/dev/vx/dsk/dapos1dg/dapos1"
# hares -modify dapos1_mnt FSType vxfs
# hares -modify dapos1_mnt FsckOpt "%-n"
# hares -modify dapos1_mnt CkptUmount 1
# hares -modify dapos1_mnt SecondLevelMonitor 0
# hares -modify dapos1_mnt SecondLevelTimeout 30
# hares -modify dapos1_mnt Enabled 1

Fine, next part is linking - while linking always remember the syntax -

# hares -link {parent} {child}

Please note in VCS, parent resource depends on child resource, for example in a group where volume depends on DiskGroup resource, DiskGroup will be child & volume would be parent...

# hares -link dapos1_vol dapos1_dg
# hares -link dapos1_mnt dapos1_vol

Upon performing the above configuration you can see the resources added to the service group dapos1_sg.

# hagrp -resources dapos1_sg
dapos1_prod_IPM
dapos1_prod_proxy
dapos1_dg
dapos1_vol
dapos1_mnt

Perform the same procedure for rest of 2 service groups. Once completed check the status of the service groups.

# hastatus -sum | egrep 'dapos1_sg|dapos2_sg|dapos3_sg'
B dapos1_sg sys1 Y N ONLINE
B dapos1_sg sys2 Y N OFFLINE
B dapos1_sg sys3 Y N OFFLINE
B dapos1_sg sys4 Y N OFFLINE
B dapos2_sg sys1 Y N OFFLINE
B dapos2_sg sys2 Y N ONLINE
B dapos2_sg sys3 Y N OFFLINE
B dapos2_sg sys4 Y N OFFLINE
B dapos3_sg sys1 Y N OFFLINE
B dapos3_sg sys2 Y N OFFLINE
B dapos3_sg sys3 Y N ONLINE
B dapos3_sg sys4 Y N OFFLINE

Once completed with the changes do not forget to get the Cluster Server configuration in read-only mode and dump the changes in the main.cf file. You can do the same using below command -

# haconf -dump -makero

Wooh.. we are done with the requirement. It was a good exercise in itself...

HTH.

Dynamically Adding a RAW device to a Solaris zone.

Dynamically Adding a RAW device to a Solaris zone.

By default it is not possible to add raw device to zone without taking a reboot on zone however there is a famous saying => "There is always a way out for those clever enough to find it". So I'm, just kidding, don't take it seriously. :)

I found a little hack to accomplish the objective of adding raw device to zone without rebooting it. Here is a way out -

1) Add the device to the zonecfg

#zonecfg -z barvozone1
zonecfg:barvozone1> add device
zonecfg:barvozone1:device> set match=/dev/rdsk/c3t60050768018A8023B8000000000000F0d0s0
zonecfg:barvozone1:device> end
zonecfg:barvozone1>exit

2) use the mknod command to create the device in the zones dev folder

#ls -l /dev/rdsk/c3t60050768018A8023B8000000000000F0d0s0
lrwxrwxrwx   1 root     root          67 Feb 18 15:34 /dev/rdsk/c3t60050768018A8023B8000000000000F0d0s0 -> ../../devices/scsi_vhci/ssd@g60050768018a8023b8000000000000f0:a,raw

#ls -l /devices/scsi_vhci/ssd@g60050768018a8023b8000000000000f0:a,raw
crw-r-----   1 root     sys      118, 128 Mar  5 23:55 /devices/scsi_vhci/ssd@g60050768018a8023b8000000000000f0:a,raw

# cd /barvozone1/zonepath/dev

# mknod c3t60050768018A8023B8000000000000F0d0s0 c 118 128

That's it. The raw device is now visible within zone and now you can start with your stuffs without any downtime. Isn't it cool?

Identify ASM Disks from OS interface

There are several ways to identify ASM disks from the database interface however sometimes or in fact most of the times it might be the case that the System Engineer/Administrator doesn't have access to Oracle (ASM instance) or he may not aware of database operations so in that case how to recognize that a particular disk is being used and under control of ASM.

There are several occasions we need to order SAN disks/LUNs to either expand volume group, zpool or metadevice to increase a filesystem within it. Now consider, there are already few devices/disks available on system and looks free but if you are unsure whether a specific device/disk/LUN is being used by ASM, you can look at the header of the device. The od (octal dump) command can be used. E.g. od --read-bytes=128 --format=c displays the 1st 128 bytes of a device in character format.

For Solaris distributions –

root:aczfild3s:/var/tmp # od -c -N 128 /dev/rdsk/c5t6006016051402800F07C0E23B0CFDF11d0s0
0000000 \0 202 001 001 \0 \0 \0 \0 200 \0 \0 017 A 200 327 272
0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000040 O R C L D I S K \0 \0 \0 \0 \0 \0 \0 \0
0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000100 \n 020 \0 \0 \0 017 001 003 V 4 S _ P E R _
0000120 D A T A 0 1 _ 0 0 1 5 \0 \0 \0 \0 \0
0000140 \0 \0 \0 \0 \0 \0 \0 \0 V 4 S _ P E R   _
0000160 D A T A 0 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000200

This is a ASM disk as the ORCLDISK string exists in the header. The ASM disk name is V4S_PER_DATA01_0015.

For Linux distributions –


# od --read-bytes=128 --format=c /dev/dm-39
0000000 001 202 001 001 \0 \0 \0 \0 \0 \0 \0 200 w 374 211 343
0000020 244 } 030 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000040 O R C L D I S K \0 \0 \0 \0 \0 \0 \0 \0
0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000100 \0 \0 020 \n \0 \0 001 003 N C A S _ D I S
0000120 K 1 _ 0 0 0 0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000140 \0 \0 \0 \0 \0 \0 \0 \0 N C A S _ D I S
0000160 K 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000200

The ASM disk name is NCAS_DISK1_0000.
 
For AIX distributions –


AIX is one of the “SENSIBLE” distributions I must say. Why? Check this out –

In Linux or Oracle-Sun Solaris the native volume manager or utilities around it does not identify if the disk is being used by ASM and it simply allow to format/erase the disk and allow to add the device to inline filesystem or volume group however AIX native volume manager is a step ahead & smart - disks being used by Oracle ASM cannot be also used with AIX LVM. The main reason for this is because the Oracle ASM software puts information on the raw disk to identify it as an Oracle disk, which wipes out any existing PVID or VGDA information.

If an attempt is made to use LVM on a disk already in an Oracle ASM disk group the following errors will be seen:

root> extendvg -f apps hdiskpower10
0516-1339 extendvg: Physical volume contains some 3rd party volume group.
0516-1397 extendvg: The physical volume hdiskpower10, will not be added to the volume group.
0516-792 extendvg: Unable to extend volume group.

root> mkvg -y lparvg hdiskpower128
0516-1339 mkvg: Physical volume contains some 3rd party volume group.
0516-1397 mkvg: The physical volume hdiskpower128, will not be added to the volume group.
0516-862 mkvg: Unable to create volume group.

LVM knows to search for the Oracle identifier on these disks, and will fail the operation. You can manually check for the identifier by running the lquerypv command.

# lquerypv -h /dev/hdiskpower7
00000000 00820101 00000000 80000000 F7BE76D7 ..............v.00000010 00000000 00000000 00000000 00000000
................00000020 4F52434C 4449534B 00000000 00000000 ORCLDISK........00000030 00000000 00000000 00000000 00000000 ................00000040 0A100000 00000103 44415441 5F303030 ........DATA_00000000050 30000000 00000000 00000000 00000000 0...............00000060 00000000 00000000 44415441 00000000 ........DATA....00000070 00000000 00000000 00000000 00000000 ................00000080 00000000 00000000 44415441 5F303030 ........DATA_00000000090 30000000 00000000 00000000 00000000 0...............000000A0 00000000 00000000 00000000 00000000 ................000000B0 00000000 00000000 00000000 00000000 ................000000C0 00000000 00000000 01F64610 3B306C00 ..........F.;0l. 000000D0 01F64610 3B9C7C00 02001000 00100000 ..F.;. .........000000E0 0001BC80 0000C800 00000002 00000001 ................000000F0 00000002 00000002 00000000 00000000 ................
 
Hope that helps.

Oracle-Solaris Patching using Live Upgrade

In today's post I'll be talking about a procedure for patching Solaris server having zones installed on it using Live Upgrade functionality.

The Solaris OS Recommended Patch Cluster provides critical Solaris OS Security, Data Corruption, and System Availability fixes & hence it is advisable to patch your Solaris systems twice in a year (atleast), as per Oracle-Sun's Critical Patch Update release schedule, I prefer to execute patch cycle for my environment in end of April and sometime late October every year.

Oracle-Sun CPUs are released on the Tuesday closest to the 17th of January, April, July, and October –

See - http://www.oracle.com/technetwork/topics/security/alerts-086861.html

In my environment, I use Live Upgrade to patch our Solaris systems. Reason behind using Live Upgrade for patching purpose are - 

1. Create a copy of the system environment; that is, a copy of the root (/) file system

2. Live Upgrade has build-in feature for splitting the mirrors of an SVM mirrored root (detach, attach, preserve options on lucreate) hence low overhead to deal with SVM mirror break stuffs separately etc.

3. Less downtime (not more than 15-20 mins) and minimal risk.

4. Better back out option. In case something breaks after patching revert to old BE and be at stage from where started, again that doesn’t take much downtime and safe option.

5. The most appropriate option for those Solaris servers who have zones/containers installed on it.

There might be many more benefits out there, however I find above benefits best fit for my purpose.

So to summarize, all tasks except the reboot can be accomplished on an operational production system; the impact on any running process is minimal. Live Upgrade is a combination of maximizing system availability when applying changes and minimizing risk by offering the ability to reboot to a known working state (your original environment).

Well. let's see how to do it in real life, in my current environment we have many servers which uses Solaris Volume Manager as their primary volume manager to manage the disk and data. So, let's take a look at patching procedure to patch servers who have SVM installed and configured on it along with zones installed on it sitting on ZFS filesystem.

Let us grab the output of metastat to understand metadevice placement -

# metastat -c
d32              p  1.0GB d4
d33              p  1.0GB d4
d36              p   40GB d4
d35              p  1.0GB d4
d34              p  4.0GB d4
d60              p   16GB d4
d30              p  1.0GB d4
d31              p  1.0GB d4
    d4           m  100GB d14 d24
        d14      s  100GB c1t0d0s4
        d24      s  100GB c1t1d0s4
d103             m   10GB d23 d13
    d23          s   10GB c1t1d0s3
    d13          s   10GB c1t0d0s3
d100             m   10GB d20 d10
    d20          s   10GB c1t1d0s0
    d10          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

Alright, my / is on d100 and /var is on d103. Let us create an alternative boot environment out of it.

# lucreate -c Sol10 -n Sol10pu -m /:/dev/md/dsk/d0:ufs,mirror -m /:/dev/md/dsk/d20:detach,attach -m /var:/dev/md/dsk/d3:ufs,mirror –m /var:/dev/md/dsk/d23:detach,attach

Here I'm trying to create a metadevice d0 representing / UFS filesystem having a sub-mirror d20 (sub-mirror d20 first gets detach from d100 and then attach to d0). Same thing applicable for /var filesystem and it's meta device configuration.

In above command I'm creating a new boot environment called Sol10pu using option “-n”, option “-m” Specifies the vfstab information for a new UFS-based BE.

NOTE: The -m option is not supported for BEs based on ZFS file systems.

NOTE: In case you're performing upgrade and patching in one go then point to be ponder - Before upgrading, you must install the Oracle Solaris Live Upgrade packages from the release to which you are upgrading. New capabilities are added to the upgrade tools, so installing the new packages from the target release is important. Example, you need to upgrade from Oracle Solaris 10 update 4 to Oracle Solaris update 8, so you must get the Oracle Solaris Live Upgrade packages from the Oracle Solaris update 8 DVD.


Once above command finishes, you will see you meta device configuration changed as follows -

# metastat -c
d32              p  1.0GB d4
d33              p  1.0GB d4
d36              p   40GB d4
d35              p  1.0GB d4
d34              p  4.0GB d4
d60              p   16GB d4
d30              p  1.0GB d4
d31              p  1.0GB d4
    d4           m  100GB d14 d24
        d14      s  100GB c1t0d0s4
        d24      s  100GB c1t1d0s4
d103             m   10GB d23
    d13          s   10GB c1t1d0s3
d100             m   10GB d20
    d10          s   10GB c1t1d0s0
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

d0 and d3 has one sub-mirror and d100 and d100 has one sub-mirror associated.

Also you will be able to see two boot environments on your Solaris system -

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10                      yes      yes    yes       no     -
Sol10pu                    yes      no     no        yes    -


Fine, so now we have 2 boot environments with us and we are going to patch the alternate BE (sol10pu) using a patching tool called PCA, BTW - I use PCA (Patch Check Advance) tool to apply patches to our Solaris systems. PCA has been setup to download patches via local web proxy to access outside systems.

PCA setup tips can be found at - http://www.par.univie.ac.at/solaris/pca/usage.html

What PCA needs in case setting it up -

- PERL distribution
- At least one server which is internet facing (this server will then act as a proxy to rest of servers)
- Patch cross-reference file called patchdiag.xref (latest one always while patching)
- Valid Oracle support (MOS) user ID and password
- If at all required, some wrapper scripts to PCA

To do so, let us mount the alternate BE on mount point say /a

# lumount Sol10pu /a
/a

Now I'll create a temporary directory to download the missing & required patches,

# mkdir -p /patchman/patches
My next job is to generate patch_order file,

# /bin/pca missingrs -R /a -H --format "%p-%c" > /patchman/patches/patch_order

Where -R stands for Alternative root directory
Where -H stands for Don't display descriptive headers
Where --format stands for Set output format to FORMAT

And go get them, download -

# /bin/pca missingrs -R /a -d -P /patchman/patches/ 

Where -d stands for download patches
Where -P stands for Patch download directory

Unmount the ABE -

# luumount /a

Now if you populate the /patchman/patches directory then you will see the list of patches in there.

Unzip all those patches -

# for i in `ls *.zip`;do
  unzip $i
  rm $i
done

Okay, at this stage we are ready upgrade ABE with patches available -

# cd /patchman/patches; luupgrade -n Sol10pu -s /patchman/patches -t `cat patch_order`

NOTE: Reactive patching may occasionally be necessary to address break-and-fix issues so in this case you can use LU with something like -

Apply single patch to ABE -

# luupgrade -n Sol10pu -s /patchman/patches -t

This will update patches on global as well as non-global zones.

Once the patches are installed it will automatically un-mounts the ABE sol10pu mounted on mount point /a.

Now it's time to activate the ABE sol10pu which just been patched using Live Upgrade utility.

# luactivate Sol10pu
A Live Upgrade Sync operation will be performed on startup of boot environment .
**********************************************************************

The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.

**********************************************************************

In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:

1. Enter the PROM monitor (ok prompt).

2. Change the boot device back to the original boot environment by typing:

     setenv boot-device /pci@0/pci@0/pci@2/scsi@0/disk@0,0:a

3. Boot to the original boot environment by typing:

     boot

**********************************************************************

Modifying boot archive service
Activation of boot environment successful.

# init 6
updating /platform/sun4v/boot_archive
SYSTEM GOING DOWN!!!!

NOTE: Live upgrade always uses init 6 or shutdown commands. Halt and reboot commands will create big time bang, be aware!!!

Once system is up it should show new kernel patch version.

Great, it's been a week after patching and application, DB owners are happy with patching stuffs and now we need to perform post patching stuffs upon certain confirmations.

POST PATCHING WORK TO DO - A Week LATER....
=========================================

Now a week later I need to delete the old boot environment and rebuild the metadevices to be in mirror layout.

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10                      yes      no     no        yes    -
Sol10pu                    yes      yes    yes       no     -

# ludelete Sol10
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.

So, the meta devices are looking like as below -

# metastat -c
d32              p  1.0GB d4
d33              p  1.0GB d4
d36              p   40GB d4
d35              p  1.0GB d4
d34              p  4.0GB d4
d60              p   16GB d4
d30              p  1.0GB d4
d31              p  1.0GB d4
    d4           m  100GB d14 d24
        d14      s  100GB c1t0d0s4
        d24      s  100GB c1t1d0s4
d103             m   10GB d23
    d13          s   10GB c1t1d0s3
d100             m   10GB d20
    d10          s   10GB c1t1d0s0
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

Now clear the d100 & d103 mirrors.

# metaclear d100 d103
d100: Mirror is cleared
d103: Mirror is cleared

# metastat -c
d32              p  1.0GB d4
d33              p  1.0GB d4
d36              p   40GB d4
d35              p  1.0GB d4
d34              p  4.0GB d4
d60              p   16GB d4
d30              p  1.0GB d4
d31              p  1.0GB d4
    d4           m  100GB d14 d24
        d14      s  100GB c1t0d0s4
        d24      s  100GB c1t1d0s4
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1
d13              s   10GB c1t1d0s3
d10              s   10GB c1t1d0s0

Next attach the sub-mirrors d10 & d13 to metadevices d0 and d3 respectively.

# metattach d0 d10
d0: submirror d10 is attached

# metattach d3 d13
d3: submirror d13 is attached

Hence my final meta device placement looks like as follows -

# metastat -c
d32              p  1.0GB d4
d33              p  1.0GB d4
d36              p   40GB d4
d35              p  1.0GB d4
d34              p  4.0GB d4
d60              p   16GB d4
d30              p  1.0GB d4
d31              p  1.0GB d4
    d4           m  100GB d14 d24
        d14      s  100GB c1t0d0s4
        d24      s  100GB c1t1d0s4
d3               m   10GB d23 d13 (resync-25%)
    d23          s   10GB c1t1d0s3
    d13          s   10GB c1t0d0s3
d0               m   10GB d20 d10 (resync-45%)
    d20          s   10GB c1t1d0s0
    d10          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

That's it. Now your done with patching your Solaris server and zones deployed on it.

Migrating VERITAS Volumes from one storage array to another storage array

Background:

This week I got a task to Migrate Veritas disk groups & their volumes on a specific server from one storage array to new storage array due to old storage array is ruining out of capacity.

To execute this task I split this into 3 phases -

1. Responsibilities with Storage

     a. Target server has to be zoned to new Storage system
     b. Provide LUNs/capacity to server (total capacity required is 3.5 TB 58x60G & 2x15G LUNs)

2. Responsibilities with me

      a. Migrate Veritas disk groups to new storage array
            A) Label all newly added SAN disks (may be using some scripted method) & make sure SAN disks are visible under VxVM
            B) Initialize all those new SAN disks with VxVM
            C) Add the SAN disk to disk group
            D) Mirror the volumes
            E) Verify if sync is completed
            F) Verify from vxprint if you see a new plex added to designated volumes
            G) If all is well then go ahead and detach old plex
            H) Once plex is disassociated from designated volumes then delete the old plex
            I) Verify if data group(s) & their volumes are on new storage array.
     b. Remove disks associated with old storage from Veritas configuration

3. Responsibilities with Storage

     a. Remove disks associated with old storage from server
     b. Take care of redundant paths

Back out -

     a. Have server full backup handy.

Execution -

List the disk groups needs to be migrated to new storage array.

# vxdg list
NAME STATE ID
xxxx_dg enabled,cds 1279726733.18.xxxxx
localswdg enabled,cds 1279726567.16.xxxxx
nass3_dg enabled 1074844579.1535.nassau3

Get the true picture of your VxVM configuration. Save output of this command for future reference

# vxprint -hrt

Now that Storage has attached SAN disks to servers HBA we need to label them and get them to VERITAS control.

List the existing Disks & new disks detected.

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
UNIX176_0a7a auto:sliced fls02c4_nass3_dg nass3_dg online <<<< All UNIX# disks are from old storage arrary
UNIX176_0a7f auto:sliced fls03c1_nass3_dg nass3_dg online
UNIX176_0a73 auto:sliced fls01c1_nass3_dg nass3_dg online
UNIX176_0a74 auto:sliced fls01c2_nass3_dg nass3_dg online
UNIX176_0a75 auto:sliced fls01c3_nass3_dg nass3_dg online
UNIX176_0a76 auto:sliced fls01c4_nass3_dg nass3_dg online
UNIX176_0a77 auto:sliced fls02c1_nass3_dg nass3_dg online
UNIX176_0a78 auto:sliced fls02c2_nass3_dg nass3_dg online
UNIX176_0a79 auto:sliced fls02c3_nass3_dg nass3_dg online
UNIX176_0a80 auto:sliced fls03c2_nass3_dg nass3_dg online
UNIX176_0a81 auto:sliced fls03c3_nass3_dg nass3_dg online
UNIX176_0a82 auto:sliced fls03c4_nass3_dg nass3_dg online
UNIX176_0dbf auto:cdsdisk xxxx_dg02 xxxxx_dg online
UNIX176_07f8 auto:cdsdisk localswdg01 localswdg online
UNIX176_07f9 auto:cdsdisk xxxx_dg01 xxxx_dg online
UNIX176_09a0 auto:sliced fls09c2_nass3_dg nass3_dg online
UNIX176_09aa auto:sliced fls11c4_nass3_dg nass3_dg online
UNIX176_09ab auto:sliced fls12c1_nass3_dg nass3_dg online
UNIX176_191c auto:cdsdisk localswdg02 localswdg online
UNIX176_0990 auto:sliced fls05c2_nass3_dg nass3_dg online
UNIX176_0991 auto:sliced fls05c3_nass3_dg nass3_dg online

[...]

disk_0 auto:none - - online invalid
disk_1 auto:none - - online invalid
emc_clariion0_1704 auto - - online invalid   <<<< All emc_* disks are from new storage arrary
emc_clariion0_1705 auto - - online invalid
emc_clariion0_1706 auto - - online invalid
emc_clariion0_1707 auto - - online invalid
emc_clariion0_1708 auto - - online invalid
emc_clariion0_1709 auto - - online invalid
emc_clariion0_1710 auto - - online invalid
emc_clariion0_1713 auto - - online invalid
emc_clariion0_1800 auto - - online invalid
emc_clariion0_1801 auto - - online invalid
emc_clariion0_1803 auto - - online invalid
emc_clariion0_1804 auto - - online invalid
emc_clariion0_1805 auto - - online invalid
emc_clariion0_1806 auto - - online invalid
emc_clariion0_1807 auto - - online invalid
emc_clariion0_1808 auto - - online invalid
emc_clariion0_1809 auto - - online invalid
emc_clariion0_1810 auto - - online invalid
emc_clariion0_3700 auto - - online invalid

[...]

emc_clariion0_5809 auto - - online invalid
emc_clariion0_5858 auto - - online invalid

Initialize the disks with VxVM and add them to appropriate disk group.

For a single disk -

# vxdisksetup -i emc_clariion0_5858 format=sliced

For multiple disks

# vxdisk list | awk '{print $1}' | grep -i emc > /tmp/EMC_disks
# for d in `cat /tmp/EMC_disks` ; do vxdisksetup -i $d format=sliced; done

Add disks to disk group

#!/bin/sh
#for e.g. vxdg -g nass3_dg adddisk nass3_dg01=emc_clariion0_1704
DG=nass3_dg
DISKS=`vxdisk list | awk '{print $3"="$1}' | grep -i emc`
#Output of command vxdisk list | awk '{print $3"="$1}' | grep -i emc is look like as = nass3_dg01=emc_clariion0_1704
for d in $DISKS;
do vxdg -g $DG adddisk $DISKS;
done

OR you can also use vxdiskadm menu based command to perfrom this activity.

Now you should see something like -

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
UNIX176_0a7a auto:sliced fls02c4_nass3_dg nass3_dg online
UNIX176_0a7f auto:sliced fls03c1_nass3_dg nass3_dg online
UNIX176_0a73 auto:sliced fls01c1_nass3_dg nass3_dg online
UNIX176_0a74 auto:sliced fls01c2_nass3_dg nass3_dg online
UNIX176_0a75 auto:sliced fls01c3_nass3_dg nass3_dg online
UNIX176_0a76 auto:sliced fls01c4_nass3_dg nass3_dg online
UNIX176_0a77 auto:sliced fls02c1_nass3_dg nass3_dg online
UNIX176_0a78 auto:sliced fls02c2_nass3_dg nass3_dg online
UNIX176_0a79 auto:sliced fls02c3_nass3_dg nass3_dg online
UNIX176_0a80 auto:sliced fls03c2_nass3_dg nass3_dg online
UNIX176_0a81 auto:sliced fls03c3_nass3_dg nass3_dg online
UNIX176_0a82 auto:sliced fls03c4_nass3_dg nass3_dg online
UNIX176_0dbf auto:cdsdisk xxxx_dg02 xxxxx_dg online
UNIX176_07f8 auto:cdsdisk localswdg01 localswdg online
UNIX176_07f9 auto:cdsdisk xxxx_dg01 xxxx_dg online
UNIX176_09a0 auto:sliced fls09c2_nass3_dg nass3_dg online
UNIX176_09aa auto:sliced fls11c4_nass3_dg nass3_dg online
UNIX176_09ab auto:sliced fls12c1_nass3_dg nass3_dg online
UNIX176_191c auto:cdsdisk localswdg02 localswdg online
UNIX176_0990 auto:sliced fls05c2_nass3_dg nass3_dg online
UNIX176_0991 auto:sliced fls05c3_nass3_dg nass3_dg online

[...]

disk_0 auto:none - - online invalid
disk_1 auto:none - - online invalid
emc_clariion0_1704 auto:sliced nass3_dg01 nass3_dg online
emc_clariion0_1705 auto:sliced nass3_dg02 nass3_dg online
emc_clariion0_1706 auto:sliced nass3_dg03 nass3_dg online
emc_clariion0_1707 auto:sliced nass3_dg04 nass3_dg online
emc_clariion0_1708 auto:sliced nass3_dg05 nass3_dg online
emc_clariion0_1709 auto:sliced nass3_dg06 nass3_dg online
emc_clariion0_1710 auto:sliced nass3_dg07 nass3_dg online
emc_clariion0_1713 auto:sliced nass3_dg08 nass3_dg online
emc_clariion0_1800 auto:sliced nass3_dg09 nass3_dg online
emc_clariion0_1801 auto:sliced nass3_dg10 nass3_dg online
emc_clariion0_1803 auto:sliced nass3_dg11 nass3_dg online
emc_clariion0_1804 auto:sliced nass3_dg12 nass3_dg online
emc_clariion0_1805 auto:sliced nass3_dg13 nass3_dg online
emc_clariion0_1806 auto:sliced nass3_dg14 nass3_dg online
emc_clariion0_1807 auto:sliced nass3_dg15 nass3_dg online
emc_clariion0_1808 auto:sliced nass3_dg16 nass3_dg online
emc_clariion0_1809 auto:sliced nass3_dg17 nass3_dg online
emc_clariion0_1810 auto:sliced nass3_dg18 nass3_dg online
emc_clariion0_3700 auto:sliced nass3_dg19 nass3_dg online

[...]

emc_clariion0_5809 auto:sliced nass3_dg59 nass3_dg online

Now that we have added new disks to the appropriate datagroups, next task to mirror the volumes.

NOTE: To decide how many disks are require to mirroe a perticular volume is very easy, put those many disks which will meet the current size of volume.

# vxassist -g nass3_dg mirror db_TESTDB_vol alloc=nass3_dg05,nass3_dg02,nass3_dg06,nass3_dg07,nass3_dg08,nass3_dg09,nass3_dg10,nass3_dg11,nass3_dg12,nass3_dg13,nass3_dg14,nass3_dg15,nass3_dg16,nass3_dg17,nass3_dg18,nass3_dg19,nass3_dg20,nass3_dg21,nass3_dg22,nass3_dg23,nass3_dg24,nass3_dg25,nass3_dg26,nass3_dg27,nass3_dg28,nass3_dg29,nass3_dg30,nass3_dg31,nass3_dg32,nass3_dg33,nass3_dg34,nass3_dg35,nass3_dg36,nass3_dg37,nass3_dg38,nass3_dg39,nass3_dg40,nass3_dg41,nass3_dg42,nass3_dg43,nass3_dg44,nass3_dg45,nass3_dg46,nass3_dg47,nass3_dg48,nass3_dg49,nass3_dg50,nass3_dg51,nass3_dg52,nass3_dg53,nass3_dg54,nass3_dg55

Repeat the previous step for rest of the data groups and volumes.

Likewise you can add disks to the remaining volumes and then we have to mirror them.

Check sync progress using -

# vxtask -l list
Task: 5912 RUNNING
Type: ATCOPY
Operation: PLXATT Vol db_TESTDB_vol Plex db_TESTDB_vol-02 Dg nass3_dg
Started: Fri Sep 16 16:05:24 2011
Throttle: 0
Progress: 4.27% 268713984 of 6291456000 Blocks
Work time: 29 minutes, 30 seconds (11:01:11 remaining)

Verify from vxprint , you should see a new plex added to db_TESTDB_vol volume

# vxprint -qthg nass3_dg db_TESTDB_vol

If everything looks good, then detach/disassociate & remove old plex, in short - break the mirror. (Before doing so get application owner consent)

# vxmend -g nass3_dg off db_TESTDB_vol-01
# vxplex -g nass3_dg -o rm dis db_TESTDB_vol-01

Repeat the previous two steps for rest of the plexes.

Well, by this time we can say we are done so verify if datagroup(s) & their volumes are on new storage arrary.

# vxprint -qthg nass3_dg db_TESTDB_vol
v db_TESTDB_vol - ENABLED ACTIVE 6291456000 SELECT db_TESTDB_vol-02 fsgen
pl db_TESTDB_vol-02 db_TESTDB_vol ENABLED ACTIVE 6291456000 STRIPE 3/128 RW
sd nass3_dg05-01 db_TESTDB_vol-02 nass3_dg05 0 125754880 0/0 emc_clariion0_1708 ENA
sd nass3_dg08-01 db_TESTDB_vol-02 nass3_dg08 0 125754880 0/125754880 emc_clariion0_1713 ENA
sd nass3_dg11-01 db_TESTDB_vol-02 nass3_dg11 0 125754880 0/251509760 emc_clariion0_1803 ENA
sd nass3_dg14-01 db_TESTDB_vol-02 nass3_dg14 0 125754880 0/377264640 emc_clariion0_1806 ENA
sd nass3_dg17-01 db_TESTDB_vol-02 nass3_dg17 0 125754880 0/503019520 emc_clariion0_1809 ENA
sd nass3_dg20-01 db_TESTDB_vol-02 nass3_dg20 0 125754880 0/628774400 emc_clariion0_3702 ENA
sd nass3_dg23-01 db_TESTDB_vol-02 nass3_dg23 0 125754880 0/754529280 emc_clariion0_3705 ENA
sd nass3_dg26-01 db_TESTDB_vol-02 nass3_dg26 0 125754880 0/880284160 emc_clariion0_3708 ENA
sd nass3_dg29-01 db_TESTDB_vol-02 nass3_dg29 0 125754880 0/1006039040 emc_clariion0_3800 ENA
sd nass3_dg32-01 db_TESTDB_vol-02 nass3_dg32 0 125754880 0/1131793920 emc_clariion0_3803 ENA

[...]

sd nass3_dg43-01 db_TESTDB_vol-02 nass3_dg43 0 125754880 2/1594132480 emc_clariion0_5704 ENA
sd nass3_dg46-01 db_TESTDB_vol-02 nass3_dg46 0 125754880 2/1719887360 emc_clariion0_5707 ENA
sd nass3_dg50-01 db_TESTDB_vol-02 nass3_dg50 0 125754880 2/1845642240 emc_clariion0_5800
ENA
sd nass3_dg53-01 db_TESTDB_vol-02 nass3_dg53 0 125754880 2/1971397120 emc_clariion0_5803 ENA

Yes, volume has been moved to new storage.

Now we are ready to remove disks associated with old storage from Veritas-configuration.

# vxdg -g nass3_dg rmdisk fls02c4_nass3_dg

Ask storage team to detach/remove the old disks permanently from the server.

This is overall procedure to migrate your data groups & their volumes from one storage to another storage.