Add VERITAS Volume Manager Disk Group & Filesystem into VERITAS Cluster service group - from the scratch orientation

In this article we will see how to create a new disk group, create 4-way stripe VxVM filesystem & then will see how do we create a new VCS service group and incorporate the newly created Disk Group & VxFS filesystem to the service group.
So let's roll.

Well, what's the requirement? so, here we go!

• Create 3 diskgroups namely dapos1dg, dapos2dg, dapos3dg.
• Create three VxVM filesystems within above three DiskGroup /twm/dapos1, /twm/dapos2, /twm/dapos3  with 4/128 stripe layout.
• Create three VERITAS service groups dapos1_sg, dapos2_sg & dapos3_sg.
• Insert/attach/link the needed infra resources (DiskGroup, filesystem, ..) into the corresponding VERITAS service groups.

Solution Delivery –

In order to create 4/128 stripe layout volumes/filesystems we need to get storage disks in bunch of 4 hence I ordered for 12x15G LUNs. Each DiskGroup will have 4 disks.

Well, below are the LUNs that we have from storage –

c2t5006048C52A8CCC3d40
c2t5006048C52A8CCC3d41
c2t5006048C52A8CCC3d42
c2t5006048C52A8CCC3d43
c2t5006048C52A8CCC3d44
c2t5006048C52A8CCC3d45
c2t5006048C52A8CCC3d46
c2t5006048C52A8CCC3d47
c2t5006048C52A8CCC3d48
c2t5006048C52A8CCC3d49
c2t5006048C52A8CCC3d50
c2t5006048C52A8CCC3d51

Before formatting the disks verify if someone hasn’t used these disks for any other purpose. (it happens sometimes, some lazy frogs uses unused disks without taking much efforts and without knowing may someone has ordered these LUNs for some purpose)

# for disk in `cat disks`;do vxdisk -eo alldgs list | grep $disk;done
EMC0_223 auto - - online c2t5006048C52A8CCC3d40s2
EMC0_224 auto - - online c2t5006048C52A8CCC3d41s2
EMC0_225 auto - - online c2t5006048C52A8CCC3d42s2
EMC0_229 auto - - online c2t5006048C52A8CCC3d43s2
EMC0_226 auto - - online c2t5006048C52A8CCC3d44s2
EMC0_227 auto - - online c2t5006048C52A8CCC3d45s2
EMC0_228 auto - - online c2t5006048C52A8CCC3d46s2
EMC0_231 auto - - online c2t5006048C52A8CCC3d47s2
EMC0_222 auto - - online c2t5006048C52A8CCC3d48s2
EMC0_221 auto - - online c2t5006048C52A8CCC3d49s2
EMC0_220 auto - - online c2t5006048C52A8CCC3d50s2
EMC0_230 auto - - online c2t5006048C52A8CCC3d51s2

Good. Disks are not being used by any VERITAS DG so let's label the disks with SMI label.

 Once done with labelling the disks make sure you run vxdctl enable command & re-scan for new disks devices so that the disks are visible to VxVM.

# vxdctl enable

Now we are all set to get all new disks to VERITAS control. Before taking disks into VxVM control verify if those are visible to VxVM.

# for lun in `cat /tmp/lun-list`; do vxdisk -eo alldgs list | grep -i $lun; done
EMC0_223 auto - - online c2t5006048C52A8CCC3d40s2
EMC0_224 auto - - online c2t5006048C52A8CCC3d41s2
EMC0_225 auto - - online c2t5006048C52A8CCC3d42s2
EMC0_229 auto - - online c2t5006048C52A8CCC3d43s2
EMC0_226 auto - - online c2t5006048C52A8CCC3d44s2
EMC0_227 auto - - online c2t5006048C52A8CCC3d45s2
EMC0_228 auto - - online c2t5006048C52A8CCC3d46s2
EMC0_231 auto - - online c2t5006048C52A8CCC3d47s2
EMC0_222 auto - - online c2t5006048C52A8CCC3d48s2
EMC0_221 auto - - online c2t5006048C52A8CCC3d49s2
EMC0_220 auto - - online c2t5006048C52A8CCC3d50s2
EMC0_230 auto - - online c2t5006048C52A8CCC3d51s2

Now let's initialize the new disks and get them under VERITAS control.

Get the list of new disks visible to VxVM using below command.

# for lun in `cat lun-list`; do vxdisk -e list | grep -i $lun; done | awk '{print $1}' > /tmp/vxvm_disks && cat /tmp/vxvm_disks
EMC0_223
EMC0_224
EMC0_225
EMC0_229
EMC0_226
EMC0_227
EMC0_228
EMC0_231
EMC0_222
EMC0_221
EMC0_220
EMC0_230

Initialize new disks using below command.

# for disk in `cat /tmp/vxvm_disks`;do vxdisksetup -i $disk format=sliced; done

See if all the new disks are online and under VxVM control.

# for lun in `cat vxvm_disks`; do vxdisk list | grep -i $lun; done | sort
EMC0_220 auto:sliced - - online
EMC0_221 auto:sliced - - online
EMC0_222 auto:sliced - - online
EMC0_223 auto:sliced - - online
EMC0_224 auto:sliced - - online
EMC0_225 auto:sliced - - online
EMC0_226 auto:sliced - - online
EMC0_227 auto:sliced - - online
EMC0_228 auto:sliced - - online
EMC0_229 auto:sliced - - online
EMC0_230 auto:sliced - - online
EMC0_231 auto:sliced - - online

Now let's create diskgroups –

NOTE: As these are not the shared disk groups make sure you create them on individual systems where these diskgroups needs to be mounted or else create all three disks groups on one system and then deport –import them on respective systems. Either way!

# vxdg init dapos1dg cds=off s01c0_dapos1dg=EMC0_220 s01c1_dapos1dg=EMC0_221 s01c2_dapos1dg=EMC0_222 s01c3_dapos1dg=EMC0_223
# vxdg init dapos2dg cds=off s01c0_dapos2dg=EMC0_224 s01c1_dapos2dg=EMC0_225 s01c2_dapos2dg=EMC0_226 s01c3_dapos2dg=EMC0_227
# vxdg init dapos3dg cds=off s01c0_dapos3dg=EMC0_228 s01c1_dapos3dg=EMC0_229 s01c2_dapos3dg=EMC0_230 s01c3_dapos3dg=EMC0_231

# vxdg list |egrep 'dapos1dg|dapos2dg|dapos3dg'
dapos1dg enabled 1330089054.668.sys1
dapos2dg enabled 1330089075.670.sys1
dapos3dg enabled 1330089087.672.sys1

Now we will create STRIPE volumes within these newly created three diskgroups.

# vxassist -g dapos1dg make dapos1 3g layout=stripe s01c0_dapos1dg s01c1_dapos1dg s01c2_dapos1dg s01c3_dapos1dg
# vxassist -g dapos2dg make dapos2 3g layout=stripe s01c0_dapos2dg s01c1_dapos2dg s01c2_dapos2dg s01c3_dapos2dg
# vxassist -g dapos3dg make dapos3 3g layout=stripe s01c0_dapos3dg s01c1_dapos3dg s01c2_dapos3dg s01c3_dapos3dg

Well, the volumes are created. Let’s check them out.

# for i in dapos1dg dapos2dg dapos3dg; do vxprint -htg $i | grep -i dapos;done
dg dapos1dg default default 16000 1330089054.668.sys1
dm s01c0_dapos1dg EMC0_220 auto 17023 31420800 -
dm s01c1_dapos1dg EMC0_221 auto 17023 31420800 -
dm s01c2_dapos1dg EMC0_222 auto 17023 31420800 -
dm s01c3_dapos1dg EMC0_223 auto 17023 31420800 -
v dapos1 - ENABLED ACTIVE 6291456 SELECT dapos1-01 fsgen
pl dapos1-01 dapos1 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos1dg-01 dapos1-01 s01c0_dapos1dg 0 1572864 0/0 EMC0_220 ENA
sd s01c1_dapos1dg-01 dapos1-01 s01c1_dapos1dg 0 1572864 1/0 EMC0_221 ENA
sd s01c2_dapos1dg-01 dapos1-01 s01c2_dapos1dg 0 1572864 2/0 EMC0_222 ENA
sd s01c3_dapos1dg-01 dapos1-01 s01c3_dapos1dg 0 1572864 3/0 EMC0_223 ENA
dg dapos2dg default default 68000 1330089075.670.sys1
dm s01c0_dapos2dg EMC0_224 auto 17023 31420800 -
dm s01c1_dapos2dg EMC0_225 auto 17023 31420800 -
dm s01c2_dapos2dg EMC0_226 auto 17023 31420800 -
dm s01c3_dapos2dg EMC0_227 auto 17023 31420800 -
v dapos2 - ENABLED ACTIVE 6291456 SELECT dapos2-01 fsgen
pl dapos2-01 dapos2 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos2dg-01 dapos2-01 s01c0_dapos2dg 0 1572864 0/0 EMC0_224 ENA
sd s01c1_dapos2dg-01 dapos2-01 s01c1_dapos2dg 0 1572864 1/0 EMC0_225 ENA
sd s01c2_dapos2dg-01 dapos2-01 s01c2_dapos2dg 0 1572864 2/0 EMC0_226 ENA
sd s01c3_dapos2dg-01 dapos2-01 s01c3_dapos2dg 0 1572864 3/0 EMC0_227 ENA
dg dapos3dg default default 20000 1330089087.672.sys1
dm s01c0_dapos3dg EMC0_228 auto 17023 31420800 -
dm s01c1_dapos3dg EMC0_229 auto 17023 31420800 -
dm s01c2_dapos3dg EMC0_230 auto 17023 31420800 -
dm s01c3_dapos3dg EMC0_231 auto 17023 31420800 -
v dapos3 - ENABLED ACTIVE 6291456 SELECT dapos3-01 fsgen
pl dapos3-01 dapos3 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos3dg-01 dapos3-01 s01c0_dapos3dg 0 1572864 0/0 EMC0_228 ENA
sd s01c1_dapos3dg-01 dapos3-01 s01c1_dapos3dg 0 1572864 1/0 EMC0_229 ENA
sd s01c2_dapos3dg-01 dapos3-01 s01c2_dapos3dg 0 1572864 2/0 EMC0_230 ENA
sd s01c3_dapos3dg-01 dapos3-01 s01c3_dapos3dg 0 1572864 3/0 EMC0_231 ENA

Next thing is to create a mount point for these volume & create a file system on this volume.

On individual servers…

# for fs in 1 2 3;do mkdir -p /twm/dapos$fs;done && ls –ld /twm/dapos*
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos1
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos2
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos3

Not required to be done on individual servers as the storage disks are shared between cluster nodes.

# mkfs -F vxfs /dev/vx/rdsk/dapos1dg/dapos1
# mkfs -F vxfs /dev/vx/rdsk/dapos2dg/dapos2
# mkfs -F vxfs /dev/vx/rdsk/dapos3dg/dapos3

First time needed only on respective individual servers.

# mount -F vxfs /dev/vx/dsk/dapos1dg/dapos1 /twm/dapos1

Now, furthermore there is requirement to create new VCS service groups & Insert/attach the needed infra resources (DiskGroup, filesystem, ..) into the corresponding VERITAS service groups. Let’s on it!

Before doing anything, first thing to check if cluster configuration is read-only or in write mode.

# haclus -display |grep -i 'readonly'
ReadOnly 1

Where,
0 = write mode
1 = read only mode

If the cluster configuration is read-only make it write mode -

# haconf –makerw

# hagrp -add dapos1_sg
# hagrp -modify dapos1_sg SystemList sys1 0 sys3 1 sys4 2 sys2 3
# hagrp -modify dapos1_sg AutoStart 0
# hagrp -modify dapos1_sg AutoStartList sys1
# hagrp -modify dapos1_sg SourceFile "./main.cf"

# hares -add dapos1_dg DiskGroup dapos1_sg
# hares -modify dapos1_dg DiskGroup dapos1dg
# hares -modify dapos1_dg PanicSystemOnDGLoss 0
# hares -modify dapos1_dg StartVolumes 1
# hares -modify dapos1_dg StopVolumes 1
# hares -modify dapos1_dg MonitorReservation 0
# hares -modify dapos1_dg tempUseFence INVALID
# hares -modify dapos1_dg DiskGroupType private
# hares -modify dapos1_dg Enabled 1

# hares -add dapos1_EMCProc EMCProc dapos1_sg
# hares -modify dapos1_EMCProc Enabled 1
# hares -add dapos1_vol Volume dapos1_sg
# hares -modify dapos1_vol Volume dapos1
# hares -modify dapos1_vol DiskGroup dapos1dg
# hares -modify dapos1_vol Enabled 1

# hares -add dapos1_mnt Mount dapos1_sg
# hares -modify dapos1_mnt Critical 0
# hares -modify dapos1_mnt MountPoint "/twm/dapos1"
# hares -modify dapos1_mnt BlockDevice "/dev/vx/dsk/dapos1dg/dapos1"
# hares -modify dapos1_mnt FSType vxfs
# hares -modify dapos1_mnt FsckOpt "%-n"
# hares -modify dapos1_mnt CkptUmount 1
# hares -modify dapos1_mnt SecondLevelMonitor 0
# hares -modify dapos1_mnt SecondLevelTimeout 30
# hares -modify dapos1_mnt Enabled 1

Fine, next part is linking - while linking always remember the syntax -

# hares -link {parent} {child}

Please note in VCS, parent resource depends on child resource, for example in a group where volume depends on DiskGroup resource, DiskGroup will be child & volume would be parent...

# hares -link dapos1_vol dapos1_dg
# hares -link dapos1_mnt dapos1_vol

Upon performing the above configuration you can see the resources added to the service group dapos1_sg.

# hagrp -resources dapos1_sg
dapos1_prod_IPM
dapos1_prod_proxy
dapos1_dg
dapos1_vol
dapos1_mnt

Perform the same procedure for rest of 2 service groups. Once completed check the status of the service groups.

# hastatus -sum | egrep 'dapos1_sg|dapos2_sg|dapos3_sg'
B dapos1_sg sys1 Y N ONLINE
B dapos1_sg sys2 Y N OFFLINE
B dapos1_sg sys3 Y N OFFLINE
B dapos1_sg sys4 Y N OFFLINE
B dapos2_sg sys1 Y N OFFLINE
B dapos2_sg sys2 Y N ONLINE
B dapos2_sg sys3 Y N OFFLINE
B dapos2_sg sys4 Y N OFFLINE
B dapos3_sg sys1 Y N OFFLINE
B dapos3_sg sys2 Y N OFFLINE
B dapos3_sg sys3 Y N ONLINE
B dapos3_sg sys4 Y N OFFLINE

Once completed with the changes do not forget to get the Cluster Server configuration in read-only mode and dump the changes in the main.cf file. You can do the same using below command -

# haconf -dump -makero

Wooh.. we are done with the requirement. It was a good exercise in itself...

HTH.