Migrating disks within coordinator disk group from one array to another.

Last week I was working on storage migration tasks in which I envisioned to migrate disks within coordinator disk group from old array to newly deployed array.

To me, it was first time doing this task so I decided to note down the steps and after successfully completing the task decided to share with my all friends out there!

There can be several methods to do this task however for me below steps worked very well but if anyone has any better set of instructions doing this then request to share those.

Let's start then -

1. If VCS is running, shut it down locally:

# hastop -all -force

2. Stop I/O fencing on all nodes. This removes any registration keys on the disks.

# /etc/init.d/vxfen stop   (on all 3 cluster nodes)

3. Import the coordinator disk group. The file /etc/vxfendg includes the name of the disk group (for example, vxfencoorddg) that contains the coordinator disks, so use the command -

# vxdg -tfC import 'cat /etc/vxfendg'
                                       OR
# vxdg -tfC import vxfencoorddg

Where:

-t specifies that the disk group is imported only until the system restarts.
-f specifies that the import is to be done forcibly, which is necessary if one or more disks is not accessible.
-C specifies that any import blocks are removed.

4. Turn off the coordinator attribute value for the coordinator disk group.

# vxdg -g vxfencoorddg set coordinator=off

5. Remove old disks and add new disks

    First remove N-1 disks from fence disk group
 

       # vxdg -g vxfencoorddg rmdisk vxfencoorddg01
       # vxdg -g vxfencoorddg rmdisk vxfencoorddg02
       # vxdg -g vxfencoorddg rmdisk vxfencoorddg03
       # vxdg -g vxfencoorddg rmdisk vxfencoorddg04
       # vxdg -g vxfencoorddg rmdisk vxfencoorddg05
       # vxdg -g vxfencoorddg rmdisk vxfencoorddg06

        Add disk for fence disk group

       # vxdisk list | egrep 'apevmx13_139|apevmx13_140|apevmx13_143|apevmx13_145|apevmx14_139|apevmx14_140|apevmx14_141'
    apevmx13_139 auto:cdsdisk    -            -            online
    apevmx13_140 auto:cdsdisk    -            -            online
    apevmx13_143 auto:cdsdisk    -            -            online
    apevmx13_145 auto:cdsdisk    -            -            online
    apevmx14_139 auto:cdsdisk    -            -            online
    apevmx14_140 auto:cdsdisk    -            -            online
    apevmx14_141 auto:cdsdisk    -            -            online

    # vxdg -g vxfencoorddg adddisk vxfencoorddg01=apevmx13_139
    # vxdg -g vxfencoorddg adddisk vxfencoorddg02=apevmx13_140
    # vxdg -g vxfencoorddg adddisk vxfencoorddg03=apevmx13_143
    # vxdg -g vxfencoorddg adddisk vxfencoorddg04=apevmx13_145
    # vxdg -g vxfencoorddg adddisk vxfencoorddg05=apevmx14_139
    # vxdg -g vxfencoorddg adddisk vxfencoorddg06=apevmx14_140
   
    Remove the remaining one disk from enclosure apedmx06

    # vxdg -g vxfencoorddg rmdisk vxfencoorddg07

    Add the 7th disk from enclosure apevmx14

    # vxdg -g vxfencoorddg adddisk vxfencoorddg07=apevmx14_141

6. Set the coordinator attribute value as "on" for the coordinator disk group.

# vxdg -g vxfencoorddg set coordinator=on

7. Run disk scan on all nodes

# vxdisk scandisks   (Run on all cluster nodes)

8. Check if fencing disks are visible on all nodes

# vxdisk -o alldgs list | grep fen

9. After replacing disks in a coordinator disk group, deport the disk group:

# vxdg deport 'cat /etc/vxfendg'
                                 OR
# vxdg deport vxfencoorddg

10. Verify if the fencing diskgroup is deported

# vxdisk -o alldgs list | grep fen

11. On each node in the cluster, start the I/O fencing driver:

# /etc/init.d/vxfen start  (on all 3 cluster nodes)

12. hastart on all cluster nodes. 

# hastart  (on all 3 cluster nodes)

That's it, these 12 steps takes you through migrating disks within coordinator disk group from one array to another.

HTH someone!

Increase Cluster File System - CVM Cluster

In this post I would like to discuss and demonstrate about increasing the Cluster File System (CFS) in CVM environment.

    - The requirement is to increase the filesystem by 1TB.

As a best practice, in a CVM/CFS environment, volume should be grown on CVM master and file system should be grown on a CFS Primary. Please note that the CVM Master and the SF CFS node can be two different nodes.

Just off the topic, to know how to grow volume on CVM master & then grow the filesystem on CFS Primary -

To increase the size of the file system, execute the following on CVM master -

# vxassist –g shared_disk_group growto volume_name newlength

And then on CFS node, execute-

# fsadm –F vxfs –b newsize –r device_name mount_point

On other hand, if the system is both CVM master and CFS primary, then "vxresize" command could be executed on the system without any issues.

The above mentioned statement is true below VERITAS version 3.5 but from and above VERITAS version 3.5 vxresize can be run on any nodes within the cluster provided that attribute named 'HacliUserLevel' is set to value "COMMANDROOT". The default value of this attribute is 'NONE' which prevents user to run vxresize command from all the nodes in the cluster.

In my case it is set to "COMMANDROOT"

# /opt/VRTSvcs/bin/haclus -display | grep Hacli
HacliUserLevel         COMMANDROOT

But if the value of attribute 'HacliUserLevel' is set to "NONE" then below is method can be used to change the value.

To change the value to COMMANDROOT, run:

# /opt/VRTSvcs/bin/haconf -makerw
# /opt/VRTSvcs/bin/haclus -modify HacliUserLevel COMMANDROOT
# /opt/VRTSvcs/bin/haconf -dump -makero

This change allows vxresize to call hacli command, which then allows any command to be run on any system within the cluster.

Okay, back to our requirement -

So first let's find out the CFS primary node -

# fsclustadm -v showprimary /u01-zz/oradata
XXXXX

Now let's find out the master CVM node in the cluster by -

# vxdctl -c mode
mode: enabled: cluster active - SLAVE
master: XXXXX

Now that I need to increase the filesystem by 1TB hence I'm checking if disk group has enough space in it.

# vxassist -g dde1ZZGO0 maxsize
Maximum volume size: 2136743936 (1043332Mb)

Well, we have enough space available under disk group dde1ZZGO0.

Let increase the FS by 1TB now -

# vxresize -b -F vxfs -g dde1ZZGO0 ZZGO0_v0 +1000g

BEFORE:

# df -kh /u01-zz/oradata
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/dde1ZZGO0/ZZGO0_v0
                       5.0T   5.0T    17G   100%    /u01-zz/oradata

AFTER:

# df -kh /u01-zz/oradata
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/dde1ZZGO0/ZZGO0_v0
                       6.0T   5.0T   954G    85%    /u01-zz/oradata

Good learning about attribute HacliUserLevel, isn't it?

Change mount point under VCS.

It's my B'day today!!! On this occasion I can’t miss an opportunity to write a blog entry as I think, being a tech savvy  guy I should write something which would be the nice gift from me to myself..

On this special day, today I would like to share a blog entry with you all. So let's start...

Past week I was requested to change mount point of /opt/vdf to /data along with applicable alteration in VCS configuration to make sure VCS functionality should work as expected as before for this volume & mount resource.

The mount point needs to be renamed to /data is as follows –

# df -kh /opt/vdf/
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/vdfdg/opt_vdf_vol
            400G 165M 375G 1% /opt/vdf

Hence in order to do so, I’ll have to –

Unmount the /opt/vdf filesystem and re-mount it on /data as mount point. Before we do so, check if any data resides under filesystem and if any application active processes running within the filesystem which needs to be renamed. In case if any application is running and using the filesystem which needs renamed then at first place you need to stop the application and verify if no application process is hooked with the filesystem & If data resides on filesystem then simply create a temporary filesystem with temporary mountpoint and then copy over the data from original filesystem needs to be renamed. I trust this is simple one and any experienced SA can certainly do it.

Now in order to remount the filesystem with new mountpoint, first you need to modify your VCS configuration a bit so that you can un-mount the filesystem successfully.

Obliviously, the first basic thing to verify that VCS cluster configuration is currently read-only mode or write mode. To verify that you can use below command.

# haclus -display | grep -i 'readonly'
ReadOnly 1

Where,

0 = write mode
1 = read only mode

Well, currently VCS configuration is in Read-only mode so let’s make it write mode.

# haconf -makerw

# haclus -display | grep -i 'readonly'
ReadOnly 0

Good, now we have VCS configuration in write mode so we can make appropriate changes to the configuration and save them.

Now change the appropriate attributes in VCS configuration for filesystem/mount point resource that we are supposed to rename. This may vary, in my case I’m not changing the volume name and changing just a mountpoint name so I’m modifying only “MountPoint” attribute for resource “optvdf_mnt”

# hares -modify optvdf_mnt MountPoint "/data"

To verify.

# hares -display optvdf_mnt | grep -i MountPoint
optvdf_mnt ArgListValues adevdf01s MountPoint 1 /data BlockDevice 1 /dev/vx/dsk/vdfdg/opt_vdf_vol FSType 1 vxfs MountOpt 1 "" FsckOpt 1 -n SnapUmount 1 0 CkptUmount 1 1 SecondLevelMonitor 1 0 SecondLevelTimeout 1 30 OptCheck 1 0 CreateMntPt 1 0 MntPtPermission 1 "" MntPtOwner 1 "" MntPtGroup 1 "" AccessPermissionChk 1 0 RecursiveMnt 1 0 VxFSMountLock 1 1
optvdf_mnt ArgListValues adevdf02s MountPoint 1 /data BlockDevice 1 /dev/vx/dsk/vdfdg/opt_vdf_vol FSType 1 vxfs MountOpt 1 "" FsckOpt 1 -n SnapUmount 1 0 CkptUmount 1 1 SecondLevelMonitor 1 0 SecondLevelTimeout 1 30 OptCheck 1 0 CreateMntPt 1 0 MntPtPermission 1 "" MntPtOwner 1 "" MntPtGroup 1 "" AccessPermissionChk 1 0 RecursiveMnt 1 0 VxFSMountLock 1 1
optvdf_mnt MountPoint global /data

At this point we are good to un-mount the filesystem with mountpoint named /opt/vdf.

# umount /opt/vdf
UX:vxfs umount: ERROR: V-3-26388: file system /opt/vdf has been mount locked

Error!!! Whenever I get any error I feel very happy as every error teaches something new & especially new errors which I never came across!

The Solaris native OS command "umount" may through below error -

Well this error occurs if the mount point is locked by VCS. This is for cases where VCS service groups having DiskGroup resources configured with UnMountVolumes attribute set and the volumes are mounted outside of VCS control. The purpose of the VERITAS File System (VxFS) Mount Lock is to prevent accidental unmounting of a VxFS file system. The feature is enabled for the VCS Mount resource by default. This feature can be disabled by setting the Mount resource attribute VxFSMountLock to 0.

# hagrp -resources vdfapp_sg
vdfapp_dg
vdfappIP
optcdvdf_mnt
optvdf_mnt
optcdvdf_vol
optvdf_vol

# hares -display vdfapp_dg | grep -i UmountVolumes
vdfapp_dg ArgListValues adevdf01s DiskGroup 1 vdfdg StartVolumes 1 1 StopVolumes 1 1 MonitorOnly 1 0 MonitorReservation 1 0 tempUseFence 1 SCSI3 PanicSystemOnDGLoss 1 0 DiskGroupType 1 private UmountVolumes 1 0 vdfapp_dg ArgListValues adevdf02s DiskGroup 1 vdfdg StartVolumes 1 1 StopVolumes 1 1 MonitorOnly 1 0 MonitorReservation 1 0 tempUseFence 1 SCSI3 PanicSystemOnDGLoss 1 0 DiskGroupType 1 private UmountVolumes 1 0 vdfapp_dg UmountVolumes global 0

To get rid of this error use the VxFS umount command for Solaris to manually unmount the file system.

# /opt/VRTS/bin/umount -o mntunlock=VCS /opt/vdf

To mount filesystem/volume on new mountpoint use –

# mount -F vxfs -o mntlock=VCS /dev/vx/dsk/vdfdg/opt_vdf_vol /data

# df -kh /data
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/vdfdg/opt_vdf_vol
              400G  165M  375G  1%   /data

Make sure to save and make cluster configuration read-only.

# haconf -dump -makero

That's it! Done, isn't this easy! :)

Add VERITAS Volume Manager Disk Group & Filesystem into VERITAS Cluster service group - from the scratch orientation

In this article we will see how to create a new disk group, create 4-way stripe VxVM filesystem & then will see how do we create a new VCS service group and incorporate the newly created Disk Group & VxFS filesystem to the service group.
So let's roll.

Well, what's the requirement? so, here we go!

• Create 3 diskgroups namely dapos1dg, dapos2dg, dapos3dg.
• Create three VxVM filesystems within above three DiskGroup /twm/dapos1, /twm/dapos2, /twm/dapos3  with 4/128 stripe layout.
• Create three VERITAS service groups dapos1_sg, dapos2_sg & dapos3_sg.
• Insert/attach/link the needed infra resources (DiskGroup, filesystem, ..) into the corresponding VERITAS service groups.

Solution Delivery –

In order to create 4/128 stripe layout volumes/filesystems we need to get storage disks in bunch of 4 hence I ordered for 12x15G LUNs. Each DiskGroup will have 4 disks.

Well, below are the LUNs that we have from storage –

c2t5006048C52A8CCC3d40
c2t5006048C52A8CCC3d41
c2t5006048C52A8CCC3d42
c2t5006048C52A8CCC3d43
c2t5006048C52A8CCC3d44
c2t5006048C52A8CCC3d45
c2t5006048C52A8CCC3d46
c2t5006048C52A8CCC3d47
c2t5006048C52A8CCC3d48
c2t5006048C52A8CCC3d49
c2t5006048C52A8CCC3d50
c2t5006048C52A8CCC3d51

Before formatting the disks verify if someone hasn’t used these disks for any other purpose. (it happens sometimes, some lazy frogs uses unused disks without taking much efforts and without knowing may someone has ordered these LUNs for some purpose)

# for disk in `cat disks`;do vxdisk -eo alldgs list | grep $disk;done
EMC0_223 auto - - online c2t5006048C52A8CCC3d40s2
EMC0_224 auto - - online c2t5006048C52A8CCC3d41s2
EMC0_225 auto - - online c2t5006048C52A8CCC3d42s2
EMC0_229 auto - - online c2t5006048C52A8CCC3d43s2
EMC0_226 auto - - online c2t5006048C52A8CCC3d44s2
EMC0_227 auto - - online c2t5006048C52A8CCC3d45s2
EMC0_228 auto - - online c2t5006048C52A8CCC3d46s2
EMC0_231 auto - - online c2t5006048C52A8CCC3d47s2
EMC0_222 auto - - online c2t5006048C52A8CCC3d48s2
EMC0_221 auto - - online c2t5006048C52A8CCC3d49s2
EMC0_220 auto - - online c2t5006048C52A8CCC3d50s2
EMC0_230 auto - - online c2t5006048C52A8CCC3d51s2

Good. Disks are not being used by any VERITAS DG so let's label the disks with SMI label.

 Once done with labelling the disks make sure you run vxdctl enable command & re-scan for new disks devices so that the disks are visible to VxVM.

# vxdctl enable

Now we are all set to get all new disks to VERITAS control. Before taking disks into VxVM control verify if those are visible to VxVM.

# for lun in `cat /tmp/lun-list`; do vxdisk -eo alldgs list | grep -i $lun; done
EMC0_223 auto - - online c2t5006048C52A8CCC3d40s2
EMC0_224 auto - - online c2t5006048C52A8CCC3d41s2
EMC0_225 auto - - online c2t5006048C52A8CCC3d42s2
EMC0_229 auto - - online c2t5006048C52A8CCC3d43s2
EMC0_226 auto - - online c2t5006048C52A8CCC3d44s2
EMC0_227 auto - - online c2t5006048C52A8CCC3d45s2
EMC0_228 auto - - online c2t5006048C52A8CCC3d46s2
EMC0_231 auto - - online c2t5006048C52A8CCC3d47s2
EMC0_222 auto - - online c2t5006048C52A8CCC3d48s2
EMC0_221 auto - - online c2t5006048C52A8CCC3d49s2
EMC0_220 auto - - online c2t5006048C52A8CCC3d50s2
EMC0_230 auto - - online c2t5006048C52A8CCC3d51s2

Now let's initialize the new disks and get them under VERITAS control.

Get the list of new disks visible to VxVM using below command.

# for lun in `cat lun-list`; do vxdisk -e list | grep -i $lun; done | awk '{print $1}' > /tmp/vxvm_disks && cat /tmp/vxvm_disks
EMC0_223
EMC0_224
EMC0_225
EMC0_229
EMC0_226
EMC0_227
EMC0_228
EMC0_231
EMC0_222
EMC0_221
EMC0_220
EMC0_230

Initialize new disks using below command.

# for disk in `cat /tmp/vxvm_disks`;do vxdisksetup -i $disk format=sliced; done

See if all the new disks are online and under VxVM control.

# for lun in `cat vxvm_disks`; do vxdisk list | grep -i $lun; done | sort
EMC0_220 auto:sliced - - online
EMC0_221 auto:sliced - - online
EMC0_222 auto:sliced - - online
EMC0_223 auto:sliced - - online
EMC0_224 auto:sliced - - online
EMC0_225 auto:sliced - - online
EMC0_226 auto:sliced - - online
EMC0_227 auto:sliced - - online
EMC0_228 auto:sliced - - online
EMC0_229 auto:sliced - - online
EMC0_230 auto:sliced - - online
EMC0_231 auto:sliced - - online

Now let's create diskgroups –

NOTE: As these are not the shared disk groups make sure you create them on individual systems where these diskgroups needs to be mounted or else create all three disks groups on one system and then deport –import them on respective systems. Either way!

# vxdg init dapos1dg cds=off s01c0_dapos1dg=EMC0_220 s01c1_dapos1dg=EMC0_221 s01c2_dapos1dg=EMC0_222 s01c3_dapos1dg=EMC0_223
# vxdg init dapos2dg cds=off s01c0_dapos2dg=EMC0_224 s01c1_dapos2dg=EMC0_225 s01c2_dapos2dg=EMC0_226 s01c3_dapos2dg=EMC0_227
# vxdg init dapos3dg cds=off s01c0_dapos3dg=EMC0_228 s01c1_dapos3dg=EMC0_229 s01c2_dapos3dg=EMC0_230 s01c3_dapos3dg=EMC0_231

# vxdg list |egrep 'dapos1dg|dapos2dg|dapos3dg'
dapos1dg enabled 1330089054.668.sys1
dapos2dg enabled 1330089075.670.sys1
dapos3dg enabled 1330089087.672.sys1

Now we will create STRIPE volumes within these newly created three diskgroups.

# vxassist -g dapos1dg make dapos1 3g layout=stripe s01c0_dapos1dg s01c1_dapos1dg s01c2_dapos1dg s01c3_dapos1dg
# vxassist -g dapos2dg make dapos2 3g layout=stripe s01c0_dapos2dg s01c1_dapos2dg s01c2_dapos2dg s01c3_dapos2dg
# vxassist -g dapos3dg make dapos3 3g layout=stripe s01c0_dapos3dg s01c1_dapos3dg s01c2_dapos3dg s01c3_dapos3dg

Well, the volumes are created. Let’s check them out.

# for i in dapos1dg dapos2dg dapos3dg; do vxprint -htg $i | grep -i dapos;done
dg dapos1dg default default 16000 1330089054.668.sys1
dm s01c0_dapos1dg EMC0_220 auto 17023 31420800 -
dm s01c1_dapos1dg EMC0_221 auto 17023 31420800 -
dm s01c2_dapos1dg EMC0_222 auto 17023 31420800 -
dm s01c3_dapos1dg EMC0_223 auto 17023 31420800 -
v dapos1 - ENABLED ACTIVE 6291456 SELECT dapos1-01 fsgen
pl dapos1-01 dapos1 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos1dg-01 dapos1-01 s01c0_dapos1dg 0 1572864 0/0 EMC0_220 ENA
sd s01c1_dapos1dg-01 dapos1-01 s01c1_dapos1dg 0 1572864 1/0 EMC0_221 ENA
sd s01c2_dapos1dg-01 dapos1-01 s01c2_dapos1dg 0 1572864 2/0 EMC0_222 ENA
sd s01c3_dapos1dg-01 dapos1-01 s01c3_dapos1dg 0 1572864 3/0 EMC0_223 ENA
dg dapos2dg default default 68000 1330089075.670.sys1
dm s01c0_dapos2dg EMC0_224 auto 17023 31420800 -
dm s01c1_dapos2dg EMC0_225 auto 17023 31420800 -
dm s01c2_dapos2dg EMC0_226 auto 17023 31420800 -
dm s01c3_dapos2dg EMC0_227 auto 17023 31420800 -
v dapos2 - ENABLED ACTIVE 6291456 SELECT dapos2-01 fsgen
pl dapos2-01 dapos2 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos2dg-01 dapos2-01 s01c0_dapos2dg 0 1572864 0/0 EMC0_224 ENA
sd s01c1_dapos2dg-01 dapos2-01 s01c1_dapos2dg 0 1572864 1/0 EMC0_225 ENA
sd s01c2_dapos2dg-01 dapos2-01 s01c2_dapos2dg 0 1572864 2/0 EMC0_226 ENA
sd s01c3_dapos2dg-01 dapos2-01 s01c3_dapos2dg 0 1572864 3/0 EMC0_227 ENA
dg dapos3dg default default 20000 1330089087.672.sys1
dm s01c0_dapos3dg EMC0_228 auto 17023 31420800 -
dm s01c1_dapos3dg EMC0_229 auto 17023 31420800 -
dm s01c2_dapos3dg EMC0_230 auto 17023 31420800 -
dm s01c3_dapos3dg EMC0_231 auto 17023 31420800 -
v dapos3 - ENABLED ACTIVE 6291456 SELECT dapos3-01 fsgen
pl dapos3-01 dapos3 ENABLED ACTIVE 6291456 STRIPE 4/128 RW
sd s01c0_dapos3dg-01 dapos3-01 s01c0_dapos3dg 0 1572864 0/0 EMC0_228 ENA
sd s01c1_dapos3dg-01 dapos3-01 s01c1_dapos3dg 0 1572864 1/0 EMC0_229 ENA
sd s01c2_dapos3dg-01 dapos3-01 s01c2_dapos3dg 0 1572864 2/0 EMC0_230 ENA
sd s01c3_dapos3dg-01 dapos3-01 s01c3_dapos3dg 0 1572864 3/0 EMC0_231 ENA

Next thing is to create a mount point for these volume & create a file system on this volume.

On individual servers…

# for fs in 1 2 3;do mkdir -p /twm/dapos$fs;done && ls –ld /twm/dapos*
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos1
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos2
drwxr-xr-x 2 root other 512 Feb 24 14:36 /twm/dapos3

Not required to be done on individual servers as the storage disks are shared between cluster nodes.

# mkfs -F vxfs /dev/vx/rdsk/dapos1dg/dapos1
# mkfs -F vxfs /dev/vx/rdsk/dapos2dg/dapos2
# mkfs -F vxfs /dev/vx/rdsk/dapos3dg/dapos3

First time needed only on respective individual servers.

# mount -F vxfs /dev/vx/dsk/dapos1dg/dapos1 /twm/dapos1

Now, furthermore there is requirement to create new VCS service groups & Insert/attach the needed infra resources (DiskGroup, filesystem, ..) into the corresponding VERITAS service groups. Let’s on it!

Before doing anything, first thing to check if cluster configuration is read-only or in write mode.

# haclus -display |grep -i 'readonly'
ReadOnly 1

Where,
0 = write mode
1 = read only mode

If the cluster configuration is read-only make it write mode -

# haconf –makerw

# hagrp -add dapos1_sg
# hagrp -modify dapos1_sg SystemList sys1 0 sys3 1 sys4 2 sys2 3
# hagrp -modify dapos1_sg AutoStart 0
# hagrp -modify dapos1_sg AutoStartList sys1
# hagrp -modify dapos1_sg SourceFile "./main.cf"

# hares -add dapos1_dg DiskGroup dapos1_sg
# hares -modify dapos1_dg DiskGroup dapos1dg
# hares -modify dapos1_dg PanicSystemOnDGLoss 0
# hares -modify dapos1_dg StartVolumes 1
# hares -modify dapos1_dg StopVolumes 1
# hares -modify dapos1_dg MonitorReservation 0
# hares -modify dapos1_dg tempUseFence INVALID
# hares -modify dapos1_dg DiskGroupType private
# hares -modify dapos1_dg Enabled 1

# hares -add dapos1_EMCProc EMCProc dapos1_sg
# hares -modify dapos1_EMCProc Enabled 1
# hares -add dapos1_vol Volume dapos1_sg
# hares -modify dapos1_vol Volume dapos1
# hares -modify dapos1_vol DiskGroup dapos1dg
# hares -modify dapos1_vol Enabled 1

# hares -add dapos1_mnt Mount dapos1_sg
# hares -modify dapos1_mnt Critical 0
# hares -modify dapos1_mnt MountPoint "/twm/dapos1"
# hares -modify dapos1_mnt BlockDevice "/dev/vx/dsk/dapos1dg/dapos1"
# hares -modify dapos1_mnt FSType vxfs
# hares -modify dapos1_mnt FsckOpt "%-n"
# hares -modify dapos1_mnt CkptUmount 1
# hares -modify dapos1_mnt SecondLevelMonitor 0
# hares -modify dapos1_mnt SecondLevelTimeout 30
# hares -modify dapos1_mnt Enabled 1

Fine, next part is linking - while linking always remember the syntax -

# hares -link {parent} {child}

Please note in VCS, parent resource depends on child resource, for example in a group where volume depends on DiskGroup resource, DiskGroup will be child & volume would be parent...

# hares -link dapos1_vol dapos1_dg
# hares -link dapos1_mnt dapos1_vol

Upon performing the above configuration you can see the resources added to the service group dapos1_sg.

# hagrp -resources dapos1_sg
dapos1_prod_IPM
dapos1_prod_proxy
dapos1_dg
dapos1_vol
dapos1_mnt

Perform the same procedure for rest of 2 service groups. Once completed check the status of the service groups.

# hastatus -sum | egrep 'dapos1_sg|dapos2_sg|dapos3_sg'
B dapos1_sg sys1 Y N ONLINE
B dapos1_sg sys2 Y N OFFLINE
B dapos1_sg sys3 Y N OFFLINE
B dapos1_sg sys4 Y N OFFLINE
B dapos2_sg sys1 Y N OFFLINE
B dapos2_sg sys2 Y N ONLINE
B dapos2_sg sys3 Y N OFFLINE
B dapos2_sg sys4 Y N OFFLINE
B dapos3_sg sys1 Y N OFFLINE
B dapos3_sg sys2 Y N OFFLINE
B dapos3_sg sys3 Y N ONLINE
B dapos3_sg sys4 Y N OFFLINE

Once completed with the changes do not forget to get the Cluster Server configuration in read-only mode and dump the changes in the main.cf file. You can do the same using below command -

# haconf -dump -makero

Wooh.. we are done with the requirement. It was a good exercise in itself...

HTH.