Migrate VERITAS (VxVM) to Solaris Disk Suite (SDS)

Hi there! busy.. busy.. busy... stuck with routine work.. lots of work! :)

Anyways, today I finally decided to write a blog entry on a bit unusual task I did recently - Migrate VERITAS (VxVM) to Solaris Disk Suite (SDS). In past have done several migrations, like UFS to ZFS, UFS/SDS to VxVM, VxVM to ZFS however this time I been contacted for some different and backward migration which is VxVM to SDS. When SDS is getting obsolete but on other hand still there are some requirement comes, in which demanding for such things… anyways, good to perform anything which you feels interesting and uncommon. Then let’s do it…

Point To Ponder - Make sure you have full backup of the system on which you're going to operate.

Let's first see which disks are part of bootdg.

Disk_0 auto rootdisk rootdg online c3t0d0s2
Disk_5 auto rootmirror rootdg online c0t0d0s2

Which all volumes needs to be converted to the SVM -

# df -kh | grep bootdg
/dev/vx/dsk/bootdg/rootvol 5.9G 4.3G 1.5G 75% /
/dev/vx/dsk/bootdg/var 5.9G 4.7G 1.1G 81% /var
/dev/vx/dsk/bootdg/opt 5.9G 1.8G 4.0G 32% /opt

Take the pre-veritas vfstab copy.

root@XXXXXX# cat /etc/vfstab.prevm

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c0t0d0s1 - - swap - no -
/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no nologging
/dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /var ufs 1 no nologging
/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /opt ufs 2 yes nologging
/dev/dsk/c0t2d0s0 /dev/rdsk/c0t2d0s0 /var/crash ufs 2 yes nologging
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes -

#/dev/dsk/c3t0d0s0 is currently mounted on /.
#/dev/dsk/c3t0d0s1 is currently used by swap.
#/dev/dsk/c3t0d0s5 is currently mounted on /opt.
#/dev/dsk/c3t0d0s6 is currently mounted on /var.

Let's first unencapsulate the rootdisk using the vxunroot.

Detach all the plexes associated with the 'rootmirror' disk if applicable & Verify the rootmirror plexes have been detached.

# vxprint -qhtg rootdg -s | grep -i rootmirror | awk '{print $3}' > /var/tmp/subs.plex && cat /var/tmp/subs.plex
rootvol-02
swapvol-02
opt-02
var-02

# for x in `cat /var/tmp/subs.plex`

> do
> vxplex -g rootdg dis $x
> vxprint -qhtg rootdg -p $x
> done
pl rootvol-02 - DISABLED - 12584484 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 12584484 0 Disk_5 ENA
pl swapvol-02 - DISABLED - 31458321 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 12584484 31458321 0 Disk_5 ENA
pl opt-02 - DISABLED - 12584484 CONCAT - RW
sd rootmirror-03 opt-02 rootmirror 44042805 12584484 0 Disk_5 ENA
pl var-02 - DISABLED - 12584484 CONCAT - RW
sd rootmirror-04 var-02 rootmirror 56627289 12584484 0 Disk_5 ENA

# /etc/vx/bin/vxunroot

VxVM vxunroot NOTICE V-5-2-1564
This operation will convert the following file systems from
   volumes to regular partitions:
opt rootvol swapvol var

   VxVM vxunroot INFO V-5-2-2011
Replacing volumes in root disk to partitions will require a system
  reboot. If you choose to continue with this operation, system
  configuration will be updated to discontinue use of the volume
  manager for your root and swap devices.

Do you wish to do this now [y,n,q,?] (default: y) y
VxVM vxunroot INFO V-5-2-287 Restoring kernel configuration...
VxVM vxunroot INFO V-5-2-78
A shutdown is now required to install the new kernel.
You can choose to shutdown now, or you can shutdown later, at your
   convenience.

Do you wish to shutdown now [y,n,q,?] (default: n) n

VxVM vxunroot INFO V-5-2-258
Please shutdown before you perform any additional volume manager
   or disk reconfiguration. To shutdown your system cd to / and type

          shutdown -g0 -y -i6

# sync;sync;sync;shutdown -g0 -y -i6

Well, after 2-3 reboots server came back online and now I've UFS filesystems for OS volumes.

# df -kh / /var /opt
Filesystem size used avail capacity Mounted on
/dev/dsk/c3t0d0s0 5.9G 4.3G 1.5G 74% /
/dev/dsk/c3t0d0s6 5.9G 4.7G 1.1G 81% /var
/dev/dsk/c3t0d0s5 5.9G 1.8G 4.0G 32% /opt

Just to be sure, once just try booting from mirror disk too.

Now let's create a partition slice for metadb.

# format c3t0d0

selecting c3t0d0
[disk formatted]
Warning: Current Disk has mounted partitions.
/dev/dsk/c3t0d0s0 is currently mounted on /. Please see umount(1M).
/dev/dsk/c3t0d0s1 is currently used by swap. Please see swap(1M).
/dev/dsk/c3t0d0s5 is currently mounted on /opt. Please see umount(1M).
/dev/dsk/c3t0d0s6 is currently mounted on /var. Please see umount(1M).

FORMAT MENU:
                disk - select a disk
                type - select (define) a disk type
                partition - select (define) a partition table
                current - describe the current disk
                format - format and analyze the disk
                repair - repair a defective sector
                label - write label to the disk
                analyze - surface analysis
                defect - defect list management
                backup - search for backup labels
                verify - read and display labels
                save - save new disk/partition definitions
                inquiry - show vendor, product and revision
                volname - set 8-character volume name
                ![cmd] - execute [cmd], then return
                quit
format> p

PARTITION MENU:
        0 - change `0' partition
        1 - change `1' partition
        2 - change `2' partition
        3 - change `3' partition
        4 - change `4' partition
        5 - change `5' partition
        6 - change `6' partition
        7 - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name - name the current table
        print - display the current table
        label - write partition map and label to the disk
        ![cmd] - execute [cmd], then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 3 - 4358 6.00GB (4356/0/0) 12584484
1 swap wu 4359 - 15247 15.00GB (10889/0/0) 31458321
2 backup wu 0 - 24619 33.92GB (24620/0/0) 71127180
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 15248 - 19603 6.00GB (4356/0/0) 12584484
6 var wm 19604 - 23959 6.00GB (4356/0/0) 12584484
7 unassigned wm 0 0 (0/0/0) 0

partition> 7
Part Tag Flag Cylinders Size Blocks
7 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 23960
Enter partition size[0b, 0c, 23960e, 0.00mb, 0.00gb]: 128mb
partition> p
Current partition table (unnamed):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 3 - 4358 6.00GB (4356/0/0) 12584484
1 swap wu 4359 - 15247 15.00GB (10889/0/0) 31458321
2 backup wu 0 - 24619 33.92GB (24620/0/0) 71127180
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 15248 - 19603 6.00GB (4356/0/0) 12584484
6 var wm 19604 - 23959 6.00GB (4356/0/0) 12584484
7 unassigned wm 23960 - 24050 128.37MB (91/0/0) 262899

partition> l
Ready to label disk, continue? yes

Good enough, now let's create metadb.

# metadb -a -f -c 3 c3t0d0s7

# metadb -i
       flags first blk block count
         a  u     16      8192     /dev/dsk/c3t0d0s7
         a  u     8208    8192     /dev/dsk/c3t0d0s7
         a  u     16400   8192     /dev/dsk/c3t0d0s7
r - replica does not have device relocation information
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input
W - replica has device write errors
a - replica is active, commits are occurring to this replica
M - replica had problem with master blocks
D - replica had problem with data blocks
F - replica had format problems
S - replica is too small to hold current data base
R - replica had device read errors

Set SDS on root file system.

# metainit -f d11 1 1 c3t0d0s0
d11: Concat/Stripe is setup

# metainit d10 -m d11
d10: Mirror is setup
# metaroot d10

After executing metaroot, check the changes in /etc/vfstab and /etc/system.

#live-upgrade: updated boot environment

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
#live-upgrade::# /dev/vx/dsk/bootdg/swapvol - - swap -no nologging
/dev/dsk/c3t0d0s1 - - swap - no -
/dev/md/dsk/d10 /dev/md/rdsk/d10 / ufs 1 no nologging
/dev/dsk/c3t0d0s6 /dev/rdsk/c3t0d0s6 /var ufs 1 no nologging,nosuid
/dev/dsk/c3t0d0s5 /dev/rdsk/c3t0d0s5 /opt ufs 2 yes nologging
/dev/vx/dsk/crashdg/crashvol /dev/dsk/crashdg/crashvol /var/crash vxfs 2 yes -
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes nosuid

* Begin MDD root info (do not edit)
rootdev:/pseudo/md@0:0,10,blk
* End MDD root info (do not edit)

Good enough.

# metastat -ac
d10           m 6.0GB d11
   d11        s 6.0GB c3t0d0s0

# metainit -f d31 1 1 c3t0d0s6
d31: Concat/Stripe is setup
# metainit d30 -m d31
d30: Mirror is setup

# metainit -f d51 1 1 c3t0d0s5
d51: Concat/Stripe is setup
# metainit d50 -m d51
d50: Mirror is setup

# metainit -f d1 1 1 c3t0d0s1
d1: Concat/Stripe is setup
# metainit d0 -m d1
d0: Mirror is setup

# metastat -ac
d0      m 15GB d1
   d1   s 15GB c3t0d0s1
d50     m 6.0GB d51
   d51  s 6.0GB c3t0d0s5
d30     m 6.0GB d31
   d31  s 6.0GB c3t0d0s6
d10     m 6.0GB d11
   d11  s 6.0GB c3t0d0s0

So in above,

d0 - swap
d10 - /
d30 - /var
d50 - /opt

Now it's time to change the vfstab to change slices into metadb devices.

# vi /etc/vfstab

# cat /etc/vfstab
#live-upgrade: updated boot environment
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
#live-upgrade::# /dev/vx/dsk/bootdg/swapvol - - swap - no nologging
/dev/md/dsk/d0 - - swap - no -
/dev/md/dsk/d10 /dev/md/rdsk/d10 / ufs 1 no nologging
/dev/md/dsk/d30 /dev/md/rdsk/d30 /var ufs 1 no nologging,nosuid
/dev/md/dsk/d50 /dev/md/rdsk/d50 /opt ufs 2 yes nologging
/dev/vx/dsk/crashdg/crashvol /dev/dsk/crashdg/crashvol /var/crash vxfs 2 yes -
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes nosuid

Reboot the box once.

Now it's time to add mirror disk to existing metadevices.

At this stage the mirror disk is part of rootdg and for rootdg it's the last disk in the disk group hence we need to destroy rootdg.

# vxdg destroy rootdg

Unsetup the disk so it will be out of VERITAS control.

# vxdiskunsetup -C Disk_5

Cool, now I'm all set to create/copy partition table on mirror disk as exisitng disk part of metadevices.

# prtvtoc /dev/rdsk/c3t0d0s2 | fmthard -s - /dev/rdsk/c0t0d0s2
fmthard: New volume table of contents now in place.

Fine, Set redundant database on mirror disk.

# metadb -a -f -c 3 c0t0d0s7

# metadb -i

        flags    first blk    block count
     a m p luo   16           8192         /dev/dsk/c3t0d0s7
     a   p luo   8208         8192         /dev/dsk/c3t0d0s7
     a   p luo   16400        8192         /dev/dsk/c3t0d0s7
     a      u    16           8192         /dev/dsk/c0t0d0s7
     a      u    8208         8192         /dev/dsk/c0t0d0s7
     a      u    16400        8192         /dev/dsk/c0t0d0s7
r - replica does not have device relocation information
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input
W - replica has device write errors
a - replica is active, commits are occurring to this replica
M - replica had problem with master blocks
D - replica had problem with data blocks
F - replica had format problems
S - replica is too small to hold current data base
R - replica had device read errors

Create metadevices on mirror.

# metainit -f d12 1 1 c0t0d0s0
d12: Concat/Stripe is setup

# metainit -f d2 1 1 c0t0d0s1
d2: Concat/Stripe is setup

# metainit -f d52 1 1 c0t0d0s5
d52: Concat/Stripe is setup

# metainit -f d32 1 1 c0t0d0s6
d32: Concat/Stripe is setup

Attach metadevices.

# metattach d10 d12
d10: submirror d12 is attached

# metattach d30 d32
d30: submirror d32 is attached

# metattach d50 d52
d50: submirror d52 is attached

# metattach d0 d2
d0: submirror d2 is attached

# metastat -ac
d0              m 15GB d1 d2 (resync-76%)
     d1         s 15GB c3t0d0s1
d2              s 15GB c0t0d0s1
     d50        m 6.0GB d51 d52
d51             s 6.0GB c3t0d0s5
     d52        s 6.0GB c0t0d0s5
d30             m 6.0GB d31 d32
     d31        s 6.0GB c3t0d0s6
d32             s 6.0GB c0t0d0s6
     d10        m 6.0GB d11 d12
d11             s 6.0GB c3t0d0s0
     d12        s 6.0GB c0t0d0s0

Install boot block on both disks.

# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t0d0s0
# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c3t0d0s0

Well, that completes VxVM to SDS migration.  I'm sure hardly anyone need to do such backward migration but just in case then this method will certainly help you... :)

Have a good weekend!

UNABLE TO MIRROR VxVM VOLUME

First of all... Merry Christmas, everybody!

Yes, again after a long time... past few weeks were very hyperactive and quite attentive but finally, yes… finally one more unsleeping late night… I decided to take some timeout to post this interesting post for all my sysadmin friends!

Few days back I got a request to mirror an existing volume. Easy enough right? - Yes, it is! but then I revealed into an interesting problem causing me to unable mirror the existing volume.

Let's see what was the issue and what logic I used to get rid of the issue.

Volume DXarchive90d was not mirrored for some unknown reason and so I've been requested to mirror it.

root:XXXXXXXX:/root # vxprint -htg asmm7 DXarchive90d

V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

v DXarchive90d - ENABLED ACTIVE 419430400 SELECT - fsgen
pl DXarchive90d-01 DXarchive90d ENABLED ACTIVE 418426880 CONCAT - RW
sd asmm7_8-01 DXarchive90d-01 asmm7_8 0 167270400 0 usp006_21 ENA
sd asmm7_9-01 DXarchive90d-01 asmm7_9 0 167270400 167270400 usp006_19 ENA
sd asmm7_10-01 DXarchive90d-01 asmm7_20 0 83886080 334540800 usp006_9 ENA

In order to mirror it I decided to create the new sub disks, new plex and attach the plex to make mirror for volume DXarchive90d. When I tried so, I got an interesting error -

1)  I created another plex of same size and tried to attach but get below message.

- Create sub-disk

Syntax - vxmake [-g diskgroup] sd subdisk diskname,offset,length

root:XXXXXXXX:/root # vxmake -g asmm7 sd asmm7_21m-01 asmm7_21m,0,167270400
root:XXXXXXXX:/root # vxmake -g asmm7 sd asmm7_22m-01 asmm7_22m,0,167270400
root:XXXXXXXX:/root # vxmake -g asmm7 sd asmm7_23m-01 asmm7_23m,0,83886080

- Create Plex of same size

Syntax - vxmake [-g diskgroup] plex plex sd=subdisk1[,subdisk2,...]

root:XXXXXXXX:/root # vxmake -g asmm7 plex DXarchive90d-02 sd=asmm7_21m-01,asmm7_22m-01,asmm7_23m-01

- Attach the plex

root:XXXXXXXX:/root # vxplex -g asmm7 att DXarchive90d DXarchive90d-02
VxVM vxplex ERROR V-5-1-4035 Plex DXarchive90d-02 would be a sparse plex of Volume DXarchive90d

Tried with "force" option.

root:XXXXXXXX:/root # vxplex -g asmm7 -o force att DXarchive90d DXarchive90d-02
VxVM vxplex ERROR V-5-1-1204 Volume DXarchive90d has no complete plexes, cannot attach

2) Tried another way,

root:XXXXXXXX:/root # vxassist -v -b -g asmm7 mirror DXarchive90d layout=mirror-concat alloc="asmm7_21m asmm7_22m asmm7_23m"
VxVM vxassist ERROR V-5-1-437 Cannot allocate space to mirror 419430400 block volume

Later, carefully looking at things - Ahh.... See that -

Volume length: 419430400
Plex length: 418426880

The mirror attach operations are failing due to the addressable size of storage in the volume being larger than that of the plex assigned to it. i.e the volume has no complete plex that will store the entire volume data.

So what is the way out to such situation?

Well, I got the link where it says -

http://www.symantec.com/business/support/index?page=content&id=TECH67644

- Remove the volume
- Then attached the plex to the volume.

Well, unfortunately this solution was not for me! So, I figured out another way -

So, there is 1003520 sector diffrence between Volume length and Plex length. So how about associating subdisk of size 1003520 sectors.... just wild idea - let's see how it goes...

root:XXXXXXXX:/root # vxmake -g asmm7 sd asmm7_10-04 asmm7_10,0,1003520
root:XXXXXXXX:/root # vxsd -g asmm7 assoc DXarchive90d-01 asmm7_10-04

BEFORE:

v DXarchive90d - ENABLED ACTIVE 419430400 SELECT - fsgen
pl DXarchive90d-01 DXarchive90d ENABLED ACTIVE 418426880 CONCAT - RW
sd asmm7_8-01 DXarchive90d-01 asmm7_8 0 167270400 0 usp006_21 ENA
sd asmm7_9-01 DXarchive90d-01 asmm7_9 0 167270400 167270400 usp006_19 ENA
sd asmm7_10-01 DXarchive90d-01 asmm7_20 0 83886080 334540800 usp006_9 ENA

AFTER:

v DXarchive90d - ENABLED ACTIVE 419430400 SELECT - fsgen
pl DXarchive90d-01 DXarchive90d ENABLED ACTIVE 419430400 CONCAT - RW
sd asmm7_8-01 DXarchive90d-01 asmm7_8 0 167270400 0 usp006_21 ENA
sd asmm7_9-01 DXarchive90d-01 asmm7_9 0 167270400 167270400 usp006_19 ENA
sd asmm7_10-01 DXarchive90d-01 asmm7_20 0 83886080 334540800 usp006_9 ENA
sd asmm7_10-04 DXarchive90d-01 asmm7_10 0 1003520 418426880 usp006_16 ENA

Good enough - the root cause of volume mirror not happening was mismatch between volume lenth and plex lenth so all what I did is matched those sizes by associating subdisk of the size of diffrence to existing plex.

Now I think remove the plex which I created for mirroring...

root:XXXXXXXX:/root # vxedit -g asmm7 -rf rm DXarchive90d-02

Execute -

root:XXXXXXXX:/root # vxassist -v -b -g asmm7 mirror DXarchive90d layout=mirror-concat alloc="asmm7_21m asmm7_22m asmm7_23m"

SUCCESS!!!! Mirroring done OK.

See the mirroring progress...

root:XXXXXXXX:/root # vxtask list

TASKID PTID TYPE/STATE PCT PROGRESS
559 ATCOPY/R 01.10% 0/419430400/4608000 PLXATT DXarchive90d DXarchive90d-02 asmm7

root:XXXXXXXX:/root # vxtask -l list
Task: 559 RUNNING
Type: ATCOPY
Operation: PLXATT Vol DXarchive90d Plex DXarchive90d-02 Dg asmm7
Started: Thu Dec XX 12:07:12 2012
Throttle: 0
Progress: 1.23% 5156864 of 419430400 Blocks
Work time: 43 seconds (57:34 remaining)

root:XXXXXXXX:/root # vxprint -htg asmm7 DXarchive90d

V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
v DXarchive90d - ENABLED ACTIVE 419430400 SELECT - fsgen
pl DXarchive90d-01 DXarchive90d ENABLED ACTIVE 419430400 CONCAT - RW
sd asmm7_8-01 DXarchive90d-01 asmm7_8 0 167270400 0 usp006_21 ENA
sd asmm7_9-01 DXarchive90d-01 asmm7_9 0 167270400 167270400 usp006_19 ENA
sd asmm7_10-01 DXarchive90d-01 asmm7_20 0 83886080 334540800 usp006_9 ENA
sd asmm7_10-04 DXarchive90d-01 asmm7_10 0 1003520 418426880 usp006_16 ENA
pl DXarchive90d-02 DXarchive90d ENABLED ACTIVE 419430400 CONCAT - RW
sd asmm7_21m-01 DXarchive90d-02 asmm7_21m 0 167270400 0 usp009_20 ENA
sd asmm7_22m-01 DXarchive90d-02 asmm7_22m 0 167270400 167270400 usp009_21 ENA
sd asmm7_23m-01 DXarchive90d-02 asmm7_23m 0 84889600 334540800 usp009_22 ENA

Well, now in good shape... Hope this helps someone who gets such a interesting issue...

Weird issue with VERITAS after replacing the disk

When performing a disk replacement in VxVM 4.1 and VxVM 5.0, the disk being replaced does not show up in the output of a "vxdisk list" command. Instead a "NONAMEs2" entry is seen.

root@XXXXX# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
VxVM vxdisk ERROR V-5-1-2272 NONAMEs2 not a valid dmpnode disk access name.
NONAMEs2 auto - - online
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
c1t0d0s2 auto rootdisk rootdg online c1t0d0s2
c1t1d0s2 auto - - online c1t1d0s2
c1t2d0s2 auto data01 data online c1t2d0s2
c1t5d0s2 auto data02 data online c1t5d0s2
c1t8d0s2 auto data03 data online c1t8d0s2
c1t9d0s2 auto data04 data online c1t9d0s2
c1t10d0s2 auto - - online c1t10d0s2
c1t11d0s2 auto - - online c1t11d0s2
c1t12d0s2 auto - - online c1t12d0s2
c1t13d0s2 auto - - online c1t13d0s2
- - rootmirror rootdg removed was:c1t3d0s2

To get rid of this, solution is to re-create device mapping, restart vxconfigd, initialize the device, add the disk back to the diskgroup.

# vxdisk rm NONAMEs2 (Remove the device entry in Volume Manager)
# cd /dev/vx/rdmp (Delete all files pertaining to NONAME*)
# cd /dev/vx/dmp (Delete all files pertaining to NONAME*)
# mv /etc/vx/disk.info /etc/vx/disk.info.old
# mv /etc/vx/array.info /etc/vx/array.info.old
# devfsadm -Cv
# vxddladm stop eventsource
# vxconfigd -k (This will recreate the device mapping and the disk.info file)

If the above solution fails, Rerun all commands up to and including: "mv /etc/vx/disk.info /etc/vx/disk.info.old" then run a reconfigure reboot.

It worked!!!! disk c1t3d0s2 is back....

root@XXXXX# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
c1t0d0s2 auto rootdisk rootdg online c1t0d0s2
c1t1d0s2 auto - - online c1t1d0s2
c1t2d0s2 auto data01 data online c1t2d0s2
c1t3d0s2 auto - - online c1t3d0s2
c1t5d0s2 auto data02 data online c1t5d0s2
c1t8d0s2 auto data03 data online c1t8d0s2
c1t9d0s2 auto data04 data online c1t9d0s2
c1t10d0s2 auto - - online c1t10d0s2
c1t11d0s2 auto - - online c1t11d0s2
c1t12d0s2 auto - - online c1t12d0s2
c1t13d0s2 auto - - online c1t13d0s2
- - rootmirror rootdg removed was:c1t3d0s2

Now as the disk is back in proper format perform action of  -

• Initialize the device,
• Add the disk to the disk group
• Recover plex.

root@XXXXX# vxdisksetup -i c1t3d0 format=sliced
root@XXXXX# vxdg -g rootdg -k adddisk rootmirror=c1t3d0

root@XXXXX# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:sliced rootdisk rootdg online
c1t1d0s2 auto:none - - online invalid
c1t2d0s2 auto:sliced data01 data online
c1t3d0s2 auto:sliced rootmirror rootdg online
c1t5d0s2 auto:sliced data02 data online
c1t8d0s2 auto:sliced data03 data online failing
c1t9d0s2 auto:sliced data04 data online
c1t10d0s2 auto:none - - online invalid
c1t11d0s2 auto:none - - online invalid
c1t12d0s2 auto:none - - online invalid
c1t13d0s2 auto:none - - online invalid

root@XXXXX# vxrecover -bg rootdg

Check if recovery is in progress (vxtask list)

root@XXXXX# vxtask -l list
Task: 436 RUNNING
Type: PARENT
Operation: VXRECOVER
Started: Fri Oct 19 18:38:17 2012
Progress: 20.00% (1 of 5 jobs, 1 active)
Task: 441 RUNNING (Parent: 436)
Type: ATCOPY
Operation: PLXATT Vol opt Plex opt-02 Dg rootdg
Started: Fri Oct 19 18:40:57 2012
Throttle: 0
Progress: 30.16% 6326272 of 20972736 Blocks
Work time: 1 minute, 54 seconds (04:23 remaining)

root@XXXXX# vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 0 1027335905.1025.XXXXX
dm rootdisk c1t0d0s2 auto 10175 143339136 -
dm rootmirror c1t3d0s2 auto 9919 143328960 -

v home - ENABLED ACTIVE 8405376 ROUND - gen
pl home-01 home ENABLED ACTIVE 8405376 CONCAT - RW
sd rootdisk-03 home-01 rootdisk 75566975 8405376 0 c1t0d0 ENA
pl home-02 home ENABLED ACTIVE 8405376 CONCAT - RW
sd rootmirror-05 home-02 rootmirror 75516096 8405376 0 c1t3d0 ENA

v opt - ENABLED ACTIVE 20972736 ROUND - gen
pl opt-01 opt ENABLED ACTIVE 20972736 CONCAT - RW
sd rootdisk-04 opt-01 rootdisk 54594239 20972736 0 c1t0d0 ENA
pl opt-02 opt ENABLED ACTIVE 20972736 CONCAT - RW
sd rootmirror-04 opt-02 rootmirror 54543360 20972736 0 c1t3d0 ENA

v rootvol - ENABLED ACTIVE 4202688 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 4202688 CONCAT - RW
sd rootdisk-02 rootvol-01 rootdisk 20351 4202688 0 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 4202688 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 4202688 0 c1t3d0 ENA

v swapvol - ENABLED ACTIVE 33560448 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 33560448 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 4233215 33560448 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 33560448 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 4202688 33560448 0 c1t3d0 ENA

v var - ENABLED ACTIVE 16780224 ROUND - gen
pl var-01 var ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdisk-05 var-01 rootdisk 37803839 16780224 0 c1t0d0 ENA
pl var-02 var ENABLED ACTIVE 16780224 CONCAT - RW
sd rootmirror-03 var-02 rootmirror 37763136 16780224 0 c1t3d0 ENA

Hope this helps someone who facing this issue..... Have a good night.

Resolving Duplicate disk/device entries in “vxdisk list” or vxdisksetup.

One fine morning I had a undertaking to replace the disk which was part of VxVM. Easy enough – just another routine stuff so my words were - “Ahh it’s simple, it’s just a disk replacement! will finish this off quickly, then go get a cup of coffee over some relax time” – Nope, I was wrong & wasn’t lucky enough to find myself relaxing in office garden over a hot coffee. Anyways, learnt and realized the fact that - There’s no such thing as ‘risk free.’  Everything you do or don’t do has an inherent risk !!!

Anyways, enough with the story – let’s come to the real one. I replaced the disk and inserted new one but after doing so, I started seeing duplicate entries for the replaced disk in the “vxdisk list” output. As per Symantec notes to get rid of this issue we should perform the reconfiguration reboot. The server I was working on was unluckily Solaris 5.8 with vintage VERITAS version 3.5

The problem can be seen when running vxdisk list command:

root@XXXXX# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
c1t0d0s2 sliced rootdisk rootdg online c1t0d0s2
c1t1d0s2 sliced - - online c1t1d0s2
c1t2d0s2 sliced - - error c1t2d0s2
c1t2d0s2 sliced - - error c1t2d0s2
c1t3d0s2 sliced rootspare rootdg online c1t3d0s2
c1t4d0s2 sliced DATA_disk1 rootdg online c1t4d0s2
c1t5d0s2 sliced DATA_disk2 rootdg online c1t5d0s2
- - rootmirror rootdg removed was:c1t2d0s2

Or the problem can be seen when running vxdisksetup command:

root@XXXXX# vxdisksetup -i c1t2d0
vxdisksetup: c1t2d0: Duplicate DA records encountered for this device.
                     Refer to the troubleshooting guide to clear them

Right above in vxdisk list output you can see disk c1t2d0 has duplicate entries.

Now let’s see how to get rid of this not often (at least in my case) seen issue.

First of all, remove c1t2d0s2 entries from VxVM control & run it for all duplicate entries. There can be 2 or more than 2 duplicate entries for the same disk.

In my case, there were two entries –

root@XXXXXXX# vxdisk rm c1t2d0s2
root@XXXXXXX# vxdisk rm c1t2d0s2

Remove the disk c1t2d0s2 using luxadm. Remove device c1t2d0s2 using ” luxadm remove_device ” command.

root@XXXXXXX# luxadm remove_device /dev/rdsk/c1t2d0s2

WARNING!!! Please ensure that no filesystems are mounted on these device(s).
All data on these devices should have been backed up.

The list of devices which will be removed is:
1: Box Name:     "FCloop" slot 2
Node WWN:        20000004cfa1b23c
Device Type:Disk device
Device Paths:
             /dev/rdsk/c1t2d0s2

Please verify the above list of devices and
then enter 'c' or to Continue or 'q' to Quit. [Default: c]:
stopping: Drive in "FCloop" slot 2....Done
offlining: Drive in "FCloop" slot 2....Done

Hit after removing the device(s).

Drive in Box Name "FCloop" slot 2
Notice: Device has not been removed from the enclosure.
It has been removed from the loop and is ready to be
removed from the enclosure, and the LED is blinking.

Logical Nodes being removed under /dev/dsk/ and /dev/rdsk:
Logical Nodes being removed under /dev/dsk/ and /dev/rdsk:
c1t2d0s0
c1t2d0s1
c1t2d0s2
c1t2d0s3
c1t2d0s4
c1t2d0s5
c1t2d0s6
c1t2d0s7

Execute “devfsadm” command to clean-up unlinked device references followed by vxdctl enable command.

root@XXXXXXX# devfsadm -Cv

devfsadm[6117]: verbose: removing node /devices/pci@8,700000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@8,700000:devctl 0l/3l/20600
devfsadm[6117]: verbose: removing node /devices/pci@8,700000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@8,700000:devctl 0l/3l/20600
devfsadm[6117]: verbose: removing node /devices/pci@9,700000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@9,700000:devctl 0l/3l/20600
devfsadm[6117]: verbose: removing node /devices/pci@9,700000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@9,700000:devctl 0l/3l/20600
devfsadm[6117]: verbose: removing node /devices/pci@9,600000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@9,600000:devctl 0l/3l/20600
devfsadm[6117]: verbose: removing node /devices/pci@9,600000:devctl. invalid st_rdev
devfsadm[6117]: verbose: mknod /devices/pci@9,600000:devctl 0l/3l/20600

root@XXXXXXX# vxdctl enable

At this point of time we have removed all dev_t associated with the device. BTW, Within the kernel, the dev_t type is used to hold device numbers—both the major and minor parts.

Okay, now we have one entry less.

root@XXXXXXX# vxdisk -e list

DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
c1t0d0s2 sliced rootdisk rootdg online c1t0d0s2
c1t1d0s2 sliced - - online c1t1d0s2
c1t2d0s2 sliced - - error c1t2d0s2
c1t3d0s2 sliced rootspare rootdg online c1t3d0s2
c1t4d0s2 sliced DATA_disk1 rootdg online c1t4d0s2
c1t5d0s2 sliced DATA_disk2 rootdg online c1t5d0s2
- - rootmirror rootdg removed was:c1t2d0s2

Again remove *ALL* duplicate c1t0d0s2 entries from VxVM control.

root@XXXXXXX# vxdisk rm c1t2d0s2

Well, no more entry for disk c1t2d0s2.

root@XXXXXXX# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
c1t0d0s2 sliced rootdisk rootdg online c1t0d0s2
c1t1d0s2 sliced - - online c1t1d0s2
c1t3d0s2 sliced rootspare rootdg online c1t3d0s2
c1t4d0s2 sliced DATA_disk1 rootdg online c1t4d0s2
c1t5d0s2 sliced DATA_disk2 rootdg online c1t5d0s2
- - rootmirror rootdg removed was:c1t2d0s2

To remove all possible stale dev_t will offline all the paths to the disk.

root@XXXXXXX# luxadm -e offline /dev/dsk/c1t2d0s2

Clean up the things using devfsadm command –

root@XXXXXXX# devfsadm -Cv
devfsadm[6369]: verbose: removing node /devices/pci@8,700000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@8,700000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing node /devices/pci@8,700000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@8,700000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing node /devices/pci@9,700000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@9,700000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing node /devices/pci@9,700000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@9,700000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing node /devices/pci@9,600000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@9,600000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing node /devices/pci@9,600000:devctl. invalid st_rdev
devfsadm[6369]: verbose: mknod /devices/pci@9,600000:devctl 0l/3l/20600
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:a
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s0
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:b
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s1
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:c
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s2
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:d
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s3
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:e
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s4
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:f
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s5
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:g
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s6
devfsadm[6369]: verbose: removing file: /devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf26c8a4,0:h
devfsadm[6369]: verbose: removing file: /dev/dsk/c1t2d0s7

root@XXXXXXX# vxdctl enable

Now it’s time to insert the disk.

root@XXXXXXX# luxadm insert_device

Once the disk have been inserted or replaced, use "vxdctl enable" and "vxdiskadm" option 5 after syncing with the remaining mirror.

root@XXXXXXX# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS c#t#d#_NAME
c1t0d0s2 sliced rootdisk rootdg online c1t0d0s2
c1t1d0s2 sliced - - online c1t1d0s2
c1t2d0s2 sliced rootmirror rootdg online c1t2d0s2
c1t3d0s2 sliced rootspare rootdg online c1t3d0s2
c1t4d0s2 sliced DATA_disk1 rootdg online c1t4d0s2
c1t5d0s2 sliced DATA_disk2 rootdg online c1t5d0s2

Here we go! Done with replacing & re-mirroring the VxVM disk in Solaris 8 with legacy VERITAS version 3.5, that too without rebooting the box. Actually, ultimate solution for this situation is a reconfiguration reboot(s), and however standard reboots have been found to occasionally solution for such condition but rebooting the host is most of the time not a practical substitute for mission-critical servers and applications hosted on it.

Just in case if you face this issue on servers with Solaris 10 with version prior or equal to VERITAS Storage Foundation 5.0 MP3 then following methods will surely will be a big help.

Solution 1:
===========

With Storage Foundation 5.0 MP3, and above the following commands can be tried first without restarting vxconfigd:

# rm /etc/vx/disk.info
# rm /dev/vx/dmp/*
# rm /dev/vx/rdmp/*
# vxddladm -c assign names

Solution 2:
============

With any version prior to 5.0 MP3 where the disk.info file exists, vxconfigd must be restarted in order to recreate the DMP nodes, and the disk.info file.

IMPORTANT: If this is part of a VCS cluster, freeze all service groups before running these commands.

# rm /etc/vx/disk.info
# rm /dev/vx/dmp/*
# rm /dev/vx/rdmp/*
# vxconfigd -k

The special thing about this episode is that I able to remove duplicate disk device entries on Solaris 8 with VERITAS 3.5 without reboot. It worked very well in my case and I’m hopeful that you will find this procedure helpful.

Slice up ASM Disks on Solaris Host.

Yes, yes… I know really a long time…. Past few days were quite demanding and quite occupied with routine work. Anyways, today I finally decided to take some time out and write up something about how to slice up ASM disks on Solaris host.

I’ll start from some basics and get some background about -

What is ASM?

It's - Automatic Storage Management.

ASM provides filesystem and volume manager capabilities built into the Oracle database kernel. With this capability, ASM simplifies storage management tasks, such as creating/laying out databases and disk space management. Since ASM allows disk management to be done using familiar create/alter/drop SQL statements, DBAs do not need to learn a new skill set or make crucial decisions on provisioning.

ASM is a management tool specifically built to simplify the job of the DBA. It provides a simple storage management interface across all server and storage platforms. ASM provides the DBA flexibility to manage a dynamic database environment with increased efficiency. This feature is a key component of Grid Computing and Database Storage Consolidation. The following are some key benefits of ASM:

• I/O is spread evenly across all available disk drives to prevent hot spots and maximize performance.
• ASM eliminates the need for over provisioning and maximizes storage resource utilization facilitating database consolidation.
• Inherent large file support.
• Performs automatic online redistribution after the incremental addition or removal of storage capacity.
• Maintains redundant copies of data to provide high availability, or leverage 3rd party RAID functionality.
• Supports Oracle Database 10g as well as Oracle Real Application Clusters (RAC).
• Capable of leveraging 3rd party Multipathing technologies & much more.

Now let's see how to format disk for ASM in Solaris.

NOTE - Currently, in 10g and 11gR1, ASM disks cannot be larger than 2TB, and needs to be a minimum of 4Mb

To prevent Oracle ASM from overwriting the partition table, you cannot use slices that start at cylinder 0 (for example, slice 2).

If you need to feed the disks to ASM, which requires you to start your partitions at cylinder > 0. The disk slices to be used must skip the first cylinder (cylinder 0) to avoid overwriting the disk VTOC (Volume Table of Contents). The VTOC is a special area of disk set aside for aside for storing information about the disk’s controller, geometry and slices. Let me explain this in little details –

It is important to move the starting cylinder for the partition from cylinder 0 to cylinder 1, which can be done in the usual way before labelling the disk. This is because cylinder 0 contains the disk label at block 0. File Systems such as UFS or VxFS etc, which are placed into such partitions will not use cylinder 0, specifically to avoid overwriting the disk label, but other, perhaps less thoughtful, applications using the raw device will overwrite cylinder 0, and therefore delete the disk partition label, making the disk unusable.

SPARC platform uses 8 slices starting from 0 to 7. Below is the common configuration of these eight slices:

slice 0: Holds files and directories that make up the operating system.*
slice 1: Swap, Provides virtual memory, or swap space.
slice 2: Refers to the entire disk, by convention. The size of this slice should not be changed.**
slice 3: /export, Holds alternative versions of the operating system.
slice 4: /export/swap. Provides virtual memory space for client systems. ***
slice 5: /opt. Holds application software added to a system.
slice 6: /usr. Holds operating system commands--also known as executables-- designed to be run by users.
slice 7: /home. Holds files created by users.

Standard practice –

* = Cannot be used as ASM disk. Using this slice causes disk corruption and may render the disk as unusable.
** = Should not be used as ASM Disk, as slice refers to the entire disk (Including partition tables).
*** = Is the recommended slice to be used for ASM disk.

Use the -f option with format to take command input from the file instead of standard input.

It is a best practice on a Solaris system, to create a partition on the disk; such as slice 4 or 6, which skips the first 1Mb into the disk. Skipping 1Mb into the disk is done to skip the OS label/VTOC, as well as to preserve alignment between ASM stripping and storage array internal stripping.

Ownership and permissions –

Take care to set the right permission & ownership for the raw devices. The user oracle must have read and write access to them.

Default owner is root:sys needs to be changed to oracle:dba

bash-3.00# ls -lhL /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4
crw-r----- 1 root sys 118, 64 Feb 16 02:10 /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4

bash-3.00# chown oracle:dba /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4

bash-3.00# ls -lhL /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4
crw-r----- 1 oracle dba 118, 64 Feb 16 03:00 /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4

Also the device file should have permissions set to 660 (recommended but not mandatory)

NOTE: - If you preparing disk to be added to ASM on RAC nodes then please make sure that you set appropriate permissions and ownership on all the nodes.

root@racnode1:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 16 16:49 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

root@racnode2:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 24 13:16 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

root@racnode3:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 24 13:16 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

root@racnode4:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 24 13:17 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

Once the disk has been discovered by the ASM instance, it is ready to be added to any DiskGroup.

SQL> select name, path, group_number from v$asm_disk
NAME PATH GROUP_NUMBER
------------------------------ --------------- ------------
          /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4 0
          /dev/rdsk/c0t60060E8015347E000001347E00001213d0s4 0
          /dev/rdsk/c0t60060E8015347E000001347E00001214d0s4 0
          /dev/rdsk/c0t60060E8015347E000001347E00001215d0s4 0

Disks have various header statuses that reflect its membership state within a DiskGroup. Disks can have the following header statuses, mount statuses & global statuses:

Header status:

    MEMBER:       Disk is member of existing diskgroup.
    CANDIDATE:    No part of diskgroup, may be added.
    PROVISIONED:  No part of diskgroup, may be added but administrator must first take additionally platform-specific action.
    INCOMPATIBLE: Version in disk header is not compatible with ASM storage.
    FORMER:       Disk was one part of diskgroup. Cleanly dropped.
    UNKNOWN:      Header has not been read.
    CONFLICT:     Disk not mounted due to a conflict.
    FOREIGN:      Disk contains data created by other products than ASM (e.g. datafile,logfile,OCR).

Mount  status:

    CACHED - Normal state for disk which are part of mounted diskgroup.(disk is present in storage system and part of a diskgroup)
    OPENED - Normal state, actively used by instance.(disk is present in storage system and accessed by ASM)
    CLOSED - Disk is present in storage system but not being accessed by ASM.
    CLOSING - ASM is in process of closing this disk.
    IGNORED - Disk is present in storage system but ignored by ASM (disk in system library is the same as in ASM library OR header no longer valid)
    MISSING - ASM metadata indicates: Disk is known in diskgroup, but no disk in storage system is found with name.

Global status:

    NORMAL - Online and operating normally.
    ADDING - Disk is being added and is pending validation by all instances that have the diskgroup mounted.
    DROPPING - Disk has been manually taken offline. Rebalancing will be commence. After this disk is expelled.
    DROPPED - Disk has been fully expelled from the diskgroup.
    HUNG - Disk drop operation hangs because insufficient space to relocate date.
    FORCING - Disk is removed without attempting to offload its data. Date will be recovered from redundant copies(where possible).
    UNKNOWN - Disk state is unknown (typical disk not mounted).

I see three main benifits out of this method, which might be quite helpful in enterprise setups where bulk stuffs needs to be carried out.

i. The disk formatting done via format command file hence this ensures the factor “standardization”. Using the format command file we make sure that all the ASM disks will have an unique VTOC across the environment which in-turns make sure standardization.

ii. Bulk disks can be formatted using simple for loop using this format command file & this will be accomplished in just a matter of one hit. For e.g.

# cat /tmp/ASM_Disks
c2t5006048452A8CCDDd1060
c2t5006048452A8CCDDd1061
c2t5006048452A8CCDDd1062
c2t5006048452A8CCDDd1063
c2t5006048452A8CCDDd1064
c2t5006048452A8CCDDd1065
c2t5006048452A8CCDDd1066
c2t5006048452A8CCDDd1067
c2t5006048452A8CCDDd1068
c2t5006048452A8CCDDd1069
c2t5006048452A8CCDDd1070
c2t5006048452A8CCDDd1071
c2t5006048452A8CCDDd1072

# for disk in `cat /tmp/ASM_Disks`
> do
> /usr/scripts/ASM_format.cmd -f $disk
> done

iii. In case of ASM disk label corruption, as we will use this format command file as a standard therefore unique VTOC will help to avoid chaos about “what was previous VTOC” and re-formatting will be easy and flawless.

Hope this helps!

Plex DISABLED RECOVER/STALE state while volume is ENABLED ACTIVE state.

Recently I had a situation where, plexes went into DISABLED RECOVER state while volumes were in ENABLED ACTIVE state. The reason that all volumes remained ENABLED ACTIVE state because, only one side of mirrored plexes were disabled however other side of mirror was in good state or enabled active. This is quite rarely seen situation (at least in my environment) hence I thought of writing a blog entry about this occurrence.

I believe, due to some temporary IO failure on disk c1t1d0 this had happened and I see some logs in diagnostic messages indicating drive c1t1d0 going offline... see below vxprint output

root:XXXXXXXX:/root # vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 29000 1218716511.6.XXXXXXXX

dm rootdisk c1t0d0s2 auto 16384 143328960 -
dm rootmirror c1t1d0s2 auto 20095 143318784 -

v crash - ENABLED ACTIVE 16780224 ROUND - fsgen
pl crash-01 crash ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdisk-03 crash-01 rootdisk 67131071 16780224 0 c1t0d0 ENA
pl crash-02 crash DISABLED RECOVER 16780224 CONCAT - RW
sd rootmirror-02 crash-02 rootmirror 16780224 16780224 0 c1t1d0 ENA

v home - ENABLED ACTIVE 2097152 SELECT - fsgen
pl home-01 home ENABLED ACTIVE 2106432 CONCAT - RW
sd rootdisk-09 home-01 rootdisk 102808127 2106432 0 c1t0d0 ENA
pl home-02 home DISABLED RECOVER 2106432 CONCAT - RW
sd rootmirror-04 home-02 rootmirror 35666880 2106432 0 c1t1d0 ENA

v networker - ENABLED ACTIVE 10485760 SELECT - fsgen
pl networker-01 networker ENABLED ACTIVE 10491456 CONCAT - RW
sd rootdisk-08 networker-01 rootdisk 92316671 10491456 0 c1t0d0 ENA
pl networker-02 networker DISABLED RECOVER 10491456 CONCAT - RW
sd rootmirror-05 networker-02 rootmirror 37773312 10491456 0 c1t1d0 ENA

[... Many lines, skipped for brevity ...]

To recover from such incident I would suggest to try below command first:

# vxrecover -bsE -g diskgroup_name

# vxrecover -bsE -g rootdg

Use vxtask list or vxtask -l list command to check the sync operation status.
Where,

vxrecover - perform volume recovery operations

-b => Performs recovery operations in the background. With this option, vxrecover runs in the background  to attach stale plexes and subdisks, and to resyn-chronize mirrored volumes and RAID-5 parity. If this is used with -s, volumes are started before recovery begins in the background.

-E => Starts disabled volumes or plexes even when they are in the EMPTY state. This is useful for start-ing up volumes restored by the vxmake utility when specified along with the -s option.

This would try to recover the plex & sync them again. Just in case if this doesn't works then I would prefer to detach the plex(es) & attach them again.. you can do so via "vxplex det" & "vxplex att".

In my case first option worked like piece of cake.

root:XXXXXXXX:/root # vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 29000 1218716511.6.XXXXXXXX

dm rootdisk c1t0d0s2 auto 16384 143328960 -
dm rootmirror c1t1d0s2 auto 20095 143318784 -

v crash - ENABLED ACTIVE 16780224 ROUND - fsgen
pl crash-01 crash ENABLED ACTIVE 16780224 CONCAT - RW
sd rootdisk-03 crash-01 rootdisk 67131071 16780224 0 c1t0d0 ENA
pl crash-02 crash ENABLED ACTIVE 16780224 CONCAT - RW
sd rootmirror-02 crash-02 rootmirror 16780224 16780224 0 c1t1d0 ENA

v home - ENABLED ACTIVE 2097152 SELECT - fsgen
pl home-01 home ENABLED ACTIVE 2106432 CONCAT - RW
sd rootdisk-09 home-01 rootdisk 102808127 2106432 0 c1t0d0 ENA
pl home-02 home ENABLED ACTIVE 2106432 CONCAT - RW
sd rootmirror-04 home-02 rootmirror 35666880 2106432 0 c1t1d0 ENA

v networker - ENABLED ACTIVE 10485760 SELECT - fsgen
pl networker-01 networker ENABLED ACTIVE 10491456 CONCAT - RW
sd rootdisk-08 networker-01 rootdisk 92316671 10491456 0 c1t0d0 ENA
pl networker-02 networker ENABLED ACTIVE 10491456 CONCAT - RW
sd rootmirror-05 networker-02 rootmirror 37773312 10491456 0 c1t1d0 ENA

[... Many lines, skipped for brevity ...]

Hope this helps someone.

BTW, just for refreshing the basics, let's take a look at Plex states and Condition flags.

EMPTY: This state indicates that you have not yet defined which plex has the good data (CLEAN), and which plex does not have the good data (STALE).

CLEAN: This state is normal and indicates that the plex has a copy of the data that represents the volume. CLEAN also means that the volume is not started and is not currently able to handle I/O (by the administrator's control).

ACTIVE: This state is the same as CLEAN, but the colume is or was currently started, and the colume is or was able to perform I/O.

SNAPDONE: This state is the same as ACTIVE or CLEAN, but is a plex that has been synchronized with the volume as a result of a “vxassist snapstart” operation. After a reboot or a manual start of the volume, a plex in the SNAPDONE state is removed along with its subdisks.

STALE: This state indicates that VxVM has reason to believe that the data in the plex is not synchronized with the data in the CLEAN plexes. This state is usually caused by taking the plex offline or by a disk failure.

SNAPATT: This state indicates that the object is a snapshot that is currently being synchronized but does not yet have a complete copy of the data.

OFFLINE: This state indicates that the administrator has issued the “vxmend off” command on the plex. When the administrator brings the plex back online using the “vxmend on” command, the plex changes to the STALE state.

TEMP: The TEMP state flags (TEMP, TEMPRM, TEMPRMSD) usually indicate that the data was never a copy of the volume’s data, and you should not use these plexes. These temporary states indicate that the plex is currently involved in a synchronization operation with the volume.

NODEVICE: This flag indicates that the disk drive below the plex has failed.

REMOVED: This flag has the same meaning as NODEVICE, but the system admin has requested that the device appear as failed.

IOFAIL: This flag is similar to NODEVICE, but it indicates that an unrecoverable failure occurred on the device, and VxVM has not yet verified whether the disk is actually bad.

Note: I/O to both the public and the private regions must fail to change the state from IOFAIL to NODEVICE.

RECOVER: This flag is set on a plex when two conditions are met:

1) A failed disk has been fixed (by using vxreattach or the vxdiskadm option, “Replace a failed or removed disk”).
2) The plex was in the ACTIVE state prior to the failure.

I'm sure above notes will stand helpful…