Slice up ASM Disks on Solaris Host.

Yes, yes… I know really a long time…. Past few days were quite demanding and quite occupied with routine work. Anyways, today I finally decided to take some time out and write up something about how to slice up ASM disks on Solaris host.

I’ll start from some basics and get some background about -

What is ASM?

It's - Automatic Storage Management.

ASM provides filesystem and volume manager capabilities built into the Oracle database kernel. With this capability, ASM simplifies storage management tasks, such as creating/laying out databases and disk space management. Since ASM allows disk management to be done using familiar create/alter/drop SQL statements, DBAs do not need to learn a new skill set or make crucial decisions on provisioning.

ASM is a management tool specifically built to simplify the job of the DBA. It provides a simple storage management interface across all server and storage platforms. ASM provides the DBA flexibility to manage a dynamic database environment with increased efficiency. This feature is a key component of Grid Computing and Database Storage Consolidation. The following are some key benefits of ASM:

• I/O is spread evenly across all available disk drives to prevent hot spots and maximize performance.
• ASM eliminates the need for over provisioning and maximizes storage resource utilization facilitating database consolidation.
• Inherent large file support.
• Performs automatic online redistribution after the incremental addition or removal of storage capacity.
• Maintains redundant copies of data to provide high availability, or leverage 3rd party RAID functionality.
• Supports Oracle Database 10g as well as Oracle Real Application Clusters (RAC).
• Capable of leveraging 3rd party Multipathing technologies & much more.

Now let's see how to format disk for ASM in Solaris.

NOTE - Currently, in 10g and 11gR1, ASM disks cannot be larger than 2TB, and needs to be a minimum of 4Mb

To prevent Oracle ASM from overwriting the partition table, you cannot use slices that start at cylinder 0 (for example, slice 2).

If you need to feed the disks to ASM, which requires you to start your partitions at cylinder > 0. The disk slices to be used must skip the first cylinder (cylinder 0) to avoid overwriting the disk VTOC (Volume Table of Contents). The VTOC is a special area of disk set aside for aside for storing information about the disk’s controller, geometry and slices. Let me explain this in little details –

It is important to move the starting cylinder for the partition from cylinder 0 to cylinder 1, which can be done in the usual way before labelling the disk. This is because cylinder 0 contains the disk label at block 0. File Systems such as UFS or VxFS etc, which are placed into such partitions will not use cylinder 0, specifically to avoid overwriting the disk label, but other, perhaps less thoughtful, applications using the raw device will overwrite cylinder 0, and therefore delete the disk partition label, making the disk unusable.

SPARC platform uses 8 slices starting from 0 to 7. Below is the common configuration of these eight slices:

slice 0: Holds files and directories that make up the operating system.*
slice 1: Swap, Provides virtual memory, or swap space.
slice 2: Refers to the entire disk, by convention. The size of this slice should not be changed.**
slice 3: /export, Holds alternative versions of the operating system.
slice 4: /export/swap. Provides virtual memory space for client systems. ***
slice 5: /opt. Holds application software added to a system.
slice 6: /usr. Holds operating system commands--also known as executables-- designed to be run by users.
slice 7: /home. Holds files created by users.

Standard practice –

* = Cannot be used as ASM disk. Using this slice causes disk corruption and may render the disk as unusable.
** = Should not be used as ASM Disk, as slice refers to the entire disk (Including partition tables).
*** = Is the recommended slice to be used for ASM disk.

Use the -f option with format to take command input from the file instead of standard input.

It is a best practice on a Solaris system, to create a partition on the disk; such as slice 4 or 6, which skips the first 1Mb into the disk. Skipping 1Mb into the disk is done to skip the OS label/VTOC, as well as to preserve alignment between ASM stripping and storage array internal stripping.

Ownership and permissions –

Take care to set the right permission & ownership for the raw devices. The user oracle must have read and write access to them.

Default owner is root:sys needs to be changed to oracle:dba

bash-3.00# ls -lhL /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4
crw-r----- 1 root sys 118, 64 Feb 16 02:10 /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4

bash-3.00# chown oracle:dba /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4

bash-3.00# ls -lhL /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4
crw-r----- 1 oracle dba 118, 64 Feb 16 03:00 /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4

Also the device file should have permissions set to 660 (recommended but not mandatory)

NOTE: - If you preparing disk to be added to ASM on RAC nodes then please make sure that you set appropriate permissions and ownership on all the nodes.

root@racnode1:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 16 16:49 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

root@racnode2:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 24 13:16 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

root@racnode3:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 24 13:16 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

root@racnode4:/dev/rdsk# ls -latr ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw
crw-r----- 1 oracle dba 118, 228 Jul 24 13:17 ../../devices/scsi_vhci/ssd@g60060e8015347e000001347e00001212:e,raw

Once the disk has been discovered by the ASM instance, it is ready to be added to any DiskGroup.

SQL> select name, path, group_number from v$asm_disk
NAME PATH GROUP_NUMBER
------------------------------ --------------- ------------
          /dev/rdsk/c0t60060E8015347E000001347E00001212d0s4 0
          /dev/rdsk/c0t60060E8015347E000001347E00001213d0s4 0
          /dev/rdsk/c0t60060E8015347E000001347E00001214d0s4 0
          /dev/rdsk/c0t60060E8015347E000001347E00001215d0s4 0

Disks have various header statuses that reflect its membership state within a DiskGroup. Disks can have the following header statuses, mount statuses & global statuses:

Header status:

    MEMBER:       Disk is member of existing diskgroup.
    CANDIDATE:    No part of diskgroup, may be added.
    PROVISIONED:  No part of diskgroup, may be added but administrator must first take additionally platform-specific action.
    INCOMPATIBLE: Version in disk header is not compatible with ASM storage.
    FORMER:       Disk was one part of diskgroup. Cleanly dropped.
    UNKNOWN:      Header has not been read.
    CONFLICT:     Disk not mounted due to a conflict.
    FOREIGN:      Disk contains data created by other products than ASM (e.g. datafile,logfile,OCR).

Mount  status:

    CACHED - Normal state for disk which are part of mounted diskgroup.(disk is present in storage system and part of a diskgroup)
    OPENED - Normal state, actively used by instance.(disk is present in storage system and accessed by ASM)
    CLOSED - Disk is present in storage system but not being accessed by ASM.
    CLOSING - ASM is in process of closing this disk.
    IGNORED - Disk is present in storage system but ignored by ASM (disk in system library is the same as in ASM library OR header no longer valid)
    MISSING - ASM metadata indicates: Disk is known in diskgroup, but no disk in storage system is found with name.

Global status:

    NORMAL - Online and operating normally.
    ADDING - Disk is being added and is pending validation by all instances that have the diskgroup mounted.
    DROPPING - Disk has been manually taken offline. Rebalancing will be commence. After this disk is expelled.
    DROPPED - Disk has been fully expelled from the diskgroup.
    HUNG - Disk drop operation hangs because insufficient space to relocate date.
    FORCING - Disk is removed without attempting to offload its data. Date will be recovered from redundant copies(where possible).
    UNKNOWN - Disk state is unknown (typical disk not mounted).

I see three main benifits out of this method, which might be quite helpful in enterprise setups where bulk stuffs needs to be carried out.

i. The disk formatting done via format command file hence this ensures the factor “standardization”. Using the format command file we make sure that all the ASM disks will have an unique VTOC across the environment which in-turns make sure standardization.

ii. Bulk disks can be formatted using simple for loop using this format command file & this will be accomplished in just a matter of one hit. For e.g.

# cat /tmp/ASM_Disks
c2t5006048452A8CCDDd1060
c2t5006048452A8CCDDd1061
c2t5006048452A8CCDDd1062
c2t5006048452A8CCDDd1063
c2t5006048452A8CCDDd1064
c2t5006048452A8CCDDd1065
c2t5006048452A8CCDDd1066
c2t5006048452A8CCDDd1067
c2t5006048452A8CCDDd1068
c2t5006048452A8CCDDd1069
c2t5006048452A8CCDDd1070
c2t5006048452A8CCDDd1071
c2t5006048452A8CCDDd1072

# for disk in `cat /tmp/ASM_Disks`
> do
> /usr/scripts/ASM_format.cmd -f $disk
> done

iii. In case of ASM disk label corruption, as we will use this format command file as a standard therefore unique VTOC will help to avoid chaos about “what was previous VTOC” and re-formatting will be easy and flawless.

Hope this helps!