Oracle-Solaris Patching using Live Upgrade

In today's post I'll be talking about a procedure for patching Solaris server having zones installed on it using Live Upgrade functionality.

The Solaris OS Recommended Patch Cluster provides critical Solaris OS Security, Data Corruption, and System Availability fixes & hence it is advisable to patch your Solaris systems twice in a year (atleast), as per Oracle-Sun's Critical Patch Update release schedule, I prefer to execute patch cycle for my environment in end of April and sometime late October every year.

Oracle-Sun CPUs are released on the Tuesday closest to the 17th of January, April, July, and October –

See - http://www.oracle.com/technetwork/topics/security/alerts-086861.html

In my environment, I use Live Upgrade to patch our Solaris systems. Reason behind using Live Upgrade for patching purpose are - 

1. Create a copy of the system environment; that is, a copy of the root (/) file system

2. Live Upgrade has build-in feature for splitting the mirrors of an SVM mirrored root (detach, attach, preserve options on lucreate) hence low overhead to deal with SVM mirror break stuffs separately etc.

3. Less downtime (not more than 15-20 mins) and minimal risk.

4. Better back out option. In case something breaks after patching revert to old BE and be at stage from where started, again that doesn’t take much downtime and safe option.

5. The most appropriate option for those Solaris servers who have zones/containers installed on it.

There might be many more benefits out there, however I find above benefits best fit for my purpose.

So to summarize, all tasks except the reboot can be accomplished on an operational production system; the impact on any running process is minimal. Live Upgrade is a combination of maximizing system availability when applying changes and minimizing risk by offering the ability to reboot to a known working state (your original environment).

Well. let's see how to do it in real life, in my current environment we have many servers which uses Solaris Volume Manager as their primary volume manager to manage the disk and data. So, let's take a look at patching procedure to patch servers who have SVM installed and configured on it along with zones installed on it sitting on ZFS filesystem.

Let us grab the output of metastat to understand metadevice placement -

# metastat -c
d32              p  1.0GB d4
d33              p  1.0GB d4
d36              p   40GB d4
d35              p  1.0GB d4
d34              p  4.0GB d4
d60              p   16GB d4
d30              p  1.0GB d4
d31              p  1.0GB d4
    d4           m  100GB d14 d24
        d14      s  100GB c1t0d0s4
        d24      s  100GB c1t1d0s4
d103             m   10GB d23 d13
    d23          s   10GB c1t1d0s3
    d13          s   10GB c1t0d0s3
d100             m   10GB d20 d10
    d20          s   10GB c1t1d0s0
    d10          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

Alright, my / is on d100 and /var is on d103. Let us create an alternative boot environment out of it.

# lucreate -c Sol10 -n Sol10pu -m /:/dev/md/dsk/d0:ufs,mirror -m /:/dev/md/dsk/d20:detach,attach -m /var:/dev/md/dsk/d3:ufs,mirror –m /var:/dev/md/dsk/d23:detach,attach

Here I'm trying to create a metadevice d0 representing / UFS filesystem having a sub-mirror d20 (sub-mirror d20 first gets detach from d100 and then attach to d0). Same thing applicable for /var filesystem and it's meta device configuration.

In above command I'm creating a new boot environment called Sol10pu using option “-n”, option “-m” Specifies the vfstab information for a new UFS-based BE.

NOTE: The -m option is not supported for BEs based on ZFS file systems.

NOTE: In case you're performing upgrade and patching in one go then point to be ponder - Before upgrading, you must install the Oracle Solaris Live Upgrade packages from the release to which you are upgrading. New capabilities are added to the upgrade tools, so installing the new packages from the target release is important. Example, you need to upgrade from Oracle Solaris 10 update 4 to Oracle Solaris update 8, so you must get the Oracle Solaris Live Upgrade packages from the Oracle Solaris update 8 DVD.


Once above command finishes, you will see you meta device configuration changed as follows -

# metastat -c
d32              p  1.0GB d4
d33              p  1.0GB d4
d36              p   40GB d4
d35              p  1.0GB d4
d34              p  4.0GB d4
d60              p   16GB d4
d30              p  1.0GB d4
d31              p  1.0GB d4
    d4           m  100GB d14 d24
        d14      s  100GB c1t0d0s4
        d24      s  100GB c1t1d0s4
d103             m   10GB d23
    d13          s   10GB c1t1d0s3
d100             m   10GB d20
    d10          s   10GB c1t1d0s0
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

d0 and d3 has one sub-mirror and d100 and d100 has one sub-mirror associated.

Also you will be able to see two boot environments on your Solaris system -

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10                      yes      yes    yes       no     -
Sol10pu                    yes      no     no        yes    -


Fine, so now we have 2 boot environments with us and we are going to patch the alternate BE (sol10pu) using a patching tool called PCA, BTW - I use PCA (Patch Check Advance) tool to apply patches to our Solaris systems. PCA has been setup to download patches via local web proxy to access outside systems.

PCA setup tips can be found at - http://www.par.univie.ac.at/solaris/pca/usage.html

What PCA needs in case setting it up -

- PERL distribution
- At least one server which is internet facing (this server will then act as a proxy to rest of servers)
- Patch cross-reference file called patchdiag.xref (latest one always while patching)
- Valid Oracle support (MOS) user ID and password
- If at all required, some wrapper scripts to PCA

To do so, let us mount the alternate BE on mount point say /a

# lumount Sol10pu /a
/a

Now I'll create a temporary directory to download the missing & required patches,

# mkdir -p /patchman/patches
My next job is to generate patch_order file,

# /bin/pca missingrs -R /a -H --format "%p-%c" > /patchman/patches/patch_order

Where -R stands for Alternative root directory
Where -H stands for Don't display descriptive headers
Where --format stands for Set output format to FORMAT

And go get them, download -

# /bin/pca missingrs -R /a -d -P /patchman/patches/ 

Where -d stands for download patches
Where -P stands for Patch download directory

Unmount the ABE -

# luumount /a

Now if you populate the /patchman/patches directory then you will see the list of patches in there.

Unzip all those patches -

# for i in `ls *.zip`;do
  unzip $i
  rm $i
done

Okay, at this stage we are ready upgrade ABE with patches available -

# cd /patchman/patches; luupgrade -n Sol10pu -s /patchman/patches -t `cat patch_order`

NOTE: Reactive patching may occasionally be necessary to address break-and-fix issues so in this case you can use LU with something like -

Apply single patch to ABE -

# luupgrade -n Sol10pu -s /patchman/patches -t

This will update patches on global as well as non-global zones.

Once the patches are installed it will automatically un-mounts the ABE sol10pu mounted on mount point /a.

Now it's time to activate the ABE sol10pu which just been patched using Live Upgrade utility.

# luactivate Sol10pu
A Live Upgrade Sync operation will be performed on startup of boot environment .
**********************************************************************

The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.

**********************************************************************

In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:

1. Enter the PROM monitor (ok prompt).

2. Change the boot device back to the original boot environment by typing:

     setenv boot-device /pci@0/pci@0/pci@2/scsi@0/disk@0,0:a

3. Boot to the original boot environment by typing:

     boot

**********************************************************************

Modifying boot archive service
Activation of boot environment successful.

# init 6
updating /platform/sun4v/boot_archive
SYSTEM GOING DOWN!!!!

NOTE: Live upgrade always uses init 6 or shutdown commands. Halt and reboot commands will create big time bang, be aware!!!

Once system is up it should show new kernel patch version.

Great, it's been a week after patching and application, DB owners are happy with patching stuffs and now we need to perform post patching stuffs upon certain confirmations.

POST PATCHING WORK TO DO - A Week LATER....
=========================================

Now a week later I need to delete the old boot environment and rebuild the metadevices to be in mirror layout.

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10                      yes      no     no        yes    -
Sol10pu                    yes      yes    yes       no     -

# ludelete Sol10
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.

So, the meta devices are looking like as below -

# metastat -c
d32              p  1.0GB d4
d33              p  1.0GB d4
d36              p   40GB d4
d35              p  1.0GB d4
d34              p  4.0GB d4
d60              p   16GB d4
d30              p  1.0GB d4
d31              p  1.0GB d4
    d4           m  100GB d14 d24
        d14      s  100GB c1t0d0s4
        d24      s  100GB c1t1d0s4
d103             m   10GB d23
    d13          s   10GB c1t1d0s3
d100             m   10GB d20
    d10          s   10GB c1t1d0s0
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

Now clear the d100 & d103 mirrors.

# metaclear d100 d103
d100: Mirror is cleared
d103: Mirror is cleared

# metastat -c
d32              p  1.0GB d4
d33              p  1.0GB d4
d36              p   40GB d4
d35              p  1.0GB d4
d34              p  4.0GB d4
d60              p   16GB d4
d30              p  1.0GB d4
d31              p  1.0GB d4
    d4           m  100GB d14 d24
        d14      s  100GB c1t0d0s4
        d24      s  100GB c1t1d0s4
d3               m   10GB d13
    d23          s   10GB c1t0d0s3
d0               m   10GB d10
    d20          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1
d13              s   10GB c1t1d0s3
d10              s   10GB c1t1d0s0

Next attach the sub-mirrors d10 & d13 to metadevices d0 and d3 respectively.

# metattach d0 d10
d0: submirror d10 is attached

# metattach d3 d13
d3: submirror d13 is attached

Hence my final meta device placement looks like as follows -

# metastat -c
d32              p  1.0GB d4
d33              p  1.0GB d4
d36              p   40GB d4
d35              p  1.0GB d4
d34              p  4.0GB d4
d60              p   16GB d4
d30              p  1.0GB d4
d31              p  1.0GB d4
    d4           m  100GB d14 d24
        d14      s  100GB c1t0d0s4
        d24      s  100GB c1t1d0s4
d3               m   10GB d23 d13 (resync-25%)
    d23          s   10GB c1t1d0s3
    d13          s   10GB c1t0d0s3
d0               m   10GB d20 d10 (resync-45%)
    d20          s   10GB c1t1d0s0
    d10          s   10GB c1t0d0s0
d1               m   16GB d11 d21
    d11          s   16GB c1t0d0s1
    d21          s   16GB c1t1d0s1

That's it. Now your done with patching your Solaris server and zones deployed on it.

Migrating VERITAS Volumes from one storage array to another storage array

Background:

This week I got a task to Migrate Veritas disk groups & their volumes on a specific server from one storage array to new storage array due to old storage array is ruining out of capacity.

To execute this task I split this into 3 phases -

1. Responsibilities with Storage

     a. Target server has to be zoned to new Storage system
     b. Provide LUNs/capacity to server (total capacity required is 3.5 TB 58x60G & 2x15G LUNs)

2. Responsibilities with me

      a. Migrate Veritas disk groups to new storage array
            A) Label all newly added SAN disks (may be using some scripted method) & make sure SAN disks are visible under VxVM
            B) Initialize all those new SAN disks with VxVM
            C) Add the SAN disk to disk group
            D) Mirror the volumes
            E) Verify if sync is completed
            F) Verify from vxprint if you see a new plex added to designated volumes
            G) If all is well then go ahead and detach old plex
            H) Once plex is disassociated from designated volumes then delete the old plex
            I) Verify if data group(s) & their volumes are on new storage array.
     b. Remove disks associated with old storage from Veritas configuration

3. Responsibilities with Storage

     a. Remove disks associated with old storage from server
     b. Take care of redundant paths

Back out -

     a. Have server full backup handy.

Execution -

List the disk groups needs to be migrated to new storage array.

# vxdg list
NAME STATE ID
xxxx_dg enabled,cds 1279726733.18.xxxxx
localswdg enabled,cds 1279726567.16.xxxxx
nass3_dg enabled 1074844579.1535.nassau3

Get the true picture of your VxVM configuration. Save output of this command for future reference

# vxprint -hrt

Now that Storage has attached SAN disks to servers HBA we need to label them and get them to VERITAS control.

List the existing Disks & new disks detected.

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
UNIX176_0a7a auto:sliced fls02c4_nass3_dg nass3_dg online <<<< All UNIX# disks are from old storage arrary
UNIX176_0a7f auto:sliced fls03c1_nass3_dg nass3_dg online
UNIX176_0a73 auto:sliced fls01c1_nass3_dg nass3_dg online
UNIX176_0a74 auto:sliced fls01c2_nass3_dg nass3_dg online
UNIX176_0a75 auto:sliced fls01c3_nass3_dg nass3_dg online
UNIX176_0a76 auto:sliced fls01c4_nass3_dg nass3_dg online
UNIX176_0a77 auto:sliced fls02c1_nass3_dg nass3_dg online
UNIX176_0a78 auto:sliced fls02c2_nass3_dg nass3_dg online
UNIX176_0a79 auto:sliced fls02c3_nass3_dg nass3_dg online
UNIX176_0a80 auto:sliced fls03c2_nass3_dg nass3_dg online
UNIX176_0a81 auto:sliced fls03c3_nass3_dg nass3_dg online
UNIX176_0a82 auto:sliced fls03c4_nass3_dg nass3_dg online
UNIX176_0dbf auto:cdsdisk xxxx_dg02 xxxxx_dg online
UNIX176_07f8 auto:cdsdisk localswdg01 localswdg online
UNIX176_07f9 auto:cdsdisk xxxx_dg01 xxxx_dg online
UNIX176_09a0 auto:sliced fls09c2_nass3_dg nass3_dg online
UNIX176_09aa auto:sliced fls11c4_nass3_dg nass3_dg online
UNIX176_09ab auto:sliced fls12c1_nass3_dg nass3_dg online
UNIX176_191c auto:cdsdisk localswdg02 localswdg online
UNIX176_0990 auto:sliced fls05c2_nass3_dg nass3_dg online
UNIX176_0991 auto:sliced fls05c3_nass3_dg nass3_dg online

[...]

disk_0 auto:none - - online invalid
disk_1 auto:none - - online invalid
emc_clariion0_1704 auto - - online invalid   <<<< All emc_* disks are from new storage arrary
emc_clariion0_1705 auto - - online invalid
emc_clariion0_1706 auto - - online invalid
emc_clariion0_1707 auto - - online invalid
emc_clariion0_1708 auto - - online invalid
emc_clariion0_1709 auto - - online invalid
emc_clariion0_1710 auto - - online invalid
emc_clariion0_1713 auto - - online invalid
emc_clariion0_1800 auto - - online invalid
emc_clariion0_1801 auto - - online invalid
emc_clariion0_1803 auto - - online invalid
emc_clariion0_1804 auto - - online invalid
emc_clariion0_1805 auto - - online invalid
emc_clariion0_1806 auto - - online invalid
emc_clariion0_1807 auto - - online invalid
emc_clariion0_1808 auto - - online invalid
emc_clariion0_1809 auto - - online invalid
emc_clariion0_1810 auto - - online invalid
emc_clariion0_3700 auto - - online invalid

[...]

emc_clariion0_5809 auto - - online invalid
emc_clariion0_5858 auto - - online invalid

Initialize the disks with VxVM and add them to appropriate disk group.

For a single disk -

# vxdisksetup -i emc_clariion0_5858 format=sliced

For multiple disks

# vxdisk list | awk '{print $1}' | grep -i emc > /tmp/EMC_disks
# for d in `cat /tmp/EMC_disks` ; do vxdisksetup -i $d format=sliced; done

Add disks to disk group

#!/bin/sh
#for e.g. vxdg -g nass3_dg adddisk nass3_dg01=emc_clariion0_1704
DG=nass3_dg
DISKS=`vxdisk list | awk '{print $3"="$1}' | grep -i emc`
#Output of command vxdisk list | awk '{print $3"="$1}' | grep -i emc is look like as = nass3_dg01=emc_clariion0_1704
for d in $DISKS;
do vxdg -g $DG adddisk $DISKS;
done

OR you can also use vxdiskadm menu based command to perfrom this activity.

Now you should see something like -

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
UNIX176_0a7a auto:sliced fls02c4_nass3_dg nass3_dg online
UNIX176_0a7f auto:sliced fls03c1_nass3_dg nass3_dg online
UNIX176_0a73 auto:sliced fls01c1_nass3_dg nass3_dg online
UNIX176_0a74 auto:sliced fls01c2_nass3_dg nass3_dg online
UNIX176_0a75 auto:sliced fls01c3_nass3_dg nass3_dg online
UNIX176_0a76 auto:sliced fls01c4_nass3_dg nass3_dg online
UNIX176_0a77 auto:sliced fls02c1_nass3_dg nass3_dg online
UNIX176_0a78 auto:sliced fls02c2_nass3_dg nass3_dg online
UNIX176_0a79 auto:sliced fls02c3_nass3_dg nass3_dg online
UNIX176_0a80 auto:sliced fls03c2_nass3_dg nass3_dg online
UNIX176_0a81 auto:sliced fls03c3_nass3_dg nass3_dg online
UNIX176_0a82 auto:sliced fls03c4_nass3_dg nass3_dg online
UNIX176_0dbf auto:cdsdisk xxxx_dg02 xxxxx_dg online
UNIX176_07f8 auto:cdsdisk localswdg01 localswdg online
UNIX176_07f9 auto:cdsdisk xxxx_dg01 xxxx_dg online
UNIX176_09a0 auto:sliced fls09c2_nass3_dg nass3_dg online
UNIX176_09aa auto:sliced fls11c4_nass3_dg nass3_dg online
UNIX176_09ab auto:sliced fls12c1_nass3_dg nass3_dg online
UNIX176_191c auto:cdsdisk localswdg02 localswdg online
UNIX176_0990 auto:sliced fls05c2_nass3_dg nass3_dg online
UNIX176_0991 auto:sliced fls05c3_nass3_dg nass3_dg online

[...]

disk_0 auto:none - - online invalid
disk_1 auto:none - - online invalid
emc_clariion0_1704 auto:sliced nass3_dg01 nass3_dg online
emc_clariion0_1705 auto:sliced nass3_dg02 nass3_dg online
emc_clariion0_1706 auto:sliced nass3_dg03 nass3_dg online
emc_clariion0_1707 auto:sliced nass3_dg04 nass3_dg online
emc_clariion0_1708 auto:sliced nass3_dg05 nass3_dg online
emc_clariion0_1709 auto:sliced nass3_dg06 nass3_dg online
emc_clariion0_1710 auto:sliced nass3_dg07 nass3_dg online
emc_clariion0_1713 auto:sliced nass3_dg08 nass3_dg online
emc_clariion0_1800 auto:sliced nass3_dg09 nass3_dg online
emc_clariion0_1801 auto:sliced nass3_dg10 nass3_dg online
emc_clariion0_1803 auto:sliced nass3_dg11 nass3_dg online
emc_clariion0_1804 auto:sliced nass3_dg12 nass3_dg online
emc_clariion0_1805 auto:sliced nass3_dg13 nass3_dg online
emc_clariion0_1806 auto:sliced nass3_dg14 nass3_dg online
emc_clariion0_1807 auto:sliced nass3_dg15 nass3_dg online
emc_clariion0_1808 auto:sliced nass3_dg16 nass3_dg online
emc_clariion0_1809 auto:sliced nass3_dg17 nass3_dg online
emc_clariion0_1810 auto:sliced nass3_dg18 nass3_dg online
emc_clariion0_3700 auto:sliced nass3_dg19 nass3_dg online

[...]

emc_clariion0_5809 auto:sliced nass3_dg59 nass3_dg online

Now that we have added new disks to the appropriate datagroups, next task to mirror the volumes.

NOTE: To decide how many disks are require to mirroe a perticular volume is very easy, put those many disks which will meet the current size of volume.

# vxassist -g nass3_dg mirror db_TESTDB_vol alloc=nass3_dg05,nass3_dg02,nass3_dg06,nass3_dg07,nass3_dg08,nass3_dg09,nass3_dg10,nass3_dg11,nass3_dg12,nass3_dg13,nass3_dg14,nass3_dg15,nass3_dg16,nass3_dg17,nass3_dg18,nass3_dg19,nass3_dg20,nass3_dg21,nass3_dg22,nass3_dg23,nass3_dg24,nass3_dg25,nass3_dg26,nass3_dg27,nass3_dg28,nass3_dg29,nass3_dg30,nass3_dg31,nass3_dg32,nass3_dg33,nass3_dg34,nass3_dg35,nass3_dg36,nass3_dg37,nass3_dg38,nass3_dg39,nass3_dg40,nass3_dg41,nass3_dg42,nass3_dg43,nass3_dg44,nass3_dg45,nass3_dg46,nass3_dg47,nass3_dg48,nass3_dg49,nass3_dg50,nass3_dg51,nass3_dg52,nass3_dg53,nass3_dg54,nass3_dg55

Repeat the previous step for rest of the data groups and volumes.

Likewise you can add disks to the remaining volumes and then we have to mirror them.

Check sync progress using -

# vxtask -l list
Task: 5912 RUNNING
Type: ATCOPY
Operation: PLXATT Vol db_TESTDB_vol Plex db_TESTDB_vol-02 Dg nass3_dg
Started: Fri Sep 16 16:05:24 2011
Throttle: 0
Progress: 4.27% 268713984 of 6291456000 Blocks
Work time: 29 minutes, 30 seconds (11:01:11 remaining)

Verify from vxprint , you should see a new plex added to db_TESTDB_vol volume

# vxprint -qthg nass3_dg db_TESTDB_vol

If everything looks good, then detach/disassociate & remove old plex, in short - break the mirror. (Before doing so get application owner consent)

# vxmend -g nass3_dg off db_TESTDB_vol-01
# vxplex -g nass3_dg -o rm dis db_TESTDB_vol-01

Repeat the previous two steps for rest of the plexes.

Well, by this time we can say we are done so verify if datagroup(s) & their volumes are on new storage arrary.

# vxprint -qthg nass3_dg db_TESTDB_vol
v db_TESTDB_vol - ENABLED ACTIVE 6291456000 SELECT db_TESTDB_vol-02 fsgen
pl db_TESTDB_vol-02 db_TESTDB_vol ENABLED ACTIVE 6291456000 STRIPE 3/128 RW
sd nass3_dg05-01 db_TESTDB_vol-02 nass3_dg05 0 125754880 0/0 emc_clariion0_1708 ENA
sd nass3_dg08-01 db_TESTDB_vol-02 nass3_dg08 0 125754880 0/125754880 emc_clariion0_1713 ENA
sd nass3_dg11-01 db_TESTDB_vol-02 nass3_dg11 0 125754880 0/251509760 emc_clariion0_1803 ENA
sd nass3_dg14-01 db_TESTDB_vol-02 nass3_dg14 0 125754880 0/377264640 emc_clariion0_1806 ENA
sd nass3_dg17-01 db_TESTDB_vol-02 nass3_dg17 0 125754880 0/503019520 emc_clariion0_1809 ENA
sd nass3_dg20-01 db_TESTDB_vol-02 nass3_dg20 0 125754880 0/628774400 emc_clariion0_3702 ENA
sd nass3_dg23-01 db_TESTDB_vol-02 nass3_dg23 0 125754880 0/754529280 emc_clariion0_3705 ENA
sd nass3_dg26-01 db_TESTDB_vol-02 nass3_dg26 0 125754880 0/880284160 emc_clariion0_3708 ENA
sd nass3_dg29-01 db_TESTDB_vol-02 nass3_dg29 0 125754880 0/1006039040 emc_clariion0_3800 ENA
sd nass3_dg32-01 db_TESTDB_vol-02 nass3_dg32 0 125754880 0/1131793920 emc_clariion0_3803 ENA

[...]

sd nass3_dg43-01 db_TESTDB_vol-02 nass3_dg43 0 125754880 2/1594132480 emc_clariion0_5704 ENA
sd nass3_dg46-01 db_TESTDB_vol-02 nass3_dg46 0 125754880 2/1719887360 emc_clariion0_5707 ENA
sd nass3_dg50-01 db_TESTDB_vol-02 nass3_dg50 0 125754880 2/1845642240 emc_clariion0_5800
ENA
sd nass3_dg53-01 db_TESTDB_vol-02 nass3_dg53 0 125754880 2/1971397120 emc_clariion0_5803 ENA

Yes, volume has been moved to new storage.

Now we are ready to remove disks associated with old storage from Veritas-configuration.

# vxdg -g nass3_dg rmdisk fls02c4_nass3_dg

Ask storage team to detach/remove the old disks permanently from the server.

This is overall procedure to migrate your data groups & their volumes from one storage to another storage.

Disable Large Segment Offload (LSO) in Solaris 10

In this blog article, I will share my understanding on Large Segment Offload (LSO). I got a task to disable the LSO on few of the servers (include zones).

Let's first understand what is LSO stands for and what is the purpose of using LSO.

As you see above LSO stands for Large Segment Offload.

TCP Offload Engine is an embryonic technology which is designed to offload TCP stack handling from the main system CPU to a processor built into NIC cards, hence the no CPU cycle and kernel time will get consumed.

LSO saves valuable CPU cycles by allowing the network protocol stack to handle large segments instead of the traditional model of MSS (TCP Maximum Segment Size) sized segments. In the traditional network stack, the TCP layer segments the outgoing data into the MSS sized segments and passes them down to the driver. This becomes computationally expensive with 10 GigE networking because of the large number of kernel functional calls required for every MSS segment. With LSO, a large segment is passed by TCP to the driver, and the driver or NIC hardware does the job of TCP segmentation (LSO offload the segmentation job on Layer 4 to the NIC driver). An LSO segment may be as large as 64 KByte. The larger the LSO segment, better the CPU efficiency since the network stack has to work with smaller number of segments for the same throughput.

So in simple words, use LSO for better network performance while reducing processor (CPU) utilization.

Segmentation is needed if a full TCP segment does not fit into the Ethernet Maximum Transmission Unit (MTU) size. With LSO, TCP segments do not need to get split in software implementation, this is done on the interface card hardware instead. Being much more effective, this improves the network performance while reducing the workload on the CPUs. LSO is most helpful for 10 Gigabit Ethernet network interfaces and on systems with slow CPU threads or lack of CPU resources.

Solaris LSO can be used if all of the three conditions are met :

1.The TCP/IP stack integrates LSO,

2.The Network Interface Card hardware supports it (for e.g. drivers like e1000g,ixgb,ixgbe etc),
3.The driver for this network card is capable of handling it.

Sadly, in most of the cases LSO seems to be not working that well hence it leads to disable the LSO support. Here is the ways to disable the LSO.

Ways to disable LSO -

Disable LSO by adding the following line in the /kernel/drv/e1000g.conf file (I’m using the e1000g interface/driver hence the file that I'm using is /kernel/drv/e1000g.conf) :

lso_enable=0,0,0,0,0,0,0,0;

After making the changes reboot is required or else if reboot is not possible then you can use ndd utility/command to disable it on a temporary basis and not persist across the reboot.

Using ndd you can disable it as shown below -

# ndd -set /dev/ip ip_lso_outbound 0

Also if you don't want to reboot the server after modifying the file /kernel/drv/e1000g.conf you can simply unplumb all of your e1000g interfaces with ifconfig, do "update_drv e1000g" to reload the .conf file, and then replumb and reconfigure the interfaces with ifconfig however still if I'm going to unplumb the network interfaces then eventually I'll be disturbing the services so reboot is the best option.

I had to disable the LSO as our application folks were experiencing slowness in their web application (response time etc.) It looks like LSO cause unstable connections & hence there are few observations like dropped sockets, dropped packets, packet reordering, packet retransmits and ultimately application folks observed slowness in their web application, NFS stuffs etc.

Installing Perl modules on Solaris 10 - An experience

In recent times I got a task to perform few Perl modules installation. To build the module in question (Compress::Raw::Zlib), I downloaded the module from CPAN (Comprehensive Perl Archive Network), verified that the MD5 checksum was correct, and used the following steps to compile the module:


# perl Makefile.PL
# make
# make install

The ‘make Makefile.PL’ completed successfully, but the make failed with the following errors:

# make
cp lib/Compress/Raw/Zlib.pm blib/lib/Compress/Raw/Zlib.pm
AutoSplitting blib/lib/Compress/Raw/Zlib.pm (blib/lib/auto/Compress/Raw/Zlib)
/usr/bin/perl /usr/perl5/5.8.4/lib/ExtUtils/xsubpp -typemap /usr/perl5/5.8.4/lib/ExtUtils/typemap -typemap typemap Zlib.xs > Zlib.xsc && mv Zlib.xsc Zlib.c
cc -c -I./zlib-src -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -xarch=v8 -D_TS_ERRNO -xO3 -xspace -xildoff -DVERSION=\"2.037\" -DXS_VERSION=\"2.037\" -KPIC "-I/usr/perl5/5.8.4/lib/sun4-solaris-64int/CORE" -DNO_VIZ -DGZIP_OS_CODE=3 -DUSE_PPPORT_H Zlib.c
/usr/ucb/cc: language optional software package not installed
*** Error code 1
make: Fatal error: Command failed for target `Zlib.o'

Above output indicate an error related to compiler & it is clearly showing that it is looking for cc complier however I was trying to build the module using gcc and not the cc (Sun studio) compiler but it seems like by default modules Makefile is referring to Sun Studio compiler and not the gcc complier. I do not have Sun Studio compiler on our systems & as this is being license software I can not even simply install it on any server due to compliance restrictions.

After doing a little bit research on the issue, I found that there is a way to fix this problem.  If you want to use gcc to build all Perl modules on a system, you can permanently remove the Sun Studio compiler references by adjusting the “cccdlflags” and “optimize” variables in /usr/perl5/5.8.4/lib/sun4-solaris-64int/Config.pm:

root@XXXX:/usr/perl5/5.8.4/lib/sun4-solaris-64int# egrep '(KPIC|O3)' Config.pm

cccdlflags='-KPIC'
optimize='-xO3 -xspace -xildoff'

After removing the above entries make was successful with gcc complier.

There is yet another method to do the Perl module installation even not performing above changes to Config.pm. You can simply build Perl module using –

- /usr/perl5/bin/perlgcc Makefile.PL
- /usr/sfw/bin/gmake
- /usr/sfw/bin/gmake test
- /usr/sfw/bin/gmake install

What we were discussing in above is how to install Perl modules manually and how to tackle with complier related issues. There is yet another automated method to install Perl module and which is most preferred and quick method to do it.

Simply export the http_proxy environment variable with your proxy server name or IP address and valid port number so that you can directly reach to the internet via proxy.


# http_proxy=myproxy:8080; export http_proxy
# perl -MCPAN -e shell

If this is the first time this script is run it will ask a lot of questions and store the results in Config.pm

Then simply run –

1. Start the CPAN shell script: /usr/perl5/bin/perl -MCPAN -e shell.
2. Install a module, e.g.: install CGI::Session
3. Exit the shell script: exit

To check the already installed modules execute –


# perl -e 'use HTML::Parser;'

If nothing is returned, Perl was able to locate the module. Otherwise, you will see “Can't locate HTML/Parser.pm in @INC”

One more experience to share here. I installed Threads Perl module however user was not able to execute his scripts due to current Perl distribution was not build or complied with “useithreads” configuration option hence he requested to recompile Perl distribution with threads option enable.

Here is a string that I use to compile Perl distribution with threads option.


# sh Configure -Dusethreads -Dprefix=/usr/perl5/5.14.1

With new Perl compilation user was able to execute his scripts with threads module.

Hope this document will help to those who are new to this work.

IBM’s Private Cloud Computing Platform (ISDM)

In last post I've discussed about Cloud Computing at a broad level and as I said would be writing the brief article on IBM private cloud platform. Since past few months I've been involved in evaluating IBM private cloud technology known as ISDM (IBM Integrated Service Delivery Manager) and herein I'm writing about my own understandings and experiences about IBM private cloud PoC that recently deployed in our test environment.

The below document gives very well information about IBM private cloud solution. I hope you will find this document helpful.

A Review of IBM Private Cloud

Cloud computing – Revisited

Well, before I write something about IBM Cloud Computing platform (IBM Integrated Service Delivery Manager) I would like to explain few of my understandings about Cloud Computing & its implementation factors.

I’m sure almost everyone knows about cloud models/types which are namely public cloud, private cloud, hybrid cloud & community cloud also these all clouds have specific service delivery models in particular, IT-as-a-Service, Platform-as-a-service & Software-as-a-Service.

Today, I would like to write an article on implementation factor(s) of cloud computing. Since past 8-9 months have been working on various cloud technology platforms/products and quite a few POC environments to test out & make out which product best suits to an organization specific requirements. There is no ready made cloud product out there which will able meet all of your organization requirements hence you need to take a call whether customizing the solution or altering the organization policies, structure i.e. whatever requirement. Trust me testing the technology is an easy job however to fix on what is the best fit product for an organization is a “not easy” job as this is something which is beyond just a technology & cloud implementers, decision makers and technical people should look at it from business viewpoint too anyways this is not a topic of discussion here. 

Let’s start this discussion with traditional IT system management portfolio & service delivery model. In ideal scenario a particular software development project gets a requirement to develop a product and resources required to deliver that product, the project manager or project lead starts identifying the resource requirements and ask IT department to procure necessary systems, OS, software, databases, toolkits or provision necessary systems, OS, software, databases via available IT infrastructures which takes significant amount of time since person who driving the project has to get many things approved by higher management, software compliance team, information security team etc  then procurement start which takes several weeks or he raise a request to IT department to provision systems with necessary OS, software, databases which also takes several days and so on. Then only actual application, product development starts, tested & deployed on that infrastructure. We can see several challenges in this model as described below – 

Need large Capex - Large investments required to procure the infrastructure required for any product development.

Poor utilization of resources - Application usage is not going to be constant yet the infrastructure is provisioned for peak demand hence the infrastructure remains under-utilized for a major part of the time.

Slow Time-to-Market - This model of procuring and provisioning infrastructure usually requires significant time and reduces the agility of an organization in creating new business solutions.

Now we can clearly see how traditional IT system management portfolio & service delivery model prove expensive in current competitive, difficult economic market.

Also on the other hand, still nothing is accountable as such. For system administrators it’s becomes a challenge to showcase or prove that a particular system or systems are under utilized which is dedicated to a specific functional group and those can be shared among the other groups. In such cases cost grows, complexities increases as number of hardware resources and software resources increases, data center space, cooling cost increases, requires more man power to handle the infrastructure, lack of standardization due to silos & much more to talk about. 

Another point of view, Over 70% of IT budgets in a typical Data Centre goes just to "keeping the lights on", especially keeping the “green” lights on! Conversely, only a small portion of each dollar spent on IT today creates a direct business benefit. Since Data Center IT assets become obsolete approximately every 5 years, the vast majority of IT investment is spent on upgrading various pieces of infrastructure and providing redundancy and recover-ability: activities that consume approximately 60 to 80% of IT expenditures without necessarily providing optimal business value or innovation.

To overcome on above challenges to a certain level virtualization & consolidation has played a major role however IT has started seen many challenges with virtualization environment like VM sprawl, costing etc consequently to overcome on those challenges & challenges  of overall service delivery methodology Cloud Computing technology evolved. 

Cloud Computing is a model of service delivery and access where dynamically scalable and virtualized resources are provided as a service over the Internet.

Cloud Computing offers an alternative approach that profoundly transforms the way in which information and services are consumed and provided and can enable businesses to:

Lower costs by using energy and resources more efficiently

Enhance agility, growth, and profitability

Simplify operations and management

Ensure elastic and trusted collaboration between various groups which results into visibility & standardized, smooth operations.

Faster time to market

On-demand elastic, dynamic infrastructure & lot more.

Cloud computing addresses many of the challenges of IT silos: inefficiencies, high costs, and ongoing support and maintenance concerns, as well as increasing user demand for services.

Evolution towards Cloud

Both "private" and "public" cloud computing is based on qualities such as self-service, pay-as-you-go charge-back, on-demand provisioning, and the appearance of unbounded scalability.  

Public cloud has its own benefits and challenges; everybody knows the benefits so we will see what all challenges of public cloud – 

Public clouds like Amazon AWS, Microsoft Azure, Google AppEngine offer infrastructure and platforms as services over the internet. In public clouds, resources and costs are shared by users who use them over the internet on pay per use model.

This model appeals especially to startups and small organizations that have not invested in hardware resources and are looking for ways to avoid the large capex involved in procuring infrastructure upfront. Even though there are several benefits like cost savings, faster time to market, etc., from this model, there are a few challenges listed below that are preventing wide scale adoption of public clouds.

Security - The biggest blockade is the possible security issues due to multi-tenant nature of public clouds. There is security and privacy concerns with sharing same physical hardware with unknown parties that need to addressed.

Control over IT – No direct control of IT infrastructure hence putting up mission critical, data sensitive applications is a potential risk.

Leveraging Existing Investment - Most large organizations that have already invested in their own data centers would see a need to leverage those investments as an important criterion in adopting cloud computing. 

Corporate Governance and Auditing: Performing governance and auditing activities with the corporate data abstracted in the public cloud poses challenges, which are yet to be addressed.  The limitation or law for storing data across different national boundaries etc. 

Outage – There are several outages of public cloud vendor and that creates a chaos to really want to adapt to public cloud for their mission critical applications.

Incident history -

Microsoft Azure: malfunction - 22 h outage on March 13/14, 2008 in Windows Azure,

S3 outage: authentication service overload - 2 h outage on Feb 15, 2008 leading to unavailability,

S3 outage: Single bit error leading to 6–8 h outage on July 20, 2008 gossip protocol blowup,

FlexiScale: core network failure 18 h outage on Oct 31, 2008  

The most up-to-date AWS outage around 48 h.

 

Well, the most important positive about private cloud offer a critical additional benefit: TRUST. This ability to offer elastic computing without sacrificing security or control that is driving many businesses to moving to this delivery model for IT services.

Initially, most private clouds will be made up almost entirely of internal resources. A private cloud can combine both external and internal cloud resources to meet the needs of an application system, and that combination, which is totally under enterprise control using unified management, can change moment by moment. This is also called as Hybrid cloud. With a private cloud, enterprises can run processes internally and externally, having established the private cloud as the control point for workloads. With control through a unified management tool and a user-centric view, the private cloud thus enables IT to make the best decisions about whether to use internal or external resources, or both, and allows that decision to be made on a real-time basis to meet user service needs.

The movement toward cloud computing began for the enterprises with data center virtualization and consolidation of server, storage, and network resources to reduce redundancy and wasted space and equipment with measured planning of both architecture (including facilities allocation and design) and process.

Below are major three stages to cloudify your organization. (There are several stages in between like requirement gathering, building service offerings & service catalogs upon gathered information, security, software compliance, information security compliance, defining billing model/charge back system etc)

Stage 1: Consolidation and Virtualization

Consolidation is a critical application of virtualization, enabling IT departments to regain control of distributed resources by creating shared pools of standardized resources that can be rationalized and centrally managed. Many IT departments already are consolidating under-utilized computing resources by running multiple applications on a single physical server with virtualization technology from IBM AIX, Sun-Oracle Zones/Container & LDOMs, and VMware, Linux KVM etc.

Stage 2: Automation and Optimized Virtualization

In this stage, virtualization optimizes IT resources and increases IT agility, thus speeding time-to market for services. Through automation, Data Centers systematically remove manual labor requirements for the run-time operation of the data center. To create a cloud service, self-service and metering (feedback about the cost of the resources allocated) are offered in addition to automation.  

 

Stage 3: Federation

Linking disparate cloud computing infrastructures with one another by connecting their individual management infrastructures allows disparate cloud IT resources and capabilities-capacity, monitoring, and management-to be shared, much like power from a power grid. It also enables unified metering and billing, one-stop self-service provisioning, and the movement of application loads between clouds, since federation can occur across data center and organization boundaries, with cloud internetworking. Cloud internetworking is the network technology enabling the linkage of disparate cloud systems in a way that accommodates the unique nature of cloud computing and the running of IT workload.

Hence after stage 3 our private cloud should look like as follows.

 

There is much more to discuss, write on this topic however for this blog entry I’ve explained a very high level overview on implementing private cloud thoughts, I hope this helps.