/var is full & causes Sun patching

NOTE: Below is information on how to free up space in /var by deleting certain patch related files.  This should only be used as a last resort of gaining space in /var.


PS: Do not use this procedure on server hosting non-global zones.  It will break the ability to install non-global zones.This procedure has fairly risky impact to the package and patch database. 

It has been observed that sometimes /var is almost full & /var/sadm eating up a lot of disk space. It leads to a problem sometimes, especially during patching. Normally we are uncertain on what to delete or move to make adequate space available so that you can start with patching. Following procedure tells what exactly /var/sadm/pkg contains & what we can delete or move to make adequate space available & get going with patching.

If /var/sadm/pkg is really big (it holds patch backout info):
----------------------------------------------------------------

Check for the existence of "obsolete.Z" files in the /var/sadm/pkg hierarchy. If you've done a lot of patching these files can take up a lot of space. They can be removed without ill effect. For example:

# find /var/sadm/pkg -type f -name obsolete.Z -exec rm {} +

This can recover several hundreds of megabytes depending on the number of patches applied.
If you don't want to backout the patches again you can just get rid of the undo backup files:

# find /var/sadm/pkg -name '1?????-??' -type d -exec rm -rf {} +
-------------------------------------------------------------------------------
There are two ways of deleting the backout information.
1. Remove all undo.Z and obsolete.Z files from the /var/sadm/pkg directory structure. This frees most of the space but also prevents any patch from being backed out. This can be catastrophic if a patch is applied that causes problems and needs to be backed out. If you have removed the backout files following the patch install, it's impossible to remove the patch.

2. Remove only the obsolete.Z files from the /var/sadm/pkg directory structure. This still frees a lot of space but it also preserves the ability to remove any of the most recent versions of any patch. This can be particularly useful if a problem is found with a freshly applied patch. If removal of backout information is unavoidable, this would be the better method of the two.
Here is an example of a command which deletes all obsolete.Z files:

# find /var/sadm/pkg -name obsolete.Z -exec rm {} \;

NOTE 1: In the Solaris 10 OS, there are additional entries located in the /var/sadm/pkg//save directory structure: specifically, the pspool directory and everything under it. These files and directories are created and utilized by the updated Solaris 10 patching utilities. For example, there will be more undo.Z files found under the pspool directory. Such as:

/var/sadm/pkg/SUNWcsr/save/pspool/SUNWcsr/save/118833-36/undo.Z

Removing those undo.Z files will free up some space but it is highly discouraged.
Infodoc 14295
----------------------------------------------------------------------------------
E.g.

# du -skh /var/sadm
1.3G /var/sadm


# find /var/sadm/pkg -type f -name obsolete.Z -exec rm {} +

# du -skh /var/sadm
391M /var/sadm

OR THE BEST WAY TO SEPARATE THE /var/sadm. Here you can have /var/sadm on internal disk as a soft partition in case you have enough space available on internal disk.

# metainit d## -p d4 4g

# newfs /dev/md/rdsk/d##
# mount /dev/md/dsk/d## /mnt
#rsync -aHv /var/sadm/ /mnt/
PS: repeat rsync to verify everything is copied.
# rsync -aHv --delete /var/sadm/ /mnt/
# cd /var/
# rm -r sadm
# mkdir /var/sadm/
# chmod 755 /var/sadm
# chown root:sys /var/sadm
# umount /mnt
# vi /etc/vfstab

Add:
/dev/md/dsk/d## /dev/md/rdsk/d## /var/sadm ufs 2 yes logging
# mount -a

That's it!!

LiveUpgrade problems

I came across with few incidents recently related to LiveUpgrade & I would like to share it to system administrators community through this blog.

Issue #1

Problem -

LiveUpgrade stuck/hang at "Validating file system requests"

# lucreate -c Sol10u7 -n Sol10u8 -m /:/dev/md/dsk/d0:ufs,mirror \
-m /:/dev/md/dsk/d20:detach,attach \
-m /var:/dev/md/dsk/d3:ufs,mirror \
-m /var:/dev/md/dsk/d23:detach,attach

Discovering physical storage devices
Discovering logical storage devices
Cross referencing storage devices with boot environment configurations
Determining types of file systems supported
Validating file system requests

HANG...HANG...HANG...HANG..YOU'RE STUCK...HANG...IT SUCKS...WHAT'S NEXT???

Solution -

Just execute -

# devfsadm -v -C

It will clear any broken device links under /dev or unreferenced device links under /dev directory. After performing this LiveUpgrade work like a piece of cake!

Issue #2

Problem -

After LiveUpgrade/patching Sol10u7 to Sol10u8, I activated the ABE Sol10u8 & booted off from it. After doing so I deleted the one of the container as we no longer needed that container & cleared up the metadevice where the zoneroot was residing. Deleting container, I certainly made changes to ABE Sol10u8 however those were not obliviously reflected to original BE Sol10u7. So now while deleting Sol10u7 LU program finds a mismatch & fails to delete the boot environment.

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10u7 yes no no yes -
Sol10u8 yes yes yes no -

# ludelete Sol10u7
ERROR: mount: /zone1: No such file or directory
ERROR: cannot mount mount point device /.alt.tmp.b-k2c.mnt/zone1 device /zone1
ERROR: failed to mount file system /zone1 on /.alt.tmp.b-k2c.mnt/zone1
ERROR: unmounting partially mounted boot environment file systems
ERROR: cannot mount boot environment by icf file /etc/lu/ICF.2
ERROR: Cannot mount BE .
Unable to delete boot environment.

Solution -

I don't know if it's a Sun supported solution or workaround. I simply edit the /etc/lu/ICF.nn file for the problematic BE, and delete the lines that reference the missing (or moved) filesystems & it worked fine.

# ludelete Sol10u7
Determining the devices to be marked free.
INFORMATION: Unable to determine size or capacity of slice .
ERROR: An error occurred during creation of configuration file.
WARNING: Target BE BE ID <2> unable to verify file systems belonging to BE are not mounted.
WARNING: Unable to determine disk partition configuration information for BE .
WARNING: Unable to determine the devices/datasets to be freed for BE .
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.

# echo $?
0

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10u8 yes yes yes no -

Adding one more tip - 31 Aug 2010

Issue #3


Problem: -

While ludelete operation I got below error -

# ludelete Sol10u8_stage1
ERROR: Read-only file system: cannot create mount point
ERROR: failed to create mount point for file system
ERROR: unmounting partially mounted boot environment file systems
ERROR: cannot mount boot environment by icf file
ERROR: Cannot mount BE .
Unable to delete boot environment.

Solution: -


As a part of solution, I used unofficial method to get rid of the Pesky BE. I modified /etc/lu/ICF.2 a bit and TRUST ME "lufslist" IS YOUR TRUE FRIEND.

# lufslist Sol10u8_stage1 && lufslist s10s_u8wos_08a


boot environment name: Sol10u8_stage1
Filesystem fstype device size Mounted on Mount Options
----------------------- -------- ------------ ------------------- --------------
/dev/md/dsk/d1 swap 34365898752 - -
/dev/md/dsk/d0 ufs 10743250944 / -
/dev/md/dsk/d3 ufs 8596684800 /var -
/dev/md/dsk/d31 ufs 1073741824 /home -
/dev/md/dsk/d30 ufs 1073741824 /opt/patrol logging
/dev/md/dsk/d32 ufs 536870912 /etc/opt/OV logging
/dev/md/dsk/d33 ufs 1073741824 /cims logging
/dev/md/dsk/d34 ufs 4294967296 /oraarch logging


boot environment name: s10s_u8wos_08a
This boot environment is currently active.
This boot environment will be active on next system boot.


Filesystem fstype device size Mounted on Mount Options
----------------------- -------- ------------ ------------------- --------------
/dev/zvol/dsk/rpool/swap swap 34366029824 - -
rpool/ROOT/s10s_u8wos_08a zfs 6566768640 / -
/dev/md/dsk/d33 ufs 1073741824 /cims -
/dev/md/dsk/d31 ufs 1073741824 /home -
rpool zfs 55018053120 /rpool -
zp00/backup zfs 15671161726 /backup -
zp00/opt-OV zfs 1750692636 /opt/OV -
zp00/pw zfs 598801456 /opt/pw -
zp00/oracle zfs 8574417038 /oracle -
/dev/md/dsk/d34 ufs 4294967296 /oraarch -
zp00/orabkup zfs 345610320 /orabkup -
zp00/oemagent zfs 1251898296 /oemagent -
zp00/oradata1 zfs 1231904784 /oradata1 -
zp00/oradata2 zfs 6858200184 /oradata2 -
zp00/oradata3 zfs 2111803452 /oradata3 -
zp00/oradata4 zfs 880723206 /oradata4 -
zp00/patrol_sw zfs 27280 /patrol_sw -
/dev/md/dsk/d32 ufs 536870912 /etc/opt/OV -
/dev/md/dsk/d30 ufs 1073741824 /opt/patrol -
zp00/var-opt-OV zfs 11173308982 /var/opt/OV -


# cat ICF.2
Sol10u8_stage1:-:/dev/md/dsk/d1:swap:67120896
Sol10u8_stage1:/:/dev/md/dsk/d0:ufs:20982912
Sol10u8_stage1:/var:/dev/md/dsk/d3:ufs:16790400
Sol10u8_stage1:/cims:/dev/md/dsk/d33:ufs:2097152
Sol10u8_stage1:/home:/dev/md/dsk/d31:ufs:2097152
Sol10u8_stage1:/rpool:rpool:zfs:0
Sol10u8_stage1:/backup:zp00/backup:zfs:0
Sol10u8_stage1:/opt/OV:zp00/opt-OV:zfs:0
Sol10u8_stage1:/opt/pw:zp00/pw:zfs:0
Sol10u8_stage1:/oracle:zp00/oracle:zfs:0
Sol10u8_stage1:/oraarch:/dev/md/dsk/d34:ufs:8388608
Sol10u8_stage1:/orabkup:zp00/orabkup:zfs:0
Sol10u8_stage1:/oemagent:zp00/oemagent:zfs:0
Sol10u8_stage1:/oradata1:zp00/oradata1:zfs:0
Sol10u8_stage1:/oradata2:zp00/oradata2:zfs:0
Sol10u8_stage1:/oradata3:zp00/oradata3:zfs:0
Sol10u8_stage1:/oradata4:zp00/oradata4:zfs:0
Sol10u8_stage1:/patrol_sw:zp00/patrol_sw:zfs:0
Sol10u8_stage1:/etc/opt/OV:/dev/md/dsk/d32:ufs:1048576
Sol10u8_stage1:/opt/patrol:/dev/md/dsk/d30:ufs:2097152
Sol10u8_stage1:/rpool/ROOT:rpool/ROOT:zfs:0
Sol10u8_stage1:/var/opt/OV:zp00/var-opt-OV:zfs:0

I wonder why ZFS filesystems listed in this file and hence it is trying to mount BR on existing ZFS BE so I just altered file with below entries -

# vi ICF.2
"ICF.2" 22 lines, 1001 characters
Sol10u8_stage1:-:/dev/md/dsk/d1:swap:67120896
Sol10u8_stage1:/:/dev/md/dsk/d0:ufs:20982912
Sol10u8_stage1:/var:/dev/md/dsk/d3:ufs:16790400
Sol10u8_stage1:/cims:/dev/md/dsk/d33:ufs:2097152
Sol10u8_stage1:/home:/dev/md/dsk/d31:ufs:2097152
Sol10u8_stage1:/oraarch:/dev/md/dsk/d34:ufs:8388608
Sol10u8_stage1:/etc/opt/OV:/dev/md/dsk/d32:ufs:1048576
Sol10u8_stage1:/opt/patrol:/dev/md/dsk/d30:ufs:2097152
:wq!

After this change ludete worked like charm!!!!
# ludelete Sol10u8_stage1
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.

Debug tip in case if container is hung while shutdown.

Debug tip in case if container is hung while shutdown.

Sometimes if you have NFS share mounted to NGZ through GZ & if you initiate shutdown to that container then most of the times it hangs & stuck at shutting_down state. If zone hung nfs mount one should be able to see it still mounted in the /etc/mntab file in the global zone: grep nfs /etc/mntab. It will be mounted under the zonepath.  You should then be able to do a umount -f / from the global zone and if you're really lucky the zone will finish shutting down.

Also see if any process is holding zone getting shutdown & try to kill them.  Yet another good command is truss, this will be a helpful while performing debugging. Like when you initiate shutdown then it start some process so you can simply truss on that process ID & see what it is exactly doing & where it is stuck.

If above tip won't work then you can use mdb to debug further -

Sun Container ref_count

# mdb -k
::walk zone | ::print zone_t zone_name zone_ref

The zone_ref > 1 means that something in the kernel is holding the zone.

# mdb -k
::fsinfo

# mdb -k
::kmem_cache | grep rnode
ffffffffa6438008 rnode_cache               0200 000000      656    70506
ffffffffa643c008 rnode4_cache              0200 000000      968        0

Then run -

ffffffffa6438008::walk kmem | ::print rnode_t r_vnode | ::wnode2path

See if this gives any hints for solution. The out put from this command may show you few files/filesystems which may be hold back from the zone & it is causing shutting down zone.

In case if nothing workout then you have to take a call & recycle GZ server.

One important thing I came to know out of this experience that - zsched process is always unkillable.  It will only exit when instructed to by zoneadmd.

Solaris Live Upgrade to Resize Filesystems

Solaris Live Upgrade Resize Filesystems

Pre-work

i. Verify Live Upgrade is installed (SUNWluu, SUNWlur, SUNWlucfg)
    Install any packages if anything is missing.

# mount jumpstart_srv:/jumpstart_dir /mnt
# cd /mnt/OS/Solaris_9U5/Solaris_9/Tools/Installers
# ./liveupgrade20 -noconsole -nodisplay

ii. Verify you have the latest patch sets ref info doc 72099 on SunSolve
iii. Verify your backups!

PS: Verify that all application processes are stopped, you must be sure about this or you risk corrupting application data.
PS: You must be in run-level 3 to use live upgrade.

FS Resize work

i. Shutdown all applications verify they are stopped my unmounting then remounting application filesystems
ii. Disable application startup scripts (check in /etc/rc2.d/ & /etc/rc3.d)
iii. Break mirror system disk mirror, delete spare metadb and re-format the disk in the desired layout

# metastat -p
d0 -m d10 d20 1
d10 1 1 c0t0d0s0
d20 1 1 c0t2d0s0
d3 -m d13 d23 1
d13 1 1 c0t0d0s3
d23 1 1 c0t2d0s3
d1 -m d11 d21 1
d11 1 1 c0t0d0s1
d21 1 1 c0t2d0s1
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 d24 1
d14 1 1 c0t0d0s4
d24 1 1 c0t2d0s4
d31 -p d4 -o 32 -b 2097152

# for i in 0 1 3 4; do
> metadetach d$i d2$i
> metaclear d2$i
> done
d0: submirror d20 is detached
d20: Concat/Stripe is cleared
d1: submirror d21 is detached
d21: Concat/Stripe is cleared
d3: submirror d23 is detached
d23: Concat/Stripe is cleared
d4: submirror d24 is detached
d24: Concat/Stripe is cleared

# metastat -p
d0 -m d10 1
d10 1 1 c0t0d0s0
d3 -m d13 1
d13 1 1 c0t0d0s3
d1 -m d11 1
d11 1 1 c0t0d0s1
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c0t0d0s4
d31 -p d4 -o 32 -b 2097152

# metadb -f -d c0t2d0s7

# metadb -i
flags first blk block count
a m p luo 16 8192 /dev/dsk/c0t0d0s7
a p luo 8208 8192 /dev/dsk/c0t0d0s7
a p luo 16400 8192 /dev/dsk/c0t0d0s7

# format c0t2d0

So after format the partition table will look like

Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 5140 10.00GB (5141/0/0) 20975280
1 swap wu 5141 - 7197 4.00GB (2057/0/0) 8392560
2 backup wm 0 - 19155 37.27GB (19156/0/0) 78156480
3 var wm 7198 - 10282 6.00GB (3085/0/0) 12586800
4 unassigned wm 10283 - 18507 16.00GB (8225/0/0) 33558000
5 unassigned wu 0 0 (0/0/0) 0
6 unassigned wu 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0

Label the disk & come out of format utility.

# for i in 0 3; do
> metainit d2$i 1 1 c0t2d0s$i
> done
d20: Concat/Stripe is setup
d23: Concat/Stripe is setup

Create soft partitions to move soft partitions to, use block size to match size exactly.

# metainit d32 -p d4 1G
d33: Soft Partition is setup
# metainit d33 -p d4 1G
d33: Soft Partition is setup

# newfs /dev/md/rdsk/d32
# newfs /dev/md/rdsk/d33

Re-Write metadevice information to new size on slice 7

# metadb -a -f -c 3 c0t2d0s7

Migrating /tmp to tmpfs

Take server to single user mode.

# init S
# who -r
. run-level 1 Jan 5 05:30 1 0 S
# chmod 1777 /tmp

Edit /etc/vfstab and add the following line
swap - /tmp tmpfs - yes size=4096m

Run: mount -a, verify that /tmp mount

# df -k /tmp
Filesystem kbytes used avail capacity Mounted on
swap 1710144 16 1710128 1% /tmp

Run init 6 to reboot to multiuser mode.

Now see where your primary dump device is.

# dumpadm
Dump content: kernel pages
Dump device: /dev/md/dsk/d1 (dedicated)
Savecore directory: /var/crash/xxxxxx
Savecore enabled: yes

Change the location.

# metainit d21 1 1 c0t2d0s1
d21: Concat/Stripe is setup

# dumpadm -c kernel -d /dev/md/dsk/d21
Dump content: kernel pages
Dump device: /dev/md/dsk/d21 (dedicated)
Savecore directory: /var/crash/xxxxxx
Savecore enabled: yes

# metastat -p
d0 -m d20 1
d20 1 1 c0t2d0s0
d3 -m d23 1
d23 1 1 c0t2d0s3
d1 -m d11 1
d11 1 1 c0t0d0s1
d21 1 1 c0t2d0s1
d33 -p d4 -o 2097216 -b 2097152
d4 -m d24 1
d24 1 1 c0t2d0s4
d32 -p d4 -o 32 -b 2097152

# metaclear -f d1
d1: Mirror is cleared


# metaclear d11
d11: Concat/Stripe is cleared


# metastat -p
d0 -m d20 1
d20 1 1 c0t2d0s0
d3 -m d23 1
d23 1 1 c0t2d0s3
d21 1 1 c0t2d0s1
d33 -p d4 -o 2097216 -b 2097152
d4 -m d24 1
d24 1 1 c0t2d0s4
d32 -p d4 -o 32 -b 2097152

Recreate the metadevice d1 & attach d21 to it.

# metainit -f d1 -m d21
d1: Mirror is setup

# metastat -p
d1 -m d21 1
d21 1 1 c0t2d0s1
d0 -m d20 1
d20 1 1 c0t2d0s0
d3 -m d23 1
d23 1 1 c0t2d0s3
d33 -p d4 -o 2097216 -b 2097152
d4 -m d24 1
d24 1 1 c0t2d0s4
d32 -p d4 -o 32 -b 2097152

Change your dump device back to metadevice d1

# dumpadm -c kernel -d /dev/md/dsk/d1
Dump content: kernel pages
Dump device: /dev/md/dsk/d1 (dedicated)
Savecore directory: /var/crash/slabtst3
Savecore enabled: yes

Use LiveUpgrade to copy the data to the new disk

# lucreate -c Sol9_orig -n Sol9_bigroot -m /:d20:ufs -m /var:d23:ufs -m /opt/patrol:d32:ufs -m /home:d33:ufs
Discovering physical storage devices
Discovering logical storage devices
Cross referencing storage devices with boot environment configurations
Determining types of file systems supported
Validating file system requests
The device name expands to device path
The device name expands to device path
The device name expands to device path
The device name expands to device path
Preparing logical storage devices
Preparing physical storage devices
Configuring physical storage devices
Configuring logical storage devices
Analyzing system configuration.
No name for current boot environment.
Current boot environment is named .
Creating initial configuration for primary boot environment .
WARNING: The device for the root file system mount point is not a physical device.
WARNING: The system boot prom identifies the physical device as the system boot device.
Is the physical device the boot device for the logical device ? (yes or no) yes
INFORMATION: Assuming the boot device obtained from the system boot prom is the physical boot device for logical device .
The device is not a root device for any boot environment.
PBE configuration successful: PBE name PBE Boot Device .
Comparing source boot environment file systems with the file
system(s) you specified for the new boot environment. Determining which
file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Searching /dev for possible boot environment filesystem devices


Updating system configuration files.
The device is not a root device for any boot environment.
Creating configuration for boot environment .
Creating boot environment .
Creating file systems on boot environment .
Creating file system for on .
Creating file system for on .
Creating file system for on .
Creating file system for on .
Mounting file systems for boot environment .
Calculating required sizes of file systems for boot environment .
Populating file systems on boot environment .
Checking selection integrity.
Integrity check OK.
Populating contents of mount point .
Populating contents of mount point .
Populating contents of mount point .
Populating contents of mount point .
Copying.
Creating shared file system mount points.
Creating compare databases for boot environment .
Creating compare database for file system .
Creating compare database for file system .
Creating compare database for file system .
Creating compare database for file system .
Updating compare databases on boot environment .
Making boot environment bootable.
Setting root slice to Solaris Volume Manager metadevice .
Population of boot environment successful.
Creation of boot environment successful.

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol9_orig yes yes yes no -
Sol9_bigroot yes no no yes -

Let’s activate the alternate boot environment.

# luactivate Sol9_bigroot




**********************************************************************


The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.


**********************************************************************


In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:


1. Enter the PROM monitor (ok prompt).


2. Change the boot device back to the original boot environment by typing:


setenv boot-device c0t0d0


3. Boot to the original boot environment by typing:


boot


**********************************************************************


Activation of boot environment successful.

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol9_orig yes yes no no -
Sol9_bigroot yes no yes no -


# init 6

Messages from console –

Configuring IPv4 interfaces: dmfe0.
Hostname: XXXXXX
Configuring /dev and /devices
Configuring the /dev directory (compatibility devices)
The system is coming up. Please wait.
checking ufs filesystems
/dev/md/rdsk/d32: is logging.
/dev/md/rdsk/d33: is logging.
Live Upgrade: Synchronizing new boot environment.
Live Upgrade: Previous boot environment was .
Live Upgrade: Current boot environment is now .
starting rpc services: rpcbind keyserv done.
Setting netmask of dmfe0 to xxx.xxx.xxx.x
Setting default IPv4 interface for multicast: add net 224.0/4: gateway xxxxxx
No such file or directory
syslog service starting.
vasd started
Idle daemon started

Well, now system is up & here you can see that the root filesystem size has been resized.

BEFORE:


# df -kh
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d20 5.0G 4.0G 1.0G 90% /
[…]


AFTER:


# df -kh
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d20 9.8G 2.0G 7.7G 21% /
[…]

Now at least for a week time keep the original BE as it is and let application owner verify the server environments, applications, databases etc. If there were no complains then just simply delete the original BE and re-mirror the disks.

Delete old boot environment

# ludelete Sol9_orig
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.

After deleting the old BE the metastat looks like as follows –

# metastat -p
d0 -m d10 1
d10 1 1 c0t0d0s0
d3 -m d13 1
d13 1 1 c0t0d0s3
d1 -m d11 1
d11 1 1 c0t0d0s1
d24 1 1 c0t2d0s4
d23 1 1 c0t2d0s3
d21 1 1 c0t2d0s1
d20 1 1 c0t2d0s0
d33 -p d4 -o 6291584 -b 2097152
d4 -m d14 1
d14 1 1 c0t0d0s4
d32 -p d4 -o 4194400 -b 2097152
d30 -p d4 -o 2097216 -b 2097152
d31 -p d4 -o 32 -b 2097152

So here we are going to clear d0, d10, d3, d13, d24, d21, d30 & d31 –

# metaclear d0;metaclear d10; metaclear d3; metaclear d13; metaclear d24; metaclear d21; metaclear d30; metaclear d31
d0: Mirror is cleared
d10: Concat/Stripe is cleared
d3: Mirror is cleared
d13: Concat/Stripe is cleared
d24: Concat/Stripe is cleared
d21: Concat/Stripe is cleared
d30: Soft Partition is cleared
d31: Soft Partition is cleared

Create non root metadevices

# for i in 1 3 ; do
> metainit -f d$i -m d2$i
> done
d1: Mirror is setup
d3: Mirror is setup

Update vfstab for non-root filesystems.

/dev/md/dsk/d1 - - swap - no -
/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no logging
/dev/md/dsk/d3 /dev/md/rdsk/d3 /var ufs 1 no logging >>>> here we have made a change. d23 replaced by d3
swap - /tmp tmpfs - yes -

Change root to mirror

# metainit -f d0 -m d20
d0: Mirror is setup


# metaroot d0


# lockfs –fa


# reboot

After reboot re-mirror soft partitions

Clean the metadb on first disk.

# metadb -f -d c0t0d0s7

Reattach the empty disk to existing metadevices so that will have 2nd submirror

# prtvtoc /dev/rdsk/c0t2d0s2 | fmthard -s - /dev/rdsk/c0t0d0s2
fmthard: New volume table of contents now in place.

Setup metadb

# metadb -a -f -c3 c0t0d0s7

# metadb -i
flags first blk block count
a u 16 8192 /dev/dsk/c0t0d0s7
a u 8208 8192 /dev/dsk/c0t0d0s7
a u 16400 8192 /dev/dsk/c0t0d0s7
a m p luo 16 8192 /dev/dsk/c0t2d0s7
a p luo 8208 8192 /dev/dsk/c0t2d0s7
a p luo 16400 8192 /dev/dsk/c0t2d0s7

# metainit -f d10 1 1 c0t0d0s0;metainit -f d11 1 1 c0t0d0s1;metainit -f d13 1 1 c0t0d0s3;
metainit -f d14 1 1 c0t0d0s4
d10: Concat/Stripe is setup
d11: Concat/Stripe is setup
d13: Concat/Stripe is setup
d14: Concat/Stripe is setup

# metattach d0 d10; metattach d1 d11; metattach d3 d13; metattach d4 d14
d0: submirror d10 is attached
d1: submirror d11 is attached
d3: submirror d13 is attached
d4: submirror d14 is attached

# metastat -p
d1 -m d21 d11 1
d21 1 1 c0t2d0s1
d11 1 1 c0t0d0s1
d0 -m d20 d10 1
d20 1 1 c0t2d0s0
d10 1 1 c0t0d0s0
d3 -m d23 d13 1
d23 1 1 c0t2d0s3
d13 1 1 c0t0d0s3
d33 -p d4 -o 2097216 -b 2097152
d4 -m d24 d14 1
d24 1 1 c0t2d0s4
d14 1 1 c0t0d0s4
d32 -p d4 -o 32 -b 2097152

Wait till devices sync up with each other & you’re almost done.

There is a downtime cost associated with this method. This method required downtime of around ~20 Mins (2/3 reboot). This is really time consuming, tedious dirty method and this will teach you how initial planning for accurate capacity planning, proper filesystem layout will help in long run!!!