LiveUpgrade problems

I came across with few incidents recently related to LiveUpgrade & I would like to share it to system administrators community through this blog.

Issue #1

Problem -

LiveUpgrade stuck/hang at "Validating file system requests"

# lucreate -c Sol10u7 -n Sol10u8 -m /:/dev/md/dsk/d0:ufs,mirror \
-m /:/dev/md/dsk/d20:detach,attach \
-m /var:/dev/md/dsk/d3:ufs,mirror \
-m /var:/dev/md/dsk/d23:detach,attach

Discovering physical storage devices
Discovering logical storage devices
Cross referencing storage devices with boot environment configurations
Determining types of file systems supported
Validating file system requests

HANG...HANG...HANG...HANG..YOU'RE STUCK...HANG...IT SUCKS...WHAT'S NEXT???

Solution -

Just execute -

# devfsadm -v -C

It will clear any broken device links under /dev or unreferenced device links under /dev directory. After performing this LiveUpgrade work like a piece of cake!

Issue #2

Problem -

After LiveUpgrade/patching Sol10u7 to Sol10u8, I activated the ABE Sol10u8 & booted off from it. After doing so I deleted the one of the container as we no longer needed that container & cleared up the metadevice where the zoneroot was residing. Deleting container, I certainly made changes to ABE Sol10u8 however those were not obliviously reflected to original BE Sol10u7. So now while deleting Sol10u7 LU program finds a mismatch & fails to delete the boot environment.

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10u7 yes no no yes -
Sol10u8 yes yes yes no -

# ludelete Sol10u7
ERROR: mount: /zone1: No such file or directory
ERROR: cannot mount mount point device /.alt.tmp.b-k2c.mnt/zone1 device /zone1
ERROR: failed to mount file system /zone1 on /.alt.tmp.b-k2c.mnt/zone1
ERROR: unmounting partially mounted boot environment file systems
ERROR: cannot mount boot environment by icf file /etc/lu/ICF.2
ERROR: Cannot mount BE .
Unable to delete boot environment.

Solution -

I don't know if it's a Sun supported solution or workaround. I simply edit the /etc/lu/ICF.nn file for the problematic BE, and delete the lines that reference the missing (or moved) filesystems & it worked fine.

# ludelete Sol10u7
Determining the devices to be marked free.
INFORMATION: Unable to determine size or capacity of slice .
ERROR: An error occurred during creation of configuration file.
WARNING: Target BE BE ID <2> unable to verify file systems belonging to BE are not mounted.
WARNING: Unable to determine disk partition configuration information for BE .
WARNING: Unable to determine the devices/datasets to be freed for BE .
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.

# echo $?
0

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10u8 yes yes yes no -

Adding one more tip - 31 Aug 2010

Issue #3


Problem: -

While ludelete operation I got below error -

# ludelete Sol10u8_stage1
ERROR: Read-only file system: cannot create mount point
ERROR: failed to create mount point for file system
ERROR: unmounting partially mounted boot environment file systems
ERROR: cannot mount boot environment by icf file
ERROR: Cannot mount BE .
Unable to delete boot environment.

Solution: -


As a part of solution, I used unofficial method to get rid of the Pesky BE. I modified /etc/lu/ICF.2 a bit and TRUST ME "lufslist" IS YOUR TRUE FRIEND.

# lufslist Sol10u8_stage1 && lufslist s10s_u8wos_08a


boot environment name: Sol10u8_stage1
Filesystem fstype device size Mounted on Mount Options
----------------------- -------- ------------ ------------------- --------------
/dev/md/dsk/d1 swap 34365898752 - -
/dev/md/dsk/d0 ufs 10743250944 / -
/dev/md/dsk/d3 ufs 8596684800 /var -
/dev/md/dsk/d31 ufs 1073741824 /home -
/dev/md/dsk/d30 ufs 1073741824 /opt/patrol logging
/dev/md/dsk/d32 ufs 536870912 /etc/opt/OV logging
/dev/md/dsk/d33 ufs 1073741824 /cims logging
/dev/md/dsk/d34 ufs 4294967296 /oraarch logging


boot environment name: s10s_u8wos_08a
This boot environment is currently active.
This boot environment will be active on next system boot.


Filesystem fstype device size Mounted on Mount Options
----------------------- -------- ------------ ------------------- --------------
/dev/zvol/dsk/rpool/swap swap 34366029824 - -
rpool/ROOT/s10s_u8wos_08a zfs 6566768640 / -
/dev/md/dsk/d33 ufs 1073741824 /cims -
/dev/md/dsk/d31 ufs 1073741824 /home -
rpool zfs 55018053120 /rpool -
zp00/backup zfs 15671161726 /backup -
zp00/opt-OV zfs 1750692636 /opt/OV -
zp00/pw zfs 598801456 /opt/pw -
zp00/oracle zfs 8574417038 /oracle -
/dev/md/dsk/d34 ufs 4294967296 /oraarch -
zp00/orabkup zfs 345610320 /orabkup -
zp00/oemagent zfs 1251898296 /oemagent -
zp00/oradata1 zfs 1231904784 /oradata1 -
zp00/oradata2 zfs 6858200184 /oradata2 -
zp00/oradata3 zfs 2111803452 /oradata3 -
zp00/oradata4 zfs 880723206 /oradata4 -
zp00/patrol_sw zfs 27280 /patrol_sw -
/dev/md/dsk/d32 ufs 536870912 /etc/opt/OV -
/dev/md/dsk/d30 ufs 1073741824 /opt/patrol -
zp00/var-opt-OV zfs 11173308982 /var/opt/OV -


# cat ICF.2
Sol10u8_stage1:-:/dev/md/dsk/d1:swap:67120896
Sol10u8_stage1:/:/dev/md/dsk/d0:ufs:20982912
Sol10u8_stage1:/var:/dev/md/dsk/d3:ufs:16790400
Sol10u8_stage1:/cims:/dev/md/dsk/d33:ufs:2097152
Sol10u8_stage1:/home:/dev/md/dsk/d31:ufs:2097152
Sol10u8_stage1:/rpool:rpool:zfs:0
Sol10u8_stage1:/backup:zp00/backup:zfs:0
Sol10u8_stage1:/opt/OV:zp00/opt-OV:zfs:0
Sol10u8_stage1:/opt/pw:zp00/pw:zfs:0
Sol10u8_stage1:/oracle:zp00/oracle:zfs:0
Sol10u8_stage1:/oraarch:/dev/md/dsk/d34:ufs:8388608
Sol10u8_stage1:/orabkup:zp00/orabkup:zfs:0
Sol10u8_stage1:/oemagent:zp00/oemagent:zfs:0
Sol10u8_stage1:/oradata1:zp00/oradata1:zfs:0
Sol10u8_stage1:/oradata2:zp00/oradata2:zfs:0
Sol10u8_stage1:/oradata3:zp00/oradata3:zfs:0
Sol10u8_stage1:/oradata4:zp00/oradata4:zfs:0
Sol10u8_stage1:/patrol_sw:zp00/patrol_sw:zfs:0
Sol10u8_stage1:/etc/opt/OV:/dev/md/dsk/d32:ufs:1048576
Sol10u8_stage1:/opt/patrol:/dev/md/dsk/d30:ufs:2097152
Sol10u8_stage1:/rpool/ROOT:rpool/ROOT:zfs:0
Sol10u8_stage1:/var/opt/OV:zp00/var-opt-OV:zfs:0

I wonder why ZFS filesystems listed in this file and hence it is trying to mount BR on existing ZFS BE so I just altered file with below entries -

# vi ICF.2
"ICF.2" 22 lines, 1001 characters
Sol10u8_stage1:-:/dev/md/dsk/d1:swap:67120896
Sol10u8_stage1:/:/dev/md/dsk/d0:ufs:20982912
Sol10u8_stage1:/var:/dev/md/dsk/d3:ufs:16790400
Sol10u8_stage1:/cims:/dev/md/dsk/d33:ufs:2097152
Sol10u8_stage1:/home:/dev/md/dsk/d31:ufs:2097152
Sol10u8_stage1:/oraarch:/dev/md/dsk/d34:ufs:8388608
Sol10u8_stage1:/etc/opt/OV:/dev/md/dsk/d32:ufs:1048576
Sol10u8_stage1:/opt/patrol:/dev/md/dsk/d30:ufs:2097152
:wq!

After this change ludete worked like charm!!!!
# ludelete Sol10u8_stage1
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.