UFS to ZFS using LiveUpgrade, all went well but it made me sad when I deleted the snapshots/clones for zonepath

UFS to ZFS using LiveUpgrade, all went well but it made me sad when I deleted the snapshots/clones for zonepath.

Few weeks ago I made a mistake, it was height of stupidity as an experienced and sensible system administrator. Few weeks back while migrating my UFS based system to ZFS I landed up in a situation where it made me upset for a long time. Below is the brief history on problem.

I'm currently working on UFS to ZFS migration on systems having zones which are already migrated to ZFS zonepath/zone root.

In last upgrade/migration, the mistake I made is - I deleted the snapshots/clones made by LiveUpgrade for zone's zonepath/zoneroot.

The story behind is - When we create a ABE which will boot to ZFS LiveUpgrade program creates a snapshot and that snapshot creates a clone of /[zonename]/zonepath as shown below -


[...]
Creating snapshot for zone1/zonepath on mailto:zone1/zonepath@s10s_u8wos_08a].
Creating clone for [zone1/zonepath@s10s_u8wos_08a] on zone1/zonepath-s10s_u8wos_08a.
Creating snapshot for zone2/zonepath on mailto:zone2/zonepath@s10s_u8wos_08a].
Creating clone for [zone2/zonepath@s10s_u8wos_08a] on zone2/zonepath-s10s_u8wos_08a.
[...]
After creating ABE for ZFS we can luactivate it and boot into ZFS.


# luactivate s10s_u8wos_08a

Till this point everything Worked fine.


# init 6
# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10u8 yes no no yes -
s10s_u8wos_08a yes yes yes no -

From above output you can see I'm in ZFS boot environment.

Now the real stupidity stunt starts onwards below portion. Unfortunately and lack of study before task execution, I deleted snapshots/clones using -


# zfs list -H -t snapshot | cut -f 1
zone1/zonepath-Sol10u8@s10s_u8wos_08a
zone2/zonepath-Sol10u8@s10s_u8wos_08a
zone3/zonepath-Sol10u8@s10s_u8wos_08a

# for snapshot in `zfs list -H -t snapshot
cut -f 1`; do zfs destroy -R -f $snapshot;done

From this point onward I lost my zonepaths and could not able to boot my zones. WOW... Horrible Sunday starts here!

After few weeks after I found the answer as described below -

Solution -

Thanks to Alex and Julien for their suggestions and help.

I removed/destroyed the snapshots/clones meant for new ZFS BE

After this point onward tried mounting the UFS BE but due to unclean zonepaths it failed and left nebulous mouintpoints. To clear those mount points you rebooted the server.
This point onward, after analyzing the ICF* files decided to re-created the zonepath using old UFs BE. Command zpool history helped me here.


# zfs clone zone1/zonepath-d100@s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set mountpoint=/zone1/zonepath-s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set zpdata:rbe=s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set zpdata:zn=zone1 zone1/zonepath-s10s_u8wos_08a
# zfs set mountpoint=/zone1/zonepath-s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set canmount=off zone1/zonepath-s10s_u8wos_08a
# zfs set canmount=on zone1/zonepath-s10s_u8wos_08a
# zfs rename zone1/zonepath-s10s_u8wos_08a zone1/zonepath
# zfs set mountpoint =/zone1/zoenpath zone1/zonepath

likewise one by one recreated/recovered zonepaths snapshots/clones for all the zones which were at fault.

This way, zonepath backed again and fall back to UFS BE went soft however the main thing here, was the patching needs to be re-done here as the zonepath were re-created using unpatched UFS BE.

After backout, deleted the ZFS BE & this point onwards I created new ZFS BE and patched it once again.

I hope no one is stupid like me to make such a mistakes but if made then here you go!