Find it

Monday, November 30, 2009

Using Live Upgrade utility - migrate Solaris 10 UFS with SVM to ZFS root mirrored configuration.

Using Live Upgrade utility - migrate Solaris 10 UFS with SVM to ZFS root mirrored configuration.


Introduction:

I’m having a server running Solaris 10 [Solaris 10 10/09], upgraded from Solaris 9 using LiveUpgrade. Currently the root filesystem is using UFS and has Solaris Volume Manager implemented with Concat/Stripe mirror configuration.

I'm going to use Live Upgrade utility to upgrade UFS to ZFS. I have 2 internal disks & both disks are currently in use by SVM. So my first task is to make one disk free where I can have my rpool created and creating ZFS boot environment.

My current metadevices are as below -

# metastat -p
d0 -m d20 d10 1
d20 1 1 c1t1d0s0
d10 1 1 c1t0d0s0
d1 -m d11 d21 1
d11 1 1 c1t0d0s1
d21 1 1 c1t1d0s1
d3 -m d23 d13 1
d23 1 1 c1t1d0s3
d13 1 1 c1t0d0s3
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 d24 1
d14 1 1 c1t0d0s4
d24 1 1 c1t1d0s4
d31 -p d4 -o 32 -b 2097152

My disks are - c1t0d0 & c1t1d0.

So, let us de-attach the mirror for all filesystems.

Detach the metadevices from c1t1d0

# metadetach d0 d20
# metadetach d3 d23
# metadetach d1 d21
# metadetach d4 d24


# metastat -p
d0 -m d10 1
d10 1 1 c1t0d0s0
d1 -m d11 1
d11 1 1 c1t0d0s1
d3 -m d13 1
d13 1 1 c1t0d0s3
d24 1 1 c1t1d0s4 >>>>> unused metadevices
d23 1 1 c1t1d0s3 >>>>> unused metadevices
d21 1 1 c1t1d0s1 >>>>> unused metadevices
d20 1 1 c1t1d0s0 >>>>> unused metadevices

d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152

Now clear the metadevices so that we will have free disk for creating root pool.

# for i in 0 1 3 4; do
> metaclear d2$i
> done

d20: Concat/Stripe is cleared
d21: Concat/Stripe is cleared
d23: Concat/Stripe is cleared
d24: Concat/Stripe is cleared

Now remove metadb from c1t1d0

# metadb -f -d /dev/dsk/c1t1d0s7

Alright, now we have free disk now so let us create the new root pool. Before creating root pool make sure you have formatted disk with SMI label and reconfigure your

slices so that s0 consists of the whole disk.

# zpool create rpool c1t1d0s0

We can now create a new boot environment named Solaris10ZFS that is a copy of the current one on our newly created ZFS pool named rpool.

# lucreate -n Solaris10ZFS -p rpool
Analyzing system configuration.
Comparing source boot environment file systems with the file
system(s) you specified for the new boot environment. Determining which
file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Updating system configuration files.
Creating configuration for boot environment .
Source boot environment is .
Creating boot environment .
Creating file systems on boot environment .
Creating file system for in zone on .
Populating file systems on boot environment .
Checking selection integrity.
Integrity check OK.
Populating contents of mount point .
Copying.
Creating shared file system mount points.
Creating compare databases for boot environment .
Creating compare database for file system .
Creating compare database for file system .
Creating compare database for file system .
Updating compare databases on boot environment .
Making boot environment bootable.
Creating boot_archive for /.alt.tmp.b-P0g.mnt
updating /.alt.tmp.b-P0g.mnt/platform/sun4u/boot_archive
Population of boot environment successful.
Creation of boot environment successful.

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10 yes yes yes no -
Solaris10ZFS yes no no yes -

Now let's activate the boot environment created for ZFS root.

# luactivate Solaris10ZFS


**********************************************************************


The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.


**********************************************************************


In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:


1. Enter the PROM monitor (ok prompt).


2. Change the boot device back to the original boot environment by typing:


setenv boot-device c1t0d0


3. Boot to the original boot environment by typing:


boot


**********************************************************************

Modifying boot archive service
Activation of boot environment successful.

Now lustatus will show below output -

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10 yes yes no no -
Solaris10ZFS yes no yes no -

Now let's boot to the ZFS boot environment

# sync;sync;sync;init 6

It's a good idea to have a console connection handy as you can see the console messges/errors while reboot and also in case the server is not responsive then you can have console to troubleshoot it.

Nice! Server came up after reboot and now it is on ZFS root. However I still have old BE available on system and it's a good practice to keep it on system while application/database team give go ahead confirmation. Once everything is fine then go ahead and delete old BE if you wish. Deleting old BE will free up the disk and then you can use that disk to mirror it to existing ZFS root pool.

After reboot you can see lustatus output -

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10 yes no no yes -
Solaris10ZFS yes yes yes no -

Here, you can see Can Delete field is "yes".

Also you can see the ZFS filesystems and zpool details -

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
rpool 33.8G 5.46G 28.3G 16% ONLINE -


# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 9.46G 23.8G 98K /rpool
rpool/ROOT 4.46G 23.8G 21K /rpool/ROOT
rpool/ROOT/Solaris10ZFS 4.46G 23.8G 4.46G /
rpool/dump 1.00G 23.8G 1.00G -
rpool/swap 4.00G 27.8G 16K -


# zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0


errors: No known data errors

Ok. I still have few filesystems like /home, /opt on metadevices so let's create ZFS filesystems and rsync/dump UFS filesystems to ZFS ones.

# zfs create -o mountpoint=/mnt/home -o quota=2g rpool/home
# zfs create -o mountpoint=/mnt/opt quota=1g rpool/opt


What I'm doing here is, for temporary purpose I have set the mount points for rpool/home & rpool/opt to /mnt/home & /mnt/opt because there are directories which are already mounted on metadevices by same name. After copying over the data I'm going to change the mount points.

Okay, now it's time to copy over the data. I'm using rsync for this purpose. One can also use ufsdump utility.

# rsync -axP --delete /home/ /mnt/home/
# rsync -axP /opt/ /mnt/opt/


Cool... now all the data is copied to ZFS filesystem and we are ok to clear up these metadevices.

Ok, So now I will go ahead and delete old BE safely as all data has been copied and verified by the application owner.

# ludelete Sol10 && lustatus
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Solaris10ZFS yes yes yes no -

Now will will clear all the metadevices so that the disk hosting old BE will be marked as a free to use and then we can add it to existing rpool as a mirror disk.

The metadevices stats are as below -

# metastat -p
d0 -m d10 1
d10 1 1 c1t0d0s0
d1 -m d11 1
d11 1 1 c1t0d0s1
d3 -m d13 1
d13 1 1 c1t0d0s3
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152

unmount the metadevices for /home & /opt

#umount /home;umount /opt

Remove or comment out entries from vfstab file and clear them out.

# for i in 0 1 3 4; do metaclear -r d${i}; done
d0: Mirror is cleared
d10: Concat/Stripe is cleared
d1: Mirror is cleared
d11: Concat/Stripe is cleared
d3: Mirror is cleared
d13: Concat/Stripe is cleared

We also have few soft partitions so we are now clearing up them now.

metastat output is as below -

# metastat -p
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152

# metaclear d30
d30: Soft Partition is cleared


# metaclear d31
d31: Soft Partition is cleared


# metaclear d14
d14: Concat/Stripe is cleared


# metaclear d4
d4: Mirror is cleared

Remove the metadb

# metadb -d -f /dev/dsk/c1t0d0s7

Verify you do not left with any SVM metadb using metadb -i command, after executing metadb -i command it should return nothing.

All right, now we have got the free disk.

Our next job is to add free disk to ZFS rpool as a mirror disk, to do so we will first format the disk c0t1d0 and make slice 0 encompass the whole disk. Write out the label, and get back to the prompt. Now, we need to make our single-disk ZPool into a two-way mirror.

# zpool status -v
pool: rpool
state: ONLINE
scrub: scrub completed after 0h4m with 0 errors on Mon Nov 30 00:00:43 2009
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0


errors: No known data errors

Let’s attach the 2nd disk to rpool

# zpool attach rpool c1t1d0s0 c1t0d0s0
Please be sure to invoke installboot(1M) to make 'c1t0d0s0' bootable.

This sets up the mirror, and automatically starts the resilvering (syncing) process. You can monitor its progress by running 'zpool status'.

# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h1m, 23.01% done, 0h4m to go
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0
c1t0d0s0 ONLINE 0 0 0 1.30G resilvered


errors: No known data errors


The final step is to actually make c1t0d0 bootable in case c1t1d0 fails.

# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c1t0d0s0

Don't forget to change the mount points of /home & /opt as currently those are mounted on /mnt mount point.

# zfs set mountpoint=/home rpool/home
# zfs set mountpoint=/opt rpool/opt


After all this clean up & verification work, just reboot the system once, however this is optional!!!

That's it! This is how you can migrate your all servers from UFS to ZFS. For datacenter environments where numbers of servers are in hundreds or thousands you may thing of having this process automated.

I'm also discovering how we can recover crashed system using ZFS root pool & will soon post the procedure on blog.

2 comments:

  1. I found this really helpful.
    Thanks for taking the time to write it up.

    ReplyDelete
  2. Very useful. Thanks a lot

    ReplyDelete