Using Live Upgrade utility - migrate Solaris 10 UFS with SVM to ZFS root mirrored configuration.

Using Live Upgrade utility - migrate Solaris 10 UFS with SVM to ZFS root mirrored configuration.

Introduction:

I’m having a server running Solaris 10 [Solaris 10 10/09], upgraded from Solaris 9 using LiveUpgrade. Currently the root filesystem is using UFS and has Solaris Volume Manager implemented with Concat/Stripe mirror configuration.

I'm going to use Live Upgrade utility to upgrade UFS to ZFS. I have 2 internal disks & both disks are currently in use by SVM. So my first task is to make one disk free where I can have my rpool created and creating ZFS boot environment.

My current metadevices are as below -

# metastat -p
d0 -m d20 d10 1
d20 1 1 c1t1d0s0
d10 1 1 c1t0d0s0
d1 -m d11 d21 1
d11 1 1 c1t0d0s1
d21 1 1 c1t1d0s1
d3 -m d23 d13 1
d23 1 1 c1t1d0s3
d13 1 1 c1t0d0s3
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 d24 1
d14 1 1 c1t0d0s4
d24 1 1 c1t1d0s4
d31 -p d4 -o 32 -b 2097152

My disks are - c1t0d0 & c1t1d0.

So, let us de-attach the mirror for all filesystems.

Detach the metadevices from c1t1d0

# metadetach d0 d20
# metadetach d3 d23
# metadetach d1 d21
# metadetach d4 d24

# metastat -p
d0 -m d10 1
d10 1 1 c1t0d0s0
d1 -m d11 1
d11 1 1 c1t0d0s1
d3 -m d13 1
d13 1 1 c1t0d0s3
d24 1 1 c1t1d0s4 >>>>> unused metadevices
d23 1 1 c1t1d0s3 >>>>> unused metadevices
d21 1 1 c1t1d0s1 >>>>> unused metadevices
d20 1 1 c1t1d0s0 >>>>> unused metadevices
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152

Now clear the metadevices so that we will have free disk for creating root pool.

# for i in 0 1 3 4; do
> metaclear d2$i
> done
d20: Concat/Stripe is cleared
d21: Concat/Stripe is cleared
d23: Concat/Stripe is cleared
d24: Concat/Stripe is cleared

Now remove metadb from c1t1d0

# metadb -f -d /dev/dsk/c1t1d0s7

Alright, now we have free disk now so let us create the new root pool. Before creating root pool make sure you have formatted disk with SMI label and reconfigure your

slices so that s0 consists of the whole disk.

# zpool create rpool c1t1d0s0

We can now create a new boot environment named Solaris10ZFS that is a copy of the current one on our newly created ZFS pool named rpool.

# lucreate -n Solaris10ZFS -p rpool
Analyzing system configuration.
Comparing source boot environment file systems with the file
system(s) you specified for the new boot environment. Determining which
file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Updating system configuration files.
Creating configuration for boot environment .
Source boot environment is .
Creating boot environment .
Creating file systems on boot environment .
Creating file system for in zone on .
Populating file systems on boot environment .
Checking selection integrity.
Integrity check OK.
Populating contents of mount point .
Copying.
Creating shared file system mount points.
Creating compare databases for boot environment .
Creating compare database for file system .
Creating compare database for file system .
Creating compare database for file system .
Updating compare databases on boot environment .
Making boot environment bootable.
Creating boot_archive for /.alt.tmp.b-P0g.mnt
updating /.alt.tmp.b-P0g.mnt/platform/sun4u/boot_archive
Population of boot environment successful.
Creation of boot environment successful.

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10 yes yes yes no -
Solaris10ZFS yes no no yes -

Now let's activate the boot environment created for ZFS root.

# luactivate Solaris10ZFS


**********************************************************************


The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.


**********************************************************************


In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:


1. Enter the PROM monitor (ok prompt).


2. Change the boot device back to the original boot environment by typing:


setenv boot-device c1t0d0


3. Boot to the original boot environment by typing:


boot


**********************************************************************

Modifying boot archive service
Activation of boot environment successful.

Now lustatus will show below output -

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10 yes yes no no -
Solaris10ZFS yes no yes no -

Now let's boot to the ZFS boot environment

# sync;sync;sync;init 6

It's a good idea to have a console connection handy as you can see the console messges/errors while reboot and also in case the server is not responsive then you can have console to troubleshoot it.

Nice! Server came up after reboot and now it is on ZFS root. However I still have old BE available on system and it's a good practice to keep it on system while application/database team give go ahead confirmation. Once everything is fine then go ahead and delete old BE if you wish. Deleting old BE will free up the disk and then you can use that disk to mirror it to existing ZFS root pool.

After reboot you can see lustatus output -

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10 yes no no yes -
Solaris10ZFS yes yes yes no -

Here, you can see Can Delete field is "yes".

Also you can see the ZFS filesystems and zpool details -

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
rpool 33.8G 5.46G 28.3G 16% ONLINE -


# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 9.46G 23.8G 98K /rpool
rpool/ROOT 4.46G 23.8G 21K /rpool/ROOT
rpool/ROOT/Solaris10ZFS 4.46G 23.8G 4.46G /
rpool/dump 1.00G 23.8G 1.00G -
rpool/swap 4.00G 27.8G 16K -


# zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0


errors: No known data errors

Ok. I still have few filesystems like /home, /opt on metadevices so let's create ZFS filesystems and rsync/dump UFS filesystems to ZFS ones.

# zfs create -o mountpoint=/mnt/home -o quota=2g rpool/home
# zfs create -o mountpoint=/mnt/opt quota=1g rpool/opt

What I'm doing here is, for temporary purpose I have set the mount points for rpool/home & rpool/opt to /mnt/home & /mnt/opt because there are directories which are already mounted on metadevices by same name. After copying over the data I'm going to change the mount points.

Okay, now it's time to copy over the data. I'm using rsync for this purpose. One can also use ufsdump utility.

# rsync -axP --delete /home/ /mnt/home/
# rsync -axP /opt/ /mnt/opt/

Cool... now all the data is copied to ZFS filesystem and we are ok to clear up these metadevices.

Ok, So now I will go ahead and delete old BE safely as all data has been copied and verified by the application owner.

# ludelete Sol10 && lustatus
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Solaris10ZFS yes yes yes no -

Now will will clear all the metadevices so that the disk hosting old BE will be marked as a free to use and then we can add it to existing rpool as a mirror disk.

The metadevices stats are as below -

# metastat -p
d0 -m d10 1
d10 1 1 c1t0d0s0
d1 -m d11 1
d11 1 1 c1t0d0s1
d3 -m d13 1
d13 1 1 c1t0d0s3
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152

unmount the metadevices for /home & /opt

#umount /home;umount /opt

Remove or comment out entries from vfstab file and clear them out.

# for i in 0 1 3 4; do metaclear -r d${i}; done
d0: Mirror is cleared
d10: Concat/Stripe is cleared
d1: Mirror is cleared
d11: Concat/Stripe is cleared
d3: Mirror is cleared
d13: Concat/Stripe is cleared

We also have few soft partitions so we are now clearing up them now.

metastat output is as below -

# metastat -p
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152

# metaclear d30
d30: Soft Partition is cleared


# metaclear d31
d31: Soft Partition is cleared


# metaclear d14
d14: Concat/Stripe is cleared


# metaclear d4
d4: Mirror is cleared

Remove the metadb

# metadb -d -f /dev/dsk/c1t0d0s7

Verify you do not left with any SVM metadb using metadb -i command, after executing metadb -i command it should return nothing.

All right, now we have got the free disk.

Our next job is to add free disk to ZFS rpool as a mirror disk, to do so we will first format the disk c0t1d0 and make slice 0 encompass the whole disk. Write out the label, and get back to the prompt. Now, we need to make our single-disk ZPool into a two-way mirror.

# zpool status -v
pool: rpool
state: ONLINE
scrub: scrub completed after 0h4m with 0 errors on Mon Nov 30 00:00:43 2009
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0


errors: No known data errors

Let’s attach the 2nd disk to rpool

# zpool attach rpool c1t1d0s0 c1t0d0s0
Please be sure to invoke installboot(1M) to make 'c1t0d0s0' bootable.

This sets up the mirror, and automatically starts the resilvering (syncing) process. You can monitor its progress by running 'zpool status'.

# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h1m, 23.01% done, 0h4m to go
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0
c1t0d0s0 ONLINE 0 0 0 1.30G resilvered


errors: No known data errors


The final step is to actually make c1t0d0 bootable in case c1t1d0 fails.

# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c1t0d0s0

Don't forget to change the mount points of /home & /opt as currently those are mounted on /mnt mount point.

# zfs set mountpoint=/home rpool/home
# zfs set mountpoint=/opt rpool/opt

After all this clean up & verification work, just reboot the system once, however this is optional!!!

That's it! This is how you can migrate your all servers from UFS to ZFS. For datacenter environments where numbers of servers are in hundreds or thousands you may thing of having this process automated.

I'm also discovering how we can recover crashed system using ZFS root pool & will soon post the procedure on blog.

rsync over NFS, issue with permission

We use rsync over NFS to migrate data from one machine to another. We accomplish this by mounting filesystems of the source machine which needs to be migrated to new one, that is target server. On source machine we have entries for filesystems in dfstab file which needs to share & we mount those on target machine like /mnt/nilesh_rjoshi etc etc.. & later we use rsync to copy over them from source to destination server.

# rsync -axP /mnt/nilesh_rjoshi /nilesh_rjoshi

While performing rsync I got errors like -

rsync: send_files failed to open…Permission denied (13)

& at the end of rsync

rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]

The problem was NFS share options was not set properly. we need to use NFS option anon.

share -F nfs -o ro=@xx.xx.xx.xx/xx,anon=0 /nilesh_rjoshi/

what anon=uid do?

If a request comes from an unknown user, use uid as the effective user ID.

Note: Root users (uid 0) are always considered ``unknown'' by the NFS server unless they are included in the root option below.

The default value for this option is 65534. Setting anon to 65535 disables anonymous access.

Also if you put value like -1 then disables anonymous access. So if you set anon to -1 it prevents access to root on clients (unless specified using the root option in /etc/exports or dfstab in case of Solaris) and also prevents access to users not defined on the server.

Thanks for solution to My UNIX Guru Alex!

Hook up the second drive to SVM to have 2nd submirror to existing SVM configuration.

I have SVM filesystems/partitions with 1-way submirror and I need to add 2nd disk to existing SVM configuration and have 2-way submirror. Below procedure talks about it.

This can be scripted however be honest I'm really bad @ scripting... one of the important area I lacking into!!!

Lets start - below are the available internal disks where all OS data is hosted.

Available disks -

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100000c507bd610,0
1. c1t1d0
/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w21000004cf9985c7,0


Current metastat -p output -

# metastat -p
d0 -m d10 1
d10 1 1 c1t0d0s0
d3 -m d13 1
d13 1 1 c1t0d0s3
d1 -m d11 1
d11 1 1 c1t0d0s1
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152

Current metadbs -

# metadb -i
flags first blk block count
a m p luo 16 8192 /dev/dsk/c1t0d0s7
a p luo 8208 8192 /dev/dsk/c1t0d0s7
a p luo 16400 8192 /dev/dsk/c1t0d0s7


Time to get our hands dirty! Let's add 2nd disk as a mirror to SVM

We start with slicing the second drive in the same way as our first drive, the master.

# prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2
fmthard: New volume table of contents now in place.

No need to newfs the second drive slices here, that will automaticaly done by the mirror syncing later.

We can now setup our metadbs on 2nd disk which needs to added.

# metadb -a -f -c3 c1t1d0s7

Now you can see metadbs looks like as follows -

# metadb -i
flags first blk block count
a m p luo 16 8192 /dev/dsk/c1t0d0s7
a p luo 8208 8192 /dev/dsk/c1t0d0s7
a p luo 16400 8192 /dev/dsk/c1t0d0s7
a u 16 8192 /dev/dsk/c1t1d0s7
a u 8208 8192 /dev/dsk/c1t1d0s7
a u 16400 8192 /dev/dsk/c1t1d0s7

PS: I'm not sure why metadb for 2nd disk has flags like this? replica's location not patched in kernel somehow.  Later update!!! Humm.... After sync & reboot metadb's for 2nd disk has flags same like 1st disk...

Now we go to setup the initial metadevices.

# metainit -f d20 1 1 c1t1d0s0
# metainit -f d21 1 1 c1t1d0s1
# metainit -f d23 1 1 c1t1d0s3
# metainit -f d24 1 1 c1t1d0s4

# metastat -p
d0 -m d10 1
d10 1 1 c1t0d0s0
d3 -m d13 1
d13 1 1 c1t0d0s3
d1 -m d11 1
d11 1 1 c1t0d0s1
d24 1 1 c1t1d0s4
d23 1 1 c1t1d0s3
d21 1 1 c1t1d0s1
d20 1 1 c1t1d0s0
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152


Now its time to hook up the second drive so we have actualy mirrored slices.

# metattach d0 d20
# metattach d1 d21
# metattach d3 d23
# metattach d4 d24

Now see the metastat -p out put -

#metastat -p
d0 -m d10 d20 1
d10 1 1 c1t0d0s0
d20 1 1 c1t1d0s0
d3 -m d13 d23 1
d13 1 1 c1t0d0s3
d23 1 1 c1t1d0s3
d1 -m d11 d21 1
d11 1 1 c1t0d0s1
d21 1 1 c1t1d0s1
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 d24 1
d14 1 1 c1t0d0s4
d24 1 1 c1t1d0s4
d31 -p d4 -o 32 -b 2097152

This will take considereble amount of time. Use metastat to check on the progress of the syncing.

All done!

VERITAS Volume Manager frequently used commands

Display disk listings

# vxdisk list

Display volume manager object listings

# vxprint -ht

Display free space in a disk group

# vxdg -g free

List all volume manager tasks currently running on the system

# vxtask list

Add a disk to Volume Manager (devicename = cXtXdX) (prompt driven)

# vxdiskadd

Take a disk offline (first remove the disk from its disk group) (devicename=cXtXdXs2)

#vxdisk offline

Display multipath information

# vxdisk list

Display disk group information

# vxdg list

# vxdg list

REF: http://www.camelrichard.org/topics/VeritasVM/Volume_Manager_CLI_Examples

df: cannot statvfs /mount_point: I/O error

We were doing storage migration for Sun servers today and after migration I experienced below error on one of our Solaris box which was running VERITAS SFS.

df: cannot statvfs /mount_point: I/O error

I guess at the time of swapping the storage fiber cables system or in fact VERITAS Volume Manager has internally executed "vxvol -g datadg stopall" command and this command stops only unmounted volumes and mounted volumes are not stopped.

One can view what's wrong using below command -

# vxprint -g datadg

If you execute df -kh command you will see - df: cannot statvfs /mount_point: I/O error

To recover the volume from this, follow the steps below:

1. Run one of these commands.

a) umount the VxVM filesystem
b) vxvol -g datadg stopall
c) vxdg deport datadg
d) vxdg import datadg
e) vxvol -g datadg startall
f) vxvol -g dg_name start vol (state a volume name)
   vxvol -g dg_name start vol-L01 (state a sub volume name)
g) vxrecover -g datadg -s
h) vxdg list

Just be sure about things going to work mount the FS and again umount it perform fsck and remount it.

2. Unmount the volume.
3. Run fsck against the file system on the volume.
4. Mount the volume.

it should be OK...

NOTE: THIS METHOD IS ALSO APPLICABLE FOR MOVING THE DISKGROUP. FEW STEPS ARE APPLICABLE FOR MOVING DISKGROUP.