Online data from storage luns in ZPOOL

Wishing you & your family very Happy New Year, may this year brings you all the success, good health & wealth.

Today will be writing about online storage (LUNs) migration for ZFS zpool. I'll be replacing the LUNs from old array to new array.

The best thing about the procedure is that during migration all the data which was present in ZFS file systems in ZPOOL nsrpool is still accessible without any issues, no performance lag. No impact to production system and no down time!!!

Each disk in zfs pool has the same size.

One can do this storage migration in two ways online.

1. One can use attach/detach options, zpool attach -f [pool] [device] [new-device] and then once re-silver done zpool detach [pool] [old-device]
2. Or simply one can simply do device replace, zpool replace [pool] [device] [new-device]

I’m going to use the 2nd option because I think this is the easiest way to achieve this LUN migration.

Newly allocated disks are as below,

/dev/rdsk/c6t6006016070312700CEB537F88C48E311d0s2 - 60G format
/dev/rdsk/c6t6006016070312700FA0468009348E311d0s2 - 60G format
/dev/rdsk/c6t600601606B312700FBA40FCCC0BCE211d0s2 - 60G format
/dev/rdsk/c6t600601606B3127002E623AB38B48E311d0s2 - 60G format
/dev/rdsk/c6t600601606B3127000283C9778948E311d0s2 - 60G format
/dev/rdsk/c6t6006016070312700008A87119148E311d0s2 - 60G format

Have formatted the above disks palcing everything on slice 0.

root@XXXXXX:/root# zpool status nsrpool
  pool: nsrpool
 state: ONLINE
 scan: none requested
config:

        NAME                                       STATE     READ WRITE CKSUM
        nsrpool                                  ONLINE       0     0     0
          c6t60060160E6C31D00625335009BF1DB11d0  ONLINE       0     0     0
          c6t60060160E6C31D00C8B0F7B898F1DB11d0  ONLINE       0     0     0
          c6t60060160E6C31D000C0C7E699AF1DB11d0  ONLINE       0     0     0
          c6t60060160E6C31D00AB24671C98F1DB11d0  ONLINE       0     0     0
          c6t60060160E6C31D0060312EA39BF1DB11d0  ONLINE       0     0     0
          c6t60060160E6C31D00CCF4057397F1DB11d0  ONLINE       0     0     0

errors: No known data errors

Let's start the migration -

root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D00625335009BF1DB11d0 c6t6006016070312700CEB537F88C48E311d0
root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D00C8B0F7B898F1DB11d0 c6t6006016070312700FA0468009348E311d0
root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D000C0C7E699AF1DB11d0 c6t600601606B312700FBA40FCCC0BCE211d0
root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D00AB24671C98F1DB11d0 c6t600601606B3127002E623AB38B48E311d0
root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D0060312EA39BF1DB11d0 c6t600601606B3127000283C9778948E311d0
root@XXXXXX:/root# zpool replace nsrpool c6t60060160E6C31D00CCF4057397F1DB11d0 c6t6006016070312700008A87119148E311d0

Now that resilvering started, will have to wait.

root@XXXXXX:/root# zpool status nsrpool
  pool: nsrpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Wed Dec  4 13:04:21 2013
    44.4G scanned out of 220G at 75.1M/s, 0h39m to go
    44.4G scanned out of 220G at 75.1M/s, 0h39m to go
    44.4G resilvered, 20.24% done
config:

        NAME                                         STATE     READ WRITE CKSUM
        nsrpool                                    ONLINE       0     0     0
          replacing-0                              ONLINE       0     0     0
            c6t60060160E6C31D00625335009BF1DB11d0  ONLINE       0     0     0
            c6t6006016070312700CEB537F88C48E311d0  ONLINE       0     0     0  (resilvering)
          replacing-1                              ONLINE       0     0     0
            c6t60060160E6C31D00C8B0F7B898F1DB11d0  ONLINE       0     0     0
            c6t6006016070312700FA0468009348E311d0  ONLINE       0     0     0  (resilvering)
          replacing-2                              ONLINE       0     0     0
            c6t60060160E6C31D000C0C7E699AF1DB11d0  ONLINE       0     0     0
            c6t600601606B312700FBA40FCCC0BCE211d0  ONLINE       0     0     0  (resilvering)
          replacing-3                              ONLINE       0     0     0
            c6t60060160E6C31D00AB24671C98F1DB11d0  ONLINE       0     0     0
            c6t600601606B3127002E623AB38B48E311d0  ONLINE       0     0     0  (resilvering)
          replacing-4                              ONLINE       0     0     0
            c6t60060160E6C31D0060312EA39BF1DB11d0  ONLINE       0     0     0
            c6t600601606B3127000283C9778948E311d0  ONLINE       0     0     0  (resilvering)
          replacing-5                              ONLINE       0     0     0
            c6t60060160E6C31D00CCF4057397F1DB11d0  ONLINE       0     0     0
            c6t6006016070312700008A87119148E311d0  ONLINE       0     0     0  (resilvering)

errors: No known data errors

After a long time, resilvering done and now we see zpool with new disks - so our zpool online migration is done.

root@XXXXXX:/root# zpool status nsrpool
errors: No known data errors
  pool: nsrpool
 state: ONLINE
 scan: resilvered 219G in 2h55m with 0 errors on Wed Dec  4 16:00:07 2013
config:

        NAME                                       STATE     READ WRITE CKSUM
        nsrpool                                  ONLINE       0     0     0
          c6t6006016070312700CEB537F88C48E311d0  ONLINE       0     0     0
          c6t6006016070312700FA0468009348E311d0  ONLINE       0     0     0
          c6t600601606B312700FBA40FCCC0BCE211d0  ONLINE       0     0     0
          c6t600601606B3127002E623AB38B48E311d0  ONLINE       0     0     0
          c6t600601606B3127000283C9778948E311d0  ONLINE       0     0     0
          c6t6006016070312700008A87119148E311d0  ONLINE       0     0     0

It was easy! Hope this helps...

The most annoying error - cannot mount '/home': directory is not empty

Was working on migrating few SVM metadevice based filesystems to ZFS filesystem on Solaris 10 U8 and for the reason I have created ZFS filesystems with temporary mountpoint under /mnt like:

# zfs create –o mountpoint=/mnt/home –o quota=2g rpool/home

After creating this filesystem I’ll sync the data from metadevice based file system to ZFS based filesystem residing under /mnt like:

# rsync -axP --delete /home/ /mnt/home/

Once the data migrated I’ll un-mount the metadevice for /home and rearrange the mount point for ZFS filesystem back to /home using:

# zfs set mountpoint=/home rpool/home

While performing above command execution I get the error –

# zfs mount -a
cannot mount '/home': directory is not empty

This error may occur due to various reasons like one but not the last is auto mount daemon and so on which I’m not aware of… :)

I got the rid of this pesky issue performing “Overlay mount”

# zfs mount -O rpool/home

BTW then what is overlay mount? – Overlay mount allow the file system to be mounted over an existing mount point, making the underlying file system inaccessible. If a mount is attempted on a pre-existing mount point without setting this flag, the mount will fail, producing the error "device busy" or in ZFS terms “directory is not empty”.

This solution seems to be working just fine for me but I cannot guaranty that same will work for you as I said before there are various factors that leads to such issue.

HTH

UFS to ZFS using LiveUpgrade, all went well but it made me sad when I deleted the snapshots/clones for zonepath

UFS to ZFS using LiveUpgrade, all went well but it made me sad when I deleted the snapshots/clones for zonepath.

Few weeks ago I made a mistake, it was height of stupidity as an experienced and sensible system administrator. Few weeks back while migrating my UFS based system to ZFS I landed up in a situation where it made me upset for a long time. Below is the brief history on problem.

I'm currently working on UFS to ZFS migration on systems having zones which are already migrated to ZFS zonepath/zone root.

In last upgrade/migration, the mistake I made is - I deleted the snapshots/clones made by LiveUpgrade for zone's zonepath/zoneroot.

The story behind is - When we create a ABE which will boot to ZFS LiveUpgrade program creates a snapshot and that snapshot creates a clone of /[zonename]/zonepath as shown below -


[...]
Creating snapshot for zone1/zonepath on mailto:zone1/zonepath@s10s_u8wos_08a].
Creating clone for [zone1/zonepath@s10s_u8wos_08a] on zone1/zonepath-s10s_u8wos_08a.
Creating snapshot for zone2/zonepath on mailto:zone2/zonepath@s10s_u8wos_08a].
Creating clone for [zone2/zonepath@s10s_u8wos_08a] on zone2/zonepath-s10s_u8wos_08a.
[...]
After creating ABE for ZFS we can luactivate it and boot into ZFS.


# luactivate s10s_u8wos_08a

Till this point everything Worked fine.


# init 6
# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10u8 yes no no yes -
s10s_u8wos_08a yes yes yes no -

From above output you can see I'm in ZFS boot environment.

Now the real stupidity stunt starts onwards below portion. Unfortunately and lack of study before task execution, I deleted snapshots/clones using -


# zfs list -H -t snapshot | cut -f 1
zone1/zonepath-Sol10u8@s10s_u8wos_08a
zone2/zonepath-Sol10u8@s10s_u8wos_08a
zone3/zonepath-Sol10u8@s10s_u8wos_08a

# for snapshot in `zfs list -H -t snapshot
cut -f 1`; do zfs destroy -R -f $snapshot;done

From this point onward I lost my zonepaths and could not able to boot my zones. WOW... Horrible Sunday starts here!

After few weeks after I found the answer as described below -

Solution -

Thanks to Alex and Julien for their suggestions and help.

I removed/destroyed the snapshots/clones meant for new ZFS BE

After this point onward tried mounting the UFS BE but due to unclean zonepaths it failed and left nebulous mouintpoints. To clear those mount points you rebooted the server.
This point onward, after analyzing the ICF* files decided to re-created the zonepath using old UFs BE. Command zpool history helped me here.


# zfs clone zone1/zonepath-d100@s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set mountpoint=/zone1/zonepath-s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set zpdata:rbe=s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set zpdata:zn=zone1 zone1/zonepath-s10s_u8wos_08a
# zfs set mountpoint=/zone1/zonepath-s10s_u8wos_08a zone1/zonepath-s10s_u8wos_08a
# zfs set canmount=off zone1/zonepath-s10s_u8wos_08a
# zfs set canmount=on zone1/zonepath-s10s_u8wos_08a
# zfs rename zone1/zonepath-s10s_u8wos_08a zone1/zonepath
# zfs set mountpoint =/zone1/zoenpath zone1/zonepath

likewise one by one recreated/recovered zonepaths snapshots/clones for all the zones which were at fault.

This way, zonepath backed again and fall back to UFS BE went soft however the main thing here, was the patching needs to be re-done here as the zonepath were re-created using unpatched UFS BE.

After backout, deleted the ZFS BE & this point onwards I created new ZFS BE and patched it once again.

I hope no one is stupid like me to make such a mistakes but if made then here you go!

ZFS Revisited

Understanding ZFS & ZFS ARC/L2ARC

It's being almost a year that I’m working on ZFS Filesystem Administration and just finished migrating all of our Solaris servers from UFS root to ZFS root, so now OS data and Application data resides on ZFS. The intention to write this document is to have some handy notes about ZFS & it’s cache mechanism which always plays a vital role in terms of system performance!

This is just a revisit the features/techniques of this great filesystem known as ZFS.

Let’s start from assumptions. In last one year I’ve experienced that there were some assumption that I and others were carrying but before we jump into anything assumption queue should get clear so let’s do that.

Some assumptions about ZFS –

• Tuning ZFS is Evil (TRUE)
• ZFS doesn’t require tuning (FALSE)
• ZFS is a memory hog (TRUE)
• ZFS is slow (FLASE)
• ZFS won’t allow corruption (FLASE)

Alright then, now we clear on few of assumptions and now known to the facts! Let’s take a look at few important features of ZFS –

• ZFS is the world’s first 128-bit file system and as such has a huge capacity.
• Capacity wise Single filesystems 512TB+ (theoretical 264 devices * 264 bytes)
• Trillions of files in a single file system (theoretical 248 files per fileset)
• ZFS is Transaction based, copy-on-write filesystem, so no "fsck" is required
• High Internal data redundancy (New RAID level called RAIDz)
• End-to-end checksum of all data/metadata with Strong algorithm (SHA-256) but CPU consuming sometimes! So can be turn off on data and ON on metadata. Checksums are used to validate blocks.
• Online integrity verification and reconstruction
• Snapshots, filesets, compression, encryption facilities and much more!

What is the ZFS filesystem capacity?

264 — Number of snapshots of any file system
248 — Number of entries in any individual directory
16 EB (264 bytes) — Maximum size of a file system
16 EB — Maximum size of a single file
16 EB — Maximum size of any attribute
256 ZB (278 bytes) — Maximum size of any zpool
256 — Number of attributes of a file (actually constrained to 248 for the number of files in a ZFS file system)
264 — Number of devices in any zpool
264 — Number of zpools in a system
264 — Number of file systems in a zpool

How about data redundancy in ZFS?

ZFS supports following RAID configurations


> Stripes (RAID-0)
> Mirroring (RAID-1)
> RAID-Z (Similar to RAID-5)
> RAID-Z2 (Double parity, similar to RAID-6)

Hey, what is it…WOW no FSCK needed? HOW?

Yes, it’s true. ZFS does not need fsck to correct filesystem errors/corruptions as due to COW behavior. ZFS maintains its records as a tree of blocks. Every block is accessible via a single block called the “uber-block”. When you change an existing block, instead of getting overwritten a copy of the data is made and then modified before being written to disk this is Copy on Write (COW). This ensures ZFS never overwrites live data. This guarantees the integrity of the file system as a system crash still leaves the on disk data in a completely consistent state. There is no need for fsck. Ever.

ZFS is self healing data feature?

Yes, provided that –


• If a Bad Block is found ZFS can repair it so long as it has another copy
• RAID-1 - ZFS can “heal” bad data blocks using the mirrored copy
• RAID-Z/Z2 - ZFS can “heal” bad data blocks using parity


Also note that self healing is avail for ZFS metadata but not to actual application data.

ZFS Best Practices –

• Tune recordsize only on fixed records DB files
• Mirror for performance
• 64-bit kernel (allows greater ZFS caches)
• configure swap (don't be scared by low memory)
• Don't slice up devices (confuses I/O scheduler)
• Isolate DB log writer if that is critical (use few devices)
• Separate Root pool (system's identify) and data pools (system's function)
• Keep pool below 80% full (helps COW)
• Usage of snapshots/clones for backup/DR purpose

Okay then, let’s talk about ARC first.

ARC stands for “Adaptive/Adjustable Replacement Cache” - ARC is a very fast cache located in the server’s memory (RAM). The amount of ARC available in a server is usually all of the memory except for 1GB or 3/4th of main memory whichever is greater so simple calculation shows 3GB on your 4GB machine.

Originally, ARC concepts has been first described and invented by two IBM researchers Megiddo and Modha in November 2003 but ZFS ARC is significantly modified version of original ARC design.

The major differences of ZFS ARC is –

• The ZFS ARC is variable in size and can react to the available memory & maybe that’s the reason it is called as “Adjustable Replacement Memory”. It can grow in size when memory is available or it can shrink in size when memory is needed for other processes/jobs.
• The designed proposed by Megiddo and Modha assumes the block size should be same but ZFS ARC works with multiple block sizes.
• Under ZFS ARC you can lock pages in the cache to excuse them from the removal. This prevents the cache to remove pages, that are currently in use. This feature is not in original ARC design.

ZFS ARC stores ZFS data and metadata information from all active storage pools in physical memory (RAM) by default as much as possible, except 1 GB of RAM or 3/4th of main memory BUT I would say this is just a thumb rule or theoretical rule and depending on the environment tuning needs to be done for better system performance. Consider limiting the maximum ARC memory footprint in the following situations:

• When a known amount of memory is always required by an application. Databases often fall into this category.
• On platforms that support dynamic reconfiguration of memory boards, to prevent ZFS from growing the kernel fence in onto all boards.
• A system that requires large memory pages might also benefit from limiting the ZFS cache, which tends to breakdown large pages into base pages.
• Finally, if the system is running another non-ZFS file system, in addition to ZFS, it is advisable to leave some free memory to host that other file system's caches.

The ARC grows and consumes memory on the theory that no need exists to return data to the system while there is still plenty of free memory. When the ARC has grown and near to go exceed memory pressure exists, for example, when a new application starts up, then the ARC releases its hold on memory. Lastly and very important to note that ZFS is not designed to steal memory from applications, ZFS is very innocent!

By default, UFS uses page caching managed virtual memory system however ZFS does not use the page caching except few type of files! ZFS use ARC. There can be only one ARC per system however caching policy can be change per dataset basis.

As I said before, to make sure application/databases has enough dedicated memory available you need to perform tuning or capping on ARC.

# prtconf | grep Mem
Memory size: 98304 Megabytes

# grep zfs /etc/system
set zfs:zfs_arc_min = 1073741824
set zfs:zfs_arc_max = 17179869184

So here, I’ve 96GB total physical memory and ARC has capped at 16G. So around ~17% has been capped for arc_max and 1G for arc_min

The following command gives the current memory size in bytes that is used by ZFS cache:

# kstat zfs::arcstats:size
module: zfs instance: 0
name: arcstats class: misc
size 15325351536

The thumb rules for tuning ARC will be –

• know your future application memory requirements say if it required 20% memory in overall then it makes sense to cap the ARC such that it does not consume more than the remaining 80% of memory.
• Understand and feel your existing applications well. If application known or indeed uses large memory pages then putting cap on ARC will be a bottleneck to that application as limiting the ARC prevents ZFS from breaking up the pages and fragmenting the memory. Limiting the ARC preserves the availability of large pages.
• It’s certainly not easy to tune ARC and need quite a bit deep understanding of applications and their needs though it always helps to have mentioned scripts handy and added to your tools/script depot - arc_summary.pl (By Ben Rockwood) & arcstat.pl (By Neelakanth Nadgir)

Now let’s have a small discussion about L2ARC

L2ARC is a new layer between Disk and the cache (ARC) in main memory for ZFS. It uses dedicated storage devices to hold cached data. The main role of this cache is to boost the performance of random read workloads. The intended L2ARC devices include 10K/15K RPM disks like short-stroked disks, solid state disks (SSD), and other media with substantially faster read latency than disk.

L2ARC, or Level 2 caching in ARC makes it possible to use a disk in addition to RAM for caching reads. This improves read performance, which can otherwise be slow because of the fragmentation caused by the copy-on-write model used by ZFS.

In actual L2ARC cache is supported in Solaris 10 update 6 however as far as my knowledge L2ARC is well supported and functional in Solaris 10 Update 8 and onwards.

The L2ARC attempts to cache data from the ARC before it is evicted so the L2ARC populates its cache by periodically reading data from the tail of the ARC. The data in the tail of the ARC is the data that hasn’t been used for while. I’m still reading through and understanding on how exactly L2ARC works and how it relates to ARC and all that stuffs.

Am I really need L2ARC?

It all depends on the situation, if you have a system where a lot of data is being read frequently and you want better performance, then yes. But if not it still won’t hurt performance, only improve it.

There is lot more things to talk about & to share however you can also find this information from where I found like go through Sun officials books, Solarisinternal.com and some great blogs!

It was a great ZFS year and I’m glad that Sun (now Oracle) folks worked and working hard to make it as a Legend in filesystem arena!

ZFS logbias property

ZFS logbias property


ZFS Synchronous write bias in short logbias property is to provide a hint to ZFS about handling of synchronous requests in particular dataset. If logging is set to "latency" (the default) ZFS will use pool log devices i.e. ZIL (if configured) to handle the requests at low latency/delay. If logging is set to "throughput" then ZFS will not use configured pool log devices kind of a separated ZIL functionality of ZFS. ZFS will instead optimize synchronous operations for global pool throughput and efficient use of resources.

Basically ZFS logbias is proposed especially for managing performance of Oracle database.
Oracle manages two major types of files, the Data Files and the Redo Log files.
Writes to Redo Log files are in the course of all transactions and low latency/delay is a vital requirement. It is critical for good performance to give high priority to these write requests.
Data Files are also the subject of writes from DB writers as a form of scrubbing dirty blocks to insure DB buffer availability. Write latency/delay is much less an issue. Of more importance is achieving an acceptable level of throughput. These writes are less critical to delivered performance.

Both types of writes are synchronous (using O_DSYNC), and thus treated equally. They compete for the same resources: separate intent log, memory, and normal pool IO. The Data File writes slow down the overall potential performance of the critical Redo Log writers.

So in short - Redo Logs need short latencies/delays, so you would use zfs set logbias=latency oradbsrv1-redo/redologs. The data files of oracle need throughput, so you would configure zfs set logbias=throughput oradbsrv1-data/datafile.

If logbias is set to 'latency' (the default) then there is no change from the current implementation. If the logbias property is set to 'throughput' then intent log blocks will be allocated from the main pool instead of any separate intent log devices (if present). Also data will be written immediately to spread the write load thus making for quicker subsequent transaction group commits to the pool. This is the important change of setting logbias to throughput property.

For Oracle datafiles specifically, using the new setting of Synchronous write bias: Throughput has potential to deliver more stable performance in general and higher performance for redo log sensitive workloads.

REF: 

http://arc.opensolaris.org/caselog/PSARC/2009/423/20090807_neil.perrin

http://www.c0t0d0s0.org/archives/5838-PSARC2009423-ZFS-logbias-property.html

Using Live Upgrade utility - migrate Solaris 10 UFS with SVM to ZFS root mirrored configuration.

Using Live Upgrade utility - migrate Solaris 10 UFS with SVM to ZFS root mirrored configuration.

Introduction:

I’m having a server running Solaris 10 [Solaris 10 10/09], upgraded from Solaris 9 using LiveUpgrade. Currently the root filesystem is using UFS and has Solaris Volume Manager implemented with Concat/Stripe mirror configuration.

I'm going to use Live Upgrade utility to upgrade UFS to ZFS. I have 2 internal disks & both disks are currently in use by SVM. So my first task is to make one disk free where I can have my rpool created and creating ZFS boot environment.

My current metadevices are as below -

# metastat -p
d0 -m d20 d10 1
d20 1 1 c1t1d0s0
d10 1 1 c1t0d0s0
d1 -m d11 d21 1
d11 1 1 c1t0d0s1
d21 1 1 c1t1d0s1
d3 -m d23 d13 1
d23 1 1 c1t1d0s3
d13 1 1 c1t0d0s3
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 d24 1
d14 1 1 c1t0d0s4
d24 1 1 c1t1d0s4
d31 -p d4 -o 32 -b 2097152

My disks are - c1t0d0 & c1t1d0.

So, let us de-attach the mirror for all filesystems.

Detach the metadevices from c1t1d0

# metadetach d0 d20
# metadetach d3 d23
# metadetach d1 d21
# metadetach d4 d24

# metastat -p
d0 -m d10 1
d10 1 1 c1t0d0s0
d1 -m d11 1
d11 1 1 c1t0d0s1
d3 -m d13 1
d13 1 1 c1t0d0s3
d24 1 1 c1t1d0s4 >>>>> unused metadevices
d23 1 1 c1t1d0s3 >>>>> unused metadevices
d21 1 1 c1t1d0s1 >>>>> unused metadevices
d20 1 1 c1t1d0s0 >>>>> unused metadevices
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152

Now clear the metadevices so that we will have free disk for creating root pool.

# for i in 0 1 3 4; do
> metaclear d2$i
> done
d20: Concat/Stripe is cleared
d21: Concat/Stripe is cleared
d23: Concat/Stripe is cleared
d24: Concat/Stripe is cleared

Now remove metadb from c1t1d0

# metadb -f -d /dev/dsk/c1t1d0s7

Alright, now we have free disk now so let us create the new root pool. Before creating root pool make sure you have formatted disk with SMI label and reconfigure your

slices so that s0 consists of the whole disk.

# zpool create rpool c1t1d0s0

We can now create a new boot environment named Solaris10ZFS that is a copy of the current one on our newly created ZFS pool named rpool.

# lucreate -n Solaris10ZFS -p rpool
Analyzing system configuration.
Comparing source boot environment file systems with the file
system(s) you specified for the new boot environment. Determining which
file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Updating system configuration files.
Creating configuration for boot environment .
Source boot environment is .
Creating boot environment .
Creating file systems on boot environment .
Creating file system for in zone on .
Populating file systems on boot environment .
Checking selection integrity.
Integrity check OK.
Populating contents of mount point .
Copying.
Creating shared file system mount points.
Creating compare databases for boot environment .
Creating compare database for file system .
Creating compare database for file system .
Creating compare database for file system .
Updating compare databases on boot environment .
Making boot environment bootable.
Creating boot_archive for /.alt.tmp.b-P0g.mnt
updating /.alt.tmp.b-P0g.mnt/platform/sun4u/boot_archive
Population of boot environment successful.
Creation of boot environment successful.

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10 yes yes yes no -
Solaris10ZFS yes no no yes -

Now let's activate the boot environment created for ZFS root.

# luactivate Solaris10ZFS


**********************************************************************


The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.


**********************************************************************


In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:


1. Enter the PROM monitor (ok prompt).


2. Change the boot device back to the original boot environment by typing:


setenv boot-device c1t0d0


3. Boot to the original boot environment by typing:


boot


**********************************************************************

Modifying boot archive service
Activation of boot environment successful.

Now lustatus will show below output -

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10 yes yes no no -
Solaris10ZFS yes no yes no -

Now let's boot to the ZFS boot environment

# sync;sync;sync;init 6

It's a good idea to have a console connection handy as you can see the console messges/errors while reboot and also in case the server is not responsive then you can have console to troubleshoot it.

Nice! Server came up after reboot and now it is on ZFS root. However I still have old BE available on system and it's a good practice to keep it on system while application/database team give go ahead confirmation. Once everything is fine then go ahead and delete old BE if you wish. Deleting old BE will free up the disk and then you can use that disk to mirror it to existing ZFS root pool.

After reboot you can see lustatus output -

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol10 yes no no yes -
Solaris10ZFS yes yes yes no -

Here, you can see Can Delete field is "yes".

Also you can see the ZFS filesystems and zpool details -

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
rpool 33.8G 5.46G 28.3G 16% ONLINE -


# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 9.46G 23.8G 98K /rpool
rpool/ROOT 4.46G 23.8G 21K /rpool/ROOT
rpool/ROOT/Solaris10ZFS 4.46G 23.8G 4.46G /
rpool/dump 1.00G 23.8G 1.00G -
rpool/swap 4.00G 27.8G 16K -


# zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0


errors: No known data errors

Ok. I still have few filesystems like /home, /opt on metadevices so let's create ZFS filesystems and rsync/dump UFS filesystems to ZFS ones.

# zfs create -o mountpoint=/mnt/home -o quota=2g rpool/home
# zfs create -o mountpoint=/mnt/opt quota=1g rpool/opt

What I'm doing here is, for temporary purpose I have set the mount points for rpool/home & rpool/opt to /mnt/home & /mnt/opt because there are directories which are already mounted on metadevices by same name. After copying over the data I'm going to change the mount points.

Okay, now it's time to copy over the data. I'm using rsync for this purpose. One can also use ufsdump utility.

# rsync -axP --delete /home/ /mnt/home/
# rsync -axP /opt/ /mnt/opt/

Cool... now all the data is copied to ZFS filesystem and we are ok to clear up these metadevices.

Ok, So now I will go ahead and delete old BE safely as all data has been copied and verified by the application owner.

# ludelete Sol10 && lustatus
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Solaris10ZFS yes yes yes no -

Now will will clear all the metadevices so that the disk hosting old BE will be marked as a free to use and then we can add it to existing rpool as a mirror disk.

The metadevices stats are as below -

# metastat -p
d0 -m d10 1
d10 1 1 c1t0d0s0
d1 -m d11 1
d11 1 1 c1t0d0s1
d3 -m d13 1
d13 1 1 c1t0d0s3
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152

unmount the metadevices for /home & /opt

#umount /home;umount /opt

Remove or comment out entries from vfstab file and clear them out.

# for i in 0 1 3 4; do metaclear -r d${i}; done
d0: Mirror is cleared
d10: Concat/Stripe is cleared
d1: Mirror is cleared
d11: Concat/Stripe is cleared
d3: Mirror is cleared
d13: Concat/Stripe is cleared

We also have few soft partitions so we are now clearing up them now.

metastat output is as below -

# metastat -p
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c1t0d0s4
d31 -p d4 -o 32 -b 2097152

# metaclear d30
d30: Soft Partition is cleared


# metaclear d31
d31: Soft Partition is cleared


# metaclear d14
d14: Concat/Stripe is cleared


# metaclear d4
d4: Mirror is cleared

Remove the metadb

# metadb -d -f /dev/dsk/c1t0d0s7

Verify you do not left with any SVM metadb using metadb -i command, after executing metadb -i command it should return nothing.

All right, now we have got the free disk.

Our next job is to add free disk to ZFS rpool as a mirror disk, to do so we will first format the disk c0t1d0 and make slice 0 encompass the whole disk. Write out the label, and get back to the prompt. Now, we need to make our single-disk ZPool into a two-way mirror.

# zpool status -v
pool: rpool
state: ONLINE
scrub: scrub completed after 0h4m with 0 errors on Mon Nov 30 00:00:43 2009
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0


errors: No known data errors

Let’s attach the 2nd disk to rpool

# zpool attach rpool c1t1d0s0 c1t0d0s0
Please be sure to invoke installboot(1M) to make 'c1t0d0s0' bootable.

This sets up the mirror, and automatically starts the resilvering (syncing) process. You can monitor its progress by running 'zpool status'.

# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h1m, 23.01% done, 0h4m to go
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0
c1t0d0s0 ONLINE 0 0 0 1.30G resilvered


errors: No known data errors


The final step is to actually make c1t0d0 bootable in case c1t1d0 fails.

# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c1t0d0s0

Don't forget to change the mount points of /home & /opt as currently those are mounted on /mnt mount point.

# zfs set mountpoint=/home rpool/home
# zfs set mountpoint=/opt rpool/opt

After all this clean up & verification work, just reboot the system once, however this is optional!!!

That's it! This is how you can migrate your all servers from UFS to ZFS. For datacenter environments where numbers of servers are in hundreds or thousands you may thing of having this process automated.

I'm also discovering how we can recover crashed system using ZFS root pool & will soon post the procedure on blog.

Replacing/Relabeling the Root Pool Disk

Unfortunately one of disk associated to rootpool went crazy and fun started for a day there!!! Let's fix it. BTW now a days I am testing a small part of rpool recovery after system crash... This experience is from rpool recovery and testing project.

One can see belwo shown error if something wrong with the root pool disk -

# zpool status -x
pool: rpool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: none requested
config:


NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror DEGRADED 0 0 0
c0t0d0s0 UNAVAIL 0 0 0 cannot open
c0t2d0s0 ONLINE 0 0 0


errors: No known data errors

Have bad disk replaced with good one or brand new disk.

# zpool offline rpool c0t0d0s0
# cfgadm -c unconfigure c1::dsk/c0t0d0

Physically replace the primary disk. In my case it is c0t0d0s0.

Now I am labeling/relabeling the disk. Label rpool disk always with SMI label format and not EFI.

# format -e
Searching for disks...done




AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@1f,0/ide@d/dad@0,0
1. c0t2d0
/pci@1f,0/ide@d/dad@2,0
Specify disk (enter its number): 0
selecting c0t0d0
[disk formatted, no defect list found]




FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
show - translate a disk address
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
volname - set 8-character volume name
! - execute , then return
quit
format> l
[0] SMI Label
[1] EFI Label
Specify Label type[0]: 0
Ready to label disk, continue? yes


format>

Reconfigure the disk and bring it online, if required.

# cfgadm -c configure c1::dsk/c0t0d0
# zpool online rpool c0t0d0

Before de-attach you can replace it however I dont see much need of this step in my case it failed so I am directly De-attach the bad disk from the pool

# zpool detach rpool c0t0d0s0

See the status

# zpool status -v
pool: rpool
state: ONLINE
scrub: none requested
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c0t2d0s0 ONLINE 0 0 0


errors: No known data errors

Now we have newly added disk to the system and we need to add/attach this to the pool.

# zpool attach rpool c0t2d0s0 c0t0d0s0
Please be sure to invoke installboot(1M) to make 'c0t0d0s0' bootable.

Check the resilvering status of the newly attached disk.

# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h2m, 25.39% done, 0h6m to go
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t2d0s0 ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0 1.62G resilvered


errors: No known data errors

After the disk resilvering is complete, install the boot blocks.

# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t0d0s0

Confirm that you can boot from the replacement disk.

REF: http://www.solarisinternals.com/

Renaming zpool.

Few days back I came to know on "How to rename zpool" One of my UNIX Guru Alex educated me on this.

Alex I would like to say million of thanks to you for your valuable guidance.

All right, so here I am creating a zpool named “oracle-db-zp00”

# zpool create oracle-db-zp00 c8t60050768018A8023B800000000000013d0

# zpool status -v
pool: oracle-db-zp00
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
oracle-db-zp00 ONLINE 0 0 0
c8t60050768018A8023B800000000000013d0 ONLINE 0 0 0

errors: No known data errors

Here I messed up with zpool name; it should be “Oracle-Apps-zp00” as per standards. To fix this mistake, I first exported the pool

# zpool export oracle-db-zp00

What zpool export does?

Export a pool from the system for importing on another system. Now at this step “zpool status –v” should not give any O/P for “oracle-db-zp00” zpool.

Now I am importing it with correct name –

# zpool import oracle-db-zp00 Oracle-Apps-zp00

# zpool status -v
pool: Oracle-Apps-zp00
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
Oracle-Apps-zp00 ONLINE 0 0 0
c8t60050768018A8023B800000000000013d0 ONLINE 0 0 0

errors: No known data errors

Cool... Isn’t it easy?

ZFS and Swap space

Hi there...

Few days back I came across a situation where, I have been told to create the swap device on ZFS filesystem. Basically, being new to ZFS - I was not at all knowing about "how-to do this" hence after some research on internet about it found some good stuffs hence thought of sharing same for you all -

NOTE - To set up a swap area, create a ZFS volume of a specific size and then enable swap on that device. Do not swap to a file on a ZFS file system. A ZFS swap file configuration is not supported.

ZFS volumes are identified as devices in the /dev/zvol/{dsk,rdsk}/path directory.

E.g. - /dev/zvol/dsk/swap-zfs

So here I wanted to add 32 GB of swap space -

# zfs create -V 32g -b 8K zfsnode1-zpool/swap
# zfs set refreservation=32g zfsnode1-zpool/swap
# swap -a /dev/zvol/dsk/zfsnode1-zpool/swap
# swap -l

>>> /etc/vfstab <<<

/dev/zvol/dsk/zfsnode1-zpool/swap    -       -       swap    -       no      -

NOTE: Setting the reservation is important, particularly if you plan on making the change permanent, eg by adding the new zvol as a swap entry in /etc/vfstab. ZFS does not reserve the space for swapping otherwise, so the swap system may think there is space which isn't actually there if you don't do this.

NOTE: The -b option sets the volblocksize to improve swap performance by aligning the volume I/O units on disk to the size of the host architecture memory page size (4 KB on x86 systems and 8KB on SPARC, as reported by the pagesize command.)

It is also possible to grow the existing swap volume. To do so, set a new size and refreservation for the existing volume like this:

# swap -d /dev/zvol/dsk/rpool/swap
# zfs set volsize=2g rpool/swap
# zfs set refreservation=2g rpool/swap
# swap -a /dev/zvol/dsk/rpool/swap

That's it! Hope this helps...

ZFS - most frequently used commands.

ZFS - most frequently used commands - below are the commands that we use on daily basis for ZFS filesystem Administration.

# Some reminders on command syntax
root@server:# zpool create oradb-1 c0d0s0
root@server:# zpool create oradb-1 mirror c0d0s3 c1d0s0
root@server:# zfs create oradb-1/oracle-10g
root@server:# zfs set mountpoint=/u01/oracle-10g oradb-1/oracle-10g
root@server:# zfs create oradb-1/home
root@server:# zfs set mountpoint=/export/home oradb-1/home
root@server:# zfs create oradb-1/home/oracle
root@server:# zfs set compression=on oradb-1/home
root@server:# zfs set quota=1g oradb-1/home/oracle
root@server:# zfs set reservation=2g oradb-1/home/oracle
root@server:# zfs set sharenfs=rw oradb-1/home

# to set the filesystem block size to 16k
root@server:# zfs set recordsize=16k oradb-1/home

zpool list, zpool status, zfs list are also useful command in case of FS monitoring.

I have lots of things to add to this however as and when I will get hands on I will modify or add to my blog.

Thanks!