Solaris Live Upgrade to Resize Filesystems

Solaris Live Upgrade Resize Filesystems

Pre-work

i. Verify Live Upgrade is installed (SUNWluu, SUNWlur, SUNWlucfg)
    Install any packages if anything is missing.

# mount jumpstart_srv:/jumpstart_dir /mnt
# cd /mnt/OS/Solaris_9U5/Solaris_9/Tools/Installers
# ./liveupgrade20 -noconsole -nodisplay

ii. Verify you have the latest patch sets ref info doc 72099 on SunSolve
iii. Verify your backups!

PS: Verify that all application processes are stopped, you must be sure about this or you risk corrupting application data.
PS: You must be in run-level 3 to use live upgrade.

FS Resize work

i. Shutdown all applications verify they are stopped my unmounting then remounting application filesystems
ii. Disable application startup scripts (check in /etc/rc2.d/ & /etc/rc3.d)
iii. Break mirror system disk mirror, delete spare metadb and re-format the disk in the desired layout

# metastat -p
d0 -m d10 d20 1
d10 1 1 c0t0d0s0
d20 1 1 c0t2d0s0
d3 -m d13 d23 1
d13 1 1 c0t0d0s3
d23 1 1 c0t2d0s3
d1 -m d11 d21 1
d11 1 1 c0t0d0s1
d21 1 1 c0t2d0s1
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 d24 1
d14 1 1 c0t0d0s4
d24 1 1 c0t2d0s4
d31 -p d4 -o 32 -b 2097152

# for i in 0 1 3 4; do
> metadetach d$i d2$i
> metaclear d2$i
> done
d0: submirror d20 is detached
d20: Concat/Stripe is cleared
d1: submirror d21 is detached
d21: Concat/Stripe is cleared
d3: submirror d23 is detached
d23: Concat/Stripe is cleared
d4: submirror d24 is detached
d24: Concat/Stripe is cleared

# metastat -p
d0 -m d10 1
d10 1 1 c0t0d0s0
d3 -m d13 1
d13 1 1 c0t0d0s3
d1 -m d11 1
d11 1 1 c0t0d0s1
d30 -p d4 -o 2097216 -b 2097152
d4 -m d14 1
d14 1 1 c0t0d0s4
d31 -p d4 -o 32 -b 2097152

# metadb -f -d c0t2d0s7

# metadb -i
flags first blk block count
a m p luo 16 8192 /dev/dsk/c0t0d0s7
a p luo 8208 8192 /dev/dsk/c0t0d0s7
a p luo 16400 8192 /dev/dsk/c0t0d0s7

# format c0t2d0

So after format the partition table will look like

Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 5140 10.00GB (5141/0/0) 20975280
1 swap wu 5141 - 7197 4.00GB (2057/0/0) 8392560
2 backup wm 0 - 19155 37.27GB (19156/0/0) 78156480
3 var wm 7198 - 10282 6.00GB (3085/0/0) 12586800
4 unassigned wm 10283 - 18507 16.00GB (8225/0/0) 33558000
5 unassigned wu 0 0 (0/0/0) 0
6 unassigned wu 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0

Label the disk & come out of format utility.

# for i in 0 3; do
> metainit d2$i 1 1 c0t2d0s$i
> done
d20: Concat/Stripe is setup
d23: Concat/Stripe is setup

Create soft partitions to move soft partitions to, use block size to match size exactly.

# metainit d32 -p d4 1G
d33: Soft Partition is setup
# metainit d33 -p d4 1G
d33: Soft Partition is setup

# newfs /dev/md/rdsk/d32
# newfs /dev/md/rdsk/d33

Re-Write metadevice information to new size on slice 7

# metadb -a -f -c 3 c0t2d0s7

Migrating /tmp to tmpfs

Take server to single user mode.

# init S
# who -r
. run-level 1 Jan 5 05:30 1 0 S
# chmod 1777 /tmp

Edit /etc/vfstab and add the following line
swap - /tmp tmpfs - yes size=4096m

Run: mount -a, verify that /tmp mount

# df -k /tmp
Filesystem kbytes used avail capacity Mounted on
swap 1710144 16 1710128 1% /tmp

Run init 6 to reboot to multiuser mode.

Now see where your primary dump device is.

# dumpadm
Dump content: kernel pages
Dump device: /dev/md/dsk/d1 (dedicated)
Savecore directory: /var/crash/xxxxxx
Savecore enabled: yes

Change the location.

# metainit d21 1 1 c0t2d0s1
d21: Concat/Stripe is setup

# dumpadm -c kernel -d /dev/md/dsk/d21
Dump content: kernel pages
Dump device: /dev/md/dsk/d21 (dedicated)
Savecore directory: /var/crash/xxxxxx
Savecore enabled: yes

# metastat -p
d0 -m d20 1
d20 1 1 c0t2d0s0
d3 -m d23 1
d23 1 1 c0t2d0s3
d1 -m d11 1
d11 1 1 c0t0d0s1
d21 1 1 c0t2d0s1
d33 -p d4 -o 2097216 -b 2097152
d4 -m d24 1
d24 1 1 c0t2d0s4
d32 -p d4 -o 32 -b 2097152

# metaclear -f d1
d1: Mirror is cleared


# metaclear d11
d11: Concat/Stripe is cleared


# metastat -p
d0 -m d20 1
d20 1 1 c0t2d0s0
d3 -m d23 1
d23 1 1 c0t2d0s3
d21 1 1 c0t2d0s1
d33 -p d4 -o 2097216 -b 2097152
d4 -m d24 1
d24 1 1 c0t2d0s4
d32 -p d4 -o 32 -b 2097152

Recreate the metadevice d1 & attach d21 to it.

# metainit -f d1 -m d21
d1: Mirror is setup

# metastat -p
d1 -m d21 1
d21 1 1 c0t2d0s1
d0 -m d20 1
d20 1 1 c0t2d0s0
d3 -m d23 1
d23 1 1 c0t2d0s3
d33 -p d4 -o 2097216 -b 2097152
d4 -m d24 1
d24 1 1 c0t2d0s4
d32 -p d4 -o 32 -b 2097152

Change your dump device back to metadevice d1

# dumpadm -c kernel -d /dev/md/dsk/d1
Dump content: kernel pages
Dump device: /dev/md/dsk/d1 (dedicated)
Savecore directory: /var/crash/slabtst3
Savecore enabled: yes

Use LiveUpgrade to copy the data to the new disk

# lucreate -c Sol9_orig -n Sol9_bigroot -m /:d20:ufs -m /var:d23:ufs -m /opt/patrol:d32:ufs -m /home:d33:ufs
Discovering physical storage devices
Discovering logical storage devices
Cross referencing storage devices with boot environment configurations
Determining types of file systems supported
Validating file system requests
The device name expands to device path
The device name expands to device path
The device name expands to device path
The device name expands to device path
Preparing logical storage devices
Preparing physical storage devices
Configuring physical storage devices
Configuring logical storage devices
Analyzing system configuration.
No name for current boot environment.
Current boot environment is named .
Creating initial configuration for primary boot environment .
WARNING: The device for the root file system mount point is not a physical device.
WARNING: The system boot prom identifies the physical device as the system boot device.
Is the physical device the boot device for the logical device ? (yes or no) yes
INFORMATION: Assuming the boot device obtained from the system boot prom is the physical boot device for logical device .
The device is not a root device for any boot environment.
PBE configuration successful: PBE name PBE Boot Device .
Comparing source boot environment file systems with the file
system(s) you specified for the new boot environment. Determining which
file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Searching /dev for possible boot environment filesystem devices


Updating system configuration files.
The device is not a root device for any boot environment.
Creating configuration for boot environment .
Creating boot environment .
Creating file systems on boot environment .
Creating file system for on .
Creating file system for on .
Creating file system for on .
Creating file system for on .
Mounting file systems for boot environment .
Calculating required sizes of file systems for boot environment .
Populating file systems on boot environment .
Checking selection integrity.
Integrity check OK.
Populating contents of mount point .
Populating contents of mount point .
Populating contents of mount point .
Populating contents of mount point .
Copying.
Creating shared file system mount points.
Creating compare databases for boot environment .
Creating compare database for file system .
Creating compare database for file system .
Creating compare database for file system .
Creating compare database for file system .
Updating compare databases on boot environment .
Making boot environment bootable.
Setting root slice to Solaris Volume Manager metadevice .
Population of boot environment successful.
Creation of boot environment successful.

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol9_orig yes yes yes no -
Sol9_bigroot yes no no yes -

Let’s activate the alternate boot environment.

# luactivate Sol9_bigroot




**********************************************************************


The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.


**********************************************************************


In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:


1. Enter the PROM monitor (ok prompt).


2. Change the boot device back to the original boot environment by typing:


setenv boot-device c0t0d0


3. Boot to the original boot environment by typing:


boot


**********************************************************************


Activation of boot environment successful.

# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
Sol9_orig yes yes no no -
Sol9_bigroot yes no yes no -


# init 6

Messages from console –

Configuring IPv4 interfaces: dmfe0.
Hostname: XXXXXX
Configuring /dev and /devices
Configuring the /dev directory (compatibility devices)
The system is coming up. Please wait.
checking ufs filesystems
/dev/md/rdsk/d32: is logging.
/dev/md/rdsk/d33: is logging.
Live Upgrade: Synchronizing new boot environment.
Live Upgrade: Previous boot environment was .
Live Upgrade: Current boot environment is now .
starting rpc services: rpcbind keyserv done.
Setting netmask of dmfe0 to xxx.xxx.xxx.x
Setting default IPv4 interface for multicast: add net 224.0/4: gateway xxxxxx
No such file or directory
syslog service starting.
vasd started
Idle daemon started

Well, now system is up & here you can see that the root filesystem size has been resized.

BEFORE:


# df -kh
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d20 5.0G 4.0G 1.0G 90% /
[…]


AFTER:


# df -kh
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d20 9.8G 2.0G 7.7G 21% /
[…]

Now at least for a week time keep the original BE as it is and let application owner verify the server environments, applications, databases etc. If there were no complains then just simply delete the original BE and re-mirror the disks.

Delete old boot environment

# ludelete Sol9_orig
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment deleted.

After deleting the old BE the metastat looks like as follows –

# metastat -p
d0 -m d10 1
d10 1 1 c0t0d0s0
d3 -m d13 1
d13 1 1 c0t0d0s3
d1 -m d11 1
d11 1 1 c0t0d0s1
d24 1 1 c0t2d0s4
d23 1 1 c0t2d0s3
d21 1 1 c0t2d0s1
d20 1 1 c0t2d0s0
d33 -p d4 -o 6291584 -b 2097152
d4 -m d14 1
d14 1 1 c0t0d0s4
d32 -p d4 -o 4194400 -b 2097152
d30 -p d4 -o 2097216 -b 2097152
d31 -p d4 -o 32 -b 2097152

So here we are going to clear d0, d10, d3, d13, d24, d21, d30 & d31 –

# metaclear d0;metaclear d10; metaclear d3; metaclear d13; metaclear d24; metaclear d21; metaclear d30; metaclear d31
d0: Mirror is cleared
d10: Concat/Stripe is cleared
d3: Mirror is cleared
d13: Concat/Stripe is cleared
d24: Concat/Stripe is cleared
d21: Concat/Stripe is cleared
d30: Soft Partition is cleared
d31: Soft Partition is cleared

Create non root metadevices

# for i in 1 3 ; do
> metainit -f d$i -m d2$i
> done
d1: Mirror is setup
d3: Mirror is setup

Update vfstab for non-root filesystems.

/dev/md/dsk/d1 - - swap - no -
/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no logging
/dev/md/dsk/d3 /dev/md/rdsk/d3 /var ufs 1 no logging >>>> here we have made a change. d23 replaced by d3
swap - /tmp tmpfs - yes -

Change root to mirror

# metainit -f d0 -m d20
d0: Mirror is setup


# metaroot d0


# lockfs –fa


# reboot

After reboot re-mirror soft partitions

Clean the metadb on first disk.

# metadb -f -d c0t0d0s7

Reattach the empty disk to existing metadevices so that will have 2nd submirror

# prtvtoc /dev/rdsk/c0t2d0s2 | fmthard -s - /dev/rdsk/c0t0d0s2
fmthard: New volume table of contents now in place.

Setup metadb

# metadb -a -f -c3 c0t0d0s7

# metadb -i
flags first blk block count
a u 16 8192 /dev/dsk/c0t0d0s7
a u 8208 8192 /dev/dsk/c0t0d0s7
a u 16400 8192 /dev/dsk/c0t0d0s7
a m p luo 16 8192 /dev/dsk/c0t2d0s7
a p luo 8208 8192 /dev/dsk/c0t2d0s7
a p luo 16400 8192 /dev/dsk/c0t2d0s7

# metainit -f d10 1 1 c0t0d0s0;metainit -f d11 1 1 c0t0d0s1;metainit -f d13 1 1 c0t0d0s3;
metainit -f d14 1 1 c0t0d0s4
d10: Concat/Stripe is setup
d11: Concat/Stripe is setup
d13: Concat/Stripe is setup
d14: Concat/Stripe is setup

# metattach d0 d10; metattach d1 d11; metattach d3 d13; metattach d4 d14
d0: submirror d10 is attached
d1: submirror d11 is attached
d3: submirror d13 is attached
d4: submirror d14 is attached

# metastat -p
d1 -m d21 d11 1
d21 1 1 c0t2d0s1
d11 1 1 c0t0d0s1
d0 -m d20 d10 1
d20 1 1 c0t2d0s0
d10 1 1 c0t0d0s0
d3 -m d23 d13 1
d23 1 1 c0t2d0s3
d13 1 1 c0t0d0s3
d33 -p d4 -o 2097216 -b 2097152
d4 -m d24 d14 1
d24 1 1 c0t2d0s4
d14 1 1 c0t0d0s4
d32 -p d4 -o 32 -b 2097152

Wait till devices sync up with each other & you’re almost done.

There is a downtime cost associated with this method. This method required downtime of around ~20 Mins (2/3 reboot). This is really time consuming, tedious dirty method and this will teach you how initial planning for accurate capacity planning, proper filesystem layout will help in long run!!!