Replacing/Relabeling the Root Pool Disk

Unfortunately one of disk associated to rootpool went crazy and fun started for a day there!!! Let's fix it. BTW now a days I am testing a small part of rpool recovery after system crash... This experience is from rpool recovery and testing project.

One can see belwo shown error if something wrong with the root pool disk -

# zpool status -x
pool: rpool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: none requested
config:


NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror DEGRADED 0 0 0
c0t0d0s0 UNAVAIL 0 0 0 cannot open
c0t2d0s0 ONLINE 0 0 0


errors: No known data errors

Have bad disk replaced with good one or brand new disk.

# zpool offline rpool c0t0d0s0
# cfgadm -c unconfigure c1::dsk/c0t0d0

Physically replace the primary disk. In my case it is c0t0d0s0.

Now I am labeling/relabeling the disk. Label rpool disk always with SMI label format and not EFI.

# format -e
Searching for disks...done




AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@1f,0/ide@d/dad@0,0
1. c0t2d0
/pci@1f,0/ide@d/dad@2,0
Specify disk (enter its number): 0
selecting c0t0d0
[disk formatted, no defect list found]




FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
show - translate a disk address
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
volname - set 8-character volume name
! - execute , then return
quit
format> l
[0] SMI Label
[1] EFI Label
Specify Label type[0]: 0
Ready to label disk, continue? yes


format>

Reconfigure the disk and bring it online, if required.

# cfgadm -c configure c1::dsk/c0t0d0
# zpool online rpool c0t0d0

Before de-attach you can replace it however I dont see much need of this step in my case it failed so I am directly De-attach the bad disk from the pool

# zpool detach rpool c0t0d0s0

See the status

# zpool status -v
pool: rpool
state: ONLINE
scrub: none requested
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c0t2d0s0 ONLINE 0 0 0


errors: No known data errors

Now we have newly added disk to the system and we need to add/attach this to the pool.

# zpool attach rpool c0t2d0s0 c0t0d0s0
Please be sure to invoke installboot(1M) to make 'c0t0d0s0' bootable.

Check the resilvering status of the newly attached disk.

# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h2m, 25.39% done, 0h6m to go
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t2d0s0 ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0 1.62G resilvered


errors: No known data errors

After the disk resilvering is complete, install the boot blocks.

# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t0d0s0

Confirm that you can boot from the replacement disk.

REF: http://www.solarisinternals.com/

Replace Faulty Disk - Solaris Volume Manager

One of disk went crazy this morning let us fix it.

 

Following is an illustrated how to replace a faulty disk.

• Configuration: -
• - all disk are SCSI disks
• - disks mirror by Solaris Volume Manager
• - faulty disk was c0t0d0, and the other sub-mirror disk was c0t1d0

The following steps and set of commands can be followed during disk replacement:

 

1. Check whether there are any replicas on faulty disk, then remove them if any;

 

# metadb

 

If yes delete them.

 

# metadb -d c0t0d0s7

 

Verify if there are no existing replicas left on faulty disk;

 

# metadb | grep c0t0d0

 

2. Run "cfgadm" command to remove the failed disk.

 

# cfgadm -c unconfigure c1::dsk/c0t0d0

 

Insert and configure the new disk.

 

# cfgadm -c configure c1::dsk/c0t0d0 >>>>> Verify that disk is properly configured;

 

# cfgadm -al

 

If necessary run following related disk commands:

 

# devfsadm
# format >>>> verifying new disk

 

 

4. Create desired partition table on the new disk with prtvtoc command;

 

# prtvtoc /dev/rdsk/c0t1d0s2 | fmthard -s - /dev/rdsk/c0t0d0s2 >>>> Make sure disk sizes are same.

 

5. Recreate replicas on new disk:

 

# metadb -a -f -c 3 /dev/dsk/c0t0d0s7

 

6. Run metareplace to enable and resync the new disk.

 

# metareplace -e d0 c0t0d0s0

 

In case SVM device-id not up-to-date, run "metadevadm" which will update the new disk device-id.

 

# metadevadm -u c0t0d0

 

NOTE: metadevadm is used for update metadevice information. Use this command when the pathname stored in the metadevice state database no longer correctly addresses the device or when a disk drive has had its device ID changed.

df -kh - df: a line in /etc/mnttab has too many fields

Getting the following error on the "df -h" command -

df: a line in /etc/mnttab has too many fields

Solution -

This issue can arise because of reasons like - wrong formatting of vfstab, auto_master files.

If you want to see if a white space is the problem, you should do a "view /etc/mnttab" and then input ":set list". This will display TABS as "^T" symbols. Spaces will show up as spaces. The issue is with files like /etc/vfstab, /etc/auto_master and such are white space delimited already so looking for spaces in there is going to be somewhat fruitless as a space between fields is perfectly acceptable.

Most of the time

Anything found in /etc/mnttab has already been accepted by the kernel as a valid mount. /etc/mnttab is a kernel created and maintained file.

I was having backup copy of vfstab file I simply replaced it with original and issue disappeared.

This is pretty hot tip!

AIX Large File support - Problem even if I am on JFS2

Problem -

Had an issue with a large queue file on one of our Production queue managers.

Solution –

Let us change the fsize value to unlimited or -1 [fsize stands for - The largest file that a user can create.]

# lsuser mqm
mqm id=30031 pgrp=mqm groups=mqm,mqbrkrs home=/var/mqm shell=/usr/bin/ksh gecos=MQM Application login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=2 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=2097151 cpu=-1 data=-1 stack=65536 core=2097151 rss=65536 nofiles=2000 data_hard=-1 time_last_unsuccessful_login=1241892309 tty_last_unsuccessful_login=ssh host_last_unsuccessful_login=asdemq3.sdde.deere.com unsuccessful_login_count=11 roles=


# chuser fsize='-1' mqm


# echo $?
0

# lsuser mqm
mqm id=30031 pgrp=mqm groups=mqm,mqbrkrs home=/var/mqm shell=/usr/bin/ksh gecos=MQM Application login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=2 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=-1 cpu=-1 data=-1 stack=65536 core=2097151 rss=65536 nofiles=2000 data_hard=-1 time_last_unsuccessful_login=1241892309 tty_last_unsuccessful_login=ssh host_last_unsuccessful_login=asdemq3.sdde.deere.com unsuccessful_login_count=11 roles=

Now login as a mqm user

$ id
uid=30031(mqm) gid=30109(mqm) groups=32989(mqbrkrs)

Now we will test/verify if we can create big files say of size 3G

So, lets create a file of size 1G to be start with

$ /usr/sbin/lmktemp test 1073741824 >>>> lmktemp is the command which can be used to create temp file or else you can also use dd command. Or one can use below small script too however can be optional as we have plenty of options...

#!/usr/bin/perl -w

my $buf = ' ' x (1024 * 1024);
my $i;

open(F,'>mybigfile') || die "can't open file\n$!\n";

for($i=0;$i<2048;$i++){
print F $buf;
}

close(F);

Create a second 1GB file:

$ cp test test01

To create a 2GB file, append the first file to the second file:

$ cat test >> test01

To create a 3GB file, append the first file to the second file again:

$ cat test >> test01

$ ls -l
total 8388608
drwxrwx--- 3 mqm mqm 256 Dec 16 2007 MQGWP2
drwxr-xr-x 2 mqm mqm 256 Jun 20 2007 lost+found
-rw-rw-r-- 1 mqm mqm 1073741824 Oct 06 09:23 test
-rw-rw-r-- 1 mqm mqm 3221225472 Oct 06 09:24 test01

Cool, now I can create big file after changing fsize value to -1.

Upgrade Veritas Storage foundation 4.1 to 5.0 on Sun Solaris

All right, After a long time I finally back on blog. Little busy with my house shifting work and setting it up along with managing office work... It was real busy week... Today I have a task of Veritas Volume Manager/Veritas Storage Foundation upgrade work so thought having this information on the blog.

let me admit before I start - I am not an expert on this product and have very small experience with this product.

I am upgrading Veritas Storage foundation 4.1 to 5.0 on Sun Solaris 5.9, below steps guide on "how to?"

1. Save your existing configuration at some safe location. This document does not consider rootdg as we dont use rootdg at all. What we will concentrate on is datadg.

Lets see what is current Veritas Volume Manager version is -

# pkginfo -l VRTSvxvm
PKGINST: VRTSvxvm
NAME: VERITAS Volume Manager, Binaries
CATEGORY: system
ARCH: sparc
VERSION: 4.0,REV=12.06.2003.01.35
BASEDIR: /
VENDOR: VERITAS Software
DESC: Virtual Disk Subsystem
PSTAMP: VERITAS-4.0_MP2_RP7.2:2007-02-22
[.....]

or you can use command -

# pkginfo -l VRTSvxvm |grep VERSION |awk '{print $2}' |awk -F, '{print $1}'

Before upgrade make sure you save your current configuration like license file, vfstab file, df -kh O/P, mount command's O/P and few VxVM commands output like vxprint, vxdisk list, vxdg list etc... This information will help you in case of upgrade fail and backout process.

# cd /etc/vx/licenses/; tar -cvf /home/nilesh_rjoshi/vx_licenses_101109.tar ./* >>>> Take backup of VxVM licenses

# cp -p /etc/vfstab /home/nilesh_rjoshi/ >>>> Backup vfstab file.

# df -kh > /home/nilesh_rjoshi/df_kh_op_101109 >>>> df -kh output

# mount > /home/nilesh_rjoshi/mount_op_101109 >>>> mount command output

# vxprint -ht > /home/nilesh_rjoshi/vxprint_ht_op_101109 >>>> Save vxprint -ht output. vxprint utility displays complete or partial information from records in VERITAS Volume Manager (VxVM) disk group configurations.

# vxprint -g datadg -mvphsrqQ > /home/nilesh_rjoshi/vxprint-g_datadg_mvphsrqQ_op_101109 >>>> In detail O/P of vxprint command

# vxdisk list > /home/nilesh_rjoshi/vxdisk_list_op_101109 >>>> Save O/P of vxdisk, it shows all disks associated with or used by VERITAS Volume MAnager.

# vxdg list > /home/nilesh_rjoshi/vxdg_list_101109 >>>> Backup the data group list

Nice one, I guess now we have enough information backed up and can be used in case of backout if something goes wrong.

Let us start with the upgrade process.

Before proceeding ask application users to stop their applications or if you have authority and knowledge on how to stop applications then please disable Applications & Unmount filesystems.

Below three steps needs to be carried out -

- umount vxfs filesystems
- comment them out of the vfstab
- disable application startup in /etc/rc2.d and /etc/rc3.d

# df -kh
/dev/md/dsk/d0 3.0G 2.4G 481M 84% /
/proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
fd 0K 0K 0K 0% /dev/fd
/dev/md/dsk/d3 2.0G 1.6G 334M 83% /var
swap 11G 160K 11G 1% /var/run
dmpfs 11G 0K 11G 0% /dev/vx/dmp
dmpfs 11G 0K 11G 0% /dev/vx/rdmp
/dev/md/dsk/d6 992M 8.2M 925M 1% /home
/dev/md/dsk/d5 2.0G 362M 1.5G 19% /tmp
/dev/vx/dsk/datadg/app1_vol
5.0G 2.6G 2.3G 54% /app1
/dev/vx/dsk/datadg/app_db_vol
1.0G 452M 536M 46% /app_db
/dev/vx/dsk/datadg/appdata_vol
40G 10G 28G 28% /app_data
[.....]

# umount /app1; umount /app_db; umount /app_data

Verify if with df -kh to see everything is fine.

# vi /etc/vfstab >>>> Comment out veritas filesystems in vfstab file.

Now disable application startup in /etc/rc2.d and /etc/rc3.d

# mv S90app_tomcat K90app_tomcat

Make sure you have comment out entries in vfstab file for all veritas filesystems, application auto startup is stoped through rc directories, and all veritas filesystems are un-mounted. Next task is export the diskgroup.

# vxvol -g datadg stopall >>>> The vxvol utility performs VERITAS Volume Manager (VxVM) operations on volumes. This is Maintenance command for VERITAS.

# vxdg deport datadg >>>> Disables access to the specified disk group. A disk group cannot be deported if any volumes in the disk group are currently open. When a disk group is deported, the host ID stored on all disks in the disk group are cleared (unless a new host ID is specified with -h), so the disk group is not reimported automatically when the system is rebooted.

# vxdg -t import datadg >>>> Dissassocaite dg from host. Where -t for a temporary name. import option imports a disk group to make the specified disk group available on the local machine.When a disk group is imported, all disks in the disk group are stamped with the host's host ID.

# vxdg deport datadg

Fine, now we are done with all applicable operations on disk group.

Start the installer/upgrader:

Mount the CD/ISO file on the server and start the installation.

Quick note on mounting ISO file on Solaris -

# lofiadm -a /nilesh_rjoshi/VERITAS_Storage_Foundation_5.iso
/dev/lofi/1
# mount -F hsfs /dev/lofi/1 /mnt

Navigate to location
# cd /mnt/Solaris/Veritas/v5.0/sf_50_base

NOTE: BE SURE YOU'RE IN RUNLEVEL 3, THIS WILL NOT WORK IN RUNLEVEL 1

# ./installer

Storage Foundation and High Availability Solutions 5.0

Symantec Product Version Installed Licensed
===========================================================================
Veritas Cluster Server no no
Veritas File System 4.0 yes
Veritas Volume Manager 4.0 yes
Veritas Volume Replicator no no
Veritas Storage Foundation 4.0 yes
Veritas Storage Foundation for Oracle no no
Veritas Storage Foundation for DB2 no no
Veritas Storage Foundation for Sybase no no
Veritas Storage Foundation Cluster File System no no
Veritas Storage Foundation for Oracle RAC no no

Task Menu:

I) Install/Upgrade a Product C) Configure an Installed Product
L) License a Product P) Perform a Pre-Installation Check
U) Uninstall a Product D) View a Product Description
Q) Quit ?) Help

Enter a Task: [I,C,L,P,U,D,Q,?] I

Storage Foundation and High Availability Solutions 5.0

1) Veritas Cluster Server
2) Veritas File System
3) Veritas Volume Manager
4) Veritas Volume Replicator
5) Veritas Storage Foundation
6) Veritas Storage Foundation for Oracle
7) Veritas Storage Foundation for DB2
8) Veritas Storage Foundation for Sybase
9) Veritas Storage Foundation Cluster File System
10) Veritas Storage Foundation for Oracle RAC
11) Veritas Cluster Management Console
12) Web Server for Storage Foundation Host Management
13) Symantec Product Authentication Service
b) Back to previous menu

Select a product to install: [1-13,b,q] 5
Enter the system names separated by spaces on which to install SF: XXXXXX

Checking system licensing

Installing licensing packages
Permanent SF license registered on sdxapp1

Do you want to enter another license key for sdxapp1? [y,n,q] (n) >>>> press enter

Storage Foundation and High Availability Solutions 5.0

installer will upgrade or install the following SF packages:
VRTSperl Veritas Perl 5.8.8 Redistribution
VRTSvlic Veritas Licensing
VRTSicsco Symantec Common Infrastructure
VRTSpbx Symantec Private Branch Exchange
VRTSsmf Symantec Service Management Framework
VRTSat Symantec Product Authentication Service
VRTSobc33 Veritas Enterprise Administrator Core Service
VRTSob Veritas Enterprise Administrator Service
VRTSobgui Veritas Enterprise Administrator
VRTSccg Veritas Enterprise Administrator Central Control Grid
VRTSmh Veritas Storage Foundation Managed Host by Symantec
VRTSaa Veritas Enterprise Administrator Action Agent
VRTSspt Veritas Software Support Tools
SYMClma Symantec License Inventory Agent
VRTSvxvm Veritas Volume Manager Binaries
VRTSdsa Veritas Datacenter Storage Agent
VRTSfspro Veritas File System Management Services Provider
VRTSvmman Veritas Volume Manager Manual Pages
VRTSvmdoc Veritas Volume Manager Documentation

Press [Return] to continue:
[........]

I got below error -

The following errors prevent SF from being installed successfully on XXXXX:
827877 KB is required in the / volume and only 490085 KB is available on XXXXX


installer log files are saved at /opt/VRTS/install/logs/installer-kPOqLK

check which directory or file eating up more disk space -

# find . -name core -xdev -exec ls -l {} \; 2>/dev/null >>>> Find core files
# find . -size +20000 -xdev -exec ls -l {} \; 2>/dev/null

Do the appropriate clean-up and start installer once again.

So make sure your root filesystem has enough space available.

CPI WARNING V-9-1-1303 Installing Veritas Storage Foundation will leave only approximately 5107 KB of free space remaining
in the /var volume on XXXXX.
Do you want to continue? [y,n,q] (n) y
installer is now ready to upgrade SF.
All SF processes that are currently running must be stopped.

Are you sure you want to upgrade SF? [y,n,q] (y) y
Checking for any AP driver issues on XXXXXX ..................................................................... None

Storage Foundation and High Availability Solutions 5.0


Storage Foundation and High Availability Solutions 5.0

Stopping SF: 100%

Shutdown completed successfully on all systems


Storage Foundation and High Availability Solutions 5.0

Uninstalling SF: 100%

Uninstall completed successfully on all systems

Storage Foundation and High Availability Solutions 5.0

Installing SF: 100%

Installation completed successfully on all systems
Upgrade log files and summary file are saved at:

/opt/VRTS/install/logs/installer-XffXHU


CPI WARNING V-9-11-2246 You have completed upgrading VxVM on some or all of the systems. Reboot your systems at this time.

Reboot all systems on which VxFS was installed or upgraded.
shutdown -y -i6 -g0

See the Veritas File System Administrators Guide for information on using VxFS.

At least one package will require a reboot prior to configuration. Please reboot and run installer -configure on the
following systems after reboot has completed:

XXXXXX
Execute '/usr/sbin/shutdown -y -i6 -g0' to properly restart your systems.

Now let us install the patch.

# cd /mnt/Solaris/Veritas/v5.0/sf_50_mp1

# ./installmp

Veritas Maintenance Pack 5.0MP1 Installation Program

Copyright (c) 2007 Symantec Corporation. All rights reserved. Symantec, the Symantec Logo are trademarks or registered
trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their
respective owners.

The Licensed Software and Documentation are deemed to be "commercial computer software" and "commercial computer software
documentation" as defined in FAR Sections 12.212 and DFARS Section 227.7202.

Logs for installmp are being created in /var/tmp/installmp-tjSj6m.

Enter the system names separated by spaces on which to install MP1: XXXXX

Initial system check:



Veritas Maintenance Pack 5.0MP1 Installation Program

Checking installed packages on XXXXXX

The following patches will be installed on XXXXXX:
123996-06 to patch package VRTSsmf
123678-04 to patch package VRTSat
122632-02 to patch package VRTSobc33
122631-02 to patch package VRTSob
122633-02 to patch package VRTSobgui
123076-02 to patch package VRTSccg
123079-02 to patch package VRTSmh
123075-02 to patch package VRTSaa
[.......]

Press [Return] to continue:
installmp is now ready to install MP1.
Required 5.0 processes that are currently running will be stopped.

Are you sure you want to install MP1? [y,n,q] (y)

Installer will now stop the process and will install MP1. Basically this will install several Sun patches required for VERITAS packages. It is just a process of fixing bugs!

Add the mp1_rp3 [VRTSvxvm 5.0MP3RP2: Rolling Patch 02 for Volume Manager 5.0MP3]

# cd /mnt/Solaris/Veritas/v5.0/sf_50_mp1_rp3/

# patchadd 123742-04

# patchadd 124361-04

Once all this done the re-import the diskdg

# vxdg import datadg

# vxdg deport datadg

# vxdg import datadg

# vxvol -g datadg startall

Verify with

# vxprint -ht >>>> Verify that the volumes are ENABLED/ACTIVE

Final touchup

Renable filesstems in /etc/vfstab
Re-enable application startup in /etc/rc2.d/ /etc/rc3.d
verify you can mount all filesystems in vfstab with "mount -a"
reboot system one final time: init 6 or shutdown -y -i6 -g0

Here you done. This upgrade is pretty straight forward if everything is good like enough disk space on root, var and tmp, system is well patched etc...

Hope it will be a useful information for someone! Have a great weekend.

Permanently disabling sendmail on AIX - a small tip

To stop sendmail from starting on reboot, comment it out in the /etc/rc.tcpip file. Then do the stopsrc -s sendmail to end the running copy.