Find it

Saturday, January 30, 2010

/var is full & causes Sun patching

NOTE: Below is information on how to free up space in /var by deleting certain patch related files.  This should only be used as a last resort of gaining space in /var.


PS: Do not use this procedure on server hosting non-global zones.  It will break the ability to install non-global zones.This procedure has fairly risky impact to the package and patch database. 


It has been observed that sometimes /var is almost full & /var/sadm eating up a lot of disk space. It leads to a problem sometimes, especially during patching. Normally we are uncertain on what to delete or move to make adequate space available so that you can start with patching. Following procedure tells what exactly /var/sadm/pkg contains & what we can delete or move to make adequate space available & get going with patching.

If /var/sadm/pkg is really big (it holds patch backout info):
----------------------------------------------------------------

Check for the existence of "obsolete.Z" files in the /var/sadm/pkg hierarchy. If you've done a lot of patching these files can take up a lot of space. They can be removed without ill effect. For example:

# find /var/sadm/pkg -type f -name obsolete.Z -exec rm {} +

This can recover several hundreds of megabytes depending on the number of patches applied.
If you don't want to backout the patches again you can just get rid of the undo backup files:

# find /var/sadm/pkg -name '1?????-??' -type d -exec rm -rf {} +
-------------------------------------------------------------------------------
There are two ways of deleting the backout information.
1. Remove all undo.Z and obsolete.Z files from the /var/sadm/pkg directory structure. This frees most of the space but also prevents any patch from being backed out. This can be catastrophic if a patch is applied that causes problems and needs to be backed out. If you have removed the backout files following the patch install, it's impossible to remove the patch.

2. Remove only the obsolete.Z files from the /var/sadm/pkg directory structure. This still frees a lot of space but it also preserves the ability to remove any of the most recent versions of any patch. This can be particularly useful if a problem is found with a freshly applied patch. If removal of backout information is unavoidable, this would be the better method of the two.
Here is an example of a command which deletes all obsolete.Z files:

# find /var/sadm/pkg -name obsolete.Z -exec rm {} \;

NOTE 1: In the Solaris 10 OS, there are additional entries located in the /var/sadm/pkg//save directory structure: specifically, the pspool directory and everything under it. These files and directories are created and utilized by the updated Solaris 10 patching utilities. For example, there will be more undo.Z files found under the pspool directory. Such as:

/var/sadm/pkg/SUNWcsr/save/pspool/SUNWcsr/save/118833-36/undo.Z

Removing those undo.Z files will free up some space but it is highly discouraged.
Infodoc 14295
----------------------------------------------------------------------------------
E.g.

# du -skh /var/sadm
1.3G /var/sadm


# find /var/sadm/pkg -type f -name obsolete.Z -exec rm {} +

# du -skh /var/sadm
391M /var/sadm

OR THE BEST WAY TO SEPARATE THE /var/sadm. Here you can have /var/sadm on internal disk as a soft partition in case you have enough space available on internal disk.

# metainit d## -p d4 4g

# newfs /dev/md/rdsk/d##
# mount /dev/md/dsk/d## /mnt
#rsync -aHv /var/sadm/ /mnt/
PS: repeat rsync to verify everything is copied.
# rsync -aHv --delete /var/sadm/ /mnt/
# cd /var/
# rm -r sadm
# mkdir /var/sadm/
# chmod 755 /var/sadm
# chown root:sys /var/sadm
# umount /mnt
# vi /etc/vfstab

Add:
/dev/md/dsk/d## /dev/md/rdsk/d## /var/sadm ufs 2 yes logging
# mount -a

That's it!!

5 comments:

  1. i have deleted obsolete.z files which are more than 300 days old on a server hosting zones. I am going to patch the server. Will there be any impact of this while installing patches?

    ReplyDelete
  2. Hi there,

    I don’t think removing obsolete.Z files should impact any further patching.
    obsolete.Z is the old file data from a previous patch install. BTW, how obsolete.Z file gets created? So let’s take an example, if you install patch 12345 with version 1 then it creates undo.Z file, the second time 12345 was installed with version 2 then the undo.Z file that existed before was renamed to obsolete.Z so this way we get obsolete.Z.

    obsolete.Z files are needed if you need to back out more than one level of patch. When you execute patchadd command to install any patch it obsoletes another patch, then the corresponding file for the
    old patch becomes /var/sadm/pkg/[package_name]/save/[patch_name]/obsolete.Z also patchadd command creates another important file called undo.Z which helps in backing out the installed patch.

    You may want to remove obsolete.Z files for patches applied sufficiently long ago or maybe for patches _obsoleted_ sufficiently long ago. In your case it’s almost a year back files so looks safe to me, though I’m not sure about your patch cycle.

    Hope this helps.

    Thank you/Nilesh

    ReplyDelete
  3. Hi,

    Yes, true. This procedure "may" cause issues installing new non global zone(s) however ideally it should not affect any future patching. I think, you're question was about impact on installing new patches...

    Thnak you/Nilesh

    ReplyDelete
  4. Thanks again. It clears everything.

    ReplyDelete
  5. i have a situation here if i am in middle of patching sudenly /var is out of space what would happen to current patching ...how can i continue with patching .

    ReplyDelete