Solaris 10 by default places /tmp on swap. This is good for speed, but not so good on a general purpose box where some applications may fill up /tmp. If you fill /tmp, you essentially reduce the amount of available swap to 0. This can lead to trouble, run out of physical ram, and new processes may not start.
You will get errors like shown below -
# ps -ef
-bash: fork: Not enough space
# prstat
-bash: fork: Not enough space
...
# dmesg
...
Jan 7 02:56:51 gz1.myhome.com tmpfs: [ID 518458 kern.warning] WARNING: /tmp: File system full, swap space limit exceeded
Jan 7 02:56:57 gz1.myhome.com genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 8223 (exim)
Jan 7 02:57:26 gz1.myhome.com genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 563 (httpd)
...
The easiest way to fix this is to immediately disable any services that eat RAM using svcadm disable, and clear out /tmp however in this case you will have to take the application downtime. You can then either move /tmp to a physical partition by editing /etc/vfstab, increase the amount of swap, or THE best way out is limit the amount of swap /tmp can use by adding a mount option to /etc/vfstab:
# grep /tmp /etc/vfstab
swap - /tmp tmpfs - yes SIZE=2048M
Unfortunately with this you have to reboot the box!!
I encountered similar issue. In this kind of situation, you can still run something provided you still have a login shell. If so, you can take advantage of shell built-in commands. That will avoid forking commands. See http://chihungchan.blogspot.com/2010/02/my-solaris-cannot-fork.html
ReplyDeleteInteresting one. However I think writing such programs for advance level commands require good C/C++ programming knowledge… BUT I personally like the idea. Thanks for reading the blog & providing very good tip.
ReplyDeleteWhy new limit as the above example won't get the same error when swap limit is reached? Thanks
ReplyDelete"SIZE" option should be specified as "size" (lower case) in /etc/vfstab - else it is ignored on reboot.
ReplyDelete