Thursday, August 5, 2010

GFS2 lockdump analysis

G:  s:EX n:2/1fda70 f:Dy t:EX d:UN/132055446000 l:0 a:0 r:5
 H: s:EX f:H e:0 p:1405 [httpd] gfs2_write_begin+0x61/0x33e [gfs2]
 H: s:EX f:W e:0 p:31819 [umount] gfs2_write_inode+0x57/0x152 [gfs2]

The content of the file is a series of lines. Each line starting with G: represents one glock, and the following lines, indented by a single space, represent an item of information relating to the glock immediately before them in the file

Lines in the debugfs file starting with H: (holders) represent lock requests either granted or waiting to be granted

The flags field on the holders line f: shows which: The 'W' flag refers to a waiting request, the 'H' flag refers to a granted request

The glocks which have large numbers of waiting requests are likely to be those which are experiencing particular contention.

Having identified a glock which is causing a problem, the next step is to find out which inode it relates to. The glock number (n: on the G: line) indicates this. It is of the form type/number and if type is 2, then the glock is an inode glock and the number is an inode number. To track down the inode, you can then run find -inum number where number is the inode number converted from the hex format in the glocks file into decimal.

If the glock that was identified was of a different type, then it is most likely to be of type 3: (resource group). If you see significant numbers of processes waiting for other types of glock under normal loads, then please report this to Red Hat support.

f you do see a number of waiting requests queued on a resource group lock there may be a number of reason for this. One is that there are a large number of nodes compared to the number of resource groups in the filesystem. Another is that the filesystem may be very nearly full (requiring, on average, longer searches for free blocks). The situation in both cases can be improved by adding more storage and using the gfs2_grow command to expand the filesystem. 

The W flag indicates that this is waiting for a glock
UN (unlocked) state
SH (shared) lock

Monday, August 2, 2010

du command usage

1. du -hs *  (shows linux directory size)
2. du * | sort -n (finding large files with sort option)