Finding and removing locked files on a GNU/Linux system

In this post we will see how you can find and remove locked files on a GNU/Linux system.

For this purpose we are going to use lsof(8) and lslk(8) tools.

Sometimes it may happen that an application is not able to start up, or even not working properly due to some files being locked by other processes. This may happen in an environment where multiple clients are using shared resources, like for example a shared home which is mounted on multiple machines.

In this post we will see how to find and remove such locked files under a Debian GNU/Linux system.

Requirements

  • root access or sudo rights

Installation

First let's install the needed tools:

$ sudo apt-get update
$ sudo apt-get install lsof lslk

Now that we have the tools, le's have a look at the files that are currently being locked.

Finding the locked files

In order to view all locked files on the current system, simply execute lslk(8).

In this document as an example, we will find and remove a locked file from a KDE session on a shared storage, where multiple clients are mounting their home partitions from an NFS server.

Now let's view the locked files on the system:

$ sudo lslk
SRC         PID   DEV      INUM     SZ TY M ST WH END LEN NAME
(unknown)  1190 254,1 308115263        r 0  0  0   0   0 /mnt/homes (/dev/mapper/vg0-homes)
(unknown)  2294 254,1 308115263        r 0  0  0   0   0 /mnt/homes (/dev/mapper/vg0-homes)
(unknown)  2392 254,1 308115263        r 0  0  0   0   0 /mnt/homes (/dev/mapper/vg0-homes)
(unknown)  2397 254,1 308115263        r 0  0  0   0   0 /mnt/homes (/dev/mapper/vg0-homes)
lpd        3028 254,0   1212425      5  w 0  0  0   0   0 /var/run/lpd.pid
master     3187 254,0    262203     17  w 0  0  0   0   0 /var/spool/postfix/pid/master.pid
atd        3260 254,0   1212456      5  w 0  0  0   0   0 /var/run/atd.pid
(unknown)  3262 254,0   1212458         w 0  0  0   0   0 /var (/dev/mapper/vg0-var)
nmbd      14654 254,0    688137  32768  r 0  4  0   4   0 /var/lib/samba/public/unexpected.tdb
nmbd      14654 254,0    688148    696  r 0  4  0   4   0 /var/lib/samba/public/messages.tdb
nmbd      14654 254,0   1212429      6  w 0  0  0   0   0 /var/run/samba/public/nmbd.pid
smbd      14656 254,0    688138   8192  r 0  4  0   4   0 /var/lib/samba/public/locking.tdb
smbd      14656 254,0    688132   8192  r 0  4  0   4   0 /var/lib/samba/public/brlock.tdb
smbd      14656 254,0    688133 163840  r 0  4  0   4   0 /var/lib/samba/public/connections.tdb
smbd      14656 254,0    688147 188416  r 0  4  0   4   0 /var/lib/samba/public/sessionid.tdb
smbd      14656 254,0    688148    696  r 0  4  0   4   0 /var/lib/samba/public/messages.tdb
smbd      14656 254,0   1212430      6  w 0  0  0   0   0 /var/run/samba/public/smbd.pid
nmbd      14669 254,0    655876  32768  r 0  4  0   4   0 /var/lib/samba/users/unexpected.tdb
nmbd      14669 254,0    655875    696  r 0  4  0   4   0 /var/lib/samba/users/messages.tdb
nmbd      14669 254,0   1212448      6  w 0  0  0   0   0 /var/run/samba/users/nmbd.pid
smbd      14671 254,0    655870   8192  r 0  4  0   4   0 /var/lib/samba/users/locking.tdb
smbd      14671 254,0    655719    696  r 0  4  0   4   0 /var/lib/samba/users/brlock.tdb
smbd      14671 254,0    655718  57344  r 0  4  0   4   0 /var/lib/samba/users/connections.tdb
smbd      14671 254,0    655843 106496  r 0  4  0   4   0 /var/lib/samba/users/sessionid.tdb
smbd      14671 254,0    655875    696  r 0  4  0   4   0 /var/lib/samba/users/messages.tdb
smbd      14671 254,0   1212450      6  w 0  0  0   0   0 /var/run/samba/users/smbd.pid

What is interesting in the above output are the first four lines, which are marked as unknown.

SRC         PID   DEV      INUM     SZ TY M ST WH END LEN NAME
(unknown)  1190 254,1 308115263        r 0  0  0   0   0 /mnt/homes (/dev/mapper/vg0-homes)
(unknown)  2294 254,1 308115263        r 0  0  0   0   0 /mnt/homes (/dev/mapper/vg0-homes)
(unknown)  2392 254,1 308115263        r 0  0  0   0   0 /mnt/homes (/dev/mapper/vg0-homes)
(unknown)  2397 254,1 308115263        r 0  0  0   0   0 /mnt/homes (/dev/mapper/vg0-homes)

We get actually quite useful information from the lslk(8) output - we know where the file resides on the file system (in our case that is /mnt/homes), what is it's inode number, a PID, etc. Please refer to the manual page of lslk(8) for more information about the different options and what the different column meaning is.

What we can also notice is that there are four processes that are using the same file from /mnt/homes.

Now let's try to find the files opened by these processes using the lsof(8) tool:

$ sudo lsof -p 1190,2294,2392,2397

The above command will list all open files for the specified processes. Sometimes it may happen that we don't actually get any result from that command.

What we can do in that case is to try to find the file using it's inode number. So let's search for the file by it's inode number:

$ sudo find /mnt/homes -inum 308115263
/mnt/homes/home/foo/.qt/.qtrc.lock

After searching for the file using it's inode number we've found the locked file and we can remove it:

$ sudo rm -f /mnt/homes/home/foo/.qt/.qtrc.lock

Now that we have removed the locked file, we can start up our applications again.

Written on January 18, 2011