lockfile — Platform-independent file locking

Note

This module is alpha software. It is quite possible that the API and implementation will change in important ways as people test it and provide feedback and bug fixes. In particular, if the mkdir-based locking scheme is sufficient for both Windows and Unix platforms, the link-based scheme may be deleted so that only a single locking scheme is used, providing cross-platform lockfile cooperation.

Note

The implementation uses the with statement, both in the tests and in the main code, so will only work out-of-the-box with Python 2.5 or later. However, the use of the with statement is minimal, so if you apply the patch in the included 2.4.diff file you can use it with Python 2.4. It’s possible that it will work in Python 2.3 with that patch applied as well, though the doctest code relies on APIs new in 2.4, so will have to be rewritten somewhat to allow testing on 2.3. As they say, patches welcome. ;-)

The lockfile module exports a FileLock class which provides a simple API for locking files. Unlike the Windows msvcrt.locking() function, the Unix fcntl.flock(), fcntl.lockf() and the deprecated posixfile module, the API is identical across both Unix (including Linux and Mac) and Windows platforms. The lock mechanism relies on the atomic nature of the link() (on Unix) and mkdir() (On Windows) system calls.

Note

The current implementation uses os.link() on Unix, but since that function is unavailable on Windows it uses os.mkdir() there. At this point it’s not clear that using the os.mkdir() method would be insufficient on Unix systems. If it proves to be adequate on Unix then the implementation could be simplified and truly cross-platform locking would be possible.

Note

The current implementation doesn’t provide for shared vs. exclusive locks. It should be possible for multiple reader processes to hold the lock at the same time.

The module defines the following exceptions:

exception lockfile.Error

This is the base class for all exceptions raised by the LockFile class.

exception lockfile.LockError

This is the base class for all exceptions raised when attempting to lock a file.

exception lockfile.UnlockError

This is the base class for all exceptions raised when attempting to unlock a file.

exception lockfile.LockTimeout

This exception is raised if the LockFile.acquire() method is called with a timeout which expires before an existing lock is released.

exception lockfile.AlreadyLocked

This exception is raised if the LockFile.acquire() detects a file is already locked when in non-blocking mode.

exception lockfile.LockFailed

This exception is raised if the LockFile.acquire() detects some other condition (such as a non-writable directory) which prevents it from creating its lock file.

exception lockfile.NotLocked

This exception is raised if the file is not locked when LockFile.release() is called.

exception lockfile.NotMyLock

This exception is raised if the file is locked by another thread or process when LockFile.release() is called.

The following classes are provided:

class lockfile.LinkFileLock(path, threaded=True)

This class uses the link(2)() system call as the basic lock mechanism. path is an object in the file system to be locked. It need not exist, but its directory must exist and be writable at the time the acquire() and release() methods are called. threaded is optional, but when set to True locks will be distinguished between threads in the same process.

class lockfile.MkdirFileLock(path, threaded=True)

This class uses the mkdir(2)() system call as the basic lock mechanism. The parameters have the same meaning as for the LinkFileLock class.

class lockfile.SQLiteFileLock(path, threaded=True)

This class uses the sqlite3 module to implement the lock mechanism. The parameters have the same meaning as for the LinkFileLock class.

By default, the FileLock object refers to the MkdirFileLock class on Windows. On all other platforms it refers to the LinkFileLock class.

When locking a file the LinkFileLock class creates a uniquely named hard link to an empty lock file. That hard link contains the hostname, process id, and if locks between threads are distinguished, the thread identifier. For example, if you want to lock access to a file named “README”, the lock file is named “README.lock”. With per-thread locks enabled the hard link is named HOSTNAME-THREADID-PID. With only per-process locks enabled the hard link is named HOSTNAME–PID.

When using the MkdirFileLock class the lock file is a directory. Referring to the example above, README.lock will be a directory and HOSTNAME-THREADID-PID will be an empty file within that directory.

See also

Module msvcrt

Provides the locking() function, the standard Windows way of locking (parts of) a file.

Module posixfile

The deprecated (since Python 1.5) way of locking files on Posix systems.

Module fcntl

Provides the current best way to lock files on Unix systems (lockf() and flock()).

Implementing Other Locking Schemes

There is a LockBase base class which can be used as the foundation for other locking schemes. For example, if shared filesystems are not available, LockBase could be subclassed to provide locking via an SQL database.

FileLock Objects

FileLock objects support the context manager protocol used by the statement:with statement. The timeout option is not supported when used in this fashion. While support for timeouts could be implemented, there is no support for handling the eventual Timeout exceptions raised by the __enter__() method, so you would have to protect the with statement with a try statement. The resulting construct would not be much simpler than just using a try statement in the first place.

FileLock has the following user-visible methods:

FileLock.acquire(timeout=None)

Lock the file associated with the FileLock object. If the timeout is omitted or None the caller will block until the file is unlocked by the object currently holding the lock. If the timeout is zero or a negative number the AlreadyLocked exception will be raised if the file is currently locked by another process or thread. If the timeout is positive, the caller will block for that many seconds waiting for the lock to be released. If the lock is not released within that period the LockTimeout exception will be raised.

FileLock.release()

Unlock the file associated with the FileLock object. If the file is not currently locked, the NotLocked exception is raised. If the file is locked by another thread or process the NotMyLock exception is raised.

lockfile.is_locked()

Return the status of the lock on the current file. If any process or thread (including the current one) is locking the file, True is returned, otherwise False is returned.

lockfile.break_lock()

If the file is currently locked, break it.

Examples

This example is the “hello world” for the lockfile module:

lock = FileLock("/some/file/or/other")
with lock:
    print lock.path, 'is locked.'

To use this with Python 2.4, you can execute:

lock = FileLock("/some/file/or/other")
lock.acquire()
print lock.path, 'is locked.'
lock.release()

If you don’t want to wait forever, you might try:

lock = FileLock("/some/file/or/other")
while not lock.i_am_locking():
    try:
        lock.acquire(timeout=60)    # wait up to 60 seconds
    except LockTimeout:
        lock.break_lock()
        lock.acquire()
print "I locked", lock.path
lock.release()

Other Libraries

The idea of implementing advisory locking with a standard API is not new with lockfile. There are a number of other libraries available:

• locknix - http://pypi.python.org/pypi/locknix - Unix only
• mx.MiscLockFile - from Marc André Lemburg, part of the mx.Base distribution - cross-platform.
• Twisted - http://twistedmatrix.com/trac/browser/trunk/twisted/python/lockfile.py
• zc.lockfile - http://pypi.python.org/pypi/zc.lockfile

Contacting the Author

If you encounter any problems with lockfile, would like help or want to submit a patch, contact me directly: Skip Montanaro (skip@pobox.com).