API

dreadlocks Package

The dreadlocks module exposes three context manager functions: path_lock, process_level_path_lock, and thread_level_path_lock.

Exception classes are part of the public API. Other exported functions are implementation details subject to change.

Caveats:

If you need to lock multiple paths you should have a total order for each subset of paths that could be locked simultaneously by the same thread and call dreadlocks.path_lock() multiple times respecting this total order. For instance via resolved absolute path lexicographical order.

The Linux implementation relies on fcntl/lockf which is infamous for being hard to work with: if you close any file descriptor for a given file in a given process, it will release the lock held by that process, even if the lock was acquired through a different file descriptor. So you cannot easily open a lock file for reading or writing without breaking the locking mechanism. We implement locks in a way that uses a single fd while a lock exists for a given (normalized) path. That fd is opened with O_RDWR and is yielded by dreadlocks.path_lock() for convenience, but the user should take extra care not to close this fd, as it would release the lock. For instance, open must be used with the closefd flag as in open(fd, closefd=False). This makes writing to the lock file, or reading several times from the lock file, a bit challenging but not impossible (using fp.seek and fp.truncate).

Another solution would be to use flock but since that can sometimes fallback to the fcntl/lockf implementation, we prefer to use the latter implementation directly, and workaround the limitations. See http://0pointer.de/blog/projects/locking.html.

Currently, on Windows, at the process level, shared locks are “simulated” by exclusive locks. Implementing true shared locks on this platform would require depending on userland libraries. This would be necessary, for instance, in situations where multiple processes need to lock shared resources and communicate simultaneously about the state of those resources. Note that the problem in this particular example can be circumvented at the cost of efficiency, by simulating locked shared resources through multiple rounds of exclusively-locked resources plus communication. Currently we conclude that, it is enough for the implementation to be simple for Windows, and correct on UNIX.

The current implementation may not even be correct on UNIX in the scenario discussed above: in a multi-process, multi-thread setting, if one thread of process A acquires an exlusive lock at the thread-level first, then attempts to exclusively acquire the process-level lock, that process-level lock may already be acquired as shared by a thread of another process B that is waiting on the other threads of A to run. And vice versa, if one thread of process A acquires an exclusive lock at the process-level first, then attempts to exclusively acquire the thread-level lock, that thread-level lock may already be acquired as shared by other threads of process A, but they are waiting on a thread of process B that is in turn waiting for the process-level lock to be downgraded to shared. So it seems that composing two isolated implementations, one for threads, one for processes, cannot work in these “share and communicate” scenarios. One solution is to forbid upgrade and downgrade of a process-level lock which would effectively be the same as having an all-or-nothing lock (either you lock at both levels or you do not).

Currently, on Windows, at the process level, we attempt to lock the entire file by passing the largest possible value to mscvrt.locking nbytes argument. If we want this implementation to be correct, we would need to fo.seek(0) before calling mscvrt.locking then perhaps restore the seek position? Also the largest possible value is 2^31-1 which might not be enough for files larger than 2GB. Again, we do not currently need this “functionality”.

It is currently not possible to specify a timeout for acquiring a lock. The only options currently are immediate failure or forever blocking.

Functions

path_lock(path[, shared, blocking, reentrant])	Locks a path both at the thread-level and process-level.
process_level_path_lock(path[, shared, ...])	Locks a path at the process-level.
thread_level_path_lock(path[, shared, ...])	Locks a key at the thread-level.

Classes

AcquiringLockWouldBlockError	Raised when acquiring lock would block and blocking is False
AcquiringProcessLevelLockWouldBlockError	Raised when acquiring a process-level lock would block and blocking is False
AcquiringThreadLevelLockWouldBlockError	Raised when acquiring a thread-level lock would block and blocking is False
RecursiveDeadlockError	Raised when recursive dead-lock is detected.

Class Inheritance Diagram