"""
Filesystem operations overview

There are many actions that can be done with data on a filesystem. We'll refer to such an action as "filesystem
operation" or operation for short. Example operations: read file, list directory, create directory, create file,
write into file, delete file, delete directory, create or delete symlink or hardlink, move file or folder, copy
file or folder, etc.

The properties of all operations are dependant on the platform and the underlying filesystem. For remote filesystems
even the network can be considered a factor. Generally speaking most filesystems operate on a best-effort principle,
which means many errors should be avoided by the coordination and good-will of user calls. For example: copying two
large different files to the same location simultaneously will not yield an error and the resulting file will be in
an undefined state. Similarly moving two files to a target location can encounter a race condition where one of the
two source files will be lost, because of the always-replacing behaviour of POSIX rename.

We've done some research on securing some operations against race conditions and improving durability. Unfortunately
even for a given platform and specific local filesystem there is often no clear-cut solution for improvement.
All platforms and each of their filesystems have certain advantages and disadvantages. To support similar consistency
and durability on most filesystems we had to analyze how competing concurrent operations behave. One of our early
realizations was that even inside a single type of system the behaviour of competing operations was not satisfactory.
In the end we have chosen to implement a custom filesystem operation serialization in our application layer. This is
the simplest way to mitigate concurrency problems in a cross-platform way. The filesystem operations will inherit the
general properties they are given on the current platform and filesystem.

This custom serialization unifies the behaviour of virtually all filesystems because it avoids competing concurrent
operations. This property does not extend to filesystem operations that originate from outside of our application.
The serialization is only necessary for writing operations to keep consistency and durability, so read-only operations
may execute in parallel.

Additionally we acknowledge that some operations may take a long time, may be interrupted/restarted/retried and for
those a stable queue is required that could also need a persistent storage. Implicitly retrying some operations is
not trivial unfortunately. The "move" of a directory could fall back to a "copy" & "delete" because the source and
destination paths are on different filesystems or even hosts. As such retrying could find a partially copied state
that could be continued, but we cannot be sure what has been copied successfully and what not. So in essence we
have to restart. Further errors that are more dangerous in this process are those that arise when the deleting of the
source is pending. Retrying from that point could lead to losing a well copied file if restarting is implemented with
delete-destination & copy & delete-source. A big discouragement from implementing implicit retry is that discerning
between errors that can be fixed with retries and not is not reliably possible. Insufficient space, permission,
network, access and other errors can possibly be encountered while retrying is only useful or even correct sometimes.
For clear operation to the user all errors of these complex operations shall be propagated as-is without retrying.
While continuation or retry of an operation is too complicated to be reliably done, interruption of a process by user
request is doable. The operation implementations must be invoked as a subprocess so they may be interrupted with
signals.

### Other investigated techniques
All other options that were worth investigating always had some caveat where they did not work at all. The most
sophisticated solution would be to use all of these or rather the best one available. Since these are dependant on
the OS and the filesystem they are not available to provide a general solution.
    - File or parent folder locking:
        - Inconsistent or not supported for network filesystems
        - POSIX systems have advisory file locking, meaning they do not provide more protection than coordinated ops.
        - Locking folders is not supported on many platforms it seems
    - Using EAFP-style API with properly strict arguments:
        - the POSIX IO API has no such functions, some of the required ones only have LBYL-style APIs
        - some UNIX systems (eg. Linux) have improved APIs but these are not available for use as the base POSIX one
        - this would not help with competing operations that have a scope outside of a singe file; for example deleting
          a directory tree would compete with a file creating inside it and the conflict could either make the file
          only to be removed soon or interrupt the deleting altogether
    - LBYL workarounds are insufficient and prone to other errors:
        - rename cannot be replaced by link & unlink for different volumes or filesystems to avoid replacing
        - rename by invoking the mv utility with no-clobber flag does not report if the moving actually took place or
          not; it is not POSIX compliant either, but seems to be supported by several UNIXes

### TOCTOU
Time-Of-Check-to-Time-Of-Use is a vulnerability inherent to the API design of the POSIX file operations. The simplest
example is that by POSIX API one has to check if a destination exists before renaming a source to it, unless they want
to overwrite it. Unfortunately the check and the move operations are distinct and the result of the first one may have
changed when the second is getting started. Such results in unintentionally replacing a file, leading to data loss.
This pre-checking design is often called Look-Before-You-Leap. A different approach is Easier-to-Ask-for-Forgiveness-
than-Permission in which the move operation would explicitly fail if the target already exists. The latter behaviour is
not available in the POSIX API unfortunately. Such variant is available on Linux with the "renameat2" system call.

Every POSIX system is vulnerable ever since '79 and there is no general solution. This operation serialization does not
and cannot solve TOCTOU issues. Numerous research papers were written on the subject and the general consensus is that
a new or improved IO API is necessary for POSIX. There is no sure-to-work mitigation method.
There is much less information about Windows, but it is not exempt of this problem.
For more information:
    https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use
    https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.83.8647
    https://www.usenix.org/legacy/events/fast05/tech/full_papers/wei/wei.pdf

### Different IO queues and workers
The issue we're trying to solve here is not new. Many desktop environments have workers and services that do actual
file operations for the UI applications. The Gnome and KDE desktop environment each has their own utility and their
interoperability (or the lack thereof) has also made related utilities emerge like KIO-GioBridge. It is not perfectly
clear what kind of serialization they do, and they are not above the TOCTOU problem. (KIO was specifically checked)
These solutions are similarly not perfect as ours, since only integrated applications will behave correctly and a
single one that "only" uses the base POSIX API will in fact not coordinate with others as desired.

"""