Add fallback implementation to blockio used when platform isn't Linux, MacOS and FreeBSD by mwihoti · Pull Request #886 · IntersectMBO/lsm-tree

mwihoti · 2026-09-17T06:53:01Z

Description

Resolves #872.

I added a generic portable fallback: serial HasBlockIO in src-unix, which gets picked when the platform isn't Linux, MacOS, Windows or FreeBSD. I also introduced a posixblockio cabal flag that forces this fallback on any platform, allowing CI to build and test it on Linux without needing extra runners.

Instead of reusing the macOS backend (as #847 did, which assumes F_NOCACHE), I chose the dedicated fallback route for Unix. The generic POSIX layer cannot assume that O_DIRECT, F_NOCACHE or posix_fallocate exist, and unix's fileAllocate and fileSetCaching throw where they're missing. Therefore I guarded those features using the HAVE_* macros from HsUnix.h (the same include used by the FreeBSD backend) so they gracefully degrade to no-ops, similar to how the macOS backend already handles hAllocate. Both modules are .hsc rather than .hs because GHC's cpp can't process HsUnix.h (it pulls in glibc headers) while hsc2hs can the same reason the FreeBSD backend's Fcntl is .hsc.

Finally, I added a +posixblockio entry to the CI matrix alongside +serialblockio, and updated the platform notes in IO.hs, the README and the changelog.

Tested locally: the default Linux build is unchanged, cabal build blockio --flag=+posixblockio compiles the fallback, and cabal test blockio:test --flag=+posixblockio passes all 11 tests against it. One limitation: on Linux all the HAVE_* macros are true, so CI only compiles the "supported" branches; the #else no-ops are only compiled on a platform that actually lacks the syscalls. Running --flag=+posixblockio on FreeBSD would be a useful second data point.

Checklist

Read our contribution guidelines at CONTRIBUTING.md, and make sure that this PR complies with the guidelines.

jorisdral

I think the plan you outline in the PR description is sensible! But something may have gone wrong with your commits, because I'm missing code changes.

P.S.: no need to post screenshots of your console with test output. CI will verify that the tests succeed.

jorisdral · 2026-09-22T08:49:54Z

+* Add a portable serial `HasBlockIO` implementation for POSIX systems that do
+  not have a dedicated one, so `blockio` builds on more than Linux, MacOS,
+  Windows, and FreeBSD. Operations the platform does not support
+  (`posix_fallocate`, `O_DIRECT`/`F_NOCACHE`) degrade to no-ops. See [issue
+  #872][issue-872] and [PR #NNN][pr-NNN].
+* Add the `posixblockio` cabal flag, which forces the portable implementation
+  on any platform, for testing it. See [PR #NNN][pr-NNN].


Fill in the [PR #NNN][pr-NNN] links

I don't see the changes yet

jorisdral · 2026-09-23T07:39:11Z

+* Add a portable serial `HasBlockIO` implementation for POSIX systems that do
+  not have a dedicated one, so `blockio` builds on more than Linux, MacOS,
+  Windows, and FreeBSD. Operations the platform does not support
+  (`posix_fallocate`, `O_DIRECT`/`F_NOCACHE`) degrade to no-ops. See [issue
+  #872][issue-872] and [PR #NNN][pr-NNN].
+* Add the `posixblockio` cabal flag, which forces the portable implementation
+  on any platform, for testing it. See [PR #NNN][pr-NNN].


I don't see the changes yet

jorisdral · 2026-09-23T07:42:54Z

+* Add a portable serial `HasBlockIO` implementation for POSIX systems that do
+  not have a dedicated one, so `blockio` builds on more than Linux, MacOS,
+  Windows, and FreeBSD. Operations the platform does not support
+  (`posix_fallocate`, `O_DIRECT`/`F_NOCACHE`) degrade to no-ops. See [issue
+  #872][issue-872] and [PR #NNN][pr-NNN].
+* Add the `posixblockio` cabal flag, which forces the portable implementation
+  on any platform, for testing it. See [PR #NNN][pr-NNN].


I'm unsure about using the name "POSIX" to describe the fallback implementation. Posix is a standard, not an operating system. I think we should call it something like "unix-like", or maybe even "fallback". Do you have thoughts on this?

unix-like fits

jorisdral · 2026-09-23T07:50:33Z

+as well. On other POSIX systems a portable serial implementation is used that
+relies only on the parts of the `unix` package available on the platform; it
+can be selected explicitly on any platform with the `posixblockio` cabal flag.
+The simulated implementation also performs each I/O operation sequentially.


Suggested change

as well. On other POSIX systems a portable serial implementation is used that

relies only on the parts of the `unix` package available on the platform; it

can be selected explicitly on any platform with the `posixblockio` cabal flag.

The simulated implementation also performs each I/O operation sequentially.

as well. If the operating system is none of the above, then the system is assumed to be Unix-like, in which case the library uses a fallback implementation that performs I/O operations sequentially.

The simulated implementation also performs each I/O operation sequentially.

jorisdral · 2026-09-23T07:55:04Z

+* Add a portable serial `HasBlockIO` implementation for POSIX systems that do
+  not have a dedicated one, so `blockio` builds on more than Linux, MacOS,
+  Windows, and FreeBSD. Operations the platform does not support
+  (`posix_fallocate`, `O_DIRECT`/`F_NOCACHE`) degrade to no-ops. See [issue
+  #872][issue-872] and [PR #NNN][pr-NNN].


This should also emphasize that the OS is now assumed to be Unix-like in case the OS is not one of the explicitly supported OS's (Linux, MacOS, Windows, FreeBSD). This is different from the previous behaviour, where the library would simply be unbuildable on unsupported OS's.

jorisdral · 2026-09-23T07:56:59Z

+flag posixblockio
+  description:
+    Use the portable POSIX HasBlockIO implementation regardless of the
+    operating system. Intended for testing that implementation on platforms
+    that have a dedicated one. Not supported on Windows.
+
+  default:     False
+  manual:      True


Suggested change

flag posixblockio

description:

Use the portable POSIX HasBlockIO implementation regardless of the

operating system. Intended for testing that implementation on platforms

that have a dedicated one. Not supported on Windows.

default: False

manual: True

flag unixlikeblockio

description:

Use the Unix-like HasBlockIO implementation regardless of the

operating system. Intended for testing that implementation on platforms

that have a dedicated one. No-op on Windows.

default: False

manual: True

jorisdral · 2026-09-23T08:22:33Z

You probably copied this module from another System.FS.BlockIO.Internal.Fcntl, but it's no longer necessary. This type of module is only necessary to support unix versions <unix-2.8.7, but we support >=unix-2.8.7.

jorisdral · 2026-09-23T08:22:56Z

+  'submitIO', just like the MacOS, Windows, FreeBSD, and portable POSIX
+  implementations do.


Suggested change

'submitIO', just like the MacOS, Windows, FreeBSD, and portable POSIX

implementations do.

'submitIO', just like the MacOS, Windows, FreeBSD, and Unix-like

implementations do.

jorisdral · 2026-09-23T08:23:13Z

    * MacOS: a mocked context using an @MVar@
    * Windows: a mocked context using an @MVar@
    * FreeBSD: a mocked context using an @MVar@
+    * Other POSIX systems: a mocked context using an @MVar@


We should update this everywhere:

Suggested change

* Other POSIX systems: a mocked context using an @MVar@

* Unix-like systems: a mocked context using an @MVar@

yes I have done updates

jorisdral · 2026-09-23T08:24:26Z

+-- | A portable, serial implementation of 'HasBlockIO' for POSIX systems that
+-- do not have a dedicated implementation (Linux, MacOS, and FreeBSD do). It
+-- uses only the parts of the @unix@ package that are available on the
+-- platform, and degrades to no-ops where an operation is not supported.


Suggested change

-- | A portable, serial implementation of 'HasBlockIO' for POSIX systems that

-- do not have a dedicated implementation (Linux, MacOS, and FreeBSD do). It

-- uses only the parts of the @unix@ package that are available on the

-- platform, and degrades to no-ops where an operation is not supported.

-- | A portable, serial implementation of 'HasBlockIO' for Unix-like systems that

-- do not have a dedicated implementation (Linux, MacOS, and FreeBSD do).

jorisdral · 2026-09-23T08:30:18Z

+hSynchronise :: Handle HandleIO -> IO ()
+hSynchronise h = FS.withOpenHandle "hSynchronise" (handleRaw h) $ \fd ->
+    Unix.fileSynchronise fd


Looks to me like fileSynchronise can also throw an error if the system does not HAVE_FSYNC. We should also make it a no-op in that case, like we do with hAllocate. We should also update the docs accordingly.

Linux, MacOS, Windows, and FreeBSD each have a dedicated HasBlockIO implementation, but every other platform couldn't build blockio at all. This adds a serial fallback in src-unix, used when the platform isn't one of those four. It relies only on the parts of the unix package that are actually available, and degrades to no-ops where the platform doesn't support an operation (O_DIRECT/F_NOCACHE, posix_fallocate, fsync). Also adds the unixlikeblockio cabal flag, which forces the fallback on any platform, so CI can build and test it on Linux without needing dedicated runners for the other platforms. Resolves IntersectMBO#872.

mwihoti requested review from dcoutts, jorisdral and mheinzel as code owners September 17, 2026 06:53

jorisdral requested changes Sep 22, 2026

View reviewed changes

mwihoti force-pushed the feature/872-posix-fallback branch from 59e4238 to e9b447d Compare September 23, 2026 05:45

jorisdral requested changes Sep 23, 2026

View reviewed changes

code-orange-dev mentioned this pull request Sep 26, 2026

Full re-verification: 96 → 131 merged, verifier tool + weekly CI code-orange-dev/PR-tracking-dashboard#2

Merged

mwihoti force-pushed the feature/872-posix-fallback branch from e9b447d to 03b89f0 Compare September 30, 2026 05:49

mwihoti requested a review from jorisdral September 30, 2026 05:59

Run cabal-fmt on blockio.cabal

6937057

mwihoti changed the title ~~Added fallback implementation to blockio used when platform isn't Linux, MacOS and FreeBSD~~ Add fallback implementation to blockio used when platform isn't Linux, MacOS and FreeBSD Oct 6, 2026

		'submitIO', just like the MacOS, Windows, FreeBSD, and portable POSIX
		implementations do.

	* Other POSIX systems: a mocked context using an @MVar@
	* Unix-like systems: a mocked context using an @MVar@

Conversation

mwihoti commented Sep 17, 2026

Description

Checklist

Uh oh!

jorisdral left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants