Uring

I released uring 2.14.0 with the recent changes:

• Prepare release #140.
• [new release] uring (2.14.0) #29954.

I also continued looking at using uring flags for better Eio performance. First, Eio now tells uring that each ring has a single submitting thread:

• eio_linux: use IORING_SETUP_SINGLE_ISSUER #841.

This allows uring to skip some locking apparently, but it didn't make any noticable difference to me. However, it does mean you get a better error if you accidentally try to submit from two threads, which can happen if something calls Unix.fork (eio/issues/801). Except that Uring.submit failed to report errors:

• Report errors from io_uring_submit #144.

Last week I tried using IORING_SETUP_COOP_TASKRUN and IORING_SETUP_DEFER_TASKRUN, but noted that DEFER caused a test to hang. The test has a fiber that's always runnable (and keeps yielding) and is checking that IO still gets handled. With DEFER, Eio sees that there are no completions ready and so just gets on with the CPU-intensive fiber. I didn't see this problem with COOP mode, but I'm not sure why not because we're not doing any syscalls.

In any case, the fix turned out to be simple: uring has a IORING_SETUP_TASKRUN_FLAG which tells the kernel to set a flag in the shared memory if there are completion tasks that need to run, and io_uring_peek_cqe checks this flag and enters the kernel automatically if needed. So, Eio is now using DEFER mode:

• eio_linux: use "IORING_SETUP_DEFER_TASKRUN" for performance #842.

I'd be interested to know if anyone else sees a speed-up with this.

According to io_uring_setup_flags(7), zero-copy receive and ring resizing also require DEFER mode.

PR reviews

I reviewed some more PRs (from Patrick, Anil and Mark):

• COMMENTED Incremental directory reading #821.
  Greatly improves the speed of scanning directories by returning the type of each entry as part of the read (saving one stat per file) and by allowing you to process the listing incrementally.

• Support chown in eio_posix and eio_linux #781.

• docker: add ?os to Peek and Pull (default linux) #472.

• Statx attribute supported check was inverted #142.
  Uring.Statx.Attr.check had a logic error, but now I'm wondering if it's even useful.

• Fix dio_offset_align statx getter returning wrong field #141.

MDX

I updated my MDX hang-detector (#476) to report accurate locations for cram tests too. The CI was failing, which turned out to be due to a recent change in dune:

• MDX CI is failing #477.

Minor PRs

I updated uring to use MSG_CMSG_CLOEXEC when receiving file descriptors, and this week I made the equivalent change in eio_posix (for other systems):

• eio_posix: use MSG_CMSG_CLOEXEC when receiving file descriptors #843.

Disappointingly, it turns out that macos doesn't support this flag and I had to add a work-around for that.

The Eio benchmarks use a Dockerfile for the tests and it hadn't been updated for ages. I was hoping that using newer dependencies might show a speed improvement, but it didn't help:

• Update Dockerfiles to recent versions #844.

tar-eio

I had a brief look at improving the eio-tar API, which has a number of problems currently:

• It requires the source tar file to be opened in read/write mode, even for extracting.
• It exposes the confusing Tar.t type that's used by the core to abstract over monads and effects, which Eio users shouldn't need to care about.
• As a consequence, Tar_eio.extract needs to take a switch argument because it doesn't actually do the extraction, but just returns a computation that runs later, and so needs to hold on to the argument.

Patrick explained to me that extract needs to return a computation so that you can wrap it with Tar_gz.in_gzipped. I think it would make a lot more sense to wrap the source file with a gzip decoder, rather than wrap the computation, but changing it all looked like a fair bit of work and I didn't go any further.