Asynchronous I/O

Asynchronous I/O is a method of performing I/O in which the application issues requests to the kernel (for example, to write some data into a file) and at some point later the application receives an event from the kernel to indicate the result of the request.

Debugging programs using asynchronous I/O with UDB

To debug a program using asynchronous I/O, start it under UDB in the same way as any other program:

$ udb --args examples/aio /dev/zero
Reading symbols from examples/aio...
not running> start
Temporary breakpoint 1 at 0x126b: file aio.c, line 29.
Starting program: examples/aio /dev/zero

Temporary breakpoint 1, main (argc=2, argv=0x7fffffffd6e8) at aio.c:29
29          if (argc != 2)

UDB inspects the program and automatically loads the asynchronous I/O preload library if it detects that libaio is in use. If the program does not link against libaio (because it uses the system calls directly, or because it loads the library dynamically), asynchronous I/O can be forced on or off using the --async-io command-line option to udb, or the UNDO_enable_async_io environment variable. These take the following values:

  • true — always enable support for asynchronous I/O when launching the program, and refuse to attach to a process that does not have the asynchronous I/O library preloaded.

  • false — never enable support for asynchronous I/O when launching the program, and always attach to a process regardless of whether it has the asynchronous I/O library preloaded.

    Note that if this mode is selected but the program uses asynchronous I/O, then UDB will not be able to correctly record or replay the program. In this case it displays a warning, for example:

    $ udb --async-io false --args examples/aio /dev/zero
Reading symbols from examples/aio...
Asynchronous I/O was disabled by user, but the environment or libraries for
the debugged process suggest it may be needed.
Continuing anyway.

  • auto (the default) — automatically detect whether the program links against libaio.so, and select true or false accordingly.

Attaching to programs using asynchronous I/O

When the Undo Engine attaches to a process that is using asynchronous I/O, there may be asynchronous I/O operations in flight. In order for the Undo Engine to correctly record and replay these operations, the process must have been using the asynchronous I/O preload library. This is a shared library that can be linked into a program at run-time using the LD_PRELOAD environment variable.

The correct preload library depends on the architecture for which your program was compiled:

OS

Architecture

Preload library

Linux

Intel i386

libundodb_aio_preload_x32.so

Linux

AMD/Intel x86-64

libundodb_aio_preload_x64.so

Warning

The preload library doesn’t support raw asynchronous I/O system calls (i.e. performed with native assembly instructions). It is however supported to use the C library syscall() function to perform asynchronous I/O system calls.

When the Undo Engine attaches to a process that is linked against libaio.so but is not using the preload library, it prints the following warning:

$ udb
not running> attach 2415104
Have reached start of recorded history.
0x00007fbeae96d03b in __GI___libc_read (fd=0, buf=0x560437a086b0, nbytes=1024)
    at ../sysdeps/unix/sysv/linux/read.c:26
26      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
UDB has detected that libaio.so is in use.
To debug calls to the Linux asynchronous I/O API ("async I/O" or "aio") you
must add UDB's preload library to the LD_PRELOAD environment variable of the
program you are debugging.
For instance:
    $ LD_PRELOAD=libundodb_aio_preload_x64.so \
        YOUR-PROGRAM [ARGUMENTS] &

In this situation the Undo Engine cannot record or replay the program:

recording 1> continue
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
Forward execution cannot continue from here.
because: The debuggee attempted to execute an asynchronous I/O system call,
passing a context that was not tracked by the Undo Engine.
because: Asynchronous I/O (in this case, the "io_getevents" system call) is
not supported.
Programs using asynchronous I/O must use the asynchronous I/O preload library
to avoid this. See https://docs.undo.io/AsynchronousIO.html
syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38      ../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory.

The solution is to use the asynchronous I/O preload library:

$ LD_PRELOAD=./libundodb_aio_preload_x64.so examples/aio /dev/zero &
pid=2415165: press any key to wait for read of 4096 bytes from fd 3...
$ udb
not running> attach 2415165
Have reached start of recorded history.
0x00007f5f2871703b in __GI___libc_read (fd=0, buf=0x55cfe1f3c730, nbytes=1024)
    at ../sysdeps/unix/sysv/linux/read.c:26
26      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
recording 1> continue
Continuing.

Program received signal SIGSTOP, Stopped (signal).

The program has exited, but is still being debugged.
You can use UDB reverse commands to go backwards; see "help udb" for details.

__GI__exit (status=0) at ../sysdeps/unix/sysv/linux/_exit.c:30
30      ../sysdeps/unix/sysv/linux/_exit.c: No such file or directory.

Asynchronous I/O in LiveRecorder

Currently, LiveRecorder does not support the checks or parameters used by UDB to enable or disable asynchronous I/O. To record a program that uses asynchronous I/O operations with LiveRecorder, the user must specify the preload library in the LiveRecorder environment. For example:

$ LD_PRELOAD=./libundodb_aio_preload_x64.so live-record examples/aio /dev/zero &
live-record: Termination recording will be written to aio-2415228-2024-02-15T18-01-28.494.undo
live-record: Maximum event log size is 1G.
pid=2415228: press any key to wait for read of 4096 bytes from fd 7...

live-record: Saving to 'aio-2415228-2024-02-15T18-01-28.494.undo'...
live-record: Termination recording written to aio-2415228-2024-02-15T18-01-28.494.undo
live-record: Detaching...

Attaching to a program with LiveRecorder is a similar to attaching to the program in UDB:

$ LD_PRELOAD=./libundodb_aio_preload_x64.so examples/aio /dev/zero &
pid=2415294: press any key to wait for read of 4096 bytes from fd 3...
$ live-record --pid 2415294 -o aio.undo &
live-record: Maximum event log size is 1G.

live-record: Saving to 'aio.undo'...
live-record: Termination recording written to aio.undo
live-record: Detaching...
$ udb
not running> uload aio.undo
0x00007f606191703b in __GI___libc_read (fd=0, buf=0x55805284b730, nbytes=1024)
    at ../sysdeps/unix/sysv/linux/read.c:26
26      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.

The debugged program is at the beginning of recorded history. Start debugging
from here or, to proceed towards the end, use:
    continue - to replay from the beginning
    ugo end  - to jump straight to the end of history

Buffers with incorrect permissions

As part of its operation the preload library accesses asynchronous I/O buffers. If your program provides asynchronous I/O buffers with incorrect permissions (for example, providing a PROT_NONE buffer) then you would normally receive an error from the kernel (generally EFAULT). However since the preload library accesses the asynchronous I/O buffers directly it is likely to experience a SIGSEGV in these cases. If you experience a crash from the preload library it may therefore indicate the buffer permissions are incorrect.