Debugging Information

UDB uses debugging information to interpret the memory and CPU state of your process and present a symbolic view of your application. The most commonly used debugging data format for compiled Linux executables is the DWARF standard.

Separate debug information files

The debug information can be stored inside the executable or libraries themselves or in separate debug files. UDB and LiveRecorder look for debug information files and save them into the Undo recording so that they are available when replaying with UDB regardless of which machine the recording is loaded on.

Custom debug file directories

By default, UDB and LiveRecorder will look for debug information files in the standard /usr/lib/debug directory. Custom debug file directories can be specified in LiveRecorder by setting the UNDO_debug_file_directories environment variable, and in UDB the by setting debug-file-directory, to a colon-separated list of paths (which in UDB must contain /usr/lib/debug if you still want to search this).

Supported schemes for separate debug files

Being able to save debug information files depends on the files being stored and referenced in a standard way. There are several ways to do this, and UDB and LiveRecorder aims to support all of them.

Build ID

This debug information scheme embeds a unique Build ID hash in the executable or shared object. This is used to look up the corresponding debug information file. Build ID files are stored in the standard location /usr/lib/debug/.build-id, with filenames corresponding to their Build ID.

The Build ID uniquely identifies each executable or shared object. It is generated by the linker program and is stored in a .note.gnu.build-id ELF section in the executable. The readelf program can be used to inspect the Build ID of an executable or shared object:

readelf -n /path/to/executable

For each of the Custom debug file directories configured UDB and LiveRecorder will look in the .build-id subdirectory for a file in a directory with the first two hexadecimal characters of the Build ID, and a file with the rest of the Build ID with a .debug extension.

For example, if the Build ID is 039f6f2f91837cfe6b31187b38ca1113a50c4aea and the Custom debug file directories are configured as the value /home/user/debuginfo" then UDB and LiveRecorder will look for:

  • /usr/lib/debug/.build-id/03/9f6f2f91837cfe6b31187b38ca1113a50c4aea.debug

  • /home/user/debuginfo/.build-id/03/9f6f2f91837cfe6b31187b38ca1113a50c4aea.debug

Split DWARF

Some compilers support a feature called “split DWARF” which separates as much debug information as possible into DWARF object (DWO) files with .dwo extensions. In contrast to the Debuglink debug information scheme, where the reference is created after the linking step, and the Build ID debug information scheme, where the reference is created during the linking step, the DWO files and references are created during the compilation step.

DWO files can be created by passing the -gsplit-dwarf option to GCC or Clang. For more information see Reduce binary size and compile time with split DWARF.

Some DWARF information is retained in the original executable, including the DW_AT_dwo_name and DW_AT_dwo_id attributes which are used to look up the separate DWO files. If relative path are used they are appended to the compilation directory which stored in the DW_AT_comp_dir attribute.

These paths are searched for under the custom debug file directories. For example, if DW_AT_dwo_name is /home/build/ls.dwo and the Custom debug file directories are configured as the value /home/user/debuginfo", then UDB and LiveRecorder will look for:

  • /home/build/ls.dwo

  • /user/lib/debug/home/build/ls.dwo

  • /home/user/debuginfo/home/build/ls.dwo

  • /user/lib/debug/ls.dwo

  • /home/user/debuginfo/ls.dwo

With a DW_AT_dwo_name of build/ls.dwo, DWO_AT_comp_dir of /home/comp and the Custom debug file directories are configured as the value /home/user/debuginfo", UDB and LiveRecorder will look for:

  • /home/comp/build/ls.dwo

  • /usr/lib/debug/build/ls.dwo

  • /home/user/debuginfo/build/ls.dwo

  • /user/lib/debug/ls.dwo

  • /home/user/debuginfo/ls.dwo

DWARF package files

When using the “Split DWARF” approach, the compiler generates a DWO file for each compilation unit. The “DWARF package file” is an approach where multiple DWO files are combined into a single DWARF package (DWP) file for each executable. UDB and LiveRecorder will save DWP files that are stored in the same directory as the original executable or in one of the Custom debug file directories, and have the same name as the original executable plus a .dwp extension.

Unsupported schemes

UDB and LiveRecorder do not currently support the DWARF-5 debug information scheme, where a .debug_sup ELF section is used to refer to the debug information file.