Debugging/Getting traces - ArchWiki

Jump to content
< Debugging

Related articles

- General troubleshooting
- Reporting bug guidelines
- Debuginfod
- Core dump

This article aims to help in creating a debugging Arch package and using it to provide trace and debug information for reporting software bugs to developers.

Usually, executable files are stripped of human readable context to make them smaller. Not only that, enhanced debugging information is usually not added to the executable in the first place, which drastically reduces the quality of the trace. So, before getting traces with debug information, one has to rebuild the package without stripping and with debugging information.

Use the complete stack trace to inform developers of a bug you have discovered before. This will be highly appreciated by them and will help to improve your favorite program.

## Package names

This article or section needs expansion.

Reason: coredumpctl can be especially useful here, compare Core_dump#Cleanup_of_core_dump_files (Discuss in Talk:Debugging/Getting traces)

The first thing to do is to obtain the names of the packages which require rebuilding.

When looking at debug messages, such as by using gdb on a core dump:

    [...]
    Backtrace was generated from '/usr/bin/epiphany'

    (no debugging symbols found)
    Using host libthread_db library "/lib/libthread_db.so.1".
    (no debugging symbols found)
    [Thread debugging using libthread_db enabled]
    [New Thread -1241265952 (LWP 12630)]
    (no debugging symbols found)
    0xb7f25410 in __kernel_vsyscall ()
    #0  0xb7f25410 in __kernel_vsyscall ()
    #1  0xb741b45b in ?? () from /lib/libpthread.so.0
    [...]

`??` shows where debugging info is missing, as well as the name of library or executable which called the function. Similarly, when `(no debugging symbols found)` appears, you should look for the stated file names. For example, with pacman:

    $ pacman -Qo /lib/libthread_db.so.1

    /lib/libthread_db.so.1 is owned by glibc 2.5-8

The package is called glibc in version 2.5-8. Repeat this step for every package that needs debugging.

To retrieve debug information, proceed with the #Debuginfod, #Install debug packages or #Rebuild packages section.

## Debuginfod

For packages in the official repositories that support it [1], debug information can be retrieved directly over HTTP with debuginfod.

If one wants to retrieve the debug symbols for zstd, along with some source files, one can utilize debuginfod-find:

    $ debuginfod-find debuginfo /usr/bin/zstd

    /home/user/.cache/debuginfod_client/70e1b456c5813658df6436a3deb71812e75a0267/debuginfo

    $ debuginfod-find source /usr/bin/zstd /usr/src/debug/zstd-1.5.2/programs/fileio.c

    /home/user/.cache/debuginfod_client/70e1b456c5813658df6436a3deb71812e75a0267/source##usr##src##debug##zstd-1.5.2##programs##fileio.c

Debuggers like gdb can download them automatically.

You can do for example:

    # coredumpctl gdb

And then inside gdb

    (gdb) bt full

and you will have the debug symbols of the last crashed application in your system.

## Install debug packages

This article or section needs expansion.

Reason: Explain the different scenarios properly. Either you install a debug package or you let debuginfod fetch the things it needs? (Discuss in Talk:Debugging/Getting traces)
Note: Debug packages are not archived by Arch Linux on Arch Linux Archive.

A few mirrors currently distribute debug packages in accessible repositories. These are sponsored mirrors controlled by Arch Linux and are given access to the debug repositories.

- https://geo.mirror.pkgbuild.com (GeoDNS mirror)

To install a package you can install it directly from the repository. For example:

    # pacman -U https://geo.mirror.pkgbuild.com/core-debug/os/x86_64/zstd-debug-1.5.2-2-x86_64.pkg.tar.zst

Warning: Debug packages from one mirror are not compatible with regular packages from another mirror, if both mirrors are not in sync, and thus have mismatching builds. In this case, avoid mixing packages from different mirrors (this would result in a partial upgrade), but point all repositories to the `debug` mirror.

This article or section is a candidate for merging with Official repositories.

Notes: Official repositories have a dedicated page. (Discuss in Talk:Debugging/Getting traces)

Another option is to add the repositories to your pacman configuration.

    /etc/pacman.conf

    # Testing Repositories

    [testing-debug]
    Include = /etc/pacman.d/mirrorlist

    [community-testing-debug]
    Include = /etc/pacman.d/mirrorlist

    [multilib-testing-debug]
    Include = /etc/pacman.d/mirrorlist

    # Stable repositories

    [core-debug]
    Include = /etc/pacman.d/mirrorlist

    [extra-debug]
    Include = /etc/pacman.d/mirrorlist

    [community-debug]
    Include = /etc/pacman.d/mirrorlist

    [multilib-debug]
    Include = /etc/pacman.d/mirrorlist

Place a mirror with debug packages as the first one in the mirrorlist file:

    /etc/pacman.d/mirrorlist

    Server = https://geo.mirror.pkgbuild.com/$repo/os/$arch
    ...

## Rebuild packages

If debug information is not exposed through debuginfod (for example, when the package originates from the AUR), then it can be rebuilt from source. See ABS for packages in the official repositories, or AUR#Acquire build files for packages in the AUR.

To set the required #Compilation options, you can modify the makepkg configuration if you will only use makepkg for debug purposes. In other cases, you should modify package's `PKGBUILD` file only for each package you would like to rebuild.

### Compilation options

As of pacman 4.1, makepkg.conf(5) has debug compilation flags in `DEBUG_CFLAGS` and `DEBUG_CXXFLAGS`. To use them, enable the `debug` makepkg option, and disable `strip`.

These settings will force compilation with debug symbols and will disable their stripping from executables.

    /etc/makepkg.conf

    OPTIONS+=(debug !strip)

To apply this setting to a single package, modify the `PKGBUILD`:

    PKGBUILD

    options=(debug !strip)

Alternatively you can put the debug information in a separate package by enabling both `debug` and `strip`, debug symbols will then be stripped from the main package and placed, together with source files to aid in stepping through the debugger, in a separate `pkgbase-debug` package. This is advantageous if the package contains very large binaries (e.g. over a GB with debug symbols included) as it might cause freezing and other strange, unwanted behavior occurring.

Note: It is insufficient to simply install the newly compiled debug package, because the debugger will check that the file containing the debug symbols is from the same build as the associated library and executable. You must install both of the recompiled packages. In Arch, the debug symbol files are installed under `/usr/lib/debug/`, and source files are installed under `/usr/src/debug`. See the GDB documentation for more information about debug packages.

#### glibc

Certain packages such as glibc are stripped regardless. Check the `PKGBUILD` for sections such as:

    strip $STRIP_BINARIES usr/bin/{gencat,getconf,getent,iconv,iconvconfig} \
                          usr/bin/{ldconfig,locale,localedef,nscd,makedb} \
                          usr/bin/{pcprofiledump,pldd,rpcgen,sln,sprof} \
                          usr/lib/getconf/*

    strip $STRIP_STATIC usr/lib/*.a

    strip $STRIP_SHARED usr/lib/{libanl,libBrokenLocale,libcidn,libcrypt}-*.so \
                        usr/lib/libnss_{compat,db,dns,files,hesiod,nis,nisplus}-*.so \
                        usr/lib/{libdl,libm,libnsl,libresolv,librt,libutil}-*.so \
                        usr/lib/{libmemusage,libpcprofile,libSegFault}.so \
                        usr/lib/{audit,gconv}/*.so

And remove them where appropriate.

#### Clang

Packages using Clang as the compiler will not build with the `debug` option due to the debug flag `-fvar-tracking-assignments'` not being handled (e.g. js78).

Add the following at the top of the `build()` function to only remove the flag for the affected package:

    build() {
      CFLAGS=${CFLAGS/-fvar-tracking-assignments}
      CXXFLAGS=${CXXFLAGS/-fvar-tracking-assignments}
    [...]

### Building and installing the package

Build the package from source using `makepkg` while in the `PKGBUILD`'s directory. This could take some time:

    $ makepkg

Then install the built package:

    # pacman -U glibc-2.26-1-x86_64.pkg.tar.gz

## Getting the trace

This article or section needs expansion.

Reason: Core dumps stored by systemd can be examined with `coredumpctl gdb` (Discuss in Talk:Debugging/Getting traces)

The actual backtrace (or stack trace) can now be obtained via e.g. gdb, the GNU Debugger. Run it either via:

    # gdb /path/to/file

or:

    # gdb
    (gdb) exec /path/to/file

The path is optional, if already set in the `$PATH` variable.

Then, within `gdb`, type `run` followed by any arguments you wish the program to start with, e.g.:

    (gdb) run --no-daemon --verbose

to start execution of the file. Do whatever necessary to evoke the bug. For the actual log, type the lines:

    (gdb) set logging file trace.log
    (gdb) set logging on

and then:

    (gdb) thread apply all bt full

to output the trace to `trace.log` into the directory `gdb` was started in. To exit, enter:

    (gdb) set logging off
    (gdb) quit

Tip: To debug an application written in Python:

    # gdb /usr/bin/python
    (gdb) run <python application>

You can also debug an already running application, e.g.:

    # gdb --pid=$(pidof firefox)
    (gdb) continue

To debug an application that has already crashed, you will want to invoke `gdb` on its core dump.

## See also

- Debian:HowToGetABacktrace
- Gentoo:Project:Quality Assurance/Backtraces
- Fedora:StackTraces
- GNOME - Getting Stack Traces
- gdb mini intro

Category:

- Package development

Hidden categories:

- Pages or sections flagged with Template:Expansion
- Pages or sections flagged with Template:Merge