From SIPfoundry sipx, The Open Source SIP PBX for Linux - Calivia

Using GDB to examine processes or core files resulting from binaries installed from RPMs presents additional challenges because the binaries usually have the symbol information stripped out. In addition, it is useful to be able to capture core files from an operational system and examine them on another system.

Of course, many of the details of these processes will vary from system to system, but the overall structure of the process should as it is described here.

[edit] Debugging information for processes or core files

In most RPMs, the symbol table information has been stripped from the executables and libraries and saved in seperate debugging information files. The debugging information files have names like sipauthproxy.debug. When GDB is examining a process or core file, it automatically uses the debugging information file corresponding to the executable.

To install the debugging information files, install the debug or debuginfo RPMs. These have names like sipxacd-debug-3.6.3-008212.i386.rpm, and can be obtained from where you obtain the sipX RPMs.

The debugging information files are installed into directories below /usr/lib/debug corresponding to the install location of the binaries and libraries. E.g., the debugging information file for /usr/bin/sipregistrar is /usr/lib/debug/usr/bin/sipregistrar.debug.

To get GDB to use the debugging information files, you have to tell it the root of the debugging information files:

set debug-file-directory /usr/lib/debug

[edit] Capturing core files for later debugging

It is often desirable to save a core file from an operational system to be examined with GDB later on another system. This process works much better than rumor has it, but you have to set GDB up correctly.

To capture the core file, the corresponding binary, and the needed dynamic libraries, use sipx-snapshot:

sipx-snapshot --core <core-file-name>

The snapshot file will include the specified core file, its binary, and the dynamic libraries.

Unpack the snapshot file starting at a directory $ROOT. Then use commands like the following to examine the core file:

$ gdb
GNU gdb Red Hat Linux (6.5-15.fc6rh)
(gdb) set solib-absolute-prefix $ROOT
(gdb) file $ROOT/usr/bin/sipregistrar
(gdb) core $ROOT/var/log/sipxpbx/core.1234

However, this won't give GDB access to the debugging information that corresponds to the binary.

[edit] Debugging information for core files from other systems

In order to give GDB access to the debugging information to match a captured core file, the appropriate debugging information files need to be installed on the debugging system, which will run GDB.

One method is to install the debug or debuginfo RPMs on the debugging system. In that case, the root of the debugging files is /usr/lib/debug.

A second method is to install or unpack (using rpm2cpio xxx.rpm | cpio -i --make-directories) the RPMs into some directory, in which case, the root of the debugging files is usr/lib/debug relative to the root of the install or unpack.

In either of these cases, remember that the RPMs need to be the ones corresponding to what is installed on the operational system, not RPMs for the debugging system.

A third method, if the debugging files are already installed on the operational system, is to add --file /usr/lib/debug to the invocation of sipx-snapshot:

sipx-snapshot --core <core-file-name> --file /usr/lib/debug

That option includes all the contents of /usr/lib/debug into the snapshot file, and once it is unpacked, the root of the debugging files is $ROOT/usr/lib/debug.

Once the debugging files have been installed, symbolic links need to be added to the unpacked snapshot tree to allow GDB to find the debugging information files. Calling the root of the debugging files $DEBUG, execute:

ln -s $DEBUG/usr/bin $ROOT/usr/bin/.debug
ln -s $DEBUG/usr/lib $ROOT/usr/lib/.debug

With these links installed, GDB does not need a set debug-file-directory command to locate the debugging information files, but rather uses the links.