Getting a Backtrace¶

This document is intended to provide information on how to obtain the backtraces required on the GitHub Asterisk Issue Tracker

The backtrace information is required by developers to help fix problems with bugs of any kind. Backtraces provide information about what was wrong when a program crashed; in our case, Asterisk.

Preparing Asterisk To Produce Core Files On Crash¶

• Asterisk MUST be started with the -g option.
  Running pgrep -xa asterisk will display the full command line used to start asterisk. If -g isn't shown, adjust your startup scripts accordingly.

• You must build Asterisk with the DONT_OPTIMIZE option selected in menuselect.
  Running core show settings from the Asterisk CLI will display the current build options. If DONT_OPTIMIZE isn't displayed under "All Build Options", you'll need to re-run menuselect and enable it under "Compiler Flags". It automatically selects COMPILE_DOUBLE which you can safely deselect. Re-compile and re-install after saving teh configuration.

• Asterisk must have the debug symbols still included in the binaries.
  This is the default unless you, or the package maintainer if installing from packages, strips the binaries. Normally a package maintainer will strip the binaries and then provide separate "debug" packages that contain just the symbols. If so, make sure those debug packages are installed.

• Understand and adjust where coredumps will be saved by the kernel.
  Run sysctl kernel.core_pattern to determine how and where coredumps are saved. This can be a simple path with replacement parameters like kernel.core_pattern = /tmp/core-%e-%t or a program like kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %d %F. We recommend using /tmp/core-%e-%t. To set it, edit your /etc/sysctl.conf, comment out any existing kernel.core_pattern statements, add a new one like kernel.core_pattern=/tmp/core-%e-%t, and run sysctl -p to make it active. See kernel.core_pattern for more information. NOTE: Coredumps are basically dumps of the memory a process is using. This could result in a coredump several gigabytes in size. Make sure the filesystem that coredumps are being saved on has enough free space.

• You must have the GNU Debugger (gdb) installed on the system where Asterisk is running.
  The debugger is required to dump the contents of a coredump to human readable form.

Getting Information After A Crash¶

Assuming you've changed kernel.core_pattern as recommended, if Asterisk crashes, a coredump will be produced with a name like /tmp/core-asterisk-1767016730.2807217.

Asterisk provides a utility named ast_coredumper that examines coredumps and produces several reports, including backtraces, lists of active channels and bridges, etc. It's automatically installed in /var/lib/asterisk/scripts and should be run immediately after a crash is detected.

$ sudo /var/lib/asterisk/scripts/ast_coredumper --tarball-coredumps <path_to_coredump>

This will create the text reports and create a tarball that contains the reports, the raw coredump file, the asterisk binary, and all of the installed asterisk modules. THIS IS ESSENTIAL FOR FURTHER TROUBLESHOOTING BY THE ASTERISK TEAM!!

Getting Information for a Running Asterisk Instance¶

There may be cases where you'll need to force a coredump on a running instance of Asterisk because it's deadlocked or in another state where it's not processing calls correctly. To facilitate this, ast_coredumper can temporarily pause the asterisk process and generate a coredump.

$ sudo /var/lib/asterisk/scripts/ast_coredumper --running --tarball-coredumps

To skip the confirmation and generate the coredump immediately, specify --RUNNING instead of --running:

$ sudo /var/lib/asterisk/scripts/ast_coredumper --RUNNING --tarball-coredumps

Reporting crashes and deadlocks¶

Subject to the warning above, you can attach the text reports generated by ast_coredumper directly to an Asterisk GitHub issue. If the full tarball is required for further troubleshooting, you'll be given instructions on how to get that to us securely.