mirror of
https://github.com/asterisk/asterisk.git
synced 2025-10-31 02:37:10 +00:00
bef07ecc16
https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r257426 | lmadsen | 2010-04-15 14:40:33 -0500 (Thu, 15 Apr 2010) | 13 lines Update backtrace.txt documentation. Update the backtrace.txt documentation so it conforms to the same layout as other documents we've been working on recently. Additionally, add a bunch of new information about gathering backtraces for crashes and deadlocks, along with ways of verifying your file before uploading it. Create a couple of one line commands for people to generate the files we need. (closes issue #17190) Reported by: lmadsen Patches: backtrace.txt.patch-2 uploaded by lmadsen (license 10) Tested by: lmadsen, pabelanger ........ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@257427 65c4cc65-6c06-0410-ace0-fbb531ad65f3
278 lines
11 KiB
Plaintext
278 lines
11 KiB
Plaintext
===============================================================================
|
|
===
|
|
=== Collecting Backtrace Information
|
|
===
|
|
=== Last updated: 2010-04-12
|
|
===============================================================================
|
|
|
|
This document is intended to provide information on how to obtain the
|
|
backtraces required on the asterisk bug tracker, available at
|
|
https://issues.asterisk.org.
|
|
|
|
-------------------------------------------------------------------------------
|
|
--- Overview
|
|
-------------------------------------------------------------------------------
|
|
|
|
The backtrace information is required by developers to help fix problem with
|
|
bugs of any kind. Backtraces provide information about what was wrong when a
|
|
program crashed; in our case, Asterisk.
|
|
|
|
-------------------------------------------------------------------------------
|
|
--- Preparing Asterisk To Produce Core Files On Crash
|
|
-------------------------------------------------------------------------------
|
|
|
|
First of all, when you start Asterisk, you MUST start it with option
|
|
-g. This tells Asterisk to produce a core file if it crashes.
|
|
|
|
If you start Asterisk with the safe_asterisk script, it automatically
|
|
starts using the option -g.
|
|
|
|
If you're not sure if Asterisk is running with the -g option, type the
|
|
following command in your shell:
|
|
|
|
debian:/tmp# ps aux | grep asterisk
|
|
root 17832 0.0 1.2 2348 788 pts/1 S Aug12 0:00 /bin/sh /usr/sbin/safe_asterisk
|
|
root 26686 0.0 2.8 15544 1744 pts/1 S Aug13 0:02 asterisk -vvvg -c
|
|
[...]
|
|
|
|
The interesting information is located in the last column.
|
|
|
|
Second, your copy of Asterisk must have been built without
|
|
optimization or the backtrace will be (nearly) unusable. This can be
|
|
done by selecting the 'DONT_OPTIMIZE' option in the Compiler Flags
|
|
submenu in the 'make menuselect' tree before building Asterisk.
|
|
|
|
Running a production server with DONT_OPTIMIZE is generally safe.
|
|
You'll notice the binary files may be a bit larger, but in terms of
|
|
Asterisk performance, and impact should be negligible.
|
|
|
|
After Asterisk crashes, a core file will be "dumped" in your /tmp/
|
|
directory. To make sure it's really there, you can just type the
|
|
following command in your shell:
|
|
|
|
debian:/tmp# ls -l /tmp/core.*
|
|
-rw------- 1 root root 10592256 Aug 12 19:40 /tmp/core.26252
|
|
-rw------- 1 root root 9924608 Aug 12 20:12 /tmp/core.26340
|
|
-rw------- 1 root root 10862592 Aug 12 20:14 /tmp/core.26374
|
|
-rw------- 1 root root 9105408 Aug 12 20:19 /tmp/core.26426
|
|
-rw------- 1 root root 9441280 Aug 12 20:20 /tmp/core.26462
|
|
-rw------- 1 root root 8331264 Aug 13 00:32 /tmp/core.26647
|
|
debian:/tmp#
|
|
|
|
In the event that there are multiple core files present (as in the
|
|
above example), it is important to look at the file timestamps in
|
|
order to determine which one you really intend to look at.
|
|
|
|
-------------------------------------------------------------------------------
|
|
--- Getting Information After A Crash
|
|
-------------------------------------------------------------------------------
|
|
|
|
There are two kind of backtraces (aka 'bt') which are useful: bt and bt full.
|
|
|
|
Now that we've verified the core file has been written to disk, the final part
|
|
is to extract 'bt' from the core file. Core files are pretty big, don't be
|
|
scared, it's normal.
|
|
|
|
******************************************************************************
|
|
*** NOTE: Don't attach core files on the bug tracker as they are only useful *
|
|
*** on the machine they were generated on. We only need the output of *
|
|
*** the 'bt' and 'bt full.' *
|
|
******************************************************************************
|
|
|
|
For extraction, we use a really nice tool, called gdb. To verify that
|
|
you have gdb installed on your system:
|
|
|
|
debian:/tmp# gdb -v
|
|
GNU gdb 6.3-debian
|
|
Copyright 2004 Free Software Foundation, Inc.
|
|
GDB is free software, covered by the GNU General Public License, and you are
|
|
welcome to change it and/or distribute copies of it under certain conditions.
|
|
Type "show copying" to see the conditions.
|
|
There is absolutely no warranty for GDB. Type "show warranty" for details.
|
|
This GDB was configured as "i386-linux".
|
|
debian:/tmp#
|
|
|
|
If you don't have gdb installed, go install gdb. You should be able to install
|
|
using something like: apt-get install gdb --or-- yum install gdb
|
|
|
|
Now load the core file in gdb with the following command. This will also save
|
|
the output of gdb to the /tmp/backtract.txt file.
|
|
|
|
# gdb -se "asterisk" -c /tmp/core.26252 | tee /tmp/backtrace.txt
|
|
|
|
******************************************************************************
|
|
*** TIP!
|
|
*** Just run the following command to get the output into the
|
|
*** backtrace.txt file, ready for uploading to the issue tracker. Be sure
|
|
*** to change the name of the core file to your actual core dump file:
|
|
***
|
|
*** gdb -se "asterisk" -ex "bt full" -ex "thread apply all bt" --batch -c /tmp/core.26252 > /tmp/backtrace.txt
|
|
***
|
|
******************************************************************************
|
|
|
|
|
|
[...]
|
|
(You would see a lot of output here.)
|
|
[...]
|
|
Reading symbols from /usr/lib/asterisk/modules/app_externalivr.so...done.
|
|
Loaded symbols for /usr/lib/asterisk/modules/app_externalivr.so
|
|
#0 0x29b45d7e in ?? ()
|
|
(gdb)
|
|
|
|
In order to make extracting the gdb output easier, you may wish to
|
|
turn on logging using "set logging on". This command will save all
|
|
output to the default file of gdb.txt, which in the end can be
|
|
uploaded as an attachment to the bug tracker.
|
|
|
|
Now at the gdb prompt, type: bt
|
|
You would see output similar to:
|
|
|
|
(gdb) bt
|
|
#0 0x29b45d7e in ?? ()
|
|
#1 0x08180bf8 in ?? ()
|
|
#2 0xbcdffa58 in ?? ()
|
|
#3 0x08180bf8 in ?? ()
|
|
#4 0xbcdffa60 in ?? ()
|
|
#5 0x08180bf8 in ?? ()
|
|
#6 0x180bf894 in ?? ()
|
|
#7 0x0bf80008 in ?? ()
|
|
#8 0x180b0818 in ?? ()
|
|
#9 0x08068008 in ast_stopstream (tmp=0x40758d38) at file.c:180
|
|
#10 0x000000a0 in ?? ()
|
|
#11 0x000000a0 in ?? ()
|
|
#12 0x00000000 in ?? ()
|
|
#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262
|
|
#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965
|
|
#15 0xbcdffbe0 in ?? ()
|
|
#16 0x40028e51 in pthread_start_thread () from /lib/libpthread.so.0
|
|
#17 0x401ec92a in clone () from /lib/libc.so.6
|
|
(gdb)
|
|
|
|
|
|
The bt's output is the information that we need on the bug tracker.
|
|
|
|
Now do a bt full as follows:
|
|
|
|
(gdb) bt full
|
|
#0 0x29b45d7e in ?? ()
|
|
No symbol table info available.
|
|
#1 0x08180bf8 in ?? ()
|
|
No symbol table info available.
|
|
#2 0xbcdffa58 in ?? ()
|
|
No symbol table info available.
|
|
#3 0x08180bf8 in ?? ()
|
|
No symbol table info available.
|
|
#4 0xbcdffa60 in ?? ()
|
|
No symbol table info available.
|
|
#5 0x08180bf8 in ?? ()
|
|
No symbol table info available.
|
|
#6 0x180bf894 in ?? ()
|
|
No symbol table info available.
|
|
#7 0x0bf80008 in ?? ()
|
|
No symbol table info available.
|
|
#8 0x180b0818 in ?? ()
|
|
No symbol table info available.
|
|
#9 0x08068008 in ast_stopstream (tmp=0x40758d38) at file.c:180
|
|
No locals.
|
|
#10 0x000000a0 in ?? ()
|
|
No symbol table info available.
|
|
#11 0x000000a0 in ?? ()
|
|
No symbol table info available.
|
|
#12 0x00000000 in ?? ()
|
|
No symbol table info available.
|
|
#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262
|
|
f = (struct ast_frame *) 0x8180bf8
|
|
trans = (struct ast_trans_pvt *) 0x0
|
|
#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965
|
|
No locals.
|
|
#15 0xbcdffbe0 in ?? ()
|
|
No symbol table info available.
|
|
#16 0x40028e51 in pthread_start_thread () from /lib/libpthread.so.0
|
|
No symbol table info available.
|
|
#17 0x401ec92a in clone () from /lib/libc.so.6
|
|
No symbol table info available.
|
|
(gdb)
|
|
|
|
The final "extraction" would be to know all traces by all threads. Even if
|
|
Asterisk runs on the same thread for each call, it could have created some new
|
|
threads.
|
|
|
|
To make sure we have the correct information, just do:
|
|
(gdb) thread apply all bt
|
|
|
|
Thread 1 (process 26252):
|
|
#0 0x29b45d7e in ?? ()
|
|
#1 0x08180bf8 in ?? ()
|
|
#2 0xbcdffa58 in ?? ()
|
|
#3 0x08180bf8 in ?? ()
|
|
#4 0xbcdffa60 in ?? ()
|
|
#5 0x08180bf8 in ?? ()
|
|
#6 0x180bf894 in ?? ()
|
|
#7 0x0bf80008 in ?? ()
|
|
#8 0x180b0818 in ?? ()
|
|
#9 0x08068008 in ast_stopstream (tmp=0x40758d38) at file.c:180
|
|
#10 0x000000a0 in ?? ()
|
|
#11 0x000000a0 in ?? ()
|
|
#12 0x00000000 in ?? ()
|
|
#13 0x407513c3 in confcall_careful_stream (conf=0x8180bf8, filename=0x8181de8 "DAHDI/pseudo-1324221520") at app_meetme.c:262
|
|
#14 0x40751332 in streamconfthread (args=0x8180bf8) at app_meetme.c:1965
|
|
#15 0xbcdffbe0 in ?? ()
|
|
#16 0x40028e51 in pthread_start_thread () from /lib/libpthread.so.0
|
|
#17 0x401ec92a in clone () from /lib/libc.so.6
|
|
(gdb)
|
|
|
|
|
|
That output tells us crucial information about each thread.
|
|
|
|
-------------------------------------------------------------------------------
|
|
--- Getting Information For A Deadlock
|
|
-------------------------------------------------------------------------------
|
|
|
|
Whenever supplying information about a deadlock (i.e. when you run the
|
|
'core show locks' command on the Asterisk console), it is useful to also have
|
|
additional information about the threads. We can generate this information by
|
|
attaching to a running Asterisk process and gathering that information.
|
|
|
|
You can easily attach to a running Asterisk process, gather the output required
|
|
and then detach from the process all in a single step. Execute the following
|
|
command and upload the resulting backtrace-threads.txt file to the Asterisk
|
|
issue tracker:
|
|
|
|
gdb -ex "thread apply all bt" --batch /usr/sbin/asterisk `pidof asterisk` > /tmp/backtrace-threads.txt
|
|
|
|
Note that this gathers information from the running Asterisk process, so you
|
|
want to make sure you run this command immediately before or after gathering
|
|
the output of 'core show locks'. You can gather that information by running the
|
|
following command:
|
|
|
|
asterisk -rx "core show locks" > /tmp/core-show-locks.txt
|
|
|
|
-------------------------------------------------------------------------------
|
|
--- Verify Your Backtraces
|
|
-------------------------------------------------------------------------------
|
|
|
|
Before uploading your backtraces to the issue tracker, you should double check
|
|
to make sure the data you have is of use to the developers. Check your
|
|
backtrace files to make sure you're not seeing several of the following:
|
|
|
|
<value optimized out>
|
|
|
|
If you are, then you likely haven't compiled with DONT_OPTIMIZE. The impact of
|
|
DONT_OPTIMIZE is negligible on most systems. Be sure you've enabled the
|
|
DONT_OPTIMIZE flag within the Compiler Flags section of menuselect. After
|
|
doing so, be sure to run 'make install' and restart Asterisk.
|
|
|
|
-------------------------------------------------------------------------------
|
|
--- Uploading Your Information To The Issue Tracker
|
|
-------------------------------------------------------------------------------
|
|
|
|
You're now ready to upload your files to the Asterisk issue tracker (located at
|
|
https://issues.asterisk.org).
|
|
|
|
******************************************************************************
|
|
*** NOTE: Please ATTACH your output! DO NOT paste it as a note! *
|
|
******************************************************************************
|
|
|
|
If you have questions or comments regarding this documentation, feel free to
|
|
pass by the #asterisk-bugs channel on irc.freenode.net.
|