This patch fixes some long-standing bugs in debug threads that were
exacerbated with recent Optional API work in Asterisk 12.
With debug threads enabled, on some systems, there's a lock ordering
problem between our mutex and glibc's mutex protecting its module list
(Ubuntu Lucid, glibc 2.11.1 in this instance). In one thread, the module
list will be locked before acquiring our mutex. In another thread, our
mutex will be locked before locking the module list (which happens in
the depths of calling backtrace()).
This patch fixes this issue by moving backtrace() calls outside of
critical sections that have the mutex acquired. The bigger change was to
reentrancy tracking for ast_cond_{timed,}wait, which wrongly assumed
that waiting on the mutex was equivalent to a single unlock (it actually
suspends all recursive locks on the mutex).
(closes issue ASTERISK-22455)
Review: https://reviewboard.asterisk.org/r/2824/
........
Merged revisions 398648 from http://svn.asterisk.org/svn/asterisk/branches/1.8
........
Merged revisions 398649 from http://svn.asterisk.org/svn/asterisk/branches/11
git-svn-id: https://origsvn.digium.com/svn/asterisk/branches/12@398651 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Review https://reviewboard.asterisk.org/r/2580/ tried to fix the mismatch
in memory pools but had a math error determining the buffer size and
didn't address other similar memory pool mismatches.
* Effectively reverted the previous patch to go in the same direction as
trunk for the returned memory pool of ast_bt_get_symbols().
* Fixed memory leak in ast_bt_get_symbols() when BETTER_BACKTRACES is
defined.
* Fixed some formatting in ast_bt_get_symbols().
* Fixed sig_pri.c freeing memory allocated by libpri when MALLOC_DEBUG is
enabled.
* Fixed __dump_backtrace() freeing memory from ast_bt_get_symbols() when
MALLOC_DEBUG is enabled.
* Moved __dump_backtrace() because of compile issues with the utils
directory.
(closes issue ASTERISK-22221)
Reported by: Matt Jordan
Review: https://reviewboard.asterisk.org/r/2778/
........
Merged revisions 397525 from http://svn.asterisk.org/svn/asterisk/branches/1.8
........
Merged revisions 397528 from http://svn.asterisk.org/svn/asterisk/branches/11
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@397570 65c4cc65-6c06-0410-ace0-fbb531ad65f3
This patch allows astmm to access the backtrace generation code in Asterisk.
When memory is allocated, a backtrace is created and stored with the memory
region that tracks the allocation. If a memory corruption is detected, the
backtrace is printed to the astmm log. The backtrace will make use of the
BETTER_BACKTRACES build option if available.
As a result, this patch moves the backtrace generation code into its own file
and uses the non-wrapped versions of the C library memory allocation routines.
This allows the memory allocation code to safely use the backtrace generation
routines without infinitely recursing.
Review: https://reviewboard.asterisk.org/r/2567
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@391012 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Similar to the patch that moved the fork earlier in the startup sequence to
prevent mutex errors in the recursive mutex surrounding the read/write thread
registration lock, this patch re-initializes the logmsgs mutex. Part of the
start up sequence before forking the process into the background includes
reading asterisk.conf; this has to occur prior to the call to daemon in order
to read startup parameters. When reading in a conf file, log statements can
be generated. Since this can't be avoided, the mutex instead is
re-initialized to ensure a reset of any thread tracking information.
This patch also includes some additional debugging to catch errors when
locking or unlocking the recursive mutex that surrounds locks when the
DEBUG_THREADS build option is enabled. DO_CRASH or THREAD_CRASH will
cause an abort() if a mutex error is detected.
(issue ASTERISK-19463)
Reported by: mjordan
Tesetd by: mjordan
........
Merged revisions 376586 from http://svn.asterisk.org/svn/asterisk/branches/1.8
........
Merged revisions 376587 from http://svn.asterisk.org/svn/asterisk/branches/10
........
Merged revisions 376588 from http://svn.asterisk.org/svn/asterisk/branches/11
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@376589 65c4cc65-6c06-0410-ace0-fbb531ad65f3
With the SCOPED_LOCK macro, you can create a variable
that locks a specific lock and unlocks the lock when the
variable goes out of scope. This is useful for situations
where many breaks, continues, returns, or other interruptions
would require separate unlock statements. With a scoped lock,
these aren't necessary.
There are specializations for mutexes, read locks, write locks,
ao2 locks, ao2 read locks, ao2 write locks, and channel locks.
Each of these is a SCOPED_LOCK at heart though.
Review: https://reviewboard.asterisk.org/r/2060
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@371582 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Add the ability to specify what kind of locking an ao2 object has when it
is allocated. The locking could be one of: MUTEX, RWLOCK, or none.
New API:
ao2_t_alloc_options()
ao2_alloc_options()
ao2_t_container_alloc_options()
ao2_container_alloc_options()
ao2_rdlock()
ao2_wrlock()
ao2_tryrdlock()
ao2_trywrlock()
The OBJ_NOLOCK and AO2_ITERATOR_DONTLOCK flags have a slight meaning
change. They no longer mean that the object is protected by an external
mechanism. They mean the lock associated with the object has already been
manually obtained by one of the ao2_lock calls. This change is necessary
for RWLOCK support since they are not reentrant. Also an operation on an
ao2 container may require promoting a read lock to a write lock by
releasing the already held read lock to re-acquire as a write lock.
Replaced API calls:
ao2_t_link_nolock()
ao2_link_nolock()
ao2_t_unlink_nolock()
ao2_unlink_nolock()
with the respective
ao2_t_link_flags()
ao2_link_flags()
ao2_t_unlink_flags()
ao2_unlink_flags()
API calls to be more flexible and to allow an anticipated enhancement to
control linking duplicate objects into a container.
The changes to format.c and format_cap.c are taking advantange of the new
ao2 locking options to simplify the use of the format capabilities
containers.
Review: https://reviewboard.asterisk.org/r/1554/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@357272 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.8
........
r304950 | tilghman | 2011-01-31 00:41:36 -0600 (Mon, 31 Jan 2011) | 18 lines
Change mutex tracking so that it only consumes memory in the core mutex object when it's actually being used.
This reduces the overall size of a mutex which was 3016 bytes before this back
down to 216 bytes (this is on 64-bit Linux with a glibc-implemented mutex).
The exactness of the numbers here may vary slightly based upon how mutexes are
implemented on a platform, but the long and short of it is that prior to this
commit, chan_iax2 held down 98MB of memory on a 64-bit system for nothing more
than a table of 32767 locks. After this commit, the same table occupies a mere
7MB of memory.
(closes issue #18194)
Reported by: job
Patches:
20110124__issue18194.diff.txt uploaded by tilghman (license 14)
Tested by: tilghman
Review: https://reviewboard.asterisk.org/r/1066
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@304951 65c4cc65-6c06-0410-ace0-fbb531ad65f3
"Bad Things" would happen if Asterisk was compiled with DEBUG_THREADS, but a
loaded module was not (or vice versa). This also immensely simplifies the
lock code, since there are no longer 2 separate versions of them.
Review: https://reviewboard.asterisk.org/r/508/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@258557 65c4cc65-6c06-0410-ace0-fbb531ad65f3
This patch finishes the implementation of OBJ_MULTIPLE in astobj2 (the
case where multiple results need to be returned; OBJ_NODATA mode
already was supported). In addition, it converts ast_channel_iterators
(only the targeted versions, not the ones that iterate over all
channels) to use this method.
During this work, I removed the 'ao2_flags' arguments to the
ast_channel_iterator constructor functions; there were no uses of that
argument yet, there is only one possible flag to pass, and it made the
iterators less 'opaque'. If at some point in the future someone really
needs an ast_channel_iterator that does not lock the container, we can
provide constructor(s) for that purpose.
Review: https://reviewboard.asterisk.org/r/379/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@225244 65c4cc65-6c06-0410-ace0-fbb531ad65f3
There is a lot that could be said about this, but the patch is a big
improvement for performance, stability, code maintainability,
and ease of future code development.
The channel list is no longer an unsorted linked list. The main container
for channels is an astobj2 hash table. All of the code related to searching
for channels or iterating active channels has been rewritten. Let n be
the number of active channels. Iterating the channel list has gone from
O(n^2) to O(n). Searching for a channel by name went from O(n) to O(1).
Searching for a channel by extension is still O(n), but uses a new method
for doing so, which is more efficient.
The ast_channel object is now a reference counted object. The benefits
here are plentiful. Some benefits directly related to issues in the
previous code include:
1) When threads other than the channel thread owning a channel wanted
access to a channel, it had to hold the lock on it to ensure that it didn't
go away. This is no longer a requirement. Holding a reference is
sufficient.
2) There are places that now require less dealing with channel locks.
3) There are places where channel locks are held for much shorter periods
of time.
4) There are places where dealing with more than one channel at a time becomes
_MUCH_ easier. ChanSpy is a great example of this. Writing code in the
future that deals with multiple channels will be much easier.
Some additional information regarding channel locking and reference count
handling can be found in channel.h, where a new section has been added that
discusses some of the rules associated with it.
Mark Michelson also assisted with the development of this patch. He did the
conversion of ChanSpy and introduced a new API, ast_autochan, which makes it
much easier to deal with holding on to a channel pointer for an extended period
of time and having it get automatically updated if the channel gets masqueraded.
Mark was also a huge help in the code review process.
Thanks to David Vossel for his assistance with this branch, as well. David
did the conversion of the DAHDIScan application by making it become a wrapper
for ChanSpy internally.
The changes come from the svn/asterisk/team/russell/ast_channel_ao2 branch.
Review: http://reviewboard.digium.com/r/203/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@190423 65c4cc65-6c06-0410-ace0-fbb531ad65f3
When dvossel and I were doing some load testing last week, we noticed that we
could make Asterisk trunk lock up instantly when we started generating a bunch
of calls. The backtraces of locked threads were bizarre, and many were stuck
on an _unlock_ of an rwlock.
The changes are:
1) Fix a number of places where a backtrace would be loaded into an invalid
index of the backtrace array. It's an off by one error, which ends up
writing over the rwlock itself.
2) Ensure that in the array of held locks, we NULL out an index once it is
not being used so that it's not confusing when analyzing its contents.
3) Remove a bunch of logging referring to an rwlock operating being done
with "deep reentrancy". It is normal for _many_ threads to hold a
read lock on an rwlock.
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@184531 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Copied from my review board description:
This is a continuation of the API changes documentation started for describing
changes between releases. Most of the API changes were pretty simple needing
only to be brought to attention via the new "Asterisk API Changes" list.
However, if you see anything that needs further explanation feel free to
supplement what is there. The current method of documenting is to add (in the
header file): \version <ver number> <description of changes> and then to add
the function to the change list in doxyref.h on the AstAPIChanges page. I also
made sure all the functions that were newly added were tagged with \since
1.6.1. I think this is a good habit to start both for the historical aspect as
well as for the future ability to easily add a "New Asterisk API" page.
Review: http://reviewboard.digium.com/r/190/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@180719 65c4cc65-6c06-0410-ace0-fbb531ad65f3
------------------------------------------------------------------------
r159808 | kpfleming | 2008-11-29 10:58:29 -0600 (Sat, 29 Nov 2008) | 7 lines
update dev-mode compiler flags to match the ones used by default on Ubuntu Intrepid, so all developers will see the same warnings and errors
since this branch already had some printf format attributes, enable checking for them and tag functions that didn't have them
format attributes in a consistent way
------------------------------------------------------------------------
in addition:
move some format attributes from main/utils.c to the header files they belong in, and fix up references to the relevant functions based on new compiler warnings
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@159818 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Reported by: murf
Most of this bug was already fixed by Tilghman before
I opened it; Many thanks to Tilghman for his fix
in svn version 125794. That fix cleared up some of the
fields in the lock_info.
This commit changes the address that is stored for the
lock in the lock_info struct, so that it is the same
as that passed into the locking macros. This makes
searching for a lock_info (as in log_show_lock())
by its lock addr possible. The lock_addr field is
infinitely more useful if it is the same as what
is 'publicly' available outside the lock_info code.
Many thanks to kpfleming, putnopvut, and Russell for their
invaluable insights earlier today.
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@131570 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r126573 | russell | 2008-06-30 11:05:08 -0500 (Mon, 30 Jun 2008) | 10 lines
Fix a typo in the non-DEBUG_THREADS version of the recently added DEADLOCK_AVOIDANCE()
macro. This caused the lock to not actually be released, and as a result, not
avoid deadlocks at all. This resolves the issues reported in the last while about
Asterisk locking up all over the place (and most commonly, in chan_iax2).
(closes issue #12927)
(closes issue #12940)
(closes issue #12925)
(potentially closes others ...)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@126574 65c4cc65-6c06-0410-ace0-fbb531ad65f3
account that multiple threads could hold the same rdlock at the same time.
As such, it expected that when a thread released a lock that it must have
been the last to acquire the lock as well. Erroneous error messages would
be sent to the console stating that a thread was attempting to unlock a lock
it did not own.
Now all threads are examined to be sure that the message is only printed
when it is supposed to be printed.
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@125133 65c4cc65-6c06-0410-ace0-fbb531ad65f3
hold tracking information for mutexes. Now, the "core show locks" output
will output information about who is holding a rwlock when a thread is
waiting on it.
(closes issue #11279)
Reported by: ys
Patches:
trunk_lock_utils.v8.diff uploaded by ys (license 281)
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@120064 65c4cc65-6c06-0410-ace0-fbb531ad65f3
If a deadlock is detected, then the typical lock information will be
printed along with a backtrace of the stack for the offending threads.
Use of this requires compiling with DETECT_DEADLOCKS and having glibc
installed.
Furthermore, issuing the "core show locks" CLI command will print the
normal lock information as well as a backtraces for each lock. This
requires that DEBUG_THREADS is enabled and that glibc is installed.
All the backtrace features may be disabled by running the configure
script with --without-execinfo as an argument
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@118173 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r116088 | mmichelson | 2008-05-13 18:47:49 -0500 (Tue, 13 May 2008) | 12 lines
A change to the way channel locks are handled when DEBUG_CHANNEL_LOCKS is defined.
After debugging a deadlock, it was noticed that when DEBUG_CHANNEL_LOCKS
is enabled in menuselect, the actual origin of channel locks is obscured
by the fact that all channel locks appear to happen in the function
ast_channel_lock(). This code change redefines ast_channel_lock to be a
macro which maps to __ast_channel_lock(), which then relays the proper
file name, line number, and function name information to the core lock
functions so that this information will be displayed in the case that
there is some sort of locking error or core show locks is issued.
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@116089 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r105116 | russell | 2008-02-28 16:23:05 -0600 (Thu, 28 Feb 2008) | 8 lines
Fix a bug in the lock tracking code that was discovered by mmichelson. The issue
is that if the lock history array was full, then the functions to mark a lock as
acquired or not would adjust the stats for whatever lock is at the end of the array,
which may not be itself. So, do a sanity check to make sure that we're updating
lock info for the proper lock.
(This explains the bizarre stats on lock #63 in BE-396, thanks Mark!)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@105144 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r92875 | mmichelson | 2007-12-13 19:24:06 -0600 (Thu, 13 Dec 2007) | 7 lines
When compiling with DETECT_DEADLOCKS, don't spam the CLI with messages
about possible deadlocks. Instead just print the intended single message every
five seconds.
(closes issue 11537, reported and patched by dimas)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@92876 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r91828 | russell | 2007-12-07 15:17:24 -0600 (Fri, 07 Dec 2007) | 6 lines
Fix another bug in the DEBUG_THREADS code. The ast_mutex_init() function had
the mutex attribute object marked as static. This means that multiple threads
initializing locks at the same time could step on each other and end up with
improperly initialized locks.
(found when tracking down locking issues related to issue #11080)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@91829 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r91826 | russell | 2007-12-07 15:11:08 -0600 (Fri, 07 Dec 2007) | 6 lines
I love fixing lock related errors in the lock debugging code. That's about as
ironic as it gets in Asterisk programming land. Anyway, I spotted this bug while
trying to track down why systems are locking up and acting weird in issue #11080.
The mutex attribute object was marked as static in this function when it should
not have been.
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@91827 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r91070 | russell | 2007-12-04 18:35:31 -0600 (Tue, 04 Dec 2007) | 11 lines
Fix some crashes in chan_iax2 that were reported as happening on Mac systems.
It turns out that the problem was the Mac version of the ast_atomic_fetchadd_int()
function. The Mac atomic add function returns the _new_ value, while this function
is supposed to return the old value. So, the crashes happened on unreferencing
objects. If the reference count was decreased to 1, ao2_ref() thought that it
had been decreased to zero, and called the destructor. However, there was still
an outstanding reference around.
(closes issue #11176)
(closes issue #11289)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@91114 65c4cc65-6c06-0410-ace0-fbb531ad65f3
