Accelerated Mac OS X Core Dump Analysis Q&A
Q. It appears that on 10.6 I had to recompile the app in order for symbols to work. Is it possible at all to debug 10.7 dumps on 10.6 (or vice versa)?
A. We only tested that core dumps we generated on Mac OS X 10.7 can be opened with symbols on Mac OS X 10.8. We provide app executables to run and also all Xcode projects with source code in case you need to recompile for your system. If you have any difficulties in dump generation please don't hesitate to contact. We can also provide all recompiled apps and newly generated core dumps for newer Mac OS X versions.
Q. The vmmap output looks similar to the output of !address in WinDbg. Is there a way to get this info out of the core dump during postmortem debugging?
A. You can look at the limited output of 2 commands
maintenance info sections
info sharedlibrary
Q. Is there a way to automatically get vmmap output like a core dump (ulimit) when a process crashes?
A. I would suggest to write your own signal or crash handler and launch vmmap from there.
Q. What did you set to get gdb to pause the long output and prompt to continue?
A. The following command limits the output to 10 row chunks:
set height 10
Q. Is there a way to look at the thread's stack limits directly like in WinDbg?
A. WinDbg also cannot do that directly: only via an extension.
Q. Is there a way to have GDB demangle C++ names?
A. By default it does that automatically for stack traces:
(gdb) show demangle-style
The current C++ demangling style is "auto".
(gdb) bt #0 0x00007fff8ed7dbf2 in __psynch_mutexwait () #1 0x00007fff89a281a1 in pthread_mutex_lock () #2 0x0000000105051833 in procA () #3 0x0000000105051919 in bar_two () #4 0x0000000105051929 in foo_two () #5 0x0000000105051941 in thread_two () #6 0x00007fff89a298bf in _pthread_start () #7 0x00007fff89a2cb75 in thread_start ()
(gdb) set demangle-style none
(gdb) bt #0 0x00007fff8ed7dbf2 in __psynch_mutexwait () #1 0x00007fff89a281a1 in pthread_mutex_lock () #2 0x0000000105051833 in _Z5procAv () #3 0x0000000105051919 in _Z7bar_twov () #4 0x0000000105051929 in _Z7foo_twov () #5 0x0000000105051941 in _Z10thread_twoPv () #6 0x00007fff89a298bf in _pthread_start () #7 0x00007fff89a2cb75 in thread_start ()
If you need to demangle output in assembly listings use the following option:
(gdb) disassemble procA Dump of assembler code for function _Z5procAv: 0x00000001050517c0 <_Z5procAv+0>: push %rbp 0x00000001050517c1 <_Z5procAv+1>: mov %rsp,%rbp 0x00000001050517c4 <_Z5procAv+4>: sub $0x30,%rsp 0x00000001050517c8 <_Z5procAv+8>: lea 0x8e9(%rip),%rdi # 0x1050520b8 <mutexA> 0x00000001050517cf <_Z5procAv+15>: callq 0x105051bb0 <dyld_stub_pthread_mutex_lock> 0x00000001050517d4 <_Z5procAv+20>: mov %eax,-0x10(%rbp) 0x00000001050517d7 <_Z5procAv+23>: jmpq 0x1050517dc <_Z5procAv+28> [...]
(gdb) show print asm-demangle
Demangling of C++/ObjC names in disassembly listings is off.
(gdb) set print asm-demangle
(gdb) disassemble procA Dump of assembler code for function _Z5procAv: 0x00000001050517c0 <procA()+0>: push %rbp 0x00000001050517c1 <procA()+1>: mov %rsp,%rbp 0x00000001050517c4 <procA()+4>: sub $0x30,%rsp 0x00000001050517c8 <procA()+8>: lea 0x8e9(%rip),%rdi # 0x1050520b8 <mutexA> 0x00000001050517cf <procA()+15>: callq 0x105051bb0 <dyld_stub_pthread_mutex_lock> 0x00000001050517d4 <procA()+20>: mov %eax,-0x10(%rbp) 0x00000001050517d7 <procA()+23>: jmpq 0x1050517dc <procA()+28> [...]
Q. Should I read assembly code from top down or in reverse?
A. For disassembled code you read down as the processor executes code sequentially from lower to higher addresses: from 0x0000000105051a70 to 0x0000000105051a71, then to 0x0000000105051a74, etc.:
(gdb) disassemble main Dump of assembler code for function main: 0x0000000105051a70 <main+0>: push %rbp 0x0000000105051a71 <main+1>: mov %rsp,%rbp 0x0000000105051a74 <main+4>: sub $0x60,%rsp 0x0000000105051a78 <main+8>: lea 0x639(%rip),%rax [...]
However, for stack traces you need to read the call sequence from bottom to top (main called sleep, sleep called nanosleep) because the stack pointer is decremented during nested function calls (in other words, a stack grows down the lower addresses, please see Thread Stack Trace presentation slide):
(gdb) bt #0 0x00007fff8ed7de42 in __semwait_signal () #1 0x00007fff899dfdea in nanosleep () #2 0x00007fff899dfc08 in sleep () #3 0x0000000105051b75 in main ()
Q. Does malloc_history work for you if you're writing your own allocator?
A. If your allocator ultimately calls down malloc (for example, by default in C++ and Objective-C Cocoa) then all such calls will be traced.
Q. What was the calling convention used there? How did you know RBX was a parameter to malloc_printf function?
A. It is RDI. Please look at Table 1 in this document: http://developer.apple.com/library/mac/#technotes/tn2124/
The parameters are passed through RDI, RSI, RDX, RCX, R8, R9
(gdb) disassemble free Dump of assembler code for function free: 0x00007fff89a7b6c7 <free+0>: push %rbp 0x00007fff89a7b6c8 <free+1>: mov %rsp,%rbp [...] 0x00007fff89a7b7ef <free+296>: lea 0x173c6(%rip),%rdi # 0x7fff89a92bbc <Malloc_Facility+3228> 0x00007fff89a7b7f6 <free+303>: xor %al,%al 0x00007fff89a7b7f8 <free+305>: mov %rbx,%rsi 0x00007fff89a7b7fb <free+308>: callq 0x7fff89a825da <dyld_stub_malloc_printf> [...]
We actually see that the first parameter here is RDI and the second is RSI.
Q. In 32-bit WinDbg, a very common pattern is to find object pointers that have been saved to the stack by using ddp on the stack. Is there anything similar in GDB?
A. In WinDbg it is advisable to use dpp command as it will work as dqp on 64-bit and as ddp on 32-bit Windows. Here we can do something similar using scripting:
(gdb) set $i = 0
(gdb) set $p = $rsp
(gdb) while $i < 10
>printf "%p: ", $p
>x/ga *(long *)$p
>set $i = $i + 1
>set $p = $p + 8
>end
The script prints 10 addresses starting from RSP in the following format (* denotes address dereference):
address: *address [symbol]: **address [symbol]
The only shortcoming is that it breaks on memory access errors:
0x7fff64c509c8: 0x7fff899dfdea <nanosleep+164>: 0xffffffb80e78c085
0x7fff64c509d0: 0xdd8147e00011565: Cannot access memory at address 0xdd8147e00011565
You can also define a custom command with parameters and put it into .gdbinit file or custom command file (-x option to gdb):
define dpp
set $i = 0
set $p = $arg0
while $i < $arg1
printf "%p: ", $p
x/ga *(long *)$p
set $i = $i + 1
set $p = $p + 8
end
end
$ gdb -c ~/Documents/AMCDA-Dumps/core.2281 -e ~/Documents/AMCDA-Dumps/Apps/App11/Build/Products/Release/App11 -x ~/Documents/AMCDA-Dumps/UserCommands.txt
GNU gdb 6.3.50-20050815 (Apple version gdb-1820) (Sat Jun 16 02:40:11 UTC 2012)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin".Reading symbols for shared libraries ... done
Reading symbols for shared libraries . done
Reading symbols for shared libraries ............................ done
#0 0x00007fff8ed7de42 in __semwait_signal ()
(gdb) dpp $rsp 10
0x7fff64c509c8: 0x7fff899dfdea
0x7fff64c509d0: 0xdd8147e00011565: Cannot access memory at address 0xdd8147e00011565
(gdb)
Q. Is there a gdb command for finding pointers to vtables?
A. Please try this formatting command which is off by default:
(gdb) show print vtbl
Printing of C++ virtual function tables is off.
Q. How can you inspect an Objective C variable post-mortem?
A. You can dump its point to see the class it belongs to:
(gdb) x/a 0x7fb811c0d6f0 0x7fb811c0d6f0: 0x1032b5970 <OBJC_CLASS_$_MyClass>
Please also see Assembly-Level Objective-C Debugging section in the following document: