Q&A from Fundamentals of Complete Crash and Hang Memory Dump Analysis Webinar
Q: What's the definition of a special process?
A: A special process is a process that you would never encounter in the normal operating environment such as WerFault.exe or drwtsn32.exe (for older systems).
Q: What is "Type information missing error for MiSystemVaTypeCount"?
A: Perhaps WinDbg needed the type information for that variable but wasn't able to get it from public symbols:
1: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 1031581 ( 4126324 Kb)
Page File: \??\C:\pagefile.sys
Current: 4433524 Kb Free Space: 4433520 Kb
Minimum: 4433524 Kb Maximum: 12378972 Kb
Type information missing error for MiSystemVaTypeCount
1: kd> x nt!MiSystemVaTypeCount
fffff800`01c45b00 nt!MiSystemVaTypeCount = <no type information>
Q: How do we know that we need to search for MessageBox in the first place?
A: It is just one of the checks from a checklist to detect blocked UI threads waiting for a user input. These threads might be blocking other threads like in the case of a spooler service installing a print driver or in the case when one of the threads pops up a message about an error or a warning.
Q: Why is the message text not available as a parameter in a 64-bit dump for MessageBoxW?
A: This is because on x64 systems the first function parameters are passed via registers instead of a stack. It is still possible to get message text manually if we disassemble the code as done here (the log file from the second webinar session):
1: kd> .thread /r /p fffffa800451db60
Implicit thread is now fffffa80`0451db60
Implicit process is now fffffa80`04514630
Loading User Symbols
1: kd> kv
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr : Args to Child : Call Site
fffff880`05b7f8b0 fffff800`01a93992 : fffff900`c27c0c30 fffffa80`0451db60 fffff900`c27c0c30 fffff880`00000008 : nt!KiSwapContext+0x7a
fffff880`05b7f9f0 fffff800`01a95cff : 00000000`00000001 fffff800`01a968c2 fffff800`00000000 00000000`00000000 : nt!KiCommitThreadWait+0x1d2
fffff880`05b7fa80 fffff960`0011b557 : fffff900`c27c0c00 00000000`0000000d 00000000`6d747301 00000000`00000000 : nt!KeWaitForSingleObject+0x19f
fffff880`05b7fb20 fffff960`0011b5f1 : 00000000`00000000 00000000`00000000 00000000`00000001 00000000`00000000 : win32k!xxxRealSleepThread+0x257
fffff880`05b7fbc0 fffff960`0012e22e : fffffa80`0451db60 fffff880`05b7fca0 00000000`00000001 00000000`00000020 : win32k!xxxSleepThread+0x59
fffff880`05b7fbf0 fffff800`01a8b993 : fffffa80`0451db60 00000000`0050d2e0 00000000`0050f1d0 00000000`00000020 : win32k!NtUserWaitMessage+0x46
fffff880`05b7fc20 00000000`775cbf5a : 00000000`775d7214 00000000`00000000 00000000`00934fb0 00000000`00000001 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`05b7fc20)
00000000`022ff7c8 00000000`775d7214 : 00000000`00000000 00000000`00934fb0 00000000`00000001 00000000`00000001 : USER32!ZwUserWaitMessage+0xa
00000000`022ff7d0 00000000`775d74a5 : 00000000`00000000 00000000`776232c4 00000000`00000000 00000000`00000000 : USER32!DialogBox2+0x274
00000000`022ff860 00000000`776227f0 : 00000000`004fe8c0 ffffffff`ffffffff 00000000`00000001 00000001`3f3c8408 : USER32!InternalDialogBox+0x135
00000000`022ff8c0 00000000`77621ae5 : 00000000`0000007e 00000000`00000000 00000000`004fe8c0 0000ffff`0000004b : USER32!SoftModalMessageBox+0x9b4
00000000`022ff9f0 00000000`7762133b : 00000000`00000000 00000000`00000000 00000000`00000000 00000001`3f3c83e8 : USER32!MessageBoxWorker+0x31d
00000000`022ffbb0 00000000`77621232 : 00000000`0032ede0 00000000`00000000 00000000`00000000 00000000`00000000 : USER32!MessageBoxTimeoutW+0xb3
*** ERROR: Module load completed but symbols could not be loaded for ApplicationA.exe
00000000`022ffc80 00000001`3f3c1089 : 00000000`0032ede0 00000000`00000000 00000000`00000000 00000000`00000000 : USER32!MessageBoxW+0x4e
00000000`022ffcc0 00000001`3f3c11fb : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ApplicationA+0x1089
00000000`022ffcf0 00000001`3f3c12a5 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ApplicationA+0x11fb
00000000`022ffd20 00000000`776cf56d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ApplicationA+0x12a5
00000000`022ffd50 00000000`77803281 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`022ffd80 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d
1: kd> ub 00000001`3f3c1089
00000001`3f3c105f cc int 3
00000001`3f3c1060 4883ec28 sub rsp,28h
00000001`3f3c1064 666666660f1f840000000000 nop word ptr [rax+rax]
00000001`3f3c1070 4c8d0571730000 lea r8,[ApplicationA+0x83e8 (00000001`3f3c83e8)]
00000001`3f3c1077 488d157a730000 lea rdx,[ApplicationA+0x83f8 (00000001`3f3c83f8)]
00000001`3f3c107e 4533c9 xor r9d,r9d
00000001`3f3c1081 33c9 xor ecx,ecx
00000001`3f3c1083 ff156f610000 call qword ptr [ApplicationA+0x71f8 (00000001`3f3c71f8)]
1: kd> du 00000001`3f3c83e8
1: kd> du 00000001`3f3c83f8
Please also see these links for additional explanation:
Q: What was the connection between MessageBox and Mutant?
A: No connection in this dump. Just both threads are blocked from further processing and in the latter case, the mutex was created by ApplicationB and a thread from ApplicationC was waiting on it blocking other threads (wait chain).
Q: Would it be possible for more detailed Presentation?
A: Yes, I plan series of mini presentations about individual patterns as well.
Q: Do you teach these courses commercially to companies?
A: Not yet, but I plan to expand them to be taught commercially very soon.
Q: The CARE system. Are you planning to only support kernel dumps or also application dumps?
A: To support all dump types. It is based on debugger log analysis so the dump type is irrelevant here.
Q: Will you publish a book about this topic?
A: Thinking about it. Certainly this material will be incorporated in already scheduled books like Windows Crash Dump Analysis and Advanced Windows Crash Dump Analysis.
Q: Will !process and !vm command work on a user dump?
A: No, they will not unless you write your own WinDbg extension :-)
Q: Could you please describe CARE a little bit more? What are the practical goals of this project?
A: The main goal is to make my life easier. To make pattern detection like a breeze. So I would then concentrate on other things :-)
Q: Can you come down to my company and teach courses like this?
A: Yes, I can. If your company is in Ireland I can do immediately. For other EU countries this service will be available later this year.
Q: Is it safe to use ub command as it may show wrong disassembled commands?
A: Usually if the last instruction is call (for ub of a return address) it increases the chances that the code is correct:
1: kd> kv
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr : Args to Child : Call Site
fffff880`05419940 fffff800`01a93992 : fffffa80`054003a0 fffffa80`04f8b240 00000000`00000000 00000000`0000000a : nt!KiSwapContext+0x7a
fffff880`05419a80 fffff800`01a95cff : 00000000`00000044 00000000`00000001 00000000`00000000 fffff880`05419ca0 : nt!KiCommitThreadWait+0x1d2
fffff880`05419b10 fffff800`01d871d2 : fffffa80`04f8b200 00000000`00000006 00000000`00000001 00000000`00000000 : nt!KeWaitForSingleObject+0x19f
fffff880`05419bb0 fffff800`01a8b993 : fffffa80`04f8b240 00000000`00000044 00000000`00000000 fffffa80`04c41fe0 : nt!NtWaitForSingleObject+0xb2
fffff880`05419c20 00000000`7781fefa : 00000000`7781d1a8 000007fe`fd8d7452 00000000`00000044 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`05419c20)
00000000`002af9f8 00000000`7781d1a8 : 000007fe`fd8d7452 00000000`00000044 00000000`00000000 00000000`00000000 : ntdll!NtWaitForSingleObject+0xa
00000000`002afa00 00000000`7781d0a2 : 00000000`00000000 00000000`00000000 00000000`00000000 00000001`3f70bf48 : ntdll!RtlpWaitOnCriticalSection+0xe8
00000000`002afab0 00000001`3f7011c2 : 00000000`00000000 00000000`00000000 00000000`00000001 00000000`00000000 : ntdll!RtlEnterCriticalSection+0xab
00000000`002afae0 00000001`3f70159a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000001 : ApplicationC+0x11c2
00000000`002afb10 00000000`776cf56d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ApplicationC+0x159a
00000000`002afb50 00000000`77803281 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`002afb80 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d
1: kd> ub 00000001`3f7011c2
00000001`3f701199 488d0d50ffffff lea rcx,[ApplicationC+0x10f0 (00000001`3f7010f0)]
00000001`3f7011a0 4533c0 xor r8d,r8d
00000001`3f7011a3 33d2 xor edx,edx
00000001`3f7011a5 e88e010000 call ApplicationC+0x1338 (00000001`3f701338)
00000001`3f7011aa b970170000 mov ecx,1770h
00000001`3f7011af ff155b5e0000 call qword ptr [ApplicationC+0x7010 (00000001`3f707010)]
00000001`3f7011b5 488d0d8cad0000 lea rcx,[ApplicationC+0xbf48 (00000001`3f70bf48)]
00000001`3f7011bc ff15465e0000 call qword ptr [ApplicationC+0x7008 (00000001`3f707008)]
Q: Do you do any Linux dump analysis?
A: I did in the past. Many patterns are applicable to Linux and Mac OS X as well and I plan to supplement them with modeling examples too.
Q: Would you make a presentation for user dumps analysis?
A: I plan this too.
Q: Do you recommend the book "Dumps, Bugs and Debugging Forensics: The Adventures of Dr. Debugalov?
A: This is a comic book so it depends on your sense of humour whether you enjoy it or not.
Q: Have you ever had a success using gflags to list stack traces for memory allocations and/or critical section?
A: Yes, I had. For example, look at this article: http://support.citrix.com/article/CTX106970
Q: Are there patterns that apply to user dumps? Just like the ones presented here apply to complete dumps.
A: Yes, of course. Please look at this table (it needs to be updated because it is 2 years old): http://www.dumpanalysis.org/CDAPatterns/CDAPatternsSM.html
Q: Is it possible to effectively debug .NET applications using WinDbg? Do you have any recommendations on how to proceed?
A: Advanced .NET Debugging book is good here. Please see my review: http://www.debuggingexperts.com/book-review-advanced-dotnet-debugging
Howevever, I found it difficult to analyze managed space from complete memory dumps.
Q: What is a good reference to read to learn more about what a "Critical section" is or that shows what it is in other operating systems?
A: There is a good MSDN article I read long time ago to learn about internals of Windows critical sections:
Q: Will you give a session of cordbg, or any other tools for debugging .NET apps?
A: Cannot promise just now. Will think about it.
Q: What is your method when you get a dump file pointing to ntkrnlmp.exe or other system process?
A: It depends on an overall stack trace shape and bugcheck code. But I don't immediately advise to send the dump to Microsoft :-)
Q: What is the ideal path I would take to become an expert debugger like you?
A: Practice as much as possible with critical thinking. You also have to wait some time and don't give up before that :-)
Q: Do you know of any crash dump repository that can be used for practice?
A: I don't know unless you mine Internet. As an alternative you can introduce defects into MSDN samples and analyze crash dumps then.
Q: Is it possible to make any use of an incomplete memory dump?
A: You probably meant inconsistent memory dumps. Sometimes it is possible like in these case studies:
Q: Otherwise do you recommend other debuggers, other than WinDbg? IDAPro or others...
A: If you are able to detect patterns using them or if they produce textual logs that can be analyzed later.
Q: What about SoftIce Debugger?
A: I used it with Windows 3.xx. Is it in a public domain now? I need to Google or Bing that :-)
Q: Syser may be the debugger you are thinking of?
A: I have to try it.
Q: Any interesting use of virtual machines combined with WinDbg on the host? Like analysing early Windows boot?
A: Some virtualization solutions allow a VM snapshot to a complete memory dump conversion like in this case study with an early boot: http://www.dumpanalysis.org/blog/index.php/2010/07/14/crash-dump-analysi...
Q: How have you done to take a "picture" representation of the memory of a particular process?
Q: I didn't get the part where you have shown the dump as BMP. How can I do that?
A: Here's the link to the tool: http://www.dumpanalysis.org/blog/index.php/2007/08/04/visualizing-memory...
Q: Do you recommend any programming languages to learn to become a better debugger?
A: C language definitely. In the past I taught Intel assembly along with some rudimentary C constructs to technical support and escalation engineers. Later I reworked and extended that course in these 2 books:
Q: Do you do any Java debugging or dump analysis?
A: I did a lot of Java debugging in the past, for example while developing these applets: http://vostokov.opentask.com/Applets.htm or while developing a Java-based word processor. At that time all debugging was done via tracing or source code level. Since then I have seen many JVM user dumps but I have never analyzed them in full depth because of the lack of symbol files except obvious issues like blocked threads, CPU spikes, leaks, etc.
Q: How is the weather in Ireland?
A: This is definitely a Turing test question :-)
Q: When I change contexts with .process command, does it change kernel portion of the process too?
A: No, the kernel portion always stays the same.