IanG on Tap

I recently needed to get a set of stack traces for all of the threads of a running process. Moreover, I wanted to be able to do this in such a way that the process would be able to continue running afterwards. And I wanted to install as little as possible on the target machine in order to achieve this.

Initially I had been looking at generating a minidump. There is a tool called 'userdump' available for Windows 2000 that can give you a dump file of a running process. However, I needed to run on Windows 2003, and it wasn't clear that this utility was supported there. Moreover, it relies on installing a device driver, and I was reluctant to install a driver on the target machine just to be able to get a dump of a running process.

Also, I knew that Windows 2003 supports the MiniDumpWriteDump API. This takes a process handle, and will write a memory dump to the file of your choice. Given that this feature is built into the OS, it seemed like it should be unnecessary to install a device driver, which is what userdump does.

I wrote a simple program to use this API, and it seemed to work fine - I could get a dump from any process...on Windows XP. However, on Windows 2003, it only worked for processes running under the same user account that I ran my dump program from - I was getting security errors for any other account. This was a problem because I need to use this tool to examine ASP.NET worker processes which would be running as a specific account, and not one that I would be able to run this tool as. Keith Brown suggested a couple of things - making sure that the SeDebugPrivilege was enabled, adjusting the DACL on the process, taking ownership of it first if necessary. But none of these things seemed to help.

Fortunately, Eric Fleischman pointed out to me that a much simpler solution exists - just use the debugger. I had initially considered this, but rejected it because I wasn't going to be allowed to install anything much on the server this would be used on, and definitely not anything resembling developer tools, so the debugger was right out.

However, the part that I had forgotten is that nstd, the command line debugger, is installed with Windows 2003 as standard.

This means that you can always defer to the debugger for tasks such as obtaining stack traces. Here's a script I'm using to get a stack trace from a running process, called DumpStacks.cmd:

@echo off
if "%1" == "" goto :usage
if NOT "%2" == "" goto :run

:usage
echo Usage: %0 ^<pid^> ^<dumpfile^>
goto :quit

:run
ntsd -pv -p %1 -cf DumpStacks.debugscript -logo %2

:quit

Notice that we're passing the -pv option to ntsd here - this indicates that it should attach non-invasively. This disables certain debugger operations, but ensures that the state of the process will not be changed by the debugger. (Although it will suspend the process if necessary.) This script relies on the a file called DumpStacks.debugscript containing the following:

.load C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\sos.dll
~*k
!EEStack
.detach
q

This loads sos.dll (Son of Strike), the .NET debugging DLL, which also appears to be installed on Windows 2003. It then runs two commands, one to get a complete list of unmanaged stack traces, and another to get a complete list of managed stack traces. Then it detaches the debugger, enabling the underlying process to continue, and then quits the debugger. The result is that you can run the command like this:

DumpStacks.cmd 1234 StackTraces.txt

This will attach to process ID 1234 (assuming you have a process running with that PID), and write a list of stack traces into StackTraces.txt. It will probably also dump a load of complaints that it can't find all the symbol files unless you set up various environment variables to tell it where those are. But the stack traces will still be reasonably informative, particularly for .NET code, as it will normally be able to use the metadata built into the executables to give you the right names for all of the functions in the stack trace.

Many thanks to Eric for pointing out that ntsd was in fact an option, and that I didn't need to go down the MiniDumpWriteDump path after all.