Setting up a free Microsoft debugging environment on Windows XP

This is a short guide on how to debug applications or dynamic link libraries created by the post free Microsoft C++ compile environment.

The following tools need to be installed:

• Microsoft Debug Tools (13MB): c:\msdev\debug
• Windows XP with Service Pack 2 symbols (195MB)

Then extend your environment’s LIB path by adding the Visual Studio .NET 2003 lib folder. Visual Studio .NET 2003 was installed during the former .NET SDK installation, unfortunately the path, specified for the .NET SDK is not taken as installation path for Visual Studio but the usual program folder is used instead, in my case C:\Programme\Microsoft Visual Studio .NET 2003. Find out the path’s short name with dir /X and add the following line with your specific path to the setenv.cmd script below the three call statements:

set LIB=C:\PROGRA~1\MICROS~1.NET\Vc7\lib;%LIB%  

Add the parameters /XP32 /DEBUG to the call c:\msdev\platsdk\SetEnv.bat:

call c:\msdev\platsdk\SetEnv.bat /XP32 /DEBUG  

For having the debugging tools on the path the following line must be added:

set PATH=C:\msdev\debug;%PATH%  

Then go to the traceser example of the Detours package for testing our debug environment setup by recompiling the example with debug information:

cd detours\samples\traceser

nmake clean  
set CFLAGS=/nologo /Zi $(CLIB) /Gi- /Gm- /W4 /WX \  
      /FR "/I$(DTRINCD)" "/I$(INCD)" /Yd  
set CLIB=/MTd  
nmake /E  
set CFLAGS=  
set CLIB=  

The /Yd compiler switch must be specified in order to compile the example in debug mode. The CLIB variable is set by the Makefile to /MT which is the release version of the multi-threaded libc. Changing the CLIB variable to /MTd uses the debug version when compiling the example. The switch /E passed to the nmake tool overrides the Makefile variables with the environment settings.

For starting the DataRecorder application together with the traceser interception DLL the following commands must be executed. First start the logging service which receives the messages from the interception layer:

start ..\bin\syelogd.exe /e /s traceser.log  

Then start the application with the following command:

start ..\bin\withdll -d:..\bin\traceser.dll \  
      c:\Programme\DataRecorder\DataRecorder.exe  

Finally start the Microsoft debugger WinDbg and attach to the DataRecorder process. For source level debugging the following settings need to be done:

• Set the symbol file path (File->Symbol File Path…, CTRL-S) to C:\msdev\detours\lib; C:\msdev\detours\samples\traceser; C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib
• Set the source path (File->Source File Path…, CTRL-P) to C:\msdev\detours\samples\traceser; C:\msdev\detours\src
• Set a breakpoint to the method Mine_WriteFile in the traceser library: bp traceser!Mine_WriteFile
• Start the application (Debug->Go, F5)

The debugger should stop at the specified breakpoint, showing the highlighted source code position in the source window.

For Just-In Time Debugging set the following registry values:

• HKLM\Software\Microsoft\Windows NT\CurrentVersion\AeDebug=Windbg
• HKLM\Software\Microsoft.NETFramework\DbgJITDebugLaunchSetting=2
• HKLM\Software\Microsoft.NETFramework\DbgManagedDebugger=Windbg

For further information on using the WinDbg debugger, see the online help. The debugger itself is a complex beast and quite cryptic to use. Although it has a graphical user interface, a strong similarity to command line GDB debugging is obvious.