NetInverse Developers Blog

April 11, 2009
Category: Debugging — Tags: , , , , — admin @ 3:14 pm

SOS Comand: !TraverseHeap

!TraverseHeap [-xml] <filename>

!TraverseHeap writes out a file in a format understood by the CLR Profiler.
You can download the CLR Profiler from this link:

http://www.microsoft.com/downloads/details.aspx?FamilyId=86CE6052-D7F4-4AEB-9B7A-94635BEEBDDA&displaylang=en

It creates a graphical display of the GC heap to help you analyze the state of your application. If you pass the “-xml” flag, the file is instead written out in an easy-to-understand xml format:

    <gcheap>
        <types>
            <type id="1" name="System.String">
            ...
        </types>
        <roots>
            <root kind="handle" address="0x00a73ff0"/>
            <root kind="stack" address="0x0069f0e0"/>
            ...
        </roots>
        <objects>
            <object address="0x00b73030" typeid="1" size="300"/>
            <object address="0x00b75054" typeid="5" size="20">
                <member address="0x00b75088" />
                ...
            </object>
            ...
        </objects>
    </gcheap>

You can break into your process, load SOS, take a snapshot of your heap with this function, then continue.

Category: Debugging — Tags: , , , , — admin @ 3:10 pm
!PrintException [-nested] [<Exception object address>]

This will format fields of any object derived from System.Exception. One of the more useful aspects is that it will format the _stackTrace field, which is a binary array. If _stackTraceString field is not filled in, that can be helpful for debugging. You can of course use !DumpObj on the same exception object to explore more fields.

If called with no parameters, PrintException will look for the last outstanding exception on the current thread and print it. This will be the same exception that shows up in a run of !Threads.

!PrintException will notify you if there are any nested exceptions on the current managed thread. (A nested exception occurs when you throw another exception within a catch handler already being called for another exception).

If there are nested exceptions, you can re-run !PrintException with the “-nested” option to get full details on the nested exception objects. The !Threads command will also tell you which threads have nested exceptions. The abbreviation !pe can be used for brevity.

Category: Debugging — Tags: , , , , — admin @ 3:04 pm

SOS Command: !FinalizeQueue

!FinalizeQueue [-detail]

This command lists the objects registered for finalization. Here is output from a simple program:

0:000> !finalizequeue

SyncBlocks to be cleaned up: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 1

generation 0 has 4 finalizable objects (0015bc90->0015bca0)
generation 1 has 0 finalizable objects (0015bc90->0015bc90)
generation 2 has 0 finalizable objects (0015bc90->0015bc90)

Ready for finalization 0 objects (0015bca0->0015bca0)

Statistics:

      MT    Count TotalSize Class Name
5ba6cf78        1        24 Microsoft.Win32.SafeHandles.SafeFileHandle
5ba5db04        1        68 System.Threading.Thread
5ba73e28        2       112 System.IO.StreamWriter
Total 4 objects

The GC heap is divided into generations, and objects are listed accordingly. We see that only generation 0 (the youngest generation) has any objects registered for finalization. The notation “(0015bc90->0015bca0)” means that if you look at memory in that range, you’ll see the object pointers that are registered:

0:000> dd 15bc90 15bca0-4

0015bc90  00a743f4 00a79f00 00a7b3d8 00a7b47c

You could run !DumpObj on any of those pointers to learn more. In this example, there are no objects ready for finalization, presumably because they still have roots (You can use !GCRoot to find out). The statistics section provides a higher-level summary of the objects registered for finalization. Note that objects ready for finalization are also included in the statistics (if any).

If you pass -detail then you get extra information on any SyncBlocks that need to be cleaned up, and on any RuntimeCallableWrappers (RCWs) that await cleanup. Both of these data structures are cached and cleaned up by the finalizer thread when it gets a chance to run.

Category: Debugging — Tags: , , , , — admin @ 9:59 am

SOS Command: !ObjSize [<Object address>]

With no parameters, !ObjSize lists the size of all objects found on managed threads. It also enumerates all GCHandles in the process, and totals the size of any objects pointed to by those handles. In calculating object size, !ObjSize includes the size of all child objects in addition to the parent.

For example, !DumpObj lists a size of 20 bytes for this Customer object:

0:000> !do a79d40

Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes
 (C:pubunittest.exe)

Fields:
      MT    Field   Offset                 Type       Attr    Value Name
009038ec  4000008        4                CLASS   instance 00a79ce4 name
009038ec  4000009        8                CLASS   instance 00a79d2c bank
009038ec  400000a        c       System.Boolean   instance        1 valid

but !ObjSize lists 152 bytes:

0:000> !ObjSize a79d40
sizeof(00a79d40) =      152 (    0x98) bytes (Customer)

This is because a Customer points to a Bank, has a name, and the Bank points to an Address string. You can use !ObjSize to identify any particularly large objects, such as a managed cache in a web server. Note: above information was from SOS help.

April 10, 2009
Category: Debugging — Tags: , , , , — admin @ 5:45 pm

SOS Command: !DumpHeap

!DumpHeap [-stat]
          [-strings]
          [-short]
          [-min <size>]
          [-max <size>]
          [-thinlock]
          [-startAtLowerBound]
          [-mt <MethodTable address>]
          [-type <partial type name>]
          [start [end]]

!DumpHeap is a powerful command that traverses the garbage collected heap, collection statistics about objects. With it’s various options, it can look for particular types, restrict to a range, or look for ThinLocks (see !SyncBlk documentation). Finally, it will provide a warning if it detects excessive fragmentation in the GC heap.

When called without options, the output is first a list of objects in the heap, followed by a report listing all the types found, their size and number:

0:000> !dumpheap

 Address       MT     Size

00a71000 0015cde8       12 Free
00a71024 5ba58328       68
...

total 619 objects

Statistics:
      MT    Count TotalSize Class Name

5ba7607c        1        12 System.Security.Permissions.HostProtectionResource
5ba75d54        1        12 System.Security.Permissions.SecurityPermissionFlag
5ba61f18        1        12 System.Collections.CaseInsensitiveComparer
...

0015cde8        6     10260      Free
5ba57bf8      318     18136 System.String
...

“Free” objects are simply regions of space the garbage collector can use later. If 30% or more of the heap contains “Free” objects, the process may suffer fromheap fragmentation. This is usually caused by pinning objects for a long time combined with a high rate of allocation. Here is example output where !DumpHeap provides a warning about fragmentation:

<After the Statistics section> Fragmented blocks larger than 1MB:

    Addr     Size Followed by
00a780c0    1.5MB    00bec800 System.Byte[]
00da4e38    1.2MB    00ed2c00 System.Byte[]
00f16df0    1.2MB    01044338 System.Byte[]

The arguments in detail:

-stat     Restrict the output to the statistical type summary
-strings  Restrict the output to a statistical string value summary
-short    Limits output to just the address of each object. This allows you
          to easily pipe output from the command to another debugger
          command for automation.
-min      Ignore objects less than the size given in bytes
-max      Ignore objects larger than the size given in bytes
-thinlock Report on any ThinLocks (an efficient locking scheme, see !SyncBlk
          documentation for more info)
-startAtLowerBound
          Force heap walk to begin at lower bound of a supplied address range.
          (During plan phase, the heap is often not walkable because objects
          are being moved. In this case, DumpHeap may report spurious errors,
          in particular bad objects. It may be possible to traverse more of
          the heap after the reported bad object. Even if you specify an
          address range, !DumpHeap will start its walk from the beginning of
          the heap by default. If it finds a bad object before the specified
          range, it will stop before displaying the part of the heap in which
          you are interested. This switch will force !DumpHeap to begin its
          walk at the specified lower bound. You must supply the address of a
          good object as the lower bound for this to work. Display memory at
          the address of the bad object to manually find the next method
          table (use !dumpmt to verify). If the GC is currently in a call to
          memcopy, You may also be able to find the next object's address by
          adding the size to the start address given as parameters.)
-mt       List only those objects with the MethodTable given
-type     List only those objects whose type name is a substring match of the
          string provided.
start     Begin listing from this address
end       Stop listing at this address

A special note about -type: Often, you’d like to find not only Strings, but System.Object arrays that are constrained to contain Strings. (”new String[100]” actually creates a System.Object array, but it can only hold System.String object pointers). You can use -type in a special way to find these arrays. Just pass “-type System.String[]” and those Object arrays will be returned. More generally, “-type <Substring of interesting type>[]“. The start/end parameters can be obtained from the output of !EEHeap -gc. For example, if you only want to list objects in the large heap segment:

0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00c32754
generation 1 starts at 0x00c32748
generation 2 starts at 0x00a71000

 segment    begin allocated     size
00a70000 00a71000  010443a8 005d33a8(6108072)
Large object heap starts at 0x01a71000
 segment    begin allocated     size
01a70000 01a71000  01a75000 0x00004000(16384)

Total Size  0x5d73a8(6124456)
------------------------------
GC Heap Size  0x5d73a8(6124456)

0:000> !dumpheap 1a71000 1a75000

 Address       MT     Size
01a71000 5ba88bd8     2064
01a71810 0019fe48     2032 Free
01a72000 5ba88bd8     4096
01a73000 0019fe48     4096 Free
01a74000 5ba88bd8     4096
total 5 objects

Statistics:
      MT    Count TotalSize Class Name
0019fe48        2      6128      Free
5ba88bd8        3     10256 System.Object[]

Total 5 objects

Finally, if gc heap corruption is present, you may see an error like this:

0:000> !dumpheap -stat

object 00a73d24: does not have valid MT
curr_object : 00a73d24
Last good object: 00a73d14
----------------

That indicates a serious problem. See the help for !VerifyHeap for more information on diagnosing the cause.

Note: above information is compiled based on SOS online help.

April 9, 2009
Category: Debugging — Tags: , , , , — admin @ 9:17 pm

SOS Command: !GCRoot

!GCRoot [-nostacks] <Object address>

!GCRoot looks for references (or roots) to an object. These can exist in four places:

  1. On the stack
  2. Within a GC Handle
  3. In an object ready for finalization
  4. As a member of an object found in 1, 2 or 3 above.

First, all stacks will be searched for roots, then handle tables, and finally the freachable queue of the finalizer. Some caution about the stack roots: !GCRoot doesn’t attempt to determine if a stack root it encountered is valid or is old (discarded) data. You would have to use !CLRStack and !U to disassemble the frame that the local or argument value belongs to in order to determine if it is still in use.

Because people often want to restrict the search to gc handles and freachable objects, there is a -nostacks option.

Sample output:

!GCRoot 012c2c10
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Error during command: Warning. Extension is using a callback which Visual Studio does not implement.

Scan Thread 580 OSTHread 244
ESP:12f05c:Root:012c2c10(System.Object[])
ESP:12f194:Root:012c2c10(System.Object[])
ESP:12f414:Root:012c2c10(System.Object[])
ESP:12f444:Root:012c2c10(System.Object[])
ESP:12f534:Root:012c2c10(System.Object[])
ESP:12f6e0:Root:012c2c10(System.Object[])
ESP:12f708:Root:012c2c10(System.Object[])
Scan Thread 3448 OSTHread d78
DOMAIN(0015D338):HANDLE(Strong):9111a8:Root:012c2c10(System.Object[])
Category: Debugging — Tags: , , , , — admin @ 9:08 pm

SOS Command: !DumpObj

!DumpObj [-nofields] <object address>

!DumpObj allows you to examine the fields of an object, as well as learn important properties of the object such as the EEClass, the MethodTable, and the size.

You might find an object pointer by running !DumpStackObjects and choosing from the resultant list. Here is a simple object:

!DumpObj a79d40
Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes

Fields:
      MT    Field   Offset                 Type  VT     Attr    Value Name
009038ec  4000008        4             Customer   0 instance 00a79ce4 name
009038ec  4000009        8                 Bank   0 instance 00a79d2c bank

Note that fields of type Customer and Bank are themselves objects, and you can run !DumpObj on them too. You could look at the field directly in memory using the offset given. "dd a79d40+8 l1" would allow you to look at the bank field directly. Be careful about using this to set memory breakpoints, since objects can move around in the garbage collected heap.

What else can you do with an object? You might run !GCRoot, to determine what roots are keeping it alive. Or you can find all objects of that type with "!DumpHeap -type Customer".

The column VT contains the value 1 if the field is a valuetype structure, and 0 if the field contains a pointer to another object. For valuetypes, you can take the MethodTable pointer in the MT column, and the Value and pass them to the command !DumpVC.

The abbreviation !do can be used for brevity.
The arguments in detail:

-nofields: do not print fields of the object, useful for objects like String

Category: Debugging — Tags: , , , , — admin @ 8:59 pm

SOS Command: !DumpStackObjects

!DumpStackObjects [-verify] [top stack [bottom stack]]

This command will display any managed objects it finds within the bounds of the current stack. Combined with the stack tracing commands like K and !CLRStack, it is a good aid to determining the values of locals and parameters.

If you use the -verify option, each non-static CLASS field of an object candidate is validated. This helps to eliminate false positives. It is not on by default because very often in a debugging scenario, you are interested in objects with invalid fields.

The abbreviation !dso can be used for brevity.

Sample output:

!dumpstackobjects
OS Thread Id: 0x244 (580)
ESP/REG  Object   Name
0012f05c 012c2c10 System.Object[]    (System.String[])
0012f194 012c2c10 System.Object[]    (System.String[])
0012f3cc 012c2c34 System.RuntimeType
0012f3d0 012c2c34 System.RuntimeType
0012f3d4 012c2c20 System.RuntimeType
0012f3e0 012c2c20 System.RuntimeType
0012f414 012c2c10 System.Object[]    (System.String[])
0012f444 012c2c10 System.Object[]    (System.String[])
0012f534 012c2c10 System.Object[]    (System.String[])
0012f6e0 012c2c10 System.Object[]    (System.String[])
0012f708 012c2c10 System.Object[]    (System.String[])
Category: Debugging — Tags: , , , , — admin @ 8:29 pm

SOS Command: !BPMD

!BPMD <module name> <method name>
!BPMD -md <MethodDesc>

!BPMD provides managed breakpoint support. If it can resolve the method name to a loaded, jitted or ngen’d function it will create a breakpoint with “bp”.

If not then either the module that contains the method hasn’t been loaded yet or the module is loaded, but the function is not jitted yet. In these cases, !bpmd asks the Windows Debugger to receive CLR Notifications, and waits to receive news of module loads and JITs, at which time it will try to resolve the function to a breakpoint.

Sample output:

!bpmd LinqSelectTest.exe IEnumerableTest.Program.Main
Found 1 methods...
MethodDesc = 00933000
Setting breakpoint: bp 00FF0070 [IEnumerableTest.Program.Main(System.String[])]

This brings up a good question: “I want to set a breakpoint on the main method of my application, but SOS doesn’t work until the runtime is loaded. How can I do this?”

  1) Start the debugger and type: 

       sxe -c "" clrn

  2) g

  3) You'll get the following notification from the debugger:

     "CLR notification: module 'mscorlib' loaded"

  4) Now you can load SOS and use commands. Type

       .loadby sos mscorwks

     then

       !bpmd myapp.exe MyNamespace.MyApp.Main

  5) g

  6) You will stop at the start of MyApp.Main. If you type "bl" you will see the breakpoint listed.

!BPMD works equally well with generic types. Adding a breakpoint on a generic type sets breakpoints on all already JIT-ted generic methods and sets a pending breakpoint for any instantiation that will be JIT-ted in the future.

Example for generics:
Given the following two classes:

	class G3<T1, T2, T3>
	{
		...
		public void F(T1 p1, T2 p2, T3 p3)
		{ ... }
	}

	public class G1<T> {
		// static method
		static public void G<W>(W w)
		{ ... }
	}

One would issue the following commands to set breapoints on G3.F() and G1.G():

	!bpmd myapp.exe G3`3.F
	!bpmd myapp.exe G1`1.G

!BPMD does not accept offsets nor parameters in the method name. If there are overloaded methods, !bpmd will set a breakpoint for all of them. In the case of hosted environments such as SQL, the module name may be complex, like 'price, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.

For this case, just be sure to surround the module name with single quotes,
like: !bpmd 'price, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' Price.M2

Category: Debugging — Tags: , , , , — admin @ 5:46 pm

Son of Strike - SOS is a debugging extension that lets developers peer into the internal workings of the Common Language Runtime (CLR) in real time and in mini dumps.

The following FAQ is compiled from the output from Son of Strike - SOS debugger extension.

.load sos
!help faq

>> Where can I get the right version of SOS for my build?

If you are running version 1.1 or 2.0 of the CLR, SOS.DLL is installed in the same directory as the main CLR dll (MSCORWKS.DLL). Newer versions of the Windows Debugger provide a command to make it easy to load the right copy of SOS.DLL:

.loadby sos mscorwks

That will load the SOS extension DLL from the same place that MSCORWKS.DLL is loaded in the process. You shouldn’t attempt to use a version of SOS.DLL that doesn’t match the version of MSCORWKS.DLL. You can find the version of MSCORWKS.DLL by running "lm v m mscorwks" in the debugger.

If you are using a dump file created on another machine, it is a little bit more complex. You need to make sure the mscordacwks.dll file that came with that install is on your symbol path, and you need to load the corresponding version of sos.dll (typing .load rather than using the .loadby shortcut). Within the Microsoft corpnet, we keep tagged versions of mscordacwks.dll, with names like mscordacwks__.dll that the Windows Debugger can load. If you have the correct symbol path to the binaries for that version of the Runtime, the Windows Debugger will load the correct mscordacwks.dll file.

>> I have a chicken and egg problem. I want to use SOS commands, but the CLR isn’t loaded yet. What can I do?

In the debugger at startup you can type: "sxe clrn" Let the program run, and it will stop with the notice

"CLR notification: module 'mscorlib' loaded"

At this time you can use SOS commands. To turn off spurious notifications, type: "sxd clrn"

>> I got the following error message. Now what?

0:000> .loadby sos mscorwks

0:000> !DumpStackObjects

Failed to find runtime DLL (mscorwks.dll), 0x80004005

Extension commands need mscorwks.dll in order to have something to do.

0:000>

This means that the CLR is not loaded yet, or has been unloaded. You need to wait until your managed program is running in order to use these commands. If you have just started the program a good way to do this is to type bp mscorwks!EEStartup "g @$ra"

in the debugger, and let it run. After the function EEStartup is finished, there will be a minimal managed environment for executing SOS commands.

>> I have a partial memory minidump, and !DumpObj doesn’t work. Why?

In order to run SOS commands, many CLR data structures need to be traversed. When creating a minidump without full memory, special functions are called at dump creation time to bring those structures into the minidump, and allow a minimum set of SOS debugging commands to work. At this time, those commands that can provide full or partial output are:

CLRStack
Threads
Help
PrintException
EEVersion

For a minidump created with this minimal set of functionality in mind, you will get an error message when running any other commands. A full memory dump (obtained with “.dump /ma ” in the Windows Debugger) is often the best way to debug a managed program at this level.

>> What other tools can I use to find my bug?

Turn on Managed Debugging Assistants. These enable additional runtime diagnostics, particularly in the area of PInvoke/Interop. Adam Nathan has written some great information about that: Http://blogs.gotdotnet.com/anathan/categoryview.aspx/Debugging

« Newer PostsOlder Posts »

©2009 NetInverse. All rights reserved. Powered by WordPress