NetInverse Developers Blog

April 11, 2009
Category: Debugging — Tags: , , , , — admin @ 9:59 am

SOS Command: !ObjSize [<Object address>]

With no parameters, !ObjSize lists the size of all objects found on managed threads. It also enumerates all GCHandles in the process, and totals the size of any objects pointed to by those handles. In calculating object size, !ObjSize includes the size of all child objects in addition to the parent.

For example, !DumpObj lists a size of 20 bytes for this Customer object:

0:000> !do a79d40

Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes
 (C:pubunittest.exe)

Fields:
      MT    Field   Offset                 Type       Attr    Value Name
009038ec  4000008        4                CLASS   instance 00a79ce4 name
009038ec  4000009        8                CLASS   instance 00a79d2c bank
009038ec  400000a        c       System.Boolean   instance        1 valid

but !ObjSize lists 152 bytes:

0:000> !ObjSize a79d40
sizeof(00a79d40) =      152 (    0x98) bytes (Customer)

This is because a Customer points to a Bank, has a name, and the Bank points to an Address string. You can use !ObjSize to identify any particularly large objects, such as a managed cache in a web server. Note: above information was from SOS help.

April 10, 2009
Category: Debugging — Tags: , , , , — admin @ 5:45 pm

SOS Command: !DumpHeap

!DumpHeap [-stat]
          [-strings]
          [-short]
          [-min <size>]
          [-max <size>]
          [-thinlock]
          [-startAtLowerBound]
          [-mt <MethodTable address>]
          [-type <partial type name>]
          [start [end]]

!DumpHeap is a powerful command that traverses the garbage collected heap, collection statistics about objects. With it’s various options, it can look for particular types, restrict to a range, or look for ThinLocks (see !SyncBlk documentation). Finally, it will provide a warning if it detects excessive fragmentation in the GC heap.

When called without options, the output is first a list of objects in the heap, followed by a report listing all the types found, their size and number:

0:000> !dumpheap

 Address       MT     Size

00a71000 0015cde8       12 Free
00a71024 5ba58328       68
...

total 619 objects

Statistics:
      MT    Count TotalSize Class Name

5ba7607c        1        12 System.Security.Permissions.HostProtectionResource
5ba75d54        1        12 System.Security.Permissions.SecurityPermissionFlag
5ba61f18        1        12 System.Collections.CaseInsensitiveComparer
...

0015cde8        6     10260      Free
5ba57bf8      318     18136 System.String
...

“Free” objects are simply regions of space the garbage collector can use later. If 30% or more of the heap contains “Free” objects, the process may suffer fromheap fragmentation. This is usually caused by pinning objects for a long time combined with a high rate of allocation. Here is example output where !DumpHeap provides a warning about fragmentation:

<After the Statistics section> Fragmented blocks larger than 1MB:

    Addr     Size Followed by
00a780c0    1.5MB    00bec800 System.Byte[]
00da4e38    1.2MB    00ed2c00 System.Byte[]
00f16df0    1.2MB    01044338 System.Byte[]

The arguments in detail:

-stat     Restrict the output to the statistical type summary
-strings  Restrict the output to a statistical string value summary
-short    Limits output to just the address of each object. This allows you
          to easily pipe output from the command to another debugger
          command for automation.
-min      Ignore objects less than the size given in bytes
-max      Ignore objects larger than the size given in bytes
-thinlock Report on any ThinLocks (an efficient locking scheme, see !SyncBlk
          documentation for more info)
-startAtLowerBound
          Force heap walk to begin at lower bound of a supplied address range.
          (During plan phase, the heap is often not walkable because objects
          are being moved. In this case, DumpHeap may report spurious errors,
          in particular bad objects. It may be possible to traverse more of
          the heap after the reported bad object. Even if you specify an
          address range, !DumpHeap will start its walk from the beginning of
          the heap by default. If it finds a bad object before the specified
          range, it will stop before displaying the part of the heap in which
          you are interested. This switch will force !DumpHeap to begin its
          walk at the specified lower bound. You must supply the address of a
          good object as the lower bound for this to work. Display memory at
          the address of the bad object to manually find the next method
          table (use !dumpmt to verify). If the GC is currently in a call to
          memcopy, You may also be able to find the next object's address by
          adding the size to the start address given as parameters.)
-mt       List only those objects with the MethodTable given
-type     List only those objects whose type name is a substring match of the
          string provided.
start     Begin listing from this address
end       Stop listing at this address

A special note about -type: Often, you’d like to find not only Strings, but System.Object arrays that are constrained to contain Strings. (”new String[100]” actually creates a System.Object array, but it can only hold System.String object pointers). You can use -type in a special way to find these arrays. Just pass “-type System.String[]” and those Object arrays will be returned. More generally, “-type <Substring of interesting type>[]“. The start/end parameters can be obtained from the output of !EEHeap -gc. For example, if you only want to list objects in the large heap segment:

0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00c32754
generation 1 starts at 0x00c32748
generation 2 starts at 0x00a71000

 segment    begin allocated     size
00a70000 00a71000  010443a8 005d33a8(6108072)
Large object heap starts at 0x01a71000
 segment    begin allocated     size
01a70000 01a71000  01a75000 0x00004000(16384)

Total Size  0x5d73a8(6124456)
------------------------------
GC Heap Size  0x5d73a8(6124456)

0:000> !dumpheap 1a71000 1a75000

 Address       MT     Size
01a71000 5ba88bd8     2064
01a71810 0019fe48     2032 Free
01a72000 5ba88bd8     4096
01a73000 0019fe48     4096 Free
01a74000 5ba88bd8     4096
total 5 objects

Statistics:
      MT    Count TotalSize Class Name
0019fe48        2      6128      Free
5ba88bd8        3     10256 System.Object[]

Total 5 objects

Finally, if gc heap corruption is present, you may see an error like this:

0:000> !dumpheap -stat

object 00a73d24: does not have valid MT
curr_object : 00a73d24
Last good object: 00a73d14
----------------

That indicates a serious problem. See the help for !VerifyHeap for more information on diagnosing the cause.

Note: above information is compiled based on SOS online help.

April 9, 2009
Category: Debugging — Tags: , , , , — admin @ 9:17 pm

SOS Command: !GCRoot

!GCRoot [-nostacks] <Object address>

!GCRoot looks for references (or roots) to an object. These can exist in four places:

  1. On the stack
  2. Within a GC Handle
  3. In an object ready for finalization
  4. As a member of an object found in 1, 2 or 3 above.

First, all stacks will be searched for roots, then handle tables, and finally the freachable queue of the finalizer. Some caution about the stack roots: !GCRoot doesn’t attempt to determine if a stack root it encountered is valid or is old (discarded) data. You would have to use !CLRStack and !U to disassemble the frame that the local or argument value belongs to in order to determine if it is still in use.

Because people often want to restrict the search to gc handles and freachable objects, there is a -nostacks option.

Sample output:

!GCRoot 012c2c10
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Error during command: Warning. Extension is using a callback which Visual Studio does not implement.

Scan Thread 580 OSTHread 244
ESP:12f05c:Root:012c2c10(System.Object[])
ESP:12f194:Root:012c2c10(System.Object[])
ESP:12f414:Root:012c2c10(System.Object[])
ESP:12f444:Root:012c2c10(System.Object[])
ESP:12f534:Root:012c2c10(System.Object[])
ESP:12f6e0:Root:012c2c10(System.Object[])
ESP:12f708:Root:012c2c10(System.Object[])
Scan Thread 3448 OSTHread d78
DOMAIN(0015D338):HANDLE(Strong):9111a8:Root:012c2c10(System.Object[])
Category: Debugging — Tags: , , , , — admin @ 9:08 pm

SOS Command: !DumpObj

!DumpObj [-nofields] <object address>

!DumpObj allows you to examine the fields of an object, as well as learn important properties of the object such as the EEClass, the MethodTable, and the size.

You might find an object pointer by running !DumpStackObjects and choosing from the resultant list. Here is a simple object:

!DumpObj a79d40
Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes

Fields:
      MT    Field   Offset                 Type  VT     Attr    Value Name
009038ec  4000008        4             Customer   0 instance 00a79ce4 name
009038ec  4000009        8                 Bank   0 instance 00a79d2c bank

Note that fields of type Customer and Bank are themselves objects, and you can run !DumpObj on them too. You could look at the field directly in memory using the offset given. "dd a79d40+8 l1" would allow you to look at the bank field directly. Be careful about using this to set memory breakpoints, since objects can move around in the garbage collected heap.

What else can you do with an object? You might run !GCRoot, to determine what roots are keeping it alive. Or you can find all objects of that type with "!DumpHeap -type Customer".

The column VT contains the value 1 if the field is a valuetype structure, and 0 if the field contains a pointer to another object. For valuetypes, you can take the MethodTable pointer in the MT column, and the Value and pass them to the command !DumpVC.

The abbreviation !do can be used for brevity.
The arguments in detail:

-nofields: do not print fields of the object, useful for objects like String

Category: Debugging — Tags: , , , , — admin @ 8:59 pm

SOS Command: !DumpStackObjects

!DumpStackObjects [-verify] [top stack [bottom stack]]

This command will display any managed objects it finds within the bounds of the current stack. Combined with the stack tracing commands like K and !CLRStack, it is a good aid to determining the values of locals and parameters.

If you use the -verify option, each non-static CLASS field of an object candidate is validated. This helps to eliminate false positives. It is not on by default because very often in a debugging scenario, you are interested in objects with invalid fields.

The abbreviation !dso can be used for brevity.

Sample output:

!dumpstackobjects
OS Thread Id: 0x244 (580)
ESP/REG  Object   Name
0012f05c 012c2c10 System.Object[]    (System.String[])
0012f194 012c2c10 System.Object[]    (System.String[])
0012f3cc 012c2c34 System.RuntimeType
0012f3d0 012c2c34 System.RuntimeType
0012f3d4 012c2c20 System.RuntimeType
0012f3e0 012c2c20 System.RuntimeType
0012f414 012c2c10 System.Object[]    (System.String[])
0012f444 012c2c10 System.Object[]    (System.String[])
0012f534 012c2c10 System.Object[]    (System.String[])
0012f6e0 012c2c10 System.Object[]    (System.String[])
0012f708 012c2c10 System.Object[]    (System.String[])
Category: Debugging — Tags: , , , , — admin @ 8:29 pm

SOS Command: !BPMD

!BPMD <module name> <method name>
!BPMD -md <MethodDesc>

!BPMD provides managed breakpoint support. If it can resolve the method name to a loaded, jitted or ngen’d function it will create a breakpoint with “bp”.

If not then either the module that contains the method hasn’t been loaded yet or the module is loaded, but the function is not jitted yet. In these cases, !bpmd asks the Windows Debugger to receive CLR Notifications, and waits to receive news of module loads and JITs, at which time it will try to resolve the function to a breakpoint.

Sample output:

!bpmd LinqSelectTest.exe IEnumerableTest.Program.Main
Found 1 methods...
MethodDesc = 00933000
Setting breakpoint: bp 00FF0070 [IEnumerableTest.Program.Main(System.String[])]

This brings up a good question: “I want to set a breakpoint on the main method of my application, but SOS doesn’t work until the runtime is loaded. How can I do this?”

  1) Start the debugger and type: 

       sxe -c "" clrn

  2) g

  3) You'll get the following notification from the debugger:

     "CLR notification: module 'mscorlib' loaded"

  4) Now you can load SOS and use commands. Type

       .loadby sos mscorwks

     then

       !bpmd myapp.exe MyNamespace.MyApp.Main

  5) g

  6) You will stop at the start of MyApp.Main. If you type "bl" you will see the breakpoint listed.

!BPMD works equally well with generic types. Adding a breakpoint on a generic type sets breakpoints on all already JIT-ted generic methods and sets a pending breakpoint for any instantiation that will be JIT-ted in the future.

Example for generics:
Given the following two classes:

	class G3<T1, T2, T3>
	{
		...
		public void F(T1 p1, T2 p2, T3 p3)
		{ ... }
	}

	public class G1<T> {
		// static method
		static public void G<W>(W w)
		{ ... }
	}

One would issue the following commands to set breapoints on G3.F() and G1.G():

	!bpmd myapp.exe G3`3.F
	!bpmd myapp.exe G1`1.G

!BPMD does not accept offsets nor parameters in the method name. If there are overloaded methods, !bpmd will set a breakpoint for all of them. In the case of hosted environments such as SQL, the module name may be complex, like 'price, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.

For this case, just be sure to surround the module name with single quotes,
like: !bpmd 'price, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' Price.M2

Category: Debugging — Tags: , , , , — admin @ 5:46 pm

Son of Strike - SOS is a debugging extension that lets developers peer into the internal workings of the Common Language Runtime (CLR) in real time and in mini dumps.

The following FAQ is compiled from the output from Son of Strike - SOS debugger extension.

.load sos
!help faq

>> Where can I get the right version of SOS for my build?

If you are running version 1.1 or 2.0 of the CLR, SOS.DLL is installed in the same directory as the main CLR dll (MSCORWKS.DLL). Newer versions of the Windows Debugger provide a command to make it easy to load the right copy of SOS.DLL:

.loadby sos mscorwks

That will load the SOS extension DLL from the same place that MSCORWKS.DLL is loaded in the process. You shouldn’t attempt to use a version of SOS.DLL that doesn’t match the version of MSCORWKS.DLL. You can find the version of MSCORWKS.DLL by running "lm v m mscorwks" in the debugger.

If you are using a dump file created on another machine, it is a little bit more complex. You need to make sure the mscordacwks.dll file that came with that install is on your symbol path, and you need to load the corresponding version of sos.dll (typing .load rather than using the .loadby shortcut). Within the Microsoft corpnet, we keep tagged versions of mscordacwks.dll, with names like mscordacwks__.dll that the Windows Debugger can load. If you have the correct symbol path to the binaries for that version of the Runtime, the Windows Debugger will load the correct mscordacwks.dll file.

>> I have a chicken and egg problem. I want to use SOS commands, but the CLR isn’t loaded yet. What can I do?

In the debugger at startup you can type: "sxe clrn" Let the program run, and it will stop with the notice

"CLR notification: module 'mscorlib' loaded"

At this time you can use SOS commands. To turn off spurious notifications, type: "sxd clrn"

>> I got the following error message. Now what?

0:000> .loadby sos mscorwks

0:000> !DumpStackObjects

Failed to find runtime DLL (mscorwks.dll), 0x80004005

Extension commands need mscorwks.dll in order to have something to do.

0:000>

This means that the CLR is not loaded yet, or has been unloaded. You need to wait until your managed program is running in order to use these commands. If you have just started the program a good way to do this is to type bp mscorwks!EEStartup "g @$ra"

in the debugger, and let it run. After the function EEStartup is finished, there will be a minimal managed environment for executing SOS commands.

>> I have a partial memory minidump, and !DumpObj doesn’t work. Why?

In order to run SOS commands, many CLR data structures need to be traversed. When creating a minidump without full memory, special functions are called at dump creation time to bring those structures into the minidump, and allow a minimum set of SOS debugging commands to work. At this time, those commands that can provide full or partial output are:

CLRStack
Threads
Help
PrintException
EEVersion

For a minidump created with this minimal set of functionality in mind, you will get an error message when running any other commands. A full memory dump (obtained with “.dump /ma ” in the Windows Debugger) is often the best way to debug a managed program at this level.

>> What other tools can I use to find my bug?

Turn on Managed Debugging Assistants. These enable additional runtime diagnostics, particularly in the area of PInvoke/Interop. Adam Nathan has written some great information about that: Http://blogs.gotdotnet.com/anathan/categoryview.aspx/Debugging

Category: Debugging — Tags: , , , , — admin @ 10:54 am

Command: !EHInfo

!EHInfo (<MethodDesc address> | <Code address>)

!EHInfo shows the exception handling blocks in a jitted method. For each handler, it shows the type, including code addresses and offsets for the clause block and the handler block. For a TYPED handler, this would be the “try” and “catch” blocks respectively.

Sample output:

!clrstack
OS Thread Id: 0xb10 (2832)
ESP       EIP
0012f3c0 00ff019a IEnumerableTest.Utils.EnumToArray[[IEnumerableTest.Program+RoleType, LinqSelectTest]]()
0012f424 00ff00b9 IEnumerableTest.Program.Main(System.String[])
0012f69c 79e71b4c [GCFrame: 0012f69c] 

!ip2md 00ff00b9
MethodDesc: 00933000

!ehinfo 00933000
MethodDesc: 00933000
Method Name: IEnumerableTest.Program.Main(System.String[])
Class: 00931338
MethodTable: 0093301c
mdToken: 06000002
Module: 00932c5c
IsJitted: yes
CodeAddr: 00ff0070

EHHandler 0: TYPED
Clause: [033bbd2b, 033bbd3c] [8b, 9c]
Handler: [033bbd3c, 033bbd50] [9c, b0]

EHHandler 1: FINALLY
Clause: [033bbd83, 033bbda3] [e3, 103]
Handler: [033bbda3, 033bbdc5] [103, 125]

EHHandler 2: TYPED
Clause: [033bbd7a, 033bbdc5] [da, 125]
Handler: [033bbdc5, 033bbdd6] [125, 136]
Category: Debugging — Tags: , , , , — admin @ 10:40 am

SOS Command: GCInfo

!GCInfo (<MethodDesc address> | <Code address>)

!GCInfo is especially useful for CLR Devs who are trying to determine if there is a bug in the JIT Compiler. It parses the GCEncoding for a method, which is a compressed stream of data indicating when registers or stack locations contain managed objects. It is important to keep track of this information, because if a garbage collection occurs, the collector needs to know where roots are so it can update them with new object pointer values.

Here is sample output where you can see the change in register state. Normally you would print this output out and read it alongside a disassembly of the method. For example, the notation “reg EDI becoming live” at offset 0×11 of the method might correspond to a “mov edi,ecx” statement.

!ip2md 00ff019a
MethodDesc: 00933198

!gcinfo 00933198
entry point 00ff0160
Normal JIT generated code
GC info 131be56000931b7c
Method info block:
    method      size   = 00E3
    prolog      size   = 26
    epilog      size   =  8
    epilog     count   =  1
    epilog      end    = yes
    callee-saved regs  = EDI ESI EBX EBP
    ebp frame          = yes
    fully interruptible= yes
    double align       = no
    arguments size     =  0 DWORDs
    stack frame size   = 20 DWORDs
    untracked count    =  1
    var ptr tab count  =  8
    security check obj = yes
    exception handlers = yes
    edit & continue    = yes
    epilog        at   00DB
    argTabOffset = 1b
81 63 AD D4 B8 |
AA 94 F2 C0 C2 |
45 08 1B       | 

Pointer table:
10             |             [EBP-10H] an untracked  local
3C 2D 81 36    | 002D..00E3  [EBP-3CH] a  pointer
40 05 81 31    | 0032..00E3  [EBP-40H] a  pointer
48 15 0B       | 0047..0052  [EBP-48H] a  pointer
4C 14 28       | 005B..0083  [EBP-4CH] a  pointer
50 0D 1B       | 0068..0083  [EBP-50H] a  pointer
5C 23 1C       | 008B..00A7  [EBP-5CH] a  pointer
54 2C 08       | 00B7..00BF  [EBP-54H] a  pointer
58 18 09       | 00CF..00D8  [EBP-58H] a  pointer
F7 44          | 0044        reg EAX becoming live
F0 4C          | 0050        reg ECX becoming live
02             | 0052        reg EAX becoming dead
46             | 0058        reg EAX becoming live
08             | 0058        reg ECX becoming dead
F2 01          | 0071        reg EAX becoming dead
F1 47          | 0088        reg EAX becoming live
F1 54          | 009C        reg EDX becoming live
4B             | 009F        reg ECX becoming live
05             | 00A4        reg EAX becoming dead
08             | 00A4        reg ECX becoming dead
10             | 00A4        reg EDX becoming dead
4B             | 00A7        reg ECX becoming live
0D             | 00AC        reg ECX becoming dead
4B             | 00AF        reg ECX becoming live
45             | 00B4        reg EAX becoming live
08             | 00B4        reg ECX becoming dead
56             | 00BA        reg EDX becoming live
F0 12          | 00C4        reg EDX becoming dead
4A             | 00C6        reg ECX becoming live
0E             | 00CC        reg ECX becoming dead
F0 04          | 00D8        reg EAX becoming dead
43             | 00DB        reg EAX becoming live
F0 00          | 00E3        reg EAX becoming dead
FF             |           |

This function is important for CLR Devs, but very difficult for anyone else to make sense of it. You would usually come to use it if you suspect a gc heap corruption bug caused by invalid GCEncoding for a particular method.

Category: Debugging — Tags: , , , , — admin @ 10:26 am

SOS Command: EEStack

!EEStack [-short] [-EE]

This command runs !DumpStack on all threads in the process. The -EE option is passed directly to !DumpStack. The -short option tries to narrow down the output to “interesting” threads only, which is defined by

  1. The thread has taken a lock.
  2. The thread has been “hijacked” in order to allow a garbage collection.
  3. The thread is currently in managed code.

See the documentation for !DumpStack for more info.

Example output:

!EEStack -short
---------------------------------------------
Thread 876
---------------------------------------------
Thread 2268
« Newer PostsOlder Posts »

©2009 NetInverse. All rights reserved. Powered by WordPress